A multi-view parallax correction method and device, augmented reality glasses and a medium

Abstract

This application discloses a multi-view misalignment correction method, device, augmented reality glasses, and medium, relating to the field of augmented reality technology. This solution models the changes in extrinsic parameters from multiple views as online estimable perturbation parameters by setting an online drift state, allowing users to continuously correct misalignments during daily use. Simultaneously, by extracting shared features from a reference viewpoint and performing geometric mapping and residual compensation on these shared features based on the online drift state, compensated features for other views are obtained. This effectively avoids running a feature extraction network independently for each viewpoint, significantly reducing computational overhead and power consumption. By matching the shared features with the compensated features and constructing a reprojection residual, the drift state is updated using this residual, achieving collaborative optimization between the correction process and feature calculation. This effectively reduces the computational cost of correction and ensures the geometric consistency of the augmented reality glasses during long-term operation.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a multi-view misalignment correction method, device, augmented reality glasses and medium. Background Technology

[0002] Augmented Reality (AR) glasses rely on multi-view cameras and inertial sensors to achieve pose tracking and virtual-real fusion. This requires precise and stable intrinsic and extrinsic parameters across different viewpoints, as well as sufficiently low latency for feature extraction. Currently, devices are typically manufactured using offline calibration, which permanently stores the camera's intrinsic and extrinsic parameters. For image feature computation, mainstream solutions generally involve running a deep convolutional network independently for each camera to extract its own visual features, which are then used for subsequent pose optimization and scene perception tasks.

[0003] However, factory-set parameters cannot handle the micro-deformations caused by impact, temperature changes, and stress during daily wear. Once drift occurs, it leads to multi-view misalignment and decreased accuracy in virtual-real fusion, requiring recalibration at the factory, resulting in unsatisfactory user experience and costs. Continuously optimizing parameters using online bundle adjustment methods is computationally complex and power-intensive, making it difficult to run in real-time on resource-constrained AR glasses. More significantly, independent convolutional feature extraction for each viewpoint causes severe redundant computation; processing latency and system power consumption increase linearly with the number of cameras, significantly shortening device battery life.

[0004] Given the above, how to solve the current problems of AR glasses' inability to continuously and automatically correct misalignment and high power consumption of multi-path feature extraction is an urgent issue for technicians in this field. Summary of the Invention

[0005] The purpose of this application is to provide a multi-view misalignment correction method, device, augmented reality glasses and medium to solve the problems that current AR glasses have difficulty in continuously and automatically correcting misalignment and have high power consumption for multi-channel feature extraction, which are urgent problems to be solved by those skilled in the art.

[0006] To address the aforementioned technical problems, this application provides a multi-view misalignment correction method for use in augmented reality glasses; the method includes:

[0007] Acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters;

[0008] Based on the backbone feature extraction network and the corresponding image frames, the shared features of the reference viewpoint are obtained;

[0009] Based on the image frames and online drift states of each non-reference viewpoint, geometric mapping and residual compensation are performed on the shared features to obtain the compensated features of each viewpoint.

[0010] The shared features are matched with each compensated feature, and the reprojection residuals are constructed based on the matching results.

[0011] The online drift states are updated based on the reprojection residuals to correct multi-view misalignment.

[0012] On the one hand, image frames from various viewpoints are acquired, a reference viewpoint is selected, and the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters is set, including:

[0013] Obtain the intrinsic parameters, distortion parameters, and initial extrinsic parameters of each camera;

[0014] Set the main camera index corresponding to the reference viewpoint, set the online drift state and covariance of each viewpoint, and set the correction trigger threshold; wherein, the correction trigger threshold includes at least the reprojection error threshold and the confidence threshold;

[0015] Acquire image frames from each camera and angular velocity and linear acceleration sequences from the inertial measurement unit, and perform data timestamp alignment;

[0016] Distortion correction and brightness normalization are performed on the image frames acquired by each camera, and usability estimation is performed on each processed image frame to obtain the frame quality score of each image frame.

[0017] Determine whether the frame quality score of the reference viewpoint is lower than the threshold;

[0018] If so, switch the reference viewpoint or reduce the correction frequency.

[0019] On the other hand, based on the image frames and online drift states of each non-reference viewpoint, geometric mapping and residual compensation are performed on the shared features to obtain the compensated features for each viewpoint, including:

[0020] Based on the initial extrinsic parameters and each online drift state, determine the current extrinsic parameters corresponding to each non-reference viewpoint;

[0021] Based on each current extrinsic parameter, construct a geometric mapping from the shared features to the corresponding non-reference viewpoint to obtain the reused features corresponding to each non-reference viewpoint;

[0022] Lightweight residual features are calculated based on the lightweight residual compensation branch and image frames from each non-reference viewpoint. These lightweight residual features are then fused with the corresponding multiplexing features to obtain the compensated features for each viewpoint.

[0023] On the other hand, the shared features are matched with each compensated feature, and the reprojection residuals are constructed based on the matching results, including:

[0024] Key points and descriptors are extracted from the shared feature space formed by the compensated features of all views.

[0025] Based on key points and descriptors, feature matching is performed between the reference viewpoint and each non-reference viewpoint to generate a set of matching point pairs containing matching confidence.

[0026] Perform outlier culling on the set of matching point pairs to generate the set of inliers;

[0027] For each non-reference viewpoint, a reprojection residual is constructed based on the corresponding interior point matching pairs in the interior point set.

[0028] On the other hand, before updating each online drift state based on the reprojection residual, after matching the shared features with each compensated feature and constructing the reprojection residual based on the matching results, the process also includes:

[0029] Determine whether there exists an interior point set where the number of interior point matching pairs corresponding to non-reference viewpoints is lower than a preset number, or the confidence level of the corresponding interior point matching pairs is lower than a confidence level threshold.

[0030] If so, skip updating the online drift status of the corresponding non-reference viewpoint;

[0031] If not, calculate the mean square error of the reprojection residuals corresponding to each non-reference viewpoint;

[0032] Determine whether each mean square error is greater than the reprojection error threshold;

[0033] If so, update the online drift status of the corresponding non-reference viewpoint.

[0034] On the other hand, the online drift states are updated based on the reprojection residuals, including:

[0035] The reprojection residuals are linearized based on the online drift state to obtain the Jacobian matrix;

[0036] The increment of the online drift state is solved based on the Jacobian matrix and the weight matrix constructed from the matching confidence.

[0037] Incremental updates to the online drift state are performed based on incremental execution to obtain the updated online drift state.

[0038] On the other hand, after updating each online drift state based on the reprojection residual, the following is also included:

[0039] The current extrinsic parameters of each non-reference viewpoint are corrected based on the updated online drift state, and the corrected current extrinsic parameters are passed to the downstream module.

[0040] In this process, the rendering side applies temporal filtering to the corrected current extrinsic parameters or the updated online drift state.

[0041] To address the aforementioned technical problems, this application also provides a multi-view misalignment correction device for use in augmented reality glasses; the device includes:

[0042] The initialization module is used to acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters.

[0043] The feature extraction module is used to obtain shared features from the reference viewpoint based on the backbone feature extraction network and the corresponding image frames.

[0044] The mapping compensation module is used to perform geometric mapping and residual compensation on shared features based on image frames and online drift states from each non-reference viewpoint, so as to obtain the compensated features from each viewpoint.

[0045] The matching module is used to match the shared features with each compensated feature and construct the reprojection residuals based on the matching results.

[0046] The update module is used to update the online drift states based on the reprojection residuals in order to correct multi-view misalignment.

[0047] To address the aforementioned technical problems, this application also provides augmented reality glasses, comprising:

[0048] Memory, used to store computer programs;

[0049] A processor is used to implement the steps of the above-described multi-view misalignment correction method when executing a computer program.

[0050] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned multi-view misalignment correction method.

[0051] The multi-view misalignment correction method provided in this application models the changes in extrinsic parameters of multiple views as perturbation parameters that can be estimated online by setting an online drift state, allowing users to continuously correct misalignment during daily use. Simultaneously, by extracting shared features from the reference viewpoint and performing geometric mapping and residual compensation on the shared features based on the online drift state, compensated features for other views are obtained. This effectively avoids running the feature extraction network independently for each viewpoint, thus significantly reducing computational overhead and power consumption. By matching the shared features with the compensated features and constructing a reprojection residual, the drift state update is driven by this residual, achieving collaborative optimization between the correction process and feature calculation. This effectively reduces the computational cost of correction and ensures the geometric consistency of augmented reality glasses across multiple views during long-term operation.

[0052] In addition, this application also provides a multi-view misalignment correction device, augmented reality glasses and medium, with the same effect as above. Attached Figure Description

[0053] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 A flowchart illustrating a multi-view misalignment correction method provided in this application embodiment;

[0055] Figure 2 A schematic diagram of a multi-view misalignment correction device provided in an embodiment of this application;

[0056] Figure 3 This is a structural diagram of an augmented reality glasses provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0058] The core of this application is to provide a multi-view misalignment correction method, device, augmented reality glasses and medium to solve the problems that current AR glasses have difficulty in continuously and automatically correcting misalignment and have high power consumption for multi-channel feature extraction, which are urgent problems to be solved by those skilled in the art.

[0059] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] In practical use of AR glasses, multi-view misalignment issues can be caused by various factors, primarily including: spatial misalignment (extrinsic parameter drift), due to tolerances in device assembly, or under the influence of drop impacts, thermal expansion and contraction caused by environmental temperature changes, and deformation and stress from long-term wear of the frames, the relative pose (rotation matrix R and translation vector t) between each camera and the depth module may undergo slight changes. Optical / imaging model errors, such as lens optical distortion parameters drifting with changes in temperature and focusing distance, or shifts in the effective imaging model due to adjustments in the image signal processor (ISP) strategy. Temporal misalignment (cross-sensor data asynchrony), where the exposure times of multiple cameras are not fully synchronized, and the rolling shutter effect and IMU data timestamp drift can all lead to errors when matching data across different viewpoints. Platform constraints resulting in limited computing power and power consumption budgets, limited by battery capacity, heat dissipation capabilities, and device weight, prevent AR glasses from continuously running computationally intensive large-scale multi-view joint optimization algorithms and from simultaneously supporting complex convolutional neural network feature extraction from multiple images.

[0061] Currently, mainstream technologies for solving multi-view calibration and feature extraction have the following limitations: Devices typically use offline calibration at the factory, storing camera intrinsic and extrinsic parameters. For image feature calculation, mainstream solutions generally involve each camera independently running a deep convolutional network to extract its own visual features, which are then used for subsequent pose optimization and scene perception tasks. However, factory-fixed parameters cannot cope with micro-deformations caused by impact, temperature changes, and stress during daily wear. Drifting leads to multi-view misalignment and decreased accuracy in virtual-real fusion, requiring recalibration at the factory, resulting in unsatisfactory experience and costs. Continuously optimizing parameters using online bundle adjustment is computationally complex and power-consuming, making it difficult to run in real-time on resource-constrained AR glasses. More significantly, independent convolutional feature extraction for each viewpoint causes severe redundant computation; processing latency and system power consumption increase linearly with the number of cameras, significantly shortening device battery life. Therefore, to address these issues, this application provides a multi-view misalignment correction method. It is understood that the method provided in this application is applied to augmented reality glasses. Augmented reality glasses include at least an inertial measurement unit and multiple cameras.

[0062] Figure 1 This is a flowchart illustrating a multi-view misalignment correction method provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0063] S10: Acquire image frames from each viewpoint, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters.

[0064] First, image frames are acquired from various viewpoints, meaning image frame acquisition is performed using each camera. Simultaneously, one viewpoint is selected from the multiple cameras as a reference viewpoint, with the rest serving as non-reference viewpoints. For each non-reference viewpoint, its drift relative to the initial extrinsic parameters is modeled as a small perturbation on a special Euclidean group, parameterized using online drift states. This modeling method allows subsequent estimations to focus only on the minute perturbation increments online, rather than recalibrating the entire extrinsic parameter matrix from zero, effectively reducing the degrees of freedom and convergence difficulty in parameter estimation. In this embodiment, the process of setting the online drift states is not limited and depends on the specific implementation.

[0065] S11: Based on the backbone feature extraction network and the corresponding image frames, obtain the shared features of the reference viewpoint.

[0066] Subsequently, a pre-constructed backbone feature extraction network is used, with the image frame corresponding to the reference viewpoint as input, to perform complete backbone extraction, and this is used as anchor point features for cross-viewpoint reuse to reduce redundant calculations for other viewpoints. The specific formula is as follows:

[0067] ;

[0068] in, For reference viewpoint image, The backbone feature extraction network, For network parameters, These are the extracted shared features.

[0069] It should be noted that this embodiment does not limit the specific type of backbone feature extraction network, but depends on the specific implementation.

[0070] S12: Based on the image frames and online drift states of each non-reference viewpoint, perform geometric mapping and residual compensation on the shared features to obtain the compensated features of each viewpoint.

[0071] Furthermore, utilizing the current geometric relationships represented by the online drift state, the shared features extracted from the reference viewpoint are spatially mapped to the image planes of each non-reference viewpoint, enabling feature-level correspondence for the same scene point across multiple views. However, due to potential approximation errors in the geometric mapping model, and factors such as occlusion and viewpoint differences causing some information loss after mapping, a residual compensation mechanism is introduced to perform lightweight processing on the image of the non-reference viewpoint to supplement the missing information in the mapped features. Through the combination of mapping and compensation, complete compensated features for each viewpoint are obtained without needing to run a heavy feature extraction network independently for each viewpoint, thereby significantly reducing the overall computational overhead. This embodiment does not limit the specific process of geometric mapping and residual compensation for shared features; it depends on the specific implementation.

[0072] S13: Match the shared features with each compensated feature, and construct the reprojection residuals based on the matching results.

[0073] After obtaining the compensated features for each viewpoint, cross-view matching is performed using shared features and the compensated features for each non-reference viewpoint to establish the correspondence between pixels in the reference viewpoint and pixels in each non-reference viewpoint. It is important to note that since the compensated features have already achieved spatial alignment through geometric mapping and residual compensation, the matching process can be completed in the shared feature space. Based on the matching results, a reprojection residual is constructed. This residual quantifies the pixel deviation when the 3D points of the reference viewpoint are projected onto the non-reference viewpoints via extrinsic parameters under the current online drift state, providing an optimization target for subsequent drift state updates.

[0074] S14: Update each online drift state based on the reprojection residual to correct multi-view misalignment.

[0075] Finally, since the reprojection residual directly reflects the deviation between the current extrinsic parameter estimate and the true geometric relationship, minimizing this residual allows for the reverse calculation of the correction amount for the drift state, thereby converging the extrinsic parameters of the non-reference viewpoint to the true value and ultimately achieving the correction of multi-view misalignment. This embodiment does not limit the specific process of multi-view misalignment correction; it depends on the specific implementation.

[0076] In this embodiment, by setting an online drift state, the changes in extrinsic parameters from multiple perspectives are modeled as perturbation parameters that can be estimated online, allowing users to continuously correct misalignments during daily use. Simultaneously, by extracting shared features from the reference perspective and performing geometric mapping and residual compensation on these shared features based on the online drift state, compensated features for other perspectives are obtained. This effectively avoids running the feature extraction network independently for each perspective, significantly reducing computational overhead and power consumption. By matching the shared features with the compensated features and constructing a reprojection residual, the drift state is updated using this residual, achieving coordinated optimization between the correction process and feature calculation. This effectively reduces the computational cost of correction and ensures the geometric consistency of the augmented reality glasses across multiple perspectives during long-term operation.

[0077] Based on the above embodiments, in some embodiments, image frames from each viewpoint are acquired, a reference viewpoint is selected, and the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters is set, including:

[0078] S101: Obtain the intrinsic parameters, distortion parameters, and initial extrinsic parameters of each camera.

[0079] S102: Set the main camera index corresponding to the reference viewpoint, set the online drift state and covariance of each viewpoint, and set the correction trigger threshold; wherein, the correction trigger threshold includes at least the reprojection error threshold and the confidence threshold.

[0080] S103: Acquire image frames from each camera and angular velocity and linear acceleration sequences from the inertial measurement unit, and perform data timestamp alignment.

[0081] S104: Perform distortion correction and brightness normalization on the image frames acquired by each camera, and perform usability estimation on each processed image frame to obtain the frame quality score of each image frame.

[0082] S105: Determine if the frame quality score of the reference viewpoint is below the threshold; if yes, switch the reference viewpoint or reduce the correction frequency. If no, end.

[0083] First, system initialization and parameter loading are performed. Specifically, factory calibration parameters are read to obtain the intrinsic parameters K of each camera. i Distortion parameter D i and initial extrinsic parameter T i0 It should be noted that the internal parameter K... i This is a matrix containing the focal length and principal point coordinates, used to describe the projection relationship from the camera coordinate system to the image coordinate system. Distortion parameter D i These are coefficients describing the radial and tangential distortions of the lens, used to correct geometric distortions in the image. Initial extrinsic parameter T i0 The spatial pose transformation matrix of the non-reference view camera relative to the reference view camera includes a rotation matrix and a translation vector.

[0084] Furthermore, set the main camera index corresponding to the reference viewpoint, for example, setting the index to 0. Simultaneously, set the online drift state and covariance for each viewpoint, specifically setting the online drift state ξ. i Set the covariance P to 0. i Let P0 be the variance. i This characterizes the uncertainty in the online drift state estimation for the i-th viewpoint. Furthermore, a correction trigger threshold needs to be set, including at least the reprojection error threshold τ. r and confidence threshold τ c .

[0085] Subsequently, image frames were acquired from each camera. i The angular velocity sequence ω(t) and linear acceleration sequence a(t) of the inertial measurement unit are obtained, and data timestamp alignment is performed. If necessary, the rolling shutter is simplified and compensated, such as the pose during exposure estimated by IMU interpolation.

[0086] Finally, distortion correction and brightness normalization are performed on the image frames acquired by each camera, and usability estimation is performed on each processed image frame. For example, a frame quality score is obtained by weighted fusion of texture intensity and motion blur indices. Then, it is determined whether the frame quality score of the reference viewpoint is below a threshold; if so, the main viewpoint is considered unusable, and the reference viewpoint is switched or the correction frequency is reduced. If not, the initialization and parameter configuration process ends. In this way, by loading the intrinsic parameters, distortion parameters, and initial extrinsic parameters of each camera, and by time alignment and quality screening of multi-sensor data, reliable initialization and input guarantees are provided for accurate estimation of online drift states, avoiding interference from low-quality image frames in subsequent correction processes.

[0087] Based on the above embodiments, in some embodiments, based on the image frames and online drift states of each non-reference viewpoint, geometric mapping and residual compensation are performed on the shared features to obtain the compensated features for each viewpoint, including:

[0088] S111: Determine the current extrinsic parameters corresponding to each non-reference viewpoint based on the initial extrinsic parameters and each online drift state.

[0089] S112: Construct a geometric mapping from the shared features to the corresponding non-reference viewpoints based on each current extrinsic parameter, so as to obtain the reused features corresponding to each non-reference viewpoint.

[0090] S113: Calculate lightweight residual features based on the lightweight residual compensation branch and image frames from each non-reference viewpoint, and fuse each lightweight residual feature with the corresponding multiplexing feature to obtain the compensated features for each viewpoint.

[0091] To achieve low-power shared feature extraction, this embodiment specifically determines the current extrinsic parameters corresponding to each non-reference viewpoint based on the initial extrinsic parameters and each online drift state, as follows:

[0092] ;

[0093] in, The initial external parameters calibrated for the factory. Let i be the online drift state of the i-th viewpoint relative to the factory extrinsic parameters. This is the corrected extrinsic parameter estimate for the current moment, i.e., the current extrinsic parameters. During initialization, set... =0, and will be continuously corrected through online updates.

[0094] Subsequently, a geometric mapping from the shared features to the corresponding non-reference viewpoints is constructed based on each current extrinsic parameter to obtain the reused features corresponding to each non-reference viewpoint. The specific formula is as follows:

[0095] ;

[0096] in, For the reuse feature of the i-th viewpoint, For the extracted shared features, Warp(⋅) is the geometric projection relationship that maps the reference viewpoint to the i-th viewpoint, constructed based on the current extrinsic parameters of the i-th viewpoint. Warp(⋅) is the feature distortion of the sampling grid constructed based on the current estimated extrinsic parameters and sparse depth / disparity. In specific implementations, it can be implemented using bilinear sampling, which is computationally lightweight.

[0097] Finally, lightweight residual features are calculated based on the lightweight residual compensation branch and image frames from each non-reference viewpoint. These lightweight residual features are then fused with the corresponding multiplexed features to obtain the compensated features for each viewpoint. The specific formula is as follows:

[0098] ;

[0099] in, This refers to the reused features derived from the reference view features and then warped to the i-th view feature. For the low-resolution input of the i-th viewpoint, For lightweight residual compensation branch, These are the compensated features from each perspective.

[0100] In this embodiment, by extracting shared features only once from the reference viewpoint and using geometric mapping and lightweight residual compensation to obtain compensated features from other viewpoints, the heavy feature extraction network is avoided from running independently for each viewpoint, thereby significantly reducing the redundancy and power consumption of multi-view feature computation.

[0101] Based on the above embodiments, in some embodiments, the shared features are matched with each compensated feature, and the reprojection residual is constructed based on the matching results, including:

[0102] S121: Extract key points and descriptors from the shared feature space formed by the compensated features of all views.

[0103] S122: Based on key points and descriptors, perform feature matching between the reference viewpoint and each non-reference viewpoint to generate a set of matching point pairs containing matching confidence.

[0104] S123: Perform outpoint culling on the set of matching point pairs to generate an inpoint set.

[0105] S124: For each non-reference viewpoint, construct the reprojection residual based on the corresponding interior point matching pairs in the interior point set.

[0106] After completing the computation and fusion of lightweight residual features, a shared feature set is obtained, which can be denoted as {F0, F1, ..., F...}. N-1}. F0 represents the anchor point feature of the reference viewpoint, and the others F iThis is the result of fusing reused features and lightweight residual compensation features. To achieve cross-view matching and effective constraint generation, keypoints and descriptors are further extracted from the shared feature space composed of the compensated features from all views. It can be understood that the shared feature space is the unified representation space of the aforementioned shared feature set. The keypoints and descriptors are expressed as follows:

[0107] ;

[0108] ;

[0109] in, As the key point, For descriptors.

[0110] Subsequently, based on key points and descriptors, feature matching is performed between the reference viewpoint and each non-reference viewpoint to determine whether points in the two viewpoints correspond to the same real scene, thereby generating a set of matching point pairs containing matching confidence scores, as shown in the following formula:

[0111] ;

[0112] in, To match a set of point pairs, To match the confidence level, Let K be the coordinates of the k-th matching point in the reference view. Let be the coordinates of the k-th matching point in the i-th viewpoint.

[0113] Furthermore, outlier culling is performed on the set of matching point pairs to generate the set of inliers. In this embodiment, there are no restrictions on the outlier removal method; for example, RANSAC and epipolar consistency filtering can be used in combination. Finally, for each non-reference viewpoint, the reprojection residual is constructed based on the corresponding inlier matching pairs in the inlier set. In this way, by performing cross-view matching and eliminating erroneous matches in a unified shared feature space, high-quality interior point matching pairs are provided for drift state updates, ensuring the reliability of reprojection residuals and the stability of subsequent corrections.

[0114] Furthermore, to ensure the reliability of the correction parameter updates, in some embodiments, before updating each online drift state based on the reprojection residuals, after matching the shared features with each compensated feature and constructing the reprojection residuals based on the matching results, the method further includes:

[0115] S125: Determine whether there exists an interior point set where the number of interior point matching pairs corresponding to a non-reference viewpoint is lower than a preset number, or the confidence level of the corresponding interior point matching pair is lower than the confidence level threshold; if yes, skip updating the online drift state of the corresponding non-reference viewpoint; if no, proceed to step S126.

[0116] S126: Calculate the mean square error of the reprojection residuals corresponding to each non-reference viewpoint.

[0117] S127: Determine if each mean square error is greater than the reprojection error threshold; if yes, update the online drift state of the corresponding non-reference viewpoint. If no, do not trigger an update.

[0118] Specifically, it determines the number of inlier matching pairs in the inlier set that correspond to non-reference viewpoints. If the number is less than a preset limit, or the confidence level of the corresponding inlier matching pair is lower than the confidence threshold τ. c If so, the online drift state update for the corresponding non-reference viewpoint is skipped. It should be noted that this embodiment does not limit the preset number; it depends on the specific implementation.

[0119] If it is confirmed that there are no interior point matching pairs corresponding to non-reference views in the interior point set. The number of matches is below a preset limit, and the confidence level of the corresponding inlier matching pairs is not lower than the confidence threshold τ. c Then calculate the mean square error of the reprojection residuals corresponding to each non-reference viewpoint:

[0120] ;

[0121] in, Let be the mean square error of the reprojection residual for the i-th viewpoint. This is the reprojection residual.

[0122] Finally, determine each mean square error. Is it greater than the reprojection error threshold τ? r If so, the online drift state of the corresponding non-reference viewpoint is updated; otherwise, no update is triggered. It's understandable that the larger the error, the higher the number of iterations or update frequency. Therefore, by using multi-layered threshold judgments based on the number of interior points, matching confidence, and reprojection error, unreliable parameter updates are avoided when the matching quality is insufficient, thereby reducing unnecessary computational overhead while ensuring correction accuracy.

[0123] Based on the above embodiments, in some embodiments, the online drift states are updated according to the reprojection residuals, including:

[0124] S131: Linearize the reprojection residuals based on the online drift state to obtain the Jacobian matrix.

[0125] S132: Solve for the increment of the online drift state based on the Jacobian matrix and the weight matrix constructed from the matching confidence.

[0126] S133: Perform incremental updates to the online drift state based on incremental operations to obtain the updated online drift state.

[0127] After obtaining the reprojection residuals, in order to solve for the drift increment and update the extrinsic parameter corrections, this embodiment specifically linearizes the reprojection residuals based on the online drift state to obtain the Jacobian matrix. The Jacobian matrix J is specifically obtained from the first derivative of the reprojection residuals with respect to the online drift state. Simultaneously, a weight matrix W is constructed based on matching confidence, image quality, residual size, and geometric consistency to reduce the impact of abnormal matches on the update results. Then, based on the Jacobian matrix J and the weight matrix W constructed from the matching confidence, the increment of the online drift state is solved, and an incremental update of the online drift state is performed based on the increment, as follows:

[0128] ;

[0129] in, For increments, This is for incremental updates on the special Euclidean group SE(3). Therefore, by linearizing the reprojected residuals and using matched confidence weighting to solve for the drift increment, a stable update of the drift state is achieved through iterative recursion, avoiding the high computational complexity and divergence risk of directly solving large-scale nonlinear optimization problems. It should also be noted that the covariance P can be updated using the EKF form. i This is used for subsequent adaptive gating.

[0130] Based on the above embodiments, in some embodiments, after updating each online drift state according to the reprojection residual, the method further includes:

[0131] S141: Correct the current extrinsic parameters of each non-reference viewpoint based on the updated online drift state, and pass the corrected current extrinsic parameters to the downstream module.

[0132] In this process, the rendering side applies temporal filtering to the corrected current extrinsic parameters or the updated online drift state.

[0133] In order to distribute the correction results to the business and rendering links, this embodiment specifically corrects the current extrinsic parameters of each non-reference viewpoint based on the updated online drift state:

[0134] ;

[0135] in, This provides the corrected current extrinsic parameters for each non-reference viewpoint. Simultaneously, the corrected current extrinsic parameters are passed to downstream modules, such as stereo depth, VIO / SLAM, multi-view fusion, and virtual-real registration. The rendering side applies temporal filtering to the corrected current extrinsic parameters or the updated online drift state to avoid jumps.

[0136] It should also be noted that during system operation, an on-demand update and keyframe triggering mechanism can be adopted, performing iterative correction only at keyframes where reprojection errors exceed limits or where observation information is abundant, while maintaining only lightweight drift tracking during other periods to reduce computation frequency. Simultaneously, fixed-point quantization and operator fusion are implemented on the shared feature extraction network, along with a memory reuse strategy, to further reduce inference energy consumption. Furthermore, the extracted shared features simultaneously serve online correction and downstream business perception tasks, achieving module reuse, avoiding repeated calls to the feature extraction network, and controlling system power consumption as a whole.

[0137] The multi-view misalignment correction method has been described in detail in the above embodiments. This application also provides embodiments of the multi-view misalignment correction device.

[0138] Figure 2 This is a schematic diagram of a multi-view misalignment correction device provided in an embodiment of this application. The device is applied to augmented reality glasses; such as... Figure 2 As shown, the device includes:

[0139] Initialization module 10 is used to acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial external parameters.

[0140] The feature extraction module 11 is used to obtain shared features from the reference viewpoint based on the backbone feature extraction network and the corresponding image frame.

[0141] The mapping compensation module 12 is used to perform geometric mapping and residual compensation on the shared features based on the image frames and online drift states of each non-reference viewpoint, so as to obtain the compensated features of each viewpoint.

[0142] Matching module 13 is used to match the shared features with each compensated feature and construct the reprojection residual based on the matching results;

[0143] The update module 14 is used to update each online drift state based on the reprojection residual in order to correct multi-view misalignment.

[0144] In some embodiments, the initialization module 10 includes:

[0145] The acquisition submodule is used to acquire the intrinsic parameters, distortion parameters, and initial extrinsic parameters of each camera;

[0146] The settings submodule is used to set the main camera index corresponding to the reference viewpoint, set the online drift status and covariance of each viewpoint, and set the correction trigger threshold; wherein, the correction trigger threshold includes at least the reprojection error threshold and the confidence threshold.

[0147] The acquisition submodule is used to acquire image frames from each camera and angular velocity and linear acceleration sequences from the inertial measurement unit, and to perform data timestamp alignment.

[0148] The first processing submodule is used to perform distortion correction and brightness normalization processing on the image frames acquired by each camera, and to perform usability estimation on each processed image frame to obtain the frame quality score of each image frame.

[0149] The first judgment submodule is used to determine whether the frame quality score of the reference view is lower than the threshold; if so, the reference view is switched or the correction frequency is reduced.

[0150] In some embodiments, the mapping compensation module 12 includes:

[0151] The determination submodule is used to determine the current extrinsic parameters corresponding to each non-reference viewpoint based on the initial extrinsic parameters and each online drift state;

[0152] The first calculation submodule is used to construct a geometric mapping from the shared features to the corresponding non-reference viewpoint based on each current extrinsic parameter, so as to obtain the reused features corresponding to each non-reference viewpoint.

[0153] The second calculation submodule is used to calculate lightweight residual features based on the lightweight residual compensation branch and the image frames of each non-reference viewpoint, and to fuse each lightweight residual feature with the corresponding multiplexing feature to obtain the compensated features of each viewpoint.

[0154] In some embodiments, the matching module 13 includes:

[0155] The extraction submodule is used to extract key points and descriptors from the shared feature space composed of compensated features from all views.

[0156] The matching submodule is used to perform feature matching between the reference view and each non-reference view based on key points and descriptors, so as to generate a set of matching point pairs containing matching confidence.

[0157] The second processing submodule is used to perform outlier removal on the set of matching point pairs to generate an inlier set;

[0158] The third calculation submodule is used to construct the reprojection residual based on the corresponding interior point matching pairs in the interior point set for each non-reference viewpoint.

[0159] In some embodiments, it also includes:

[0160] The second judgment submodule is used to determine whether there is a set of inlier sets where the number of inlier matching pairs corresponding to non-reference views is lower than a preset number, or the confidence level of the corresponding inlier matching pairs is lower than the confidence level threshold; if so, the online drift state of the corresponding non-reference view is skipped and updated; if not, the mean square error of the reprojection residuals corresponding to each non-reference view is calculated.

[0161] The third judgment submodule is used to determine whether each mean square error is greater than the reprojection error threshold; if so, the online drift status of the corresponding non-reference viewpoint is updated.

[0162] In some embodiments, the updating module 14 includes:

[0163] The third processing submodule is used to linearize the reprojection residuals based on the online drift state to obtain the Jacobian matrix;

[0164] The fourth calculation submodule is used to solve for the increment of the online drift state based on the Jacobian matrix and the weight matrix constructed from the matching confidence.

[0165] The update submodule is used to perform incremental updates to the online drift state based on incremental changes, so as to obtain the updated online drift state.

[0166] In some embodiments, after updating each online drift state according to the reprojection residual, the method further includes:

[0167] The correction module is used to correct the current extrinsic parameters of each non-reference viewpoint based on the updated online drift state, and then pass the corrected current extrinsic parameters to the downstream module.

[0168] In this process, the rendering side applies temporal filtering to the corrected current extrinsic parameters or the updated online drift state.

[0169] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0170] Figure 3 This is a structural diagram of augmented reality glasses provided in an embodiment of this application. Figure 3 As shown, augmented reality glasses include:

[0171] Memory 20 is used to store computer programs;

[0172] The processor 21 is used to implement the steps of the multi-view misalignment correction method mentioned in the above embodiments when executing a computer program.

[0173] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0174] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the multi-view misalignment correction method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the multi-view misalignment correction method. The augmented reality glasses also include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0175] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on augmented reality glasses and may include more or fewer components than illustrated.

[0176] Finally, this application also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps described in the above method embodiments.

[0177] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The foregoing has provided a detailed description of a multi-view misalignment correction method, apparatus, augmented reality glasses, and medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

[0179] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A multi-view misalignment correction method, characterized in that, Applied to augmented reality glasses; the method includes: Acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters; Based on the backbone feature extraction network and the corresponding image frames, the shared features of the reference viewpoint are obtained; Based on the image frames of each non-reference viewpoint and the online drift state, geometric mapping and residual compensation are performed on the shared features to obtain the compensated features of each viewpoint. The shared features are matched with each of the compensated features, and the reprojection residuals are constructed based on the matching results; The online drift states are updated based on the reprojection residuals to correct multi-view misalignment.

2. The multi-view misalignment correction method according to claim 1, characterized in that, Acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters, including: Obtain the intrinsic parameters, distortion parameters, and initial extrinsic parameters of each camera; Set the main camera index corresponding to the reference viewpoint, set the online drift state and covariance of each viewpoint, and set the correction trigger threshold; wherein, the correction trigger threshold includes at least a reprojection error threshold and a confidence threshold; Acquire image frames from each camera and angular velocity and linear acceleration sequences from the inertial measurement unit, and perform data timestamp alignment; Distortion correction and brightness normalization are performed on the image frames acquired by each camera, and usability estimation is performed on each processed image frame to obtain the frame quality score of each image frame. Determine whether the frame quality score of the reference viewpoint is lower than a threshold; If so, switch the reference viewpoint or reduce the correction frequency.

3. The multi-view misalignment correction method according to claim 2, characterized in that, Based on the image frames from each of the non-reference viewpoints and the online drift state, geometric mapping and residual compensation are performed on the shared features to obtain the compensated features for each viewpoint, including: Based on the initial extrinsic parameters and each of the online drift states, determine the current extrinsic parameters corresponding to each of the non-reference viewpoints; Based on each of the current extrinsic parameters, construct the geometric mapping from the shared feature to the corresponding non-reference viewpoint to obtain the reused feature corresponding to each non-reference viewpoint; Lightweight residual features are calculated based on the lightweight residual compensation branch and the image frames of each non-reference viewpoint, and each lightweight residual feature is fused with the corresponding multiplexing feature to obtain the compensated features of each viewpoint.

4. The multi-view misalignment correction method according to claim 2, characterized in that, Matching the shared features with each of the compensated features, and constructing reprojection residuals based on the matching results, includes: Key points and descriptors are extracted from the shared feature space formed by the compensated features from all perspectives. Based on the key points and the descriptors, feature matching is performed between the reference viewpoint and each of the non-reference viewpoints to generate a set of matching point pairs containing matching confidence. Outside point culling is performed on the set of matching point pairs to generate an inside point set; For each of the aforementioned non-reference viewpoints, the reprojection residual is constructed based on the corresponding interior point matching pairs in the interior point set.

5. The multi-view misalignment correction method according to claim 4, characterized in that, Before updating each of the online drift states based on the reprojection residuals, after matching the shared features with each of the compensated features and constructing the reprojection residuals based on the matching results, the method further includes: Determine whether there exists an interior point set where the number of interior point matching pairs corresponding to the non-reference viewpoint is lower than a preset number, or the confidence level of the corresponding interior point matching pair is lower than the confidence level threshold. If so, then skip updating the online drift state corresponding to the non-reference viewpoint; If not, calculate the mean square error of the reprojection residuals corresponding to each of the non-reference viewpoints; Determine whether each of the mean square errors is greater than the reprojection error threshold; If so, then update the online drift state corresponding to the non-reference viewpoint.

6. The multi-view misalignment correction method according to claim 5, characterized in that, Updating each of the online drift states based on the reprojection residuals includes: The reprojection residual is linearized based on the online drift state to obtain the Jacobian matrix; Based on the Jacobian matrix and the weight matrix constructed from the matching confidence, the increment of the online drift state is solved; Based on the increment, perform an incremental update on the online drift state to obtain the updated online drift state.

7. The multi-view misalignment correction method according to claim 6, characterized in that, After updating each of the online drift states according to the reprojection residuals, the method further includes: The current extrinsic parameters of each non-reference viewpoint are corrected based on the updated online drift state, and the corrected current extrinsic parameters are passed to the downstream module. In this process, the rendering side applies temporal filtering to the corrected current extrinsic parameters or the updated online drift state.

8. A multi-view misalignment correction device, characterized in that, Applied to augmented reality glasses; the device includes: The initialization module is used to acquire image frames from various viewpoints, select a reference viewpoint, and set the online drift state of each non-reference viewpoint relative to the initial extrinsic parameters. The feature extraction module is used to obtain the shared features of the reference viewpoint based on the backbone feature extraction network and the corresponding image frame; The mapping compensation module is used to perform geometric mapping and residual compensation on the shared features based on the image frames of each non-reference viewpoint and the online drift state, so as to obtain the compensated features of each viewpoint. The matching module is used to match the shared features with each of the compensated features, and to construct the reprojection residuals based on the matching results; An update module is used to update each of the online drift states based on the reprojection residuals to correct multi-view misalignment.

9. An augmented reality glasses, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multi-view misalignment correction method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the multi-view misalignment correction method as described in any one of claims 1 to 7.

Images