A human eye camera alignment method for active alignment of an XR device

Abstract

The present application relates to the technical field of optical equipment adjustment, and more particularly to a human eye camera positioning method for active alignment of an XR device, which comprises: acquiring a tilt ellipticity index to evaluate spot geometric distortion of a ROI region; using the tilt ellipticity index to dynamically determine the number of Gaussian components of GMM fitting; obtaining a gradient factor of a pixel point in the ROI region based on the gradient amplitude of the pixel point; for each Gaussian component, obtaining a center distance based on the distance between the Gaussian component and the ideal center of the image, calculating the target weight of each Gaussian component; weighting the Gaussian component center based on the target weight to obtain an optical center; inputting the tilt ellipticity index and the optical center into a preset angle mapping model to back-propagate to obtain an optical tilt angle of the human eye camera, so as to obtain a monocular positioning instruction to correct the position and posture of the human eye camera, thereby effectively improving the positioning accuracy of the human eye camera.

Description

Technical Field

[0001] This invention relates to the field of optical device adjustment technology. More specifically, this invention relates to a method for adjusting the human eye camera in an active alignment device for XR devices. Background Technology

[0002] For XR devices, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) devices, the assembly quality directly determines the user's immersive visual experience and safety. Active alignment (AA) is a crucial step in ensuring the accurate relative positioning of the optical engine with optical components such as lenses or waveguides.

[0003] In the AA (Assembly-Action) process, the human-eye camera is used to capture test images projected by the optomechanism to provide feedback for adjusting its position. Its assembly quality directly impacts the user's visual experience. To ensure accuracy, the human-eye camera must be calibrated beforehand to ensure its optical center aligns with the optical center of the waveguide sheet. Currently, the industry typically calibrates the human-eye camera's position through external alignment. However, due to manufacturing tolerances, the physical and optical centers of the camera often deviate, potentially leading to inaccurate calibration results. This reduces the final optomechanism alignment accuracy, affecting the optical performance and yield of XR equipment.

[0004] Existing technologies typically employ laser spot subpixel localization algorithms based on Gaussian mixture models (GMMs), which determine the subpixel center by probabilistically fitting the energy distribution of the spot. However, energy distribution-based GMM algorithms have limitations. This algorithm only extracts positional information and struggles to handle the two-dimensional geometric distortions of the spot caused by camera attitude deviations due to optical tilt, such as ellipticization and principal axis tilt. Consequently, it is ultimately impossible to achieve high-precision simultaneous correction of both position and attitude. Summary of the Invention

[0005] To address the limitations of the aforementioned energy distribution-based GMM algorithm, which only extracts position information and struggles to handle the two-dimensional geometric distortions of the light spot caused by camera attitude deviations due to optical tilt, such as ellipticization and principal axis tilt, thus failing to achieve simultaneous high-precision correction of both position and attitude, this invention proposes a method for aligning the human eye camera in an XR device's active alignment device. This method includes the following steps:

[0006] The Region of Interest (ROI) is acquired from the laser spot image captured by the human eye camera. Based on the ratio of eigenvalues ​​of the covariance matrix of pixel intensities within the ROI, a tilt ellipticity index is obtained to evaluate the geometric distortion of the laser spot within the ROI. This tilt ellipticity index is then used to dynamically determine the number of Gaussian components fitted to the Gaussian Mixture Model (GMM). , The gradient factor of a pixel is obtained based on the gradient magnitude of the pixel in the ROI region, and the gradient factor is negatively correlated with the gradient magnitude in the ROI region. For each Gaussian component, the center distance is obtained based on the distance between the Gaussian component and the ideal center of the image. The target weight of each Gaussian component is calculated. The target weight is negatively correlated with the center distance and positively correlated with the mean gradient factor in the ROI region. The optical center is obtained by weighting the center of the Gaussian component based on the target weight. The optical tilt angle of the human eye camera is obtained by inputting the tilt ellipticity index and the optical center into a preset angle mapping model. Based on the deviation between the optical center and the ideal center of the image, and the optical tilt angle, a monocular adjustment command is obtained to correct the position and orientation of the human eye camera.

[0007] This invention extracts position information using the Gaussian Mixture Model (GMM) algorithm, improving the accuracy of position and attitude adjustment for human-eye cameras. During the adjustment process, this invention addresses the issue that GMM algorithms based solely on energy distribution ignore two-dimensional geometric distortions caused by optical tilt, such as ellipticization and principal axis tilt, which cannot accurately correct camera attitude. Therefore, this invention calculates the tilt ellipticity index by analyzing the eigenvalues ​​of the ROI region covariance matrix, thereby assessing the degree of spot distortion and dynamically adjusting the number of Gaussian components fitted by the GMM, solving the underfitting or overfitting problem caused by a fixed number of components. Furthermore, this invention introduces gradient factors and distance weights to filter and weight the Gaussian components, effectively suppressing interference from stray light and edge aberrations, thus accurately locating the true optical center. Finally, it combines an angle mapping model to inversely deduce the optical tilt angle, achieving high-precision correction of both positional deviation and attitude tilt of the human-eye camera, improving the positioning accuracy of the human-eye camera in XR devices.

[0008] According to the present invention, a method for adjusting the position of a human eye camera in an active alignment device for an XR device, wherein the method for obtaining the Region of Interest (ROI) in a laser spot image acquired by the human eye camera includes: placing a laser in a product contour block; acquiring an original image of the laser spot projected onto the contour block by the human eye camera; preprocessing the original laser spot image to obtain a laser spot image; extracting the ROI region of the spot in the laser spot image using an adaptive threshold segmentation algorithm; obtaining the initial physical center of the spot using the gray-scale centroid method; and obtaining the center distance of the spot by the difference between the initial physical center and the ideal center of the image.

[0009] This invention effectively removes random noise and accurately extracts regions of interest by preprocessing the original image and using adaptive threshold segmentation. It uses the gray-scale centroid method to obtain the initial physical center of the light spot, which provides a basis for subsequent evaluation of the macroscopic offset of the light spot and calculation of the distance penalty factor, and facilitates the subsequent differentiation between the core energy region and stray light far from the center.

[0010] According to the present invention, a method for adjusting the human eye camera of an active alignment device in an XR device is provided. The method for obtaining the tilt ellipticity index includes: obtaining two eigenvalues ​​of the covariance matrix, denoted as the first eigenvalue and the second eigenvalue, respectively, wherein the first eigenvalue is greater than the second eigenvalue, the first eigenvalue corresponds to the square of the major axis of the fitted ellipse, and the second eigenvalue corresponds to the square of the minor axis of the fitted ellipse; and subtracting 1 from the square root of the ratio of the first eigenvalue to the second eigenvalue to obtain the tilt ellipticity index.

[0011] According to the present invention, a method for aligning a human eye camera in an active alignment device for an XR device is provided, wherein the tilt ellipticity index is used to dynamically determine the number of Gaussian components fitted by the GMM. ,include:

[0012] ;

[0013] The number of Gaussian components fitted to the GMM. The minimum number of Gaussian components, This is the proportionality coefficient. The ellipticity index is used to indicate the tilt. The baseline Gaussian component number, To find the maximum value function, This is the rounding function.

[0014] This invention establishes a dynamic functional relationship between the number of Gaussian components and the tilt ellipticity index. Compared to setting a fixed number of components, this invention can automatically increase the number of components to capture subtle features when the light spot distortion is severe, and reduce the number of components to prevent overfitting when the distortion is small, thereby ensuring the real-time efficiency of the algorithm while ensuring the fitting accuracy.

[0015] According to the present invention, a method for adjusting the human eye camera of an active alignment device for an XR device is provided. The method for obtaining the gradient factor of a pixel based on the gradient magnitude of a pixel in the ROI region includes: multiplying the ratio of the square of the gradient magnitude of the pixel to the maximum gradient magnitude in the ROI region by a gradient suppression coefficient and taking the negative number as the exponent input of an exponential function to obtain the gradient factor of the pixel.

[0016] According to the present invention, a method for adjusting the human eye camera of an active alignment device for an XR device is provided, which calculates the target weight of each Gaussian component, including: taking the negative of the ratio of the square of the center distance of the Gaussian component to the square of the center distance of the spot, and using it as the exponential input of the exponential function to obtain the distance penalty factor of the Gaussian component; and obtaining the target weight of the Gaussian component by multiplying the distance penalty factor of the Gaussian component, the mean gradient factor in the ROI region, and the original mixing weight.

[0017] This invention provides a precise method for calculating the target weight of Gaussian components. By reducing the target weight of spurious features and gradient anomaly components, it effectively eliminates the interference of unreasonable Gaussian components on the optical center calculation, and effectively improves the positioning accuracy in environments with stray light or edge aberrations.

[0018] According to the present invention, a method for aligning a human eye camera in an XR device's active alignment device, the method for obtaining the angle mapping model includes:

[0019] ;

[0020] The optical tilt angle of a human eye camera. The ellipticity index is used to indicate the tilt. The location of the optical center. The location of the ideal center of the image. , , These are the first, second, and third calibration coefficients, respectively. It is the arctangent function.

[0021] According to the present invention, a method for adjusting the position of a human eye camera in an active alignment device for an XR device is provided. The method for obtaining a monocular adjustment command to correct the position and orientation of the human eye camera includes: using the difference between the position of the optical center and the position of the ideal center of the image as the position adjustment amount in the X and Y directions; decomposing the optical tilt angle into components in the X and Y directions to obtain the orientation correction amount for rotation around the X and Y axes; and sending the position adjustment amount and the orientation correction amount to a six-axis robot control system to drive the human eye camera to adjust its position and orientation simultaneously.

[0022] According to the present invention, a method for adjusting the human eye camera of an active alignment device for an XR device is provided, wherein the optical tilt angle is decomposed into components in the X and Y directions to obtain the attitude correction amount of rotation around the X and Y axes, including: obtaining the principal axis direction angle and the spot center moment based on the covariance matrix of pixel intensity in the ROI region;

[0023] ; ;

[0024] , These are the attitude correction values ​​for rotation around the X-axis and Y-axis, respectively. The optical tilt angle of a human eye camera. The main axis direction angle, Let the center moment of the light spot be , , These are the sine and cosine functions, respectively. It is a symbolic function.

[0025] This invention utilizes the principal axis orientation angle of the light spot obtained by the covariance matrix to accurately decompose the total optical tilt angle into independent rotational components around the X and Y axes. This solves the problem that a single tilt angle cannot directly guide the motion of a multi-axis robot, ensuring the precise correspondence between the attitude correction command and the robot coordinate system, thereby realizing synchronous decoupling adjustment of multiple degrees of freedom.

[0026] According to the present invention, a method for adjusting the human eye camera of an active alignment device for an XR device is provided, wherein obtaining a monocular adjustment command to correct the position and orientation of the human eye camera, and then further comprising: calculating a modulation transfer function value based on the corrected human eye camera to evaluate the optomechanical performance.

[0027] The present invention has the following beneficial effects:

[0028] Based on the above technical solutions, this invention provides a method for aligning the human eye camera in an XR device's active alignment system. By calculating the eigenvalues ​​of the ROI region's covariance matrix, a tilt ellipticity index is obtained to assess the degree of light spot distortion. Based on this, the number of Gaussian components fitted by the GMM is dynamically adjusted, solving the underfitting or overfitting problem caused by a fixed number of components. Furthermore, this invention introduces gradient factors and distance weights to filter and weight the Gaussian components, effectively suppressing interference from stray light and edge aberration pseudo-features, thereby accurately locating the true optical center. Finally, the optical tilt angle is inferred by combining an angle mapping model, achieving high-precision correction of both the human eye camera's positional deviation and attitude tilt, thus improving the alignment accuracy of the human eye camera in an XR device's active alignment system. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of a method for adjusting the position of a human eye camera in an XR device actively aligning with an XR device, according to an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of an XR device active alignment process provided in an embodiment of the present invention.

[0031] Figure 3 A schematic diagram showing the positions of the laser, the product contour block, and the human eye camera provided in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a process for determining the optical center, provided as an embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0034] In the AA process of XR equipment, the laser spot is used to adjust the axis of the human eye camera, ensuring that the center of the spot falls precisely in the center of the image. That is, the center of the laser spot pattern coincides with the optical center of the camera, and both are aligned with the optical center of the waveguide sheet. However, there may be an optical tilt angle between the human eye camera and the product profilograph, i.e., a deviation in camera orientation. This tilt causes significant ellipticization and principal axis tilt in the laser spot pattern, resulting in two-dimensional geometric distortions. This means that although the laser spot is aligned with the physical center of the image during calibration, this physical center itself is already deviated from the camera's true optical center.

[0035] Based on this, embodiments of the present invention provide a method for optimizing the positioning and image alignment of human eye cameras in XR devices based on geometrically enhanced model (GMM). This method is applied to the debugging stage of active alignment (AA) devices and can effectively improve the accuracy of calibrating the position and orientation of human eye cameras.

[0036] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a method for adjusting the human eye camera in an XR device's active alignment device, according to an embodiment of the present invention. The method includes the following steps:

[0037] S1: Obtain the ROI region in the laser spot image captured by the human eye camera.

[0038] For example, in an embodiment of the present invention, obtaining the ROI region in a laser spot image acquired by a human eye camera includes: placing a laser in a product contour block, and acquiring an original image of the laser spot projected onto the contour block by the human eye camera; after preprocessing the original laser spot image to obtain a laser spot image, using an adaptive threshold segmentation algorithm to extract the ROI region of the light spot in the laser spot image.

[0039] Specifically, when preprocessing the original laser spot image to obtain the laser spot image, median filtering or Gaussian filtering can be applied to the original laser spot image to remove random noise; and an adaptive threshold segmentation algorithm is used to generate a binary mask image of the laser spot to obtain the region of interest (ROI) of the spot. The pixel intensity is then normalized to obtain the intensity-normalized ROI region.

[0040] The ROI region can be extracted using methods such as local Otsu's method and background subtraction. The specific settings can be configured according to actual needs, and this embodiment of the invention does not impose too many restrictions here.

[0041] See also Figure 2 and Figure 3 As shown, Figure 2This is a schematic diagram of an active alignment process for an XR device provided in an embodiment of the present invention. During image acquisition, a six-axis robot first moves the optical engine to a preset position, typically at the connection between the temple and the lens. Then, the optical engine is activated, projecting a test image containing features such as a target. The image is refracted through the lens or transmitted via a waveguide to form the image to be tested, and a human eye camera captures the image in real time.

[0042] Figure 3 This is a schematic diagram showing the positions of the laser, product profiling block, and human eye camera provided in an embodiment of the present invention. When adjusting the position of the human eye camera, the laser can be placed inside the product profiling block, which is then placed in a fixture. The human eye camera captures the speckle pattern from the laser, and the axis of the human eye camera is adjusted to ensure that the center of the speckle is at the center of the image.

[0043] Optical tilt between the human eye camera and the product contour block can cause two-dimensional geometric distortion in the laser spot image. Therefore, a sub-pixel localization algorithm for laser spots based on Gaussian mixture model (GMM) is required. This algorithm uses EM (Expectation Maximization) iterative optimization to fit the pixel intensity distribution of the spot ROI region as a mixture of multiple Gaussian components and extracts the mean of the Gaussian component with the highest fitting degree as the position coordinate of the sub-pixel optical center. However, GMM cannot adapt to the two-dimensional geometric distortion in the laser spot image.

[0044] Based on this, embodiments of the present invention can obtain two-dimensional geometric feature indices of the laser spot pattern, and by combining the two-dimensional geometric feature indices of the laser spot pattern with the GMM, it is possible to simultaneously achieve center offset correction of the human eye camera and reverse correction of the optical tilt angle of the camera, i.e., to perform the following steps.

[0045] S2: Based on the ratio of eigenvalues ​​of the covariance matrix of pixel intensity within the ROI region, the tilt ellipticity index is obtained to evaluate the geometric distortion of the spot in the ROI region; the tilt ellipticity index is used to dynamically determine the number of Gaussian components fitted by the GMM.

[0046] Among them, the number of Gaussian components fitted by GMM It is positively correlated with the tilt ellipticity index.

[0047] It should be noted that, since GMM is less sensitive to optical tilt angles, embodiments of the present invention can construct and extract distortion geometric indices to enhance GMM fitting by acquiring the two-dimensional geometry of the light spot. Furthermore, the camera's optical tilt angle causes the light spot to be projected as an ellipse, and its aspect ratio and principal axis direction angle contain attitude information. Moreover, the AA (Alignment and Automation) process environment may introduce stray light or edge aberrations, causing the GMM to fit some Gaussian components far from the ideal image center. These Gaussian components are spurious features, and if included in the final optical center calculation, they will reduce its accuracy. In addition, the ellipticity of the light spot is mainly caused by the camera's optical tilt angle, but it may also be affected by center position deviations, such as the different manifestations of aberrations at different field-of-view locations.

[0048] Based on this, embodiments of the present invention can obtain the center distance of the light spot to assess the degree of macroscopic center offset of the light spot, which can be used as a normalization benchmark. If the distance of the fitted component from the ideal center is significantly greater than the overall macroscopic offset of the light spot, it indicates that it is likely to be a spurious feature. By reducing the weight of such Gaussian components that are significantly deviated from the core energy region of the light spot, the true core optical center of the light spot can be located more accurately. At the same time, the influence of the center position deviation on the ellipticity measurement is reduced, so that the final optical tilt angle can accurately reflect the pure attitude deviation, rather than the geometric error caused by the coupling of position and attitude.

[0049] For example, in an embodiment of the present invention, the method for obtaining the center distance of the light spot in the ROI region includes: obtaining the initial physical center of the light spot using the gray centroid method, and obtaining the center distance of the light spot by the difference between the initial physical center of the image and the ideal center of the image.

[0050] The grayscale centroid method obtains the weighted average coordinates of the ROI region by multiplying the pixel coordinates by their normalized intensity, which is used to reflect the physical center of the light spot intensity. This can be achieved through existing technologies, and will not be elaborated on in this embodiment of the invention.

[0051] The center distance of the light spot obtained from the above steps can be used to evaluate the macroscopic offset of the light spot. The larger the center distance, the more severe the offset of the light spot center.

[0052] It should be understood that, since the optical tilt angle of the camera causes the light spot to be projected into an ellipse, the embodiments of the present invention can evaluate its pose information by the degree of ellipticity of the ROI region of the light spot.

[0053] Specifically, the second-order central moments are calculated using the pixel intensities within the ROI region of the light spot to construct the covariance matrix. ,in, , Describe the variance of the intensity distribution of the light spot in the X and Y directions, respectively. It is used to describe the correlation between the X and Y directions and is the center moment of the light spot.

[0054] It should be noted that the above-mentioned spot center moment and spot center distance are not the same concept. Spot center moment is an indicator used in statistics and image processing to describe the shape, distribution and characteristics of data or images, while spot center distance is used to evaluate the macroscopic offset of the spot and is a geometric distance concept.

[0055] For example, in an embodiment of the present invention, obtaining the tilt ellipticity index includes: obtaining two eigenvalues ​​of the covariance matrix, denoted as the first eigenvalue and the second eigenvalue, respectively, wherein the first eigenvalue is greater than the second eigenvalue, the first eigenvalue corresponds to the square of the major axis of the fitted ellipse, and the second eigenvalue corresponds to the square of the minor axis of the fitted ellipse; and subtracting 1 from the square root of the ratio of the first eigenvalue to the second eigenvalue to obtain the tilt ellipticity index.

[0056] Specifically, the closer the tilt ellipticity index is to 0, the closer the ratio of the first eigenvalue to the second eigenvalue is to 1, and the closer the light spot is to a circle. Conversely, the closer the tilt ellipticity index is to 1, the closer the ratio of the first eigenvalue to the second eigenvalue is to 0, and the greater the geometric distortion of the light spot, the more severe the ellipticization of the light spot. Since a larger optical tilt angle results in a greater degree of projection stretching of the light spot onto the camera's imaging plane, a larger tilt ellipticity index corresponds to a larger optical tilt angle, and the two are positively correlated.

[0057] The tilt ellipticity index of the ROI region is obtained based on the above steps. The larger the tilt ellipticity index, the more complex the energy distribution of the ROI region. The purpose of GMM is to fit the complex energy distribution of the light spot with multiple Gaussian components. The larger the tilt ellipticity index, the more Gaussian components the GMM needs to capture its subtle distortion features in order to achieve accurate positioning. In a conventional GMM, the number of Gaussian components is a fixed value. If the number of Gaussian components is fixed and too small, it will be unable to fit the light spot with severe distortion, which may result in underfitting; if the number of Gaussian components is fixed and too large, it may cause overfitting to simple and regular light spots, and it is easy to introduce noise components.

[0058] Based on this, embodiments of the present invention can adjust the fixed reference Gaussian component number in a conventional GMM by using the tilt ellipticity index to accurately adapt to the degree of spot distortion in the current ROI region.

[0059] Optionally, in this embodiment of the invention, the number of Gaussian components fitted by the GMM can be calculated using the following formula:

[0060] ;

[0061] The number of Gaussian components fitted to the GMM. The minimum number of Gaussian components, This is the proportionality coefficient. The ellipticity index is used to indicate the tilt. The baseline Gaussian component number, To find the maximum value function, This is the rounding function.

[0062] The minimum Gaussian component count can be set to 1, the reference Gaussian component count can be set to 2, and the scaling factor can be set to 15; the specific settings can be adjusted according to actual needs.

[0063] In the above relation, This is the component adjustment amount, used to adjust the number of reference Gaussian components. The larger the tilt ellipticity index, the higher the adjustment amount for the number of reference Gaussian components.

[0064] The minimum number of Gaussian components is the lower limit of the number of Gaussian components fitted by GMM. By taking the maximum value, we can ensure that the number of Gaussian components obtained based on the above relationship will not be lower than the minimum number of Gaussian components.

[0065] Thus, by dynamically determining the number of Gaussian components, the model can adaptively adjust according to the real-time distortion complexity of the light spot, avoiding the underfitting or overfitting problems caused by fixing the number of Gaussian components, and improving the positioning accuracy and real-time performance.

[0066] S3: Based on the gradient factor of the pixels in the ROI region and the distance penalty factor of the Gaussian component, the target weight of each Gaussian component is obtained to obtain the optical center.

[0067] It should be noted that the true optical center of the laser spot should be near the ideal center of the image. Even if there is a shift, it will still converge in the core region of the spot. Therefore, the greater the distance between a Gaussian component and the ideal center of the image, the more likely that the Gaussian component is stray light from a distance and is a spurious feature. Thus, during the iteration process, Gaussian components that are far from the ideal center of the image need to be assigned a smaller weight to suppress the influence of spurious features with unreasonable positions on the confirmation of the optical center position.

[0068] Furthermore, if the local gradient magnitude of a pixel in the ROI region is larger, it indicates that the pixel may be located at the edge of the spot, a noise spike, or a region with severe aberrations. Therefore, during the iteration process, for pixels with drastic and unstable gradient changes, a lower weight needs to be set to suppress the influence of high-frequency noise and local optical defects on the GMM parameter fitting, increase the GMM's focus on the core energy region of the spot, and improve the accuracy of optical center acquisition.

[0069] Based on this, in determining whether a Gaussian component is a true or false feature, embodiments of the present invention can obtain the distance penalty factor of each Gaussian component and the gradient factor of the pixel in the ROI region, respectively. By combining these two factors to adjust the weight of each Gaussian component, the determination of the optical center can be biased towards the higher-quality true features in the Gaussian components. For details, please refer to... Figure 4 As shown, Figure 4 The following is a flowchart illustrating the process of determining the optical center, as provided in an embodiment of the present invention.

[0070] S401: The gradient factor of a pixel is obtained based on the gradient magnitude of the pixel in the ROI region. The gradient factor is negatively correlated with the gradient magnitude in the ROI region.

[0071] For example, in an embodiment of the present invention, obtaining the gradient factor of a pixel based on the gradient magnitude of a pixel in the ROI region includes: multiplying the ratio of the square of the gradient magnitude of the pixel to the maximum gradient magnitude in the ROI region by the gradient suppression coefficient and taking the negative number as the exponential input of the exponential function to obtain the gradient factor of the pixel.

[0072] The exponential function can be an exponential function with base e, where e is the natural constant. Optionally, the gradient suppression coefficient can be set to 3, which can be adjusted according to actual needs.

[0073] Pixel gradient magnitude measures the drasticness of brightness changes around a pixel. A larger square indicates a faster brightness change at that pixel, potentially located at image edges, high-frequency noise spikes, or optical defects; data stability and reliability are lower in such areas. Conversely, a smaller square indicates a smoother brightness change at that pixel, possibly located in the core region of a bright spot or on a flat background; data stability and reliability are higher in such areas.

[0074] By mapping high gradients to low weights and low gradients to high weights using an exponential function, the resulting gradient factor of a pixel can be used to characterize the local stability and reliability of the image data at that pixel.

[0075] The gradient factor of each pixel in the ROI region can be obtained by following the steps above.

[0076] S402: For each Gaussian component, obtain the center distance based on the distance between the Gaussian component and the ideal center of the image, so as to obtain the distance penalty factor of the Gaussian component.

[0077] For example, in an embodiment of the present invention, calculating the target weight of each Gaussian component includes: taking the negative of the ratio of the square of the center distance of the Gaussian component to the square of the center distance of the spot, and using it as the exponential input of the exponential function to obtain the distance penalty factor of the Gaussian component.

[0078] The mean of the Gaussian component can be used as the center of the Gaussian component.

[0079] For ease of understanding, embodiments of the present invention provide a method for calculating the distance penalty factor of Gaussian components, as shown in the following formula:

[0080] ;

[0081] For the first Distance penalty factor for each Gaussian component, The penalty coefficient is... For the first The center of each Gaussian component The location of the ideal center of the image. The center distance of the light spot It is an exponential function with base e. The norm symbol for a vector. It is a constant used to avoid a denominator of 0, and can be set to a very small positive real number.

[0082] The penalty coefficient controls the intensity of the distance penalty and can be set to 1, depending on the actual needs.

[0083] For the first The center distance of each Gaussian component, due to As a vector, representing positional differences on a two-dimensional plane, the length of the two-dimensional vector is calculated using the norm notation and then amplified by squaring. A larger center distance indicates a greater distance between the Gaussian component and the ideal center of the image, increasing the likelihood of spurious features generated by stray light or edge aberrations. The corresponding distance penalty factor is smaller, resulting in a smaller impact on the weights.

[0084] The distance penalty factor for each Gaussian component can be obtained by following the steps above. The weights are adjusted by the distance penalty factor so that the final determined optical center can be closer to the core energy distribution center of the light spot.

[0085] S403: Calculate the target weight of each Gaussian component. The target weight is negatively correlated with the center distance and positively correlated with the mean gradient factor in the ROI region. The optical center is obtained by weighting the centers of the Gaussian components based on the target weight.

[0086] For example, in an embodiment of the present invention, the target weight of the Gaussian component is obtained by multiplying the distance penalty factor of the Gaussian component, the mean gradient factor in the ROI region, and the original mixed weight.

[0087] The original mixture weights can be calculated in the GMM fitting using the expectation-maximization algorithm, which will not be elaborated here in the embodiments of the present invention.

[0088] For example, when obtaining the optical center by weighting the Gaussian component centers based on the target weight, the weighted average of all Gaussian component centers can be used as the optical center.

[0089] Thus, this embodiment of the invention employs intensity distribution-based GMM fitting and geometric feature enhancement to extract a more robust optical center, rather than relying on a physical center susceptible to tolerances. This effectively improves calibration accuracy and provides a high-precision reference for subsequent AA processes.

[0090] S4: The optical tilt angle of the human eye camera is obtained by back-calculating the tilt ellipticity index and the optical center input preset angle mapping model.

[0091] It should be noted that the stretching of the laser spot into an ellipse is a geometric result of tilt projection; therefore, tilt ellipticity is a primary image feature representing the tilt angle. Furthermore, the degree of elliptic distortion in the image depends not only on the camera's tilt angle but may also be affected by the actual position of the camera's optical center. A significant center offset can have a secondary impact on the measurement results of the optical tilt angle of the human eye camera.

[0092] Based on this, in determining the optical tilt angle of the human eye camera, the embodiments of the present invention can obtain the optical tilt angle of the human eye camera by combining the tilt ellipticity index and the degree of offset between the optical center of the camera and the ideal center of the image, so as to reduce the secondary error caused by the position deviation to the attitude feature measurement. If the center offset is zero, the final optical tilt angle is determined by the attitude feature tilt ellipticity index; if the center offset is large, the optical tilt angle can be corrected by the degree of offset, thereby improving the accuracy of the model.

[0093] Optionally, in this embodiment of the invention, the method for obtaining the angle mapping model can be found in the following relation:

[0094] ;

[0095] The optical tilt angle of a human eye camera. The ellipticity index is used to indicate the tilt. The location of the optical center. The location of the ideal center of the image. The first calibration coefficient, This is the second calibration coefficient. This is the third calibration coefficient. It is the arctangent function.

[0096] The first calibration coefficient is the amplitude scaling coefficient, which controls the total output amplitude of the arctangent function and can be set to 4. The second calibration coefficient is the tilt sensitivity coefficient, which controls the sensitivity of the tilt ellipticity index to the input of the arctangent function and can be set to 6. The third calibration coefficient is the center offset penalty coefficient, which evaluates the contribution weight of the center offset to the attitude angle calculation and can be set to 0.3. It can be obtained through offline calibration and can be set according to actual needs.

[0097] The optical tilt angle of the human eye camera can be obtained by following the steps above.

[0098] S5: Based on the deviation between the optical center and the ideal image center, as well as the optical tilt angle, a monocular adjustment command is obtained to correct the position and orientation of the human eye camera.

[0099] It should be noted that the optical tilt angle of the human eye camera obtained in the above steps is used by the six-axis robot to correct its posture in the AA process by rotating around the X and Y axes of the human eye camera coordinate system. The rotation correction around the X and Y axes together constitute the posture correction amount.

[0100] Based on this, embodiments of the present invention can decompose the attitude correction amount into components in the X and Y directions to realize the position and attitude adjustment of a six-axis robot.

[0101] For example, in an embodiment of the present invention, obtaining a monocular adjustment command to correct the position and attitude of the human eye camera includes: using the difference between the position of the optical center and the position of the ideal center of the image as the position adjustment amount in the X and Y directions; decomposing the optical tilt angle into components in the X and Y directions to obtain the attitude correction amount of rotation around the X and Y axes; and sending the position adjustment amount and attitude correction amount to the six-axis robot control system to drive the human eye camera to adjust its position and attitude simultaneously.

[0102] For example, in an embodiment of the present invention, obtaining the attitude correction amount for rotation around the X-axis and Y-axis includes: obtaining the principal axis direction angle and spot center moment based on the covariance matrix of pixel intensity within the ROI region, decomposing the optical tilt angle into components in the X and Y directions, and obtaining the attitude correction amount for rotation around the X-axis and Y-axis.

[0103] Optionally, the attitude correction amount for rotation around the X-axis can be calculated using the following formula:

[0104] ;

[0105] This is the attitude correction amount for rotation around the X-axis. The optical tilt angle of a human eye camera. The main axis direction angle, Let the center moment of the light spot be , It is a sine function. It is a symbolic function.

[0106] in, Used to eliminate the tilt deviation of the camera's projected components in the X-axis direction. If A value greater than 0 indicates that the camera is tilted excessively in the X direction, and the robot needs to rotate around the negative X-axis. The angle is reset to zero.

[0107] Optionally, the attitude correction amount for rotation around the X-axis can be calculated using the following formula:

[0108] ;

[0109] This is the attitude correction amount for rotation around the Y-axis. The optical tilt angle of a human eye camera. The main axis direction angle, Let the center moment of the light spot be , It is a cosine function. It is a symbolic function.

[0110] in, Used to eliminate the tilt deviation of the camera's projected components in the Y-axis direction. If A value greater than 0 indicates that the camera is tilted excessively in the Y direction, and the robot needs to rotate around the negative Y-axis. The angle is reset to zero.

[0111] In the above relationship, the sign function is used to evaluate the correlation, and the positive or negative sign of the spot center moment indicates the correlation between the X and Y coordinates of the spot intensity distribution. When the value is greater than 0, it indicates a positive correlation, and the main axis of the light spot extends from the lower left to the upper right. When the value is less than 0, it indicates a negative correlation, and the main axis of the light spot extends from the upper left to the lower right.

[0112] The complete adjustment command obtained from the above steps includes position adjustment amount and attitude correction amount. These are packaged, associated, and sent to the six-axis platform. The six-axis robot performs displacement and rotation based on the adjustment command, which can realize synchronous adjustment of position and attitude.

[0113] Specifically, both position adjustment and attitude correction include adjustments along three axes: X, Y, and Z. For position adjustment, the adjustments along the X and Y axes are the differences between the optical center and the ideal image center, while the adjustment along the Z axis is zero. For attitude correction, the adjustments along the X and Y axes are obtained using the formulas described above. and , This is the attitude correction amount for rotation around the Z-axis, and its value is also 0.

[0114] After the XR device is actively aligned with the human eye camera of the device according to the above steps, subsequent processing can be performed based on the accurate pose.

[0115] For example, in an embodiment of the present invention, a monocular tuning command is obtained to correct the position and orientation of the human eye camera, and then the method further includes: calculating the modulation transfer function value based on the corrected human eye camera to evaluate the optomechanical performance.

[0116] The calculation of the modulation transfer function value can be achieved through existing steps, which will not be elaborated here in the embodiments of the present invention.

[0117] As can be seen in this embodiment of the invention, when the XR device actively aligns with the human eye camera of the device, the Region of Interest (ROI) in the laser spot image acquired by the human eye camera can be obtained; based on the ratio of the eigenvalues ​​of the covariance matrix of pixel intensity within the ROI region, the tilt ellipticity index is obtained to evaluate the geometric distortion of the spot in the ROI region; the tilt ellipticity index is used to dynamically determine the number of Gaussian components fitted by the GMM. , The gradient factor of a pixel is obtained based on the gradient magnitude of the pixel in the ROI region, and the gradient factor is negatively correlated with the gradient magnitude in the ROI region. For each Gaussian component, the center distance is obtained based on the distance between the Gaussian component and the ideal center of the image. The target weight of each Gaussian component is calculated. The target weight is negatively correlated with the center distance and positively correlated with the mean gradient factor in the ROI region. The optical center is obtained by weighting the center of the Gaussian component based on the target weight. The optical tilt angle of the human eye camera is obtained by inputting the tilt ellipticity index and the optical center into a preset angle mapping model. Based on the deviation between the optical center and the ideal center of the image, and the optical tilt angle, a monocular adjustment command is obtained to correct the position and orientation of the human eye camera, which effectively improves the adjustment accuracy of the human eye camera.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for adjusting the position of a human eye camera in an active alignment device for an XR device, characterized in that, include: Obtain the Region of Interest (ROI) in the laser spot image captured by the human eye camera; The tilt ellipticity index is obtained based on the ratio of eigenvalues ​​of the covariance matrix of pixel intensity within the ROI region to evaluate the geometric distortion of the spot in the ROI region. Specifically, two eigenvalues ​​of the covariance matrix are obtained and denoted as the first eigenvalue and the second eigenvalue, respectively. The first eigenvalue is greater than the second eigenvalue. The first eigenvalue corresponds to the square of the major axis of the fitted ellipse, and the second eigenvalue corresponds to the square of the minor axis of the fitted ellipse. The square root of the ratio of the first eigenvalue to the second eigenvalue is then subtracted from 1. The tilt ellipticity index is used to dynamically determine the number of Gaussian components fitted to the Gaussian mixture model (GMM). ,include: ; The number of Gaussian components fitted to the GMM. The minimum number of Gaussian components, This is the proportionality coefficient. The ellipticity index is used to indicate the tilt. The baseline Gaussian component number, To find the maximum value function, This is the rounding function. Positively correlated with the tilt ellipticity index; The gradient factor of a pixel is obtained based on the gradient magnitude of the pixel in the ROI region. The gradient factor is negatively correlated with the gradient magnitude in the ROI region. For each Gaussian component, the center distance is obtained based on the distance between the Gaussian component and the ideal center of the image. The target weight of each Gaussian component is calculated, including: taking the negative of the ratio of the square of the center distance of the Gaussian component to the square of the center distance of the spot, and using it as the exponent input of the exponential function to obtain the distance penalty factor of the Gaussian component. The target weight of the Gaussian component is obtained by multiplying the distance penalty factor of the Gaussian component, the mean gradient factor in the ROI region, and the original mixing weight. The target weight is negatively correlated with the center distance and positively correlated with the mean gradient factor in the ROI region. The optical center is obtained by weighting the center of the Gaussian component based on the target weight. The tilt ellipticity index and optical center are input into a preset angle mapping model to back-calculate the optical tilt angle of the human eye camera; based on the deviation between the optical center and the ideal center of the image, and the optical tilt angle, a monocular adjustment command is obtained to correct the position and orientation of the human eye camera. Methods for obtaining the angle mapping model include: ; The optical tilt angle of a human eye camera. The ellipticity index is used to indicate the tilt. The location of the optical center. The location of the ideal center of the image. , , These are the first, second, and third calibration coefficients, respectively. It is the arctangent function.

2. The method for adjusting the human eye camera in an active alignment device for an XR device according to claim 1, characterized in that, The acquisition of the Region of Interest (ROI) in the laser spot image captured by the human eye camera includes: The laser is placed in the product contour block, and the human eye camera captures the original image of the laser spot projected onto the contour block. After preprocessing the original image of the laser spot to obtain the laser spot image, the ROI region of the spot in the laser spot image is extracted using an adaptive threshold segmentation algorithm. The initial physical center of the spot is obtained using the gray-scale centroid method, and the center distance of the spot is obtained by the difference between the initial physical center of the image and the ideal center of the image.

3. The method for adjusting the human eye camera in an active alignment device for an XR device according to claim 1, characterized in that, The gradient factor of a pixel is obtained based on the gradient magnitude of the pixel in the ROI region, including: The gradient factor of a pixel is obtained by multiplying the ratio of the square of the gradient magnitude of a pixel to the maximum gradient magnitude within the ROI region by the gradient suppression coefficient and taking the negative number. This negative number is used as the exponent input of the exponential function.

4. The method for adjusting the human eye camera in an active alignment device for an XR device according to claim 1, characterized in that, The process of obtaining monocular adjustment commands to correct the position and orientation of the human eye camera includes: The difference between the position of the optical center and the position of the ideal center of the image is used as the position adjustment amount in the X and Y directions; the optical tilt angle is decomposed into components in the X and Y directions to obtain the attitude correction amount for rotation around the X and Y axes; the position adjustment amount and attitude correction amount are sent to the six-axis robot control system to drive the human eye camera to adjust its position and attitude simultaneously.

5. A method for adjusting the position of a human eye camera in an active alignment device for an XR device according to claim 4, characterized in that, The process of decomposing the optical tilt angle into components in the X and Y directions to obtain attitude correction amounts for rotation around the X and Y axes includes: The principal axis orientation angle and spot center moment are obtained based on the covariance matrix of pixel intensity within the ROI region; ; ; , These are the attitude correction values ​​for rotation around the X-axis and Y-axis, respectively. The optical tilt angle of a human eye camera. The main axis direction angle, Let the center moment of the light spot be , , These are the sine and cosine functions, respectively. It is a symbolic function.

6. A method for adjusting the position of a human eye camera in an XR device's active alignment device according to claim 1, characterized in that, The process of obtaining the monocular adjustment command to correct the position and orientation of the human eye camera further includes: The modulation transfer function value is calculated based on the calibrated human eye camera to evaluate optomechanical performance.

Images