Video see-through camera auto focus method and system for XR headsets
By combining a neural network model based on the Transformer architecture with eye-tracking technology, single-shot focus parameter prediction for XR head-mounted display video perspective cameras was achieved, solving the problems of slow focusing speed and difficulty in depth information mapping in existing technologies, and improving focusing speed and accuracy in dynamic scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU HUIJIAN ZHILIAN TECH CO LTD
- Filing Date
- 2026-05-11
- Publication Date
- 2026-06-05
AI Technical Summary
Existing video perspective cameras have slow autofocus methods in dynamic scenes, and the depth information of monocular images cannot be effectively mapped to focus parameters, making it difficult for the device to adapt to moving targets or rapidly changing scenes.
It employs a neural network model based on the Transformer architecture, combined with eye tracking of the XR headset to obtain the coordinates of the user's gaze point. By using the depth map of the gaze area and the coordinates of the gaze point, it performs single-shot focus parameter prediction and directly maps the target focus parameters, avoiding repeated attempts and reliance on depth sensors.
It achieves fast and accurate single-shot focusing, adapts to moving targets and dynamic scenes, improves the real-time performance and user experience of XR headset video perspective, and reduces focusing speed and computing power consumption.
Smart Images
Figure CN122160625A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to an autofocus method and system for a video perspective camera used in XR head-mounted displays. Background Technology
[0002] With the rapid development of extended reality (XR) devices such as virtual reality (VR) and mixed reality (MR), video see-through (VST) technology has become a key approach to achieving a seamless blend of virtual and real experiences. In a VST system, the camera module is responsible for capturing images of the real scene, and its autofocus performance directly affects the clarity and immersion of the user's viewing experience.
[0003] Currently, common autofocus methods for VST cameras include contrast-detection autofocus and single-shot autofocus. Contrast-detection autofocus moves the focusing mechanism to different positions, acquires images in real time, and calculates the contrast or sharpness evaluation value of the image area, selecting the position with the highest evaluation value as the optimal focus point. Single-shot autofocus acquires scene depth information, establishes a mapping relationship between depth and focal length, and thus calculates the appropriate focus parameters in one operation.
[0004] However, contrast-based autofocus requires multiple adjustments to focus parameters and repeated attempts during the focusing process, resulting in a slow focusing speed and difficulty adapting to moving targets or rapidly changing scenes. Therefore, contrast-based autofocus is mainly suitable for static shooting scenarios and has significant limitations in dynamic XR environments. Single-shot autofocus methods generally suffer from the problem of missing mapping relationships between depth information and focal length parameters. In addition, devices without depth estimation sensors cannot obtain effective depth data; if depth information is recovered using monocular vision technology, the resulting depth data lacks true scale and cannot be directly used for focal length parameter calculation; while using binocular vision technology to recover depth information can obtain scaled depth data, this solution consumes a large amount of computing resources, which is not conducive to device miniaturization and low-power deployment. Summary of the Invention
[0005] Based on the above analysis, the present invention aims to provide an autofocus method and system for a video see-through camera for XR headsets, in order to solve the problems of slow focusing speed and ineffective mapping of depth information and focusing parameters in existing video see-through cameras.
[0006] On one hand, embodiments of the present invention provide an autofocus method for a video see-through camera used in XR headsets, comprising the following steps: Acquire the current image captured by the XR headset's video perspective camera and its corresponding depth map; Based on the user's gaze coordinates in the current image, extract the local region centered on the gaze coordinates from the depth map as the gaze region depth map; The depth map of the gaze region and the coordinates of the gaze point are input into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera. Adjust the video perspective camera according to the target focus parameters to complete a single focus.
[0007] Based on further improvements to the above method, the focus parameter prediction model is a neural network model based on the Transformer architecture, which includes, in sequence: an image patch embedding layer, a position encoding layer, multiple encoder layers, and a prediction layer; wherein, each encoder layer includes: a self-attention mechanism module and a feedforward network module.
[0008] Based on the further improvements of the above method, the image patch embedding layer is used to divide the depth map of the gaze region into multiple image patches and encode each image patch into a feature vector; the position encoding layer is used to obtain the position encoding vector of each image patch and the position encoding vector of the gaze point coordinates through the position encoding algorithm; the output of the image patch embedding layer and the output of the position encoding layer are concatenated into a fusion vector, which is then passed into multiple stacked encoder layers; each encoder layer models the feature relationship of the fusion vector through a self-attention mechanism module, and performs nonlinear transformation and dimension adjustment through a feedforward network module; the prediction layer is used to map the output of the last encoder layer into target focus parameters.
[0009] Based on the further improvement of the above method, the prediction layer of the focus parameter prediction model is either a classification prediction layer or a regression prediction layer; the classification prediction layer includes: a first multi-layer fully connected network and a Softmax layer, used to output the focus category number; the regression prediction layer includes: a second multi-layer fully connected network and a linear mapping layer, used to output parameter values normalized to the [0,1] interval.
[0010] Based on the further improvement of the above method, during the training of the focus parameter prediction model, a sample set is constructed by iteratively executing acquisition operations, wherein each acquisition operation includes: The user wears an XR headset and gazes at the observation area, keeping the gaze point stationary. The system iterates through the focus parameters of the video perspective camera and captures corresponding training images. It calculates the image sharpness of the gaze region in the training image based on the coordinates of the corresponding gaze point. After the iteration is complete, it obtains the index number of the focus parameter with the highest image sharpness as the annotation result of this acquisition operation. Obtain the depth map of each training image, and extract the gaze region depth map from the depth map based on the gaze point coordinates; combine the gaze region depth map, gaze point coordinates, and the annotation results of the acquisition operation of each training image into a sample and put it into the sample set.
[0011] Based on the further improvement of the above method, the focus parameters of the video perspective camera are generated iteratively, including: Based on a preset number N, N discrete values are uniformly selected between the minimum and maximum values of the focus parameters of the video perspective camera, and each is assigned an index number from 0 to N-1. The discrete value corresponding to each index number is used as the focus parameter generated during the traversal in this acquisition operation.
[0012] Based on the further improvements of the above method, the image sharpness of the gaze region in the training image is obtained by calculating the average sharpness score of the gaze region using the Sobel operator or the Laplacian operator.
[0013] Further improvements to the above method include adjusting the video perspective camera based on the target focus parameters, including: Obtain the minimum and maximum values of the focus parameters minParam and maxParam of the video perspective camera; If the target focus parameter is the focus category number cls, and the total number of focus categories is the preset number N, then the actual focus parameter is minParam + cls × (maxParam - minParam) / N; If the target focus parameter is a normalized parameter value reg, then the actual focus parameter is minParam + reg × (maxParam - minParam); The actual focus parameters are sent to the video perspective camera to complete the focusing process.
[0014] Based on the further improvement of the above method, the local region is a square region or a circular region. The side length of the square region and the radius of the circular region are calculated from the short side length of the current image according to different preset ratios.
[0015] On the other hand, embodiments of the present invention provide an autofocus system for a video see-through camera in an XR headset, comprising: The image acquisition module is used to acquire the current image and its corresponding depth map captured by the video perspective camera of the XR headset; The gaze region extraction module is used to extract a local region centered on the gaze point coordinates from the depth map as the gaze region depth map based on the user's gaze point coordinates in the current image. The parameter prediction module is used to input the depth map of the gaze region and the coordinates of the gaze point into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera. The focus control module is used to adjust the video perspective camera according to the target focus parameters to complete a single focus.
[0016] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. By acquiring the current image and depth map of the video perspective camera, and combining it with the user's gaze coordinates obtained from eye tracking, the depth map of the gaze area is extracted. The target focus parameters are then directly mapped using a pre-trained focus parameter prediction model, allowing the camera to focus in a single adjustment. This method eliminates the need for repeated attempts and adjustments like contrast-based focusing, and it does not rely on depth sensors or binocular vision for explicit depth-focal length physical modeling. The end-to-end mapping from image to focus parameters is completed in a single forward inference, resulting in fast focusing speed, adaptability to moving targets and dynamic scenes, and significantly improved real-time performance and user experience of XR head-mounted display video perspective.
[0017] 2. Using the depth map of the gaze region and the coordinates of the gaze point as input, the model based on the Transformer architecture captures the spatial dependencies between different depth regions. At the same time, the gaze point information is incorporated as a global condition into the feature representation of each image block, enabling the model to accurately focus on the user's attention region and avoid background interference, thereby achieving higher accuracy in focus parameter prediction.
[0018] 3. The prediction layer of the focus parameter prediction model can be flexibly set as a classification prediction layer or a regression prediction layer. The two modes can be flexibly selected according to the camera hardware characteristics and system requirements, and can be used in combination under certain conditions, which greatly enhances the versatility and engineering adaptability of the autofocus method.
[0019] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0020] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart of an automatic focusing method for a video perspective camera used in an XR head-mounted display according to Embodiment 1 of the present invention. Detailed Implementation
[0021] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0022] Example 1 One specific embodiment of the present invention discloses an autofocus method for a video see-through camera used in XR headsets, such as... Figure 1As shown, it includes steps S1-S4.
[0023] S1. Obtain the current image and its corresponding depth map captured by the video perspective camera of the XR headset.
[0024] It should be noted that XR headsets typically feature two video see-through cameras (VST cameras) on the left and right sides to simulate human binocular vision. In this embodiment, autofocus is performed independently for each VST camera. Taking one VST camera as an example, its current output image I is acquired in real time, with a frame rate preferably of 60Hz to ensure real-time focus response.
[0025] To obtain the depth map corresponding to the current image I, this embodiment employs a monocular depth estimation model. Specifically, the current image I is input into a pre-trained monocular depth estimation model (such as MiDaS, Depth-Anything, or other deep learning models). This model directly outputs a pixel-level depth map D of the same size as the input image, where each pixel value is the scene's 3D spatial depth value corresponding to each pixel in the image.
[0026] Compared to existing technologies, this embodiment employs a monocular depth estimation model, eliminating the need for additional depth sensors and the high computational cost of binocular stereo matching. It can reconstruct the depth structure of a scene using only a single RGB image, which is beneficial for the miniaturization and low-power deployment of XR devices. Furthermore, this embodiment is insensitive to the scale accuracy of the depth map; subsequent learning directly establishes the mapping relationship between the depth map and focus parameters, eliminating the need for explicit scale recovery and further reducing reliance on depth sensors, thus broadening the applicability of this embodiment.
[0027] S2. Based on the user's gaze coordinates in the current image, extract the local region centered on the gaze coordinates from the depth map as the gaze region depth map.
[0028] It should be noted that the eye-tracking module on the XR headset (such as the Tobii Eye Tracking SDK or 7invensun Eye Tracking SDK) acquires the user's gaze point position in real time. The output of this eye-tracking module is the user's pixel coordinates g(x,y) on the current image I, indicating which area in the image the user is currently focusing on. The frequency of eye-tracking data acquisition is preferably consistent with the image acquisition frequency, i.e., 60Hz, to ensure that the gaze point information is synchronized with the image frame and reduce latency.
[0029] After obtaining the gaze coordinates g(x,y), a local region centered on the gaze coordinates is cropped from the corresponding full depth map D to obtain the gaze region depth map.
[0030] It should be noted that local region extraction is performed because: on the one hand, users' visual attention is usually concentrated in a small area around the point of fixation, and background information in the distance or at the edge contributes little to the current focus decision; on the other hand, it facilitates a significant reduction in data volume, lowers computational overhead, and improves inference speed, which is especially important for XR devices with limited computing power.
[0031] Specifically, the local area can be a square or a circular region. The side length of the square region and the radius of the circular region are calculated from the shorter side length of the current image according to different preset ratios. For example, the side length of the square region is set to 1 / 4 of the shorter side length of the current image; the radius of the circular region is set to 1 / 8 of the shorter side length of the current image. These ratios are just examples. In actual applications, they should be adjusted appropriately according to factors such as the field of view of the XR device and the user's viewing habits, as long as the cropped local area can cover the area of the scene that the user is interested in.
[0032] This step yields a gaze region depth map much smaller than the original depth map, preserving the depth information of key objects in the scene the user is currently focused on, while removing a large amount of irrelevant background, thus preparing for the next processing step.
[0033] S3. Input the depth map of the gaze region and the coordinates of the gaze point into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera.
[0034] It's important to note that the focus parameter prediction model is a neural network model based on the Transformer architecture. Its core function is to jointly map the "depth distribution of the user's attention region" and the "precise location of the gaze point" to the optimal focus parameters, achieving end-to-end learning from monocular visual information to camera control variables. Compared to traditional contrast-based autofocus, which requires multiple attempts or relies on depth sensors to build explicit mapping relationships, the focus parameter prediction model can complete the focus decision in a single forward inference, significantly improving response speed while avoiding the high computational cost of binocular vision.
[0035] The focus parameter prediction model consists of, in sequence: an image patch embedding layer, a position coding layer, multiple encoder layers, and a prediction layer.
[0036] (1) The image patch embedding layer is used to divide the depth map of the gaze region into multiple image patches and use a multilayer perceptron to encode each image patch into a feature vector.
[0037] First, the input gaze region depth map is uniformly scaled to a predetermined size, so that the width and height of the new image are both integer multiples of the image patch size, expressed by the following formula: , in, and These represent the width and height of the new image after scaling the depth map of the gaze region, respectively. and These represent the original width and height of the depth map of the gaze region, respectively. Indicates the size of the image block to be segmented; This represents integer division by rounding down to the nearest integer.
[0038] Specifically, a resize operation is performed on the depth map of the gaze region, recording the scaling ratios of width and height to ensure that the spatial relative relationship of depth information remains unchanged.
[0039] Then, the scaled gaze depth map is arranged according to... The image is divided into patches of varying sizes. For example, The value is set to 28. Each image patch contains depth distribution information within its local neighborhood, and there is no overlap between adjacent patches. The segmented image patch sequence preserves the spatial order, laying the foundation for subsequent location encoding and self-attention modeling.
[0040] Furthermore, each image patch is fed into a multilayer perceptron, and within the image patch... Each depth value is mapped to a fixed-length feature vector, capturing the depth statistics of the local region (such as average depth, depth gradient, etc.). For example, the length of the feature vector is set to 64.
[0041] (2) The position coding layer is used to obtain the position coding vector of each image block and the position coding vector of the gaze point coordinates through the position coding algorithm.
[0042] To preserve the absolute spatial position information of image patches, position encoding is performed on the position index of each image patch. The position encoding algorithm uses sine and cosine functions to encode the position index into a vector, which serves as the position encoding vector for the image patch. For example, the length of the position encoding vector for the image patch is set to 8.
[0043] In addition, the gaze coordinates g(x,y) are also encoded as vectors. Specifically, the x and y coordinates of the gaze point are rounded down, and then encoded into vectors of equal length using sine and cosine functions. These two encoded vectors are then concatenated to obtain the gaze point's position encoded vector. For example, the length of the encoded vector for each coordinate is set to 4, and the length of the gaze point's position encoded vector is 8.
[0044] Furthermore, the feature vector of each image patch, the location encoding vector of that image patch, and the location encoding vector of the gaze point are directly concatenated along the feature dimension to form a fusion vector.
[0045] For example, the total length of the fusion vector corresponding to each image patch is 64+8+8=80.
[0046] It should be noted that the position encoding vector of the gaze point is the same for all image patches (because there is only one gaze point), but through the concatenation operation, the gaze point information is fused into the features of each image patch, enabling the model to perceive the spatial relationship between the gaze point and each image patch.
[0047] (3) Each encoder layer includes a self-attention mechanism module and a feedforward network module; each encoder layer models the feature relationship of the fused vector through the self-attention mechanism module, and performs nonlinear transformation and dimension adjustment through the feedforward network module.
[0048] The fusion vectors corresponding to all image patches are arranged in spatial order to form a sequence, which is then input into multiple stacked Transformer encoder layers. For example, the number of encoder layers is set to 10.
[0049] The self-attention mechanism module models the global feature relationships of the input fused vector sequence. The features of each image patch can focus on the features of all other image patches in the sequence, thus capturing the dependencies between different depth regions. For example, when the gaze point is on a foreground object, the model can learn the correlation between foreground and background depths, helping to determine the optimal focusing distance. The self-attention mechanism is permutation invariant, but introduces spatial order information through positional encoding, enabling the model to distinguish image patches at different locations.
[0050] The feedforward network module performs non-linear transformations and dimensionality adjustments on the output of the self-attention module. It consists of two fully connected layers and an activation function (such as GELU). This module enhances the model's expressive power, enabling it to learn more complex depth-focus mapping patterns.
[0051] After iterative processing through multiple encoder layers, the model outputs a sequence of vectors with fully extracted features, where each vector corresponds to an original image patch but incorporates global contextual information and gaze guidance information.
[0052] (4) The prediction layer is used to map the output of the last encoder layer to the target focus parameters.
[0053] It should be noted that the target focus parameters output by the focus parameter prediction model are focus category numbers or normalized parameter values. Correspondingly, the prediction layer is a classification prediction layer or a regression prediction layer.
[0054] Specifically, the classification prediction layer includes: a first multilayer fully connected network (e.g., a multilayer perceptron MLP) and a Softmax layer; the first multilayer fully connected network maps the output of the last encoder layer to the logical values of N focus category numbers, and the Softmax layer converts the logical values into a probability distribution, taking the focus category number cls with the highest probability as the output; the value range of the focus category number is 0, 1, ..., N-1.
[0055] The regression prediction layer consists of a second multilayer fully connected network (e.g., a multilayer perceptron MLP) and a linear mapping layer. The second multilayer fully connected network maps the output of the last encoder to a scalar, and the linear mapping layer restricts this scalar within the interval [0,1], outputting a normalized parameter value reg. This value represents the relative position of the optimal focus parameter within the adjustable range [0,1] (0 corresponds to the minimum focus parameter minParam, and 1 corresponds to the maximum focus parameter maxParam).
[0056] It should be noted that when training the focus parameter prediction model, a sample set is constructed by iteratively executing acquisition operations. Each acquisition operation includes: ① The user wears an XR headset and gazes at the observation area, keeping the gaze point still.
[0057] Specifically, multiple typical usage scenarios are selected, such as offices, home environments, outdoor and indoor sports fields, with no fewer than 10 different scenarios to cover common usage environments for XR devices. The user wears an XR headset and initiates the training data acquisition program to trigger a data acquisition operation. The user gazes at a fixed observation area in the scene (such as an object on a table, a sticker on a wall, etc.) while keeping their head and eyes basically still, ensuring that the coordinates of the gaze point remain stable within the target area during the acquisition process.
[0058] ② Iterate through the focus parameters of the generated video perspective camera and capture the corresponding training images. Calculate the image sharpness of the gaze area in the training images based on the coordinates of the corresponding gaze point.
[0059] For the current gaze scene, the acquisition program automatically controls the VST camera to traverse all preset focus parameters. Specifically, first, the focus parameter range of the VST camera [minParam, maxParam] is determined, which can be obtained through camera hardware parameters or driver interface. Then, according to a preset number N, N discrete values (e.g., N=50) are evenly selected between the minimum and maximum focus parameters of the video perspective camera, and each is assigned an index number from 0 to N-1. The discrete value corresponding to each index number is used as the focus parameter generated in this traversal; where, the focus parameter param_n generated in the nth traversal is param_n=n×(maxParam-minParam) / N.
[0060] For each focus parameter, the camera captures an image as the current training image, while simultaneously recording the current gaze coordinates g(x,y) (since the user remains stationary, the gaze coordinates remain essentially unchanged). The acquisition program then crops the gaze region centered on the gaze coordinates from the current training image and calculates the image sharpness of that region.
[0061] It should be noted that the image sharpness of the gaze region in the training image is determined by using the Sobel or Laplacian operator to perform edge detection filtering on the gaze region image to obtain a gradient magnitude map, and then calculating the average score of the gradient magnitude map (i.e., the arithmetic mean of the gradient magnitudes of all pixels). The higher the score, the sharper the image.
[0062] ③ After the traversal is completed, obtain the index number corresponding to the focus parameter with the highest image clarity, and use it as the annotation result of this acquisition operation.
[0063] After N iterations, the acquisition program compares the sharpness scores obtained from each iteration, selects the one with the highest score, and records its index number as best_n. The corresponding focus parameter is bestParam=param_{best_n}. This best_n is the index number of the best focus parameter corresponding to the current scene and the current gaze point position, and is used as the annotation result of this acquisition operation.
[0064] To improve the reliability of annotation, the acquisition program automatically displays training images taken with optimal focus parameters, which are then manually verified to ensure they are indeed sharp. Only after the user confirms the training image is sharp and acceptable through any interactive input method, such as an XR controller or voice, is all data obtained from that acquisition operation saved. If the image is unacceptable (e.g., the user moved their head or the scene lighting changed), all data from that acquisition operation is discarded, and the acquisition is repeated. This manual verification step effectively ensures the accuracy of the training data.
[0065] ④ Obtain the depth map of each training image, and extract the gaze region depth map from the depth map based on the gaze point coordinates; combine the gaze region depth map, gaze point coordinates, and the annotation results of the acquisition operation of each training image into a sample and put it into the sample set.
[0066] According to the method described in step S1, the depth map of each training image is obtained using the trained monocular depth estimation model, and the depth map of the gaze region is extracted from the depth map based on the gaze point coordinates.
[0067] The annotation result of each image is set to best_n (for classification tasks) or best_n / N (normalized value for regression tasks, range [0,1]), thus forming a triplet data as a sample: (gaze region depth map, gaze point coordinates, annotation result).
[0068] It's important to note that all N training samples in the same traversal share the same annotation result (i.e., the best index number in that group) and gaze coordinates, but the depth maps of the gaze regions are different for each. This approach allows the model to learn the mapping relationship that "regardless of the current focus state, it should eventually adjust to the optimal parameters," thus enhancing the model's robustness.
[0069] Repeat steps ①-④ above to collect a sufficient number of samples in different scenarios and at different gaze points. For example, the total number of samples should be no less than 50,000.
[0070] The above sample set construction process can generate multiple training samples in one traversal. This process does not require manual annotation of depth or focus parameters for each image. Only simple manual confirmation is needed after the collection is completed to obtain a large-scale, high-quality labeled dataset, which significantly reduces the data preparation cost for model training and facilitates rapid iteration and deployment in XR headsets.
[0071] Furthermore, the sample set is randomly shuffled and divided into a training set, a validation set, and a test set in an 8:1:1 ratio.
[0072] Configure hyperparameters for model training, including batch size, initial learning rate, number of training epochs, and select an optimizer.
[0073] Specifically, the batch size is set according to the GPU memory capacity. If the memory is insufficient, a gradient accumulation strategy is adopted, updating the parameters every 2-4 batches to effectively increase the batch size. For example, it is set to 16; the initial learning rate is set to 1e-4 to 5e-4; the decay strategy uses cosine annealing decay, gradually reducing the learning rate in the later stages of training to improve convergence stability; or a step decay is used, reducing the learning rate to 1 / 10 of its original value every 10-20 rounds; the number of training rounds is set to 30-100 rounds, combined with an early stopping strategy, terminating training when the validation set loss does not decrease for 5-8 consecutive rounds to avoid overfitting; the AdamW optimizer is used, with the weight decay coefficient set to 1e-5, which suppresses weight overfitting while optimizing the gradient. The decay rates β1 and β2 of the first and second moments of the gradient are set to 0.9 and 0.999, respectively, balancing the momentum of the gradient with the adaptive learning rate characteristics.
[0074] If the focus parameter prediction model is used to perform a classification task, a weighted cross-entropy loss function is used, with each class weighted by 1 / N to balance the number of samples from different classes. The loss function calculates the difference between the probability distribution predicted by the model and the true class number.
[0075] If the focus parameter prediction model is used to perform a regression task, a smooth L1 loss function is used with a smoothing coefficient of 0.001. This loss function is insensitive to outliers and is beneficial for the stable convergence of normalized parameter values in regression tasks.
[0076] During training, the validation set loss and accuracy (for classification tasks) or mean squared error (for regression tasks) are monitored in real time. When the validation set performance no longer improves, the trained focus parameter prediction model is obtained and deployed to the XR headset.
[0077] It should be noted that when building a focus parameter prediction model, whether the prediction layer is set as a classification prediction layer (used to perform classification tasks and output focus category numbers) or a regression prediction layer (used to perform regression tasks and output normalized parameter values), it is applicable to various types of video perspective scenes.
[0078] Specifically, for low-end cameras that only support discrete focus levels, the focus category number can be directly output through classification prediction (the total number of categories can be set according to the number of hardware focus levels), or continuous parameter values can be output through regression prediction, and then the continuous value can be mapped to the nearest hardware focus level through nearest neighbor quantization or interval judgment. The latter method can sometimes obtain smoother prediction results, especially when the training data sampling density is high.
[0079] For high-end cameras that support continuous focus, normalized parameter values can be directly output through regression prediction, or "virtual focus category numbers" can be output through classification prediction. The total number of categories corresponding to this number can be much larger than the actual number of hardware levels (for example, N is 50 during training, but the hardware only supports 10 levels). During inference, the output number is first converted into a continuous offset value, then mapped to the actual parameter range, and finally the hardware interface is called for continuous adjustment.
[0080] Therefore, in practical applications, the type of prediction layer can be flexibly selected by comprehensively considering factors such as the convenience of the camera's underlying driver, the accuracy of the training data annotation, the computing power for model deployment, and the requirements for prediction smoothness. It's even possible to train both classification and regression models simultaneously on the same device and dynamically switch between them based on runtime strategies. This flexibility adapts to various practical engineering needs, significantly enhancing the ability to adapt to different hardware configurations and performance requirements.
[0081] S4. Adjust the video perspective camera according to the target focus parameters to complete a single focus.
[0082] This step is used to convert the target focus parameters into actual executable hardware instructions for the video perspective camera, and drive the camera to complete the focusing in one go.
[0083] Specifically, if the target focus parameter is the focus category number cls, and the total number of focus categories is a preset number N, then the actual focus parameter is minParam + cls × (maxParam - minParam) / N; thus achieving a linear mapping from discrete categories to actual parameter values.
[0084] If the target focus parameter is a normalized parameter value reg, then the actual focus parameter is minParam + reg × (maxParam - minParam); this achieves linear scaling of the normalized parameter value to the actual parameter range of the camera for continuous stepless adjustment.
[0085] The actual focus parameters are sent to the video perspective camera. The driver module of the video perspective camera calls the underlying interface to perform the focusing operation based on the actual focus parameters. These parameters are already the optimal focus parameters, so focusing can be completed with only one adjustment, without the need for repeated attempts.
[0086] In practice, steps S1 to S4 are executed independently for each of the two cameras on the XR headset. This involves acquiring the current images from the left and right cameras, acquiring the gaze coordinates of the eye-tracking point (which is typically a unified gaze point in the screen coordinate system but can be mapped to the image coordinates of the left and right cameras respectively), predicting the optimal target focus parameters for each camera, and driving each camera to focus. Since the left and right cameras capture different perspectives of the same scene, their optimal target focus parameters are usually similar but may differ slightly due to parallax. Adjusting these parameters separately yields optimal binocular consistent sharpness. The method of this embodiment is executed cyclically during XR headset operation, updating the focus state in real time at a frequency of 60Hz or higher to ensure the user always receives a clear perspective image when their head moves or their gaze changes.
[0087] Compared to existing technologies, this embodiment provides an automatic focusing method for video perspective cameras in XR headsets. By acquiring the current image and depth map of the video perspective camera, combined with the user's gaze coordinates obtained through eye tracking, the depth map of the gaze region is extracted. A pre-trained focus parameter prediction model is then used to directly map the target focus parameters, allowing for a single camera adjustment to achieve focus. This method eliminates the need for repeated trials and adjustments required by contrast-based focusing, and avoids explicit depth-focal length physical modeling using depth sensors or binocular vision. The end-to-end mapping from image to focus parameters is completed in a single forward inference, resulting in fast focusing speed, adaptability to moving targets and dynamic scenes, and significantly improved real-time performance and user experience for XR headset video perspective. Using the gaze region depth map and gaze coordinates as input, a Transformer-based model captures the spatial dependencies between different depth regions. Simultaneously, gaze information is incorporated as a global condition into the feature representation of each image block, enabling the model to accurately focus on the user's attention area, avoiding background interference, and thus achieving higher focus parameter prediction accuracy. The prediction layer of the focus parameter prediction model can be flexibly set as either a classification prediction layer or a regression prediction layer. The two modes can be flexibly selected according to the camera hardware characteristics and system requirements, and can be used in combination under certain conditions, which greatly enhances the versatility and engineering adaptability of the autofocus method.
[0088] Example 2 Another embodiment of the present invention discloses an autofocus system for a video see-through camera in an XR headset, thereby implementing the autofocus method for a video see-through camera in an XR headset described in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes: The image acquisition module is used to acquire the current image and its corresponding depth map captured by the video perspective camera of the XR headset; The gaze region extraction module is used to extract a local region centered on the gaze point coordinates from the depth map as the gaze region depth map based on the user's gaze point coordinates in the current image. The parameter prediction module is used to input the depth map of the gaze region and the coordinates of the gaze point into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera. The focus control module is used to adjust the video perspective camera according to the target focus parameters to complete a single focus.
[0089] Since the video perspective camera autofocus system for XR headsets described in this embodiment and the aforementioned video perspective camera autofocus method for XR headsets are related and can be mutually referenced, this description is redundant and will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.
[0090] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0091] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. An autofocus method for a video see-through camera used in XR headsets, characterized in that, Includes the following steps: Acquire the current image captured by the XR headset's video perspective camera and its corresponding depth map; Based on the user's gaze point coordinates in the current image, a local region centered on the gaze point coordinates is extracted from the depth map as a gaze region depth map. The depth map of the gaze region and the coordinates of the gaze point are input into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera. The video perspective camera is adjusted according to the target focus parameters to complete a single focus.
2. The automatic focusing method for a video perspective camera for an XR head-mounted display according to claim 1, characterized in that, The focus parameter prediction model is a neural network model based on the Transformer architecture, which includes, in sequence: an image patch embedding layer, a position encoding layer, multiple encoder layers, and a prediction layer; wherein, each encoder layer includes: a self-attention mechanism module and a feedforward network module.
3. The automatic focusing method for a video perspective camera for an XR headset according to claim 2, characterized in that, The image patch embedding layer is used to divide the depth map of the gaze region into multiple image patches and encode each image patch into a feature vector; the position encoding layer is used to obtain the position encoding vector of each image patch and the position encoding vector of the gaze point coordinates through a position encoding algorithm; the output of the image patch embedding layer and the output of the position encoding layer are concatenated into a fusion vector, which is then passed to multiple stacked encoder layers; each encoder layer models the feature relationship of the fusion vector through a self-attention mechanism module, and performs nonlinear transformation and dimension adjustment through a feedforward network module; the prediction layer is used to map the output of the last encoder layer into target focus parameters.
4. The automatic focusing method for a video perspective camera for an XR headset according to claim 2, characterized in that, The prediction layer of the focus parameter prediction model is either a classification prediction layer or a regression prediction layer. The classification prediction layer includes a first multi-layer fully connected network and a Softmax layer, used to output the focus category number; the regression prediction layer includes a second multi-layer fully connected network and a linear mapping layer, used to output parameter values normalized to the [0,1] interval.
5. The automatic focusing method for a video perspective camera for an XR head-mounted display according to claim 1, characterized in that, During the training of the focus parameter prediction model, a sample set is constructed by iteratively executing acquisition operations, wherein each acquisition operation includes: The user wears an XR headset and gazes at the observation area, keeping the gaze point stationary. The system iterates through the focus parameters of the video perspective camera and captures corresponding training images. It calculates the image sharpness of the gaze region in the training image based on the coordinates of the corresponding gaze point. After the iteration is complete, it obtains the index number of the focus parameter with the highest image sharpness as the annotation result of this acquisition operation. Obtain the depth map of each training image, and extract the gaze region depth map from the depth map based on the gaze point coordinates; combine the gaze region depth map, gaze point coordinates, and the annotation results of the acquisition operation of each training image into a sample and put it into the sample set.
6. The automatic focusing method for a video perspective camera for an XR headset according to claim 5, characterized in that, The traversal generates the focus parameters of the video perspective camera, including: Based on a preset number N, N discrete values are uniformly selected between the minimum and maximum values of the focus parameters of the video perspective camera, and each is assigned an index number from 0 to N-1. The discrete value corresponding to each index number is used as the focus parameter generated during the traversal in this acquisition operation.
7. The automatic focusing method for a video perspective camera for an XR headset according to claim 5, characterized in that, The image sharpness of the gaze region in the training image is obtained by calculating the average sharpness score of the gaze region using the Sobel operator or the Laplacian operator.
8. The automatic focusing method for a video perspective camera for an XR head-mounted display according to claim 1, characterized in that, The step of adjusting the video perspective camera according to the target focus parameters includes: Obtain the minimum and maximum values of the focus parameters minParam and maxParam of the video perspective camera; If the target focus parameter is the focus category number cls, and the total number of focus categories is a preset number N, then the actual focus parameter is minParam + cls × (maxParam - minParam) / N; If the target focusing parameter is a normalized parameter value reg, then the actual focusing parameter is minParam + reg × (maxParam - minParam); The actual focus parameters are sent to the video perspective camera to complete the focusing process.
9. The automatic focusing method for a video perspective camera for an XR headset according to claim 1, characterized in that, The local area is either a square area or a circular area. The side length of the square area and the radius of the circular area are calculated from the short side length of the current image according to different preset ratios.
10. An autofocus system for a video see-through camera used in XR headsets, characterized in that, include: The image acquisition module is used to acquire the current image and its corresponding depth map captured by the video perspective camera of the XR headset; The gaze region extraction module is used to extract a local region centered on the gaze point coordinates from the depth map as a gaze region depth map based on the user's gaze point coordinates in the current image. The parameter prediction module is used to input the depth map of the gaze region and the coordinates of the gaze point into the trained focus parameter prediction model to obtain the target focus parameters of the video perspective camera. The focus control module is used to adjust the video perspective camera according to the target focus parameters to complete a single focus.