Exposure time adjustment method and device for a positioning head-mounted display device

Abstract

This application relates to the field of VR / AR technology, and provides a method and apparatus for adjusting the exposure time of a positioning head-mounted display device to improve the accuracy and response speed of the interaction between the head-mounted display device and the controller. Since the first exposure time of the positioning head-mounted display device and the second exposure time of the positioning controller alternate periodically, the length of the first exposure time affects the response efficiency and accuracy of the interaction. Therefore, this method dynamically adjusts the first exposure time in three stages according to the actual usage environment, thereby reducing the interference of environmental factors on the first exposure time, improving the positioning accuracy of the head-mounted display device, and thus improving the accuracy of the interaction. Furthermore, by weighting the accumulated first exposure time, the changes in subsequent first exposure times are made more stable, avoiding the impact of an excessively long first exposure time on the second exposure time, thus improving both positioning accuracy and interaction response speed.

Description

Technical Field

[0001] This application relates to the field of Virtual Reality (VR) / Augmented Reality (AR) technology, and provides a method and device for adjusting the exposure time of a positioning head-mounted display device. Background Technology

[0002] With the development of VR and AR technologies, head-mounted display devices (HMDs) such as VR glasses and AR glasses have been widely used in various industries, such as education and training, fire drills, virtual driving, and real estate.

[0003] Currently, in VR / AR interactive scenarios, when performing real-time localization and tracking of moving targets within a certain spatial range, it is also necessary to simultaneously locate the head-mounted display device. The localization of the moving head-mounted display device typically employs Simultaneous Localization and Mapping (SLAM) technology, and the localization results of the head-mounted display device affect the localization and tracking results of the moving target, thus impacting the interaction quality. Therefore, head-mounted display localization has become a primary technology in VR / AR interactive scenarios.

[0004] When simultaneously locating a head-mounted display and a moving target, to achieve SLAM positioning of the head-mounted display and thus quickly and accurately track the moving target, it is necessary to ensure the accuracy of real-time acquisition of user environmental information by the camera on the head-mounted display. However, in practical applications, the camera not only needs to use long exposures to capture environmental images for positioning the head-mounted display, but also needs to switch to short exposures to capture light ring spots for positioning the handle due to optical positioning. This makes it difficult to control the long exposure time for capturing environmental images. Excessive long exposure time will affect the interaction speed, while insufficient long exposure time may result in insufficient environmental information acquisition. Furthermore, the environmental images captured by long exposure are also greatly affected by environmental factors (such as light intensity). All of these factors will affect the positioning accuracy of the head-mounted display and thus the interaction results.

[0005] Therefore, adjusting the exposure time of environmental images to ensure the positioning accuracy and efficiency of head-mounted display devices is an important aspect of VR / AR interactive scenarios. Summary of the Invention

[0006] This application provides a method and apparatus for adjusting the exposure time of a positioning head-mounted display device, which is used to improve the positioning accuracy and efficiency of the head-mounted display device.

[0007] On one hand, this application provides a method for adjusting the exposure time of a positioning head-mounted display device, applied to the head-mounted display device. The camera on the head-mounted display device is configured with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is used to acquire an environmental image for positioning the head-mounted display device, and the second exposure time is used to acquire an image of a light ring spot for positioning a handle. The handle interacts with the head-mounted display device. The first exposure time is greater than the second exposure time. The method includes:

[0008] The actual exposure situation is obtained by acquiring the current environmental image based on the initial exposure value set for the first exposure time, and the initial exposure value of the subsequent environmental images exposed n times at the first exposure time is adjusted according to the actual exposure situation using a preset time interval, where n is an integer greater than 1.

[0009] The initial exposure values ​​of the first exposure time are weighted to obtain the (n+1)th first exposure time;

[0010] After a preset exposure time, the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device are updated for each region to calculate the subsequent first exposure time.

[0011] On the other hand, embodiments of this application provide a head-mounted display device, including a processor, a memory, a display, a communication interface, and a camera, wherein the camera, the communication interface, the display, the memory, and the processor are connected via a bus;

[0012] The camera is set with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is used to acquire an environmental image for positioning the head-mounted display device, and the second exposure time is used to acquire an image of a light ring spot for positioning the handle. The first exposure time is longer than the second exposure time.

[0013] The communication interface is used to interact with the gamepad, and the gamepad is used to control the screen displayed on the monitor.

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] The actual exposure situation is obtained by acquiring the current environmental image based on the initial exposure value set for the first exposure time, and the initial exposure value of the subsequent environmental images exposed n times at the first exposure time is adjusted according to the actual exposure situation using a preset time interval, where n is an integer greater than 1.

[0016] The initial exposure values ​​of the first exposure time are weighted to obtain the (n+1)th first exposure time;

[0017] After a preset exposure time, the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device are updated for each region to calculate the subsequent first exposure time.

[0018] Optionally, the processor adjusts the initial exposure value of subsequent environmental images exposed n times with the first exposure time based on the actual exposure conditions and using a preset time interval. Specifically, the operation is as follows:

[0019] If the actual exposure is underexposed, the sum of the initial exposure value of the current environmental image and the preset step size is used as the initial exposure value of the environmental image for subsequent exposures of the first exposure time n times.

[0020] If the actual exposure is overexposed, the difference between the initial exposure value of the current environmental image and the preset step size is used as the initial exposure value for subsequent exposures of the environmental image n times with the first exposure time.

[0021] Wherein, the preset step size is less than or equal to the time interval.

[0022] Optionally, the processor weights the n initial exposure values ​​of the first exposure time to obtain the (n+1)th first exposure time, specifically by:

[0023] Obtain the first average exposure value of the n initial exposure values, and the second average exposure value of the m subsequent initial exposure values ​​of the n initial exposure values;

[0024] The first exposure mean and the second exposure mean are weighted according to the first weight and the second weight to obtain the initial exposure value for the (n+1)th exposure.

[0025] Based on the coefficient representing environmental changes and the camera's frame rate, the initial exposure value for the (n+1)th exposure is adjusted to obtain the first exposure time for the (n+1)th exposure.

[0026] Optionally, the formula for calculating the first exposure time of the (n+1)th exposure is:

[0027]

[0028] Among them, a n For the first weighted weight, β n This is the second weighting weight. The average value of the first exposure. γ is the second average exposure value, γ is the coefficient of the environmental change, and f is the frame rate of the camera.

[0029] Optionally, the formulas for determining the first weighted weight and the second weighted weight are as follows:

[0030]

[0031] α n =1-β n

[0032] Among them, t n-1 Let t be the initial exposure value of the environmental image exposed for the (n-1)th time with the first exposure time. n-2 Let t0 be the initial exposure value of the environmental image exposed for the (n-2)th time with the first exposure time, and β be the initial exposure value of the current environmental image. n-1 β is the second weighting weight for the (n-1)th iteration. n For the second weighting in the nth iteration, α n This is the first weighted weight for the nth iteration.

[0033] Optionally, after an exposure of a preset duration, the processor updates the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device, specifically as follows:

[0034] After exposure for a preset duration, a 3D map representing the motion trajectory of the head-mounted display device is established based on the pose of the head-mounted display device determined from the collected environmental images.

[0035] Based on the actual environmental space, the 3D map is divided into regions, and for each region, the following operations are performed:

[0036] Calculate the average time of the first exposure within the preset duration and the weighted average of the weighted weights;

[0037] The average time and the previous first exposure time are weighted according to the weighted average value, and the weighted result is adjusted with a coefficient representing environmental change and the frame rate of the camera to obtain the next first exposure time of the environmental image.

[0038] Optionally, the formula for calculating the next first exposure time within each region is:

[0039]

[0040] Where a0 is the average weight of the first weighted weights of the n first exposure times within the preset duration, β nγ is the average weight of the second weighted weight of m first exposure times within the preset duration, γ is the coefficient of the environmental change, and f is the frame rate of the camera.

[0041] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a method for adjusting the exposure time of a positioning head-mounted display device provided in embodiments of this application.

[0042] The beneficial effects of the exposure time adjustment method for positioning head-mounted display devices provided in this application embodiment are as follows:

[0043] During the interaction between the head-mounted display device and the controller, a first exposure time is set for the camera used to position the head-mounted display device and the controller to acquire an environmental image for positioning the head-mounted display device, and a second exposure time is set for acquiring an image of the light ring spot for positioning the controller. The first exposure time is longer than the second exposure time, and the first and second exposure times alternate periodically. Thus, the length of the first exposure time affects the positioning efficiency of the head-mounted display device, which in turn affects the response speed of the interaction between the head-mounted display device and the controller, as well as the positioning accuracy of the head-mounted display device, which in turn affects the accuracy of the interaction between the head-mounted display device and the controller. Therefore, this embodiment dynamically adjusts the first exposure time in three stages based on the actual exposure situation. First, based on the current environmental image collected according to the initial exposure value set for the first exposure time, the actual exposure situation is obtained to adjust the initial exposure value of the subsequent environmental images exposed n times with the first exposure time. This allows the exposure value of the subsequent environmental images to be dynamically adjusted within a preset time interval based on the initial exposure value, thereby reducing interference from environmental factors and improving the positioning accuracy of the head-mounted display device. Second, the n initial exposure values ​​of the first exposure time are weighted to obtain the (n+1)th first exposure time, making the change of the first exposure time more stable and avoiding the impact of an excessively long first exposure time on the second exposure time. This improves both positioning accuracy and interaction response speed. Finally, after an exposure of a preset duration, the initial exposure value and weighting weight used to calculate the subsequent first exposure time are updated for the divided regions of the 3D map representing the motion trajectory of the head-mounted display device. This saves the resource consumption of adjusting the first exposure time, thereby improving positioning efficiency and, consequently, interaction response speed.

[0044] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram illustrating the interaction between the handle and the head-mounted display device provided in an embodiment of this application.

[0047] Figure 2 This is a schematic diagram of the structure of the camera and handle LED light ring of the head-mounted display device provided in the embodiments of this application;

[0048] Figure 3 This is a schematic diagram of a camera exposure method provided in an embodiment of this application;

[0049] Figure 4 A flowchart illustrating a method for adjusting the exposure time of a positioning head-mounted display according to an embodiment of this application;

[0050] Figure 5A A schematic diagram illustrating an adjustment of the first exposure time based on actual exposure conditions, provided as an embodiment of this application;

[0051] Figure 5B A schematic diagram illustrating another adjustment of the first exposure time based on actual exposure conditions, provided for an embodiment of this application;

[0052] Figure 5C A schematic diagram illustrating another adjustment of the first exposure time based on actual exposure conditions, provided for an embodiment of this application;

[0053] Figure 6 A flowchart illustrating a method for adjusting the first exposure time in the first stage, as provided in an embodiment of this application.

[0054] Figure 7 A flowchart illustrating a method for second-stage adjustment of the first exposure time provided in an embodiment of this application;

[0055] Figure 8 A flowchart illustrating a method for adjusting the first exposure time in a third stage, as provided in an embodiment of this application;

[0056] Figure 9 This is an overall architecture diagram of SLAM spatial motion positioning for head-mounted display devices provided in an embodiment of this application;

[0057] Figure 10 This is a structural diagram of a head-mounted display device provided in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0059] Currently, interaction methods for head-mounted display devices include head control and handheld control. Handheld control of head-mounted display devices is as important as mouse control of a PC. For example... Figure 1 The diagram shows the interaction between the controller and the head-mounted display device. The left and right controllers may correspond to different distances and angles from the head-mounted display device. The user interacts with the head-mounted display device by moving the cursor on the application interface through the ray emitted by the controller. During the interaction, the user can switch between the left and right controllers used for interaction.

[0060] Typically, head-mounted display devices have at least two external cameras, and the handle has an external ring of multiple LEDs that flash at a fixed frequency, such as... Figure 2 As shown, simultaneous positioning of the head-mounted display and the controller is achieved through environmental images and light ring spot images captured by the camera. Therefore, in practical applications, the camera not only needs to use long exposures to capture environmental images for positioning the head-mounted display, but also needs to switch to short exposures to capture light ring spot images for positioning the controller. This makes it difficult to control the long exposure time for capturing environmental images; excessively long exposure times will affect the interaction speed, while excessively short exposure times may result in insufficient environmental information collection. Furthermore, the environmental images captured by long exposures are significantly affected by environmental factors (such as light intensity). All of these factors will affect the positioning accuracy of the head-mounted display, and consequently, the interaction results.

[0061] In view of this, this application provides a method for adjusting the exposure time of a positioning head-mounted display device. In actual use environment, the method adjusts the long exposure time of the camera in three stages based on the historical data of the long exposure time of the positioning head-mounted display device. This ensures that the environmental image captured by the camera contains rich environmental information and reduces the interference of environmental factors, thereby improving the positioning accuracy of the head-mounted display device and thus improving the interaction quality between the head-mounted display device and the controller, so that the user can obtain a better immersive experience.

[0062] In this embodiment of the application, for the camera on the head-mounted display device, the long exposure time set for acquiring environmental images for positioning the head-mounted display device is denoted as the first exposure time, and the short exposure time set for acquiring light ring spot images for positioning the handle is denoted as the second exposure time. The first and second exposure times alternate periodically, with the first exposure time being longer than the second exposure time. Figure 3 As shown.

[0063] When simultaneously locating a head-mounted display (HUD) and a controller using images captured by a camera, the initial exposure duration of the environmental image used for HUD positioning affects the response speed of the interaction between the HUD and the controller. Furthermore, the initial exposure duration determines the amount of environmental information in the image; insufficient environmental information reduces the positioning accuracy of the HUD, thus affecting the pose of the controller relative to the HUD and consequently the accuracy of the interaction between the HUD and the controller. Therefore, the length of the initial exposure duration has a significant impact on the interaction quality between the controller and the HUD. Consequently, this embodiment focuses on adjusting the initial exposure time for locating the HUD to improve the positioning accuracy and efficiency of SLAM for the HUD.

[0064] See Figure 4 This is a flowchart of a method for adjusting the exposure time of a positioning head-mounted display according to an embodiment of this application. The process is executed by the head-mounted display device and mainly includes the following steps:

[0065] S401: Based on the current environmental image acquired according to the initial exposure value set for the first exposure time, obtain the actual exposure situation, and adjust the initial exposure value of the subsequent environmental images exposed n times at the first exposure time according to the actual exposure situation using a preset time interval.

[0066] The preset exposure value is set according to the camera's frame rate. For example, if the camera's frame rate f is 25Hz, then the initial exposure value for the first exposure time is...

[0067] In practical applications, factors such as the shooting environment and shooting angle may cause overexposure, image shift, and deviation. Therefore, after setting the initial exposure value for the first exposure time, the camera acquires a frame of the environment image using this initial exposure value and performs exposure detection on the current environment image to determine whether there is overexposure or underexposure. This allows for dynamic adjustment of the first exposure time. The adjusted first exposure time may be the same or different in different periods. Figures 5A to 5C As shown, when the first exposure time is different, the value of the first exposure time fluctuates within a certain range in different periods.

[0068] In practice, the dynamic adjustment process of the first exposure time in S401 is the first stage, see [link to relevant documentation]. Figure 6 The main steps include the following:

[0069] S4011: Determine the type of actual exposure based on the current environmental image. If it is underexposed, execute S4012; if it is overexposed, execute S4013.

[0070] When the current environment image acquired with the initial exposure value is underexposed, it indicates that the first exposure time of the current environment image is too short, which may lead to insufficient environmental information acquisition and thus affect the positioning accuracy of the head-mounted display device. Therefore, it is necessary to increase the first exposure time of subsequent environment images to enrich the environmental information. When the current environment image acquired with the initial exposure value is overexposed, it indicates that the first exposure time of the current environment image is too long, which will increase the positioning time of the head-mounted display device and thus increase the interaction response speed between the head-mounted display device and the controller. Therefore, it is necessary to reduce the first exposure time of subsequent environment images to improve positioning efficiency.

[0071] S4012: Use the sum of the initial exposure value of the current environmental image and the preset step size as the initial exposure value of the subsequent environmental images exposed n times with the first exposure time.

[0072] When the actual exposure is underexposed, the initial exposure value of the environmental image exposed at the first exposure time can be increased by a preset step size based on the initial exposure value of the current environmental image. The preset step size is less than or equal to the time interval, and n is an integer greater than 1.

[0073] S4013: Use the difference between the initial exposure value of the current environmental image and the preset step size as the initial exposure value of the subsequent environmental images exposed n times with the first exposure time.

[0074] When the actual exposure is overexposed, the initial exposure value of the environmental image exposed at the first exposure time can be reduced by a preset step size based on the initial exposure value of the current environmental image, wherein the preset step size is less than or equal to the time interval.

[0075] For example, suppose the initial exposure value of the current environment image is acquired at the first exposure time. The time interval Δt = 1ms is the initial exposure value t of the environmental images that are subsequently exposed n times at the first exposure time when the actual exposure of the current environmental image is underexposed. i =5ms (i=1,2,...,n), when the actual exposure of the current environmental image is overexposed, the initial exposure value t of the subsequent environmental images exposed n times with the first exposure time. i =3ms(i=1,2,...,n).

[0076] In some embodiments, the initial exposure value t of the environmental image subsequently exposed n times at the first exposure time i The steps can be different, meaning they can be adjusted gradually with a preset step size. The step size for each adjustment can be the same or different.

[0077] For example, when the actual exposure of the current environmental image is underexposed, the initial exposure value t of the subsequent environmental images exposed for the first exposure time n1 times is... i =4.5ms (i = 1, 2, ..., n1), then, the environmental image acquired with the initial exposure value of n1 at the first exposure time is used as the new current environmental image, and the actual exposure situation is obtained again. If the actual exposure situation of the new current environmental image is overexposed, the initial exposure value t of the subsequent environmental images with the first exposure time n2 is changed. i =4.2ms (i=n2, n2+1,..., n, n1+n2=n).

[0078] In the first stage of the first exposure time adjustment embodiment of this application, the actual exposure situation is determined by acquiring the current environmental image with the initial exposure value of the first exposure time. This allows the initial exposure value of the subsequent environmental images exposed n times with the first exposure time to be adjusted. As a result, the first exposure time is dynamically adjusted within the range of t0±Δt during the interaction between the head-mounted display device and the handle, as the shooting environment changes. This reduces the interference of environmental factors and improves the positioning accuracy of the head-mounted display device.

[0079] S402: Weight the n initial exposure values ​​of the first exposure time to obtain the (n+1)th first exposure time.

[0080] After obtaining n initial exposure values ​​of the environmental image acquired at the first exposure time, the n initial exposure values ​​can be used as historical data and divided into long-term historical data and short-term historical data, thereby enabling a second-stage dynamic adjustment of the first exposure time.

[0081] In specific implementation, the process of adjusting the first exposure time in the second stage in S402 is described in [reference needed]. Figure 7 The main steps include the following:

[0082] S4021: Obtain the first average exposure value of n initial exposure values, and the second average exposure value of the m subsequent initial exposure values ​​of the n initial exposure values.

[0083] In one example, the n initial exposure values ​​of the first exposure time of the environmental image are used as long-term historical data, and the average first exposure value of the long-term historical data is calculated. Furthermore, taking the m (m < n) initial exposure values ​​after the n initial exposure values ​​as short-term historical data, the second exposure mean of the short-term historical data is calculated.

[0084] S4022: Based on the first weighted weight and the second weighted weight, the first exposure mean and the second exposure mean are weighted to obtain the initial exposure value for the (n+1)th exposure.

[0085] In one example, the first weighted weight α of the first exposure mean. n And the second weighted weight β of the second exposure mean n This is a dynamically changing weighted index. Specifically, the second weighting weight β of the second exposure mean... n It will be based on the second weighted weight β from the previous time. n-1 And the initial exposure values ​​t from the previous two exposures. n-1 t n-2 The ratio of the deviation value to t0 changes dynamically.

[0086] Specifically, the first weighting weight and the second weighting weight are determined as follows:

[0087]

[0088] α n =1-β n Formula 2

[0089] Among them, t n-1 Let t be the initial exposure value of the environmental image exposed for the (n-1)th time with the first exposure time. n-2 Let t0 be the initial exposure value of the environmental image exposed for the (n-2)th time with the first exposure time, and β be the initial exposure value of the current environmental image. n-1 β is the second weighting weight for the (n-1)th iteration. n For the second weighting in the nth iteration, α n The first weighted weight is used for the nth iteration. The weighted result of the first weighted weight and the second weighted weight is:

[0090] S4023: Adjust the initial exposure value for the (n+1)th exposure based on the coefficient representing environmental changes and the camera's frame rate to obtain the first exposure time for the (n+1)th exposure.

[0091] During the interaction between the head-mounted display and the controller, the head-mounted display is constantly in motion. At this time, the synchronous localization pressure of SLAM is relatively high, and the construction of a new map model may be carried out during this process, which requires more image data. Moreover, environmental factors such as lighting may also change. Therefore, a dynamic constant representing the environmental changes can be set according to the changes in the shooting environment of the environmental image. Combined with the camera frame rate, the initial exposure value of the environmental image of the (n+1)th exposure acquired at the first exposure time can be adjusted.

[0092] Specifically, the formula for the first exposure time of the (n+1)th environmental image is expressed as:

[0093]

[0094] Among them, a n β is the first weighted weight for the nth iteration. n As the second weighting weight, The first mean, γ is the second mean, f is the coefficient of environmental change, and f is the camera's frame rate.

[0095] In the second stage of the first exposure time adjustment embodiment of this application, during the nth exposure of the light ring spot image to locate and track the handle after the nth exposure of the environmental image by the camera, the first exposure time of the (n+1)th exposure of the environmental image by the camera is determined. This makes the change of the first exposure time more stable, ensuring the real-time acquisition of environmental information by the camera and avoiding the impact of an excessively long first exposure time on the subsequent second exposure time. This allows for more accurate positioning of the handle and improves the efficiency of the head-mounted display device, ensuring the accuracy and response speed of the interaction between the head-mounted display device and the handle. Furthermore, by dynamically correcting the first exposure time using a coefficient γ that characterizes environmental changes, more image data can be acquired when SLAM builds a new map model, thereby completing map building and synchronous positioning faster and more accurately. This further improves the positioning accuracy and efficiency of the head-mounted display device, and thus further improves the accuracy and response speed of the interaction between the head-mounted display device and the handle.

[0096] S403: After an exposure of a preset duration, update the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device for calculating the subsequent first exposure time.

[0097] After accumulating sufficient positioning data over a certain period of time, a 3D map representing the motion trajectory of the head-mounted display device can be obtained using SLAM. Considering that the head-mounted display device may pass through different environmental areas (such as bedrooms, living rooms, and studies) during the interaction process, the 3D map constructed by SLAM can be divided into regions, and the first exposure time can be dynamically adjusted in the third stage by updating the weighted weights and initial exposure values ​​of each region.

[0098] In specific implementation, the process of adjusting the first exposure time in the third stage in S403 is described in [reference needed]. Figure 8 The main steps include the following:

[0099] S4031: After exposure for a preset duration, a 3D map representing the motion trajectory of the head-mounted display device is established based on the pose of the head-mounted display device determined by the acquired environmental images.

[0100] In one example, the pose of the head-mounted display device is a 6-degree-of-freedom (DOF) pose, which includes the translation vector of the 3D head-mounted display device's position in three-dimensional space and the rotation matrix of the 3D head-mounted display device's orientation in three-dimensional space. Therefore, the pose of the head-mounted display device can characterize the motion trajectory of the head-mounted display device, thereby constructing a 3D map of the head-mounted display device.

[0101] S4032: Divide the 3D map into regions based on the actual environmental space.

[0102] During the interaction between the head-mounted display device and the controller, the head-mounted display device may pass through different spatial areas, such as bedrooms, living rooms, and studies. The environments (such as lighting) of these spatial areas may be different. Therefore, the first exposures of each spatial area may also be different. Thus, the 3D map constructed by SLAM can be divided into regions based on the actual environmental space passed by the head-mounted display device.

[0103] S4033: For each region, calculate the average time of the first exposure time of that region within a preset duration and the weighted average of the weighted weights.

[0104] S4034: Based on the weighted mean, the average time and the previous first exposure time are weighted, and the weighted result is adjusted using coefficients representing environmental changes and the camera's frame rate to obtain the next first exposure time of the environmental image.

[0105] Specifically, the formula for the next exposure time of the environmental image within each region is expressed as:

[0106]

[0107] Where a0 is the first weighted weight α of the n first exposure times within the preset duration. n The weighted mean, β n The second weighted weight β is the first exposure time of m times within a preset duration. n The weighted mean is γ, where γ is the coefficient of environmental change and f is the camera's frame rate.

[0108] In the third stage of the first exposure time adjustment in this embodiment, after an exposure of a preset duration, the weighted average of the first exposure time for exposing the environmental image in each region divided in the 3D map is calculated. The weighted average is then used to weight the average of the first exposure time in the corresponding region with the previous first exposure time. Compared with the second adjustment stage, this saves computational resources for the first exposure time adjustment, further improves positioning efficiency, and thus improves the interaction response speed of the head-mounted display device and the controller.

[0109] See Figure 9 This document presents an overall architecture diagram for SLAM spatial motion positioning of a head-mounted display device, as provided in an embodiment of this application. In the first stage, the first exposure time of subsequent n environmental images is adjusted based on actual exposure conditions, and exposure data is accumulated. When the accumulated exposure data does not reach a preset duration, a second stage adjustment is performed on the first exposure time. In this second stage, the first exposure time of the (n+1)th exposure is calculated using a weighted average of the n first exposure times (long-term historical data) and the subsequent m (m < n) first exposure times (short-term historical data). When the accumulated exposure data reaches the preset duration, a third stage adjustment is performed on the first exposure time. In this third stage, the 3D map constructed by SLAM is divided into regions. For each region, the average time and weighted average of the first exposure times within a preset duration are calculated. The next first exposure time is then calculated using a weighted average of the average first exposure time within the preset duration and the previous first exposure time. After the first exposure time adjustment, the user's environmental information is exposed according to the final first exposure time to obtain an environmental image. The pose of the head-mounted display device is then calculated based on the environmental image, achieving SLAM spatial motion positioning.

[0110] In the embodiments of this application, the dynamic adjustment of the first exposure time of the environmental image is optimized in three stages according to the actual usage environment, so as to quickly and accurately perform SLAM spatial motion positioning of the head-mounted display device, thereby improving the response speed and accuracy of the interaction between the head-mounted display device and the controller, and effectively enhancing the user experience.

[0111] Based on the same technical concept, this application provides a head-mounted display device, which can be a virtual reality device or an augmented reality device. The head-mounted display device can implement the steps of the above-mentioned method for adjusting the exposure time of a positioning head-mounted display device and achieve the same technical effect.

[0112] See Figure 10 The head-mounted display device includes a processor 1001, a memory 1002, a display 1003, a communication interface 1004, and a camera 1005. The camera 1005, the communication interface 1004, the display 1003, the memory 1002, and the processor 1001 are connected via a bus 1006.

[0113] Camera 1005 is set with a first exposure time and a second exposure time. The first exposure time and the second exposure time alternate periodically. The first exposure time is used to acquire environmental images for positioning the head-mounted display device, and the second exposure time is used to acquire light spot images of the light ring for positioning the handle. The first exposure time is longer than the second exposure time.

[0114] The communication interface 1004 is used to interact with the gamepad, which is used to control the screen displayed on the display 1003;

[0115] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0116] The actual exposure situation is obtained by acquiring the current environmental image based on the initial exposure value set for the first exposure time, and the initial exposure value of the subsequent environmental images exposed n times at the first exposure time is adjusted according to the actual exposure situation using a preset time interval. n is an integer greater than 1.

[0117] We weight the n initial exposure values ​​for the first exposure time to obtain the (n+1)th first exposure time;

[0118] After a preset exposure time, the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device are updated for calculation of the subsequent first exposure time.

[0119] Optionally, the processor 1001 adjusts the initial exposure value of subsequent environmental images exposed n times with the first exposure time based on the actual exposure conditions and using a preset time interval. Specifically, the operation is as follows:

[0120] If the actual exposure is underexposed, the sum of the initial exposure value of the current environmental image and the preset step size will be used as the initial exposure value of the environmental image for subsequent exposures of the first exposure time n times.

[0121] If the actual exposure is overexposed, the difference between the initial exposure value of the current environmental image and the preset step size will be used as the initial exposure value for subsequent exposures of the environmental image n times with the first exposure time.

[0122] The preset step size is less than or equal to the time interval.

[0123] Optionally, the processor 1001 weights the n initial exposure values ​​of the first exposure time to obtain the (n+1)th first exposure time. The specific operation is as follows:

[0124] Obtain the first average exposure value of n initial exposure values, and the second average exposure value of the m subsequent initial exposure values ​​of the n initial exposure values;

[0125] The first exposure mean and the second exposure mean are weighted according to the first weight and the second weight to obtain the initial exposure value for the (n+1)th exposure.

[0126] Based on the coefficients representing environmental changes and the camera's frame rate, the initial exposure value for the (n+1)th exposure is adjusted to obtain the first exposure time for the (n+1)th exposure.

[0127] Optionally, the formula for calculating the first exposure time of the (n+1)th exposure is:

[0128]

[0129] Among them, a n As the first weighted weight, β n As the second weighting weight, The average value of the first exposure. γ is the average of the second exposure, γ is the coefficient of environmental change, and f is the camera's frame rate.

[0130] Optionally, the formula for determining the first weighting weight and the second weighting weight is as follows:

[0131]

[0132] α n =1-β n

[0133] Among them, t n-1 Let t be the initial exposure value of the environmental image exposed for the (n-1)th time with the first exposure time. n-2 Let t0 be the initial exposure value of the environmental image exposed for the (n-2)th time with the first exposure time, and β be the initial exposure value of the current environmental image. n-1 β is the second weighting weight for the (n-1)th iteration. n For the second weighting in the nth iteration, α n This is the first weighted weight for the nth iteration.

[0134] Optionally, after an exposure of a preset duration, the processor 1001 updates the initial exposure value and weighting weight of each region in the 3D map representing the motion trajectory of the head-mounted display device, specifically as follows:

[0135] After exposure for a preset duration, a 3D map representing the motion trajectory of the head-mounted display device is created based on the pose of the head-mounted display device determined from the collected environmental images.

[0136] Based on the actual environmental space, the 3D map is divided into regions. For each region, the following operations are performed:

[0137] Calculate the average time of the first exposure within the preset duration and the weighted average of the weighted averages;

[0138] The average time and the previous first exposure time are weighted according to the weighted mean, and the weighted result is adjusted by the coefficient representing environmental changes and the camera frame rate to obtain the next first exposure time of the environmental image.

[0139] Optionally, the formula for calculating the next first exposure time within each region is:

[0140]

[0141] Where a0 is the average weight of the first weighted weights of n first exposure times within a preset duration, and β n γ is the average weight of the second weighted weight of m first exposure times within a preset duration, γ is the coefficient of environmental change, and f is the frame rate of the camera.

[0142] The head-mounted display device provided in this application embodiment, during its interaction with the controller, has a first exposure time for capturing an environmental image of the head-mounted display device and a second exposure time for capturing an image of a light ring spot of the controller. The first exposure time is longer than the second exposure time, and the first and second exposure times alternate periodically. Thus, the length of the first exposure time affects the positioning efficiency of the head-mounted display device, thereby affecting the response speed of the interaction between the head-mounted display device and the controller, as well as the positioning accuracy of the head-mounted display device, thereby affecting the accuracy of the interaction between the head-mounted display device and the controller. Therefore, this embodiment dynamically adjusts the first exposure time in three stages based on the actual exposure situation. First, based on the current environmental image collected according to the initial exposure value set for the first exposure time, the actual exposure situation is obtained to adjust the initial exposure value of the subsequent environmental images exposed n times with the first exposure time. This allows the exposure value of the subsequent environmental images to be dynamically adjusted within a preset time interval based on the initial exposure value, thereby reducing interference from environmental factors and improving the positioning accuracy of the head-mounted display device. Second, the n initial exposure values ​​of the first exposure time are weighted to obtain the (n+1)th first exposure time, making the change of the first exposure time more stable and avoiding the impact of an excessively long first exposure time on the second exposure time. This improves both positioning accuracy and interaction response speed. Finally, after an exposure of a preset duration, the initial exposure value and weighting weight used to calculate the subsequent first exposure time are updated for the divided regions of the 3D map representing the motion trajectory of the head-mounted display device. This saves the resource consumption of adjusting the first exposure time, thereby improving positioning efficiency and, consequently, interaction response speed.

[0143] in, Figure 10 The memory 1002 in the memory can be volatile memory, such as random-access memory (RAM); the memory 1002 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 1002 can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-described memories;

[0144] The processor 1001 may include one or more central processing units (CPUs) or digital processing units, etc.

[0145] It should be noted that, Figure 10 This is merely an example illustrating the hardware necessary for a head-mounted display device to perform the exposure time adjustment method steps provided in this application embodiment for positioning a head-mounted display device. Not shown, the head-mounted display device may also include hardware from conventional VR / AR devices such as speakers, microphones, IMUs, and power supplies.

[0146] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a method for adjusting the exposure time of a positioning head-mounted display device as described in the foregoing embodiments.

[0147] This application also provides a computer program product for storing a computer program for executing a method for adjusting the exposure time of a positioning head-mounted display device as described in the foregoing embodiments.

[0148] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for adjusting the exposure time of a positioning head-mounted display device, characterized in that, An application to a head-mounted display device, wherein a camera on the head-mounted display device is configured with a first exposure time and a second exposure time, the first exposure time and the second exposure time alternating periodically, the first exposure time being used to acquire an environmental image for positioning the head-mounted display device, the second exposure time being used to acquire an image of a light ring spot for positioning a handle, the handle interacting with the head-mounted display device, the first exposure time being longer than the second exposure time, the method comprising: In the first stage, based on the current actual exposure situation, the current initial exposure value of the first exposure time is adjusted using a preset time interval to obtain the next initial exposure value of the first exposure time, until a preset duration is reached, and then the second stage of adjustment begins; the actual exposure situation is determined based on the current environmental image collected based on the current initial exposure value; In the second stage, the first exposure mean of the adjusted n initial exposure values ​​and the second exposure mean of the m subsequent initial exposure values ​​of the n initial exposure values ​​are obtained; the first exposure mean and the second exposure mean are weighted according to the first weighting weight and the second weighting weight of the nth exposure, and the weighted result is adjusted with the coefficient representing the environmental change and the frame rate of the camera to obtain the initial exposure value of the (n+1)th exposure time of the first exposure time; After another exposure of a preset duration, the exposure value is adjusted in the third stage. In the third stage, the 3D map representing the motion trajectory of the head-mounted display device is divided into multiple regions. The initial exposure value of the first exposure time in each region is adjusted separately. During each adjustment process: the average time of the n initial exposure values ​​of the first exposure time is calculated, as well as the first weight average of the n first weighted values ​​and the second weight average of the n second weighted values; the average time and the nth first exposure time are weighted according to the first weight average and the second weight average, and the weighted result is adjusted with the coefficient representing environmental changes and the frame rate of the camera to obtain the (n+1)th initial exposure value of the first exposure time. Wherein, the initial exposure value of the (n+1)th exposure time in the second stage. The calculation method is as follows: The initial exposure value of the (n+1)th exposure time in the third stage. The calculation method is as follows: in, This is the first weighted weight during the nth adjustment. This is the second weighting weight during the nth adjustment. The average value of the first exposure. The second exposure average, The coefficient for the aforementioned environmental change. The frame rate of the camera. The first weight mean of the first weighted average in n weighted averages. The second weight mean of the second weighted average in n weighted averages. The average time is represented by n, where n is an integer greater than 1.

2. The method as described in claim 1, characterized in that, The step of adjusting the initial exposure value of the environmental image exposed n times with the first exposure time based on the actual exposure situation using a preset time interval includes: If the actual exposure is underexposed, the sum of the initial exposure value of the current environmental image and the preset step size is used as the initial exposure value of the environmental image for subsequent exposures of the first exposure time n times. If the actual exposure is overexposed, the difference between the initial exposure value of the current environmental image and the preset step size is used as the initial exposure value for subsequent exposures of the environmental image n times with the first exposure time. Wherein, the preset step size is less than or equal to the time interval.

3. The method as described in claim 1 or 2, characterized in that, The formula for determining the first and second weights in the nth adjustment is: in, The initial exposure value of the environmental image exposed for the (n-1)th time with the first exposure time. The initial exposure value of the environmental image exposed for the (n-2)th time with the first exposure time. The preset exposure value for the current environmental image. The second weighting weight for the (n-1)th iteration. The second weighting weight for the nth iteration. This is the first weighted weight for the nth iteration.

4. The method as described in claim 1, characterized in that, The 3D map representing the motion trajectory of the head-mounted display device is divided into multiple regions, including: After exposure in the first and second stages, a 3D map representing the motion trajectory of the head-mounted display device is established based on the pose of the head-mounted display device determined from the collected environmental images. The 3D map is divided into regions based on the actual environmental space.

5. A head-mounted display device, characterized in that, It includes a processor, a memory, a display, a communication interface, and a camera, wherein the camera, the communication interface, the display, the memory, and the processor are connected via a bus; The camera is set with a first exposure time and a second exposure time, which alternate periodically. The first exposure time is used to acquire an environmental image for positioning the head-mounted display device, and the second exposure time is used to acquire an image of a light ring spot for positioning the handle. The first exposure time is longer than the second exposure time. The communication interface is used to interact with the handle, and the handle is used to control the screen displayed on the monitor; In the first stage, based on the current actual exposure situation, the current initial exposure value of the first exposure time is adjusted using a preset time interval to obtain the next initial exposure value of the first exposure time, until the preset duration is reached and then the second stage of adjustment is entered. The actual exposure is determined based on the current environmental image acquired from the current initial exposure value; In the second stage, the first exposure average of the adjusted n initial exposure values ​​is obtained, and the second exposure average of the subsequent m initial exposure values ​​of the n initial exposure values ​​is obtained; Based on the first and second weighting weights of the nth time, the first exposure mean and the second exposure mean are weighted, and the weighted result is adjusted using a coefficient representing environmental change and the frame rate of the camera to obtain the initial exposure value of the (n+1)th time of the first exposure time. After another exposure of a preset duration, the exposure value is adjusted in the third stage. In the third stage, the 3D map representing the motion trajectory of the head-mounted display device is divided into multiple regions. The initial exposure value of the first exposure time in each region is adjusted separately. During each adjustment process: the average time of the n initial exposure values ​​of the first exposure time is calculated, as well as the first weight average of the n first weighted values ​​and the second weight average of the n second weighted values; the average time and the nth first exposure time are weighted according to the first weight average and the second weight average, and the weighted result is adjusted with the coefficient representing environmental changes and the frame rate of the camera to obtain the (n+1)th initial exposure value of the first exposure time. Wherein, the initial exposure value of the (n+1)th exposure time in the second stage. The calculation method is as follows: The initial exposure value of the (n+1)th exposure time in the third stage. The calculation method is as follows: in, This is the first weighted weight during the nth adjustment. This is the second weighting weight during the nth adjustment. The average value of the first exposure. The second exposure average, The coefficient for the aforementioned environmental change. The frame rate of the camera. The first weight mean of the first weighted average in n weighted averages. The second weight mean of the second weighted average in n weighted averages. The average time is represented by n, where n is an integer greater than 1.

Images