Control method of head-mounted display device, head-mounted display device and medium

Abstract

The invention discloses a control method of head-mounted display equipment, the head-mounted display equipment and a medium, and relates to the technical field of head-mounted equipment. The head-mounted display equipment comprises a zooming module, a driving module and an eyeball information acquisition module, and the method comprises the following steps: when a preset trigger event occurs, acquiring first eyeball information within a preset duration from the current moment through the eyeball information acquisition module; generating a zooming mode of the zooming module according to the first eyeball information and a pre-trained AI optical prediction model; the AI optical prediction model is used for predicting the zooming mode according to the eyeball information within the preset duration, and the zooming mode is that the zooming amount of the zooming module is controlled to reach the target zooming amount from the zooming moment. According to the method, the adjustment focal length of the eyeballs of the wearer after compensation of the zoom module is just matched with the virtual image distance of the virtual image displayed by the display screen, so that the convergence conflict, namely the VAC problem, can be avoided.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more specifically, to a control method for a head-mounted display device, a head-mounted display device, and a medium. Background Technology

[0002] For example, AR and XR head-mounted display devices enable interaction between people and the virtual world by merging the virtual and real worlds. During the interaction, the display screen in the head-mounted display device plays the role of displaying the virtual world to the wearer's retina, projecting the virtual images corresponding to the virtual world onto the wearer's retina for imaging.

[0003] Currently, the distance between the wearer's eyes and the display screen of the head-mounted display device is fixed. However, the depth of the virtual image relative to the human eye varies, and the wearer's eyes dynamically converge on the same object based on these changes in depth. This leads to convergence conflict, which can cause visual discomfort, eye fatigue, and other adverse experiences, affecting the usability and user experience of the head-mounted display device. Summary of the Invention

[0004] One objective of this application is to provide a new technical solution for the control of head-mounted display devices.

[0005] According to a first aspect of this application, a control method for a head-mounted display device is provided, the head-mounted display device including a zoom module, a driving module, and an eye information acquisition module, the method comprising: When a preset trigger event occurs, the eye information acquisition module acquires first eye information within a preset time period starting from the current moment. Based on the first eyeball information and the pre-trained AI optical prediction model, the zoom mode of the zoom module is generated. The zoom module is controlled to deform according to the zoom method described above, wherein the AI ​​optical prediction model is used to predict the zoom method based on eye information within a preset time period, and the zoom method is to control the zoom amount of the zoom module to reach the target zoom amount at the zoom moment.

[0006] Optionally, the zoom module includes a zoom lens and a grating, and controlling the deformation of the zoom module via the drive module according to the zoom method includes: According to the first zoom method, the first driver controls the deformation of the zoom lens so that the curvature of the zoom lens is the target curvature; According to the second zoom method, the grating deformation is controlled by the second driver to generate the target optical path offset; The drive module includes a first driver and a second driver, and the zoom mode includes a first zoom mode and a second zoom mode. The first zoom mode is to control the deformation of the zoom lens at a first zoom moment so that the curvature of the zoom lens reaches the target curvature, and the second zoom mode is to control the deformation of the grating at a second zoom moment to achieve the target optical path offset.

[0007] Optionally, the step of collecting first eye information within a preset time period starting from the current moment through the eye information acquisition module includes: Within a preset time period starting from the current moment, the eye movement coordinates within the preset time period are determined based on the first eye movement data collected by the first eye tracker and the second eye movement data collected by the second eye tracker. Within a preset time period starting from the current moment, the depth image within the preset time period is determined based on the first eye image data acquired by the first image sensor and the second eye image data acquired by the second image sensor. Based on the eye movement coordinates and depth images within a preset time period starting from the current moment, determine the first eyeball information within the preset time period starting from the current moment.

[0008] Optionally, controlling the deformation of the zoom lens via a first driver according to the first zoom method to make the curvature of the zoom lens the target curvature includes: Starting from the first zoom moment, a first deviation between the actual curvature of the zoom lens and the target curvature is determined; If the first deviation is greater than the first preset deviation, the first driving quantity of the first driver is determined based on the actual curvature, the target curvature, and the PID control algorithm. The first driver is driven based on the first driving quantity.

[0009] Optionally, the step of controlling the grating deformation via a second driver to generate the target optical path offset according to the second zoom method includes: Starting from the second zoom moment, a second deviation between the actual optical path offset generated by the grating and the target optical path offset is determined; If the second deviation is greater than the second preset deviation, the second driving quantity of the second driver is determined based on the actual optical path offset, the target optical path offset, and the PID control algorithm. The second driver is driven based on the second driving quantity.

[0010] Optionally, the method further includes: When the zoom module reaches the target zoom level, the wearer's current gaze point is obtained; If the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than the third preset deviation, the first eye information, the zoom time, and the required zoom amount are used as a set of training samples to update the AI ​​optical prediction model.

[0011] Optionally, the method further includes: If the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than the third preset deviation, an image blurring compensation operation is performed based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze region outside the gaze region where the current gaze point is located.

[0012] Optionally, before generating the zoom method for the zoom module based on the first eyeball information and a pre-trained AI optical prediction model, the method further includes: A trained AI optical prediction model is obtained. The AI ​​optical prediction model is trained by a training sample set and a preset loss function. The training sample set includes multiple sets of training samples. Each set of training samples includes eye sample information and corresponding zoom sample mode. The multiple sets of training samples include simulation training samples and real training samples.

[0013] Optionally, the AI ​​optical prediction model is a generator in an adversarial network, and obtaining the trained AI optical prediction model includes: The training operation includes: inputting eye sample information from the training samples into the generator to obtain a zoom prediction method generated by the generator based on the eye sample information; inputting the eye sample information and the corresponding zoom prediction method into the discriminator in the adversarial network to obtain a first discrimination result of the discriminator on the zoom prediction method; inputting the eye sample information and the corresponding zoom prediction method into the discriminator to obtain a second discrimination result of the discriminator on the zoom prediction method; updating the discriminator based on the first discrimination result, the second discrimination result, and a preset loss function; inputting the eye sample information and the corresponding zoom prediction method into the updated discriminator to obtain a third discrimination result of the updated discriminator on the zoom prediction method; and updating the generator based on the third discrimination result. Repeat the training operation until the preset training cutoff condition is reached to obtain a trained generator, which can be used as a trained AI optical prediction model.

[0014] Optionally, the method further includes: If the information from the first eyeball is not obtained, perform any of the following operations: The first step is to obtain the pose of the head-mounted display device, and when the pose changes, to generate first eye information based on the amount of pose change and historical eye information. The second step involves determining the predicted zoom amount of the zoom module based on the virtual image distance of the virtual image to be displayed on the head-mounted display device, and controlling the deformation of the zoom module through the drive module according to the predicted zoom amount. The third step involves obtaining the wearer's current gaze point and performing an image blurring compensation operation based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

[0015] According to a second aspect of this application, a head-mounted display device is provided, the head-mounted display device including a zoom module, a driving module, an eye information acquisition module, a memory, and a processor. The zoom module, the driving module, the eye information acquisition module, and the memory are respectively connected to the processor. The driving module is used to control the deformation of the zoom module under the control of the processor. The eye information acquisition module is used to acquire first eye information within a preset time period starting from the current moment under the control of the processor. The memory is used to store computer instructions. The processor is used to call the computer instructions from the memory to execute the method as described in any one of the first aspects.

[0016] According to a third aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the first aspects.

[0017] This application provides a control method for a head-mounted display device, which includes a zoom module, a drive module, and an eye information acquisition module. The method includes: upon the occurrence of a preset trigger event, acquiring first eye information within a preset duration starting from the current moment through the eye information acquisition module; generating a zoom mode for the zoom module based on the first eye information and a pre-trained AI optical prediction model; and controlling the deformation of the zoom module according to the zoom mode through the drive module. The AI ​​optical prediction model is used to predict the zoom mode based on the eye information within the preset duration, and the zoom mode is to control the zoom amount of the zoom module to reach a target zoom amount starting at the zoom moment. This method, on the one hand, ensures that the adjusted focal length of the wearer's eye after adjustment by the zoom module perfectly matches the virtual image distance of the virtual image displayed on the screen, thus avoiding convergence conflict, i.e., the VAC problem. On the other hand, by driving the zoom module to deform in advance through the drive module, the problem of the wearer's eye not adjusting the focus in time can be avoided, which would occur if the virtual image distance of the virtual image displayed on the screen changes and then the zoom module is driven to deform through the drive module.

[0018] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0020] Figure 1 This is a schematic diagram of the structure of a head-mounted display device provided in this application; Figure 2 This is a flowchart illustrating a control method for a head-mounted display device provided in this application; Figure 3 This application provides the structure of an AI optical prediction model; Figure 4 This is a schematic diagram of the structure of a control device for a head-mounted display device provided in this application; Figure 5 This is a schematic diagram of the structure of a head-mounted display device provided in this application. Detailed Implementation

[0021] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0026] This application provides a control method for a head-mounted display device, wherein, as Figure 1 As shown, the head-mounted display device includes a zoom module 10 and a drive module ( Figure 1 (Not shown) and eye information acquisition module 20. For example... Figure 2 As shown, the control method for the head-mounted display device provided in this application includes the following steps S2100 to S2300.

[0027] Step S2100: When a preset trigger event occurs, the eye information acquisition module 20 acquires the first eye information within a preset time period starting from the current moment.

[0028] In one embodiment of this application, the preset trigger event may specifically be an event in which the head-mounted display device receives voice or gesture input from the wearer to trigger the control method of the head-mounted display device provided in this application. The preset trigger event occurs when the wearer inputs voice or gesture input to the head-mounted display device to trigger the control method of the head-mounted display device provided in this application. It should be noted that this application does not limit the specific implementation of the preset trigger event.

[0029] In this embodiment, as Figure 1 As shown, the zoom module 10 is positioned on the side of the display screen 30 of the head-mounted display device closer to the wearer's eyes. Driven by the drive module, the zoom module 10 deforms to change its focal length, thereby adjusting the convergence point of the light rays corresponding to the virtual image displayed on the head-mounted display screen. This adjusts the focus of the wearer's eyes, allowing the virtual image to form a clear image on the retina. Specifically, when the focal length of the zoom module 10 shortens, the convergence point of the light rays corresponding to the virtual image displayed on the head-mounted display screen is closer to the eyeball, and the eye's focusing distance decreases accordingly. Conversely, when the focal length of the zoom module 10 lengthens, the convergence point of the light rays corresponding to the virtual image displayed on the head-mounted display screen is farther from the eyeball, and the eye's focusing distance increases accordingly.

[0030] In one example, zoom module 10 is a liquid zoom lens, such as a liquid crystal zoom lens.

[0031] The driving module is used to drive the zoom module 10 to deform and change its focal length. In one example, when the zoom module 10 is a liquid zoom lens, the driving module is a voltage source with a variable output voltage. Furthermore, this application does not limit the position of the driving module, as long as it does not obstruct the wearer's line of sight from the display screen 30.

[0032] like Figure 1 As shown, the eye information acquisition module 20 is disposed on the side of the zoom module 10 closer to the wearer's eyeball, and is used to acquire the wearer's eye information, which reflects the characteristic information of the eyeball. In one embodiment, the eye information acquisition module 20 is an eye-tracking module. Furthermore, in this embodiment, the eye information acquired by the eye information acquisition module 20 within a preset time period from the current moment is recorded as the first eye information.

[0033] Step S2200: Based on the first eyeball information and the pre-trained AI optical prediction model, generate the zoom mode of the zoom module 10.

[0034] Among them, the AI ​​optical prediction model is used to predict the zoom mode based on eye information within a preset time period. The zoom mode is to control the zoom amount of the zoom module 10 to reach the target zoom amount at the zoom moment.

[0035] In this embodiment, the first eye information is input into the trained AI optical prediction model. The trained AI optical prediction model predicts the target zoom amount and zoom time of the zoom module 10 based on the first eye information. It predicts when (zoom time) the zoom module will start to control the deformation of the zoom module to form a certain amount of zoom (target zoom amount). The zoom mode is such that the adjusted focal length of the wearer's eye after adjustment by the zoom module 10 matches the virtual image distance of the changed virtual image displayed on the display screen 30 (virtual image distance refers to the distance between the virtual image displayed on the display screen 13 and the wearer's eye).

[0036] Step S2300: Control the deformation of the zoom module through the drive module according to the zoom method.

[0037] After obtaining the zoom mode of the zoom module 10, the drive module is controlled to deform the zoom module 10 at the zoom moment to achieve the target zoom amount. At this time, the virtual image displayed on the screen 30 changes to a virtual image distance that matches the adjusted focal length of the wearer's eye after adjustment by the zoom module 10. This ensures that the adjusted focal length of the wearer's eye after adjustment by the zoom module 10 matches the virtual image distance of the virtual image displayed on the screen 30, thus avoiding convergence conflict (VAC) problems. Furthermore, by driving the zoom module 10 to deform in advance, the problem of untimely focal length adjustment by the wearer's eye is avoided, which occurs when the virtual image distance of the virtual image displayed on the screen 30 changes before the zoom module 10 is deformed.

[0038] This application provides a control method for a head-mounted display device, which includes a zoom module, a drive module, and an eye information acquisition module. The method includes: upon the occurrence of a preset trigger event, acquiring first eye information within a preset duration starting from the current moment through the eye information acquisition module; generating a zoom mode for the zoom module based on the first eye information and a pre-trained AI optical prediction model; and controlling the deformation of the zoom module according to the zoom mode through the drive module. The AI ​​optical prediction model is used to predict the zoom mode based on the eye information within the preset duration, and the zoom mode is to control the zoom amount of the zoom module to reach a target zoom amount starting at the zoom moment. This method, on the one hand, ensures that the adjusted focal length of the wearer's eye after adjustment by the zoom module perfectly matches the virtual image distance of the virtual image displayed on the screen, thus avoiding convergence conflict, i.e., the VAC problem. On the other hand, by driving the zoom module to deform in advance through the drive module, the problem of the wearer's eye not adjusting the focus in time can be avoided, which would occur if the virtual image distance of the virtual image displayed on the screen changes and then the zoom module is driven to deform through the drive module.

[0039] In one embodiment of this application, the control method for the head-mounted display device provided in this application further includes the following step S2210 before the above step S2200.

[0040] Step S2210: Obtain the trained AI optical prediction model.

[0041] The AI ​​optical prediction model is trained using a training sample set and a preset loss function. The training sample set includes multiple training samples. Each training sample set includes eye sample information and the corresponding zoom sample mode. The multiple training samples include simulation training samples and real training samples.

[0042] In this embodiment, a simulated training sample refers to eye information and its corresponding zoom mode simulated using, for example, Blender. A real training sample refers to eye information and its corresponding zoom mode obtained by collecting eye information from a real eye. The eye information in the simulated or real training sample is denoted as eye sample information, and the zoom mode corresponding to the eye information in the simulated or real training sample is denoted as zoom sample mode. It should be noted that simulated training samples are used to compensate for the insufficient number of real training samples, thereby increasing the number of training samples in the training sample set and improving the prediction accuracy of the AI ​​optical prediction model. Furthermore, based on the training samples, a neural network model with prediction capabilities is trained using traditional neural network model training methods to obtain the AI ​​optical prediction model.

[0043] It should be noted that, for a real training sample, the acquisition method of the zoom sample mode corresponding to the eye sample information is as follows: the R&D personnel wear the head-mounted display device, and the eye information acquisition module collects the wearer's eye information to obtain the eye sample information; the starting time of zooming of the zoom module 10 from the time when the eye sample information is collected (referred to as time 0) is recorded as the zoom time in the zoom sample mode; the zoom module 10 is controlled to zoom; during the zooming process of the zoom module 10, the adjustment focus of the wearer's eyeball is detected, and the zoom amount of the zoom module 10 when the adjustment focus of the wearer's eyeball exactly matches the virtual image distance of the virtual image after the change in virtual image distance is recorded as the target zoom amount in the zoom sample mode.

[0044] A simulated training sample can be generated using sample simulation software. For example, it can be randomly generated according to a preset formula using the software, or it can be generated after learning from real samples using the software, or it can be generated based on user-defined parameters. The data format of the simulated training sample is the same as that of the real training sample.

[0045] In one embodiment of this application, taking the AI ​​optical prediction model as an adversarial network and the AI ​​optical prediction model as a generator in the adversarial network as an example, the above step S2210 is specifically implemented through the following steps S2211 and S2212.

[0046] Step S2211: Perform the training operation. The training operation includes the following steps S2211-1 to S2211-6: Step S2211-1: Input the eye sample information from the training samples into the generator to obtain the zoom prediction method generated by the generator based on the eye sample information.

[0047] Initially, the generator's model parameters are set to default values. For step S2211-1 above, the eye sample information from the training samples is input into the generator, which then predicts the zoom mode corresponding to the eye sample information and outputs the zoom prediction mode. It should be noted that typically, the eye sample information from the training samples, along with random noise, needs to be input into the generator, which then outputs the zoom prediction mode.

[0048] Step S2211-2: Input the eye sample information and the corresponding zoom prediction method into the discriminator in the adversarial network to obtain the first discrimination result of the discriminator on the zoom prediction method.

[0049] For step S2211-2 above, the eye sample information and the corresponding zoom prediction method are input into the discriminator. The discriminator determines the probability that the zoom prediction method corresponding to the eye sample information is true, and obtains the judgment result, which is recorded as the first discrimination result.

[0050] Step S2211-3: Input the eyeball sample information and the corresponding zoom sample mode into the discriminator to obtain the discriminator's second discrimination result for the zoom sample mode.

[0051] For step S2211-3 above, the eye sample information and the corresponding zoom sample mode are input into the discriminator. The discriminator determines the probability that the zoom sample mode corresponding to the eye sample information is true, and obtains the judgment result, which is recorded as the second discrimination result.

[0052] Step S2211-4: Update the discriminator based on the first discrimination result, the second discrimination result, and the preset loss function.

[0053] In one embodiment of this application, the preset loss function in steps S2211-4 above is shown in Formula 1 below.

[0054] (Formula 1) in, and These are weighting coefficients used to balance the adversarial training loss term. and user feedback loss items This can be set based on experience. In one example, =0.7, =0.3.

[0055] as well as, (Formula 2) Where D() is the discriminator; G() is the generator; and X is the parameter input to the generator (such as eye sample information and random noise). That is, the first discrimination result; These are the training samples (including eyeball sample information and corresponding zoom sample modes) input into the discriminator. That is, the second discrimination result; For example, the gradient penalty coefficient is... =10.

[0056] It should be noted that traditional methods only measure pixel-level differences, ignoring the perceptual characteristics of the human visual system, resulting in overly smooth output (especially in high-frequency detail areas); and they cannot solve the physiological adaptation problems of retinal imaging (such as VAC conflict). However, by introducing the adversarial loss of the discriminator D through the above formula, D can learn human visual perception preferences (such as focus sharpness and chromatic aberration tolerance), and the generator is forced to synthesize a zoom mode that conforms to physiological comfort.

[0057] as well as, (Formula 3) in, This is the optimal set of parameters for the historical output of the AI ​​optical prediction model; is the current parameter set of the AI ​​optical prediction model; S is the real-time feedback score, which ranges from [1, 5]. The higher the score, the larger the value of S.

[0058] It should be noted that after the generator is trained iteratively once, the head-mounted display device runs the generator. The user determines the value of S in Formula 3 by observing at least one of the following: whether the text in the image displayed by the head-mounted display device (such as the image of a fast-moving checkerboard pattern, the image of high-contrast edge text, the image of a virtual object with a depth mutation, etc.) is blurry, whether the boundaries are clear, whether the edges of objects have colored borders, whether there is ghosting when turning the head quickly, whether there is abrupt transition from the center of the field of vision to the edge, whether there is eye strain or dizziness when looking at objects at different distances, and whether there is a headache after using it continuously for 15 minutes.

[0059] Furthermore, the discriminator update method in step S2211-4 above can be achieved by minimizing the loss function to obtain the updated discriminator. It is understandable that the model parameters of the updated discriminator are not different from those of the original discriminator.

[0060] Step S2211-5: Input the eye sample information and the corresponding zoom prediction method into the updated discriminator to obtain the third discrimination result of the updated discriminator for the zoom prediction method.

[0061] For step S2211-5 above, the eye sample information and the corresponding zoom prediction method are input into the updated discriminator obtained based on step S2211-5 above. The updated discriminator determines the probability that the zoom sample method corresponding to the eye sample information is true, and obtains the judgment result, which is recorded as the third discrimination result.

[0062] Step S2211-6: Update the generator based on the third discrimination result.

[0063] The generator's loss is determined based on the third discrimination result, and the generator is updated using backpropagation based on the generator's loss.

[0064] Step S2212: Repeat the training operation until the preset training cutoff condition is reached, and obtain the trained generator as the trained AI optical prediction model.

[0065] In one embodiment of this application, the preset training cutoff condition is a preset number of repetitions of the training operation, wherein the preset number of repetitions can be set based on experience. Of course, the preset training cutoff condition can also be other, and this application does not limit it.

[0066] In one embodiment of this application, the zoom module includes a zoom lens and a grating, and the above step S2300 is specifically implemented through the following steps S2310 and S2320.

[0067] Step S2310: Control the deformation of the zoom lens through the first driver according to the first zoom mode so that the curvature of the zoom lens is the target curvature.

[0068] Step S2320: Control the grating deformation through the second driver according to the second zoom method to generate the target optical path offset.

[0069] The drive module includes a first driver and a second driver, and the zoom mode includes a first zoom mode and a second zoom mode. The first zoom mode is to control the deformation of the zoom lens at the first zoom moment so that the curvature of the zoom lens reaches the target curvature, and the second zoom mode is to control the deformation of the grating at the second zoom moment to achieve the target optical path offset.

[0070] In this embodiment, the grating, zoom lens, and display screen are arranged sequentially. After obtaining the first zoom mode and the second zoom mode, at the first zoom moment, the first driver controls the zoom lens 12 to deform so that the curvature of the zoom lens 12 reaches the target curvature. At the second zoom moment, the second driver controls the grating 11 to deform to generate the target optical path offset. Thus, since the focal length adjustment accuracy of the zoom lens 12 is less than that of the grating 11, based on the adjustment of the eye's focusing distance through the zoom lens 12, further fine-tuning of the eye's focusing distance is achieved through the grating 11. This allows for precise adjustment of the zoom module 10's focus distance for the wearer's eyes.

[0071] In one embodiment of this application, the grating 11 is specifically a subwavelength grating. The zoom lens 12 is a liquid zoom lens array, for example, 30×30. A piezoelectrically driven liquid zoom lens array, with each liquid zoom lens being 1×1 .

[0072] In one embodiment of this application, the first driver is a voltage source with a variable output voltage value. Specifically, when controlling the deformation of the zoom lens 12 via the first driver, the first driver outputs a voltage that matches the target curvature. At this time, the zoom lens 12 changes the light deflection angle according to the applied voltage until the light deflection angle stops changing and matches the target curvature. The voltage matching the target curvature is determined according to Formula 4 below.

[0073] (Formula 4) That is, (Formula 5) in, The output voltage of the first driver; For target curvature; For the material property constants of zoom lens 12, such as It is a dielectric elastomer. =0.12; The vacuum permittivity, such as =8.85e-12 F / m; The relative permittivity of the material, such as =2.8; The curvature of the zoom lens 12 at 0 voltage is determined by the initial shape of the zoom lens 12, and is typically 200 μm.

[0074] It should be noted that, since there is a hysteresis and nonlinear relationship between the voltage applied to the zoom lens 12 and the curvature response of the zoom lens 12, it is necessary to perform real-time calibration and correction on the output voltage of the first driver calculated by the above formula. For the specific calibration and correction method, please refer to Formula 6 below.

[0075] (Formula 6) in, This is a non-linear threshold, typically 3V; To calibrate the corrected output voltage of the first driver; Together they describe the nonlinear hysteresis characteristics of voltage and curvature response; For voltage-curvature conversion gain in the linear region; The curvature saturation slope of the nonlinear region; As a reference for curvature offset in the linear region; This is a hysteresis compensation term for the nonlinear region; It is a hysteresis asymmetric correction factor; and The ratio of dielectric anisotropy to elastic modulus, which characterizes liquid crystals, determines the voltage-curvature conversion efficiency. and The anchoring energy of the orientation layer and the concentration of ionic impurities cause zero bias and hysteresis; The difference in voltage ramping / depressing paths is due to the asymmetry in the rotational viscosity of liquid crystal molecules.

[0076] In one embodiment of this application, the second driver is also a voltage source with a variable output voltage value. Specifically, when controlling the deformation of the grating 11 through the second driver to achieve the target optical path offset, the second driver is controlled to output a voltage matching the target optical path offset. The voltage matching the target optical path offset is determined according to Formula Seven below.

[0077] (Formula 7) That is, (Formula 8) in, Target optical path offset (unit: μm); This is the half-wave voltage, when grating 11 is a lithium niobate grating. =5V, This is the scaling factor related to the grating period 11.

[0078] In one embodiment of this application, the zoom lens 12 includes sub-zoom lenses in multiple regions. The first driver includes sub-drivers corresponding to each region. A sub-driver is used to drive the deformation of the sub-zoom lens in the corresponding region. The first zoom mode includes at least the target sub-zoom mode of the target sub-zoom lens and the adjustment focal length of the wearer's eyeball after adjustment by the target sub-zoom lens matches the virtual image distance of the virtual image displayed on the screen. The target sub-zoom lens is the sub-zoom lens corresponding to the wearer's gaze point.

[0079] In one example, zoom lens 12 includes 16 1×1 Sub-zoom lens.

[0080] In this embodiment, the zoom lens 12 is divided into different regions, and the zoom lens portion corresponding to each region is denoted as a sub-zoom lens. Each sub-zoom lens is driven by a sub-driver of the first driver. Furthermore, each sub-zoom lens corresponds to a discrete viewing angle.

[0081] Furthermore, the target sub-zoom method involves controlling the deformation of the target sub-zoom lens at the zoom moment to achieve the target sub-zoom amount. Specifically, the target sub-zoom amount refers to the amount of zoom (target sub-zoom amount) achieved when the processor 40, via the sub-driver, drives the deformation of the target sub-zoom lens in the zoom module 10, and the adjusted focal length of the wearer's eye after adjustment by the target sub-zoom lens precisely matches the virtual image distance of the changed virtual image displayed on the screen.

[0082] In this embodiment, the target sub-zoom lens corresponds to a target sub-zoom method that matches the virtual image distance of the virtual image displayed on the screen with the adjustable focal length adjusted by the wearer using the target sub-zoom lens. The target sub-zoom lens is the sub-zoom lens corresponding to the wearer's gaze point. Therefore, when the wearer views the virtual image displayed on the head-mounted display device's screen, the virtual image distance of the virtual image at the gaze point matches the adjustable focal length adjusted by the wearer using the target sub-zoom lens. This ensures that the adjustable focal length of the wearer's eyeball after adjustment by the target sub-zoom lens exactly matches the virtual image distance of the virtual image displayed at the wearer's gaze point.

[0083] In one embodiment of this application, step S2100 is specifically implemented through the following steps S2110 to S2130.

[0084] Step S2110: Within a preset time period starting from the current moment, determine the eye movement coordinates within the preset time period starting from the current moment based on the first eye movement data collected by the first eye tracker and the second eye movement data collected by the second eye tracker.

[0085] Step S2120: Within a preset time period starting from the current moment, determine the depth image within the preset time period starting from the current moment based on the first eye image data collected by the first image sensor and the second eye image data collected by the second image sensor. Step S2130: Determine the first eyeball information within the preset time period starting from the current moment based on the eye movement coordinates within the preset time period starting from the current moment and the depth image within the preset time period starting from the current moment.

[0086] In this embodiment, the eye information acquisition module includes: a first eye tracker, a second eye tracker, a first image sensor, and a second image sensor.

[0087] like Figure 1 As shown, the eye information acquisition module 20 includes: a first eye tracker 21 for acquiring first eye movement data of the first eye; a second eye tracker 22 for acquiring second eye movement data of the second eye; a first image sensor 23 for acquiring first image data of the first eye; and a first image sensor 24 for acquiring second image data of the second eye.

[0088] In this case, the first eyeball is the left eyeball and the second eyeball is the right eyeball, or the first eyeball is the right eyeball and the second eyeball is the left eyeball.

[0089] The first eye tracker 21 and the second eye tracker 22 can specifically be iris cameras. Eye coordinates can be represented as the position of the gaze point on the display screen, expressed by components on the x-axis and y-axis. Furthermore, the gaze direction is determined based on the first and second eye track data, and the coordinates of the intersection of the gaze direction and the display screen 30 are further defined as eye coordinates.

[0090] Furthermore, the first and second image sensors can specifically be infrared cameras. The depth image is used to reflect the distance between the wearer's gaze and the eyeball. And based on the first and second eye image data, the depth image can be obtained using the principle of parallax.

[0091] After obtaining the eye movement coordinates within a preset duration from the current moment through step S2110 and the depth image within a preset duration from the current moment through step S2120, the eye movement coordinates within a preset duration from the current moment and the depth image within a preset duration from the current moment are used as the first eyeball information within the preset duration from the current moment.

[0092] In conjunction with the above embodiments, in one embodiment of this application, such as Figure 3 As shown, the AI ​​optical prediction model includes: a CNN visual branch and an LSIM temporal branch, as well as a feature fusion layer that combines the outputs of the two branches.

[0093] The CNN vision branch is used for feature extraction from the depth image. That is, the input to the CNN vision branch is the depth image. In one example, the depth image is a 256×256 depth map, and the CNN vision branch is a lightweight, improved version of MobileNetV3. The CNN vision branch extracts features from the 256×256 depth map to obtain a 512-dimensional depth feature vector. The specific structure of the CNN vision branch includes: a first convolutional layer with a 3×3 kernel, a stride of 2, and 16 output channels; five depthwise separable convolutional layers, each with doubled channels to 128; and an attention mechanism with a compression ratio of 4 and an inserted SE module.

[0094] The LSTM temporal branch is used for feature extraction from eye-tracking coordinates. That is, the input to the LSTM temporal branch is the eye-tracking coordinates. In one example, the eye-tracking coordinates are from 10 consecutive frames. The structure of the LSTM temporal branch includes a bidirectional LSTM (64 hidden units) and a self-attention layer. The LSTM temporal branch extracts features from the eye-tracking coordinates of the 10 consecutive frames and outputs a 128-dimensional eye-tracking feature vector.

[0095] The input to the feature fusion layer is a 512-dimensional depth feature vector output from the CNN vision branch and a 128-dimensional eye-tracking feature vector output from the LSTM temporal branch. The output of the feature fusion layer is a first zoom mode that controls the deformation of the zoom lens 12 from the first zoom moment so that the curvature of the zoom lens 12 reaches the target curvature, and a second zoom mode that controls the deformation of the grating 11 from the second zoom moment to achieve the target optical path offset.

[0096] In one embodiment of this application, step S2310 is specifically implemented by the following steps S2311 to S2313.

[0097] Step S2311: Starting from the first zoom moment, determine the first deviation between the actual curvature and the target curvature of the zoom lens.

[0098] In one embodiment of this application, the head-mounted display device includes a curvature sensor disposed on the side of the zoom lens closest to the wearer's eyeball, for acquiring the actual curvature of the zoom lens. Based on this, the actual curvature of the zoom lens can be obtained. Starting from the first zoom moment, after obtaining the actual curvature of the zoom lens, the deviation between the actual curvature and the target curvature of the zoom lens is calculated and denoted as the first deviation.

[0099] Step S2312: If the first deviation is greater than the first preset deviation, determine the first driving quantity of the first driver based on the actual curvature, the target curvature and the PID control algorithm.

[0100] Step S2313: Drive the first driver based on the first driving quantity.

[0101] After the zoom lens 12 is deformed according to the first zoom mode by the first driver, due to driving errors and other reasons, there is a possibility that the actual curvature of the zoom lens 12 may not be the target curvature after the curvature of the zoom lens 12 reaches the target curvature. To address this, in order to ensure that the actual curvature of the zoom lens 12 is the target curvature, the actual curvature of the zoom lens 12 is acquired by a curvature sensor. If the first deviation between the actual curvature and the target curvature is greater than a first preset deviation, a first driving amount of the first driver is determined based on the actual curvature, the target curvature, and a PID control algorithm, and the first driver is driven according to the first driving amount. This feedback method ensures that the actual curvature of the zoom lens 12 is the target curvature, where the first deviation is the maximum allowable deviation between the actual curvature and the target curvature.

[0102] In the case where the first driver is a voltage source with a variable output voltage value, the determination of the first driving quantity of the first driver in step S2312 based on the actual curvature, the target curvature, and the PID control algorithm is specifically achieved through the following formula nine.

[0103] (Formula Nine) in, For the target curvature, For actual curvature, , , As the primary driving force, The voltage corresponding to the target curvature is used as the reference input for PID control. It should be noted that step S2312 above can be executed at a fixed period (e.g., a period of 1ms).

[0104] Similarly, in one embodiment of this application, the above step S2320 is specifically implemented through the following steps S2321 to S2323.

[0105] Step S2321: Starting from the second zoom moment, determine the second deviation between the actual optical path offset generated by the grating and the target optical path offset.

[0106] In one embodiment of this application, the head-mounted display device includes a diffraction efficiency sensor 60, disposed on the side of the grating 11 near the wearer's eyeball, for collecting the actual diffraction efficiency of the grating 11. Based on this, the actual diffraction efficiency of the grating 11 can be obtained, and the actual optical path offset can be further determined based on the actual diffraction efficiency. Starting from the second zoom moment, after obtaining the actual optical path offset generated by the grating, the deviation between the actual optical path offset of the grating 11 and the target optical path offset is calculated and recorded as the second deviation.

[0107] Step S2322: If the second deviation is greater than the second preset deviation, determine the second driving quantity of the second driver based on the actual optical path offset, the target optical path offset, and the PID control algorithm.

[0108] Step S2323: Drive the second driver based on the second driving quantity.

[0109] After the grating 11 is deformed by the second driver according to the second zoom method, due to driving errors and other reasons, there is a possibility that the actual optical path offset generated by the grating 11 after the optical path offset reaches the target optical path offset is not the target optical path offset. To address this, in order to ensure that the actual optical path offset of the grating 11 equals the target optical path offset, the actual diffraction efficiency of the grating 11 is collected by the diffraction efficiency sensor 60, and the actual optical path offset is determined based on the actual diffraction efficiency. Furthermore, if the second deviation between the actual optical path offset and the target optical path offset is greater than a second preset deviation, a second driving quantity of the second driver is determined based on the actual optical path offset, the target optical path offset, and the PID control algorithm, and the second driver is driven according to the second driving quantity. This feedback mechanism ensures that the actual optical path offset generated by the grating 11 is the target optical path offset. The second deviation is the maximum allowable deviation between the actual optical path offset and the target optical path offset.

[0110] In one embodiment of this application, the control method for a head-mounted display device provided in this application further includes the following steps S2400 to S2500.

[0111] Step S2400: When the zoom module reaches the target zoom level, obtain the wearer's current gaze point.

[0112] In one embodiment of this application, the wearer's gaze point can be determined based on the eye information collected by the eye information acquisition module 20 and the eye-tracking algorithm.

[0113] In step S2500, if the third deviation between the required zoom amount and the target zoom amount corresponding to the current gaze point is greater than the third preset deviation, the first eye information, zoom time, and required zoom amount are used as a set of training samples to update the AI ​​optical prediction model.

[0114] If the third deviation between the desired zoom amount and the target zoom amount corresponding to the wearer's current gaze point is greater than the third preset deviation, then the zoom mode predicted by the AI ​​optical prediction model is determined to be inaccurate. In this case, the eye information input into the AI ​​optical prediction model in step S2100, the zoom time in the zoom mode obtained in step S2200, and the corresponding desired zoom amount are used as a set of training samples to continue training the AI ​​optical prediction model, thereby updating the AI ​​optical prediction model. This allows for further training of the AI ​​optical prediction model, enabling it to predict zoom modes more accurately while conforming to the wearer's current eye characteristics. The third preset deviation is the maximum allowable difference between the desired zoom amount and the target zoom amount.

[0115] In one embodiment of this application, the control method for the head-mounted display device provided in this application further includes the following step S2600.

[0116] Step S2600: If the third deviation between the required zoom amount and the target zoom amount corresponding to the current gaze point is greater than the third preset deviation, perform an image blurring compensation operation based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

[0117] If the third deviation between the desired zoom amount and the target zoom amount corresponding to the wearer's current gaze point is greater than a third preset deviation, then the zoom method predicted by the AI ​​optical prediction model is determined to be inaccurate. In this case, an image blurring compensation operation is performed on the virtual image to be displayed via software. Specifically, this operation involves adding a progressive Gaussian blur to the annular region around the gaze point to simulate a depth-of-field effect. The specific operations for adding a progressive Gaussian blur to the annular region around the gaze point can be: sequentially performing gaze point radial gradient blur, depth adaptive intensity control, edge preservation and temporal smoothing, and hardware delay window precise coverage processing.

[0118] By using the above step S2600, zoom compensation can be performed via software when the zoom mode predicted by the AI ​​optical prediction model is inaccurate.

[0119] Furthermore, in one embodiment of this application, for steps S2500 and S2600 above, the actual curvature and actual optical path offset are determined by the optical core algorithms shown in Formula 10 and Formula 11 below.

[0120] (Formula 10) in, Let t be the radius of curvature of the lens at time t (unit: μm). This is the focal length compensation value corresponding to the target curvature of the AI ​​optical prediction model, where n is the refractive index of the lens material (usually 1.45). The coordinates of the gaze point at time t can be determined using an infrared eye-tracking camera and stereo vision algorithms. Here are the coordinates of the gaze point at time t-1. The curvature precompensation amount is the unit pixel displacement.

[0121] (Formula Eleven) Where Δxij is the x-direction offset of the (i,j)th sub-pixel (unit: nm); Dij is the scene depth corresponding to the pixel (unit: mm); Dfocus is the gaze depth (unit: mm); and λ is the wavelength of the light source. For dynamic calibration coefficients.

[0122] After obtaining the actual curvature, it is compared with the target curvature to determine whether the deviation between the two is greater than the preset deviation. If so, it indicates that the AI ​​optical prediction model is accurate; otherwise, it is inaccurate. Similarly, the actual optical path offset is also determined in this way.

[0123] In one embodiment of this application, when the eye information acquisition module fails and eye information cannot be acquired, the control of the head-mounted display device provided in this application further includes a zoom degradation strategy corresponding to at least one of the following steps S2700, S2800, or S2900.

[0124] Step S2700: Obtain the pose of the head-mounted display device. If the pose changes, generate first eye information based on the amount of pose change and historical eye information.

[0125] In this embodiment, the pose can be acquired by a pose sensor installed on the head-mounted display device, wherein the pose sensor can be exemplarily an IMU capable of acquiring 6DOF information.

[0126] When the wearer's pose changes, it is determined that the focal length of their eyeballs changes, requiring the zoom module 10 to recompensate for this change. Therefore, if eyeball information is unavailable, the first eyeball information is inferred based on the pose change and historical eyeball information. Specifically, the first eyeball information used in the previous call to the AI ​​optical prediction model is recorded as historical eyeball information. Furthermore, after inferring the first eyeball information, the AI ​​optical prediction model is invoked using this information to generate the zoom mode of the zoom module 10.

[0127] Furthermore, based on the change in posture and historical eye information, the principle for inferring the first eye information is as follows: by activating the vestibular-ocular reflex (VOR) model through changes in head posture, the gaze point prediction is expanded from precise coordinates to a probability distribution area (the diameter of this probability distribution area is dynamically adjusted with the head angular velocity (5°-10°), and a gaze probability heatmap is generated by combining historical eye information; eye information is obtained based on the gaze probability heatmap.

[0128] Step S2800: Based on the virtual image distance of the virtual image to be displayed on the head-mounted display device, determine the predicted zoom amount of the zoom lens 12, and control the deformation of the zoom module through the drive module according to the predicted zoom amount.

[0129] In this embodiment, when the zoom amount includes the curvature of the zoom lens 12 and the optical path offset generated by the grating 11, the specific implementation of step S2800 is as follows: For different virtual image distances, the curvature and optical path offset corresponding to different virtual image distance ranges are divided. The curvature and optical path offset corresponding to the range of virtual image distances to be displayed by the head-mounted display device are recorded as the target curvature and target optical path offset, respectively, as the predicted zoom amount. Further, the zoom module deformation is controlled by the drive module according to the predicted zoom amount.

[0130] Step S2900: Obtain the wearer's current gaze point and perform an image blurring compensation operation based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

[0131] The specific implementation of step S2900 can be found in the relevant description in step S2600 above, and will not be repeated here.

[0132] Through steps 2700 and S2900 above, this application also provides three zoom degradation strategies when the eye information acquisition module fails and eye information cannot be obtained.

[0133] This application also provides a control device 400 for a head-mounted display device, wherein the head-mounted display device includes a zoom module, a drive module, and an eye information acquisition module, such as... Figure 4 As shown, the device 400 includes: The acquisition module 410 is used to acquire first eye information within a preset time period starting from the current moment through the eye information acquisition module when a preset trigger event occurs. The generation module 420 is used to generate the zoom mode of the zoom module based on the first eyeball information and the pre-trained AI optical prediction model. Control module 430 is used to control the deformation of the zoom module through the drive module according to the zoom mode, wherein the AI ​​optical prediction model is used to predict the zoom mode based on eye information within a preset time period, and the zoom mode is to control the zoom amount of the zoom module to reach the target zoom amount at the zoom moment.

[0134] In one embodiment of this application, the zoom module includes a zoom lens and a grating, and the control module 430 is specifically used for: According to the first zoom method, the first driver controls the deformation of the zoom lens so that the curvature of the zoom lens is the target curvature; According to the second zoom method, the grating deformation is controlled by the second driver to generate the target optical path offset; The drive module includes a first driver and a second driver, and the zoom mode includes a first zoom mode and a second zoom mode. The first zoom mode is to control the deformation of the zoom lens at a first zoom moment so that the curvature of the zoom lens reaches the target curvature, and the second zoom mode is to control the deformation of the grating at a second zoom moment to achieve the target optical path offset.

[0135] In one embodiment of this application, the acquisition module 410 is specifically used for: Within a preset time period starting from the current moment, the eye movement coordinates within the preset time period are determined based on the first eye movement data collected by the first eye tracker and the second eye movement data collected by the second eye tracker. Within a preset time period starting from the current moment, the depth image within the preset time period is determined based on the first eye image data acquired by the first image sensor and the second eye image data acquired by the second image sensor. Based on the eye movement coordinates and depth images within a preset time period starting from the current moment, determine the first eyeball information within the preset time period starting from the current moment.

[0136] In one embodiment of this application, the control module 430 is specifically used for: Starting from the first zoom moment, a first deviation between the actual curvature of the zoom lens and the target curvature is determined; If the first deviation is greater than the first preset deviation, the first driving quantity of the first driver is determined based on the actual curvature, the target curvature, and the PID control algorithm. The first driver is driven based on the first driving quantity.

[0137] In one embodiment of this application, the control module 430 is specifically used for: Starting from the second zoom moment, a second deviation is determined between the actual optical path offset generated by the grating and the target optical path offset; If the second deviation is greater than the second preset deviation, the second driving quantity of the second driver is determined based on the actual optical path offset, the target optical path offset, and the PID control algorithm. The second driver is driven based on the second driving quantity.

[0138] In one embodiment of this application, the control device 400 for the head-mounted display device provided in this application further includes: The update module is used to obtain the wearer's current gaze point when the zoom module reaches the target zoom amount; If the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than the third preset deviation, the first eye information, the zoom time, and the required zoom amount are used as a set of training samples to update the AI ​​optical prediction model.

[0139] In one embodiment of this application, the control device 400 for the head-mounted display device provided in this application further includes: The compensation module is used to perform an image blurring compensation operation based on the current gaze point when the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than a third preset deviation. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

[0140] In one embodiment of this application, the acquisition module 410 is further configured to: A trained AI optical prediction model is obtained. The AI ​​optical prediction model is trained by a training sample set and a preset loss function. The training sample set includes multiple sets of training samples. Each set of training samples includes eye sample information and corresponding zoom sample mode. The multiple sets of training samples include simulation training samples and real training samples.

[0141] In one embodiment of this application, the acquisition module 410 is specifically used for: performing a training operation, the training operation including: inputting eye sample information from the training samples into the generator to obtain a zoom prediction method generated by the generator based on the eye sample information; inputting the eye sample information and the corresponding zoom prediction method into the discriminator in the adversarial network to obtain a first discrimination result of the discriminator on the zoom prediction method; inputting the eye sample information and the corresponding zoom prediction method into the discriminator to obtain a second discrimination result of the discriminator on the zoom prediction method; updating the discriminator based on the first discrimination result, the second discrimination result, and a preset loss function; inputting the eye sample information and the corresponding zoom prediction method into the updated discriminator to obtain a third discrimination result of the updated discriminator on the zoom prediction method; and updating the generator based on the third discrimination result. Repeat the training operation until the preset training cutoff condition is reached to obtain a trained generator, which can be used as a trained AI optical prediction model.

[0142] In one embodiment of this application, the control device 400 for the head-mounted display device provided in this application further includes: The degradation module is used to perform any of the following operations when the first eyeball information is not obtained: If the information from the first eyeball is not obtained, perform any of the following operations: The first step is to obtain the pose of the head-mounted display device, and when the pose changes, to generate first eye information based on the amount of pose change and historical eye information. The second step involves determining the predicted zoom amount of the zoom module based on the virtual image distance of the virtual image to be displayed on the head-mounted display device, and controlling the deformation of the zoom module through the drive module according to the predicted zoom amount. The third step involves obtaining the wearer's current gaze point and performing an image blurring compensation operation based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

[0143] This application also provides a head-mounted display device 500, such as... Figure 5 As shown, the head-mounted display device 500 includes a zoom module 510, a drive module 520, an eye information acquisition module 530, a memory 540, and a processor 550. The zoom module 510, the drive module 520, the eye information acquisition module 530, and the memory 540 are respectively connected to the processor 550. The drive module 520 is used to control the deformation of the zoom module 510 under the control of the processor 550. The eye information acquisition module 530 is used to acquire first eye information within a preset time period starting from the current moment under the control of the processor 550. The memory 540 is used to store computer instructions. The processor 550 is used to call the computer instructions from the memory 540 to execute the method as described in any of the above method embodiments.

[0144] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the above-described method embodiments.

[0145] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0146] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0147] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0148] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0149] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0153] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A control method for a head-mounted display device, characterized in that, The head-mounted display device includes a zoom module, a driving module, and an eye information acquisition module; the method includes: When a preset trigger event occurs, the eye information acquisition module acquires first eye information within a preset time period starting from the current moment. Based on the first eyeball information and the pre-trained AI optical prediction model, the zoom mode of the zoom module is generated. The zoom module is controlled to deform according to the zoom method described above, wherein the AI ​​optical prediction model is used to predict the zoom method based on eye information within a preset time period, and the zoom method is to control the zoom amount of the zoom module to reach the target zoom amount at the zoom moment.

2. The method according to claim 1, characterized in that, The zoom module includes a zoom lens and a grating. Controlling the deformation of the zoom module via the drive module according to the zoom method includes: According to the first zoom method, the first driver controls the deformation of the zoom lens so that the curvature of the zoom lens is the target curvature; According to the second zoom method, the grating deformation is controlled by the second driver to generate the target optical path offset; The drive module includes a first driver and a second driver, and the zoom mode includes a first zoom mode and a second zoom mode. The first zoom mode is to control the deformation of the zoom lens at a first zoom moment so that the curvature of the zoom lens reaches the target curvature, and the second zoom mode is to control the deformation of the grating at a second zoom moment to achieve the target optical path offset.

3. The method according to claim 1, characterized in that, The step of collecting first eye information within a preset time period starting from the current moment through the eye information acquisition module includes: Within a preset time period starting from the current moment, the eye movement coordinates within the preset time period are determined based on the first eye movement data collected by the first eye tracker and the second eye movement data collected by the second eye tracker. Within a preset time period starting from the current moment, the depth image within the preset time period is determined based on the first eye image data acquired by the first image sensor and the second eye image data acquired by the second image sensor. Based on the eye movement coordinates and depth images within a preset time period starting from the current moment, determine the first eyeball information within the preset time period starting from the current moment.

4. The method according to claim 2, characterized in that, The step of controlling the deformation of the zoom lens via a first driver according to a first zoom method to make the curvature of the zoom lens the target curvature includes: Starting from the first zoom moment, a first deviation between the actual curvature of the zoom lens and the target curvature is determined; If the first deviation is greater than the first preset deviation, the first driving quantity of the first driver is determined based on the actual curvature, the target curvature, and the PID control algorithm. The first driver is driven based on the first driving quantity.

5. The method according to claim 2, characterized in that, The step of controlling the grating deformation via a second driver to generate the target optical path offset according to the second zoom method includes: Starting from the second zoom moment, a second deviation between the actual optical path offset generated by the grating and the target optical path offset is determined; If the second deviation is greater than the second preset deviation, the second driving quantity of the second driver is determined based on the actual optical path offset, the target optical path offset, and the PID control algorithm. The second driver is driven based on the second driving quantity.

6. The method according to claim 1, characterized in that, The method further includes: When the zoom module reaches the target zoom level, the wearer's current gaze point is obtained; If the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than the third preset deviation, the first eye information, the zoom time, and the required zoom amount are used as a set of training samples to update the AI ​​optical prediction model.

7. The method according to claim 6, characterized in that, The method further includes: If the third deviation between the required zoom amount corresponding to the current gaze point and the target zoom amount is greater than the third preset deviation, an image blurring compensation operation is performed based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze region outside the gaze region where the current gaze point is located.

8. The method according to claim 1, characterized in that, Before the method for generating the zoom module based on the first eyeball information and the pre-trained AI optical prediction model, the method further includes: A trained AI optical prediction model is obtained. The AI ​​optical prediction model is trained by a training sample set and a preset loss function. The training sample set includes multiple sets of training samples. Each set of training samples includes eye sample information and corresponding zoom sample mode. The multiple sets of training samples include simulation training samples and real training samples.

9. The method according to claim 8, characterized in that, The AI ​​optical prediction model is a generator in an adversarial network. Obtaining the trained AI optical prediction model includes: The training operation includes: inputting eye sample information from the training samples into the generator to obtain a zoom prediction method generated by the generator based on the eye sample information; inputting the eye sample information and the corresponding zoom prediction method into the discriminator in the adversarial network to obtain a first discrimination result of the discriminator on the zoom prediction method; inputting the eye sample information and the corresponding zoom prediction method into the discriminator to obtain a second discrimination result of the discriminator on the zoom prediction method; updating the discriminator based on the first discrimination result, the second discrimination result, and a preset loss function; inputting the eye sample information and the corresponding zoom prediction method into the updated discriminator to obtain a third discrimination result of the updated discriminator on the zoom prediction method; and updating the generator based on the third discrimination result. Repeat the training operation until the preset training cutoff condition is reached to obtain a trained generator, which can be used as a trained AI optical prediction model.

10. The method according to claim 1, characterized in that, The method further includes: If the information from the first eyeball is not obtained, perform any of the following operations: The first step is to obtain the pose of the head-mounted display device, and when the pose changes, to generate first eye information based on the amount of pose change and historical eye information. The second step involves determining the predicted zoom amount of the zoom module based on the virtual image distance of the virtual image to be displayed on the head-mounted display device, and controlling the deformation of the zoom module through the drive module according to the predicted zoom amount. The third step is to obtain the wearer's current gaze point and perform an image blurring compensation operation based on the current gaze point. The image blurring compensation operation includes at least blurring the non-gaze area outside the gaze area where the current gaze point is located.

11. A head-mounted display device, characterized in that, The head-mounted display device includes a zoom module, a drive module, an eye information acquisition module, a memory, and a processor. The zoom module, the drive module, the eye information acquisition module, and the memory are respectively connected to the processor. The drive module is used to control the deformation of the zoom module under the control of the processor. The eye information acquisition module is used to acquire first eye information within a preset time period starting from the current moment under the control of the processor. The memory is used to store computer instructions. The processor is used to call the computer instructions from the memory to execute the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images