Display unit control method, head-mounted display device and storage medium

Abstract

The invention relates to the field of head-mounted display devices, in particular to a control method of a display unit, a head-mounted display device and a storage medium. The method is applied to a head-mounted display device, a display unit of the head-mounted display device comprises a metasurface optical array and a grating displacement platform, and the method comprises the following steps: acquiring eye image information and eye muscle information of a user of the head-mounted display device; based on the eye image information and the eye muscle information, predicting a focal length compensation value at a target moment and a target curvature of a metasurface optical array; and based on the focal length compensation value and the target curvature, driving the metasurface optical array, and driving the grating to move.

Description

Technical Field

[0001] This disclosure relates to the field of head-mounted display devices, and more specifically, to a control method for a display unit, a head-mounted display device, and a storage medium. Background Technology

[0002] Head-mounted displays, especially virtual reality and augmented reality headsets, are integrating into everyday life and industry applications at an unprecedented pace. From immersive gaming and remote virtual collaboration to industrial design simulations and medical surgery training, their application scenarios continue to expand. Technological upgrades and the continued decline in consumer product prices have further accelerated their popularization, gradually transforming them into a key entry point for next-generation human-computer interaction and a standard terminal connecting the digital world and the physical world.

[0003] However, current mainstream head-mounted displays face a fundamental physiological adaptation challenge—the convergence-accommodation conflict. Traditional fixed-focal-length optical systems cannot dynamically match the natural focusing behavior of the human eye: when users gaze at virtual objects at different distances, the convergence angle of the eyes misaligns with the required accommodative refractive power of the lens. This sensory conflict forces the ciliary muscle to remain tense, causing dizziness, visual fatigue, and even headaches, severely limiting the duration and comfort of the user experience, becoming a core bottleneck hindering the large-scale adoption of head-mounted display technology. Solving the convergence-accommodation conflict problem is a key breakthrough in achieving a truly natural and healthy immersive experience. Summary of the Invention

[0004] One object of this disclosure is to provide a control method, electronic device, and storage medium for a display unit that can resolve the problem of visual convergence-accommodation conflict.

[0005] According to a first aspect of this disclosure, a control method for a display unit is provided, applied to a head-mounted display device, wherein the display unit of the head-mounted display device includes a metasurface optical array and a grating displacement platform, the method comprising:

[0006] Acquire eye image information and eye muscle information of users of head-mounted display devices;

[0007] Based on the eye image information and eye muscle information, the focal length compensation value and the target curvature of the metasurface optical array at the target time are predicted.

[0008] Based on the focal length compensation value and the target curvature, the metasurface optical array is driven, and the grating is driven to shift.

[0009] Optionally, predicting the focal length compensation value and the target curvature of the metasurface optical array at the target time based on the eye image information and eye muscle information includes:

[0010] The eye image information and eye muscle information are input into a pre-trained dual-stream spatiotemporal network, and the user's gaze characteristics are determined based on the eye image information and the eye muscle information.

[0011] Based on the gaze features, the focal length compensation value and the target curvature are output, wherein the dual-stream spatiotemporal network is trained based on a preset loss function, and the preset loss function L total for:

[0012] L total =λ1L adv +λ2L physics +λ3L comfort

[0013] Where λ1, λ2, and λ3 are weighting coefficients, and L adv To combat the losses, L physics For physical constraint loss, L comfort This results in a loss of user comfort.

[0014] Optionally, the physical constraint loss is:

[0015]

[0016] Among them, f t Here, n is the focal length compensation value, and R is the refractive index of the material of the metasurface optical array. t Let R be the target curvature. fix Let be the radius of the fixed curvature surface.

[0017] Optionally, driving the metasurface optical array based on the focal length compensation value and the target curvature includes:

[0018] Based on the focal length compensation value and the first preset relationship, the first driving voltage of the metasurface optical array is determined, wherein the first preset relationship is the correspondence between the focal length compensation value and the driving voltage of the metasurface optical array.

[0019] The first driving voltage is verified based on the target curvature. If the verification is successful, the metasurface optical array is driven based on the first driving voltage.

[0020] Optionally, the gaze feature includes the user's gaze depth at the current moment and the user's gaze depth at the target moment;

[0021] The driving grating is displaced, including:

[0022] Determine the displacement Δx of the grating:

[0023]

[0024] Among them, D current D represents the depth of gaze at the current moment. target Let f be the depth of gaze at the target moment, α be the grating sensitivity coefficient, and f be the depth of gaze at the target moment. meta The focal length of the metasurface optical array after being driven by the first driving voltage; the displacement of the grating is controlled based on the displacement of the grating.

[0025] Optionally, after driving the metasurface optical array and displacing the grating based on the focal length compensation value and the target curvature at the target time, the method further includes:

[0026] Obtain the light field uniformity of the light field image output by the display unit;

[0027] If the uniformity of the light field is less than a preset uniformity threshold, adjust the grating displacement and the driving voltage of the metasurface optical array.

[0028] Optionally, obtaining the uniformity of the light field output by the display unit includes:

[0029] The light field image is divided into multiple sub-blocks, and the light field uniformity of each sub-block is determined:

[0030]

[0031] Among them, U ij For sub-block I ij Light field uniformity, μ(I ij ) is sub-block I ij The average light intensity, σ(I ij ) is sub-block I ij The standard deviation of light intensity.

[0032] Optionally, adjusting the grating displacement and the driving voltage of the metasurface optical array when the light field uniformity is less than a preset uniformity threshold includes:

[0033] If the optical field uniformity of the sub-block is less than a preset uniformity threshold, determine the change in driving voltage ΔV of the metasurface optical array corresponding to the sub-block. ij :

[0034] ΔV ij =K p ·ΔU ij +K i ·∑ΔU ij

[0035] Where, ΔU ij For sub-block I ijThe deviation value of the light field uniformity, K p K is the proportionality coefficient. i The integral coefficient;

[0036] Determine the displacement Δx of the grating unit corresponding to the sub-block. ij :

[0037] Δx ij =K p ·(U th -U ij )

[0038] The displacement of the grating unit is adjusted based on the displacement of the grating unit, and the metasurface optical array corresponding to the sub-block is driven according to the change in the driving voltage.

[0039] According to a second aspect of this disclosure, a head-mounted display device is provided, comprising: a display unit, a data acquisition unit, a processor, and a memory;

[0040] The display unit includes a metasurface optical array and a grating displacement platform;

[0041] The data acquisition unit includes an image acquisition module and an electromyography (EMG) signal sensor.

[0042] The image acquisition module is used to acquire the user's eye image information, and the electromyography signal sensor is used to acquire the user's eye muscle information.

[0043] The memory stores computer instructions that, when executed by the processor, implement the method described in any of the first aspects.

[0044] Optionally, the data acquisition unit further includes: a photosensor or a diffraction efficiency sensor;

[0045] The photosensitive sensor or diffraction efficiency sensor is used to obtain the light field uniformity of the light field image output by the display unit.

[0046] According to a third aspect of this disclosure, a storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the control method of any of the display units of the first aspect.

[0047] One technical advantage of this disclosure is that it provides a control method for a display unit. During operation, the head-mounted display device can continuously collect the user's eye information and predict the focal length compensation value and the target curvature of the metasurface optical array at subsequent moments based on the information. Based on the predicted value, the display unit of the head-mounted display device is adjusted at the target moment to adjust the curvature of the metasurface optical array and the displacement of the grating, thereby adapting it to the user's eye physiological condition at future moments to compensate for the problem of visual convergence and accommodation conflict caused by sensory misalignment and dizziness.

[0048] Other features and advantages of the embodiments of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which form part of this specification, illustrate embodiments of the present disclosure and, together with the specification, serve to explain the principles of the embodiments of the present disclosure.

[0050] Figure 1 This is a flowchart of the control method for the display unit provided in this application; Figure 2 This is a schematic diagram of the structure of an example head-mounted display device provided in this application; Figure 3 This is a schematic diagram of the hardware structure of another example of a head-mounted display device provided in this application. Detailed Implementation

[0052] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0053] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0054] Technologies and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such technologies and equipment should be considered part of the instruction manual.

[0055] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0056] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0057] It should be noted that all actions involving the acquisition of signals, information, or data in this embodiment are carried out in compliance with the relevant data protection laws and regulations of the country where the location is situated, and with authorization from the owner of the relevant equipment.

[0058] In one example of this embodiment, a control method for a display unit is also provided, which can be applied to head-mounted display devices, such as... Figure 1 As shown, the method includes steps S11-S13:

[0059] Step S11: Obtain eye image information and eye muscle information of the user of the head-mounted display device.

[0060] In this embodiment, the head-mounted display device has a display unit, which may incorporate a metasurface optical array to replace a traditional curved lens. The metasurface optical array can exhibit different optical curvatures under different driving voltages, thus achieving lens functionality. In this example, the display unit also includes a grating displacement platform, which can be a photonic crystal grating. In this example, the grating can be divided into multiple grating units, each of which can be driven independently and perform displacement.

[0061] In this embodiment, the head-mounted display device may further include an image acquisition module and a ciliary muscle EMG (Electromyography) sensor. The image acquisition module may be a bionic retinal event camera used to acquire image information of the user's eyes. A bionic retinal event camera is a neuromorphic visual sensor that simulates the working mechanism of the human retina. Its working principle is that each pixel independently monitors brightness changes, and only when the light intensity sensed by a pixel exceeds a threshold is an event asynchronously output. In this example, the bionic retinal event camera, through ON / OFF channel separation, simulates the response of retinal ganglion cells and outputs a pulse signal format (x, y, t, p), where x and y represent pixel coordinates, t is time, and p represents brightness increase or decrease.

[0062] In this embodiment, the ciliary muscle EMG sensor can be a non-contact sensor used to collect information about the ciliary muscle, such as the intensity of muscle contraction, the total energy of sustained contraction, the median frequency of the frequency component shifting to lower frequencies when the muscle is fatigued, and the average power frequency of the concentrated frequency band of the spectrum energy.

[0063] Step S12: Based on eye image information and eye muscle information, predict the focal length compensation value and the target curvature of the metasurface optical array at the target time.

[0064] In this embodiment, the user's eye condition can be predicted using image and muscle information. For example, eye fatigue can be determined using eye images and muscle information, and a fatigue threshold can be used to determine whether the user is about to experience saccades, thus predicting the user's gaze depth at a future target time. This allows for the calculation of the required focal length compensation value and target curvature for the user's eyes. In this example, the focal length compensation value refers to the difference between the user's ideal focal length at the target time and the current actual focal length. The target time can be a future time with a set interval from the current time.

[0065] In one example of this embodiment, predicting the focal length compensation value of a user's eye and the target curvature of a metasurface optical array at a target time based on eye image information and eye muscle information includes: inputting the eye image information and eye muscle information into a pre-trained dual-stream spatiotemporal network; determining the user's gaze characteristics based on the eye image information and the eye muscle information; and outputting the focal length compensation value and target curvature based on the gaze characteristics. The dual-stream spatiotemporal network is trained based on a preset loss function, L. total for:

[0066] L total =λ1L adv +λ2L physics +λ3l comfort

[0067] Where λ1, λ2, and λ3 are weighting coefficients, which can be obtained according to the actual situation, L adv To counteract the loss, a generator and discriminator work together to ensure that the focal length compensation value output by the generator approximates the true optical condition in terms of the target curvature. physics For physical constraint loss, L comfort This results in a loss of user comfort.

[0068] In one example of this embodiment, the physical constraint loss is:

[0069]

[0070] Among them, f t Here, n is the focal length compensation value, and R is the refractive index of the material of the metasurface optical array. t Let R be the target curvature. fix Let be the radius of the fixed curvature surface.

[0071] In this embodiment, the curvature of the metasurface optical array is strongly correlated with the focal length compensation value through optical physics laws. Therefore, the focal length compensation value f can be used as a basis for... t The target radius of curvature R of the metasurface optical array t The relationship is used as the physical constraint loss in the loss function.

[0072] Furthermore, in one example, the user comfort loss L can be determined based on the user's comfort score and the comfort score predicted by the two-stream spatiotemporal network. confort :

[0073] L confort =KL(p user ||p model )

[0074] Where, p user For the distribution of user comfort ratings, p model This example demonstrates how to predict the corresponding comfort distribution for a two-stream spatiotemporal network. In this case, the difference in probability distribution between the user's actual comfort level and the predicted comfort level can be quantified using a relative entropy loss function. In this example, the user's comfort score can be obtained through the head-mounted display device during user interaction.

[0075] In this embodiment, the dual-stream spatiotemporal network can be pre-trained using the aforementioned loss function.

[0076] In one example of this embodiment, the dual-stream spatiotemporal network includes a retinal encoder and a spiking neural network (SNN). The retinal encoder can extract spatiotemporal features of the gaze point based on eye image information, simulating the response characteristics of human ocular ganglion cells. These features are temporally sequential, such as gaze point depth, gaze point coordinates, and scene depth maps from the most recent multiple frames. Simultaneously, the spiking neural network can process eye image information and eye muscle information input from a ciliary muscle EMG sensor to predict the gaze point trajectory at a target time and obtain gaze point features at that time, such as gaze point depth. Furthermore, the user's physiological signals, muscle contraction intensity, and pupil diameter can be used as input features. Based on these input features, the dual-stream spatiotemporal network determines the focal length compensation value and the target curvature.

[0077] Step S13: Based on the focal length compensation value and the target curvature, drive the metasurface optical array and drive the grating to shift.

[0078] In one example of this embodiment, driving a metasurface optical array based on a focal length compensation value and a target curvature includes: determining a first driving voltage for the metasurface optical array according to a focal length compensation value and a first preset relationship, wherein the first preset relationship is the correspondence between the focal length compensation value and the driving voltage of the metasurface optical array; verifying the first driving voltage based on the target curvature; and driving the metasurface optical array based on the first driving voltage if the verification is successful.

[0079] In this embodiment, the head-mounted display device can pre-test and store the correspondence between focal length compensation values ​​and the driving voltage of the metasurface optical array's driving circuit. After predicting the focal length compensation value at the target time using a dual-stream spatiotemporal network, the driving voltage of the metasurface optical array can be determined based on this focal length compensation value and the pre-stored correspondence. Then, based on the target curvature R... t To verify the driving voltage, refer to the following formula:

[0080]

[0081] Where R0 is the curvature of the metasurface optical array before actuation. d is the thickness, k is the stiffness coefficient, ∈ 0 ∈ r These parameters, such as dielectric constant, can be determined by the material of the driving circuit.

[0082] When the driving voltage V calculated from the target curvature meta If the value is equal to the first driving voltage, or the difference between the first driving voltage and the first driving voltage is less than the threshold, the first driving voltage is determined to be verified as passed. The metasurface optical array is then driven by the first driving voltage to make its optical curvature change to the target curvature.

[0083] In the first example of this embodiment, the gaze features include the user's gaze depth at the current moment and the user's gaze depth at the target moment; driving the grating to displacement includes: determining the displacement amount Δx of the grating:

[0084]

[0085] Among them, D current D represents the depth of gaze at the current moment. target Let f be the depth of gaze at the target time, α be the grating sensitivity coefficient, and f be the depth of gaze at the target time. meta The focal length of the metasurface optical array after being driven by the first driving voltage; the displacement of the grating is controlled based on the displacement of the grating.

[0086] In this example, a control method for a display unit is provided. The head-mounted display device can continuously collect the user's eye information during operation and predict the focal length compensation value and the target curvature of the metasurface optical array at subsequent moments based on the information. Based on the predicted value, the display unit of the head-mounted display device is adjusted at the target moment to adjust the curvature of the metasurface optical array and the displacement of the grating, so as to adapt to the user's eye physiological condition at future moments to compensate for sensory misalignment and dizziness.

[0087] In one example of this embodiment, at the target time, after driving the metasurface optical array and driving the grating to displacement based on the focal length compensation value and the target curvature, the method further includes: obtaining the light field uniformity of the light field image output by the display unit; and adjusting the grating displacement and the driving voltage of the metasurface optical array when the light field uniformity is less than a preset uniformity threshold.

[0088] In this embodiment, a photosensor or diffraction efficiency sensor may also be provided in the head-mounted display device to detect the uniformity of the light field image output by the display unit. In this embodiment, the uniformity U of the light field image I can be determined based on the following formula:

[0089]

[0090] Where σ(I) is the standard deviation of light intensity of light field image I, and μ(I) is the mean light intensity of light field image I, which can be determined based on the sensing data of diffraction efficiency sensor or photosensitive sensor.

[0091] In this example, to avoid the influence of noise, the light field uniformity within a preset time window can be averaged and filtered. For example, the light field uniformity of the light field images within 5 frames can be averaged and used as the light field uniformity at the current moment.

[0092] After determining the uniformity of the light field image, it can be compared with a preset uniformity threshold to determine whether it meets the display requirements. When the light field uniformity is less than the threshold, the light field uniformity can be improved by adjusting the grating and / or metasurface optical array.

[0093] In this embodiment, by using the above method, the effect of the previously predicted and driven display unit can be further judged at the target time. If the light field uniformity does not meet the requirements, the metasurface optical array and grating displacement can be fine-tuned again to provide users with a better display effect. Furthermore, by adjusting before the target time, only fine-tuning is needed at the target time, which can effectively reduce the adjustment time of the display unit and overcome the contradiction between display accuracy and adjustment speed in traditional display unit control methods.

[0094] In one example of this embodiment, obtaining the light field uniformity output by the display unit includes: dividing the light field image into multiple sub-blocks and determining the light field uniformity of each sub-block.

[0095]

[0096] Among them, U ij For sub-block I ij Light field uniformity, μ(I ij ) is sub-block I ij The average light intensity, σ(Iij ) is sub-block I ij The standard deviation of light intensity.

[0097] In this embodiment, the output light field of the display unit may only have local non-uniformity. Therefore, the uniformity of each sub-block can be determined by segmenting the light field image and comparing it with a threshold to determine which specific part needs adjustment. In this example, the light field image can be segmented into... i Row j, column j.

[0098] In one example of this embodiment, when the optical field uniformity is less than a preset uniformity threshold, adjusting the grating displacement and the driving voltage of the metasurface optical array includes: determining the change in the driving voltage ΔV of the metasurface optical array corresponding to the sub-block when the optical field uniformity of the sub-block is less than the preset uniformity threshold. ij :

[0099] ΔV ij =K p ·ΔU ij +K i ·∑ΔU ij

[0100] Where, ΔU ij For sub-block I ij The deviation value of the light field uniformity, K p K is the proportionality coefficient. i The integral coefficients are used to determine the displacement Δx of the grating element corresponding to the sub-block. ij :

[0101] Δx ij =K p ·(U th -U ij )

[0102] The displacement of the grating unit is adjusted based on the displacement of the grating unit, and the metasurface optical array corresponding to the sub-block is driven according to the change in driving voltage.

[0103] In this example, each sub-region, during pre-division, can be determined through testing of the corresponding metasurface optical array and grating units. This allows for local fine-tuning of the sub-region when its optical field uniformity is low. When the optical field uniformity of a sub-block is less than a preset uniformity threshold, the difference between the uniformity threshold and the optical field uniformity of the sub-region, i.e., ΔU, can be used as the basis for adjustment. ij This allows for fine-tuning of the metasurface optical array and grating unit corresponding to the sub-region, thereby improving the display effect of the sub-region.

[0104] This application also provides a head-mounted display device 200, including: a display unit 201, a data acquisition unit 202, a processor, and a memory; the display unit 201 includes a metasurface optical array 2011 and a grating displacement platform 2012; the data acquisition unit 202 includes an image acquisition module 2021 and an electromyography (EMG) sensor 2022; wherein, the image acquisition module 2021 is used to acquire the user's eye image information, the EMG sensor 2022 is used to acquire the user's eye muscle information, and the memory 204 stores computer instructions. When the computer instructions are executed by the processor 203, they implement any one of the above-described control method embodiments of the display unit 201 and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0105] In this embodiment, as Figure 3 As shown, the head-mounted display device 200 has a display unit 201, which may contain a metasurface optical array 2011 to replace a traditional curved lens. The metasurface optical array 2011 is driven by a driving circuit and can exhibit different optical curvatures under different driving voltages, thus realizing the lens function. In this example, the display unit 201 also has a grating displacement platform 2012, which can be a photonic crystal grating. In this example, the grating can be divided into multiple grating units, each of which can be driven independently and perform displacement.

[0106] In this example, the display unit 201 can also be equipped with one or more other types of lenses according to actual needs, such as Figure 3 The elliptical portion. Furthermore, when the head-mounted display device 200 is a VR, MR, or other smart glasses, a screen 2013 may also be provided in the display unit 201 for imaging.

[0107] It should be noted that, Figure 3 The hardware configuration of the head-mounted display device 200 is only illustrative. The specific shape, specifications and position of the metasurface optical array 2011 and the grating can be configured by those skilled in the art according to the actual needs of the head-mounted display device 200.

[0108] In one example of this embodiment, the head-mounted display device 200 is provided with a data acquisition unit 202, which includes an image acquisition module 2021 and an electromyography signal sensor 2022. In one example, the image acquisition module 2021 may be a bionic retinal event camera used to acquire the user's eye image information.

[0109] In this embodiment, the electromyography (EMG) signal sensor 2022 can be a sensor used to collect the EMG signal of the user's ciliary muscle. To make it more comfortable to wear, it can be set as a non-contact sensor to collect information about the user's ciliary muscle, such as the muscle contraction intensity, the total energy of continuous contraction, the median frequency of the frequency component shifting to lower frequencies when the muscle is fatigued, and the average power frequency of the concentrated frequency band of the spectrum energy.

[0110] In one example of this embodiment, the data acquisition unit 202 further includes a photosensitive sensor or a diffraction efficiency sensor 2023; the photosensitive sensor or diffraction efficiency sensor 2023 is used to acquire the light field uniformity of the light field image output by the display unit 201.

[0111] This application also provides a storage medium storing computer instructions. When the computer instructions are executed by a processor, they implement any one of the above-described control method embodiments for the display unit and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0112] The various embodiments in this disclosure are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and apparatus embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0113] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] Embodiments of this disclosure may be systems, methods, and / or computer program products. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of this disclosure.

[0115] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0116] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0117] Computer program instructions used to perform the operations of embodiments of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of embodiments of this disclosure.

[0118] Various aspects of embodiments of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0119] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation in a combination of software and hardware are equivalent.

[0122] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A control method for a display unit, characterized in that, Applied to a head-mounted display device, wherein the display unit of the head-mounted display device includes a metasurface optical array and a grating displacement platform, the method includes: Acquire eye image information and eye muscle information of users of head-mounted display devices; Based on the eye image information and eye muscle information, the focal length compensation value and the target curvature of the metasurface optical array at the target time are predicted. Based on the focal length compensation value and the target curvature, the metasurface optical array is driven, and the grating is driven to shift.

2. The method according to claim 1, characterized in that, The step of predicting the focal length compensation value and the target curvature of the metasurface optical array at the target time based on the eye image information and eye muscle information includes: The eye image information and eye muscle information are input into a pre-trained dual-stream spatiotemporal network, and the user's gaze characteristics are determined based on the eye image information and the eye muscle information. Based on the gaze features, the focal length compensation value and the target curvature are output, wherein the dual-stream spatiotemporal network is trained based on a preset loss function, and the preset loss function L total for: L total =λ1L adv +λ2L physics +λ3L comfort Where λ1, λ2, and λ3 are weighting coefficients, and L adv To combat the losses, L physics For physical constraint loss, L comfort This results in a loss of user comfort. The physical constraint loss is: Among them, f t Here, n is the focal length compensation value, and R is the refractive index of the material of the metasurface optical array. t Let R be the target curvature. fix Let be the radius of the fixed curvature surface.

3. The method according to claim 1, characterized in that, Based on the focal length compensation value and the target curvature, the metasurface optical array is driven, including: Based on the focal length compensation value and the first preset relationship, the first driving voltage of the metasurface optical array is determined, where the first preset relationship is the correspondence between the focal length compensation value and the driving voltage of the metasurface optical array. The first driving voltage is verified based on the target curvature. If the verification is successful, the metasurface optical array is driven based on the first driving voltage.

4. The method according to claim 3, characterized in that, The gaze feature includes the user's gaze depth at the current moment and the user's gaze depth at the target moment. The displacement of the driving grating includes: determining the displacement amount Δx of the grating. Among them, D current D represents the depth of gaze at the current moment. target Let f be the depth of gaze at the target moment, α be the grating sensitivity coefficient, and f be the depth of gaze at the target moment. meta The focal length of the metasurface optical array after being driven by the first driving voltage; The displacement of the grating is controlled based on the displacement amount of the grating.

5. The method according to claim 1, characterized in that, After driving the metasurface optical array and the grating to displacement based on the focal length compensation value and the target curvature at the target time, the method further includes: Obtain the light field uniformity of the light field image output by the display unit; When the uniformity of the light field is less than a preset uniformity threshold, the displacement of the grating and the driving voltage of the metasurface optical array are adjusted.

6. The method according to claim 5, characterized in that, The step of obtaining the uniformity of the light field output by the display unit includes: The light field image is divided into multiple sub-blocks, and the light field uniformity of each sub-block is determined: Among them, U ij For sub-block I ij The uniformity of the light field, μ(I ij ) is sub-block I ij The average light intensity, σ(I ij ) is sub-block I Ij The standard deviation of light intensity.

7. The method according to claim 6, characterized in that, The step of adjusting the grating displacement and the driving voltage of the metasurface optical array when the light field uniformity is less than a preset uniformity threshold includes: If the optical field uniformity of the sub-block is less than a preset uniformity threshold, determine the change in driving voltage ΔV of the metasurface optical array corresponding to the sub-block. ij : ΔV ij =K p ·D.U. iJ +K i ·∑ΔU ij Where, ΔU ij For sub-block I ij The deviation value of the light field uniformity, K p K is the proportionality coefficient. i The integral coefficient; Determine the displacement Δx of the grating unit corresponding to the sub-block. ij : Δx ij =K p ·(IN th -IN ij ) The displacement of the grating unit is adjusted based on the displacement of the grating unit, and the metasurface optical array corresponding to the sub-block is driven according to the change in the driving voltage.

8. A head-mounted display device, characterized in that, include: Display unit, data acquisition unit, processor, and memory; The display unit includes a metasurface optical array and a grating displacement platform; The data acquisition unit includes an image acquisition module and an electromyography (EMG) signal sensor. The image acquisition module is used to acquire the user's eye image information, and the electromyography signal sensor is used to acquire the user's eye muscle information. The memory stores computer instructions, which, when executed by the processor, implement the method described in any one of claims 1-7.

9. The display device according to claim 8, characterized in that, The data acquisition unit also includes: a photosensitive sensor or a diffraction efficiency sensor; The photosensitive sensor or diffraction efficiency sensor is used to obtain the light field uniformity of the light field image output by the display unit.

10. A storage medium, characterized in that, It stores computer instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1-7.

Images