Head-mounted display equipment, control method and device of head-mounted display equipment and medium

Abstract

The invention discloses head-mounted display equipment, a control method and device of the head-mounted display equipment and a medium, and relates to the technical field of wearable equipment. The device comprises an image sensor used for collecting an eyeball image of a wearer of the head-mounted display device; the controller is connected with the image sensor and used for determining the vision state of the wearer according to the eyeball image; the diopter adjusting lens is arranged on one side, close to eyeballs of a wearer, of a display module of the head-mounted display equipment; the first driver is electrically connected with the diopter adjusting lens and the controller; the controller is further used for controlling the first driver to drive the diopter adjusting lens to deform according to the deformation vector matched with the vision state. The head-mounted display device can be used by wearers with visual defects.

Description

Technical Field

[0001] This application relates to the field of wearable technology, and more specifically, to a head-mounted display device, a control method for the head-mounted display device, an apparatus, and a medium. Background Technology

[0002] The essence of XR head-mounted display devices such as AR / VR / MR is to achieve interaction between people and the virtual world by merging the virtual world with the real world. During the interaction, the display module in the head-mounted display device plays the role of displaying the virtual world to the near eye, projecting the virtual image corresponding to the virtual world onto the retina of the human eye to form an image.

[0003] However, in traditional technology, the focal length of display modules is fixed, generally designed for normal, healthy human eyes. This means that current display modules can only be adapted to normal, healthy human eyes. For users with visual impairments, such as nearsighted users, the image displayed by the module cannot be seen clearly without their glasses, but wearing glasses makes them unsuitable for head-mounted display devices.

[0004] Therefore, how to provide a head-mounted display device that can be used by visually impaired users has become an urgent technical problem to be solved. Summary of the Invention

[0005] One objective of this application is to provide a new technical solution for head-mounted display devices.

[0006] According to a first aspect of this application, a head-mounted display device is provided, comprising:

[0007] An image sensor is used to acquire images of the eyeballs of the wearer of the head-mounted display device;

[0008] A controller, connected to the image sensor, is used to determine the wearer's visual status based on the eye image;

[0009] A diopter-adjustable lens, wherein the diopter-adjustable lens is disposed on the display module of the head-mounted display device on the side closest to the wearer's eyeball;

[0010] A first actuator is connected to both the diopter-adjusting lens and the controller.

[0011] The controller is also used to control the first driver to drive the diopter-adjusting lens to undergo deformation of a deformation vector that matches the visual state.

[0012] Optionally, the head-mounted display device further includes:

[0013] A zoom lens, wherein the zoom lens is disposed between the display module and the diopter-adjustable lens;

[0014] The second driver is connected to both the controller and the zoom lens;

[0015] The controller is also configured to determine the current virtual image distance based on the eye image, and control the second driver to adjust the focal length of the zoom lens to match the current virtual image distance.

[0016] Optionally, the diopter-adjustable lens is an Alvarez zoom system.

[0017] Optionally, the zoom lens is a liquid crystal zoom lens.

[0018] Optionally, the first driver includes: a motor drive unit and a motor, wherein:

[0019] The motor drive unit is connected to the controller and the motor respectively, and the motor is connected to the diopter-adjustable lens.

[0020] According to a second aspect of this application, a control method for a head-mounted display device is provided, the method being applied to a head-mounted display device as described in any one of the first aspects, comprising:

[0021] Acquire images of the wearer's eyes captured by the image sensor in the head-mounted display device;

[0022] The wearer's visual status is determined based on the eye image;

[0023] The first driver in the head-mounted display device is controlled to drive the diopter-adjusting lens to produce a deformation vector that matches the visual state.

[0024] Optionally, determining the wearer's visual status based on the eye image includes:

[0025] The wearer's identity information is determined based on the eye image;

[0026] Based on the identity information, determine whether there exists a deformation vector in the preset mapping relationship that matches the identity information;

[0027] In the absence of such an image, the wearer's visual status is determined based on the eye image.

[0028] The preset mapping relationship stores the correspondence between different identity information and matching deformation vectors.

[0029] Optionally, the method further includes:

[0030] The deformation vectors that match the identity information with the vision state are stored as a set of correspondences in the preset mapping relationship.

[0031] Optionally, the method further includes:

[0032] Based on the eye image, determine the current virtual image distance;

[0033] Based on the current virtual image distance, the second driver in the head-mounted display device is controlled to adjust the focal length of the zoom lens to match the current virtual image distance.

[0034] Optionally, after the method adjusts the focal length of the zoom lens to match the current virtual image distance using the second driver in the head-mounted display device, it further includes:

[0035] Acquire the current eye image of the wearer captured by the image sensor;

[0036] Based on the current eye image, determine whether the wearer's eye is in a preset state, wherein the preset state is a state that causes a change in the virtual image distance;

[0037] In the case of yes, the current eye image is used as the eye image, and the step of determining the current virtual image distance based on the eye image is repeated.

[0038] Optionally, the preset state is a blinking state or an eye fatigue state.

[0039] According to a third aspect of this application, a control device for a head-mounted display device is provided, the device being applied to a head-mounted display device as described in any one of the first aspects, comprising:

[0040] The acquisition module is used to acquire eye images of the wearer of the head-mounted display device collected by the image sensor in the head-mounted display device;

[0041] A determining module is used to determine the wearer's visual status based on the eye image;

[0042] A control module is used to control the first driver in the head-mounted display device to drive the diopter-adjusting lens to produce a deformation vector that matches the visual state.

[0043] According to a fourth aspect of this application, another head-mounted display device is provided, the head-mounted display device including the control device for the head-mounted display device as described in the third aspect;

[0044] Alternatively, the head-mounted display device includes a memory and a processor, the memory for storing computer instructions and the processor for retrieving the computer instructions from the memory to perform the control method of the head-mounted display device as described in any of the second aspects.

[0045] According to a fifth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a control method for a head-mounted display device according to any one of the second aspects.

[0046] This application provides a head-mounted display device, comprising: an image sensor for acquiring eye images of a wearer; a controller connected to the image sensor for determining the wearer's visual state based on the eye images; a diopter-adjustable lens disposed on the display module of the head-mounted display device closer to the wearer's eyeball; and a first driver connected to both the diopter-adjustable lens and the controller. The controller further controls the first driver to drive the diopter-adjustable lens to undergo deformation of a deformation vector matching the visual state. The head-mounted display device provided by this application can be used by wearers with visual impairments.

[0047] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0049] Figure 1 This is a schematic diagram of the structure of a head-mounted display device provided in this application. Figure 1 ;

[0050] Figure 2 This is a schematic diagram of the structure of a head-mounted display device provided in this application. Figure 2 ;

[0051] Figure 3 This is a schematic flowchart of a control method for a head-mounted display device provided in this application;

[0052] Figure 4 This is a schematic diagram of the structure of a control device for a head-mounted display device provided in this application;

[0053] Figure 5 This is a schematic diagram of the structure of a head-mounted display device provided in this application. Figure 3 . Detailed Implementation

[0054] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0055] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0056] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0057] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0059] This application provides a head-mounted display device 100, which may be exemplarily AR / VR / MR head-mounted glasses or helmets. Figure 1 and Figure 2 As shown, the head-mounted display device 100 includes:

[0060] Image sensor 101 is used to acquire images of the eyes of the wearer of the head-mounted display device;

[0061] Controller 102, which is connected to image sensor 101, is used to determine the wearer's visual status based on eye images;

[0062] The diopter-adjustable lens 103 is disposed on the side of the display module 107 of the head-mounted display device closer to the wearer's eyes;

[0063] The first driver 104 is connected to the diopter-adjustable lens 103 and the controller 102 respectively;

[0064] The controller 102 is also used to control the first driver 104 to drive the diopter-adjusting lens 103 to deform a deformation vector that matches the visual state.

[0065] It should be noted that, as Figure 2As shown, the head-mounted display device 100 also includes a display module 107, which consists of a panel module 1071 and a pancake module 1072.

[0066] In this embodiment, the image sensor 101 acquires images of the wearer's eyes while the display module 107 is displaying an image. Specifically, the image sensor 101 is used to acquire images of the wearer's eyes when viewing the image displayed on the display module 107. The image sensor 101 is connected to the controller 102, specifically through an electrical connection. After acquiring the wearer's eye images, the image sensor 101 sends the images to the controller 102 based on the electrical connection. The image sensor 101 is exemplarily an eye-tracking camera (ETcamera). The electrical connection between the image sensor 101 and the controller 102 can be exemplarily as follows: Figure 2 The image shows the electrical connection method based on CSI.

[0067] The wearer's eye image corresponds to a first eye feature, which is an eye feature reflecting the wearer's corresponding visual state, such as pupil size. Based on this, after acquiring the wearer's eye image from the image sensor 101, the controller 102 analyzes the first eye feature corresponding to the eye image and determines the wearer's visual state based on the first eye feature. The visual state is one of: normal visual acuity, myopic refractive error when nearsighted, or hyperopic refractive error when farsighted. Furthermore, the controller 102 can be, for example, an application processor or a microprocessor.

[0068] 103 diopter-adjustable lenses Figure 2 As shown, the display module 107 of the head-mounted display device is positioned on the side closest to the wearer's eyeball. That is, the wearer's eyeball views the display module 107 through the diopter-adjustable lens 103. In this embodiment, the diopter corresponding to the diopter-adjustable lens 103 can be adjusted by controlling the deformation of the diopter-adjustable lens 103; that is, the diopter changes when the diopter-adjustable lens 103 is deformed.

[0069] In one embodiment of this application, the diopter-adjustable lens 103 can be exemplarily an Alvarez zoom system. The Alvarez zoom system consists of two lenses, and zoom functionality is achieved by moving the two lenses in opposite directions perpendicular to the optical axis. Each lens includes a plane and a freeform surface; the freeform surface in the second lens is obtained by rotating the freeform surface in the first lens by 180°. When the two lenses are fully aligned, they are equivalent to parallel glass plates, corresponding to a diopter of 0. When the convex portions of the two lenses are facing each other, they are equivalent to convex lenses, converging the light beam, corresponding to a hyperopic diopter; when the concave portions of the two lenses are facing each other, they are equivalent to concave lenses, diverging the light beam, corresponding to a myopic diopter.

[0070] The first actuator 104 is connected to both the diopter-adjusting lens 103 and the controller 102. Specifically, the first actuator 104 is mechanically connected to the diopter-adjusting lens 103 and electrically connected to the controller 102. Under the control of the controller 102, the first actuator 104 can drive the diopter-adjusting lens 103 to deform. When the diopter-adjusting lens 103 deforms, its diopter changes. In the case of the diopter-adjusting lens 103 being an Alvarez zoom system, driving the diopter-adjusting lens to deform specifically involves driving the two lenses in the Alvarez zoom system to move relative to each other along a direction perpendicular to the optical axis.

[0071] In one embodiment of this application, such as Figure 1 As shown, the first driver 104 specifically includes a motor drive unit 1042 and a motor 1041. The motor drive unit 1042 is connected to the controller 102 and the motor 1041, respectively, and the motor 1041 is connected to the diopter adjustment lens 103.

[0072] In this embodiment, specifically, the motor drive unit 1042 is electrically connected to both the controller 102 and the motor 1041, and the motor 1041 is mechanically connected to the diopter-adjustable lens 103. The controller 102 controls the operation of the motor 1041 through the motor drive unit 1042. When the motor 1041 is running, it drives the diopter-adjustable lens 103 to deform. In one example, the electrical connection between the controller 102 and the motor drive unit 1042 can be exemplarily an SPI / I2C-based electrical connection, and the electrical connection between the motor drive unit 1042 and the motor 1041 can be exemplarily an PWM / Sine Signal-based electrical connection.

[0073] Furthermore, upon determining the visual acuity, the controller 102 determines the deformation vector of the diopter-adjusting lens 103 that matches the wearer's visual acuity. Further, the controller 102 controls the first actuator 104 to drive the diopter-adjusting lens 103 to deform according to the aforementioned deformation vector. At this time, the diopter of the diopter-adjusting lens 103 matches the visual acuity determined by the controller 102.

[0074] Taking the diopter-adjustable lens 103 as an example of the Alvarez zoom system, the deformation vector includes the direction of relative movement of the two lenses in the Alvarez zoom system and the amount of relative movement.

[0075] Based on the above, when the refractive power of the adjusting lens 103 corresponds to myopia, the adjusting lens 103 can be used as a lens for myopia glasses that matches the wearer's visual acuity. When the refractive power of the adjusting lens 103 corresponds to hyperopia, the adjusting lens 103 can be used as a lens for hyperopia glasses that matches the wearer's visual acuity. When the refractive power of the adjusting lens 103 is 0, the adjusting lens 103 can be used as a regular clear lens without refractive power.

[0076] Based on the above, it can be seen that the head-mounted display device 100 provided in this application can be used by wearers with visual impairments.

[0077] This application provides a head-mounted display device, comprising: an image sensor for acquiring eye images of a wearer; a controller connected to the image sensor for determining the wearer's visual state based on the eye images; a diopter-adjustable lens disposed on the display module of the head-mounted display device closer to the wearer's eyeball; and a first driver electrically connected to both the diopter-adjustable lens and the controller. The controller further controls the first driver to drive the diopter-adjustable lens to undergo deformation of a deformation vector matching the visual state. The head-mounted display device provided by this application can be used by wearers with visual impairments.

[0078] In one embodiment of this application, such as Figure 1 and Figure 2 As shown, the head-mounted display device 100 provided in this application also includes:

[0079] A zoom lens 106 is disposed between the display module 107 and the diopter-adjustable lens 103.

[0080] The second driver 105 is electrically connected to the controller 102 and the zoom lens 106 respectively.

[0081] The controller 102 is also used to determine the current virtual image distance based on the eye image and control the second driver 105 to adjust the focal length of the zoom lens 106 to match the current virtual image distance.

[0082] In this embodiment, as Figure 2 As shown, the zoom lens 106 is disposed between the display module 107 and the diopter-adjusting lens 103. The focal length of the zoom lens 106 can be adjusted under the control of the second driver 105. In one example, the zoom lens 106 can be a liquid crystal zoom lens, and the deflection of liquid crystal molecules in the liquid crystal zoom lens can be controlled by applying a voltage signal to the liquid crystal zoom lens, thereby realizing the focusing of the liquid crystal zoom lens.

[0083] When the zoom lens 106 can be a liquid crystal zoom lens, the second driver 105 can specifically be a voltage source capable of outputting different voltage signals. The electrical connection between the second driver 105 and the controller 102 can be exemplarily an SPI / I2C-based connection. And the electrical connection between the second driver 105 and the zoom lens 106 can be exemplarily a PWM / Sine Signal-based connection.

[0084] Furthermore, based on the eye image acquired by the image sensor, the controller 102 determines the depth value corresponding to the wearer's gaze point along the optical axis, and records the reciprocal of the depth value corresponding to the wearer's gaze point along the optical axis as the current virtual image distance. Even further, the controller controls the second driver 105 to adjust the focal length of the zoom lens 106 to match the current virtual image distance.

[0085] Based on this embodiment, the current virtual image distance matches the focal length of the zoom lens 106, thus avoiding convergence conflict, i.e., the VAC problem. In other words, the head-mounted display device provided in this application can also avoid convergence conflict, i.e., the VAC problem.

[0086] In relation to the head-mounted display device provided in any of the above embodiments, this application also provides a control method for the head-mounted display device, such as... Figure 3 As shown, the method includes the following steps S310 to S330.

[0087] Step S310: Obtain an image of the wearer's eyeballs from the image sensor in the head-mounted display device.

[0088] In one embodiment of this application, step S310 can be implemented as follows: acquiring an eye image of the wearer of the head-mounted display device when viewing an image displayed on the display module, captured by an image sensor.

[0089] Step S320: Determine the wearer's visual status based on the eye image.

[0090] In this embodiment, the wearer's eye image corresponds to a first eye feature, which is an eye feature reflecting the wearer's corresponding visual acuity, such as pupil size. Based on this, after acquiring the wearer's eye image from the image sensor, the first eye feature corresponding to the eye image is analyzed, and the wearer's visual acuity is determined according to the first eye feature. The visual acuity is one of the following: normal visual acuity, myopic refractive error when nearsighted, or hyperopic refractive error when farsighted.

[0091] In step S330, the first driver in the head-mounted display device is controlled to drive the diopter adjustment lens to undergo a deformation that matches the visual state.

[0092] In this embodiment, after determining the visual acuity, a deformation vector of the diopter-adjusting lens that matches the wearer's visual acuity is determined. Further, a first actuator is controlled to drive the diopter-adjusting lens to deform according to the aforementioned deformation vector. At this time, the diopter of the diopter-adjusting lens matches the visual acuity determined by the controller.

[0093] Taking the Alvarez zoom system as an example, the deformation vector includes the direction of relative movement of the two lenses in the Alvarez zoom system and the amount of relative movement.

[0094] Based on the above, when the refractive power of the accommodative lens corresponds to myopia, it can be used as a lens for myopia glasses that matches the wearer's visual acuity. When the refractive power of the accommodative lens corresponds to hyperopia, it can be used as a lens for hyperopia glasses that matches the wearer's visual acuity. When the refractive power of the accommodative lens is 0, it can be used as a regular clear lens without refractive power.

[0095] Based on the above, it can be seen that the head-mounted display device controlled by the control method of the head-mounted display device provided in this application can be used by wearers with visual impairments.

[0096] In one embodiment of this application, step S320 is implemented by the following steps S321 to S323.

[0097] Step S321: Determine the wearer's identity information based on the eye image.

[0098] In this application, the eye image corresponds to a second eye feature, which is an eye feature that reflects the wearer's identity, such as interpupillary distance. The second eye features corresponding to the eye images of different wearers are not entirely the same. Therefore, the wearer's identity information can be determined based on the second eye features corresponding to the wearer's eye image. In one embodiment, the second eye features corresponding to the eye image can be directly used as the wearer's identity information.

[0099] Step S322: Based on the identity information, determine whether there is a deformation vector in the preset mapping relationship that matches the identity information.

[0100] The preset mapping relationship stores the correspondence between different identity information and the matching deformation vector.

[0101] In this embodiment, the preset mapping relationship is initially empty. For a wearer of the head-mounted display device controlled by the control method of the head-mounted display device provided in this application for the first time, the head-mounted display device executes the control method of the head-mounted display device provided in this application, records the wearer's identity information determined based on step S321 above, and records the deformation vector determined based on step S330 above. The previously recorded wearer's identity information and deformation vector are stored as a set of correspondences in the preset mapping relationship. Based on this, as different wearers wear the head-mounted display device, the correspondences in the preset mapping relationship continuously increase.

[0102] Step S323: If the condition is not present, determine the wearer's visual status based on the eye image.

[0103] In this embodiment, if the device is not present, it indicates that the current wearer is a user of the head-mounted display device controlled by the control method provided in this application for the first time. In this case, step S320 is executed to determine the wearer's visual state based on the eye image. Further, step S330 is executed.

[0104] Corresponding to step S323 above, if the condition exists, it indicates that the current wearer is not a first-time user of the head-mounted display device controlled by the control method provided in this application. In this case, the deformation vector corresponding to the identity information that matches the identity information determined in step S321 is directly searched in the preset mapping relationship, and the found deformation vector is used as the deformation vector generated by the first driver driving the diopter adjustment lens. This eliminates the need to perform the step of determining the wearer's visual state based on the eye image. This reduces the computational resource overhead of the head-mounted display device.

[0105] Based on the steps S321 to S323 above, the control method for the head-mounted device provided in this application further includes the following step S340.

[0106] Step S340: The deformation vectors that match the identity information and the visual state are stored as a set of correspondences in the preset mapping relationship.

[0107] It is understandable that the identity information and vision status in step S340 above correspond to the same wearer.

[0108] Based on the above step S340, the corresponding relationships in the preset mapping relationship can be added. When the wearer with the identity information in this set of corresponding relationships uses the head-mounted display device next time, the data in the preset mapping relationship can be directly called, which provides a basis for reducing the computing resource overhead of the head-mounted display device.

[0109] In one embodiment of this application, the control method for a head-mounted display device provided in this application further includes the following steps S350 and S360.

[0110] Step S350: Determine the current virtual image distance based on the eye image.

[0111] Step S360: Based on the current virtual image distance, control the second driver in the head-mounted display device to adjust the focal length of the zoom lens to match the current virtual image distance.

[0112] In this embodiment, based on the eye image acquired by the image sensor, the depth value corresponding to the wearer's gaze point along the optical axis is determined, and the reciprocal of the depth value corresponding to the wearer's gaze point along the optical axis is recorded as the current virtual image distance. Furthermore, the second driver is controlled to adjust the focal length of the zoom lens to match the current virtual image distance. In this way, the current virtual image distance matches the focal length of the zoom lens, avoiding convergence conflict, i.e., the VAC problem. That is to say, the control method for the head-mounted display device provided in this application can also avoid the occurrence of convergence conflict, i.e., the VAC problem.

[0113] It is understood that steps S310 to S360 are performed during the calibration phase at the beginning of the user's use of the head-mounted display device. In one embodiment of this application, after the aforementioned calibration is completed, the control method for the head-mounted display device provided in this application further includes steps S370 to S390 after step S360.

[0114] Step S370: Acquire the current eye image of the wearer captured by the image sensor.

[0115] Step S380: Based on the current eye image, determine whether the wearer's eye is in a preset state.

[0116] The preset state is the state that causes a change in the distance of the virtual image.

[0117] In one embodiment, the aforementioned preset state can be a state that causes a change in the distance of the virtual image.

[0118] It should be noted that step S380 can be implemented using conventional techniques, and this embodiment does not limit the specific implementation of step S380.

[0119] Step S390: If so, use the current eye image as the eye image and repeat the step of determining the current virtual image distance based on the eye image.

[0120] In this embodiment, if the wearer's eye is in a preset state, it indicates that the wearer's current virtual image distance has changed. At this time, the current virtual image distance is no longer matched with the focal length of the zoom lens. The current eye image is then used as the eye image for the calibration phase, i.e., as the eye image in step S350 above. Step S350 is then repeated to readjust the focal length of the zoom lens in the head-mounted display device to match the new current virtual image distance, controlled by the second driver. This avoids convergence conflict (VAC) problems during subsequent use of the head-mounted display device.

[0121] Corresponding to step S390 above, if the wearer's eyeballs are not in the preset state, it indicates that the wearer's distance to the current virtual image has not changed. In this case, step S380 above is then executed to determine whether the wearer's eyeballs are in the preset state.

[0122] This application also provides a control device 400 for a head-mounted display device, such as... Figure 4 As shown, the device is applied to any of the head-mounted display devices provided in the above-described head-mounted display device embodiments, comprising:

[0123] The acquisition module 410 is used to acquire eye images of the wearer of the head-mounted display device collected by the image sensor in the head-mounted display device;

[0124] The determining module 420 is used to determine the wearer's visual status based on the eye image;

[0125] The control module 430 is used to control the first driver in the head-mounted display device to drive the diopter-adjusting lens to generate a deformation vector that matches the visual state.

[0126] In one embodiment of this application, the determining module 420 is specifically used to determine the wearer's identity information based on the eyeball image;

[0127] Based on the identity information, determine whether there exists a deformation vector in the preset mapping relationship that matches the identity information;

[0128] In the absence of such an image, the wearer's visual status is determined based on the eye image.

[0129] The preset mapping relationship stores the correspondence between different identity information and matching deformation vectors.

[0130] In one embodiment of this application, the control device 400 for the head-mounted display device provided in this application further includes:

[0131] The storage module is used to store the deformation vectors that match the identity information and the vision state as a set of correspondences in the preset mapping relationship.

[0132] In one embodiment of this application, the determining module 420 is further configured to determine the current virtual image distance based on the eye image;

[0133] The control module 430 is also configured to control the second driver in the head-mounted display device to adjust the focal length of the zoom lens to match the current virtual image distance based on the current virtual image distance.

[0134] In one embodiment of this application, the acquisition module 410 is further configured to acquire the current eye image of the wearer collected by the image sensor;

[0135] The determining module 420 is further configured to determine, based on the current eye image, whether the wearer's eye is in a preset state, wherein the preset state is a state that causes a change in the virtual image distance;

[0136] In this embodiment, the control device 400 for the head-mounted display device provided in this application further includes:

[0137] A repeat module is used, if the condition is met, to use the current eye image as the eye image and repeat the step of determining the current virtual image distance based on the eye image.

[0138] In one embodiment of this application, the preset state is a blinking state or an eye fatigue state.

[0139] This application also provides a head-mounted display device 500, which includes a control device 400 for a head-mounted display device as described in any of the above-described device embodiments.

[0140] Or, such as Figure 5As shown, the head-mounted display device includes a memory 510 and a processor 520. The memory 510 is used to store computer instructions, and the processor 520 is used to retrieve the computer instructions from the memory 510 to execute the control method of the head-mounted display device as described in any of the above method embodiments.

[0141] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method for a head-mounted display device according to any one of the above method embodiments.

[0142] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0143] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0144] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0145] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0146] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0148] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0150] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A head-mounted display device, characterized in that, include: An image sensor is used to acquire images of the eyeballs of the wearer of the head-mounted display device; A controller, connected to the image sensor, is used to determine the wearer's visual status based on the eye image; A diopter-adjustable lens, wherein the diopter-adjustable lens is disposed on the display module of the head-mounted display device on the side closest to the wearer's eyeball; A first actuator is connected to both the diopter-adjusting lens and the controller. The controller is also used to control the first driver to drive the diopter-adjusting lens to undergo deformation of a deformation vector that matches the visual state.

2. The head-mounted display device according to claim 1, characterized in that, The head-mounted display device also includes: A zoom lens, wherein the zoom lens is disposed between the display module and the diopter-adjustable lens; The second driver is connected to both the controller and the zoom lens; The controller is also configured to determine the current virtual image distance based on the eye image, and control the second driver to adjust the focal length of the zoom lens to match the current virtual image distance.

3. The head-mounted display device according to claim 1, characterized in that, The diopter-adjustable lens is an Alvarez zoom system.

4. The head-mounted display device according to claim 2, characterized in that, The zoom lens is a liquid crystal zoom lens.

5. The head-mounted display device according to claim 1, characterized in that, The first driver includes: a motor drive unit and a motor, wherein: The motor drive unit is connected to the controller and the motor respectively, and the motor is connected to the diopter-adjustable lens.

6. A control method for a head-mounted display device, characterized in that, The method is applied to a head-mounted display device as described in any one of claims 1 to 5, comprising: Acquire images of the wearer's eyes captured by the image sensor in the head-mounted display device; The wearer's visual status is determined based on the eye image; The first driver in the head-mounted display device is controlled to drive the diopter-adjusting lens to produce a deformation vector that matches the visual state.

7. The method according to claim 6, characterized in that, Determining the wearer's visual status based on the eye image includes: The wearer's identity information is determined based on the eye image; Based on the identity information, determine whether there exists a deformation vector in the preset mapping relationship that matches the identity information; In the absence of such an image, the wearer's visual status is determined based on the eye image. The preset mapping relationship stores the correspondence between different identity information and matching deformation vectors.

8. The method according to claim 7, characterized in that, The method further includes: The deformation vectors that match the identity information with the vision state are stored as a set of correspondences in the preset mapping relationship.

9. The method according to claim 6, characterized in that, The method further includes: Based on the eye image, determine the current virtual image distance; Based on the current virtual image distance, the second driver in the head-mounted display device is controlled to adjust the focal length of the zoom lens to match the current virtual image distance.

10. The method according to claim 9, characterized in that, After the second driver in the head-mounted display adjusts the focal length of the zoom lens to match the current virtual image distance, the method further includes: Acquire the current eye image of the wearer captured by the image sensor; Based on the current eye image, determine whether the wearer's eye is in a preset state, wherein the preset state is a state that causes a change in the virtual image distance; In the case of yes, the current eye image is used as the eye image, and the step of determining the current virtual image distance based on the eye image is repeated.

11. The method according to claim 10, characterized in that, The preset state is either blinking or visual fatigue.

12. A control device for a head-mounted display device, characterized in that, The device is applied to a head-mounted display device as described in any one of claims 1 to 5, comprising: The acquisition module is used to acquire eye images of the wearer of the head-mounted display device collected by the image sensor in the head-mounted display device; A determining module is used to determine the wearer's visual status based on the eye image; A control module is used to control the first driver in the head-mounted display device to drive the diopter-adjusting lens to produce a deformation vector that matches the visual state.

13. A head-mounted display device, characterized in that, The head-mounted display device includes the control device for the head-mounted display device as described in claim 12; Alternatively, the head-mounted display device includes a memory and a processor, the memory for storing computer instructions, and the processor for retrieving the computer instructions from the memory to execute the control method of the head-mounted display device as claimed in any one of claims 6-11.

14. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the control method for the head-mounted display device according to any one of claims 6-11.

Images