Visual aid device, control device and head-mounted display system

Abstract

The application discloses a visual auxiliary device, a control device and a head-mounted display system, and relates to the field of wearable technology. The visual auxiliary device comprises an image sensor, the image sensor is used for collecting eyeball images of a wearer of a head-mounted display device; a refractive power adjusting lens; a first driver connected with the refractive power adjusting lens; the visual auxiliary device is in communication connection with the control device; wherein the control device comprises a processor, the processor acquires the eyeball images collected by the image sensor from the visual auxiliary device based on the communication connection, determines the vision state of the wearer according to the eyeball images, and controls the first driver to drive the refractive power adjusting lens to deform in a manner matched with the vision state. When the head-mounted display device is used in cooperation with the visual auxiliary device, the head-mounted display device can be used by a wearer with vision defects.

Description

Technical Field

[0001] This application relates to the field of wearable technology, and more specifically, to a visual assistive device, control device, and head-mounted display system. Background Technology

[0002] AR / VR / MR / XR head-mounted display devices enable interaction between humans and the virtual world by merging the virtual and real worlds. During this interaction, the optical engine module in the head-mounted display device plays the role of displaying the virtual world near the eye, projecting the corresponding virtual images onto the retina of the human eye for imaging.

[0003] Currently, the focal length of the optical engine module in existing head-mounted displays is designed based on the normal, healthy human eye. Therefore, the focal length of the optical engine module is fixed and can only be adapted to the normal, healthy human eye. However, for users with visual impairments such as myopia, they cannot see a clear image when using head-mounted displays.

[0004] Therefore, how to enable visually impaired users to see a clear picture when using head-mounted display devices has become an urgent technical problem to be solved. Summary of the Invention

[0005] One objective of this application is to provide a new technical solution for a visual aid device for a head-mounted display device.

[0006] According to a first aspect of this application, a visual aid device for a head-mounted display device is provided, comprising:

[0007] An image sensor is used to acquire images of the eyeballs of the wearer of the head-mounted display device;

[0008] Adjustable lenses;

[0009] A first actuator, which is connected to the diopter-adjustable lens;

[0010] The visual aid device is communicatively connected to the control device;

[0011] The control device includes a processor that acquires eye images collected by the image sensor from the vision assistive device via the communication connection, determines the wearer's visual state based on the eye images, and controls the first driver to drive the diopter-adjustable lens to undergo a deformation that matches the visual state.

[0012] Optionally, the vision assist device includes a first interface, the first interface including a first electrical connection terminal and a second electrical connection terminal, the first electrical connection terminal being connected to the image sensor, and the second electrical connection terminal being connected to the first driver;

[0013] The control device includes a second connector, which includes a third electrical connection terminal and a fourth electrical connection terminal. The third electrical connection terminal and the fourth electrical connection terminal are respectively connected to the processor. When the first connector is plugged into the second connector, the first electrical connection terminal is connected to the third electrical connection terminal, and the second electrical connection terminal is connected to the fourth electrical connection terminal. The control device is communicatively connected to the visual aid device. The processor receives the eye image acquired by the image sensor through the third electrical connection terminal and controls the first driver through the fourth electrical connection terminal.

[0014] Optionally, the visual aid device further includes:

[0015] A zoom lens, wherein the zoom lens is disposed on the inner surface of the diopter-adjustable lens, and the inner surface is the surface of the diopter-adjustable lens that is close to the wearer's eyeball;

[0016] A second driver is connected to the zoom lens;

[0017] The processor determines the current virtual image distance based on the eye image and controls the second driver to adjust the focal length of the zoom lens to match the current virtual image distance via the communication connection.

[0018] Optionally, the vision assist device includes a first interface, the first interface including a fifth electrical connection terminal, the fifth electrical connection terminal being connected to the second driver;

[0019] The control device includes a second connector, which includes a sixth electrical connection terminal. The sixth electrical connection terminal is connected to the processor. When the first connector is plugged into the second connector, the fifth electrical connection terminal is connected to the sixth electrical connection terminal. The processor controls the second driver through the sixth electrical connection terminal to adjust the focal length of the zoom lens to match the current virtual image distance.

[0020] Optionally, the image sensor includes:

[0021] An eye-tracking camera, wherein the eye-tracking camera is disposed at the first end of the inner surface;

[0022] An infrared emitter is disposed at the second end of the inner surface;

[0023] The zoom lens is located between the first end and the second end of the inner surface. The processor receives eye images captured by the eye-tracking camera while controlling the infrared emitter to emit infrared light toward the eyeball via the communication connection.

[0024] Optionally, the visual aid device includes:

[0025] First connecting part;

[0026] Wherein, the first connecting part is used to mechanically connect with the second connecting part in the head-mounted display device. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device includes a display screen, the visual aid device is fixed to the side of the display screen near the wearer's eyeball. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device does not include a display screen, the visual aid device is fixed to the side of the light guide device of the head-mounted display device near the wearer's eyeball.

[0027] Optionally, the vision assist device includes a battery and a first connector, the first connector including a power connection terminal, the battery being connected to the power connection terminal, the image sensor and the first driver respectively, and the power connection terminal being used to connect to an external charging power source.

[0028] According to a second aspect of this application, a control device is provided, the control device being communicatively connected to a vision assistive device, comprising:

[0029] The processor is configured to receive eye images acquired by an image sensor in the visual aid device via the communication connection, determine the visual state of the wearer of the head-mounted display device based on the eye images, and control a first driver in the visual aid device via the communication connection to drive the diopter-adjusting lens in the visual aid device to undergo a deformation that matches the visual state.

[0030] Optionally, the control device further includes:

[0031] The second connector includes a third electrical connection terminal and a fourth electrical connection terminal, which are connected to the processor.

[0032] The visual assistive device includes a first connector, which includes a first electrical connection terminal and a second electrical connection terminal. When the first connector is plugged into the second connector, the first electrical connection terminal of the first connector is connected to the third electrical connection terminal, and the second electrical connection terminal of the first connector is connected to the fourth electrical connection terminal. The control device is communicatively connected to the visual assistive device. The processor is used to receive eye images acquired by the image sensor through the third electrical connection terminal and to control the first driver through the fourth electrical connection terminal.

[0033] Optionally, the processor is further configured to determine the current virtual image distance based on the eye image, and control the second driver in the visual aid device via the communication connection to adjust the focal length of the zoom lens in the visual aid device to match the current virtual image distance.

[0034] Optionally, the control device further includes:

[0035] The second connector includes a sixth electrical connection terminal, which is connected to the processor.

[0036] The vision assistive device includes a first connector, which includes a fifth electrical connection terminal. When the first connector is plugged into the second connector, the sixth electrical connection terminal is connected to the fifth electrical connection terminal. The processor is used to control the second driver in the vision assistive device through the sixth electrical connection terminal.

[0037] Optionally, the processor is specifically used for:

[0038] The wearer's identity information is determined based on the eye image;

[0039] Based on the identity information, determine whether there is a deformation in the preset mapping relationship that matches the identity information;

[0040] In the absence of the eye image, the wearer's visual status is determined.

[0041] The processor is further configured to: store the deformations that match the identity information with the visual state as a set of correspondences into the preset mapping relationship;

[0042] If present, the deformation in the preset mapping relationship that matches the identity information is used as the first driver to drive the diopter-adjusting lens to produce a deformation that matches the visual state.

[0043] Optionally, the processor is further configured to:

[0044] Based on the eye image, determine whether the wearer's eye is in a preset state, wherein the preset state is a state that causes a change in the virtual image distance;

[0045] In this case, repeat the steps of determining the current virtual image distance based on the eye image.

[0046] According to a third aspect of this application, a head-mounted display system is provided, the head-mounted display system comprising a visual assistive device as described in any one of the first aspects, a control device as described in any one of the second aspects, and a head-mounted display device;

[0047] Alternatively, if the control device as described in any of the second aspects is integrated into the head-mounted display device, the head-mounted display system includes a visual aid device for the head-mounted display device as described in any of the first aspects and the head-mounted display device.

[0048] This application provides a visual assistance device for a head-mounted display device, comprising: an image sensor for acquiring eye images of a wearer of the head-mounted display device; a diopter-adjusting lens; a first driver connected to the diopter-adjusting lens; and a communication connection between the visual assistance device and a control device. The control device includes a processor that acquires the eye images acquired by the image sensor from the visual assistance device via the communication connection, determines the wearer's visual state based on the eye images, and controls the first driver to drive the diopter-adjusting lens to undergo a deformation matching the visual state. When the visual assistance device provided in this application is placed between the wearer's eyeball and the display screen or light guide device of the head-mounted display device, and a communication connection is established between the visual assistance device and the control device, the head-mounted display device can be used by a wearer with visual impairments.

[0049] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0051] Figure 1 This is a structural schematic diagram of a visual assistive device provided in this application. Figure 1 ;

[0052] Figure 2 This is a schematic diagram of the structure of a control device provided in this application;

[0053] Figure 3 This application provides a structural schematic diagram of a control device and a vision-aiding device. Figure 1 ;

[0054] Figure 4 This application provides a structural schematic diagram of a control device and a vision-aiding device. Figure 2 ;

[0055] Figure 5 This is a structural schematic diagram of a visual assistive device provided in this application. Figure 2 ;

[0056] Figure 6 This is a schematic diagram of the structure of a head-mounted display system provided in this application. Detailed Implementation

[0057] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0058] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0059] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0060] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0061] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0062] This application provides a visual assistance device for a head-mounted display device. When used in conjunction with the head-mounted display device, the head-mounted display device can be used by a wearer with visual impairments. The head-mounted display device can be an integrated head-mounted display device or a separate head-mounted display device. In the case of a separate head-mounted display device, the head-mounted display device includes a head-mounted display and a control device (e.g., a puck device). The control device is communicatively connected to the head-mounted display and controls the head-mounted display.

[0063] like Figure 1 As shown, the visual aid device 100 for a head-mounted display device provided in this application includes:

[0064] Image sensor 101 is used to acquire images of the eyes of the wearer of the head-mounted display device;

[0065] 102 diopter-adjustable lenses;

[0066] The first driver 103 is connected to the diopter-adjustable lens 102;

[0067] Among them, such as Figure 2 As shown, the control device 200 includes a processor 201, which is communicatively connected to the visual aid device 100. The processor 201 acquires eye images collected by the image sensor 101 from the visual aid device 100 based on the communication connection, determines the wearer's visual state based on the eye images, and controls the first driver 103 to drive the diopter adjustment lens 102 to undergo deformation that matches the visual state.

[0068] In this embodiment, the visual aid device 100 and the control device 200 are separate, rather than the control device 200 being integrated into the visual aid device 100. The visual aid device 100 and the control device 200 can communicate with each other, meaning they can exchange data. The visual aid device 100 and the control device 200 can communicate directly or indirectly; this application does not limit the specific communication method used.

[0069] When the head-mounted display device includes a display screen, such as an MR / VR head-mounted display device, the visual aid device 100, when used in conjunction with the head-mounted display device, needs to be placed between the wearer's eyeballs and the display screen of the head-mounted display device. In this case, the wearer views the display screen of the head-mounted display device through the visual aid device 100.

[0070] Correspondingly, in cases where the head-mounted display device does not include a display screen, such as an AR head-mounted display device, the visual aid device 100, when used in conjunction with the head-mounted display device, can be placed between the wearer's eyeball and the light guide device of the head-mounted display device. The light guide device can be an optical waveguide that transmits image light emitted by the optical engine to the wearer's eyeball. In this case, the wearer can see the light guide device of the head-mounted display device through the visual aid device 100.

[0071] In one embodiment of this application, the diopter-adjustable lens 102 can be exemplarily an Alvarez zoom system. The Alvarez zoom system consists of two lenses, and zoom functionality is achieved by moving the two lenses in opposite directions perpendicular to the optical axis. Each lens includes a plane and a freeform surface; the freeform surface in the second lens is obtained by rotating the freeform surface in the first lens by 180°. When the two lenses are fully aligned, they are equivalent to parallel glass plates, corresponding to a diopter of 0. When the convex portions of the two lenses are facing each other, they are equivalent to convex lenses, converging the light beam, corresponding to a hyperopic diopter; when the concave portions of the two lenses are facing each other, they are equivalent to concave lenses, diverging the light beam, corresponding to a myopic diopter.

[0072] The first actuator 103 is connected to the diopter-adjusting lens 102. Specifically, the first actuator 103 is mechanically connected to the diopter-adjusting lens 102. Under the control of the processor 201, the first actuator 103 drives the diopter-adjusting lens 102 to deform. When the diopter-adjusting lens 102 deforms, its diopter changes. In the case of the diopter-adjusting lens 102 being an Alvarez zoom system, driving the diopter-adjusting lens 102 to deform specifically involves driving the two lenses in the Alvarez zoom system to move relative to each other along a direction perpendicular to the optical axis.

[0073] In one embodiment of this application, such as Figure 3 As shown, the first driver 103 specifically includes a motor drive unit 1032 and a motor 1031, wherein the motor drive unit 1032, the motor 1031, and the diopter-adjusting lens 102 are sequentially connected. Furthermore, the motor 1031 and the diopter-adjusting lens 102 are specifically mechanically connected.

[0074] The working principle of the visual assist device 100 and the control device 200 of the head-mounted display device provided in this application will be explained below, with the visual assist device 100 placed between the wearer's eyeball and the display screen or light guide device of the head-mounted display device, and the visual assist device 100 establishing a communication connection with the control device 200.

[0075] When an image is displayed on a head-mounted display device, the image sensor 101 specifically captures eye images of the wearer while viewing the image displayed on the head-mounted display device. The visual aid device 100 transmits the eye images to the processor 201 via a communication connection with the control device 200. Upon receiving the eye images, the processor 201 analyzes the eye features in the images that reflect the wearer's visual state, thereby determining the wearer's visual state. These eye features reflecting the wearer's visual state may include, for example, pupil size, and the visual state may be one of normal vision, myopic refractive error in myopia, or hyperopic refractive error in hyperopia.

[0076] Furthermore, upon determining the visual acuity state, the processor 201 determines the deformation of the diopter-adjusting lens 102 that matches the wearer's visual acuity state. Further, the processor 201, through a communication connection between the visual aid device 100 and the control device 200, controls the first driver 103 to drive the diopter-adjusting lens 102 to undergo the aforementioned deformation matching the wearer's visual acuity state. At this time, the diopter of the diopter-adjusting lens 102 matches the visual acuity state determined by the processor 201. The processor 201 pre-stores a correspondence between visual acuity states and the deformation of the matching diopter-adjusting lens 102, which is obtained experimentally.

[0077] Taking the diopter-adjustable lens 102 as an example of the Alvarez zoom system, the deformation of the diopter-adjustable lens 102 that matches the wearer's visual state is a vector, including the direction of relative movement of the two lenses in the Alvarez zoom system and the amount of relative movement.

[0078] Based on the above, when the refractive power of the adjusting lens 102 corresponds to myopia, the adjusting lens 102 can be used as a lens for myopia glasses that matches the wearer's visual acuity. When the refractive power of the adjusting lens 102 corresponds to hyperopia, the adjusting lens 102 can be used as a lens for hyperopia glasses that matches the wearer's visual acuity. When the refractive power of the adjusting lens 102 is 0, the adjusting lens 102 can be used as a regular clear lens without refractive power.

[0079] Based on the above, it can be seen that when the visual aid device 100 is placed between the wearer's eyeball and the display screen or light guide device of the head-mounted display device, and the visual aid device 100 establishes a communication connection with the control device 200, the head-mounted display device and the visual aid device work together, and the head-mounted display device can be used by wearers with visual impairments.

[0080] This application provides a visual assistance device for a head-mounted display device, comprising: an image sensor for acquiring eye images of a wearer of the head-mounted display device; a diopter-adjusting lens; a first driver connected to the diopter-adjusting lens; and a communication connection between the visual assistance device and a control device. The control device includes a processor that acquires the eye images acquired by the image sensor from the visual assistance device via the communication connection, determines the wearer's visual state based on the eye images, and controls the first driver to drive the diopter-adjusting lens to undergo a deformation matching the visual state. When the visual assistance device provided in this application is placed between the wearer's eyeball and the display screen or light guide device of the head-mounted display device, and a communication connection is established between the visual assistance device and the control device, the head-mounted display device can be used by a wearer with visual impairments.

[0081] In one embodiment of this application, such as Figure 3 As shown, the visual assistive device 100 includes a first interface 104, which includes a first electrical connection terminal 1041 and a second electrical connection terminal 1042. The first electrical connection terminal 1041 is connected to the image sensor 101, and the second electrical connection terminal 1042 is connected to the first driver 103.

[0082] Among them, such as Figure 4 As shown, the control device 200 includes a second connector 202, which includes a third electrical connection terminal 2021 and a fourth electrical connection terminal 2022. The third electrical connection terminal 2021 and the fourth electrical connection terminal 2022 are respectively connected to the processor 201. When the first connector 104 is plugged into the second connector 202, the first electrical connection terminal 1041 is connected to the third electrical connection terminal 2021, and the second electrical connection terminal 1042 is connected to the fourth electrical connection terminal 2022. The control device 200 is communicatively connected to the visual aid device 100. The processor 201 receives the eye image collected by the image sensor 101 through the third electrical connection terminal 2021 and controls the first driver 103 through the fourth electrical connection terminal 2022.

[0083] In this embodiment, as Figure 3 As shown, the visual assistive device 100 includes a first connector 104, which includes a first electrical connection terminal 1041 and a second electrical connection terminal 1042. The control device 200 includes a second connector 202, which includes a third electrical connection terminal 2021 and a fourth electrical connection terminal 2022. The first connector 104 is adapted to the second connector 202.

[0084] For example, the first plug interface 104 and the second plug interface 202 are USB type interfaces. Of course, the first plug interface 104 and the second plug interface 202 can also be other types of interfaces, such as MIPI interfaces, which are not limited in this application.

[0085] It is understandable that the first connector 104 may also include other electrical connection terminals. Similarly, the second connector 202 includes other compatible electrical connection terminals.

[0086] When the visual aid device 100 for the head-mounted display device is used in conjunction with the head-mounted display device, the first connector 104 is plugged into the second connector 202. The first connector 104 can be plugged into the second connector 202 directly or via a wire.

[0087] When the first connector 104 of the vision assist device 100 is plugged into the second connector 202, the first electrical connection terminal 1041 is connected to the third electrical connection terminal 2021, and the second electrical connection terminal 1042 is connected to the fourth electrical connection terminal. Since the first electrical connection terminal 1041 is connected to the image sensor 101 in the vision assist device 100, the second electrical connection terminal 1042 is connected to the first driver 103 in the vision assist device 100, and the third electrical connection terminal 2021 and the fourth electrical connection terminal 2022 are connected to the processor 201, when the first connector 104 of the vision assist device 100 is plugged into the second connector 202 of the control device 200, the processor 201 communicates with the image sensor 101 and the first driver 103 respectively, establishing a communication connection.

[0088] It should be noted that, in this embodiment, as Figure 3 As shown, based on the first driver 103 including a motor 1031 and a motor drive unit 1032, the second electrical connection terminal 1042, the motor drive unit 1032, and the motor 1031 are connected in sequence.

[0089] Based on the above embodiments, this application provides a method for establishing a communication connection between a visual assistive device and a control device.

[0090] Furthermore, in one embodiment of this application, the head-mounted display device is a split-type head-mounted display device, including a head-mounted display and a control device, such as a PUCK device. Based on this, the control device 200 can be the control device for the split-type head-mounted display device. Alternatively, the head-mounted display device is an integrated head-mounted display device, where the processor 201 in the control device 200 is the processor of the integrated head-mounted display device, and the second interface 202 is disposed on the integrated head-mounted display device; that is, the control device 200 is integrated into the head-mounted display device. Or, the control device 200 can be a terminal device independent of the head-mounted display device, such as a personal computer.

[0091] In one embodiment of this application, when the control device 200 is a terminal device independent of the head-mounted display device, an interface such as the second interface 202 may be provided on the head-mounted display device to allow the head-mounted display device to acquire eye images from the first interface of the visual aid device 100. Based on this, the head-mounted display device acts as a relay device, sending the eye images to the control device 200. Furthermore, the control device 200 controls the first driver 103 via the head-mounted display device.

[0092] It should be noted that the first electrical connection terminal 1041 and the second electrical connection terminal 1042 in the first connector 104 can be the same electrical connection terminal, such as a bus-based communication terminal, thus reducing the number of electrical connection terminals in the first connector 104. Similarly, the third electrical connection terminal 2021 and the fourth electrical connection terminal 2022 in the second connector 202 are also like this. Furthermore, the first electrical connection terminal 1041, the second electrical connection terminal 1042, the third electrical connection terminal 2021, and the fourth electrical connection terminal 2022 can be implemented using pins.

[0093] In one embodiment of this application, such as Figure 4 As shown, the visual aid device 100 also includes:

[0094] A zoom lens 106 is disposed on the inner surface of the diopter-adjusting lens 102, and the inner surface is the surface of the diopter-adjusting lens 102 that is close to the wearer's eyeball.

[0095] The second driver 105 is connected to the zoom lens 106;

[0096] The processor 201 determines the current virtual image distance based on the eye image and controls the second driver 105 via a communication connection to adjust the focal length of the zoom lens 106 to match the current virtual image distance.

[0097] In this embodiment, as Figure 4As shown, the zoom lens 106 is disposed on the inner surface of the diopter-adjusting lens 102, which is the surface of the diopter-adjusting lens 102 closest to the wearer's eyeball. This brings the zoom lens 106 closer to the wearer's eyeball, reducing the impact on the field of view (FOV) of the head-mounted display device. Furthermore, because the zoom lens 106 is disposed on the inner surface of the diopter-adjusting lens 102, the wearer's eyeball sees the display screen or light guide device of the head-mounted display device sequentially through the zoom lens 106 and the diopter-adjusting lens 102. Thus, although a change in the diopter of the diopter-adjusting lens 102 will cause a slight change in the virtual image distance, the current virtual image distance is determined based on the change in virtual image distance caused by the change in the diopter of the diopter-adjusting lens 102. Therefore, the current virtual image distance is the accurate distance of the virtual image actually seen by the wearer's eyeball.

[0098] In this embodiment, the focal length of the zoom lens 106 can be adjusted under the control of the second driver 105. In one example, the zoom lens 106 can be a liquid crystal zoom lens, and the deflection of liquid crystal molecules in the liquid crystal zoom lens can be controlled by applying a voltage signal to the liquid crystal zoom lens, thereby realizing the focusing of the liquid crystal zoom lens.

[0099] When the zoom lens 106 can be a liquid crystal zoom lens, the second driver 105 can be specifically a voltage source that can output different voltage signals.

[0100] In this embodiment, the processor 201 determines the depth value corresponding to the wearer's gaze point along the optical axis based on the eye image acquired by the image sensor 101, and records the reciprocal of the depth value corresponding to the wearer's gaze point along the optical axis as the current virtual image distance. Furthermore, based on the communication connection with the visual aid device 100, the processor 201 controls the second driver 105 to adjust the focal length of the zoom lens 106 to match the current virtual image distance.

[0101] Based on this embodiment, the current virtual image distance matches the focal length of the zoom lens 106, thus avoiding the occurrence of convergence conflict, i.e., VAC problems. In other words, when the visual aid 100 for the head-mounted display device provided in this application is used in conjunction with the head-mounted display device, the occurrence of convergence conflict, i.e., VAC problems, can also be avoided.

[0102] In one embodiment of this application, such as Figure 4 As shown, the visual assistive device 100 includes a first interface 104, the first interface 104 includes a fifth electrical connection terminal 1043, and the fifth electrical connection terminal 1043 is connected to the second driver 105.

[0103] Among them, such as Figure 4As shown, the control device 200 includes a second connector 202, which includes a sixth electrical connection terminal 2023. The sixth electrical connection terminal 2023 is connected to the processor 201. When the first connector 104 is plugged into the second connector 202, the fifth electrical connection terminal 1043 is connected to the sixth electrical connection terminal 2023. The processor 202 determines the current virtual image distance based on the eye image and controls the second driver 105 to adjust the focal length of the zoom lens to match the current virtual image distance through the sixth electrical connection terminal 2023.

[0104] In this embodiment, when the first connector 104 is connected to the second connector 202, the fifth electrical connection terminal 1043 is connected to the sixth electrical connection terminal 2023. Since the sixth electrical connection terminal 2023 is connected to the processor 201 and the fifth electrical connection terminal 1043 is connected to the second driver 105, the processor 201 is connected to the second driver 105, and the second driver 106 can be controlled to adjust the focal length of the zoom lens 105 to match the current virtual image distance.

[0105] It should be noted that the first electrical connection terminal 1041, the second electrical connection terminal 1042, and the fifth connection terminal 1043 in the first connector 104 can be the same electrical connection terminal, such as a bus-based communication terminal, thus reducing the number of electrical connection terminals in the first connector 104. Similarly, the third electrical connection terminal 2021, the fourth electrical connection terminal 2022, and the sixth electrical connection terminal 2023 in the second connector 202 are also like this. In addition, the fifth electrical connection terminal 2022 and the sixth electrical connection terminal 2023 can be implemented using pins.

[0106] Based on the above, the previous embodiment provides a method in which the processor 201 controls the second driver 105 to adjust the focal length of the zoom lens to match the current virtual image distance.

[0107] In one embodiment of this application, such as Figure 5 As shown, the image sensor 101 includes:

[0108] An eye-tracking camera 1012 is disposed at the first end of the inner surface;

[0109] Infrared emitter 1011, the infrared emitter is disposed at the second end of the inner surface;

[0110] The zoom lens 106 is located between the first end and the second end of the inner surface. The processor 201 receives eye images captured by the eye-tracking camera 1012 when controlling the infrared emitter 1011 to emit infrared light toward the eyeball via a communication connection.

[0111] In this embodiment, the processor 201 controls the infrared emitter 1011 to emit infrared light toward the eyeball. At this time, the infrared light reaches the eyeball and is reflected and received by the eye-tracking camera 1012. The eye-tracking camera 1012 generates an eyeball image based on the received infrared light. Furthermore, the processor 201 acquires the eyeball image from the eye-tracking camera 1012.

[0112] Furthermore, in this embodiment, the eye-tracking camera 1012 is disposed at the first end of the inner surface, the infrared emitter 1011 is disposed at the second end of the inner surface, and the zoom lens 106 is located between the first end and the second end of the inner surface. This allows for a compact structure of the visual aid device 100, thereby reducing the thickness of the head-mounted display device. Additionally, it avoids interference between the zoom lens 106 and the diopter-adjusting lens 102 on the emitted infrared light and the infrared light reflected by the eyeball.

[0113] In one embodiment of this application, the above-described... Figure 3 or Figure 4 Based on the illustrated embodiment, the first electrical connection terminal 1041 includes a first sub-electrical connection terminal and a second sub-electrical connection terminal, and the third electrical connection terminal 2021 includes a third sub-electrical connection terminal and a fourth sub-electrical connection terminal. When the first connector 104 is plugged into the second connector 202, the third sub-electrical connection terminal is connected to the first sub-electrical connection terminal, and the fourth sub-electrical connection terminal is connected to the second sub-electrical connection terminal. Since the eye-tracking camera 1012 is connected to the first sub-electrical connection terminal and the infrared emitter 1011 is connected to the second sub-electrical connection terminal, the processor 201 is connected to both the infrared emitter 1011 and the eye-tracking camera 1012.

[0114] In one embodiment of this application, the visual assistive device 100 further includes:

[0115] First connecting part;

[0116] The first connecting part is used to mechanically connect with the second connecting part in the control device 200. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device includes a display screen, the visual aid device 100 is fixed to the side of the display screen near the wearer's eyeball. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device does not include a display screen, the visual aid device 100 is fixed to the side of the light guide device of the head-mounted display device near the wearer's eyeball.

[0117] The head-mounted display device is provided with a second connecting part. In this embodiment, by providing a first connecting part on the visual aid device 100, the first connecting part is adapted to the second connecting part on the head-mounted display device for mechanical connection with the second connecting part in the head-mounted display device, so as to stably fix the visual aid device 100 to the side of the display screen or light guide device of the head-mounted display device closer to the wearer's eyeball.

[0118] It is understood that the vision assist device 100 also includes a housing, and the various components in the vision assist device 100 can be placed inside, on the surface or outside the housing as needed.

[0119] In one embodiment of this application, the first connecting part is a hook, and the second connecting part is a retaining ring. In another embodiment of this application, the first connecting part is a first magnetic attracting member, and the second connecting part is a second magnetic attracting member that magnetically attracts the first magnetic attracting member.

[0120] In one embodiment of this application, the visual assistive device 100 includes a battery and a first connector 104. The first connector 104 includes a power connection terminal. The battery is connected to the power connection terminal, the image sensor 101, and the first driver 103, respectively. The power connection terminal is used to connect to an external charging power source.

[0121] In this embodiment, the visual aid device 100 is provided with a battery that powers various components within the visual aid device 100, such as the image sensor 101 and the first driver 103. Of course, if the visual aid device 100 includes a second driver 105, the battery is also connected to the second driver 105 to power the second driver 105.

[0122] Furthermore, the battery can be powered via the first connector 104. Specifically, when an external charging source is connected to the power connection terminal of the first connector 104, the external charging source charges the battery through the power connection terminal of the first connector 104. The external charging source can be a control device capable of reverse charging.

[0123] This application also provides a control device 200, such as... Figure 2 As shown, the control device 200 is communicatively connected to the visual aid device 100, and includes:

[0124] The processor 201 is used to receive eye images collected by the image sensor 101 in the visual aid device 100 via a communication connection, determine the wearer's visual state based on the eye images, and control the first driver 103 in the visual aid device 100 via the communication connection to drive the diopter adjustment lens 102 in the visual aid device 100 to undergo deformation that matches the visual state.

[0125] In one embodiment of this application, such as Figure 3 As shown, the control device 200 also includes:

[0126] The second interface 202 includes a third electrical connection terminal 2021 and a fourth electrical connection terminal 2022, which are connected to the processor 201.

[0127] The visual aid device 100 includes a first connector 104, which includes a first electrical connection terminal 1041 and a second electrical connection terminal 1042. When the first connector 104 is plugged into the second connector 202, the first electrical connection terminal 1041 in the first connector 104 is connected to the third electrical connection terminal 2021, and the second electrical connection terminal 1042 in the first connector 104 is connected to the fourth electrical connection terminal 2022. The control device 200 is communicatively connected to the visual aid device 100. The processor 201 is used to receive the eye image collected by the image sensor 101 through the third electrical connection terminal 2021 and control the first driver 103 through the fourth electrical connection terminal 2022.

[0128] In one embodiment of this application, the processor 201 is further configured to determine the current virtual image distance based on the eye image, and control the second driver 105 in the visual aid device 100 via a communication connection to adjust the focal length of the zoom lens 106 in the visual aid device 100 to match the current virtual image distance.

[0129] In one embodiment of this application, such as Figure 4 As shown, the control device 200 also includes:

[0130] The second connector 202 includes a sixth electrical connection terminal 2023, which is connected to the processor 201.

[0131] The visual aid device 100 includes a first connector 104, which includes a fifth electrical connection terminal 1043. When the first connector 104 is connected to the second connector 202, the sixth electrical connection terminal 2023 is connected to the fifth electrical connection terminal 1043. The processor 201 is used to control the second driver 105 in the visual aid device 100 through the sixth electrical connection terminal 2023.

[0132] It should be noted that the specific implementation of the above control device embodiment can be found in the specific implementation of the control device involved in the above vision assistive device embodiment, and will not be repeated here.

[0133] In one embodiment of this application, the processor 201 specifically determines the wearer's visual state based on the eye image through the following steps S11 to S13. That is, the processor 201 is specifically used to execute the following steps S11 to S13.

[0134] Step S11: Determine the wearer's identity information based on the eye image.

[0135] In this application, the processor 201 determines eye features, such as interpupillary distance, that distinguish the wearer's identity based on an eye image. Based on this, the processor 201 can determine the wearer's identity information according to the determined eye features. In one embodiment, the eye features corresponding to the eye image that distinguish the wearer's identity can be directly used as the wearer's identity information.

[0136] Step S12: Based on the identity information, determine whether there is a deformation in the preset mapping relationship that matches the identity information.

[0137] The preset mapping relationship stores the correspondence between different identity information and the matching deformation.

[0138] In this embodiment, the preset mapping relationship is empty in the initial state. For the wearer of the visual assistive device 100 provided in this application for the first time, there is no deformation in the preset mapping relationship that matches the identity information.

[0139] Step S13: If the condition is not present, determine the wearer's visual status based on the eye image.

[0140] In this embodiment, if the visual aid device 100 provided in this application is not present, it indicates that the current wearer is a first-time user of the device. In this case, the wearer's visual state is determined based on the eye image.

[0141] Based on the above step S13, the processor 201 is also used to execute the following step S14.

[0142] Step S14: The deformations that match the identity information and the visual state are stored as a set of correspondences in the preset mapping relationship.

[0143] It is understandable that the identity information and vision status in step 14 above correspond to the same wearer.

[0144] Based on the above step S14, the corresponding relationships in the preset mapping relationship can be added. When the wearer uses the head-mounted display device and visual aid device again, the identity information in this set of corresponding relationships can directly call the data in the preset mapping relationship, which provides a basis for reducing the computing resource overhead of the control device. That is, the processor 201 is also used to execute the following step S15.

[0145] Step S15: If present, the deformation that matches the identity information in the preset mapping relationship is used as the first driver 103 to drive the diopter adjustment lens 102 to undergo a deformation that matches the visual state.

[0146] Corresponding to step S13 above, if it is determined that there is a deformation in the preset mapping relationship that matches the identity information, it indicates that the current wearer is not using the visual assistive device 100 provided in this application for the first time. In this case, the preset mapping relationship directly searches for a deformation that matches the identity information determined in step S11 above, and uses the found deformation as the first actuator 103 to drive the diopter adjustment lens 102 to produce a deformation that matches the visual state. In this way, it is not necessary to perform the step of determining the wearer's visual state based on the eye image again. At this time, the computational resource overhead of the control device can be reduced.

[0147] In one embodiment of this application, the processor is further configured to perform the following steps S16 and S17.

[0148] Step S16: Determine whether the wearer's eyes are in a preset state based on the eye image.

[0149] The preset state is the state that causes a change in the distance of the virtual image.

[0150] In one embodiment, the preset state may be, for example, a fatigue state or a blinking state. It should be noted that the above step S16 can be implemented by conventional technology, such as determining whether the eyeball is in a fatigue state based on the degree of eyelid closure and blinking frequency within a preset time period before the corresponding moment of the eyeball image. This embodiment does not limit the specific implementation of the above step S16.

[0151] Step S17: If yes, repeat the step of determining the current virtual image distance based on the eye image.

[0152] In this embodiment, if the wearer's eye is in a preset state, it indicates that the wearer's current virtual image distance has changed. At this time, the current virtual image distance no longer matches the focal length of the zoom lens 106. A new current virtual image distance is then determined based on the new eye image, and the second driver 105 in the visual aid device is controlled to adjust the focal length of the zoom lens 106 to match the new current virtual image distance. This avoids the occurrence of convergence conflict (VAC) problems during subsequent use of the head-mounted display device.

[0153] Corresponding to step S17 above, if the wearer's eyeballs are not in the preset state, it indicates that the wearer's distance to the current virtual image has not changed. In this case, step S16 above is then executed to monitor whether the wearer's eyeballs are in the preset state.

[0154] This application also provides a head-mounted display system, which includes any of the visual aids provided in the visual aids embodiment, any of the control devices provided in the control device embodiment, and a head-mounted display device;

[0155] Alternatively, in the case where the control device as described in any of the control device embodiments is integrated into the head-mounted display device, such as... Figure 6 As shown, the head-mounted display system 6000 includes any of the visual assistive devices 100 provided in the visual assistive device embodiments and the head-mounted display device 300.

[0156] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0157] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A visual aid device for a head-mounted display device, characterized in that, include: An image sensor is used to acquire images of the eyeballs of the wearer of the head-mounted display device; Adjustable lenses; A first actuator, which is connected to the diopter-adjustable lens; The visual aid device is communicatively connected to the control device; The control device includes a processor that acquires eye images collected by the image sensor from the vision assistive device via the communication connection, determines the wearer's visual state based on the eye images, and controls the first driver to drive the diopter-adjustable lens to undergo a deformation that matches the visual state.

2. The visual assistive device according to claim 1, characterized in that, The visual assistive device includes a first interface, which includes a first electrical connection terminal and a second electrical connection terminal. The first electrical connection terminal is connected to the image sensor, and the second electrical connection terminal is connected to the first driver. The control device includes a second connector, which includes a third electrical connection terminal and a fourth electrical connection terminal. The third electrical connection terminal and the fourth electrical connection terminal are respectively connected to the processor. When the first connector is plugged into the second connector, the first electrical connection terminal is connected to the third electrical connection terminal, and the second electrical connection terminal is connected to the fourth electrical connection terminal. The control device is communicatively connected to the visual aid device. The processor receives the eye image acquired by the image sensor through the third electrical connection terminal and controls the first driver through the fourth electrical connection terminal.

3. The visual assistive device according to claim 1, characterized in that, The visual aid device also includes: A zoom lens, wherein the zoom lens is disposed on the inner surface of the diopter-adjustable lens, and the inner surface is the surface of the diopter-adjustable lens that is close to the wearer's eyeball; A second driver is connected to the zoom lens; The processor determines the current virtual image distance based on the eye image and controls the second driver to adjust the focal length of the zoom lens to match the current virtual image distance via the communication connection.

4. The visual assistive device according to claim 3, characterized in that, The visual assistive device includes a first interface, the first interface including a fifth electrical connection terminal, the fifth electrical connection terminal being connected to the second driver; The control device includes a second connector, which includes a sixth electrical connection terminal. The sixth electrical connection terminal is connected to the processor. When the first connector is plugged into the second connector, the fifth electrical connection terminal is connected to the sixth electrical connection terminal. The processor controls the second driver through the sixth electrical connection terminal to adjust the focal length of the zoom lens to match the current virtual image distance.

5. The visual assistive device according to claim 3, characterized in that, The image sensor includes: An eye-tracking camera, wherein the eye-tracking camera is disposed at the first end of the inner surface; An infrared emitter is disposed at the second end of the inner surface; The zoom lens is located between the first end and the second end of the inner surface. The processor receives eye images captured by the eye-tracking camera while controlling the infrared emitter to emit infrared light toward the eyeball via the communication connection.

6. The visual assistive device according to any one of claims 1-5, characterized in that, The visual aid device includes: First connecting part; Wherein, the first connecting part is used to mechanically connect with the second connecting part in the head-mounted display device. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device includes a display screen, the visual aid device is fixed to the side of the display screen near the wearer's eyeball. When the first connecting part and the second connecting part are mechanically connected and the head-mounted display device does not include a display screen, the visual aid device is fixed to the side of the light guide device of the head-mounted display device near the wearer's eyeball.

7. The visual assistive device according to claim 6, characterized in that, The vision assist device includes a battery and a first connector. The first connector includes a power connection terminal. The battery is connected to the power connection terminal, the image sensor, and the first driver. The power connection terminal is used to connect to an external charging source.

8. A control device, characterized in that, The control device is communicatively connected to the vision assistive device, including: The processor is configured to receive eye images acquired by an image sensor in the visual aid device via the communication connection, determine the visual state of the wearer of the head-mounted display device based on the eye images, and control a first driver in the visual aid device via the communication connection to drive the diopter-adjusting lens in the visual aid device to undergo a deformation that matches the visual state.

9. The control device according to claim 8, characterized in that, The control device further includes: The second connector includes a third electrical connection terminal and a fourth electrical connection terminal, which are connected to the processor. The visual assistive device includes a first connector, which includes a first electrical connection terminal and a second electrical connection terminal. When the first connector is plugged into the second connector, the first electrical connection terminal of the first connector is connected to the third electrical connection terminal, and the second electrical connection terminal of the first connector is connected to the fourth electrical connection terminal. The control device is communicatively connected to the visual assistive device. The processor is used to receive eye images acquired by the image sensor through the third electrical connection terminal and to control the first driver through the fourth electrical connection terminal.

10. The control device according to claim 8, characterized in that, The processor is also configured to determine the current virtual image distance based on the eye image, and control the second driver in the visual aid device via the communication connection to adjust the focal length of the zoom lens in the visual aid device to match the current virtual image distance.

11. The control device according to claim 10, characterized in that, The control device further includes: The second connector includes a sixth electrical connection terminal, which is connected to the processor. The vision assistive device includes a first connector, which includes a fifth electrical connection terminal. When the first connector is plugged into the second connector, the sixth electrical connection terminal is connected to the fifth electrical connection terminal. The processor is used to control the second driver in the vision assistive device through the sixth electrical connection terminal.

12. The control device according to claim 8, characterized in that, The processor is specifically used for: The wearer's identity information is determined based on the eye image; Based on the identity information, determine whether there is a deformation in the preset mapping relationship that matches the identity information; In the absence of the eye image, the wearer's visual status is determined. The processor is further configured to: store the deformations that match the identity information with the visual state as a set of correspondences into the preset mapping relationship; If present, the deformation in the preset mapping relationship that matches the identity information is used as the first driver to drive the diopter-adjusting lens to produce a deformation that matches the visual state.

13. The control device according to claim 10, characterized in that, The processor is also used for: Based on the eye image, determine whether the wearer's eye is in a preset state, wherein the preset state is a state that causes a change in the virtual image distance; In this case, repeat the steps of determining the current virtual image distance based on the eye image.

14. A head-mounted display system, characterized in that, The head-mounted display system includes a visual aid device as described in any one of claims 1-7, a control device as described in any one of claims 8-13, and a head-mounted display device; Alternatively, if the control device as described in any one of claims 8-13 is integrated into the head-mounted display device, the head-mounted display system includes the visual aid device of the head-mounted display device as described in any one of claims 1-7 and the head-mounted display device.

Images