Augmented reality system integrating display and eye tracking and electronic device

Abstract

The present disclosure relates to an augmented reality system integrating display and eye movement tracking and an electronic device, the display system comprising: a display, an optical waveguide, a gaze point determination unit; the display is configured to emit a first light signal, the first light signal containing image information of a current virtual image; the optical waveguide is configured to transmit the first light signal and transmit the first light signal out of the optical waveguide; the optical waveguide is further configured to transmit a second light signal reflected by an eye and transmit the second light signal out of the optical waveguide to form a third light signal, the third light signal being a distorted light signal; the gaze point determination unit is arranged on a side of the optical waveguide where the second light signal is transmitted out; the gaze point determination unit is configured to determine a gaze point coordinate of the eye based on the third light signal. In this way, the display system of the present disclosure can still obtain accurate gaze point coordinates, improve the accuracy of eye movement judgment, and improve the user experience.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of augmented reality display technology, and in particular to an augmented reality system integrating display and eye movement tracking and an electronic device. BACKGROUND

[0002] Augmented reality technology applies a variety of technical means to simulate virtual information such as computer-generated text, images, three-dimensional models, music, and videos, and applies them to the real world, so that virtual information and real-world information complement each other, thereby achieving "augmentation" of the real world.

[0003] With the rapid development of augmented reality technology, augmented reality application products have shown broad application potential in medical, industrial, educational, and entertainment fields. Augmented reality application products can effectively integrate virtual information with the real world, providing users with immersive experiences and improving information use efficiency and convenience.

[0004] Currently, augmented reality application products have the problems of inaccurate human-computer interaction and unsmooth interaction in use, which reduces the user experience. SUMMARY

[0005] Therefore, the present disclosure provides a display scheme.

[0006] According to an aspect of the present disclosure, a display system is provided, comprising: a display, an optical waveguide, and a gaze point determination unit; the display is configured to emit a first light signal, the first light signal containing image information of a current virtual image; the optical waveguide is configured to transmit the first light signal and transmit the first light signal out of the optical waveguide; the optical waveguide is further configured to transmit a second light signal reflected by an eye and transmit the second light signal out of the optical waveguide to form a third light signal, the third light signal being a distorted light signal; the gaze point determination unit is arranged on one side of the optical waveguide where the second light signal is transmitted out; the gaze point determination unit is configured to determine a gaze point coordinate of the eye based on the third light signal.

[0007] In a possible implementation, the gaze point determination unit comprises: a correction unit, a first imaging unit, and a first identification unit,

[0008] The correction unit is configured to correct the third light signal to obtain a fourth light signal, the difference between the fourth light signal and the second light signal being smaller than the difference between the third light signal and the second light signal;

[0009] The first imaging unit is configured to generate an eye image based on the fourth light signal;

[0010] The first identification unit is configured to identify the gaze point coordinates based on the eye image.

[0011] In a possible implementation, the gaze point determination unit comprises a second imaging unit, a second identification unit

[0012] The second imaging unit is configured to generate a distorted eye image based on the third light signal.

[0013] The second identification unit is configured to perform gaze point identification on the distorted eye image by using an identification model to determine the gaze point coordinates, wherein the identification model is obtained by training based on distorted eye image samples and real gaze point coordinates of eyes represented by the distorted eye image samples.

[0014] In a possible implementation, the correction unit comprises a model acquisition unit configured to acquire an imaging characteristic model, wherein the imaging characteristic model is used to simulate propagation of the second light signal by the optical waveguide to obtain a fifth light signal; and a target independent variable acquisition unit configured to determine a target independent variable of the imaging characteristic model in a manner that minimizes a difference between the third light signal and the fifth light signal, and use the target independent variable as the fourth light signal.

[0015] In a possible implementation, the system further comprises:

[0016] a degradation model acquisition unit configured to acquire a degradation model;

[0017] a distorted eye image sample acquisition unit configured to input an eye image sample into the degradation model to obtain the distorted eye image sample;

[0018] a real gaze point coordinate determination unit configured to acquire first gaze point coordinates represented by the eye image sample and use the first gaze point coordinates as real gaze point coordinates of an eye represented by the distorted eye image sample.

[0019] In a possible implementation, the gaze point determination unit is in communication connection with the display, and the display is further configured to:

[0020] determine a scene of interest of the eye based on the gaze point coordinates;

[0021] generate a first virtual image matched with the scene, and replace the current virtual image with the first virtual image.

[0022] In a possible implementation, the gaze point determination unit further comprises an infrared light source, and the second light signal is a light signal formed by reflection of a light beam emitted by the infrared light source by the eye.

[0023] In a possible implementation, the display system further comprises a support device configured to fix the optical waveguide in front of the eye.

[0024] The support device comprises an outer frame support configured to support the optical waveguide, and a positioning support connected to the outer frame support, the positioning support being arranged in a direction avoiding the line of sight.

[0025] The display and the gaze point determination unit are arranged on the positioning support.

[0026] In a possible implementation, the display system further comprises a collimating device and a light splitting element arranged between the display and the optical waveguide, and a focusing device arranged between the gaze point determination unit and the light splitting element.

[0027] The collimating device is closer to the display than the light splitting element, and the light splitting element is closer to the optical waveguide than the collimating device.

[0028] The collimating device is configured to propagate the first light signal in a first direction so that the first light signal reaches the light splitting element.

[0029] The light splitting element is configured to propagate the first light signal in a second direction so that the first light signal reaches the optical waveguide.

[0030] The light splitting element is configured to propagate the third light signal in a third direction so that the third light signal reaches the focusing device.

[0031] The focusing device is configured to propagate and converge the third light signal in a fourth direction so that the third light signal reaches the gaze point determination unit.

[0032] According to another aspect of the present disclosure, an electronic device is provided, comprising a processor, a memory for storing processor-executable instructions, wherein the processor is configured to implement the functions of the display system described above when executing the instructions stored in the memory.

[0033] In the embodiments of the present disclosure, the second light signal is transmitted through the optical waveguide to form a distorted light signal, i.e., the third light signal. The gaze point determination unit can determine the gaze point coordinates of the eye based on the third light signal. In this way, even if the obtained light signal cannot accurately reflect the gaze state of the eye, the display system of the present disclosure can still obtain accurate gaze point coordinates, improve the accuracy of the judgment of the eye movement, and improve the matching degree between the generated virtual image and the real scene seen by the eye, thereby improving the user experience. Other features and aspects of the present disclosure will become apparent based on the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present disclosure, and together with the description, serve to explain the principles of the present disclosure.

[0035] Figure 1 A structural schematic diagram of a display system provided by the present disclosure.

[0036] Figure 2 A structural schematic diagram of another display system provided by the present disclosure.

[0037] Figure 3 A structural schematic diagram of an electronic device for display provided by the present disclosure. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings represent the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0039] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0040] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated herein can not be described in detail. It should be appreciated that the present disclosure can be practiced with the exact details as

[0041] In the process of tracking eye movement, an augmented reality application product can not be able to accurately determine the eye movement of a person. There can be several reasons for this, one of which is that the gaze point coordinates that accurately reflect the real state of the person's eye are not obtained. Then, the real world scene seen by the person's eye cannot be accurately determined. That is, there is a difference between the scene actually seen by the person's eye and the scene determined according to the eye movement, so that the generated virtual image is not highly matched or is incorrectly matched with the scene actually seen by the person's eye. In this way, the human-computer interaction is not accurate and smooth, and the user experience is reduced.

[0042] Figure 1 A structural schematic diagram of a display system provided by the present disclosure. As shown in Figure 1 The display system includes a display 101, an optical waveguide 102, and a gaze point determination unit 103.

[0043] As shown in Figure 1The dotted line and the unfilled arrow represent the light signal emitted by the display 101, i.e., the first light signal. The display 101 can display a virtual image. The first light signal can contain image information of the current virtual image. The solid line and the filled arrow represent the light signal reflected by the eye, i.e., the second light signal. The second light signal can be the reflection of the first light signal by the eye, and / or the reflection of ambient light, and / or the reflection of light emitted by other light sources.

[0044] The optical waveguide 102 can transmit the first light signal and transmit the first light signal out. The position at which the first light signal is transmitted out of the optical waveguide is a position at which the eye can receive the transmitted first light signal. In addition, the optical waveguide 102 can also transmit the second light signal and transmit the second light signal out to form a third light signal. The position at which the second light signal is transmitted out of the optical waveguide 102 is a position at which the gaze point determination unit 103 can receive the third light signal. The optical waveguide 102 distorts the second light signal, so that the third light signal is a distorted light signal. The optical waveguide 102 can cause the first light signal and the second light signal to propagate in the optical waveguide 102 without interfering with each other, i.e., to form a common path propagation.

[0045] The gaze point determination unit 103 can determine the gaze point coordinates of the eye directly using the third light signal, or can correct the third light signal and then determine the gaze point coordinates of the eye based on the corrected third light signal. The gaze point coordinates here are coordinates that can accurately reflect the gaze state of the eye.

[0046] In the embodiments of the present disclosure, the second light signal is transmitted through the optical waveguide to form a distorted light signal, i.e., the third light signal. The gaze point determination unit can determine the gaze point coordinates of the eye based on the third light signal. In this way, even if the obtained light signal cannot accurately reflect the gaze state of the eye, the display system using the present disclosure can still obtain accurate gaze point coordinates, improve the accuracy of the judgment of the eye movement, and improve the matching degree between the generated virtual image and the real scene seen by the eye, thereby improving the user experience.

[0047] Still taking Figure 1 For example, in another example, the display system can further include a first coupling device disposed on the optical waveguide 102 and facing the side of the display 101, and a second coupling device disposed on the optical waveguide 102 and facing the side of the eye. The first coupling device can control the first light signal to be incident on the optical waveguide 102 and control the second light signal to be transmitted out of the optical waveguide 102. The second coupling device can control the second light signal to be incident on the optical waveguide 102 and control the first light signal to be transmitted out of the optical waveguide 102.

[0048] Due to the first coupling device and the second coupling device, the display and / or the gaze point determination unit can be arranged in more positions, which improves the design flexibility of the display system of the present disclosure, and also allows the display and the gaze point determination unit to be arranged in a position that does not block the line of sight, thereby improving the user experience.

[0049] In a possible implementation, the gaze point determination unit comprises: a correction unit, a first imaging unit, and a first identification unit. The correction unit is configured to correct the third light signal to obtain a fourth light signal, wherein a difference between the fourth light signal and the second light signal is smaller than a difference between the third light signal and the second light signal. The first imaging unit is configured to generate an eye image based on the fourth light signal. The first identification unit is configured to identify the gaze point coordinates based on the eye image.

[0050] In the embodiments of the present disclosure, the eye gaze state reflected by the fourth light signal is more similar to the real eye gaze state. The eye gaze state reflected by the fourth light signal is closer to the real eye gaze state than the eye gaze state reflected by the third light signal. This is because the difference between the fourth light signal and the second light signal is smaller than the difference between the third light signal and the second light signal. Compared with the third light signal, the fourth light signal is more similar to the second light signal.

[0051] Exemplarily, the propagation parameter can be determined based on the structure of the optical waveguide and through experiments. The correction unit can determine a compensation coefficient based on the propagation parameter, and use the compensation coefficient to compensate the third light signal to obtain the fourth light signal. The fourth light signal can also be obtained by using the method in the present disclosure below.

[0052] The above is only an example, and the embodiments of the present disclosure do not limit the method for determining the fourth light signal.

[0053] The first imaging unit can comprise a photosensitive element, such as an electrically coupled device, a complementary metal oxide semiconductor, etc. The first imaging unit can convert the fourth light signal into a first electrical signal, and use the first electrical signal to generate the eye image.

[0054] The first identification unit can perform gaze point identification on the eye image to obtain the gaze point coordinates. The gaze point identification method can be various, such as determining the gaze point coordinates according to the positions of the eyes in multiple eye images, or using an improved dense trajectory algorithm to determine the gaze point coordinates. The gaze point identification is not the focus of the present disclosure, and thus will not be described in detail.

[0055] In the embodiments of the present disclosure, the correction unit can correct the third optical signal to obtain a fourth optical signal. The accuracy of the optical signal used to generate the eye movement image is improved, and thus the eye image generated by the first imaging unit can accurately reflect the eye fixation state. Furthermore, the accuracy of identifying the fixation point coordinates is improved.

[0056] In a possible implementation, the correction unit comprises: a model obtaining unit, configured to obtain an imaging characteristic model, the imaging characteristic model being used to simulate propagation of the second optical signal by the optical waveguide to obtain a fifth optical signal; and a target independent variable obtaining unit, configured to determine a target independent variable of the imaging characteristic model in a manner that minimizes the difference between the third optical signal and the fifth optical signal, and take the target independent variable as the fourth optical signal.

[0057] In the embodiments of the present disclosure, the optical waveguide can be analyzed to determine the transmission matrix of the optical waveguide. The transmission matrix of the optical waveguide is used to represent the transformation relationship between the tangential components of the electric field intensity and the magnetic field intensity of the optical wave (optical signal) at both ends of the waveguide when the optical wave propagates in the waveguide.

[0058] In one example, the independent variable of the imaging characteristic model can represent the second optical signal, and the dependent variable can represent the fifth optical signal. Furthermore, the transmission matrix is taken as a parameter of the imaging characteristic model.

[0059] The imaging characteristic model can simulate the distorted optical signal obtained after the optical signal is transmitted by the optical waveguide. For example, the second optical signal can be input into the imaging characteristic model to obtain the fifth optical signal. The fifth optical signal is a distorted optical signal simulated by the imaging characteristic model based on the second optical signal. In theory, the fifth optical signal can be the same as the third optical signal. However, in practice, the fifth optical signal can be different from the third optical signal. For ease of description, the difference is named as the first difference. The numerical value of the first difference is negatively correlated with the restoration degree of the imaging characteristic model in restoring the transmission process of the optical waveguide. The independent variable in the case of the minimum first difference can be solved by analytical inversion or an iterative algorithm. For ease of description, the independent variable in the case of the minimum first difference is named as the target independent variable. Furthermore, the target independent variable is taken as the fourth optical signal.

[0060] In this way, the third optical signal can be corrected to obtain the fourth optical signal that is more similar to the second optical signal after correction. The imaging characteristic model can simulate the distorted optical signal (fifth optical signal) after the optical waveguide transmission. The third optical signal can be captured, and thus the target independent variable of the imaging characteristic model can be solved according to the third optical signal and the fifth optical signal. The similarity between the fourth optical signal and the second optical signal is improved. In this way, the eye image obtained by using the fourth optical signal can more accurately reflect the real eye fixation state.

[0061] In a possible implementation, the gaze point determination unit comprises: a second imaging unit, configured to generate a distorted eye image based on the third light signal; and a second recognition unit, configured to determine the gaze point coordinates by performing gaze point recognition on the distorted eye image using a recognition model, wherein the recognition model is obtained by training based on distorted eye image samples and real gaze point coordinates of eyes represented by the distorted eye image samples.

[0062] The second imaging unit can comprise a light sensing element, such as an electric coupling device or a complementary metal oxide semiconductor. The first imaging unit can be used as the second imaging unit, or the second imaging unit and the first imaging unit can each be an imaging unit.

[0063] The second imaging unit can convert the third light signal into a second electrical signal and generate an image based on the second electrical signal. The third light signal is a distorted light signal. Therefore, the image generated by the second imaging unit is a distorted eye image. The distorted eye image cannot accurately reflect the gaze state of the eye. The second imaging unit can include a recognition model or can call a recognition model.

[0064] The recognition model can be a trained neural network, for example, a trained deep neural network. The distorted eye image samples can be used as training samples, and the real gaze point coordinates of the eyes represented by the real eye images can be used as the true values of the training samples. The deep neural network is trained using the distorted eye image samples and their corresponding true values to obtain the recognition model.

[0065] In this way, in use, the distorted eye image can be input into the recognition model to obtain gaze point coordinates that can reflect the real gaze state of the eye.

[0066] In the embodiments of the present disclosure, the third light signal can be directly used without correction, and the gaze point coordinates can be directly recognized on the distorted eye image. On the premise of obtaining accurate gaze point coordinates, the computational burden of the system is reduced, and the accuracy and response speed of determining the gaze state of the eye are improved.

[0067] In a possible implementation, the method further comprises: obtaining a degradation model; inputting an eye image sample into the degradation model to obtain the distorted eye image sample; and obtaining first gaze point coordinates represented by the eye image sample and using the first gaze point coordinates as the real gaze point coordinates of the eye represented by the distorted eye image sample.

[0068] In the embodiments of the present disclosure, the degradation parameter can be obtained by geometric optics ray tracing, or optical diffraction propagation, and the degradation model can be obtained. In addition, the degradation parameter can also be obtained by constructing a point light source, performing optical point spread function measurement, or constructing an orthogonal image basis (for example, Hadamard basis), and the degradation model can be obtained. The degradation model can process the eye pattern sample reflecting the real eye gaze state into a distorted eye pattern sample. In order to facilitate understanding, the degradation model is represented by formula (1).

[0069] y = Qx (1)

[0070] Wherein, x represents the eye pattern sample, y represents the distorted eye pattern sample, and Q represents the degradation parameter, which can be represented in the form of a matrix.

[0071] In one example, a first gaze point can be specified, and the eye pattern sample can be generated according to the first gaze point. At this time, the first gaze point is known, and the eye pattern sample needs to be processed into a distorted eye pattern sample using the degradation model, and the distorted eye pattern sample is taken as a training sample, and the first gaze point is taken as the true value of the training sample.

[0072] In another example, the display system in the present disclosure can be used to obtain the eye pattern sample based on the light signal reflected by the eye. At the same time, the first gaze point corresponding to the eye pattern sample is obtained by using a high-precision eye movement tracking device. The distorted eye pattern sample is taken as a training sample, and the first gaze point is taken as the true value of the training sample.

[0073] In the embodiments of the present disclosure, accurate training samples (distorted eye pattern sample and first gaze point) can be obtained, the feasibility of training the degradation model is improved, and the accuracy of the degradation model is improved.

[0074] In a possible implementation, the gaze point determination unit is in communication connection with the display, and the display is further configured to: determine a scene of interest of the eye based on the gaze point coordinates; generate a first virtual image matched with the scene; and replace the current virtual image with the first virtual image.

[0075] The display can include a processor, or the display can call an external processor. Thus, the display can determine the scene of interest of the eye based on the gaze point coordinates. For example, the scene image of the eye gaze is determined based on the gaze point coordinates by using a YOLO algorithm, and then the scene of interest of the eye is determined from the scene image. The above is only an example, and the embodiments of the present disclosure are not limited in this regard.

[0076] The display can generate a first virtual image matched with the scene of interest of the eye, and the virtual image can include a textual description of the scene, a graphical annotation, etc. And the display of the currently displayed virtual image is stopped, and the first virtual image is displayed.

[0077] In this embodiment, a first virtual image matching the scene being focused on by the eyes can be generated based on the coordinates of the gaze point, improving the matching accuracy between the first virtual image and the scene. Furthermore, displaying the first virtual image as the current virtual image allows for rapid response to eye gaze states, enabling the user to quickly obtain information matching the scene being focused on, thus improving the user experience.

[0078] In one possible implementation, the gaze point determination unit further includes an infrared light source, and the second light signal is a light signal formed by the reflection of a beam of light emitted by the infrared light source by the eye.

[0079] In the embodiments of this disclosure, even in cases of insufficient or complex ambient light, a sufficiently strong second light signal can be generated, thereby improving the environmental versatility of the display system in this disclosure.

[0080] In one example, the first coupling device controls the infrared light signal incident waveguide 102, and the second coupling device controls the infrared light signal output waveguide 102. The infrared light signal is output from a location that the eye can receive.

[0081] In one possible implementation, the display system further includes: a support device for fixing the optical waveguide in front of the eye; the support device includes: an outer frame bracket for supporting the optical waveguide, and a positioning bracket connected to the outer frame bracket, the positioning bracket being positioned in a direction that avoids the line of sight; the display and the gaze point determination unit are disposed on the positioning bracket.

[0082] The optical waveguide is positioned in front of the eye. Its four sides—top, bottom, left, and right—form its edges. An outer frame can cover all or part of these edges. A positioning bracket can connect to the outer frame. The positioning bracket can fix the outer frame and the optical waveguide in a preset position. For example, it can be fixed at a certain distance from a table or the ground, or attached to clothing. When the optical waveguide is used as a lens, the outer frame can be the lens rim, and the positioning bracket can be the temple.

[0083] In this embodiment, the display and gaze point determination unit can be mounted on a positioning bracket to avoid the direction of the line of sight and reduce the impact on the viewing effect. This allows for accurate acquisition of gaze point coordinates and a virtual image that accurately matches the viewed real-world scene, without affecting the eye's ability to view the real-world scene. This improves the user experience.

[0084] In a possible implementation, the display system further includes: a collimating device and a light splitting element arranged between the display and the optical waveguide; and a focusing device arranged between the gaze point determination unit and the light splitting element; wherein the collimating device is closer to the display than the light splitting element, and the light splitting element is closer to the optical waveguide than the collimating device; the collimating device is configured to propagate the first light signal in a first direction, so that the first light signal reaches the light splitting element; the light splitting element is configured to propagate the first light signal in a second direction, so that the first light signal reaches the optical waveguide; the light splitting element is configured to propagate the third light signal in a third direction, so that the third light signal reaches the focusing device; and the focusing device is configured to propagate and converge the third light signal in a fourth direction, so that the third light signal reaches the gaze point determination unit.

[0085] The first light signal emitted by the display passes through the collimating device, then passes through the light splitting element, and then is emitted out of the light splitting element and enters the optical waveguide. The first light signal can be formed by light beams emitted by each pixel of the display. The collimating device can make the light beams emitted by each pixel of the display into parallel light beams. The first direction can be a direction in which the first light signal reaches a region A on the light splitting element.

[0086] The focusing device can be arranged between the gaze point determination unit and the light splitting element. The focusing device can make the third light signal propagate and converge in a fourth direction. The fourth direction can be a direction in which the third light signal reaches a region B on the gaze point determination unit.

[0087] The light splitting device can make the first light signal propagate in a second direction. The second direction can be a direction in which the first light signal reaches a region C on the optical waveguide, for example, the region C can be a first specific region on the first coupling device.

[0088] The light splitting device can also make the third light signal propagate in a third direction. The third direction can be a direction in which the third light signal reaches a region D on the focusing device.

[0089] In a case where the gaze point determination unit includes an infrared light source, the light splitting device can also propagate the infrared light in a fifth direction. The fifth direction can be a direction in which the infrared light reaches a region E on the optical waveguide. For example, the region E can be a second specific region on the first coupling device. Moreover, the light splitting device can also transmit only the infrared light signal reflected by the eye as the third light signal to the focusing device.

[0090] The regions A-E, the first specific region, and the second specific region can be regions that are set in advance according to characteristics of the optical waveguide and other components in the system.

[0091] In the embodiments of the present disclosure, the light splitting element can propagate different light signals in respective preset directions, reducing interference between the light signals. Moreover, only one light splitting element is provided, saving cost and reducing the volume of the entire system. Furthermore, the collimating device, the light splitting element, the focusing device, and the gaze point determination unit can be designed in one module and installed on the positioning support, achieving plug and play, improving the accuracy of the light waveguide augmented reality device that has been put into use, and reducing the cost of modification or redesign of the existing light waveguide augmented reality device.

[0092] Figure 2 Another structural schematic diagram of a display system is provided in the present disclosure. As shown in Figure 2 a possible implementation, the display system includes a display 101, a light waveguide 102, a gaze point determination unit 103, a collimating device 104, a light splitting element 105, and a focusing device 106. The light waveguide 102 includes a first coupling device 1021 and a second coupling device 1022, and the gaze point determination unit 103 includes an infrared light source (not shown in the figure). Figure 2 In the figure, the first light signal is represented by a dashed line and an unfilled arrow; the second light signal is represented by a solid line and a filled arrow; and the infrared signal is represented by a dashed line and a filled arrow.

[0093] The display 101 displays a current virtual image and emits a first light signal. The first light signal contains image information of the current virtual image. The collimating device 104 propagates the first light signal in a first direction and to the light splitting element 105. The light splitting element 105 propagates the first light signal in a second direction, so that the first light signal reaches the first coupling device 1021 and enters the light waveguide 102. The first light signal is transmitted out of the light waveguide 102 from the second coupling device 1022. The transmission position can be such that the eye can receive it.

[0094] The gaze point determination unit 103 emits an infrared light signal, which reaches the light splitting element 105. The light splitting element propagates the infrared light signal to the first coupling device 1021 and into the light waveguide 102. The infrared light signal is transmitted out of the light waveguide 102 from the second coupling device 1022. The transmission position can be such that the eye can receive it.

[0095] The second light signal reflected by the eye can include at least one of the following: ambient light of an environment in which the user is located, a light signal formed by reflecting the first light signal, and a light signal formed by reflecting the infrared light signal.

[0096] The second light signal enters the optical waveguide 102 from the second coupling device. The third light signal is output from the optical waveguide 102 from the first coupling device. The light splitting device 105 causes the third light signal to propagate in the third direction and reach the focusing device 106. In the case that the third light signal contains the infrared light signal of the eye reflection, the light splitting device 105 can cause the third light signal to contain only the infrared light signal and reach the focusing device 106.

[0097] The optical waveguide 102 can cause the first light signal, the second light signal, and the infrared light signal to propagate in the optical waveguide 102 without interfering with each other, i.e., to form common-path propagation.

[0098] The gaze point determination unit 103 can generate a distorted eye image based on the third light signal, and then input the distorted eye image to the recognition model to obtain the gaze point coordinates.

[0099] Alternatively, the gaze point determination unit 103 can correct the third light signal using an imaging characteristic model to obtain a fourth light signal. And, an eye image is generated based on the fourth light signal. The gaze point coordinates are recognized from the eye image.

[0100] The gaze point determination unit 103 is in communication connection with the display 101. The display 101 can determine the scene of interest of the eye based on the gaze point coordinates, and generate a first virtual image matched with the scene, stop displaying the current image, and display the first virtual image as a new current image.

[0101] The embodiments of the present disclosure also provide an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the functions of the above display system when executing the instructions stored in the memory.

[0102] Figure 3 A structural schematic diagram of an electronic device for display provided by the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0103] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM , or the like.

[0104] In an exemplary embodiment, there is also provided a non-transitory computer readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0105] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0106] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, a

[0107] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0108] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0109] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operations to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0110] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage cycles that change state. The instructions can be executed by one or more processors of a computer, to cause a series of operational steps to be performed on the computer to produce a computer-implemented process. The instructions can also cause one or more processors of a computer or other programmable data processing apparatus to

[0111] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0112] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0113] Embodiments of the present disclosure have been described above, and the description is intended to be illustrative of the embodiments and not restrictive of the disclosure. Many modifications and variations of the described embodiments are possible in light of this disclosure. It is intended that the scope of the disclosure be limited not by this detailed description, but rather by the claims appended hereto. The choice of words in the description is intended to be a choice of emphasis on features deemed to be key in expressing the principles of the present disclosure. Other words can be substituted therefor without departing from the essence of the disclosure.

Claims

1. A display system, characterized in that, include: Display, optical waveguide, gaze point determination unit; The display is used to emit a first light signal, the first light signal containing image information of the current virtual image; The optical waveguide is used to transmit the first optical signal and to transmit the first optical signal out of the optical waveguide; The optical waveguide is also used to transmit a second optical signal reflected by the eye, and to transmit the second optical signal out of the optical waveguide to form a third optical signal, which is a distorted optical signal; The gaze point determination unit is located on the side of the optical waveguide from which the second optical signal is transmitted; The gaze point determination unit is used to determine the gaze point coordinates of the eye based on the third light signal; The fixation point determination unit includes a correction unit, the correction unit comprising: The model acquisition unit is used to acquire an imaging characteristic model, which is used to simulate the propagation of the second optical signal in the optical waveguide to obtain the fifth optical signal. The target independent variable acquisition unit is used to determine the target independent variable of the imaging characteristic model with the goal of minimizing the difference between the third optical signal and the fifth optical signal, and to use the target independent variable as the fourth optical signal, wherein the difference between the fourth optical signal and the second optical signal is less than the difference between the third optical signal and the second optical signal.

2. The system according to claim 1, characterized in that, The gaze point determination unit includes: a first imaging unit and a first recognition unit. The first imaging unit is used to generate an eye image based on the fourth optical signal; The first recognition unit is used to identify the coordinates of the gaze point based on the eye image.

3. The system according to claim 1, characterized in that, The gaze point determination unit includes: a second imaging unit and a second recognition unit. The second imaging unit is used to generate a distorted eye image based on the third optical signal; The second recognition unit is used to use a recognition model to perform gaze point recognition on the distorted eye image and determine the gaze point coordinates. The recognition model is obtained by training based on distorted eye map samples and the real gaze point coordinates of the eyes represented by the distorted eye map samples.

4. The system according to claim 3, characterized in that, Also includes: Degradation model acquisition unit, used to acquire degradation models; The distorted eye diagram sample acquisition unit is used to input the eye diagram sample into the degradation model to obtain the distorted eye diagram sample; The true gaze point coordinate determination unit is used to obtain the first gaze point coordinates represented by the eye diagram sample, and use the first gaze point coordinates as the true gaze point coordinates of the eye represented by the distorted eye diagram sample.

5. The system according to claim 1, characterized in that, The gaze point determination unit is communicatively connected to the display, and the display is further configured to: Based on the coordinates of the gaze point, the object of the eye's focus is determined; Generate a first virtual image that matches the scene, and replace the current virtual image with the first virtual image.

6. The system according to claim 1, characterized in that, The gaze point determination unit further includes an infrared light source, and the second light signal is a light signal formed by the reflection of the light beam emitted by the infrared light source by the eye.

7. The system according to any one of claims 1-6, characterized in that, The display system further includes a support device for fixing the optical waveguide to the front of the eye. The support device includes: an outer frame bracket for supporting the optical waveguide, and a positioning bracket connected to the outer frame bracket, wherein the positioning bracket is positioned in a direction that avoids the line of sight; The display and the gaze point determination unit are mounted on the positioning bracket.

8. The system according to any one of claims 1-6, characterized in that, The display system further includes: a collimation device and a beam splitter disposed between the display and the optical waveguide; and a focusing device disposed between the gaze point determination unit and the beam splitter. Wherein, the collimating device is closer to the display than the beam splitter, and the beam splitter is closer to the optical waveguide than the collimating device; The collimation device is used to propagate the first optical signal along a first direction, so that the first optical signal reaches the beam splitter. The beam splitter is used to propagate the first optical signal along the second direction, so that the first optical signal reaches the optical waveguide. The beam splitter propagates the third optical signal along a third direction, so that the third optical signal reaches the focusing device; The focusing device propagates and converges the third light signal along the fourth direction, so that the third light signal reaches the gaze point determination unit.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the functions of the system according to any one of claims 1 to 8 when executing instructions stored in the memory.

Images