Augmented reality system integrating display and eye movement tracking and electronic equipment

Abstract

The invention relates to an augmented reality system integrating display and eye movement tracking and electronic equipment. The display system comprises a display, an optical waveguide and a fixation point determination unit, the display is used for emitting a first optical signal, and the first optical signal comprises image information of a current virtual image; the optical waveguide is used for transmitting the first optical signal and transmitting the first optical signal out of the optical waveguide; the optical waveguide is also used for transmitting a second optical signal reflected by the eye and transmitting the second optical signal out of the optical waveguide to form a third optical signal, and the third optical signal is a distorted optical signal; the fixation point determination unit is arranged on one side, where the second optical signal is transmitted, of the optical waveguide; and the fixation point determination unit is used for determining a fixation point coordinate of the eye based on the third optical signal. Therefore, the display system disclosed by the invention can still obtain accurate coordinates of the fixation point, so that the accuracy of judging the eye movement condition is improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of augmented reality display technology, and in particular to an augmented reality system and electronic device integrating display and eye tracking. Background Art

[0002] Augmented reality technology uses a variety of technical means to simulate computer-generated virtual information such as text, images, three-dimensional models, music, video, etc., and apply it to the real world. By complementing virtual information and information in the real world, it achieves "enhancement" of the real world.

[0003] With the rapid development of augmented reality technology, augmented reality application products have shown wide application potential in many fields such as medicine, industry, education and entertainment. Augmented reality application products can effectively integrate virtual information with the real world, bring immersive experience to users, and improve the efficiency and convenience of information use.

[0004] At present, augmented reality application products have problems such as inaccurate human-computer interaction and unsmooth interaction, which reduces the user experience. Summary of the invention

[0005] In view of this, the present disclosure proposes a display solution.

[0006] According to one aspect of the present disclosure, there is provided a display system, including: a display, an optical waveguide, and a gaze point determination unit; the display is used to emit a first light signal, wherein the first light signal contains image information of a current virtual image; the optical waveguide is used to transmit the first light signal and transmit the first light signal out of the optical waveguide; the optical waveguide is also used to transmit a second light signal reflected by an eye, and transmit the second light signal out of the optical waveguide to form a third light signal, wherein the third light signal is a distorted light signal; the gaze point determination unit is arranged on a side of the optical waveguide where the second light signal is transmitted; the gaze point determination unit is used to determine the gaze point coordinates of the eye based on the third light signal.

[0007] In a possible implementation, the gaze point determination unit includes: a correction unit, a first imaging unit, a first recognition unit,

[0008] The correction unit is used to correct the third optical signal to obtain a fourth optical signal, wherein the difference between the fourth optical signal and the second optical signal is smaller than the difference between the third optical signal and the second optical signal;

[0009] The first imaging unit is configured to generate an eye image based on the fourth light signal;

[0010] The first recognition unit is used to recognize the gaze point coordinates based on the eye image.

[0011] In a possible implementation, the gaze point determination unit includes: a second imaging unit, a second recognition unit

[0012] The second imaging unit is used to generate a distorted eye image based on the third light signal;

[0013] The second recognition unit is used to use a recognition model to perform gaze point recognition on the distorted eye image and determine the gaze point coordinates. The recognition model is trained based on the distorted eye diagram samples and the real gaze point coordinates of the eyes represented by the distorted eye diagram samples.

[0014] In a possible implementation, the correction unit includes: a model acquisition unit, used to acquire an imaging characteristic model, wherein the imaging characteristic model is used to simulate the optical waveguide propagating the second optical signal to obtain a fifth optical signal; and a target independent variable acquisition unit, used to determine a target independent variable of the imaging characteristic model with the goal of minimizing the difference between the third optical signal and the fifth optical signal, and use the target independent variable as the fourth optical signal.

[0015] In a possible implementation, the method further includes:

[0016] A degradation model acquisition unit, used for acquiring a degradation model;

[0017] A distorted eye diagram sample acquisition unit, used for inputting the eye diagram sample into the degradation model to obtain the distorted eye diagram sample;

[0018] The real gaze point coordinate determining unit is used to obtain the first gaze point coordinate represented by the eye diagram sample, and use the first gaze point coordinate as the real gaze point coordinate of the eye represented by the distorted eye diagram sample.

[0019] In a possible implementation manner, the gaze point determination unit is communicatively connected to the display, and the display is further used to:

[0020] Determining the scene that the eye is focusing on based on the gaze point coordinates;

[0021] A first virtual image matching the scene is generated, and the current virtual image is replaced by the first virtual image.

[0022] In a possible implementation, the gaze point determination unit further includes an infrared light source, and the second light signal is a light signal formed by the eye reflecting the light beam emitted by the infrared light source.

[0023] In a possible implementation, the display system further includes: a supporting device, the supporting device being used to fix the optical waveguide in front of the eye;

[0024] The supporting device comprises: an outer frame support for supporting the optical waveguide, and a positioning support connected to the outer frame support, wherein the positioning support is arranged in a direction avoiding the line of sight;

[0025] The display and the gaze point determination unit are arranged on the positioning bracket.

[0026] In a possible implementation, the display system further includes: a collimating device and a light splitting element disposed between the display and the optical waveguide; and a focusing device disposed between the gaze point determination unit and the light splitting element;

[0027] Wherein, the collimating device is closer to the display than the light splitting element, and the light splitting element is closer to the optical waveguide than the collimating device;

[0028] The collimating device is used to propagate the first optical signal along a first direction so that the first optical signal reaches the beam splitting element;

[0029] The light splitting element is used to propagate the first light signal along a second direction so that the first light signal reaches the optical waveguide;

[0030] The beam splitter element propagates the third optical signal along a third direction so that the third optical signal reaches the focusing device;

[0031] The focusing device propagates and converges the third optical signal along a fourth direction, so that the third optical signal reaches the gaze point determination unit.

[0032] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the functions of the above-mentioned display system when executing the instructions stored in the memory.

[0033] In an embodiment of the present disclosure, the second light signal is transmitted through the optical waveguide to form a distorted light signal, namely, a third light signal. The gaze point determination unit can determine the gaze point coordinates of the eye based on the third light signal. In this way, even if the obtained light signal cannot accurately reflect the gaze state of the eye, accurate gaze point coordinates can still be obtained using the display system of the present disclosure, thereby improving the accuracy of eye movement judgment, and also improving the matching degree between the generated virtual image and the real scene seen by the eye, thereby improving the user experience. Other features and aspects of the present disclosure will become clear based on the detailed description of exemplary embodiments with reference to the accompanying drawings below. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0035] Figure 1 A schematic diagram of the structure of a display system provided by the present invention.

[0036] Figure 2 A schematic diagram of the structure of another display system provided by the present invention.

[0037] Figure 3 A schematic diagram of the structure of an electronic device for display provided by the present disclosure. DETAILED DESCRIPTION

[0038] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0039] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0040] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0041] When tracking eye movements, augmented reality application products may not be able to accurately determine a person's eye movements. This may be due to a variety of reasons, one of which is that accurate gaze point coordinates that can reflect the true state of the human eye are not obtained. Then, it is impossible to accurately judge the real-world scenery seen by the human eye. In other words, there is a difference between the scenery actually seen by the human eye and the scenery judged based on eye movements, so that the generated virtual image does not match the scenery actually seen by the human eye, or it matches incorrectly. As a result, human-computer interaction is not accurate and the interaction is not smooth, which reduces the user experience.

[0042] Figure 1 Schematic diagram of a display system provided by the present disclosure. Figure 1 As shown, the display system includes: a display 101, an optical waveguide 102, and a gaze point determination unit 103.

[0043] like Figure 1As described above, dotted lines and unfilled arrows are used to represent light signals emitted by the display 101, i.e., first light signals. The display 101 can display a virtual image. The first light signal can include image information of the current virtual image. Solid lines and filled arrows are used to represent light signals reflected by the eye, i.e., second light signals. The second light signal can be generated by the eye reflecting the first light signal, and / or ambient light, and / or light emitted by other light sources.

[0044] The optical waveguide 102 can transmit a first optical signal and transmit the first optical signal out. The position where the first optical signal is transmitted from the optical waveguide is a position where the eye can receive the transmitted first optical signal. In addition, the optical waveguide 102 can also transmit a second optical signal and transmit the second optical signal out to form a third optical signal. The position where the second optical signal is transmitted from the optical waveguide 102 is a position where the gaze point determination unit 103 can receive the third optical signal. The optical waveguide 102 distorts the second optical signal, so that the third optical signal is a distorted optical signal. The optical waveguide 102 can prevent the first optical signal and the second optical signal from interfering with each other during the propagation process in the optical waveguide 102, that is, forming common path propagation.

[0045] The gaze point determination unit 103 can directly use the third light signal to determine the gaze point coordinates of the eye, or can correct the third light signal and then determine the gaze point coordinates of the eye based on the corrected third light signal. The gaze point coordinates here are coordinates that can accurately reflect the gaze state of the eye.

[0046] In the embodiment of the present disclosure, the second optical signal is transmitted through the optical waveguide to form a distorted optical signal, namely, a third optical signal. The gaze point determination unit can determine the gaze point coordinates of the eye based on the third optical signal. In this way, even if the obtained optical signal cannot accurately reflect the gaze state of the eye, the display system of the present disclosure can still obtain accurate gaze point coordinates, thereby improving the accuracy of eye movement judgment, and also improving the matching degree between the generated virtual image and the real scene seen by the eye, thereby improving the user experience.

[0047] Still Figure 1 For example, in another example, the display system may further include: a first coupling device disposed on the optical waveguide 102 and facing the display 101; and a second coupling device disposed on the optical waveguide 102 and facing the eye. The first coupling device may control the first optical signal to enter the optical waveguide 102, and control the second optical signal to exit the optical waveguide 102. The second coupling device may control the second optical signal to enter the optical waveguide 102, and control the first optical signal to exit the optical waveguide 102.

[0048] Due to the provision of the first coupling device and the second coupling device, the number of selectable setting positions for the display and / or the gaze point determination unit increases, thereby improving the design flexibility of the display system of the present invention, and also allowing the display and the gaze point determination device to be set at positions that do not block the line of sight, thereby improving the user experience.

[0049] In one possible implementation, the gaze point determination unit includes: a correction unit, a first imaging unit, and a first recognition unit, wherein the correction unit is used to correct the third light signal to obtain a fourth light signal, and the difference between the fourth light signal and the second light signal is smaller than the difference between the third light signal and the second light signal; the first imaging unit is used to generate an eye image based on the fourth light signal; and the first recognition unit is used to identify the gaze point coordinates based on the eye image.

[0050] In the disclosed embodiment, the eye gaze state reflected by the fourth optical signal is more similar to the real eye gaze state. The eye gaze state reflected by the fourth optical signal is closer to the real eye gaze state than the eye gaze state reflected by the third optical signal. This is because the difference between the fourth optical signal and the second optical signal is less than the difference between the third optical signal and the second optical signal. Compared with the third optical signal, the fourth optical signal is more similar to the second optical signal.

[0051] Exemplarily, the propagation parameters can be determined based on the structure of the optical waveguide and through experiments. The correction unit can determine the compensation coefficient based on the propagation parameters, and use the compensation coefficient to compensate the third optical signal to obtain the fourth optical signal. The fourth optical signal can also be obtained using the method described below in the present disclosure.

[0052] The above is only an example, and the embodiment of the present disclosure does not limit the method for determining the fourth optical signal.

[0053] The first imaging unit may include a photosensitive element, such as a coupled device, a complementary metal oxide semiconductor, etc. The first imaging unit may convert the fourth light signal into a first electrical signal, and generate an eye image using the first electrical signal.

[0054] The first recognition unit can perform gaze point recognition on the eye image to obtain the gaze point coordinates. There can be multiple methods for gaze point recognition, for example: determining the gaze point coordinates according to the positions of the eyes in multiple eye images. Another example: using an improved dense trajectory algorithm to determine the gaze point coordinates. Gaze point recognition is not the focus of the present disclosure, so it will not be introduced in detail.

[0055] In the embodiment of the present disclosure, the correction unit can correct the third light signal to obtain the fourth light signal. The accuracy of the light signal used to generate the eye movement image is improved, so that the eye image generated by the first imaging unit can accurately reflect the eye gaze state. Furthermore, the accuracy of identifying the gaze point coordinates is improved.

[0056] In a possible implementation, the correction unit includes: a model acquisition unit, used to acquire an imaging characteristic model, wherein the imaging characteristic model is used to simulate the optical waveguide propagating the second optical signal to obtain a fifth optical signal; and a target independent variable acquisition unit, used to determine a target independent variable of the imaging characteristic model with the goal of minimizing the difference between the third optical signal and the fifth optical signal, and use the target independent variable as the fourth optical signal.

[0057] In the disclosed embodiment, the optical waveguide may be subjected to structural analysis to determine the transmission matrix of the optical waveguide, which is used to represent the transformation relationship between the tangential components of the electric field intensity and the magnetic field intensity of the optical signal at both ends of the waveguide when the optical wave (optical signal) propagates in the waveguide.

[0058] In one example, the independent variable of the imaging characteristic model may represent the second optical signal, and the dependent variable may represent the fifth optical signal. In addition, the transmission matrix is ​​used as a parameter of the imaging characteristic model.

[0059] The imaging characteristic model can simulate the distorted optical signal obtained after the optical signal is transmitted in the optical waveguide. For example, the second optical signal can be input into the imaging characteristic model to obtain the fifth optical signal. The fifth optical signal is a distorted optical signal simulated by the imaging characteristic model based on the second optical signal. Theoretically, the fifth optical signal can be the same as the third optical signal. However, in practice, there may be differences between the fifth optical signal and the third optical signal. For the sake of convenience of description, the difference is named the first difference. The numerical value of the first difference is negatively correlated with the degree of restoration of the optical waveguide transmission process by the imaging characteristic model. The independent variable in the case where the first difference is minimized can be solved by analytical inversion or iterative algorithm. For the sake of convenience of description, the independent variable in the case where the first difference is minimized is named the target independent variable. And the target independent variable is used as the fourth optical signal.

[0060] In this way, the third optical signal can be corrected to obtain a corrected fourth optical signal that is more similar to the second optical signal. The imaging characteristic model can simulate the distorted optical signal (fifth optical signal) after transmission through the optical waveguide. The third optical signal can be captured, and thus, the imaging characteristic model can be solved based on the third optical signal and the fifth optical signal to obtain the target independent variable. The similarity between the fourth optical signal and the second optical signal is improved. In this way, the eye image obtained subsequently using the fourth optical signal can more accurately reflect the true eye gaze state.

[0061] In one possible implementation, the gaze point determination unit includes: a second imaging unit and a second recognition unit. The second imaging unit is used to generate a distorted eye image based on the third light signal; the second recognition unit is used to use a recognition model to perform gaze point recognition on the distorted eye image to determine the gaze point coordinates, and the recognition model is trained based on distorted eye diagram samples and real gaze point coordinates of the eye represented by the distorted eye diagram samples.

[0062] The second imaging unit may include a photosensitive element, such as a coupled device, a complementary metal oxide semiconductor, etc. The first imaging unit may be used as the second imaging unit; or the second imaging unit and the first imaging unit are each one imaging unit.

[0063] The second imaging unit may convert the third optical signal into a second electrical signal, and generate an image based on the second electrical signal. The third optical signal is a distorted optical signal. Therefore, the image generated by the second imaging unit is a distorted eye image. The distorted eye image cannot accurately reflect the gaze state of the eye. The second imaging unit may include a recognition model, or may call a recognition model.

[0064] The recognition model may be a trained neural network, for example, the recognition model may be a trained deep neural network. Distorted eye samples may be used as training samples, and the coordinates of the real gaze points of the eyes represented by the real eye diagram may be used as the true values ​​of the training samples. The deep neural network is trained using the distorted eye diagram samples and their corresponding true values ​​to obtain the recognition model.

[0065] In this way, when in use, the distorted eye image can be input into the recognition model to obtain the gaze point coordinates that can reflect the actual eye gaze state.

[0066] In the disclosed embodiment, the third light signal can be used directly without correction, and the gaze point coordinates can be identified directly on the distorted eye image. Under the premise of obtaining accurate gaze point coordinates, the calculation burden of the system is reduced, and the accuracy and response speed of determining the eye gaze state are improved.

[0067] In a possible implementation, it also includes: a degradation model acquisition unit, used to obtain a degradation model; a distorted eye diagram sample acquisition unit, used to input an eye diagram sample into the degradation model to obtain the distorted eye diagram sample; a real gaze point coordinate determination unit, used to obtain a first gaze point coordinate represented by the eye diagram sample, and use the first gaze point coordinate as the real gaze point coordinate of the eye represented by the distorted eye diagram sample.

[0068] In the disclosed embodiment, the degradation parameters can be obtained by geometric optical ray tracing, or optical diffraction propagation, and then the degradation model can be obtained. In addition, the degradation parameters can be obtained by constructing a point light source, measuring the optical point spread function, or constructing an orthogonal image basis (for example, Hadamard basis), and then the degradation model can be obtained. The degradation model can process the eye diagram samples reflecting the real eye gaze state into distorted eye diagram samples. For ease of understanding, the degradation model is represented by formula (1).

[0069] y=Qx (1)

[0070] Wherein, x represents the eye diagram sample, y represents the distorted eye diagram sample, Q represents the degradation parameter, and Q can be expressed in the form of a matrix.

[0071] In one example, a first fixation point may be specified, and an eye diagram sample may be generated according to the first fixation point. Then, the first fixation point is known, and the eye diagram sample needs to be processed into a distorted eye diagram sample using a degradation model, and the distorted eye diagram sample is used as a training sample, and the first fixation point is used as the true value of the training sample.

[0072] In another example, the display system of the present disclosure can be used to obtain an eye diagram sample based on the light signal reflected by the eye. At the same time, a high-precision eye tracking device is used to obtain a first gaze point corresponding to the eye diagram sample. The distorted eye diagram sample is used as a training sample, and the first gaze point is used as the true value of the training sample.

[0073] In the disclosed embodiment, accurate training samples (distorted eye diagram samples and first fixation points) can be obtained, which improves the feasibility of training a degradation model and improves the accuracy of the degradation model.

[0074] In a possible implementation, the gaze point determination unit is communicatively connected to the display, and the display is further used to: determine the scene that the eye is focused on based on the gaze point coordinates; generate a first virtual image matching the scene, and replace the current virtual image with the first virtual image.

[0075] The display may include a processor, or the display may call an external processor. Thus, the display may determine the scene that the eye is focusing on based on the gaze point coordinates. For example, using the YOLO algorithm, based on the gaze point coordinates, the scene image that the eye is focusing on is determined, and then the scene that the eye is focusing on is determined from the scene image. The above is only an example, and the embodiments of the present disclosure are not limited to this.

[0076] The display may generate a first virtual image matching the scene focused on by the eyes, and the virtual image may include text descriptions of the scene, graphic annotations, etc. Furthermore, the display of the currently displayed virtual image is stopped, and the first virtual image is displayed.

[0077] In the disclosed embodiment, a first virtual image matching the scene being focused on by the eye can be generated based on the gaze point coordinates, thereby improving the matching degree between the first virtual image and the scene. Furthermore, the first virtual image is displayed as the current virtual image, and as the eye gaze state responds quickly, the user quickly obtains information matching the scene being focused on, thereby improving the user experience.

[0078] In a possible implementation, the gaze point determination unit further includes an infrared light source, and the second light signal is a light signal formed by the eye reflecting the light beam emitted by the infrared light source.

[0079] In the embodiment of the present disclosure, even when the ambient light is insufficient or the ambient light is complex, a sufficiently strong second light signal can be generated, thereby improving the universality of the display system of the present disclosure to the environment.

[0080] In one example, the first coupling device can control the infrared light signal to enter the optical waveguide 102, and the second coupling device can control the infrared light signal to exit the optical waveguide 102. The position where the infrared light signal is exited is the position where the eye can receive it.

[0081] In a possible implementation, the display system further includes: a supporting device, the supporting device being used to fix the optical waveguide in front of the eye; the supporting device comprising: an outer frame bracket for supporting the optical waveguide, and a positioning bracket connected to the outer frame bracket, the positioning bracket being arranged in a direction avoiding the line of sight; the display and the gaze point determination unit are arranged on the positioning bracket.

[0082] The optical waveguide is arranged in front of the eye. The upper, lower, left and right sides of the optical waveguide form the edge of the optical waveguide. The outer frame bracket can wrap all or part of the edge. The positioning bracket can be connected to the outer frame bracket. The positioning bracket can fix the outer frame bracket and the optical waveguide at a preset position. For example: fixed at a certain distance from the desktop or the ground, or fixed on a person's clothing. In the case where the optical waveguide is used as a spectacle lens, the outer frame bracket can be a lens ring, and the positioning bracket can be a temple.

[0083] In the disclosed embodiment, the display and the gaze point determination unit can be arranged on the positioning bracket to avoid the line of sight and reduce the influence on the viewing effect of the eyes. The gaze point coordinates can be accurately obtained, and the virtual image accurately matches the scene in the real world being viewed can be obtained without affecting the scene in the real world being viewed by the eyes. The user experience is improved.

[0084] In a possible implementation, the display system further includes: a collimating device and a spectroscopic element disposed between the display and the optical waveguide; and a focusing device disposed between the gaze point determination unit and the spectroscopic element; wherein the collimating device is closer to the display than the spectroscopic element, and the spectroscopic element is closer to the optical waveguide than the collimating device; the collimating device is used to propagate the first light signal along a first direction so that the first light signal reaches the spectroscopic element; the spectroscopic element is used to propagate the first light signal along a second direction so that the first light signal reaches the optical waveguide; the spectroscopic element propagates the third light signal along a third direction so that the third light signal reaches the focusing device; the focusing device propagates and converges the third light signal along a fourth direction so that the third light signal reaches the gaze point determination unit.

[0085] The first light signal emitted by the display passes through the collimator, then passes through the beam splitter, and then passes out of the beam splitter to enter the optical waveguide. The first light signal can be formed by the light beams emitted by each pixel of the display. The collimator can make the light beams emitted by each pixel of the display become parallel light beams. The first direction can be a direction that allows the first light signal to reach area A on the beam splitter.

[0086] A focusing device may be provided between the gaze point determination unit and the light splitting element. The focusing device may cause the third light signal to propagate and converge along a fourth direction. The fourth direction may be a direction in which the third light signal reaches area B on the gaze point determination unit.

[0087] The light splitting device can make the first optical signal propagate in a second direction. The second direction can be a direction that makes the first optical signal reach area C on the optical waveguide, for example, area C can be a first specific area on the first coupling device.

[0088] The light splitting device can also make the third light signal propagate along a third direction. The third direction can be a direction that makes the third light signal reach the area D on the focusing device.

[0089] In the case where the gaze point determination unit includes an infrared light source, the spectroscopic device may further propagate the infrared light along a fifth direction. The fifth direction may be a direction in which the infrared light reaches region E on the optical waveguide. For example, region E may be a second specific region on the first coupling device. Moreover, the spectroscopic device may further transmit only the infrared light signal reflected by the eye as the third light signal to the focusing device.

[0090] The above-mentioned area AE, the first specific area, and the second specific area may be areas pre-set according to the characteristics of the optical waveguide and other components in the system.

[0091] In the disclosed embodiment, the spectroscopic element can propagate different optical signals in their respective preset directions, reducing interference between optical signals. Moreover, only one spectroscopic element is required, which saves costs and reduces the volume of the entire system. In addition, the collimating device, the spectroscopic element, the focusing device, and the gaze point determination unit can be designed in one module and installed on a positioning bracket to achieve plug-and-play, thereby improving the accuracy of the optical waveguide augmented reality device that has been put into use and reducing the cost of modifying or redesigning the existing optical waveguide augmented reality device.

[0092] Figure 2 FIG. 1 is a schematic diagram of another display system provided by the present disclosure. Figure 2 As shown, in a possible implementation, the display system includes: a display 101, an optical waveguide 102, a gaze point determination unit 103, a collimating device 104, a light splitting element 105, and a focusing device 106. The optical waveguide 102 includes a first coupling device 1021 and a second coupling device 1022, and the gaze point determination unit 103 includes: an infrared light source (not shown in the figure). Figure 2 In the figure, a first optical signal is represented by a dash-dot line and an unfilled arrow; a second optical signal is represented by a solid line and a filled arrow; and an infrared signal is represented by a dotted line and a filled arrow.

[0093] The display 101 displays the current virtual image and emits a first light signal. The first light signal contains image information of the current virtual image. The collimating device 104 causes the first light signal to propagate along a first direction and reach the beam splitter 105. The beam splitter 105 causes the first light signal to propagate along a second direction, causing the first light signal to reach the first coupling device 1021 and enter the optical waveguide 102. The first light signal is transmitted from the optical waveguide 102 from the second coupling device 1022. And the transmission position can be received by the eye.

[0094] The gaze point determination unit 103 sends out an infrared light signal, which reaches the spectroscopic element 105. The spectroscopic element allows the infrared light signal to reach the first coupling device 1021 and enter the optical waveguide 102. The infrared light signal is transmitted out of the optical waveguide 102 from the second coupling device 1022, and the transmission position can be received by the eyes.

[0095] The second light signal reflected by the eye may include at least one of the following: ambient light of the user's environment, a light signal formed by reflecting the first light signal, and a light signal formed by reflecting an infrared light signal.

[0096] The second optical signal enters the optical waveguide 102 from the second coupling device. The third optical signal is output from the optical waveguide 102 from the first coupling device. The optical splitter 105 allows the third optical signal to propagate along the third direction and reach the focusing device 106. In the case where the third optical signal includes an infrared light signal reflected by the eye, the optical splitter 105 can allow the third optical signal to only include the infrared light signal and reach the focusing device 106.

[0097] The optical waveguide 102 can prevent the first optical signal, the second optical signal, and the infrared optical signal from interfering with each other during the propagation process in the optical waveguide 102 , that is, form common path propagation.

[0098] The gaze point determination unit 103 may generate a distorted eye image based on the third light signal, and then input the distorted eye image into a recognition model to obtain the gaze point coordinates.

[0099] Alternatively, the gaze point determination unit 103 may correct the third light signal using an imaging characteristic model to obtain a fourth light signal, and generate an eye image based on the fourth light signal, and identify the gaze point coordinates from the eye image.

[0100] The gaze point determination unit 103 is in communication with the display 101. The display 101 can determine the scene that the eye focuses on based on the gaze point coordinates, generate a first virtual image matching the scene, stop displaying the current image, display the first virtual image, and use the first virtual image as a new current image.

[0101] The embodiment of the present disclosure also proposes an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the functions of the above-mentioned display system when executing the instructions stored in the memory.

[0102] Figure 3 The electronic device 1900 is a schematic diagram of a structure of an electronic device for display provided by the present disclosure. For example, the electronic device 1900 can be provided as a server or a terminal device. Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0103] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0104] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0105] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0106] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0107] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0108] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0109] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0112] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0113] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A display system, characterized in that: include: Display, optical waveguide, gaze point determination unit; The display is used to emit a first light signal, wherein the first light signal includes image information of a current virtual image; The optical waveguide is used to transmit the first optical signal and transmit the first optical signal out of the optical waveguide; The optical waveguide is further used to transmit a second optical signal reflected by the eye, and transmit the second optical signal out of the optical waveguide to form a third optical signal, wherein the third optical signal is a distorted optical signal; The gaze point determination unit is arranged at a side of the optical waveguide where the second optical signal is transmitted; The gaze point determination unit is used to determine the gaze point coordinates of the eye based on the third light signal.

2. The system according to claim 1, characterized in that The gaze point determination unit includes: a correction unit, a first imaging unit, a first recognition unit, The correction unit is used to correct the third optical signal to obtain a fourth optical signal, wherein the difference between the fourth optical signal and the second optical signal is smaller than the difference between the third optical signal and the second optical signal; The first imaging unit is configured to generate an eye image based on the fourth light signal; The first recognition unit is used to recognize the gaze point coordinates based on the eye image.

3. The system according to claim 1, characterized in that The gaze point determination unit includes: a second imaging unit, a second recognition unit, The second imaging unit is used to generate a distorted eye image based on the third light signal; The second recognition unit is used to use a recognition model to perform gaze point recognition on the distorted eye image and determine the gaze point coordinates. The recognition model is trained based on the distorted eye diagram samples and the real gaze point coordinates of the eyes represented by the distorted eye diagram samples.

4. The system according to claim 2, characterized in that The correction unit comprises: a model acquisition unit, configured to acquire an imaging characteristic model, wherein the imaging characteristic model is used to simulate the optical waveguide propagating the second optical signal to obtain a fifth optical signal; The target independent variable acquisition unit is used to determine the target independent variable of the imaging characteristic model with the goal of minimizing the difference between the third optical signal and the fifth optical signal, and use the target independent variable as the fourth optical signal.

5. The system according to claim 3, characterized in that Also includes: A degradation model acquisition unit, used for acquiring a degradation model; A distorted eye diagram sample acquisition unit, used for inputting the eye diagram sample into the degradation model to obtain the distorted eye diagram sample; The real gaze point coordinate determining unit is used to obtain the first gaze point coordinate represented by the eye diagram sample, and use the first gaze point coordinate as the real gaze point coordinate of the eye represented by the distorted eye diagram sample.

6. The system according to claim 1, characterized in that The gaze point determination unit is communicatively connected to the display, and the display is further used for: Determining the scene that the eye is focusing on based on the gaze point coordinates; A first virtual image matching the scene is generated, and the current virtual image is replaced by the first virtual image.

7. The system according to claim 1, characterized in that The gaze point determination unit also includes an infrared light source, and the second light signal is a light signal formed by the eye reflecting the light beam emitted by the infrared light source.

8. The system according to any one of claims 1 to 7, characterized in that: The display system further comprises: a supporting device, the supporting device being used to fix the optical waveguide in front of the eye; The supporting device comprises: an outer frame support for supporting the optical waveguide, and a positioning support connected to the outer frame support, wherein the positioning support is arranged in a direction avoiding the line of sight; The display and the gaze point determination unit are arranged on the positioning bracket.

9. The system according to any one of claims 1 to 7, characterized in that: The display system further comprises: a collimating device and a light splitting element arranged between the display and the optical waveguide; and a focusing device arranged between the gaze point determination unit and the light splitting element; Wherein, the collimating device is closer to the display than the light splitting element, and the light splitting element is closer to the optical waveguide than the collimating device; The collimating device is used to propagate the first optical signal along a first direction so that the first optical signal reaches the beam splitting element; The light splitting element is used to propagate the first light signal along a second direction so that the first light signal reaches the optical waveguide; The beam splitter element propagates the third optical signal along a third direction so that the third optical signal reaches the focusing device; The focusing device propagates and converges the third optical signal along a fourth direction, so that the third optical signal reaches the gaze point determination unit.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the functions of the system described in any one of claims 1 to 9 when executing the instructions stored in the memory.

Images