Display method and near-to-eye display equipment

Abstract

The invention provides a display method and near-to-eye display equipment. The display method comprises the following steps: displaying reference objects respectively positioned on each edge of a screen on the screen of the near-to-eye display equipment; obtaining a human eye image when the user looks towards each reference object; marking two opposite reference objects as a reference object group, and marking two human eye images when a user looks towards the two reference objects as an image group; identifying pupils and irises in the human eye image as feature objects; judging whether the feature object in any one human eye image in each image group is complete or not; if not, judging whether the number of the human eye images with incomplete feature objects in each image group is one or not; and if yes, adjusting the horizontal position and / or the pitching angle of the camera based on the human eye images until the feature objects in the two human eye images in each image group are complete. According to the method, the application range of the eye movement tracking function and the use effect of the eye movement tracking function can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of extended reality, and in particular to a display method and a display device. Background Art

[0002] Extended Reality (XR) technology is a technology designed to provide an immersive experience, which includes virtual reality (VR), augmented reality (AR), ， XR technology usually requires the use of near-eye display devices.

[0003] Eye tracking technology is an important function in near-eye display devices. When users use near-eye display devices, eye tracking technology can track the user's gaze point, thereby providing users with a more natural and intuitive extended reality interactive experience.

[0004] In order to realize the eye tracking function, near-eye display devices usually need to use a camera installed on the near-eye display device to capture the human eye to identify the point of the human eye's gaze. However, the near-eye display devices in the prior art are difficult to widely adapt to the individual physiological differences of different users, which affects the use effect of the eye tracking function. Summary of the invention

[0005] The present application provides a display method and a near-eye display device, aiming to improve the adaptability of the near-eye display device to the physiological differences of individual users, thereby improving the scope of application of the eye tracking function and improving the use effect of the eye tracking function.

[0006] In a first aspect, the present application provides a display method, comprising the following steps:

[0007] Displaying reference objects respectively located at respective edges of the screen on a screen of a near-eye display device;

[0008] Acquire a human eye image when the user looks at each of the reference objects;

[0009] marking two reference objects located at two opposite edges of the screen as a reference object group, and marking two human eye images when a user looks at the two reference objects in the same reference object group as an image group;

[0010] Recognize the pupil and iris in the human eye image as feature objects;

[0011] Determining whether a feature object in any one of the human eye images in each of the image groups is complete;

[0012] If not, determining whether the number of the human eye images with incomplete feature objects in each image group is one;

[0013] If so, adjust the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each group of the image groups are complete.

[0014] In the above technical solution, a series of reference objects can be displayed on the screen and the camera can be adjusted based on the eye images of the human eye looking at these reference objects. The camera can be adjusted so that the near-eye display device can more widely adapt to the physiological differences in the faces and eyes of different display individuals, thereby improving the scope of application of the eye tracking function and improving the use effect of the eye tracking function.

[0015] In a possible implementation, if yes, adjusting the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each group of the image groups are complete, specifically includes:

[0016] Marking the human eye image with incomplete feature objects as an unqualified image;

[0017] The camera is translated in a horizontal direction toward the position where the reference object corresponding to the unqualified image is located, and / or the camera is deflected in a pitch direction toward the position where the reference object corresponding to the unqualified image is located.

[0018] In a possible implementation, displaying reference objects respectively located at respective edges of the screen on the screen of the near-eye display device specifically includes:

[0019] The reference objects are located at the centers of the upper, lower, left and right edges of the screen respectively.

[0020] In a possible implementation, the method further includes: determining whether the number of the human eye images with incomplete feature objects in each image group is one;

[0021] If not, for the image group in which the characteristic objects in the two human eye images are incomplete, adjusting the camera to a first camera position to capture a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete;

[0022] The camera is adjusted to a second camera position to capture a human eye image of a human eye looking at another reference object of the two reference objects corresponding to the image group, and the feature object in the human eye image is made complete.

[0023] In a possible implementation manner, if the step of adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete;

[0024] When the user uses the near-eye display device, the display method further includes:

[0025] Collect user's eye images in real time;

[0026] identifying a pupil and an iris in the eye image as feature objects;

[0027] Determining whether the feature object is complete;

[0028] If not, based on the incomplete eye image of the feature object, adjust the horizontal position and / or horizontal angle of the camera so that the camera is translated and / or deflected in the direction in which the feature object deviates from the center of the eye image.

[0029] In a possible implementation manner, if the step of adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete;

[0030] When the user uses the near-eye display device, the display method further includes:

[0031] Collect user's eye images in real time;

[0032] identifying a pupil and an iris in the eye image as feature objects;

[0033] Determining whether the distance between the feature object and the edge of the eye image is less than a preset threshold;

[0034] If yes, predict the user's sight point within the next preset time period based on the current display screen of the near-eye display device;

[0035] Determine whether the user's feature object will continue to shift toward the edge of the eye image within the next preset time period;

[0036] If so, adjust the horizontal position and / or pitch angle of the camera so that the camera is translated and / or deflected toward the direction in which the feature object will continue to deviate.

[0037] In a second aspect, the present application provides a near-eye display device, including a screen, a camera, a driving device, and a processor;

[0038] Wherein, the screen is used to display the reference object as described in the above display method;

[0039] The camera is used to collect the human eye image as described in the above display method;

[0040] The driving device is used to drive the camera to adjust the horizontal position and the pitch angle respectively;

[0041] The processor is used to execute the steps of marking two reference objects located at two opposite edges of the screen as a reference object group, and marking two human eye images when a user looks at two of the reference objects in the same reference object group as an image group; identifying pupils and irises in the human eye images as feature objects; determining whether the feature objects in any one of the human eye images in each of the image groups are complete; if not, determining whether the number of the human eye images in each image group with incomplete feature objects is one; and if so, instructing the driving device to adjust the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each of the image groups are complete.

[0042] In the above technical solution, a series of reference objects can be displayed on the screen and the camera can be adjusted based on the eye images of the human eye looking at these reference objects. The camera can be adjusted so that the near-eye display device can more widely adapt to the physiological differences in the faces and eyes of different display individuals, thereby improving the scope of application of the eye tracking function and improving the use effect of the eye tracking function.

[0043] In a possible implementation, the driving device includes a horizontal adjustment device and a pitch adjustment mechanism;

[0044] The camera is simultaneously connected to the horizontal adjustment device and the pitch adjustment mechanism;

[0045] The horizontal adjustment device is used to drive the camera to translate in the horizontal direction, and the pitch adjustment mechanism is used to drive the camera to rotate.

[0046] In a possible implementation, the level adjustment device includes a first driving member, a traction member, and a guide member; wherein,

[0047] The traction member is slidably connected to the guide member, and the traction member can slide in a horizontal direction relative to the guide member; the first driving member is used to drive the traction member to slide in a horizontal direction; the traction member is connected to the camera and can pull the camera to slide in a horizontal direction.

[0048] In a possible implementation manner, the guide member is a sliding rod, and the sliding rod is slidably connected to the traction member;

[0049] The horizontal adjustment device further includes a screw rod, and the first driving member drives the traction member to slide in a horizontal direction through the screw rod.

[0050] In a possible implementation, the pitch adjustment mechanism includes a second driving member, a first gear and a second gear; wherein:

[0051] The second driving member is used to drive the first gear to rotate; the first gear is meshed with the second gear; the second gear is fixed to the camera and is located at an end of the camera away from the lens;

[0052] The first gear drives the camera to pitch in the vertical direction by driving the second gear to rotate.

[0053] In a possible implementation, the near-eye display device further includes an infrared fill light module, which is disposed inside a lens barrel of the near-eye display device and is used to fill light for the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is one of the flow charts showing the method in the embodiments of the present application;

[0055] Figure 2 This is an example of a complete feature object in the embodiment of the present application;

[0056] Figure 3 It is an example of an incomplete feature object in the embodiment of the present application;

[0057] Figure 4 This is the second schematic diagram of the method flow chart in the embodiment of the present application;

[0058] Figure 5 This is the third schematic diagram of the method flow chart in the embodiment of the present application;

[0059] Figure 6 This is an overall schematic diagram of a near-eye display device in an embodiment of the present application;

[0060] Figure 7 This is a schematic diagram of a horizontal adjustment device in an embodiment of the present application;

[0061] Figure 8 Schematic diagram of the pitch adjustment mechanism in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings.

[0063] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of this specification should be understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in one or more embodiments of this specification do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0064] To facilitate understanding of the display method provided in the embodiment of the present application, its application scenario is first described. The display method provided in the embodiment of the present application can be applied to a near-eye display device.

[0065] Extended reality (XR) technology is a technology designed to provide an immersive experience, which includes virtual reality (VR), augmented reality (AR) and mixed reality (MR). XR technology usually requires the use of near-eye display devices.

[0066] Eye tracking technology is an important function in near-eye display devices. When users use near-eye display devices, eye tracking technology can track the user's gaze point, thereby providing users with a more natural and intuitive extended reality interactive experience.

[0067] In order to realize the eye tracking function, near-eye display devices usually need to use a camera installed on the near-eye display device to capture the human eye to identify the point of the human eye's gaze. However, the near-eye display devices in the prior art are difficult to widely adapt to the individual physiological differences of different users, which affects the use effect of the eye tracking function.

[0068] To facilitate understanding of the display method in the embodiment of the present application, the corresponding hardware structure of the eye tracking technology is first introduced. To implement the eye tracking technology, a camera for capturing human eye images is provided in the near-eye display device. The camera is used to capture human eye images when the user wears the near-eye display device for the first time and during the use of the near-eye display device, and calculate the landing point of the human eye's line of sight on the screen based on the position and shape of the iris and pupil in the human eye image.

[0069] Different users have different physiological structures of faces and eyes. This means that when some users use eye tracking technology and look at certain parts of the screen, the camera cannot capture complete pupil and iris images, resulting in an inability to accurately determine the landing point, affecting the effectiveness of the eye tracking technology.

[0070] Based on this, the present application provides a display method, which aims to improve the adaptability of near-eye display devices to the physiological differences of individual users, thereby improving the scope of application of eye tracking functions and improving the use effect of eye tracking functions. The display method provided by the present application is described in detail below in conjunction with the accompanying drawings.

[0071] refer to Figure 1 , Figure 1 The overall flow chart of the display method provided by this application is as follows:

[0072] S101: Displaying reference objects located at respective edges of the screen on a screen of a near-eye display device.

[0073] Here, the near-eye display device may be glasses, helmets, etc. with near-eye display functions, such as VR glasses. These near-eye display devices all have screens, and their screens may be circular screens, rectangular screens, polygonal screens, etc. It should be understood that for a circular screen, its edge is a continuous closed curve as a whole. At this time, the continuous closed curve can be divided by artificial division and the multiple curves divided into segments are identified as the edges of the circular screen.

[0074] The reference object described here plays a guiding role, mainly used to guide the user to look at the edge of the screen, because the reference object here can be in various forms. In some possible implementations, the reference object can be displayed as a circular color block, a rectangular color block, a triangular color block, etc. located at the edge of the screen. Of course, in some other possible implementations, the reference object can also be an image that is deviated from other display parts of the screen. The entire screen can be displayed as a piece of grass, and the reference object is displayed as a flower on the grass and located at the edge of the screen.

[0075] As an optional implementation, when the reference objects located at the edges of the screen are displayed on the screen of the near-eye display device, the reference objects may be located at the centers of the top, bottom, left and right edges of the screen respectively.

[0076] It should be understood that the reference object can be displayed at the center of the four edges of top, bottom, left and right at the same time, or it can be displayed in the center of the four edges of top, bottom, left and right in sequence. By displaying the reference object at the center of four opposite edges, when judging whether the characteristic objects of the human eye image corresponding to the reference objects at the left and right edges are complete, the risk of the human eye's angular deflection in the vertical direction affecting the judgment result can be reduced; similarly, when judging whether the characteristic objects of the human eye image corresponding to the reference objects at the upper and lower edges are complete, the risk of the human eye's angular deflection in the horizontal direction affecting the judgment result can also be reduced; thereby, the accuracy of the result of judging whether the characteristic object is complete can be improved. And the risk of reducing the overall efficiency of the camera adjustment process due to the deflection of the line of sight in other directions affecting the judgment result can be reduced.

[0077] S102: Acquire a human eye image when the user looks at each reference object.

[0078] In this step, the human eye image when the user looks at each reference object is obtained. Specifically, when the user looks at a certain reference object, the camera takes a picture of the human eye to obtain the human eye image. When the user looks at each reference object, a picture is taken once to complete the collection of all eye images.

[0079] Since the reference objects are distributed at various edges of the screen, when the reference objects are specifically displayed, the corresponding prompts can be displayed on the screen to instruct the user to look at one of all the reference objects. For example, all the reference objects are displayed on the screen at one time, and then an indicator arrow is displayed on the screen to instruct the user to look at one of the many reference objects, or the user can be instructed to look at one of the many reference objects by highlighting a certain reference object. Of course, in some possible implementations, all the reference objects can also be set to be displayed in sequence to instruct the user to look at different reference objects. For example, a reference object can be displayed first, and a human eye image can be taken when the user looks at the reference object, and then the position of the reference object can be moved to other edge parts of the screen. When the user looks at the reference object again, another human eye image can be taken. In this way, the human eye image when the human eye looks at all the reference objects can be collected.

[0080] S103: Mark two reference objects located at two opposite edges of the screen as a reference object group, and mark two human eye images when the user looks at the two reference objects in the same reference object group as an image group. As an example, when a reference object is displayed at each of the upper, lower, left, and right edges of the screen, the two reference objects located at the upper and lower edges of the screen may be marked as a reference object group, and the two reference objects located at the left and right edges of the screen may be marked as a reference object group.

[0081] In this step, the reference objects are divided into reference object groups and the human eye images are divided into image groups, which can facilitate the adjustment of the camera based on the human eye images in each image group in the subsequent steps. Here, the reference objects located at two opposite edges are divided into one reference object group and the human eye images obtained based on the two references are divided into one image group, in order to improve the accuracy of the camera adjustment by allowing the captured human eye images to correspond to the extreme position of the human eye viewing the screen in the arrangement direction of the two reference objects in the reference object group.

[0082] S104: Identify the pupil and iris in the human eye image as feature objects.

[0083] In this step, the human eye image can be processed by a processor to obtain a feature object, which can be specifically realized by machine recognition. The specific recognition process can be based on the color distinction between the pupil and iris and other parts of the eyeball.

[0084] S105: Determine whether the feature object in any one of the human eye images in each of the image groups is complete.

[0085] In this step, the specific judgment process can be implemented according to the shape of the feature object. In a specific example, since the pupil and iris of a person are mostly circular or elliptical, the complete feature object in the human eye image is also a complete circle or ellipse. For details, please refer to Figure 2 , Figure 2 This is an example of a complete feature object in an embodiment of the present application.

[0086] There may be two situations of incomplete feature objects. One situation is that there is no feature object at all in the human eye image, that is, when the user looks at the reference object corresponding to the human eye image, the camera is completely unable to capture the pupil and iris of the human eye. This means that under the current position and orientation of the camera, when the user looks at the reference object, it is completely unable to determine the point of sight based on the position and shape of the pupil and iris.

[0087] The second situation is that the human eye image has characteristic objects, but the characteristic objects are incomplete, which can be manifested as the pupil or iris appearing as an incomplete circle or ellipse. Figure 3 , Figure 3 This is an example of an incomplete feature object in the embodiment of the present application. When there is only part of the feature object in the human eye image, when the user looks at the reference object, the camera can only capture part of the pupil and iris, which will increase the error in judging the sight point or even make it impossible to judge the sight point.

[0088] Optionally, when specifically judging whether the feature object is complete, it can also be achieved by comparing it with a reference image. As an example, before the display method step is implemented, a reference object can be displayed in the center of the screen, and when the human eye looks at the reference object, the camera takes an image of the human eye as a reference image. The pupil and iris are also identified as feature objects in the reference image, and at this time, the feature object in the reference image can be regarded as complete. By comparing the feature object in the human eye image in the image group with the feature object in the reference image, it can also be judged whether the feature object is complete.

[0089] It should be understood that if the judgment in this step is yes, that is, the feature objects in all human eye images in all image groups are complete, the step can be terminated. Since these human eye images are distributed at each edge of the screen, it means that when the human eye looks at each edge of the screen, the camera can capture the complete pupil and iris, and it can be inferred that when the user looks at all positions in the screen, the complete pupil and iris can be captured, that is, when the user looks at all positions on the screen, the line of sight can be accurately judged.

[0090] S106: If not, determine whether the number of the human eye images with incomplete feature objects in each image group is one.

[0091] Here, "if not" means that not all feature objects in each human eye image in each human eye image group are complete, which means that if the position and direction of the camera are not adjusted, it is impossible to accurately track the line of sight when the human eye looks at all positions on the screen.

[0092] There may be two results for judging whether the number of human eye images with incomplete characteristic objects in each image group is one, one result is that the number of human eye images with incomplete characteristic objects in one image group is one, and the other result is that the number of human eye images with incomplete characteristic objects in one image group is two. Different subsequent steps will be performed for the two different judgment results.

[0093] S107: If yes, adjust the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each group of the image groups are complete.

[0094] In this step, the camera is adjusted based on the human eye image. Specifically, by judging from the captured image that the camera cannot fully capture the pupil and iris when the human eye looks at a certain reference object, the horizontal position and / or pitch angle of the camera can be adjusted so that the camera can change its position and transfer angle, so that the pupil and iris can be completely located in the camera's image field when the human eye looks at the edge of the screen.

[0095] During the specific adjustment, the adjustment may be made according to a set adjustment step length, or an adjustment reference value may be given by estimating the size of the missing reference object, and then the horizontal position and / or pitch angle may be adjusted according to the reference value.

[0096] As an example, if in the image group corresponding to the reference objects located at the upper and lower edges of the screen, the feature object in the human eye image corresponding to the reference object located at the upper edge of the screen is incomplete, and the feature object in the human eye image corresponding to the reference object located at the lower edge of the screen is complete, then the processor can issue an instruction to adjust the pitch angle of the camera.

[0097] After adjusting the horizontal position and / or pitch angle of the camera, the process may return to step S102 and repeat the above-mentioned determination and adjustment process until the characteristic objects in the human eye images in each image group are complete, and then the step is terminated.

[0098] In the above technical solution, a series of reference objects can be displayed on the screen and the camera can be adjusted based on the eye images of the human eye looking at these reference objects. The camera can be adjusted so that the near-eye display device can more widely adapt to the physiological differences in the faces and eyes of different display individuals, thereby improving the scope of application of the eye tracking function and improving the use effect of the eye tracking function.

[0099] As an optional implementation, the horizontal position and / or pitch angle of the camera is adjusted based on the human eye image until the feature objects in the two human eye images in each group of the image groups are complete, specifically including: marking the human eye image with incomplete feature objects as unqualified images; translating the camera in the horizontal direction toward the position of the reference object corresponding to the unqualified image, and / or deflecting the camera in the pitch direction toward the position of the reference object corresponding to the unqualified image.

[0100] Specifically, if only one human eye image in the image group corresponding to the reference objects at the upper and lower edges of the screen is an unqualified image, the camera can be deflected only in the pitch direction toward the position of the reference object corresponding to the unqualified image, so that the camera image field is offset in the direction of the reference object corresponding to the unqualified image, thereby achieving the purpose of capturing the complete feature object.

[0101] If only one human eye image in the image group corresponding to the reference objects at the left and right edges of the screen is an unqualified image, the camera can be translated only in the horizontal direction toward the position of the reference object corresponding to the unqualified image, so that the camera image field is offset in the direction of the reference object corresponding to the unqualified image, thereby achieving the purpose of capturing the complete feature object.

[0102] As an example, still taking the case where, in the image group corresponding to the reference objects located at the upper and lower edges of the screen, only the feature object in the human eye image corresponding to the reference object located at the upper edge of the screen is incomplete, the processor can instruct the camera to deflect upward by a certain distance along the pitch adjustment route, so that the camera's image field moves upward, so as to ensure that the camera can capture the complete feature object when the human eye looks at the reference object at the upper edge of the screen.

[0103] Of course, if there is an unqualified image in both of the above two groups of images, or there is an unqualified image in the image group corresponding to a group of reference objects at the bevel edge, the horizontal position and the pitch angle can be adjusted at the same time.

[0104] After adjusting the camera so that the characteristic objects in all human eye images in the image group are complete, these eye images and another human eye image of the human eye looking at the center of the screen and the eye image captured by the camera in real time when the user uses the near-eye display device can be used for eye tracking. Specifically, the above-mentioned human eye image of the user looking at the characteristic object on the screen and the human eye image looking at the center of the screen can be used as a reference, and the eye image captured in real time using the mute display device can be compared with the human eye image used as a reference to detect the user's gaze point.

[0105] As an example, when the reference images include five human eye images, namely, images of human eyes when looking at the reference objects at the four edges of the screen, the left and the right, and images of human eyes when looking at the center of the screen, the key coordinates of the characteristic objects in the five human eye images can be first identified and extracted. For example, the characteristic coordinates of the characteristic objects can be extracted, such as the center coordinates of the pupil (x p ,y p ), the center coordinate of the iris is (x i ,y i ), the edge point coordinates of the iris Then, the feature coordinates of the feature objects extracted above are combined into a feature vector f = [x p ,y p , x i ,y i , x 1 , x 1 ,y 1 , ..., x n ,y n ], and this feature vector can be used as a dimensionality reduction representation of the human eye image. The five human eye images each have their own feature vectors. The feature vectors of these five images can be used as reference data. The feature vector of the human eye image looking at the center of the screen is recorded as f c , and the other four eigenvectors are denoted as f u 、f d 、fl 、f r , corresponding to the human eye images at the upper, lower, left and right edges of the screen respectively.

[0106] When using the near-eye display device, the user's eye image is collected in real time, and the characteristic objects in the eye image are identified, and the same image processing as in the above reference data is performed to obtain the characteristic vector of the current eye image, which is recorded as f t .

[0107] Calculate the cosine similarity between the feature vector of the current eye image and the five feature vectors in the reference data:

[0108]

[0109] Use the softmax function (normalized exponential function) to convert the similarity into a normalized similarity percentage:

[0110]

[0111] Then according to the coordinates (x c ,y c ), (x u ,y u ), (x d ,y d ), (x l ,y l ), (x r ,y r ), calculate the weighted average coordinates:

[0112]

[0113] The obtained coordinates (x t ,y t ), which is the final calculated coordinate of the sight point.

[0114] As an optional implementation, the display method provided by the present application further includes:

[0115] S106: Determine whether the number of the human eye images with incomplete feature objects in each image group is one; the explanation of this step can be referred to the above description, which will not be repeated here.

[0116] If the above judgment is no, it means that at least one image group has two human eye images with incomplete feature objects, that is, at least one image group includes two unqualified images. When it is judged that there is an unqualified image in an image group, the horizontal position or pitch angle of the camera can be adjusted so that the feature objects in the two human eye images in the image group remain intact when the human eye images are acquired again after the camera is adjusted. However, when both human eye images in an image group are unqualified images, it can be foreseen that no matter which direction the camera is adjusted in which the reference object corresponding to the unqualified image is located, two human eye images with complete feature objects cannot be obtained after the camera is adjusted.

[0117] Near-eye display devices are mostly mass-produced products, and their structure and shape design can certainly adapt to the majority of user groups. However, for a small number of users, due to the special physiological structure of their eyes and faces, when the reference object is displayed at the edge of the screen, no matter how the camera is adjusted, the feature objects in the two human eye images in one or several image groups cannot be complete.

[0118] At this time, in order to allow these special users to also gain the technical experience of eye tracking, the position and / or angle of the camera can be adjusted so that the camera can capture images of the human eye when the human eye looks at different reference objects from different camera positions, and the reference objects in these images of the human eye are all complete.

[0119] Please continue to perform the following steps:

[0120] S108: If not, for the image group in which the feature objects in the two human eye images are incomplete, adjust the camera to a first camera position to capture a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and make the feature object in the human eye image complete.

[0121] In this step, adjusting the camera to the first position may include adjusting the horizontal position and / or pitch angle of the camera, and the adjusted position and angle should make the characteristic objects in the human eye image captured by the camera complete when the human eye looks at one of the two reference objects corresponding to the image group. As an example, if it is determined that the characteristic objects in the two human eye images when the human eye looks at the two reference objects at the upper and lower edges of the screen are incomplete, the camera can be adjusted downward to the first position first, and the camera at the first position can capture the human eye image when the human eye looks at the reference object at the lower edge of the screen, and the characteristic objects in the human eye image are complete. It should be understood that the adjustment range of adjusting the camera downward to the first position here can be reversed according to the estimated missing range of the characteristic objects in the human eye image with incomplete characteristic objects, or the human eye image can be captured by gradually adjusting the angle and judging whether the characteristic objects are complete.

[0122] S109: Adjust the camera to a second camera position to capture a human eye image of a human eye looking at another reference object of the two reference objects corresponding to the image group, and make the feature object in the human eye image complete.

[0123] In this step, adjusting the camera to the second position may also include adjusting the horizontal position and / or pitch angle of the camera, and the adjusted position and angle should make the characteristic objects in the human eye image captured by the camera complete when the human eye looks at the other reference object of the two reference images corresponding to the image group. Continuing with the above example, if it is determined that the characteristic objects in the two human eye images when the human eye looks at the two reference objects at the upper and lower edges of the screen are incomplete, the camera can be adjusted downward to the first position first, and the camera at the first position can capture the human eye image when the human eye looks at the reference object at the lower edge of the screen, and the characteristic objects in the human eye image are complete. It should be understood that the adjustment range of adjusting the camera downward to the first position here can be reversed according to the estimated missing range of the characteristic objects in the human eye image with incomplete characteristic objects, or the human eye image can be captured by gradually adjusting the angle and judging whether the characteristic objects are complete.

[0124] After the shooting of the first camera and the second camera, the complete human eye images of the characteristic objects of the two reference objects at the upper edge and the lower edge of the screen can be obtained respectively. And the two images can also be used for subsequent tracking. For the two human eye images of the first camera and the second camera, the characteristic coordinates of the characteristic objects in the two human eye images can be extracted respectively, and the human eye image taken by one of the cameras is used as the initial image, and the characteristic coordinates of the characteristic objects in the human eye image taken by the other camera are converted to the coordinate values ​​in the initial image coordinate system, and the coordinate values ​​in the converted initial image coordinate system are used to form a characteristic vector for eye tracking.

[0125] Specifically, for the display method in the embodiment of the present application, the transformation between the two positions of the camera may be a translation in the horizontal direction or a deflection angle in the pitch direction. Here, the horizontal direction is set as the X direction, that is, a translation in the X direction and a rotation around the X axis. The characteristic coordinate transformation in these two directions can be referred to as follows.

[0126] If the human eye image taken by the first camera is the initial image, the second camera rotates around the X axis by an angle of θ compared to the first camera. x , the feature coordinates of a feature point in the human eye image taken by the second camera are (u r ,v r ), the camera’s intrinsic parameter matrix is

[0127]

[0128] where fx and f y is the focal length in pixels, c x and c y The coordinates of the principal point in pixels.

[0129] The camera rotates around the X axis with an angle of θ x , then the rotation matrix of the camera is:

[0130]

[0131] The feature vector in the human eye image of the second camera is mapped to the coordinate system of the human eye image of the first camera, which can be calculated by the unit matrix. The unit matrix H = K·R·K -1 , where H is a 3×3 matrix. For the convenience of calculation, the feature coordinates to be converted in the second camera are marked as x r =[u r ,v r ,1] T ; Then the feature point is mapped to the coordinate x′ in the human eye image corresponding to the first camera position through the unit matrix i =H·x r Here x′ i is a homogeneous coordinate and needs to be normalized:

[0132]

[0133] Get(u r ,v r ) is mapped to the pixel coordinates of the human eye image of the first camera in the coordinate system as follows:

[0134]

[0135] The coordinates obtained by the mapping transformation can be used to form a feature vector group and perform subsequent eye tracking calculations.

[0136] It should be understood that in the embodiment of the present application, if the adjacent first camera position and the second camera position are translated in the horizontal direction, corresponding to the above-mentioned conversion process, it is only necessary to convert the rotation matrix of the camera into the translation matrix, and the specific conversion process will not be repeated here.

[0137] It should also be pointed out that the above-mentioned setting of the first camera as the reference camera and mapping the coordinates in the human eye image taken by the second camera to the human eye image taken by the first camera is only an implementable exemplary scheme. In other possible implementations, setting the second camera as the reference camera and mapping the coordinates in the human eye image taken by the first camera to the human eye image taken by the second camera is also a feasible scheme.

[0138] refer to Figure 4 , Figure 4 This is the second schematic diagram of the method flow chart in the embodiment of the present application.

[0139] As an optional implementation, if the step of adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; during the user using the near-eye display device, the display method further includes:

[0140] S201, collecting eye images of the user in real time;

[0141] Among them, the step of collecting the user's eye image in real time can still be completed by the camera. For details, please refer to the relevant description of collecting eye images above, which will not be repeated here.

[0142] S202, identifying the pupil and iris in the eye image as feature objects;

[0143] Among them, specific possible implementation methods of identifying feature objects in eye images can refer to the above description of identifying feature objects in human eye images, which will not be repeated here.

[0144] S203, determining whether the distance between the feature object and the edge of the eye image is less than a preset threshold;

[0145] Here, the distance between the feature object and the eye image can be obtained by the processor performing image processing on the eye image, and the preset threshold can be determined according to the actual eye tracking technology requirements. If the distance between the feature object and the edge of the eye image is less than the preset threshold, it means that the feature object is already relatively close to the edge of the eye image. If the feature object continues to move toward the edge of the image in the next period, the feature object may be incomplete in the eye image taken at a certain time point in the next period.

[0146] S204: If yes, predict the user's sight point within the next preset time period based on the current display screen of the near-eye display device.

[0147] Predicting the user's sight point in the next period can be used as a basis for judging whether the feature object in the next period will continue to approach the edge of the image. As an example, based on the display content of the current image and the current sight point, it can be judged which item the user is looking at in the display screen, so that the location where the item appears in the next period may be the sight point in the next period. Alternatively, if there are items in the display screen in the next period that are strongly correlated with the items at the current sight point, or if an event that occurs in the display screen in the next period has a strong correlation with the item, it can also be judged that the location of the above-mentioned strongly correlated items or the location where the event with a strong correlation occurs may be the user's sight point in the next period.

[0148] S205 , determining whether the user's feature object will continue to shift toward the edge of the eye image within the next preset time period.

[0149] If this object will move to the edge of the screen at the next moment, or an event related to the object may occur at a position closer to the edge of the screen than the object at the next moment, it means that the user's line of sight in the next period of time may fall on a position closer to the edge of the screen, and it can be determined that the feature object will continue to move to the edge of the eye image.

[0150] S206: If yes, adjust the horizontal position and / or pitch angle of the camera so that the camera is translated and / or deflected toward the direction in which the feature object will continue to deviate.

[0151] If it is determined that the feature object in the eye image will move further to the edge of the screen, the camera can be adjusted in advance or synchronously to shift the camera to the position where the line of sight may fall in the next period of time. In this way, when the user looks at a part closer to the edge of the screen, the risk of inaccurate or even impossible line of sight judgment due to incomplete feature objects in the eye image can be reduced.

[0152] This step is applicable to the case where complete human eye images are captured by the first camera and the second camera respectively because the feature objects in the two human eye images in the same image group are incomplete. Because the execution of capturing human eye images by the first camera and the second camera is based on the situation that no matter how the horizontal position and / or pitch angle of the camera is adjusted, the camera's image field cannot completely cover the feature objects when the human eye looks at the reference objects at the two opposite edges of the screen. Therefore, when using a near-eye display device, if the human eye's line of sight falls on one side edge of the screen or falls near the edge, the feature objects in the eye image captured by the camera may be incomplete, which will affect the accuracy of judging the landing point of the line of sight and may even make it impossible to judge the landing point of the line of sight.

[0153] refer to Figure 5 , Figure 5This is the third schematic diagram of the method flow chart in the embodiment of the present application.

[0154] As an optional implementation, if the step of adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; during the user using the near-eye display device, the display method further includes:

[0155] S301, collecting user's eye images in real time.

[0156] Among them, the step of collecting the user's eye image in real time can still be completed by the camera. For details, please refer to the relevant description of collecting eye images above, which will not be repeated here.

[0157] S302: Identify the pupil and iris in the eye image as feature objects.

[0158] Among them, specific possible implementation methods of identifying feature objects in eye images can refer to the above description of identifying feature objects in human eye images, which will not be repeated here.

[0159] S303, determining whether the feature object is complete;

[0160] The specific implementation of determining whether the feature object in the eye image is complete can refer to the above description of determining whether the feature object in the human eye image is complete, which will not be repeated here.

[0161] S304: If not, based on the incomplete eye image of the feature object, adjust the horizontal position and / or horizontal angle of the camera so that the camera translates and / or deflects in the direction in which the feature object deviates from the center of the eye image. Then, return to real-time acquisition of the user's eye image until the feature object in the eye image is complete.

[0162] From the above description, it can be seen that if the feature objects of the two human eye images in the same human eye image group are incomplete, it is possible that the feature objects in the eye image will be incomplete when the user's line of sight falls on certain parts of the screen close to the edge. This step can mainly be used to deal with the situation where the feature objects in the eye image are incomplete due to the accidental movement of the human eye's line of sight during the operation of the near-eye display device.

[0163] In this step, the eye image collected in real time is first judged whether the feature object is complete. The method for judging whether the feature object is complete can refer to the method for judging whether the feature object in the human eye image is complete. If the feature object is judged to be incomplete, the camera can be quickly translated and / or deflected in the direction where the feature object deviates from the center of the eye image, so as to quickly collect the eye image with the complete feature object, thereby reducing the risk of inaccurate judgment of the sight point or even failure to judge the sight point due to incomplete feature objects in the eye image.

[0164] It is worth noting that, in the case where the camera angle needs to be adjusted during use, the feature vector f of the feature object in the eye image taken by the camera is t , it should also be mapped to the human eye image taken by the reference camera selected in the calibration process, and then perform the corresponding cosine similarity judgment and other steps. The specific conversion process can be referred to above, and will not be repeated here.

[0165] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0166] Based on the same inventive concept, a near-eye display device corresponding to the display method is also provided in the embodiment of the present application. Since the principle of solving the problem by the near-eye display device in the embodiment of the present application is similar to the above-mentioned display method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0167] refer to Figure 6 , Figure 6 1 is a schematic diagram of the near-eye display device in the embodiment of the present application. The near-eye display device provided in the embodiment of the present application includes a screen, a camera 1, a drive device 2 and a processor. The appearance of the screen and the processor of the near-eye display device is the same as that of the near-eye display device in the prior art, so there is no need to Figure 6 Indicated in the middle.

[0168] The screen is used to display the reference objects as described above. These reference objects include reference objects displayed at the edges of the screen, and may also include reference objects displayed inside the edges of the screen. Of course, the reference object located at the center of the screen may also be displayed to obtain a human eye image of the human eye looking toward the center of the screen.

[0169] Both the camera 1 and the driving device 2 can be arranged in the lens barrel of the near-eye display device, and can be specifically arranged between the screen and the optical lens module of the near-eye display device. Among them, the camera 1 can be used to collect human eye images and eye images required for any embodiment of the above-mentioned display method. The driving device 2 is used to drive the camera 1 to adjust the horizontal position and the pitch position respectively. Specifically, the driving device 2 may include a fixed part and a movable part, wherein the fixed part is used to be fixed inside the lens barrel, the movable part is used to fix the camera 1, and the fixed part and the movable part are movably connected together. In this way, when the movable part moves relative to the fixed part, the camera 1 can be driven to adjust the horizontal position and the pitch position. It should be understood that the camera 1 can be driven to adjust the horizontal position and the pitch position respectively here, which means that the driving device 2 can drive the camera 1 to adjust the horizontal position alone, can also drive the camera 1 to adjust the pitch angle alone, and can also drive the camera 1 to adjust the horizontal position and the pitch angle at the same time.

[0170] The processor is used to execute steps such as marking and judging in the above-mentioned display method and send instructions to the screen and the driving device 2.

[0171] Specifically, the processor executes marking two reference objects located at two opposite edges of the screen as a reference object group, and marking two human eye images when the user looks at the two reference objects in the same reference object group as an image group; identifying the pupil and iris in the human eye image as feature objects; judging whether the feature objects in any one of the human eye images in each of the image groups are complete; if not, judging whether the number of the human eye images with incomplete feature objects in each image group is one; if so, instructing the driving device 2 to adjust the horizontal position and / or pitch angle of the camera 1 based on the human eye image until the feature objects in the two human eye images in each of the image groups are complete.

[0172] And, in some optional embodiments, the processor is used to execute: marking the human eye image with incomplete feature object as an unqualified image; issuing an instruction to the driving device 2 to "translate the camera 1 in the horizontal direction toward the position of the reference object corresponding to the unqualified image, and / or deflect the camera 1 in the pitch direction toward the position of the reference object corresponding to the unqualified image."

[0173] In some optional embodiments, it is determined whether the number of human eye images in which the characteristic objects are incomplete in each image group is one. If not, the processor can also be used to execute: issuing instructions to the driving device: "For image groups in which the characteristic objects in both human eye images are incomplete, adjusting the camera to the first camera position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to the second camera position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete", and for executing subsequent coordinate conversion steps.

[0174] In some optional embodiments, if "for an image group in which the feature objects in both of the human eye images are incomplete, adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the feature object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the feature object in the human eye image complete", during the user's use of the near-eye display device, the processor may also be used to execute: real-time acquisition of the user's eye image; identifying the pupil and iris in the eye image as feature objects; determining whether the feature object is complete; if not, based on the incomplete eye image of the feature object, issuing an instruction to the drive device 2 to adjust the horizontal position and / or horizontal angle of the camera 1 so that the camera 1 translates and / or deflects in the direction in which the feature object deviates from the center of the eye image.

[0175] In some optional embodiments, if "for an image group in which the feature objects in both of the human eye images are incomplete, adjusting the camera to a first position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the feature object in the human eye image complete; adjusting the camera to a second position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the feature object in the human eye image complete", during the user's use of the near-eye display device, the processor may also be used to execute: real-time acquisition of the user's eye image; identifying the pupil and iris in the eye image as feature objects; determining whether the distance between the feature object and the edge of the eye image is less than a preset threshold; if so, predicting the user's line of sight within the next preset time period based on the current display screen of the near-eye display device; determining whether the user's feature object will continue to shift toward the edge of the eye image within the next preset time period; if so, issuing an instruction to the drive device 2 to "adjust the horizontal position and / or pitch angle of the camera 1 so that the camera 1 translates and / or deflects in the direction in which the feature object will continue to shift".

[0176] It is worth noting that the processor here may refer to a central processing unit, which can execute all the control steps in any of the above-mentioned display methods; of course, the processor may also be a distributed processor, and be located in different positions according to the module division of the actual steps. For example, the processor may include a screen processor, a camera 1 processor, a drive device 2 processor, a line of sight point calculation processor, etc.

[0177] For reference Figure 7 and Figure 8 , Figure 7 This is a schematic diagram of a horizontal adjustment device in an embodiment of the present application. Figure 8 Schematic diagram of the pitch adjustment mechanism in the embodiment of the present application.

[0178] As an optional implementation, the driving device 2 provided in the embodiment of the present application includes a horizontal adjustment device 21 and a pitch adjustment mechanism, and the camera 1 is connected to the horizontal adjustment device 21 and the pitch adjustment mechanism at the same time. Among them, the horizontal adjustment device 21 is used to drive the camera 1 to translate in the horizontal direction to adjust the position of the camera 1 in the horizontal direction; the pitch adjustment mechanism is used to drive the camera 1 to rotate to adjust the pitch angle of the camera 1 in the vertical direction. It should be understood that the horizontal direction and the vertical direction mentioned in the embodiment of the present application are a pair of mutually perpendicular directions. Here, only a pair of mutually perpendicular directions are used to represent the relative relationship of the adjustment of the camera 1 in the two directions. In fact, according to the specific space design inside the camera 1 and the different placement angles of the camera 1, this pair of mutually perpendicular directions can be any two mutually perpendicular directions in the space, and does not necessarily always refer to the absolute horizontal direction and the absolute vertical direction.

[0179] It should be understood that when the camera 1 is connected to the horizontal adjustment device 21 and the pitch adjustment device 22 at the same time, the horizontal adjustment device 21 and the pitch adjustment device 22 can be connected to the camera 1 separately and independently, or the horizontal adjustment device 21 and the pitch adjustment device 22 can be superimposed and then connected to the camera 1. Moreover, when the horizontal adjustment device 21 and the pitch adjustment device 22 are superimposed, the camera 1 can be connected to the horizontal adjustment device 21 and the horizontal adjustment device 21 can be connected to the pitch adjustment device 22, or the camera 1 can be connected to the pitch adjustment device 22 and the pitch adjustment device 22 can be connected to the horizontal adjustment device 21.

[0180] When the horizontal adjustment device 21 and the pitch adjustment device 22 are implemented, the adjustment methods may include but are not limited to a screw-nut mechanism, a linear motor mechanism, a gear rack transmission mechanism, etc., and may be provided in conjunction with a corresponding drive device 2.

[0181] As an optional embodiment, the horizontal adjustment device 21 includes a first driving member 211, a traction member 212 and a guide member 213. The traction member 212 is slidably connected to the guide member 213, and the traction member 212 can slide in the horizontal direction relative to the guide member 213. The first driving member 211 is used to drive the traction member 212 to slide in the horizontal direction, and the traction member 212 is connected to the camera 1 and can pull the camera 1 to slide in the horizontal direction.

[0182] The first driving member 211 may be a stepping motor, a servo motor, etc. The first driving member 211 and the traction member 212 may be connected by a screw nut, a gear rack, etc., so that the rotational motion output by the first driving member 211 can be converted into the horizontal translation of the traction member 212. The traction member 212 is slidably connected to the guide member 213, and the guide member 213 can be used to guide the moving direction of the traction member 212. Specifically, the connection method between the guide member 213 and the traction member 212 may include but is not limited to linear bearings, guide rail sliders, etc. Such a horizontal adjustment device 21 has the characteristics of simple structure, small space occupation, and fast adjustment response.

[0183] Optionally, when the horizontal adjustment device 21 is specifically set, the guide member 213 can be a sliding rod, which is slidably connected to the traction member 212, so that the sliding rod can be used to guide the sliding direction of the traction member 212 and improve the movement stability of the traction member 212.

[0184] The horizontal adjustment device also includes a screw rod 214, and the first driving member 211 drives the traction member 212 to slide in the horizontal direction through the screw rod 214. Specifically, the screw rod 214 is connected to the output shaft of the first driving member 211, so that the screw rod 214 can rotate under the drive of the first driving member 211. The screw rod 214 and the guide member 213 are connected by threads, which can be achieved by opening a threaded hole on the guide member 213 or fixing a nut on the guide member 213, so that the screw rod 214 can drive the guide member 213 to slide in the horizontal direction.

[0185] As an optional implementation, when the pitch adjustment mechanism is implemented, the pitch adjustment mechanism includes a second driving member 221, a first gear 222, and a second gear 223. The second driving member 221 is used to drive the first gear 222 to rotate, the first gear 222 is meshed with the second gear 223, and the second gear 223 is fixed to the camera 1 and is located at the end of the camera 1 away from the lens, so that the first gear 222 drives the second gear 223 to rotate and drives the camera 1 to pitch in the vertical direction.

[0186] Specifically, the second driving member 221 may also be a servo motor, a stepping motor, etc., and the output shaft of the second driving member 221 may be coaxially connected to the first gear 222, so that the second driving member 221 can drive the first gear 222 to rotate. The second gear 223 is fixed to the camera 1, where the second gear 223 may be integrally formed with the rear of the camera 1, or the second gear 223 may be produced separately and then fixed to the camera 1 by welding, bonding, etc.

[0187] When the camera 1 is specifically connected to the traction piece, the camera 1 and the traction piece may be hingedly connected, so that the hinged connection between the camera 1 and the traction piece can be used as the rotation center for adjusting the pitch angle of the camera 1. That is, the camera head of the camera 1 and the second gear 223 are respectively located on opposite sides of the hinged connection between the camera 1 and the traction piece. In this way, the first gear 222 drives the second gear 223 to rotate in a certain direction, which can drive the camera head to swing in the opposite direction.

[0188] By setting the pitch adjustment mechanism to a gear transmission mode, it also has the characteristics of simple structure, small space occupation and fast adjustment response.

[0189] As an optional implementation, the near-eye display device provided in the embodiment of the present application further includes an infrared fill light module. The infrared fill light module is disposed inside the lens barrel of the near-eye display device and is used to fill light for the camera 1. Specifically, the infrared fill light module can emit infrared light toward the human eye, and then partially reach the camera 1 after being reflected by the human eye, thereby improving the image quality of the human eye image and the eye image taken by the camera 1, and helping to improve the accuracy of the judgment of the line of sight. At the same time, since the infrared light is invisible to the human eye, it will not affect the display effect of the near-eye display device.

[0190] One or more embodiments of this specification are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification should be included in the scope of protection of this disclosure.

[0191] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A display method, applied to a near-eye display device, comprising the following steps: Displaying reference objects respectively located at respective edges of the screen on a screen of a near-eye display device; Acquire a human eye image when the user looks at each of the reference objects; marking two reference objects located at two opposite edges of the screen as a reference object group, and marking two human eye images when a user looks at the two reference objects in the same reference object group as an image group; Recognize the pupil and iris in the human eye image as feature objects; Determining whether a feature object in any one of the human eye images in each of the image groups is complete; If not, determining whether the number of the human eye images with incomplete feature objects in each image group is one; If so, adjust the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each group of the image groups are complete.

2. The display method according to claim 1, characterized in that: If so, adjusting the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each group of the image groups are complete, specifically includes: Marking the human eye image with incomplete feature objects as an unqualified image; The camera is translated in a horizontal direction toward the position where the reference object corresponding to the unqualified image is located, and / or the camera is deflected in a pitch direction toward the position where the reference object corresponding to the unqualified image is located.

3. The display method according to claim 1, characterized in that: Displaying reference objects respectively located at respective edges of the screen on the screen of the near-eye display device specifically includes: The reference objects are located at the centers of the upper, lower, left and right edges of the screen respectively.

4. The display method according to any one of claims 1 to 3, characterized in that: The method further comprises: determining whether the number of the human eye images in which the characteristic objects are incomplete in each image group is one; If not, for the image group in which the characteristic objects in the two human eye images are incomplete, adjusting the camera to a first camera position to capture a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; The camera is adjusted to a second camera position to capture a human eye image of a human eye looking at another reference object of the two reference objects corresponding to the image group, and the feature object in the human eye image is made complete.

5. The display method according to claim 4, characterized in that: If the step of adjusting the camera to the first camera position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to the second camera position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; When the user uses the near-eye display device, the display method further includes: Collect user's eye images in real time; identifying a pupil and an iris in the eye image as feature objects; Determining whether the feature object is complete; If not, based on the incomplete eye image of the feature object, adjust the horizontal position and / or horizontal angle of the camera so that the camera is translated and / or deflected in the direction in which the feature object deviates from the center of the eye image.

6. The display method according to claim 4, characterized in that: If the step of adjusting the camera to the first camera position to shoot a human eye image of a human eye looking at one of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; adjusting the camera to the second camera position to shoot a human eye image of a human eye looking at the other of the two reference objects corresponding to the image group, and making the characteristic object in the human eye image complete; When the user uses the near-eye display device, the display method further includes: Collect user's eye images in real time; identifying a pupil and an iris in the eye image as feature objects; Determining whether the distance between the feature object and the edge of the eye image is less than a preset threshold; If yes, predict the user's sight point within the next preset time period based on the current display screen of the near-eye display device; Determine whether the user's feature object will continue to shift toward the edge of the eye image within the next preset time period; If so, adjust the horizontal position and / or pitch angle of the camera so that the camera is translated and / or deflected toward the direction in which the feature object will continue to deviate.

7. A near-eye display device, characterized in that: Including screen, camera, drive device and processor; wherein the screen is used to display the reference object as claimed in claim 1; The camera is used to collect the human eye image as claimed in claim 1; The driving device is used to drive the camera to adjust the horizontal position and the pitch angle respectively; The processor is used to execute the steps of marking two reference objects located at two opposite edges of the screen as a reference object group, and marking two human eye images when a user looks at two of the reference objects in the same reference object group as an image group; identifying pupils and irises in the human eye images as feature objects; determining whether the feature objects in any one of the human eye images in each of the image groups are complete; if not, determining whether the number of the human eye images in each image group with incomplete feature objects is one; and if so, instructing the driving device to adjust the horizontal position and / or pitch angle of the camera based on the human eye images until the feature objects in the two human eye images in each of the image groups are complete.

8. The near-eye display device according to claim 7, characterized in that: The driving device includes a horizontal adjustment device and a pitch adjustment mechanism; The camera is simultaneously connected to the horizontal adjustment device and the pitch adjustment mechanism; The horizontal adjustment device is used to drive the camera to translate in the horizontal direction, and the pitch adjustment mechanism is used to drive the camera to rotate.

9. The near-eye display device according to claim 8, characterized in that: The horizontal adjustment device includes a first driving member, a traction member and a guide member; wherein, The traction member is slidably connected to the guide member, and the traction member can slide in a horizontal direction relative to the guide member; the first driving member is used to drive the traction member to slide in a horizontal direction; the traction member is connected to the camera and can pull the camera to slide in a horizontal direction.

10. The near-eye display device according to claim 9, characterized in that: The guide member is a sliding rod, and the sliding rod is slidably connected to the traction member; The horizontal adjustment device further includes a screw rod, and the first driving member drives the traction member to slide in a horizontal direction through the screw rod.

11. The near-eye display device according to claim 9, characterized in that: The pitch adjustment mechanism includes a second driving member, a first gear and a second gear; wherein, The second driving member is used to drive the first gear to rotate; the first gear is meshed with the second gear; the second gear is fixed to the camera and is located at an end of the camera away from the lens; The first gear drives the camera to pitch in the vertical direction by driving the second gear to rotate.

12. The near-eye display device according to any one of claims 7 to 11, characterized in that: The near-eye display device further includes an infrared fill light module, which is disposed inside the lens barrel of the near-eye display device and is used to fill light for the camera.

Images