Display object interaction method, display device and electronic device

Abstract

The invention discloses a display object interaction method, display equipment and electronic equipment, and belongs to the technical field of mixed reality. The method comprises the following steps: acquiring a first left eye image and a first right eye image of a user under the condition that at least two display objects which are displayed in an overlapping manner are displayed; determining a first interpupillary distance value of the user according to the first left eye image and the first right eye image; determining the sight line direction of the user according to the first left eye image and the first right eye image; and determining a target display object watched by the user from the at least two display objects displayed in an overlapping manner according to the sight direction and the first interpupillary distance value.

Description

Technical Field

[0001] This application belongs to the field of mixed reality technology, specifically relating to a display object interaction method, display device, and electronic device. Background Technology

[0002] Mixed Reality (MR) devices are wearable computing devices that deeply integrate virtual content with the real environment and support real-time user interaction.

[0003] MR devices can simultaneously present multiple interactive display objects at different spatial depths within the user's field of vision using left and right optical engines. These objects may visually overlap or superimpose. When the user gazes in a certain direction, because multiple display objects are located at different depths in that line of sight, the system has difficulty accurately determining which display object the user actually wants to interact with, resulting in an inability to accurately identify the display object the user wants to interact with. Summary of the Invention

[0004] The purpose of this application is to provide a display object interaction method, display device, and electronic device that can solve the technical problem that existing MR devices cannot accurately identify the display object that the user wants to interact with.

[0005] In a first aspect, embodiments of this application provide a method for interacting with a display object, the method comprising:

[0006] In the case of displaying at least two overlapping display objects, obtain the user's first left-eye image and first right-eye image;

[0007] The user's first interpupillary distance value is determined based on the first left eye image and the first right eye image;

[0008] Determine the user's gaze direction based on the first left-eye image and the first right-eye image;

[0009] The target display object that the user is looking at is determined from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

[0010] Secondly, embodiments of this application provide a display device, the display device comprising:

[0011] An eye-tracking detection module is used to acquire a first left-eye image and a first right-eye image of a user when at least two overlaid display objects are displayed on the display device, and to determine the user's gaze direction based on the first left-eye image and the first right-eye image.

[0012] The processing module is used to determine the user's first interpupillary distance value based on the first left eye image and the first right eye image;

[0013] The processing module is further configured to determine the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

[0014] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method provided in the first aspect.

[0015] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0016] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the method provided in the first aspect.

[0017] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as provided in the first aspect.

[0018] In the display object interaction method, display device, and electronic device of this application, when displaying at least two overlapping display objects, the user's left-eye and right-eye images can be acquired, and the user's current first interpupillary distance (IPD) and gaze direction can be determined. Then, combining the user's gaze direction and IPD, the target display object currently being gazed at by the user is selected from the at least two overlapping display objects as the user's current interaction target. Through the above scheme, the user's gaze can be combined with the real-time measured IPD to accurately identify the target display object that the user truly wants to interact with. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a display object interaction method provided in one embodiment of this application;

[0020] Figure 2 This is a schematic diagram of the interaction between a user and a display device provided in one embodiment of this application;

[0021] Figure 3 This is a schematic diagram of the display interface of a display device provided in one embodiment of this application;

[0022] Figure 4 This is a flowchart illustrating a display object interaction method provided in one embodiment of this application;

[0023] Figure 5This is a schematic diagram of the structure of a display device provided in another embodiment of this application;

[0024] Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0025] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0027] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0028] To address the aforementioned technical problems, this application provides a method for interacting with displayed objects. The following detailed description, in conjunction with the accompanying drawings, through specific embodiments and application scenarios, illustrates the method for interacting with displayed objects provided in this application.

[0029] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a display object interaction method according to an embodiment of this application. This application provides a display object interaction method applied to a display device with an eye-tracking detection module. The method may include:

[0030] S101, when displaying at least two overlapping display objects, acquire the user's first left-eye image and first right-eye image;

[0031] In this embodiment, the eye-tracking detection module refers to a hardware module that integrates components such as an eye-tracking camera and an image processing chip, used to collect and analyze the user's eye features in real time. The display device with the eye-tracking detection module can perceive the user's eye movement behavior in real time while displaying content.

[0032] Taking a Mixed Reality (MR) device with an eye-tracking detection module as an example, an MR device is a wearable computing device that deeply integrates virtual content with the real environment and supports real-time user interaction. When a user wears the MR device, the MR device can present virtual images on the display screen in front of the user through the built-in optical module and analyze the user's eye features in real time through the built-in eye-tracking detection module.

[0033] For example, an MR device can contain multiple display planes, which are positioned at different display depths to support various virtual display objects. Since each display object is rendered onto a specific display plane, the display depth corresponding to that object is also the display depth of the display plane containing that object. Figure 2 As shown, the MR device can simultaneously display a first plane 201, a second plane 202, and a third plane 203, wherein the first plane, the second plane, and the third plane are from the user's eyes in order of distance from the user's eyes, and there is at least partial overlap in display.

[0034] Therefore, MR devices can display multiple overlapping objects on the screen. These overlapping objects are located at different display depths but overlap in the same viewing direction. The objects are interactive interfaces displayed on the screen, such as virtual menus, control panels, or application views.

[0035] When a user wears an MR device and the device's display shows at least two overlapping objects, in order to determine which object the user is looking at, the device's built-in eye-tracking camera can capture images of the user's left eye and right eye respectively. Based on these images, pupil information can be extracted to further calculate the user's gaze direction and current interpupillary distance, thus determining which object the user is actually looking at.

[0036] S102, determine the user's first interpupillary distance value based on the first left eye image and the first right eye image;

[0037] In this embodiment, interpupillary distance (IPD) refers to the spatial distance between the center of the user's left pupil and the center of the user's right pupil. After capturing the user's first left-eye image and first right-eye image, the coordinates of the pupil centers of both eyes can be extracted using image processing algorithms based on these images. Then, combined with the physical distance between the left and right optical cameras, the three-dimensional distance between the centers of the left and right pupils can be calculated to obtain the current IPD value.

[0038] Specifically, in some embodiments, determining the user's first interpupillary distance value based on the first left-eye image and the first right-eye image includes:

[0039] Determine the center point of the first pupil of the user's left eye in the first left eye image, and the center point of the second pupil of the user's right eye in the first right eye image;

[0040] The first pupil center point is converted into a first coordinate point in a first optomechanical coordinate system; and the second pupil center point is converted into a second coordinate point in a second optomechanical coordinate system, wherein the first optomechanical coordinate system is a spatial coordinate system established with the location of the left eye optomechanical system as the origin, and the second optomechanical coordinate system is a spatial coordinate system established with the location of the right eye optomechanical system as the origin;

[0041] The first interpupillary distance value is determined based on the first coordinate point, the second coordinate point, and the distance between the left and right optical transducers.

[0042] In this embodiment, the first left-eye image and the first right-eye image are first processed by an image processing algorithm to extract the edge contours of the pupils of each eye in the first left-eye image and the first right-eye image. Then, based on the edge contours of the pupils, the two-dimensional coordinates of the pupil center point in the image coordinate system are determined, namely the first pupil center point in the first left-eye image and the second pupil center point in the first right-eye image.

[0043] After determining the first and second pupil center points in the image coordinate system, these two-dimensional points can be converted into three-dimensional coordinates. Specifically, the first pupil center point can be converted into a first coordinate point in a first optomechanical coordinate system, and the second pupil center point can be converted into a second coordinate point in a second optomechanical coordinate system. The left-eye and right-eye optomechanical systems are independent optical modules in the MR device used to provide image display for the left and right eyes, respectively. Based on the optical parameters of the cameras that capture the left-eye and right-eye images in the optomechanical systems, the first pupil center point can be projected into the first optomechanical coordinate system to obtain the first coordinate point, and the second pupil center point can be projected into the second optomechanical coordinate system to obtain the second coordinate point.

[0044] Finally, based on the first coordinate point in the first optomechanical coordinate system, the second coordinate point in the second optomechanical coordinate system, and the actual physical distance between the left and right eye optical machines, the system unifies the first coordinate point and the second coordinate point into the same reference coordinate system and calculates the three-dimensional spatial distance between the first coordinate point and the second coordinate point. This distance is the user's first interpupillary distance value at the current moment.

[0045] For example, the output value of the Hall sensor in the MR device at the current moment can be read, since this output value reflects the relative positional state between the left and right optical transducers. Therefore, this output value can be converted into the distance between the left and right optical transducers.

[0046] In the above scheme, the pupil center points in the left and right eye images can be transformed into three-dimensional coordinate points in the optomechanical coordinate system, and the user's current pupillary distance value can be calculated based on the three-dimensional coordinate points and the distance between the left and right eye optomechanical systems, thus achieving high-precision pupillary distance measurement.

[0047] In some embodiments, determining the center point of the first pupil of the user's left eye in the first left-eye image includes:

[0048] Based on the brightness difference between the pupil and iris of the user's left eye in the first left eye image, the edge contour of the user's left eye pupil in the first left eye image is identified;

[0049] Determine a first ellipse that matches the pupil edge contour;

[0050] The center point of the first ellipse is determined as the center point of the first pupil of the user's left eye.

[0051] In this embodiment, in the first left-eye image, since the pupil area is darker than the iris area, there is a significant brightness difference between the pupil and the iris. Based on this brightness difference, the MR device can process the first left-eye image using the Canny edge detection operator to quickly identify the pupil edge contour at the junction of the user's left pupil and iris in the first left-eye image. This pupil edge contour is composed of continuous edge pixels.

[0052] An ellipse fitting method can be used to fit the first ellipse that best matches the shape of the pupil based on these edge pixels, and the geometric center of this first ellipse can be used as the center point of the user's left pupil. For example, based on the pupil edge contour, the cv2.fitEllipse() function of OpenCV can be used to perform ellipse fitting, calculate the ellipse that best matches the pupil edge contour as the first ellipse, and use the center of this first ellipse as the center point of the user's left pupil.

[0053] Using the above method, the pupil edge contour can be determined by the brightness difference between the human eye pupil and iris, and the pupil center point can be determined based on the pupil edge contour. This method can accurately locate the center of the human eye pupil.

[0054] S103, determine the user's gaze direction based on the first left-eye image and the first right-eye image.

[0055] In this embodiment, the user's gaze direction refers to the direction in which both eyes are currently focused. After acquiring the user's current first left-eye image and first right-eye image, the center point of the first pupil of the user's left eye in the first left-eye image and the center point of the second pupil of the user's right eye in the first right-eye image can be determined based on the first left-eye image and the first right-eye image. Then, the center point of the first pupil is converted into a first coordinate point in a first optomechanical coordinate system, and the center point of the second pupil is converted into a second coordinate point in a second optomechanical coordinate system.

[0056] Using these two 3D coordinate points, a vector can be constructed that originates from the user's left or right pupil and points towards the point where the lines of sight intersect. This vector represents the user's gaze direction. The system further calculates the angle between this vector and the center of the screen to determine which angle range the user is currently looking at.

[0057] S104, determine the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

[0058] In this embodiment of the application, after determining the user's gaze direction and the first interpupillary distance value between the user's left and right eyes at the current moment, the system can infer the user's gaze depth through the user's current gaze direction and the first interpupillary distance value, and compare the user's gaze depth with the display depth of each display object, thereby determining the target display object that the user is actually gazing at among multiple overlapping display objects.

[0059] In this application, when at least two overlapping display objects are shown, the user's left-eye and right-eye images are acquired, and the user's current first interpupillary distance (IPD) and gaze direction are determined. Then, combining the user's gaze direction and IPD, the target display object currently being gazed at by the user is selected from the at least two overlapping display objects as the user's current interaction target. Through this method, the user's gaze can be combined with the real-time measured IPD to accurately identify the target display object that the user truly wants to interact with.

[0060] In some embodiments, determining the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value includes:

[0061] The user's line of sight is determined according to the line of sight direction, wherein multiple angle intervals are divided according to the angle between the user's line of sight direction and the center line of the display screen, and the multiple angle intervals include the first angle interval;

[0062] The database is used to query the first depth where there is a mapping relationship between the first angle interval and the first interpupillary distance value. The database includes multiple sets of mapping relationships between angle intervals, interpupillary distance values ​​and depths.

[0063] Determine the display depth of each display object in the at least two overlapping display objects;

[0064] Determine the depth difference between each display depth and the first depth;

[0065] The display object corresponding to the display depth with the smallest absolute value of the depth difference is determined as the target display object.

[0066] In this embodiment, when the target being gazed at by the user is closer to the center line of the screen, the eyes converge more, resulting in a smaller interpupillary distance (IPD) between the user's left and right pupils. Conversely, when the target being gazed at is further away from the center line, the user's eyes tend to look straight ahead or outward, resulting in a larger IPD. Therefore, at the same depth, the larger the angle between the user's line of sight and the center line of the display screen, the larger the user's IPD.

[0067] Therefore, multiple angle ranges can be divided according to the angle between the user's line of sight and the center line of the display screen. For example, as... Figure 3 As shown, multiple angle ranges can include a central area 301, a middle area 302, and an edge area 303. The center line 304 of the display plane can be defined. The angle between the central area and the center line is less than or equal to 15 degrees; the angle between the middle area and the center line is greater than 15 degrees and less than or equal to 25 degrees; and the angle between the edge area and the center line is greater than 25 degrees. Therefore, when a user gazes at the central area, their interpupillary distance (IPD) will be less than their natural IPD; when a user gazes at the middle area, their IPD will be close to their natural IPD; and when a user gazes at the edge area, their IPD will be greater than their natural IPD.

[0068] Furthermore, even if the user's gaze direction remains unchanged, when the user gradually shifts their focus from a distant target to a nearby target, the eyes will move in opposite directions, and the interpupillary distance between the eyes will decrease accordingly. Conversely, when the target being gazed at gradually moves away, the interpupillary distance between the eyes will gradually increase.

[0069] In summary, we can assume that a user's interpupillary distance (IPD) is inversely proportional to the depth of fixation and directly proportional to the angle of gaze. Therefore, after the system divides the screen into multiple angle intervals based on the angle between the user's gaze direction and the center line, a mapping relationship can be established between these angle intervals, IPD, and depth of fixation. A database is then built based on these pre-tested mapping relationships, and this database can be stored in tabular form. Therefore, by retrieving any two of these three parameters, the third parameter can be retrieved from the database.

[0070] Therefore, after determining the first angular interval to which the user's current gaze belongs and obtaining the user's current first interpupillary distance value, the first depth that has a mapping relationship between the first angular interval and the first interpupillary distance value can be queried from the database as the user's current gaze depth.

[0071] Furthermore, MR devices contain multiple display planes, which are positioned at different display depths to support various virtual display objects. Since each display object is rendered onto a specific display plane, the display depth of that object is also the display depth of the display plane on which it resides. Therefore, the system can determine the display depth of each display object based on the display depth of the display plane on which it resides.

[0072] Given the display depths of all overlapping display objects and the first depth obtained from the database, the display depths of multiple display objects in the current viewing direction can be compared one by one with the first depth. The absolute value of the difference between the display depth of each display object and the first depth can be calculated. Multiple absolute values ​​are then compared, and the display depth with the smallest absolute difference is considered the display object corresponding to that display depth as the target display object currently being viewed by the user.

[0073] For example, if the user's query result shows a first depth of 1.2 meters, and there are three overlapping display objects with display depths of 1.0 meters, 1.3 meters, and 1.6 meters respectively, and the system calculates the differences between these three objects and 1.2 meters to be 0.2, 0.1, and 0.4 meters respectively, then the display object with the smallest difference, a display depth of 1.3 meters, is selected as the target display object that the user is actually looking at.

[0074] The above scheme can be used to calculate the first depth of the user's current gaze based on the user's current gaze angle range and interpupillary distance value, and select the display object whose display depth is closest to the first depth from multiple overlapping display objects as the target display object for the user's current interaction, thereby accurately identifying the target display object of the MR device in a multi-layer spatial interface.

[0075] Specifically, in some embodiments, the database includes multiple mapping tables corresponding to multiple angle intervals, wherein each mapping table includes a mapping relationship between each pupil distance value and depth within a corresponding angle interval, and the multiple angle intervals include the first angle interval;

[0076] The step of querying the database for a first depth that has a mapping relationship with the first angle range and the first interpupillary distance value includes:

[0077] Determine the first mapping table corresponding to the first angle interval in the database;

[0078] Determine each pupil distance value included in the first mapping table and the pupil distance difference of the first pupil distance value;

[0079] The depth that has a mapping relationship with the interpupillary distance value that has the smallest absolute value of the interpupillary distance difference is determined as the first depth.

[0080] In this embodiment, the database may include multiple preset mapping tables, each recording the interpupillary distance (IPD) values ​​recorded by the user when focusing at different depths within a certain angular range. Therefore, each mapping table contains the mapping relationship between different depth values ​​and different IPD values ​​within the corresponding angular range. For example... Figure 3 As shown, multiple angle intervals may include a central region 301, a middle region 302, and an edge region 303. The angle between the central region and the center line 304 is less than or equal to 15 degrees, the angle between the middle region and the center line is greater than 15 degrees and less than or equal to 25 degrees, and the angle between the edge region and the center line is greater than 25 degrees. Mapping table 1 can represent the mapping relationship between the interpupillary distance value and depth of the "central region", mapping table 2 can represent the mapping relationship between the interpupillary distance value and depth of the "middle region", and mapping table 3 can represent the mapping relationship between the interpupillary distance value and depth of the "edge region".

[0081] In actual use, if it is determined that the user's gaze is within a first angular interval, a first mapping table corresponding to that first angular interval can be selected from the database. Then, the interpupillary distance (IPD) value closest to the current first IPD value can be found in the first mapping table. The depth value that has a mapping relationship with the closest IPD value is then determined as the first depth. This first depth is the most likely interaction depth in the current user's gaze direction.

[0082] In this embodiment, an interpupillary distance-depth mapping table corresponding to the angle range of the user's gaze can be determined. Then, based on the user's current interpupillary distance value, the depth value of the user's current gaze can be queried in the interpupillary distance-depth mapping table, which can accurately determine the actual depth of the user's gaze.

[0083] In some embodiments, the display screen has multiple display planes of different depths. Before querying the database for the first depth where the first angle range and the first interpupillary distance value have a mapping relationship, the method further includes:

[0084] For each display plane, a gaze point is displayed sequentially in each angular interval on the display plane;

[0085] When the first gaze point is displayed in the second angular range of the display plane, the user's first input is received;

[0086] In response to the first input, the user's second left-eye image and second right-eye image at the time of input of the first input are obtained;

[0087] The second interpupillary distance value of the user at the input time is determined based on the second left-eye image and the second right-eye image;

[0088] Construct a mapping relationship between the second interpupillary distance value, the depth of the display plane, and the second angle range.

[0089] In this embodiment, since the MR device includes multiple display planes of different depths, and can be divided into multiple angle intervals according to the angle between the user's line of sight and the center line of the display screen, it is necessary to obtain the interpupillary distance value when the user is looking at each angle interval on each display plane during the database construction process to ensure the comprehensiveness of the data.

[0090] The system can then sequentially display gaze points on each display plane according to angular intervals to guide the user's gaze at a specific location. For example, if the system displays a first gaze point in the second angular interval of a certain display plane, when the user gazes at this first gaze point, a first input can be made to the MR device to indicate that the user has gazed at the first gaze point. This first input can be a specific gesture, voice input, or button input on the MR device.

[0091] Upon receiving the first input, the MR device can respond by capturing a second left-eye image and a second right-eye image at the moment of input using its built-in camera. Based on these images, it calculates the user's second interpupillary distance (IPD) at the moment of input and constructs a mapping relationship between the IPD, the depth of the display plane, and the second angular interval. Subsequently, the user can perform repeated sampling based on different display planes and angular intervals to obtain multiple mapping relationships, forming a complete mapping table. This table is then stored in a database, which can be used to subsequently calculate the user's gaze depth based on their real-time IPD and gaze direction, thereby achieving precise interaction.

[0092] like Figure 3As shown, multiple angle intervals can include a central area 301, a middle area 302, and an edge area 303. Fixation point 1 can be displayed sequentially in the central area of ​​a display plane, fixation point 2 in the middle area, and fixation point 3 in the edge area. When fixation point 1 is displayed and the user is looking at it, the user can make a "pinch" gesture as the first input to confirm looking at the currently displayed fixation point. The eye-tracking module within the MR device will then capture the user's left and right eye images at that moment to record the user's eye state at the current fixation point. Subsequently, the interface automatically jumps to the next fixation point 2 and repeats the process. Furthermore, after completing the current display plane, the displayed user interface switches to a display plane of another depth, continuing to display fixation points of different depths sequentially on that display plane.

[0093] By using the above method, users can be guided to gaze and interpupillary distance can be collected within various angle ranges on different display planes, thus establishing a precise mapping relationship between interpupillary distance, display depth, and angle ranges, providing a high-quality reference for subsequent depth inference.

[0094] In some embodiments, before determining the target display object being gazed at by the user from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value, the method further includes:

[0095] Display multiple gaze points with different depths;

[0096] Obtain eye images of the user while they are gazing at each of the stated fixation points;

[0097] The interpupillary distance (IPD) value of the user when gazing at each of the stated fixation points is determined based on the eye image and used as the first calibrated IPD value.

[0098] In this embodiment, the MR device can display multiple different gaze points on multiple display planes of different depths and prompt the user to gaze at each gaze point in sequence. The user can gaze at each gaze point in sequence as prompted and make a "pinch" gesture when gazing at a certain prompted gaze point to inform the MR device that the user is gazing at the currently prompted gaze point.

[0099] During a user's fixation on a specific point, the MR device can acquire real-time images of the user's left and right eyes using an eye-tracking module. The system then extracts the center points of the pupils of the user's left and right eyes from these images. Based on these center points, it calculates the interpupillary distance (IPD) for each fixation point, which serves as the first calibrated IPD value for that fixation point. This first calibrated IPD value is the distance between the center points of the pupils of the user's two eyes at a known display depth, calculated by the MR device from the eye images. This first calibrated IPD value can be used as a reference for determining the user's fixation depth during subsequent interactions.

[0100] Since the depth of each fixation point is different, the interpupillary distance value will also change when the user fixates on different fixation points. Therefore, by displaying multiple fixation points with different depths and guiding the user to fixate, the first calibrated interpupillary distance value of the user at multiple fixation depths can be recorded.

[0101] These initial interpupillary distance values ​​will be mapped to the corresponding gaze depth to form a set of reference data. Then, when the user gazes in a certain direction, the user's current interpupillary distance value can be combined to determine the user's actual gaze depth, thereby accurately identifying the target display object being gazed at.

[0102] Exemplary, as an optional embodiment, such as Figure 4 As shown, after the user puts on the MR device, the device initiates an eye-tracking calibration process. During this process, the user interface first displays several target gaze points at different display depths, guiding the user to focus on each gaze point sequentially. While the user is focusing on each gaze point, the MR device can capture images of the user's left and right eyes as they gaze at those points.

[0103] Next, the system analyzes the eye images of the left and right eyes through an image processing workflow. First, it quickly locates the pupil region of the eyes in the left and right eye images, and then uses the Canny edge detection algorithm to identify the pupil edge contour. Based on the pupil edge contour, an ellipse fitting algorithm is used to fit the pupil edge contour into an ellipse, and the center point of the ellipse is determined as the center coordinate of the pupil.

[0104] Subsequently, using the center coordinates and the currently read Hall sensor value, the center coordinates of the pupil can be projected onto the corresponding optomechanical coordinate system to calculate the user's current interpupillary distance.

[0105] Finally, based on the pre-built relationship table between depth and interpupillary distance, the depth value that best matches the current interpupillary distance can be found as the current user's gaze depth, and the user's final interaction focus can be determined based on the gaze depth, thereby achieving accurate identification and response to the user's gaze content.

[0106] In some embodiments, before determining the target display object being gazed at by the user from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value, the method further includes:

[0107] Display multiple gaze points located at the same depth, wherein the distances between the multiple gaze points and the center line of the display area are different;

[0108] Obtain eye images of the user while they are gazing at each of the stated fixation points;

[0109] The interpupillary distance value determined based on the eye image when the user fixates on each of the fixation points is used as the second calibrated interpupillary distance value.

[0110] In this embodiment, the MR device can display multiple gaze points on a display plane with the same display depth. These gaze points are located in different angle ranges, meaning they have different angles relative to the center line of the display area. The system can prompt the user to gaze at each gaze point in sequence. The user follows the prompts to gaze at these gaze points in sequence and performs a confirmation gesture such as "pinch" when gazing at a specific gaze point to notify the MR device that they are currently gazing at that gaze point.

[0111] During each fixation point, the MR device acquires real-time images of the user's left and right eyes using an eye-tracking module. Based on the pupil center coordinates extracted from the left and right eye images, it calculates the interpupillary distance (IPD) value for the current fixation state. Since the fixation point depth is fixed but the angle range varies, the system records the IPD changes at different fixation angles at the same depth, using the IPD value at each fixation point as the second calibrated IPD value for that point. Therefore, the second calibrated IPD value is the distance between the pupil centers of the user's two eyes at a known viewing angle at a fixed depth, calculated by the MR device from the eye images. This second calibrated IPD value can serve as a reference for determining the user's fixation depth in subsequent interactions.

[0112] These second calibrated interpupillary distance values ​​will be mapped together with their corresponding angle intervals to form a set of reference data for subsequent interaction recognition. When the user's gaze direction is identified during actual interaction, the second calibrated interpupillary distance values ​​corresponding to the current angle interval can be combined to further assist in determining the display depth of the user's gaze, thereby improving the accuracy of target display object recognition.

[0113] Figure 5 This is a schematic diagram of the structure of a display device 500 provided in another embodiment of this application, as shown below. Figure 5 As shown, the display device may include:

[0114] The eye-tracking detection module 501 is used to acquire a first left-eye image and a first right-eye image of the user when at least two superimposed display objects are displayed on the display device, and to determine the user's gaze direction based on the first left-eye image and the first right-eye image.

[0115] Processing module 502 is used to determine the user's first interpupillary distance value based on the first left eye image and the first right eye image;

[0116] The processing module 502 is further configured to determine the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

[0117] In this application, when at least two overlapping display objects are shown, the user's left-eye and right-eye images are acquired, and the user's current first interpupillary distance (IPD) and gaze direction are determined. Then, combining the user's gaze direction and IPD, the target display object currently being gazed at by the user is selected from the at least two overlapping display objects as the user's current interaction target. Through this method, the user's gaze can be combined with the real-time measured IPD to accurately identify the target display object that the user truly wants to interact with.

[0118] In another alternative example, the processing module 502 includes:

[0119] The first determining unit is used to determine the center point of the first pupil of the user's left eye in the first left eye image and the center point of the second pupil of the user's right eye in the first right eye image.

[0120] The conversion unit is used to convert the center point of the first pupil into a first coordinate point in a first optomechanical coordinate system, and to convert the center point of the second pupil into a second coordinate point in a second optomechanical coordinate system. The first optomechanical coordinate system is a spatial coordinate system established with the location of the left eye optomechanical system as the origin, and the second optomechanical coordinate system is a spatial coordinate system established with the location of the right eye optomechanical system as the origin.

[0121] The second determining unit is used to determine the first interpupillary distance value based on the first coordinate point, the second coordinate point, and the distance between the left and right optical transducers.

[0122] In another alternative example, the first determining unit includes:

[0123] The recognition subunit is used to identify the edge contour of the pupil of the user's left eye in the first left eye image based on the brightness difference between the pupil and iris of the user's left eye in the first left eye image.

[0124] The first determining subunit is used to determine a first ellipse that matches the pupil edge contour;

[0125] The second determining subunit is used to determine the center point of the first ellipse as the center point of the first pupil of the user's left eye.

[0126] In another alternative example, the processing module 502 includes:

[0127] The third determining unit is used to determine the first angle interval where the user's line of sight is located according to the line of sight direction, wherein multiple angle intervals are divided according to the angle between the user's line of sight direction and the center line of the display screen, and the multiple angle intervals include the first angle interval.

[0128] The query unit is used to query a database for a first depth that has a mapping relationship with the first angle interval and the first interpupillary distance value, wherein the database includes multiple sets of mapping relationships between angle intervals, interpupillary distance values ​​and depths;

[0129] The fourth determining unit is used to determine the display depth of each display object in the at least two overlapping display objects;

[0130] The fifth determining unit is used to determine the depth difference between each display depth and the first depth;

[0131] The sixth determining unit is used to determine the display object corresponding to the display depth with the smallest absolute value of the depth difference as the target display object.

[0132] In another alternative example, the query unit includes:

[0133] The third determining subunit is used to determine the first mapping table corresponding to the first angle interval in the database;

[0134] The fourth determining subunit is used to determine each pupil distance value included in the first mapping table and the pupil distance difference of the first pupil distance value;

[0135] The fifth determining subunit is used to determine the depth that has a mapping relationship with the pupil distance value with the smallest absolute value of the pupil distance difference as the first depth.

[0136] In another alternative example, the display device includes:

[0137] The display module is used to sequentially display a gaze point in each angle interval on each display plane;

[0138] The receiving module is configured to receive a first input from the user when a first gaze point is displayed in the second angular range of the display plane;

[0139] The acquisition module is configured to, in response to the first input, acquire the user's second left-eye image and second right-eye image at the time of input of the first input;

[0140] The processing module is also configured to determine the second interpupillary distance value of the user at the input time based on the second left-eye image and the second right-eye image;

[0141] The processing module is also used to construct a mapping relationship between the second interpupillary distance value, the depth of the display plane, and the second angle range.

[0142] In another alternative example, the display device further includes:

[0143] The display module is used to display multiple gaze points with different depths;

[0144] The acquisition module is used to acquire eye images of the user when he / she is gazing at each of the aforementioned gaze points;

[0145] The processing module is further configured to determine the interpupillary distance value of the user when gazing at each of the fixation points based on the eye image as a first calibrated interpupillary distance value.

[0146] In another alternative example, the display device further includes:

[0147] The display module is also used to display multiple gaze points located at the same depth, wherein the multiple gaze points are at different distances from the center line of the display area;

[0148] The acquisition module is also used to acquire eye images of the user when he is looking at each of the fixation points;

[0149] The processing module is further configured to determine the interpupillary distance value of the user when gazing at each of the fixation points based on the eye image as a second calibrated interpupillary distance value.

[0150] The display device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific type of device.

[0151] The display device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0152] The display device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0153] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 100, including a processor 110, a memory 119, and a program or instructions stored in the memory 119 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described display object interaction method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0154] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0155] Please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. The electronic device 100 includes, but is not limited to, components such as: a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0156] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 120 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0157] The input unit 124 is used to acquire a first left-eye image and a first right-eye image of the user when the display device displays at least two superimposed display objects, and to determine the user's gaze direction based on the first left-eye image and the first right-eye image.

[0158] Processor 120 is configured to determine a user's first interpupillary distance value based on the first left-eye image and the first right-eye image;

[0159] The processor 120 is further configured to determine the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

[0160] In this application, when at least two overlapping display objects are shown, the user's left-eye and right-eye images are acquired, and the user's current first interpupillary distance (IPD) and gaze direction are determined. Then, combining the user's gaze direction and IPD, the target display object currently being gazed at by the user is selected from the at least two overlapping display objects as the user's current interaction target. Through this method, the user's gaze can be combined with the real-time measured IPD to accurately identify the target display object that the user truly wants to interact with.

[0161] In another alternative example, the processor 120 is specifically used for:

[0162] Determine the center point of the first pupil of the user's left eye in the first left eye image, and the center point of the second pupil of the user's right eye in the first right eye image;

[0163] The first pupil center point is converted into a first coordinate point in a first optomechanical coordinate system, and the second pupil center point is converted into a second coordinate point in a second optomechanical coordinate system. The first optomechanical coordinate system is a spatial coordinate system established with the location of the left eye optomechanical system as the origin, and the second optomechanical coordinate system is a spatial coordinate system established with the location of the right eye optomechanical system as the origin.

[0164] The first interpupillary distance value is determined based on the first coordinate point, the second coordinate point, and the distance between the left and right optical transducers.

[0165] In another alternative example, the processor 120 is specifically used for:

[0166] Based on the brightness difference between the pupil and iris of the user's left eye in the first left eye image, the edge contour of the user's left eye pupil in the first left eye image is identified;

[0167] Determine a first ellipse that matches the pupil edge contour;

[0168] The center point of the first ellipse is determined as the center point of the first pupil of the user's left eye.

[0169] In another alternative example, the processor 120 is specifically used for:

[0170] The user's line of sight is determined according to the line of sight direction, wherein multiple angle intervals are divided according to the angle between the user's line of sight direction and the center line of the display screen, and the multiple angle intervals include the first angle interval.

[0171] The database is used to query the first depth that has a mapping relationship with the first angle interval and the first interpupillary distance value. The database includes multiple sets of mapping relationships between angle intervals, interpupillary distance values ​​and depths.

[0172] Determine the display depth of each display object in the at least two overlapping display objects;

[0173] Determine the depth difference between each display depth and the first depth;

[0174] The display object corresponding to the display depth with the smallest absolute value of the depth difference is determined as the target display object.

[0175] In another alternative example, the processor 120 is specifically used for:

[0176] Determine the first mapping table corresponding to the first angle interval in the database;

[0177] Determine each pupil distance value included in the first mapping table and the pupil distance difference of the first pupil distance value;

[0178] The depth that has a mapping relationship with the interpupillary distance value that has the smallest absolute value of the interpupillary distance difference is determined as the first depth.

[0179] In another alternative example, the electronic device includes:

[0180] Display unit 126 is used to sequentially display a gaze point in each angle interval on each display plane;

[0181] User input unit 127 is used to receive first input from the user when a first gaze point is displayed in the second angular range of the display plane;

[0182] Input unit 124 is configured to, in response to the first input, acquire the user's second left-eye image and second right-eye image at the time of input of the first input;

[0183] The processor 120 is further configured to determine the second interpupillary distance value of the user at the input time based on the second left-eye image and the second right-eye image;

[0184] The processor 120 is also used to construct a mapping relationship between the second interpupillary distance value, the depth of the display plane, and the second angle range.

[0185] In another alternative example, the electronic device further includes:

[0186] Display unit 126 is used to display multiple gaze points with different depths;

[0187] Input unit 124 is used to acquire eye images of the user when he / she is looking at each of the fixation points;

[0188] The processor 120 is further configured to determine, based on the eye image, the interpupillary distance value of the user when gazing at each of the fixation points as a first calibrated interpupillary distance value.

[0189] In another alternative example, the display unit 126 is also configured to display a plurality of gaze points located at the same depth, wherein the plurality of gaze points are at different distances from the center line of the display area;

[0190] The input unit 124 is also used to acquire eye images of the user when he is looking at each of the fixation points;

[0191] The processor 120 is further configured to determine, based on the eye image, the interpupillary distance value of the user when gazing at each of the fixation points as a second calibrated interpupillary distance value.

[0192] It should be understood that, in this embodiment, the input unit 124 may include a graphics processing unit (GPU) 1241 and a microphone 1242. The GPU 1241 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 126 may include a display panel 1261, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 127 includes at least one of a touch panel 1271 and other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0193] The memory 129 can be used to store software programs and various data. The memory 129 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 129 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 129 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0194] Processor 120 may include one or more processing units; optionally, processor 120 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 120.

[0195] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described display object interaction method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0196] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0197] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described display object interaction method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0198] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0199] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described display object interaction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0200] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0201] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0202] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A display object interaction method, applied to a display device with an eye-tracking detection module, characterized in that, include: In the case of displaying at least two overlapping display objects, obtain the user's first left-eye image and first right-eye image; The user's first interpupillary distance value is determined based on the first left eye image and the first right eye image; Determine the user's gaze direction based on the first left-eye image and the first right-eye image; The target display object that the user is looking at is determined from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

2. The method according to claim 1, characterized in that, Determining the user's first interpupillary distance value based on the first left-eye image and the first right-eye image includes: Determine the center point of the first pupil of the user's left eye in the first left eye image, and the center point of the second pupil of the user's right eye in the first right eye image; The first pupil center point is converted into a first coordinate point in a first optomechanical coordinate system, and the second pupil center point is converted into a second coordinate point in a second optomechanical coordinate system. The first optomechanical coordinate system is a spatial coordinate system established with the location of the left eye optomechanical system as the origin, and the second optomechanical coordinate system is a spatial coordinate system established with the location of the right eye optomechanical system as the origin. The first interpupillary distance value is determined based on the first coordinate point, the second coordinate point, and the distance between the left and right optical transducers.

3. The method according to claim 2, characterized in that, Determining the center point of the first pupil of the user's left eye in the first left eye image includes: Based on the brightness difference between the pupil and iris of the user's left eye in the first left eye image, the edge contour of the user's left eye pupil in the first left eye image is identified; Determine a first ellipse that matches the pupil edge contour; The center point of the first ellipse is determined as the center point of the first pupil of the user's left eye.

4. The method according to claim 1, characterized in that, The step of determining the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value includes: The user's line of sight is determined according to the line of sight direction, wherein multiple angle intervals are divided according to the angle between the user's line of sight direction and the center line of the display screen, and the multiple angle intervals include the first angle interval. The database is used to query the first depth that has a mapping relationship with the first angle interval and the first interpupillary distance value. The database includes multiple sets of mapping relationships between angle intervals, interpupillary distance values ​​and depths. Determine the display depth of each display object in the at least two overlapping display objects; Determine the depth difference between each display depth and the first depth; The display object corresponding to the display depth with the smallest absolute value of the depth difference is determined as the target display object.

5. The method according to claim 4, characterized in that, The display screen has multiple display planes at different depths. Before querying the database for the first depth where the first angle range and the first interpupillary distance value have a mapping relationship, the method further includes: For each display plane, a gaze point is displayed sequentially in each angular interval on the display plane; When a first gaze point is displayed in the second angular range of the display plane and the user gazes at the first gaze point, the user's first input is received. In response to the first input, the user's second left-eye image and second right-eye image are acquired; The second interpupillary distance value is determined based on the second left-eye image and the second right-eye image when the user is fixating on the first fixation point; Construct a mapping relationship between the second interpupillary distance value, the depth of the display plane, and the second angle range.

6. The method according to claim 1, characterized in that, Before determining the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value, the method further includes: Display multiple gaze points with different depths; Obtain eye images of the user while they are gazing at each of the stated fixation points; The interpupillary distance (IPD) value of the user when gazing at each of the stated fixation points is determined based on the eye image and used as the first calibrated IPD value.

7. The method according to claim 6, characterized in that, Before determining the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value, the method further includes: Display multiple gaze points located at the same depth, wherein the distances between the multiple gaze points and the center line of the display area are different; Obtain eye images of the user while they are gazing at each of the stated fixation points; The interpupillary distance value determined based on the eye image when the user fixates on each of the fixation points is used as the second calibrated interpupillary distance value.

8. A display device, characterized in that, include: An eye-tracking detection module is used to acquire a first left-eye image and a first right-eye image of a user when at least two overlaid display objects are displayed on the display device, and to determine the user's gaze direction based on the first left-eye image and the first right-eye image. The processing module is used to determine the user's first interpupillary distance value based on the first left eye image and the first right eye image; The processing module is further configured to determine the target display object that the user is looking at from at least two overlapping display objects based on the gaze direction and the first interpupillary distance value.

9. The display device according to claim 8, characterized in that, The display device further includes: The display module is used to display multiple gaze points with different depths; The acquisition module is used to acquire eye images of the user when he / she is gazing at each of the aforementioned fixation points; The processing module is further configured to determine the interpupillary distance value of the user when gazing at each of the fixation points based on the eye image as a first calibrated interpupillary distance value.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the display object interaction method as described in any one of claims 1-7.

Images