Method and apparatus for detecting visual attention region, and electronic device

Abstract

A method and apparatus for detecting a visual attention region, and an electronic device, which belong to the technical field of computer vision. The method comprises: determining two eye corner points and a pupil center point of at least one eye in an image to be subjected to detection (S101); determining a first connecting line segment of the at least one eye in the image to be subjected to detection, wherein the first connecting line segment is used for connecting the two eye corner points (S102); on the basis of a positional relationship between a line segment midpoint of the first connecting line segment of the at least one eye and the pupil center point of the at least one eye, determining a visual attention region of a user (S103); and when the visual attention region is not a central region of a display screen of an electronic device, outputting prompt information, and re-acquiring an image to be subjected to detection, wherein the prompt information is used for instructing a human eye to gaze at the central region of the display screen (S104).

Description

Methods, devices and electronic equipment for detecting visual attention regions

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410963167.1, filed on July 17, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of computer vision technology, specifically relating to a method, apparatus, and electronic device for detecting visually relevant regions. Background Technology

[0004] When users use Mixed Reality (MR) devices, it is usually necessary to detect the user's interpupillary distance and perform iris recognition. During the interpupillary distance detection or iris recognition process, the user is usually required to look directly at the center area of ​​the MR device's screen.

[0005] In existing technologies, a user interface is generated in the center of the MR device's screen to attract the user's attention during pupil distance detection and iris recognition. However, since it is uncertain whether the user will correctly follow the instructions, recognition may fail if the user does not cooperate properly, thus affecting the user experience. Summary of the Invention

[0006] The purpose of this application is to provide a method, apparatus, and electronic device for detecting visual attention areas, which can solve the technical problem of low accuracy in existing iris recognition due to the inability to detect whether the user is cooperating with guidance.

[0007] In a first aspect, embodiments of this application provide a method for detecting a visually attention-grabbing region, the method comprising:

[0008] Determine the two outer corners of at least one eye and the center point of the pupil in the image to be detected;

[0009] A first connecting line segment is determined for at least one eye in the image to be detected, wherein the first connecting line segment is used to connect the two corner points of the eye;

[0010] The user's visual attention area is determined based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil;

[0011] If the visual attention area is not the center area of ​​the display screen of the electronic device, a prompt message is output and the image to be detected is reacquired. The prompt message is used to instruct the human eye to focus on the center area of ​​the display screen.

[0012] Secondly, embodiments of this application provide a device for detecting a visually attentional region, the device comprising:

[0013] The first determining module is used to determine the two corner points of the eye and the center point of the pupil of at least one eye in the image to be detected;

[0014] The second determining module is used to determine a first connecting line segment for at least one eye in the image to be detected, wherein the first connecting line segment is used to connect the two corner points of the eye;

[0015] The third determining module is used to determine the user's visual attention area based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil;

[0016] The output module is configured to output a prompt message and reacquire the image to be detected when the visual attention area is not the central area of ​​the display screen of the electronic device. The prompt message is used to instruct the human eye to focus on the central area of ​​the display screen.

[0017] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method provided in the first aspect.

[0018] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method provided in the first aspect.

[0019] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface, the communication interface and the processor being coupled together, the processor being used to run programs or instructions to implement the steps of the method provided in the first aspect.

[0020] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the steps of the method provided in the first aspect.

[0021] In the visual attention area detection method, apparatus, and electronic device of this application, a user's image to be detected can be acquired, and the positions of the two corner points of at least one eye and the center point of the pupil of the user can be determined in the image. Then, the user's visual attention area is determined based on the position of the midpoint of the first connecting line segment formed by the two corner points and the center point of the pupil. Since the relative position of the center point of the pupil in the eye can characterize the user's visual attention area, and the midpoint of the connecting line segment can be approximated as the center of the human eye, the midpoint of the line segment can be used as a reference point, and the user's actual visual attention area is detected based on the positional relationship between the center point of the pupil and the midpoint of the line segment. If the user's visual attention area is the center area of ​​the display screen, subsequent steps can proceed normally; if the user's visual attention area is not the center area of ​​the display screen, it can be considered that the user has not cooperated correctly with the guidance, and then a prompt message can be output to the user to prompt the user to shift the area of ​​attention to the center area of ​​the display screen. This ensures that when performing pupil distance detection and iris recognition, the area that the user focuses on is the center of the display screen, thereby improving the accuracy of recognition and detection and enhancing the user experience. Attached Figure Description

[0022] Figure 1 is a flowchart illustrating a method for detecting a region of visual interest according to an embodiment of this application;

[0023] Figure 2 is a schematic diagram of an image to be detected provided in an embodiment of this application;

[0024] Figure 3 is a schematic diagram of an image to be detected provided in another embodiment of this application;

[0025] Figure 4 is a schematic diagram of the structure of a visual interest region detection device provided in another embodiment of this application;

[0026] Figure 5 is a schematic diagram of the structure of an electronic device provided in another embodiment of this application;

[0027] Figure 6 is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] To address the aforementioned technical problems, this application provides a method for detecting visually attention-grabbing regions. The method for detecting visually attention-grabbing regions provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] As shown in Figure 1, Figure 1 is a flowchart illustrating a method for detecting a visually attention-based region according to an embodiment of this application. This embodiment of the application provides a method for detecting a visually attention-based region, which may include:

[0032] S101, determine the two corner points of at least one eye and the center point of the pupil in the image to be detected.

[0033] In this embodiment, when a user uses an electronic device, the user interface and main content of the electronic device are displayed in a layout on the target area of ​​the electronic device's display screen. Therefore, it can be determined whether the user is paying attention to the content displayed on the electronic device's display screen by detecting whether the user's visual focus area is the target area.

[0034] Specifically, the electronic device includes a camera that can capture an image of the user to be detected, including at least one of the user's eyes. After capturing the image, taking any one eye in the image as an example, the two inner corners of that eye and the center point of the pupil can be located within the image. The inner corners of the eye refer to the inner corners of the eye, i.e., the intersection of the upper and lower eyelids. Both the inner corners and the center point of the pupil in the image are individual pixels within the image.

[0035] In some embodiments, S101 includes:

[0036] The upper and lower eyelid edges of at least one eye in the image to be detected are determined using an edge detection algorithm.

[0037] The first curve corresponding to the upper eyelid edge and the second curve corresponding to the lower eyelid edge in the image to be detected are determined by curve fitting.

[0038] The two intersection points of the first curve and the second curve are determined as the two corner points of the eye.

[0039] In this embodiment, in determining the two corner points of an eye in a user's image to be detected, a Gaussian filter is first used to smooth the image to reduce noise. After filtering, the Canny edge detection algorithm can be used to locate the upper and lower eyelid edges of the eye in the image to be detected.

[0040] After locating the upper and lower eyelid edges, as shown in Figure 2, quadratic curve fitting can be performed on the upper and lower eyelid edges to obtain two quadratic curves, namely the first curve s1 and the second curve s2. The first curve represents the upper eyelid edge of the user's eye in the image to be detected, and the second curve represents the lower eyelid edge of the user's eye in the image to be detected. Therefore, the two intersection points of the first curve s1 and the second curve s2 can be further calculated, namely the first intersection point a and the second intersection point b shown in Figure 2. These two intersection points are the two corners of the user's eye.

[0041] For example, a two-dimensional image coordinate system can be constructed on the image to be detected. Then, after determining the first curve corresponding to the upper eyelid edge and the second curve corresponding to the lower eyelid edge in the image to be detected through curve fitting, the first curve function corresponding to the first curve in the image coordinate system and the second curve function corresponding to the second curve in the image coordinate system can be obtained. By combining the first curve function and the second curve function, the first coordinate value and the second coordinate value of the two intersection points of the first curve and the second curve in the image coordinate system can be calculated. In this way, the positions of the two corner points of the eyes on the image to be detected can be determined.

[0042] Using the above method, the position of the eyelid edge in the image to be detected can be determined by edge detection, and the user's upper and lower eyelids can be fitted into a curve by curve fitting. The intersection of the two curves is determined as the user's corner of the eye. In this way, the corner of the user's eye can be accurately located in the image to be detected.

[0043] In some embodiments, S101 includes:

[0044] The edge of the pupil of at least one eye in the image to be detected is determined by an edge detection algorithm, and multiple pupil edge feature points are obtained on the pupil edge.

[0045] The pupil center point is determined based on multiple pupil edge feature points.

[0046] In this embodiment, when determining the pupil center point of the eye in the user's image to be detected, Gaussian filtering can be used first to smooth the image and reduce noise. After filtering, taking one eye in the image to be detected as an example, as shown in Figure 2, the Canny edge detection algorithm can be used to locate the pupil edge s3 in the image. Then, multiple pupil edge feature points are extracted on the pupil edge s3, and the position of the pupil center point in the image to be detected is determined based on the positions of the multiple pupil edge feature points.

[0047] For example, in a two-dimensional image coordinate system on the image to be detected, the third coordinate value of each pupil edge feature point among multiple pupil edge feature points can be determined in the image coordinate system. Then, the fourth coordinate value of the pupil center point can be determined based on the multiple third coordinate values. Specifically, the average of the abscissas of the multiple third coordinate values ​​can be determined as the abscissa of the fourth coordinate value, and the average of the ordinates of the multiple fourth coordinate values ​​can be determined as the ordinate of the fourth coordinate value.

[0048] In this embodiment, the position of the pupil edge in the image to be detected can be determined by edge detection, and a set of pupil edge feature points can be taken on the pupil edge. Then, the center point of the pupil of the eye can be located based on this set of pupil edge feature points. In this way, the center point of the user's pupil can be accurately located in the image to be detected.

[0049] As an optional embodiment, determining the pupil center point based on a plurality of pupil edge feature points includes:

[0050] The geometric midpoint of the plurality of pupil edge feature points is determined as the pupil center point, or,

[0051] The point whose sum of distances to the multiple pupil edge feature points is the smallest is determined as the pupil center point.

[0052] In this embodiment, after determining multiple pupil edge feature points on one of the user's eyes in the image to be detected, the geometric midpoint of the multiple pupil edge feature points can be determined as the pupil center point of the eye, or the Fermat point with the smallest sum of distances to all pupil edge feature points can be determined as the pupil center point.

[0053] For example, in a two-dimensional image coordinate system on the image to be detected, the third coordinate value of each pupil edge feature point in the image coordinate system can be determined. Then, the first average value of all third coordinate points on the x-axis and the second average value on the y-axis are calculated. The point whose x-coordinate is the first average value and whose y-coordinate is the second average value is determined as the geometric midpoint of the plurality of pupil edge feature points.

[0054] The geometric midpoint can be directly determined as the center point of the pupil; alternatively, it can be used as an initial candidate point for the Fermat point, and then an iterative algorithm (such as simulated annealing or gradient descent) can be used to optimize the position of the candidate point, minimizing the sum of distances from the candidate point to all pupil edge feature points. After iterative optimization, the candidate point with the minimum sum of distances to all pupil edge feature points is the determined pupil center point.

[0055] Using the methods described above, the geometric midpoint or the Fermat point with the smallest sum of distances among multiple pupil edge feature points can be determined as the pupil center point. The calculation of the geometric midpoint is simple and quick, while the Fermat point method is more suitable for situations where the pupil edge feature points are unevenly distributed or contain noise, and can more accurately determine the pupil center point.

[0056] S102, determine the first connecting line segment of the at least one eye in the image to be detected, the first connecting line segment being used to connect the two corner points of the eye.

[0057] In this embodiment, after determining at least one eye in the image to be detected, a first connecting line segment corresponding to each eye can be determined. Taking any eye in the image to be detected as an example, after determining the positions of the two corner points of the eye in the image to be detected, the two corner points can be connected by a line segment, thus generating a first connecting line segment on the image to be detected, the two endpoints of which are exactly the two corner points.

[0058] S103, determine the user's visual attention area based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil.

[0059] In this embodiment, the relative position of the pupil within the eye can characterize the user's visual focus area. For example, if the user's pupil is in the center of the eye, the user's visual focus area can be considered to be directly in front; if the user's pupil is slightly to the left of the eye, the user's visual focus area can be considered to be slightly to the left; and if the user's pupil is slightly to the right of the eye, the user's visual focus area can be considered to be slightly to the right.

[0060] For any eye in the image to be detected, the midpoint of the first connecting line segment corresponding to that eye is actually the midpoint of the two corners of the eye, and the midpoint of the two corners can be approximated as the center of the human eye. Therefore, the midpoint of the line segment can be used as a reference point for analyzing the user's pupil offset and determining the user's visual focus area. By analyzing the positional relationship between the pupil center point and the midpoint of the line segment, the user's visual focus area can be determined.

[0061] For example, as shown in Figure 3, the two corners of the eye are corner a and corner b. The midpoint of the first connecting line segment ab, which connects corner a and corner b, is the midpoint m of the line segment. The center point of the pupil is point p. If the pupil center p is close to the midpoint m, the user's visual focus area can be considered to be directly in front. If the pupil center p is located to the left of the midpoint m, the user's visual focus area can be considered to be shifted to the left. If the pupil center p is located to the right of the midpoint m, the user's visual focus area can be considered to be shifted to the right.

[0062] In this application, a user's image to be detected can be acquired, and the positions of the two corner points of the user's eyes and the center point of the pupil in the image can be determined. Then, the user's visual attention area can be determined based on the position of the midpoint of the first connecting line segment formed by the two corner points and the center point of the pupil. Since the relative position of the center point of the pupil in the eye can characterize the user's visual attention area, and the midpoint of the connecting line segment between the two corner points can be approximated as the center of the human eye, the midpoint of the line segment can be used as a reference point, and the user's actual visual attention area can be detected based on the positional relationship between the center point of the pupil and the midpoint of the line segment.

[0063] Furthermore, if the image to be detected only includes one of the user's eyes, the midpoint of the line segment corresponding to that eye and the center point of the pupil can be determined using the method described above. The midpoint of the line segment is then used as the reference point for analyzing the user's pupil offset and determining the user's visual focus area. By analyzing the positional relationship between the pupil center point and the midpoint of the line segment, the user's visual focus area can be determined.

[0064] If the image to be detected includes images of the user's two eyes, and the visual attention regions determined by the two eyes are the same, then the visual attention region corresponding to both eyes is determined as the user's visual attention region.

[0065] If the image to be detected includes images of the user's two eyes, and the visual attention regions determined by the user's two eyes are different, then the center point of the first visual attention region corresponding to the left eye and the center point of the second visual attention region corresponding to the right eye can be calculated separately. Then, the average position of these two center points is taken as the final visual attention region of the user. Alternatively, the confidence scores of the user's left eye and right eye can be set in advance. When the visual attention regions corresponding to the user's two eyes are different, the visual attention region corresponding to the eye with the higher confidence score is determined as the user's visual attention region.

[0066] In some embodiments, S103 includes:

[0067] If the distance between the midpoint of the line segment and the center point of the pupil is less than or equal to a distance threshold, the user's visual attention area is determined to be the central area of ​​the display screen of the electronic device.

[0068] If the distance between the midpoint of the line segment and the center point of the pupil is greater than the distance threshold, it is determined that the user's visual attention area is not the central area of ​​the display screen.

[0069] In this embodiment, when a user is using an electronic device, the user is typically facing the display screen. Therefore, when the user is using the electronic device, an image of at least one eye can be captured for detection. Then, the user's visual attention area can be determined based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil. Taking one eye in the image as an example, the midpoint of the first connecting line segment of that eye can be used as a reference point. By analyzing the relative positional relationship between the center point of the pupil and the midpoint of the line segment, the relationship between the user's visual attention area and the display screen can be determined.

[0070] As an alternative embodiment, the relationship between the user's visual focus area and the display screen can be determined based on the distance between the center point of the pupil and the midpoint of the line segment.

[0071] Specifically, a distance threshold can be set in advance. If the distance between the midpoint of the line segment and the center point of the pupil is less than or equal to the distance threshold, the user's pupil can be considered to be roughly in the center of the eye. Since the user is facing the display screen of the electronic device, the user's visual focus area can be considered to be the central area of ​​the display screen of the electronic device. If the distance between the midpoint of the line segment and the center point of the pupil is greater than the distance threshold, the user's pupil can be considered to be at the edge of the eye, and the user's visual focus area can be considered to be not the central area of ​​the display screen of the electronic device.

[0072] As shown in Figure 3, if the distance pm between the pupil center point p and the midpoint m of the line segment is less than or equal to the distance threshold, the user's visual attention area is considered to be the central area of ​​the electronic device's display screen; if the distance pm between the pupil center point p and the midpoint m of the line segment is greater than the distance threshold, the user's visual attention area is considered not to be the central area of ​​the electronic device's display screen.

[0073] In this embodiment, the user's visual attention area can be determined as the central area of ​​the display screen based solely on the distance between the center point of the pupil and the midpoint of the line segment. This method for detecting the visual attention area is relatively simple and can therefore efficiently determine the user's visual attention area.

[0074] In some embodiments, S103 includes:

[0075] A second connecting line segment is determined on the image to be detected, connecting the midpoint of the first connecting line segment and the center point of the pupil, wherein there are two included angles between the first connecting line segment and the second connecting line segment at the midpoint of the first connecting line segment;

[0076] If the absolute value of the difference between the two included angles is less than an angle threshold, the user's visual attention area is determined to be the central area of ​​the display screen of the electronic device.

[0077] If the absolute value of the difference between the two included angles is greater than or equal to an angle threshold, it is determined that the user's visual attention area is not the central area of ​​the display screen of the electronic device.

[0078] In this embodiment, since the user is assuming they are facing the display screen of the electronic device when using it, the visual attention area of ​​the user can be determined by capturing an image of at least one eye of the user and then analyzing the positional relationship between the midpoint of the first connecting line segment of the eye and the center point of the pupil. Taking one eye in the image as an example, the midpoint of the first connecting line segment of that eye can be used as a reference point. By analyzing the relative positional relationship between the center point of the pupil and the midpoint of the line segment, the relationship between the user's visual attention area and the display screen can be determined.

[0079] As an optional embodiment, the midpoint of a line segment in the image to be detected and the center point of the pupil can be connected to obtain a second connecting line segment. The two endpoints of the second connecting line segment are the midpoint of the line segment and the center point of the pupil, respectively. The second connecting line segment intersects the first connecting line segment at the midpoint of the line segment, and the second connecting line segment does not extend beyond the midpoint of the intersecting line segment. In this intersection manner, the first connecting line segment and the second connecting line segment have two included angles at the midpoint of the intersecting line segment.

[0080] The relationship between the user's visual focus area and the display screen can be determined based on the absolute value of the difference between the two included angles.

[0081] Specifically, an angle threshold can be set. If the absolute value of the difference between two included angles is less than the angle threshold, the two included angles are considered to be not significantly different, and the user's pupils are considered to be roughly in the center of the eyes. Since the user is facing the display screen of the electronic device, the user's visual focus area is considered to be the central area of ​​the display screen. If the absolute value of the difference between two included angles is greater than or equal to the angle threshold, the two included angles are considered to be significantly different, and the user's pupils are considered to be roughly in the edge of the eyes. The user's visual focus area is considered to be not the central area of ​​the display screen.

[0082] As shown in Figure 3, if the absolute value of the angle difference between ∠amp and ∠bmp is less than the angle threshold, then the user's visual attention area is considered to be the central area of ​​the electronic device's display screen; if the absolute value of the angle difference between ∠amp and ∠bmp is greater than or equal to the angle threshold, then the user's visual attention area is considered not to be the central area of ​​the electronic device's display screen.

[0083] In this embodiment, the difference between the two angles formed by the corner of the eye, the midpoint of the line segment, and the center of the pupil can be used to determine whether the user's visual attention area is the central area of ​​the display screen. This method of detecting the visual attention area is relatively simple and can therefore efficiently determine the user's visual attention area.

[0084] S104, if the visual attention area is not the central area of ​​the display screen of the electronic device, output a prompt message and reacquire the image to be detected, the prompt message being used to instruct the human eye to focus on the central area of ​​the display screen.

[0085] In this embodiment, when a user performs visual attention-related detections such as pupil distance detection and iris recognition using an electronic device, a user interface is typically generated in the center of the electronic device's display screen to attract the user's attention.

[0086] After generating the user interface, the user's visual attention area can be determined by taking an image of the user's at least one eye and determining the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil.

[0087] If the user's visual focus area is the center of the electronic device's display screen, then interpupillary distance detection and iris recognition can be performed directly. Therefore, a guiding video is played in the center of the display screen. By responding to the guiding video and performing corresponding actions, the user can accurately complete interpupillary distance detection and adjustment, as well as iris recognition.

[0088] If the user's visual focus area is not in the center of the mixed reality device's display screen, it means that interpupillary distance detection and iris recognition cannot be performed. In this case, a prompt message can be output to the user, prompting them to adjust their focus area and gaze at the center of the display screen. This prompt message can be text displayed on the screen or an audio prompt played by the mixed reality device.

[0089] After outputting the prompt message, the image to be detected can be reacquired, and the user's visual attention area can be determined again through the image to determine whether the user has responded to the prompt message and shifted the visual attention area to the center of the display screen.

[0090] In the visual attention region detection method of this application, a user's image to be detected can be acquired, and the positions of the two corner points of at least one eye and the center point of the pupil in the image can be determined. Then, the user's visual attention region is determined based on the midpoint of the first connecting line segment formed by the two corner points and the position of the center point of the pupil. Since the relative position of the center point of the pupil in the eye can characterize the user's visual attention region, and the midpoint of the connecting line segment can be approximated as the center of the human eye, the midpoint of the line segment can be used as a reference point, and the user's actual visual attention region is detected based on the positional relationship between the center point of the pupil and the midpoint of the line segment. If the user's visual attention region is the center area of ​​the display screen, subsequent steps can proceed normally; if the user's visual attention region is not the center area of ​​the display screen, it can be considered that the user has not cooperated correctly with the guidance, and a prompt message can be output to the user to prompt the user to shift the area of ​​attention to the center area of ​​the display screen. In this way, it can be ensured that the area of ​​attention of the user is the center area of ​​the display screen when performing pupil distance detection and iris recognition, thereby improving the accuracy of recognition and detection and enhancing the user experience.

[0091] In some embodiments, after S103, the method further includes:

[0092] If the electronic device is a mixed reality device, and the mixed reality device is being worn and the user's visual focus area is not the center area of ​​the display screen of the mixed reality device, then a prompt message is output, which is used to prompt the user to focus on the center area of ​​the display screen.

[0093] In this embodiment, a Mixed Reality (MR) device refers to a device that combines the real world with the virtual world. After the user turns on and wears the mixed reality device, they can see the real world and simultaneously see virtual objects on the display screen of the mixed reality device. The mixed reality device is typically a head-mounted electronic device. After the user turns on and wears the mixed reality device, the MR device first needs to detect the user's interpupillary distance (IPD) and perform iris recognition. Based on the IPD detection result, the position of the MR device's display screen is adjusted, and based on the iris recognition result, the user can quickly log in and personalize settings. During the IPD detection or iris recognition process, the user is required to look directly at the center area of ​​the MR device's screen, because only when the user looks directly at the center area of ​​the MR device's display screen can the accuracy of the IPD detection and iris recognition results be guaranteed.

[0094] Therefore, when the hybrid display device is turned on and worn, it can be assumed that the user is wearing and using the hybrid display device. At this time, the hybrid display device can capture the user's image to be detected. Using the midpoint of the line segment connecting the two corners of the eyes in the image to be detected as a reference, the user's visual attention area is determined by the relative positional relationship between the pupil center point and the midpoint of the line segment.

[0095] If the user's visual focus area is not in the center of the mixed reality device's display screen, it means that interpupillary distance detection and iris recognition cannot be performed. In this case, a prompt message can be output to the user, prompting them to adjust their focus area to the center of the display screen. This prompt message can be text displayed on the screen or an audio prompt played by the mixed reality device.

[0096] In this way, when a user first puts on the mixed reality device, the system can detect the user's visual focus area. If the user's visual focus area is not in the center of the mixed reality device's display screen, a prompt message will be output to remind the user to focus on the center of the display screen, so as to ensure that the user can use the mixed reality device correctly.

[0097] In some embodiments, after S103, the method further includes:

[0098] When the electronic device is a mixed reality device, if the mixed reality device is being worn and the user's visual focus area is the center area of ​​the display screen of the mixed reality device, a guide video is played on the display screen. The guide video is used to detect the user's interpupillary distance and perform iris recognition on the user.

[0099] In this embodiment, after the user turns on and wears the hybrid display device, it can be assumed that the user has worn and used the hybrid display device. At this time, the hybrid display device can capture the user's image to be detected. Taking the midpoint of the line segment connecting the two corners of the eyes in the image to be detected as a reference, the user's visual attention area is determined by the relative positional relationship between the pupil center point and the midpoint of the line segment.

[0100] If the user's visual focus area is the center of the mixed reality device's display screen, then pupillary distance detection and iris recognition can be performed directly on the user. Therefore, a guide video is played in the center of the display screen. By responding to the guide video and performing corresponding operations, the user can accurately complete pupillary distance detection and adjustment, as well as iris recognition.

[0101] By using the above method, it can be ensured that the guiding video is played to perform pupil distance detection and iris recognition only when the user is wearing the mixed reality device and the user's visual focus area is the center area of ​​the display screen, thus ensuring the accuracy of pupil distance detection and iris recognition.

[0102] Figure 4 is a schematic diagram of a visual attention region detection device according to another embodiment of this application. As shown in Figure 4, the visual attention region detection device may include:

[0103] The first determining module 401 is used to determine the two corner points of the eye and the center point of the pupil of at least one eye in the image to be detected;

[0104] The second determining module 402 is used to determine a first connecting line segment of at least one eye in the image to be detected, wherein the first connecting line segment is used to connect the two corner points of the eye;

[0105] The third determining module 403 is used to determine the user's visual attention area based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil;

[0106] The output module 404 is configured to output a prompt message and reacquire the image to be detected when the visual attention area is not the central area of ​​the display screen of the electronic device. The prompt message is used to instruct the human eye to focus on the central area of ​​the display screen.

[0107] In this application, a user's image to be detected can be acquired, and the positions of the two corner points of at least one eye and the center point of the pupil in the image can be determined. Then, the user's visual attention area is determined based on the midpoint of the first connecting line segment formed by the two corner points and the position of the center point of the pupil. Since the relative position of the center point of the pupil in the eye can characterize the user's visual attention area, and the midpoint of the connecting line segment can be approximated as the center of the human eye, the midpoint of the line segment can be used as a reference point. The user's actual visual attention area is detected based on the positional relationship between the center point of the pupil and the midpoint of the line segment. If the user's visual attention area is the center area of ​​the display screen, subsequent steps can proceed normally; if the user's visual attention area is not the center area of ​​the display screen, it can be considered that the user has not cooperated correctly with the guidance, and a prompt message can be output to the user to prompt the user to shift the area of ​​attention to the center area of ​​the display screen. In this way, it can be ensured that the area of ​​attention of the user is the center area of ​​the display screen when performing pupil distance detection and iris recognition, thereby improving the accuracy of recognition and detection and enhancing the user experience.

[0108] In another alternative example, the third determining module 403 includes:

[0109] The first determining unit is configured to determine the user's visual attention area as the central area of ​​the display screen of the electronic device when the distance between the midpoint of the line segment and the center point of the pupil is less than or equal to a distance threshold.

[0110] The second determining unit is configured to determine that the user's visual attention area is not the central area of ​​the display screen when the distance between the midpoint of the line segment and the center point of the pupil is greater than the distance threshold.

[0111] In another alternative example, the third determining module 403 further includes:

[0112] The third determining unit is used to determine a second connecting line segment on the image to be detected, connecting the midpoint of the line segment and the center point of the pupil, wherein there are two included angles between the first connecting line segment and the second connecting line segment at the midpoint of the line segment;

[0113] The fourth determining unit is used to determine the user's visual attention area as the central area of ​​the display screen of the electronic device when the absolute value of the difference between the two included angles is less than an angle threshold.

[0114] The fifth determining unit is used to determine that the user's visual attention area is not the central area of ​​the display screen of the electronic device when the absolute value of the difference between the two included angles is greater than or equal to the angle threshold.

[0115] In another alternative example, the first determining module 401 further includes:

[0116] The first detection unit is used to determine the upper eyelid edge and lower eyelid edge of at least one eye in the image to be detected by using an edge detection algorithm;

[0117] The fitting unit is used to determine, through curve fitting, a first curve corresponding to the upper eyelid edge and a second curve corresponding to the lower eyelid edge in the image to be detected;

[0118] The sixth determining unit is used to determine the two intersection points of the first curve and the second curve as the two corner points of the eye.

[0119] In another alternative example, the first determining module 401 further includes:

[0120] The second detection unit is used to determine the pupil edge of at least one eye in the image to be detected by an edge detection algorithm, and to obtain multiple pupil edge feature points on the pupil edge;

[0121] The seventh determining unit is used to determine the pupil center point based on a plurality of pupil edge feature points.

[0122] In another alternative example, the seventh determining unit is specifically used for:

[0123] The geometric midpoint of the plurality of pupil edge feature points is determined as the pupil center point, or,

[0124] The point whose sum of distances to the multiple pupil edge feature points is the smallest is determined as the pupil center point.

[0125] The detection device for the visual attention area in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific type of device.

[0126] The visual attention region detection device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit it.

[0127] The visual attention region detection device provided in this application embodiment can implement the various processes implemented in the method embodiment of FIG1. ​​To avoid repetition, it will not be described again here.

[0128] Optionally, as shown in FIG5, this application embodiment also provides an electronic device 100, including a processor 110, a memory 119, and a program or instructions stored in the memory 119 and executable on the processor 110. When the program or instructions are executed by the processor 110, they implement the various processes of the above-described visual attention area detection method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0130] Please refer to Figure 6, which is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of this application. The electronic device 100 includes, but is not limited to, components such as: a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0131] Those skilled in the art will understand that the electronic device 120 may also include a power supply (such as a battery) for powering various components. The power supply can be logically connected to the processor 120 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in Figure 6 does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0132] The processor 120 is used to determine the two corner points of the eye and the center point of the pupil of at least one eye in the image to be detected.

[0133] Processor 120 is configured to determine a first connecting line segment for at least one eye in the image to be detected, the first connecting line segment being used to connect the two corner points of the eye;

[0134] Processor 120 is configured to determine the user's visual attention area based on the positional relationship between the midpoint of the first connecting line segment of the at least one eye and the center point of the pupil;

[0135] The display unit 126 is configured to output a prompt message and reacquire the image to be detected when the visual attention area is not the central area of ​​the display screen of the electronic device. The prompt message is used to instruct the human eye to focus on the central area of ​​the display screen.

[0136] In this application, a user's image to be detected can be acquired, and the positions of the two corner points of at least one eye and the center point of the pupil in the image can be determined. Then, the user's visual attention area is determined based on the midpoint of the first connecting line segment formed by the two corner points and the position of the center point of the pupil. Since the relative position of the center point of the pupil in the eye can characterize the user's visual attention area, and the midpoint of the connecting line segment can be approximated as the center of the human eye, the midpoint of the line segment can be used as a reference point. The user's actual visual attention area is detected based on the positional relationship between the center point of the pupil and the midpoint of the line segment. If the user's visual attention area is the center area of ​​the display screen, subsequent steps can proceed normally; if the user's visual attention area is not the center area of ​​the display screen, it can be considered that the user has not cooperated correctly with the guidance, and a prompt message can be output to the user to prompt the user to shift the area of ​​attention to the center area of ​​the display screen. In this way, it can be ensured that the area of ​​attention of the user is the center area of ​​the display screen when performing pupil distance detection and iris recognition, thereby improving the accuracy of recognition and detection and enhancing the user experience.

[0137] In another alternative example, the processor 120 is also used for:

[0138] If the distance between the midpoint of the line segment and the center point of the pupil is less than or equal to a distance threshold, the user's visual attention area is determined to be the central area of ​​the display screen of the electronic device.

[0139] If the distance between the midpoint of the line segment and the center point of the pupil is greater than the distance threshold, it is determined that the user's visual attention area is not the central area of ​​the display screen.

[0140] In another alternative example, processor 120 is also used for:

[0141] A second connecting line segment is determined on the image to be detected, connecting the midpoint of the first connecting line segment and the center point of the pupil, wherein there are two included angles between the first connecting line segment and the second connecting line segment at the midpoint of the first connecting line segment;

[0142] If the absolute value of the difference between the two included angles is less than an angle threshold, the user's visual attention area is determined to be the central area of ​​the display screen of the electronic device.

[0143] If the absolute value of the difference between the two included angles is greater than or equal to the angle threshold, it is determined that the user's visual attention area is not the central area of ​​the display screen of the electronic device.

[0144] In another alternative example, the processor 120 is also used for:

[0145] The upper and lower eyelid edges of at least one eye in the image to be detected are determined using an edge detection algorithm.

[0146] The first curve corresponding to the upper eyelid edge and the second curve corresponding to the lower eyelid edge in the image to be detected are determined by curve fitting.

[0147] The two intersection points of the first curve and the second curve are determined as the two corner points of the eye.

[0148] In another alternative example, the processor 120 is also used for:

[0149] The edge of the pupil of at least one eye in the image to be detected is determined by an edge detection algorithm, and multiple pupil edge feature points are obtained on the pupil edge.

[0150] The pupil center point is determined based on multiple pupil edge feature points.

[0151] In another alternative example, the processor 120 is also used for:

[0152] The geometric midpoint of the plurality of pupil edge feature points is determined as the pupil center point, or,

[0153] The point whose sum of distances to the multiple pupil edge feature points is the smallest is determined as the pupil center point.

[0154] It should be understood that, in this embodiment, the input unit 124 may include a graphics processing unit (GPU) 1241 and a microphone 1242. The GPU 1241 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 126 may include a display panel 1261, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 127 includes at least one of a touch panel 1271 and other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0155] The memory 129 can be used to store software programs and various data. The memory 129 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 129 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 129 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0156] Processor 120 may include one or more processing units; optionally, processor 120 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 120.

[0157] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described visual attention area detection method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0158] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0159] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described visual attention area detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0160] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0161] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described visual attention region detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0164] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for detecting a visual attention region, executed by an electronic device, the method comprising: determining two canthus points and a pupil center point of at least one eye in a to-be-detected image; determining a first connection line segment of the at least one eye in the to-be-detected image, the first connection line segment being used for connecting the two canthus points; determining a visual attention region of a user according to a positional relationship between a line segment midpoint of the first connection line segment of the at least one eye and the pupil center point; in a case where the visual attention region is not a central region of a display screen of the electronic device, outputting prompt information and re-acquiring the to-be-detected image, the prompt information being used for instructing a human eye to gaze at the central region of the display screen. The determining a visual attention region of a user according to a positional relationship between a line segment midpoint of the first connection line segment of the at least one eye and the pupil center point comprises: in a case where a distance between the line segment midpoint and the pupil center point is less than or equal to a distance threshold, determining that the visual attention region of the user is the central region of the display screen of the electronic device; in a case where the distance between the line segment midpoint and the pupil center point is greater than the distance threshold, determining that the visual attention region of the user is not the central region of the display screen. The determining a visual attention region of a user according to a positional relationship between a line segment midpoint of the first connection line segment of the at least one eye and the pupil center point comprises: determining a second connection line segment connecting the line segment midpoint and the pupil center point on the to-be-detected image, wherein there are two included angles between the first connection line segment and the second connection line segment at the line segment midpoint; in a case where an absolute value of a difference between the two included angles is less than an angle threshold, determining that the visual attention region of the user is the central region of the display screen of the electronic device; in a case where the absolute value of the difference between the two included angles is greater than or equal to the angle threshold, determining that the visual attention region of the user is not the central region of the display screen of the electronic device. The determining two canthus points of at least one eye in a to-be-detected image comprises: determining an upper eyelid edge and a lower eyelid edge of the at least one eye in the to-be-detected image by an edge detection algorithm; determining a first curve corresponding to the upper eyelid edge and a second curve corresponding to the lower eyelid edge in the to-be-detected image by curve fitting; and determining two intersection points of the first curve and the second curve as the two canthus points. The determining a pupil center point of at least one eye in a to-be-detected image comprises: determining a pupil edge of the at least one eye in the to-be-detected image by an edge detection algorithm, and acquiring a plurality of pupil edge feature points on the pupil edge; and determining the pupil center point according to the plurality of pupil edge feature points.

2. The method of claim 1, wherein, The determining the pupil center point according to the plurality of pupil edge feature points comprises: determining a geometric midpoint of the plurality of pupil edge feature points as the pupil center point, or determining a point with a smallest sum of distances to each of the plurality of pupil edge feature points as the pupil center point. ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ 4. The method of claim 1, wherein, ​ ​ ​ ​ 5. The method of claim 1, wherein, ​ ​ ​ 6. The method of claim 5, wherein, ​ ​ ​ 7. An apparatus for detecting a visual attention area, comprising: a first determining module configured to determine two canthus points and a pupil center point of at least one eye in a to-be-detected image; a second determining module configured to determine a first connection line segment of the at least one eye in the to-be-detected image, the first connection line segment being configured to connect the two canthus points; a third determining module configured to determine a visual attention area of a user according to a positional relationship between a line segment midpoint of the first connection line segment of the at least one eye and the pupil center point; an output module configured to output a prompt information and reacquire the to-be-detected image in a case that the visual attention area is not a center area of a display screen of the electronic device, the prompt information being configured to indicate that a human eye gazes at the center area of the display screen.

8. The apparatus of claim 7, wherein, The third determining module comprises: a first determining unit configured to determine that the visual attention area of the user is the center area of the display screen of the electronic device in a case that a distance between the line segment midpoint and the pupil center point is less than or equal to a distance threshold value; a second determining unit configured to determine that the visual attention area of the user is not the center area of the display screen in a case that the distance between the line segment midpoint and the pupil center point is greater than the distance threshold value.

9. The apparatus of claim 7, wherein, The third determining module comprises: a third determining unit configured to determine a second connection line segment connecting the line segment midpoint and the pupil center point on the to-be-detected image, wherein two included angles exist between the first connection line segment and the second connection line segment at the line segment midpoint; a fourth determining unit configured to determine that the visual attention area of the user is the center area of the display screen of the electronic device in a case that an absolute value of a difference between the two included angles is less than an angle threshold value; a fifth determining unit configured to determine that the visual attention area of the user is not the center area of the display screen of the electronic device in a case that the absolute value of the difference between the two included angles is greater than or equal to the angle threshold value.

10. An electronic device comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions being executed by the processor to implement the steps of the method for detecting a visual attention area according to any one of claims 1-6.

11. A readable storage medium, the readable storage medium storing programs or instructions, the programs or instructions being executed by a processor to implement the steps of the method for detecting a visual attention area according to any one of claims 1-6.

12. A computer program product stored in a storage medium, the program product being executed by at least one processor to implement the steps of the method for detecting a visual attention area according to any one of claims 1-6.

13. A chip comprising a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to run programs or instructions to implement the steps of the method for detecting a visual attention area according to any one of claims 1-6.

14. An electronic device configured to perform the steps of the method of visual attention region detection of any one of claims 1-6.

Images