Visual focus-based interface display method, device, equipment and storage medium

Abstract

The application relates to the field of artificial intelligence and discloses an interface display method and device based on a visual focus, equipment and a storage medium. The method comprises the following steps: collecting an eye image of a user, identifying an eyeball region in the eye image of the user, obtaining an eyeball contour image corresponding to the eyeball region, detecting a visual state of the user based on the eyeball contour image, obtaining a visual state category, detecting a visual interference item in the eyeball contour image based on the visual state category, obtaining a detection result, if the detection result is that there is no visual interference item, positioning an eyeball focus region of the user in a display interface by using a preset focus positioning method according to the eyeball contour image, calculating a current display parameter value of the display interface based on the eyeball focus region, and adjusting the display interface according to the current display parameter value. The application improves the privacy of viewing private data in public places and reduces the risk of exposure of private data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an interface display method and device based on a visual focus point, an equipment and a storage medium. BACKGROUND

[0002] With the continuous development of technology, mobile terminals such as mobile phones and tablet computers have become an indispensable part of people's life and work. People are used to using mobile terminals to read, entertain, and even work at any time. However, especially in public places, when checking data information on the terminal, the data information displayed on the terminal is inevitably and easily snooped by others, which leads to the leakage of private information and highly confidential information.

[0003] At present, the measures taken are to paste an anti-peeping protective film on the surface of the display screen of the terminal to change the corresponding visual angle on the display screen by using the physical optical polarization principle technology, thereby preventing others from snooping the data information on the display screen from the side of the terminal. However, the existing anti-peeping protective film is generally used only once, and the corresponding anti-peeping effect is different for protective films of different materials. Therefore, for different use scenarios, users can only replace different materials of protective films to meet different needs. That is, the existing anti-peeping terminal polarization display method cannot adapt to different scenarios. SUMMARY

[0004] The main purpose of the present application is to solve the technical problem that the existing anti-peeping terminal polarization display method cannot adapt to different scenarios.

[0005] The first aspect of the present application provides an interface display method based on a visual focus point, comprising: collecting a user's eye image and identifying an eyeball region in the user's eye image to obtain an eyeball contour image corresponding to the eyeball region; performing user visual state detection based on the eyeball contour image to obtain a visual state category, and performing visual interference item detection on the eyeball contour image based on the visual state category to obtain a detection result; if the detection result is that there is no visual interference item, then according to the eyeball contour image, a pre-set focus point positioning method is used to position the eyeball focus point region of the user in the display interface; based on the eyeball focus point region, the current display parameter value of the display interface is calculated, and the display interface is adjusted according to the current display parameter value.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, identifying the eyeball area in the user's eye image and obtaining an eyeball contour image corresponding to the eyeball area includes: detecting each eye key point in the user's eye image, and connecting each eye key point in sequence to obtain a mask corresponding to the eyeball area; identifying the white of the eye area in the user's eye image based on each eye key point and preset auxiliary identification information; combining the mask and the white of the eye area, and segmenting the user's eye image based on the result of the combination to obtain an eyeball contour image corresponding to the eyeball area.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, identifying the white of the eye area in the user's eye image based on each eye key point and preset auxiliary identification information includes: calculating the relative coordinate mean of each eye key point in the user's eye image, and determining the eyeball reference midpoint based on the relative coordinate mean; extracting the eyeball texture information in the preset auxiliary identification information, and calculating the white of the eye area in the user's eye image based on the eyeball texture information and the eyeball reference midpoint.

[0008] Optionally, in a third implementation of the first aspect of the present invention, user visual state detection is performed based on an eyeball contour image to obtain a visual state category, including: identifying the current user eye movement based on the eyeball contour image, and identifying the original position information of the current eyeball contour in the user eye image; calculating the size ratio of a preset eyeball contour reference image and the eyeball contour image, and determining the position adjustment information of the eyeball contour based on the size ratio; identifying the focus area of ​​the user vision in the display interface based on the current user eye movement, and mapping the current eyeball contour to the display interface based on the position adjustment information; calculating the relative distance between the eyeball contour and the focus area in the display interface, and determining whether the relative distance is greater than a preset distance threshold; if greater, determining that the visual state category corresponding to the eyeball contour image is a strabismus category, otherwise it is an emmetropia category.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the visual state category also includes a monocular vision category and a binocular vision category. Based on the visual state category, visual interference item detection is performed on the eyeball contour image, and the detection results include: judging that the visual state category is a strabismus category or an emmetropia category, and judging that the visual state category is a monocular vision category or a binocular vision category; if the visual state category is an emmetropia category or a binocular vision category, then determining that the detection result of the visual interference item of the eyeball contour image is that there is no visual interference item, otherwise it is that there is a visual interference item.

[0010] Optionally, in the fifth implementation method of the first aspect of the present invention, after visual interference item detection is performed on the eyeball contour image based on the visual state category and the detection result is obtained, it also includes: if the detection result is that there is a visual interference item and there is a monocular vision category, then according to the eyeball contour reference image and the position adjustment information, the simulated position information of the other monocular is calculated; according to the size ratio and the simulated position information, the simulated position information is adjusted, and according to the adjusted simulated position information, the eyeball contour image is symmetrically rendered.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, calculating the current display parameter value of the display interface based on the eye focus area includes: calculating the current viewing angle of the eye focus area in the display interface, and obtaining the current initial display parameter value of the display interface; according to the viewing angle, adjusting the initial display parameter value of the eye focus area in the display interface to obtain the current display parameter value of the display interface after adjustment.

[0012] The second aspect of the present invention provides an interface display device based on visual focus, including: an identification module for collecting a user's eye image, identifying the eyeball area in the user's eye image, and obtaining an eyeball contour image corresponding to the eyeball area; a detection module for detecting the user's visual state based on the eyeball contour image, obtaining a visual state category, and performing visual interference item detection on the eyeball contour image based on the visual state category to obtain a detection result; a positioning module for locating the user's eyeball focus area in the display interface based on the eyeball contour image using a preset focus positioning method if the detection result is that there is no visual interference item; and an adjustment module for calculating the current display parameter value of the display interface based on the eyeball focus area, and adjusting the display interface according to the current display parameter value.

[0013] Optionally, in a first implementation of the second aspect of the present invention, the recognition module includes: a connection unit, used to detect each eye key point in the user's eye image, and connect each eye key point in sequence to obtain a mask corresponding to the eyeball area; a first recognition unit, used to identify the white of the eye area in the user's eye image based on each eye key point and preset auxiliary recognition information; a segmentation unit, used to combine the mask and the white of the eye area, and segment the user's eye image based on the result of the combination to obtain an eyeball contour image corresponding to the eyeball area.

[0014] Optionally, in a second implementation of the second aspect of the present invention, the first recognition unit is also used to: calculate the relative coordinate mean of each eye key point in the user's eye image, and determine the eyeball reference midpoint based on the relative coordinate mean; extract the eyeball texture information from the preset auxiliary recognition information, and calculate the white of the eye area in the user's eye image based on the eyeball texture information and the eyeball reference midpoint.

[0015] Optionally, in a third implementation form of the second aspect of the present application, the detection module comprises: a second identification unit, configured to identify a current eye movement of the user based on the eyeball contour image, and identify original position information of the current eyeball contour in the user eye image; a first calculation unit, configured to calculate a size ratio of the preset eyeball contour reference image and the eyeball contour image, and determine position adjustment information of the eyeball contour according to the size ratio; a mapping unit, configured to identify a focus area of the user vision in the display interface according to the current eye movement of the user, and map the current eyeball contour to the display interface based on the position adjustment information; a second calculation unit, configured to calculate a relative distance between the eyeball contour and the focus area in the display interface, and determine whether the relative distance is greater than a preset distance threshold; and a first determination unit, configured to determine that the visual state category of the eyeball contour image is the strabismus category if the relative distance is greater than the preset distance threshold, and otherwise, the orthophoric category.

[0016] Optionally, in a fourth implementation form of the second aspect of the present application, the visual state category further comprises a monocular vision category and a binocular vision category, and the detection module comprises: a discrimination unit, configured to determine whether the visual state category is the strabismus category or the orthophoric category, and determine whether the visual state category is the monocular vision category or the binocular vision category; and a second determination unit, configured to determine that the detection result of the visual interference item of the eyeball contour image is that there is no visual interference item if the visual state category is the orthophoric category and the binocular vision category, and otherwise, there is a visual interference item.

[0017] Optionally, in a fifth implementation form of the second aspect of the present application, the interface display device based on the visual focus further comprises a rendering module, configured to: if the detection result is that there is a visual interference item and there is a monocular vision category, calculate simulation position information of another monocular eye according to the eyeball contour reference image and the position adjustment information; adjust the simulation position information according to the size ratio and the simulation position information, and perform symmetric rendering of the eyeball contour on the eyeball contour image according to the adjusted simulation position information.

[0018] Optionally, in a sixth implementation form of the second aspect of the present application, the adjustment module comprises: a third calculation unit, configured to calculate a current visible angle of the eyeball focus area in the display interface, and obtain an initial display parameter value of the display interface; and an adjustment unit, configured to adjust the initial display parameter value of the eyeball focus area in the display interface according to the visible angle, to obtain a current display parameter value of the adjusted display interface.

[0019] The third aspect of the present application provides an interface display device based on a visual focus, comprising a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to enable the interface display device based on the visual focus to perform the interface display method based on the visual focus.

[0020] A fourth aspect of the present invention provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is caused to execute the above-mentioned interface display method based on visual focus.

[0021] In the technical solution provided by the present invention, the eyeball area in the user's eye image is identified by collecting the user's eye image, and then the identified eyeball contour image is used to detect the user's visual state to obtain the visual state category, and then the eyeball contour image is used to detect visual interference items to determine whether there are no visual interference items in the eyeball contour image. When there are no interference items, the user's eye focus area in the display interface can be further located to calculate the current display parameter value of the display interface, and the display interface is adjusted according to the current display parameter value, thereby realizing the automation of the polarized display of the anti-peeping terminal to adapt to different scenarios, while improving the confidentiality of viewing private data in public places and reducing the risk of private data exposure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of a first embodiment of the interface display method based on visual focus of the present invention;

[0023] Figure 2 Schematic diagram of a second embodiment of the interface display method based on visual focus of the present invention;

[0024] Figure 3 Schematic diagram of a third embodiment of the interface display method based on visual focus of the present invention;

[0025] Figure 4 A schematic diagram of an embodiment of an interface display device based on visual focus according to the present invention;

[0026] Figure 5 Schematic diagram of another embodiment of the interface display device based on visual focus of the present invention;

[0027] Figure 6 Schematic diagram of an embodiment of an interface display device based on visual focus of the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present invention provide a method, apparatus, device, and storage medium for displaying an interface based on visual focus. The method collects a user's eye image, identifies the eyeball area in the user's eye image, and obtains an eyeball contour image corresponding to the eyeball area. The method detects the user's visual state based on the eyeball contour image to obtain a visual state category, and based on the visual state category, detects visual interference items on the eyeball contour image to obtain a detection result. If the detection result shows that no visual interference items exist, the method locates the user's eyeball focus area in the display interface using a preset focus positioning method based on the eyeball contour image. Based on the eyeball focus area, the method calculates the current display parameter value of the display interface, and adjusts the display interface according to the current display parameter value. The present invention improves the confidentiality of viewing private data in public places and reduces the risk of private data exposure.

[0029] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0030] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the interface display method based on visual focus in the embodiment of the present invention includes:

[0031] 101. Collect an eye image of the user, identify an eyeball region in the eye image of the user, and obtain an eyeball contour image corresponding to the eyeball region;

[0032] It is understandable that the execution subject of the present invention can be an interface display device based on visual focus, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0033] In this embodiment, when the user is viewing the content preset as private data in each APP or web page, the front camera of the device is turned on with permission to use the front camera of the device to capture the user's eye image in real time, so as to adjust the color intensity, grayscale value, etc. of the mobile phone screen accordingly to the user's visual convergence, thereby changing the visualization angle of the display interface, so that others cannot peek at the content of the private data currently displayed on the display interface.

[0034] First, after collecting the user's eye image, the eye region of the user in the image is identified, and the corresponding eye contour image is cropped. When the surrounding light source of the display interface is strong, it can assist in positioning the area where the current user focuses on the eye. When identifying the eye region of the user's eye image, an image recognition model can be used for identification, key points of the eye can be used for identification, or texture information of the eye can be used for identification, which is not limited here.

[0035] 102. Based on the eye contour image, the user's visual state is detected to obtain a visual state category, and based on the visual state category, the eye contour image is detected for visual interference items to obtain a detection result;

[0036] In this embodiment, for the state of the user using the device to look at the display interface that may occur in the real scene, the user's visual state is detected based on the eye contour image to determine the visual state category of the current user when looking at the display interface, such as looking at the display interface obliquely or looking at the display interface normally, and looking at the display interface with one eye or looking at the display interface with two eyes, etc., wherein looking at the display interface with two eyes normally is the user's visual category in the normal state.

[0037] 103. If the detection result is that there is no visual interference item, then according to the eye contour image, a pre-set focal point positioning method is used to position the eye focal point area of the user in the display interface;

[0038] In this embodiment, the user's eye movement can include the user's visual focal point staying on the screen and / or the blinking action. When identifying the focal point area of the user's vision in the display interface, a visual line tracking method such as pupil tracking, pupil-corneal reflection, and Purkinje imaging can be used to determine.

[0039] In addition, 1) if the detection result is that there is a visual interference item and there is a monocular visual category, then according to the eye contour reference image and the position adjustment information, the simulation position information of the other monocular is calculated; 2) according to the size ratio and the simulation position information, the simulation position information is adjusted, and according to the adjusted simulation position information, the eye contour of the eye contour image is symmetrically rendered.

[0040] In this embodiment, in the eye contour image of the monocular visual category, only one eye contour is detected, and then the simulation position information of the other missing monocular is calculated according to the position of the eye contour in the eye contour reference image, such as calculating according to the relative distance and the size of the existing eye contour, so as to obtain the simulation position information of the position where the other missing monocular should be located, which can be represented in plane coordinates.

[0041] In this embodiment, after the simulated position information of the other missing monocular is calculated, the simulated position information is inversely scaled according to the size ratio calculated in the previous step, so as to map the position of the eyeball contour of the other missing monocular to the eyeball contour image, and then the eyeball contour of the existing monocular is symmetrically rendered to the position corresponding to the missing monocular according to the midpoint normal line of the distance between the existing monocular and the missing monocular.

[0042] 104. Calculate the display parameter value of the display interface according to the eyeball focal point area, and adjust the display interface according to the current display parameter value.

[0043] In this embodiment, by adjusting the display parameter value of the display interface, such as display brightness, gray value, and display proportion of three primary colors, the visual angle of the screen can be adjusted, wherein the visual angle refers to the angle between the vertical direction on the outer surface of the display interface, for example, the visual angle is 10°, that is, the user can clearly see the content displayed on the display interface within an angle of 15° deviated from the vertical direction, but if it exceeds 10°, only sporadic pixel points or even gray can be seen, so that others cannot see the content in the display interface, preventing others from peeping the content on the display interface, and achieving the protection of private data.

[0044] In the embodiment of the application, the eyeball region in the user's eye image is recognized by collecting the user's eye image, and then the recognized eyeball contour image is used for user visual state detection to obtain a visual state category, and then the eyeball contour image is used for visual interference item detection to determine whether there is no visual interference item in the eyeball contour image. If there is no interference item, the eyeball focal point area of the user in the display interface can be further located to calculate the current display parameter value of the display interface, and the display interface is adjusted according to the current display parameter value, so as to realize the automation of the anti-peeping terminal polarized display, adapt to different scenes, and improve the privacy of viewing private data in public places and reduce the risk of exposure of private data.

[0045] Please refer to Figure 2 The second embodiment of the interface display method based on visual focus in the embodiment of the application includes:

[0046] 201. Collect the user's eye image, detect each eye key point in the user's eye image, and connect each eye key point in sequence to obtain a mask corresponding to the eyeball region;

[0047] 202. Identify the white part of the eye in the user's eye image according to each eye key point and the preset auxiliary identification information;

[0048] 203. Combining the mask and the white eye area, and segmenting the user's eye image based on the combined result to obtain an eyeball contour image corresponding to the eyeball area;

[0049] In this embodiment, face detection can be performed on the user's eye image using a face detection algorithm to detect various eye key points in the user's facial image, where the eye key points include the key points of the inner and outer contours of the whites of the eyes (i.e., the key points of the pupils).

[0050] The white of the eye area can be represented by a mask. If the key points extracted from the user's eye image include 20 key points a1-a20 for the outer contour of the white of the eye and 10 key points b1-b10 for the inner contour of the white of the eye, then connecting a1-a20 and b1-b10 in sequence will generate a mask corresponding to the eyeball area, representing two areas: the white of the eye and the pupil. Furthermore, after generating the mask, blurring can be performed, such as Gaussian blurring, to create a transition area between the pupil and the white of the eye.

[0051] Alternatively, the white area of ​​the eye can be directly identified in the user's eye image. For example, in the auxiliary identification information, the white area of ​​the eye is represented as a light color, close to white or white, while the through hole is dark, such as blue, brown, or even black. Therefore, the user's eye image can be binarized. After the binarization process, the white area of ​​the eye can be converted to white, and the pupil area can be converted to black to obtain the white area of ​​the eye.

[0052] Finally, the resulting mask and the white of the eye area are combined, and the intersection or union of the two is used as the eyeball area, which is not specifically limited here. If the white of the eye area is not within the transition area between the masked white of the eye and the pupil, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area is within the transition area between the masked white of the eye and the pupil but does not exceed the transition area, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area exceeds the masked transition area, the inner edge of the transition area is used as the inner contour of the eyeball area.

[0053] 204. Based on the eyeball outline image, identify the current eye movement of the user, and identify the original position information of the current eyeball outline in the user's eye image;

[0054] 205. Calculate the size ratio of the preset eyeball outline reference image and the eyeball outline image, and determine the position adjustment information of the eyeball outline according to the size ratio;

[0055] 206. Identify the user's visual focus area on the display interface based on the current user eye movement, and map the current eyeball contour to the display interface based on the position adjustment information;

[0056] 207. Calculate the relative distance between the eyeball outline and the focus area in the display interface, and determine whether the relative distance is greater than a preset distance threshold;

[0057] 208. If it is greater than, determining that the visual state category corresponding to the eyeball outline image is strabismus, otherwise it is emmetropia, and performing visual interference item detection on the eyeball outline image based on the visual state category to obtain a detection result;

[0058] In this embodiment, after capturing the eye outline image, it is used, on the one hand, to identify the user's current eye movement and further identify the user's visual focus area on the display interface. On the other hand, it is used to calculate the size ratio between the eye outline image and a pre-captured and stored eye outline reference image, thereby proportionally adjusting the position information of the eye outline on the display interface. Because the eye outline reference image is captured according to a preset standard size, after calculating the size ratio between the two, it is possible to determine whether the eye outline image is larger or smaller than the standard size, and then proportionally reduce or enlarge it to obtain an image of the standard size. This is then combined with the original position information of the original eye outline image to project the eye outline onto the display interface.

[0059] Specifically, in the display area, when the relative distance between the eyeball outline and the focus area is larger, it can be determined that the angle at which the user's eyes are looking at the focus area is larger. When the relative distance between the two is within a preset distance threshold, the angle of the squint is small and can still be regarded as normal vision; when the relative distance between the two exceeds the preset threshold, the visual state when viewing the screen can be regarded as squint.

[0060] In addition, when further detecting visual interference items on the eyeball contour image, the following steps can be included: 2.6) determining whether the visual state category is a strabismus category or an emmetropia category, and determining whether the visual state category is a monocular vision category or a binocular vision category; 2.7) if the visual state category is an emmetropia category or a binocular vision category, then determining that the detection result of the visual interference items of the eyeball contour image is that there are no visual interference items, otherwise, there are visual interference items.

[0061] 209. If the detection result shows that there is no visual interference, locate the user's eye focus area on the display interface using a preset focus positioning method based on the eyeball outline image;

[0062] 210. Based on the eye focus area, calculate the current display parameter value of the display interface, and adjust the display interface according to the current display parameter value.

[0063] In an embodiment of the present invention, a mask of the eyeball area is generated and the white-of-the-eye area is identified through each eye key point in the user's eye image. The two are combined and segmented to obtain an eyeball contour image of the eyeball area. Then, the intersection area of ​​the user's vision in the display interface is further identified to determine whether the visual state category of the eyeball contour image is a strabismus category or an emmetropia category, thereby improving the recognition accuracy of the user's visual focus area and improving the accuracy of the adjustment when adjusting the display parameter values ​​of the display interface subsequently.

[0064] See also Figure 3 A third embodiment of the interface display method based on visual focus in the embodiment of the present invention includes:

[0065] 301. Collect the user's eye image, detect each eye key point in the user's eye image, and connect each eye key point in sequence to obtain a mask corresponding to the eyeball area;

[0066] 302. Calculate the relative coordinate mean of each eye key point in the user's eye image, and determine the eyeball reference midpoint based on the relative coordinate mean;

[0067] 303. Extract eyeball texture information from the preset auxiliary identification information, and calculate the white of the eye area in the user's eye image based on the eyeball texture information and the eyeball reference midpoint;

[0068] In this embodiment, a rectangular coordinate system is constructed in the entire user's eye image, and the relative coordinates of each eye key point in the index coordinate system are calculated. The average relative coordinates of each eye key point are further calculated, and the point where the relative coordinate average is located is used as the reference midpoint of the eyeball.

[0069] In this embodiment, the eyeball texture information may include one or more of the following: eyeball color, shape, and distributed appearance texture. Specifically, the eyeball color may be directly applied, and the eyeball reference midpoint may be used as a reference point to perform binarization processing on the user's eye image to obtain the white of the eye area. The specific application of one or more eyeball textures is not limited here.

[0070] 304. Combining the mask and the white of the eye area, and segmenting the user's eye image based on the combined result to obtain an eyeball contour image corresponding to the eyeball area;

[0071] 305. Performing a user visual state detection based on the eyeball outline image to obtain a visual state category, and performing a visual interference item detection on the eyeball outline image based on the visual state category to obtain a detection result;

[0072] 306. If the detection result shows that no visual interference exists, locate the user's eye focus area on the display interface using a preset focus positioning method based on the eyeball outline image;

[0073] 307. Calculate the current viewing angle of the eye focus area on the display interface, and obtain the current initial display parameter value of the display interface;

[0074] 308. Adjust the initial display parameter value of the eye focus area in the display interface according to the viewing angle, obtain the current display parameter value of the display interface after adjustment, and adjust the display interface according to the current display parameter value.

[0075] In this embodiment, the current display interface is displayed using initial display parameter values ​​adjusted by the system, such as screen brightness or grayscale values ​​corresponding to the brightness of the current environment. Here, the current viewing angle of the display interface can be calculated based on the size of the eye's focus area, such as 10° or 20°. Based on the calculated viewing angle, the intensity values ​​corresponding to the red channel, green channel, and blue channel in the current RGB color mode of the display interface are adjusted; and / or the current grayscale value of the display interface is adjusted based on the calculated viewing angle. The calculated color intensity value and grayscale value are used as display parameter values ​​to adjust the display parameter values ​​of the eye's focus area in the display interface.

[0076] In an embodiment of the present invention, the eye reference center is determined by the key points of the eye, and then the white of the eye area in the user's eye image is calculated in combination with the eye texture information. At the same time, when only one eye of the user is identified in the user's eye image, the position of the other eye is determined by calculating the simulated position information of the other eye, and using the simulated position information to perform symmetrical rendering of the eye contour, thereby reducing the impact of visual interference items on the recognition of the visual focus area and improving the accuracy of the adjustment of the display parameter values ​​in the subsequent display interface.

[0077] The above describes the interface display method based on visual focus in the embodiment of the present invention. The following describes the interface display device based on visual focus in the embodiment of the present invention. Figure 4 In one embodiment of the present invention, an interface display device based on visual focus includes:

[0078] The recognition module 401 is used to collect an eye image of the user, identify the eyeball area in the eye image of the user, and obtain an eyeball contour image corresponding to the eyeball area;

[0079] Detection module 402, configured to detect the user's visual state based on the eyeball outline image to obtain a visual state category, and perform visual interference item detection on the eyeball outline image based on the visual state category to obtain a detection result;

[0080] The positioning module 403 is configured to locate the user's eye focus area on the display interface using a preset focus positioning method based on the eye contour image if the detection result shows that no visual interference items exist;

[0081] The adjustment module 404 is configured to calculate the current display parameter value of the display interface based on the eye focus area, and adjust the display interface according to the current display parameter value.

[0082] In an embodiment of the present invention, an eyeball area in the user's eye image is identified by collecting a user's eye image, and then the identified eyeball contour image is used to detect the user's visual state to obtain a visual state category. Then, a visual interference item detection is performed on the eyeball contour image to determine whether there are no visual interference items in the eyeball contour image. If there are no interference items, the user's eye focus area in the display interface can be further located to calculate the current display parameter value of the display interface, and the display interface is adjusted according to the current display parameter value, thereby realizing the automation of the polarized display of the anti-peeping terminal to adapt to different scenarios, while improving the confidentiality of viewing private data in public places and reducing the risk of private data exposure.

[0083] See also Figure 5 Another embodiment of the interface display device based on visual focus in the embodiment of the present invention includes:

[0084] The recognition module 401 is used to collect an eye image of the user, identify the eyeball area in the eye image of the user, and obtain an eyeball contour image corresponding to the eyeball area;

[0085] Detection module 402, configured to detect the user's visual state based on the eyeball outline image to obtain a visual state category, and perform visual interference item detection on the eyeball outline image based on the visual state category to obtain a detection result;

[0086] The positioning module 403 is configured to locate the user's eye focus area on the display interface using a preset focus positioning method based on the eye contour image if the detection result shows that no visual interference items exist;

[0087] The adjustment module 404 is configured to calculate the current display parameter value of the display interface based on the eye focus area, and adjust the display interface according to the current display parameter value.

[0088] Specifically, the identification module 401 includes:

[0089] The connection unit 4011 is used to detect each eye key point in the user's eye image and connect each eye key point in sequence to obtain a mask corresponding to the eye area;

[0090] The first recognition unit 4012 is configured to recognize the white of the eye area in the user's eye image based on the eye key points and the preset auxiliary recognition information;

[0091] The segmentation unit 4013 is used to combine the mask and the white eye area, and segment the user's eye image based on the combination result to obtain an eyeball contour image corresponding to the eyeball area.

[0092] Specifically, the identification unit 4012 is further configured to:

[0093] Calculate the relative coordinate mean of each eye key point in the user's eye image, and determine the eyeball reference midpoint based on the relative coordinate mean;

[0094] The eyeball texture information in the preset auxiliary identification information is extracted, and the white of the eye area in the user's eye image is calculated based on the eyeball texture information and the eyeball reference midpoint.

[0095] Specifically, the detection module 402 includes:

[0096] The second recognition unit 4021 is used to recognize the current eye movement of the user based on the eyeball outline image, and recognize the original position information of the current eyeball outline in the user's eye image;

[0097] The first calculation unit 4022 is used to calculate the size ratio of the preset eyeball outline reference image and the eyeball outline image, and determine the position adjustment information of the eyeball outline according to the size ratio;

[0098] A mapping unit 4023 is configured to identify the focus area of ​​the user's vision on the display interface according to the current user's eye movement, and map the current eye contour to the display interface based on the position adjustment information;

[0099] The second calculation unit 4024 is used to calculate the relative distance between the eye contour and the focus area in the display interface, and determine whether the relative distance is greater than a preset distance threshold;

[0100] The first determining unit 4025 is configured to determine that the visual state category corresponding to the eyeball outline image is strabismus if , and emmetropia otherwise.

[0101] Specifically, the visual state category further includes a monocular vision category and a binocular vision category, and the detection module 402 further includes:

[0102] a determination unit 4026, configured to determine whether the visual state category is strabismus or emmetropia, and whether the visual state category is monocular vision or binocular vision;

[0103] The second determining unit 4027 is configured to determine that the detection result of the visual interference item of the eyeball outline image is that the visual interference item does not exist if the visual state category is the emmetropia category and the binocular vision category, and that the visual interference item exists otherwise.

[0104] Specifically, the interface display device based on the visual focus point further comprises a rendering module 405, configured to:

[0105] If the detection result is that there is a visual interference item and there is a monocular vision category, simulated position information of the other monocular is calculated according to the eye contour reference image and the position adjustment information;

[0106] According to the size ratio and the simulated position information, the simulated position information is adjusted, and the eye contour image is symmetrically rendered according to the adjusted simulated position information.

[0107] Specifically, the adjustment module 404 comprises:

[0108] A third calculation unit 4041 is configured to calculate a current visible angle of the eye focus area in the display interface, and obtain an initial display parameter value of the display interface;

[0109] An adjustment unit 4042 is configured to adjust the initial display parameter value of the eye focus area in the display interface according to the visible angle, to obtain an adjusted display parameter value of the display interface.

[0110] In the embodiment of the application, the mask of the eyeball region and the white eye region are generated through each eye key point in the user eye image, and the eyeball contour image of the eyeball region is obtained by combining and segmenting the two. Then, the intersection region of the user vision in the display interface is further identified to determine whether the visual state category of the eyeball contour image is a strabismus category or a normal vision category. In addition, the eyeball reference center is determined through the eye key point, and the white eye region in the user eye image is calculated in combination with the eyeball texture information. When only one eye of the user is identified in the user eye image, the simulated position information of the other monocular is calculated, and the eyeball contour is symmetrically rendered by using the simulated position information to determine the position of the other eyeball, thereby reducing the influence of the visual interference item on the recognition of the visual focus area and improving the accuracy of the subsequent display parameter value adjustment in the display interface.

[0111] The above Figure 4 And Figure 5 The interface display device based on the visual focus point in the embodiment of the application is described in detail from the perspective of modular functional entities, and the interface display device based on the visual focus point in the embodiment of the application is described in detail from the perspective of hardware processing.

[0112] Figure 6is a structural schematic diagram of an embodiment of the visual focus based interface display device provided by the present application. The visual focus based interface display device 600 can have great differences due to different configurations or performances, and can include one or more central processing units (CPUs) 610 (for example, one or more processors) and a memory 620, one or more storage media 630 (for example, one or more mass storage devices) for storing applications 633 or data 632. The memory 620 and the storage media 630 can be temporary storage or persistent storage. The programs stored in the storage media 630 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the visual focus based interface display device 600. Further, the processor 610 can be configured to communicate with the storage media 630 and execute the series of instruction operations in the storage media 630 on the visual focus based interface display device 600.

[0113] The visual focus based interface display device 600 can further include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input / output interfaces 660, and / or one or more operating systems 631, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art can understand that the visual focus based interface display device can include more or fewer components than those shown, or combine certain components, or arrange the components differently. Figure 6 The visual focus based interface display device structure shown does not constitute a limitation on the visual focus based interface display device, and can include more or fewer components than those shown, or combine certain components, or arrange the components differently.

[0114] The present application also provides a visual focus based interface display device, which includes a memory and a processor. The memory stores computer readable instructions, and the processor executes the computer readable instructions to perform the steps of the visual focus based interface display method in the above embodiments.

[0115] The present application also provides a computer readable storage medium, which can be a non-volatile computer readable storage medium or a volatile computer readable storage medium. The computer readable storage medium stores instructions, and the instructions make the computer perform the steps of the visual focus based interface display method when the instructions are run on the computer.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0117] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0118] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for displaying an interface based on visual focus, characterized in that: The interface display method based on visual focus includes: Collecting a user's eye image, identifying an eyeball area in the user's eye image, and obtaining an eyeball contour image corresponding to the eyeball area; performing a user visual state detection based on the eyeball outline image to obtain a visual state category, and determining whether the visual state category is a strabismus category or an emmetropia category, and determining whether the visual state category is a monocular vision category or a binocular vision category, and if the visual state category is the emmetropia category or the binocular vision category, determining a result of detecting a visual interference item of the eyeball outline image as no visual interference item exists, otherwise determining that a visual interference item exists; If the detection result indicates that a visual interference item exists and a monocular vision category exists, then calculating simulated position information of the other eye based on a preset eye outline reference image and position adjustment information, adjusting the simulated position information based on a size ratio of the eye outline image and the simulated position information, and performing symmetrical eye contour rendering on the eye outline image based on the adjusted simulated position information, wherein the position adjustment information is determined based on a size ratio of the preset eye outline reference image and the eye outline image; If the detection result shows that there is no visual interference item, locating the eye focus area of ​​the user in the display interface using a preset focus positioning method based on the eye contour image; Calculating a current viewing angle of the eye focus area on the display interface, and obtaining a current initial display parameter value of the display interface, adjusting the initial display parameter value of the eye focus area in the display interface according to the viewing angle, obtaining a current display parameter value of the display interface after adjustment, and adjusting the display interface according to the current display parameter value; The identifying of the eyeball area in the user's eye image and obtaining the eyeball contour image corresponding to the eyeball area includes: detecting each eye key point in the user's eye image, and connecting each eye key point in sequence to obtain a mask corresponding to the eyeball area; identifying the white of the eye area in the user's eye image according to each eye key point and preset auxiliary identification information; combining the mask and the white of the eye area, and segmenting the user's eye image based on the result of the combination to obtain the eyeball contour image corresponding to the eyeball area, wherein, if the white of the eye area is not in the transition area between the white of the eye and the pupil of the mask, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area is in the transition area between the white of the eye and the pupil of the mask but does not exceed the transition area, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area exceeds the transition area of ​​the mask, the inner edge of the transition area is used as the inner contour of the eyeball area.

2. The interface display method based on visual focus according to claim 1, characterized in that: The step of identifying the white area of ​​the eye in the user's eye image according to each of the eye key points and the preset auxiliary identification information includes: Calculating the relative coordinate mean of each eye key point in the user's eye image, and determining the eyeball reference midpoint according to the relative coordinate mean; Eyeball texture information is extracted from the preset auxiliary identification information, and the white of the eye area in the user's eye image is calculated based on the eyeball texture information and the eyeball reference midpoint.

3. The interface display method based on visual focus according to claim 1, characterized in that: The user visual state detection is performed based on the eyeball outline image, and the visual state category obtained includes: Based on the eyeball outline image, identifying the current eye movement of the user, and identifying original position information of the current eyeball outline in the user's eye image; Calculating a size ratio between a preset eyeball outline reference image and the eyeball outline image, and determining position adjustment information of the eyeball outline according to the size ratio; Identifying a focus area of ​​the user's vision on the display interface according to the current user's eye movement, and mapping the current eyeball contour to the display interface based on the position adjustment information; Calculating the relative distance between the eyeball outline and the focus area in the display interface, and determining whether the relative distance is greater than a preset distance threshold; If it is greater than, it is determined that the visual state category corresponding to the eyeball outline image is strabismus, otherwise it is emmetropia.

4. An interface display device based on visual focus, characterized in that: The interface display device based on visual focus includes: A recognition module, configured to collect an eye image of the user, identify an eyeball region in the eye image of the user, and obtain an eyeball contour image corresponding to the eyeball region; a detection module for detecting a user's visual state based on the eyeball outline image to obtain a visual state category, and determining whether the visual state category is a strabismus category or an emmetropia category, and determining whether the visual state category is a monocular vision category or a binocular vision category, and if the visual state category is the emmetropia category or the binocular vision category, determining that a result of detecting a visual interference item of the eyeball outline image is that no visual interference item exists; otherwise, determining that a visual interference item exists; a rendering module configured to, if the detection result indicates the presence of visual interference and a monocular vision category, calculate simulated position information of the other eye based on a preset eye outline reference image and position adjustment information, adjust the simulated position information based on a size ratio of the eye outline image and the simulated position information, and perform symmetrical eye contour rendering on the eye outline image based on the adjusted simulated position information, wherein the position adjustment information is determined based on a size ratio of a preset eye outline reference image and the eye outline image; a positioning module, configured to locate the user's eye focus area on the display interface using a preset focus positioning method based on the eye outline image if the detection result shows that no visual interference items exist; an adjustment module, configured to calculate a current viewing angle of the eye focus area on the display interface, obtain a current initial display parameter value of the display interface, adjust the initial display parameter value of the eye focus area in the display interface according to the viewing angle, obtain a current display parameter value of the display interface after adjustment, and adjust the display interface according to the current display parameter value; The identifying of the eyeball area in the user's eye image and obtaining the eyeball contour image corresponding to the eyeball area includes: detecting each eye key point in the user's eye image, and connecting each eye key point in sequence to obtain a mask corresponding to the eyeball area; identifying the white of the eye area in the user's eye image according to each eye key point and preset auxiliary identification information; combining the mask and the white of the eye area, and segmenting the user's eye image based on the result of the combination to obtain the eyeball contour image corresponding to the eyeball area, wherein, if the white of the eye area is not in the transition area between the white of the eye and the pupil of the mask, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area is in the transition area between the white of the eye and the pupil of the mask but does not exceed the transition area, the outer edge of the transition area is used as the inner contour of the eyeball area; if the white of the eye area exceeds the transition area of ​​the mask, the inner edge of the transition area is used as the inner contour of the eyeball area.

5. An interface display device based on visual focus, characterized in that: The visual focus-based interface display device includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the visual focus-based interface display device to perform the steps of the visual focus-based interface display method according to any one of claims 1 to 3.

6. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the interface display method based on visual focus as described in any one of claims 1 to 3 are implemented.

Images