Image processing method, device and equipment

Abstract

The embodiment of the present application provides an image processing method, device and apparatus, the method comprising: obtaining multiple eye images of a user collected at each collection moment in a historical period; determining multiple types of eye features corresponding to each collection moment according to the multiple eye images; and determining eye information of the user according to the multiple types of eye features corresponding to each collection moment. The accuracy of determining eye information is improved.

Description

Technical Field

[0001] The present application relates to the field of visual technology, and in particular to an image processing method, device and equipment. Background Art

[0002] With the popularization of electronic devices, users need to use electronic devices in many aspects of life (such as office, entertainment, etc.), and the screens of electronic devices can cause damage to users' eyes. Users need to understand eye information in a timely manner.

[0003] Currently, users can determine the user's eye information based on manual experience. For example, if the user's eyes ache and shed tears when the user watches the mobile phone screen for a long time, it means that the user's eyes are highly fatigued. However, manual experience cannot accurately determine the user's eye information, which leads to low accuracy in determining the eye information. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus and device for solving the technical problem of low accuracy in determining eye information in the prior art.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0006] Acquire multiple eye images of the user collected at each collection time in the historical period;

[0007] Determining, according to the plurality of eye images, a plurality of types of eye features corresponding to each acquisition moment;

[0008] The user's eye information is determined according to multiple types of eye features corresponding to each collection moment.

[0009] In a possible implementation, determining the user's eye information according to multiple types of eye features corresponding to each acquisition moment includes:

[0010] determining at least one eye feature of a first type and at least one eye feature of a second type among the plurality of types of eye features;

[0011] For any one of the first-type eye features, determining, according to the collection time corresponding to the first-type eye feature, change information of the first-type eye feature within the historical period;

[0012] The eye information is determined according to the change information of each first type of eye feature in the historical period and / or the at least one second type of eye feature.

[0013] In a possible implementation, the first type of eye features includes an upper eyelid position and an eye opening area; and determining the eye information according to change information of each first type of eye feature within the historical period and the at least one second type of eye feature includes:

[0014] determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area;

[0015] The eye information is determined according to the eye fatigue degree and / or the second type of eye features.

[0016] In a possible implementation manner, the second type of eye features includes the distance between pupils and the eye shape; and determining the eye information according to the eye fatigue degree and / or the second type of eye features includes:

[0017] If the eye fatigue degree is greater than or equal to a first threshold, determining that the eye information is dry eyes;

[0018] If the similarity between the two eye shapes of the user is less than a second threshold, determining that the eye information is conjunctivitis;

[0019] If the distance between the pupils is greater than a third threshold, or the distance between the pupils is less than a fourth threshold, it is determined that the eye information is strabismus, and the third threshold is greater than the fourth threshold;

[0020] If the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, it is determined that the eye information is amblyopia.

[0021] In a possible implementation, determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area includes:

[0022] Determining the average position of the upper eyelid according to the change information of the upper eyelid position;

[0023] Determining an average open area of ​​the eyes according to the change information of the open area of ​​the eyes;

[0024] The degree of eye fatigue is determined according to the average position and the average open area.

[0025] In a possible implementation, for any eye feature of the first type, determining, according to a collection time corresponding to the eye feature of the first type, change information of the eye feature of the first type within the historical period includes:

[0026] Determine, according to the order of the collection moments corresponding to the eye features of the first type, a waveform graph corresponding to the eye features of the first type, wherein the horizontal axis of the waveform graph is time and the vertical axis is the eye features of the first type;

[0027] The waveform graph is determined as the change information of the first type of eye feature.

[0028] In a possible implementation, determining a plurality of eye features corresponding to each acquisition moment according to the plurality of eye images includes:

[0029] Processing the multiple eye images by using a preset model to obtain multiple eye features corresponding to each acquisition moment;

[0030] The preset model is obtained by learning multiple groups of samples, and each group of samples includes sample eye images and sample eye features.

[0031] In a possible implementation, after determining the user's eye information according to multiple types of eye features corresponding to each acquisition moment, the method further includes:

[0032] If the eye information is dry eyes, conjunctivitis, strabismus or amblyopia, prompt information is generated, and the prompt information includes: doctor information and object recommendation information.

[0033] In a possible implementation, after determining multiple types of eye features corresponding to each acquisition moment, the method further includes:

[0034] Acquire eye features collected at at least two target moments, where the similarity between environmental information at the at least two target moments is greater than or equal to a first preset threshold, and the environmental information includes ambient light information and screen display information of a terminal device viewed by the user;

[0035] If the similarity between the eye features at the at least two target moments is less than or equal to a second preset threshold, it is determined that the eye information of the user is a vision change.

[0036] In a second aspect, an embodiment of the present application provides an image processing device, including a first acquisition module, a first determination module, and a second determination module, wherein:

[0037] The first acquisition module is used to acquire multiple eye images of the user acquired at each acquisition time within a historical period;

[0038] The first determination module is used to determine multiple types of eye features corresponding to each acquisition moment according to the multiple eye images;

[0039] The second determination module is used to determine the user's eye information according to multiple types of eye features corresponding to each acquisition moment.

[0040] In a possible implementation manner, the second determining module is specifically configured to:

[0041] determining at least one eye feature of a first type and at least one eye feature of a second type among the plurality of types of eye features;

[0042] For any one of the first-type eye features, determining, according to the collection time corresponding to the first-type eye feature, change information of the first-type eye feature within the historical period;

[0043] The eye information is determined according to the change information of each first type of eye feature in the historical period and / or the at least one second type of eye feature.

[0044] In a possible implementation manner, the second determining module is specifically configured to:

[0045] determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area;

[0046] The eye information is determined according to the eye fatigue degree and / or the second type of eye features.

[0047] In a possible implementation manner, the second determining module is specifically configured to:

[0048] If the eye fatigue degree is greater than or equal to a first threshold, determining that the eye information is dry eyes;

[0049] If the similarity between the two eye shapes of the user is less than a second threshold, determining that the eye information is conjunctivitis;

[0050] If the distance between the pupils is greater than a third threshold, or the distance between the pupils is less than a fourth threshold, it is determined that the eye information is strabismus, and the third threshold is greater than the fourth threshold;

[0051] If the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, it is determined that the eye information is amblyopia.

[0052] In a possible implementation manner, the second determining module is specifically configured to:

[0053] Determining the average position of the upper eyelid according to the change information of the upper eyelid position;

[0054] Determining an average open area of ​​the eyes according to the change information of the open area of ​​the eyes;

[0055] The degree of eye fatigue is determined according to the average position and the average open area.

[0056] In a possible implementation manner, the second determining module is specifically configured to:

[0057] Determine, according to the order of the collection moments corresponding to the eye features of the first type, a waveform graph corresponding to the eye features of the first type, wherein the horizontal axis of the waveform graph is time and the vertical axis is the eye features of the first type;

[0058] The waveform graph is determined as the change information of the first type of eye feature.

[0059] In a possible implementation manner, the first determining module is specifically configured to:

[0060] Processing the multiple eye images by using a preset model to obtain multiple eye features corresponding to each acquisition moment;

[0061] The preset model is obtained by learning multiple groups of samples, and each group of samples includes sample eye images and sample eye features.

[0062] In another possible implementation, the device further includes a generating module, wherein the generating module is configured to:

[0063] If the eye information is dry eyes, conjunctivitis, strabismus or amblyopia, prompt information is generated, and the prompt information includes: doctor information and object recommendation information.

[0064] In another possible implementation, the device further includes a second acquisition module, where the second acquisition module is configured to:

[0065] Acquire eye features collected at at least two target moments, where the similarity between environmental information at the at least two target moments is greater than or equal to a first preset threshold, and the environmental information includes ambient light information and screen display information of a terminal device viewed by the user;

[0066] If the similarity between the eye features at the at least two target moments is less than or equal to a second preset threshold, it is determined that the eye information of the user is a vision change.

[0067] In a third aspect, an embodiment of the present application provides an image processing device, including: a processor and a memory;

[0068] The memory stores computer-executable instructions;

[0069] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method described in the first aspect and various possible designs of the first aspect.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0071] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method described in the first aspect and various possible designs of the first aspect.

[0072] The embodiments of the present application provide an image processing method, apparatus and device, which obtain multiple eye images of a user collected at each collection time in a historical period, determine multiple types of eye features corresponding to each collection time based on the multiple eye images, and determine the user's eye information based on the multiple types of eye features corresponding to each collection time. In the above method, the terminal device can accurately determine multiple eye features based on the multiple eye images of the user, and accurately determine the user's eye information by combining the collection time and the eye features, thereby improving the accuracy of determining the user's eye information. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0074] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0075] Figure 3 A schematic diagram of a process for collecting eye images provided in an embodiment of the present application;

[0076] Figure 4A A waveform diagram corresponding to a first type of eye feature provided in an embodiment of the present application;

[0077] Figure 4B A waveform diagram corresponding to another first type of eye feature provided in an embodiment of the present application;

[0078] Figure 5A flowchart of another image processing method provided in an embodiment of the present application;

[0079] Figure 6 A schematic diagram of a process for generating prompt information provided in an embodiment of the present application;

[0080] Figure 7 A schematic flow chart of another method for determining eye information provided in an embodiment of the present application;

[0081] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0082] Fig. 9 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0083] Fig.10 A schematic diagram of the hardware structure of the image processing device provided in this application. DETAILED DESCRIPTION

[0084] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0085] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0086] In the related art, users need to understand the information of their eyes and then protect their eyes through the information of their eyes. Currently, users can determine the information of their eyes based on manual experience. For example, when a user watches a mobile phone screen for a long time, if the user's eyes are sore and tearful, it means that the user's eyes are very tired, and the user should stop using the mobile phone to protect his eyes. However, this method cannot detect that the eyes have been damaged in time, and manual experience cannot accurately determine the information of the user's eyes, which leads to low accuracy in determining the eye information.

[0087] In order to solve the technical problem of low accuracy in determining eye information in the related art, an embodiment of the present application provides an image processing method, which obtains multiple eye images of a user collected at each collection moment in a historical period, determines multiple types of eye features corresponding to each collection moment according to the multiple eye images, determines at least one first type of eye feature and at least one second type of eye feature from the multiple types of eye features, and for any first type of eye feature, determines the change information of the first type of eye feature in the historical period according to the collection moment corresponding to the first type of eye feature, and determines the eye information according to the change information of each first type of eye feature in the historical period and / or at least one second type of eye feature. In the above method, the terminal device can accurately determine multiple eye features according to the multiple eye images of the user, and because the first type of eye features and the second type of eye features can experience the user's eye information from multiple dimensions, therefore, the user's eye information can be determined timely and accurately by combining the first type of eye features and the second type of eye features, thereby improving the accuracy of determining the user's eye information.

[0088] Next, combine Figure 1 , which introduces in detail the application scenarios to which the embodiments of the present application are applicable.

[0089] Figure 1 This is a schematic diagram of an application scenario provided by the embodiment of this application. Figure 1 , including a terminal device and an eye image. The terminal device can collect multiple eye images of the user, and extract multiple types of eye features corresponding to each eye image based on the eye images collected at multiple times. For example, the terminal device collects eye images of the user within 1 day, and determines, based on the eye images, that the position of the user's upper eyelid is low and the eye opening area is small. At this time, the terminal device can determine that the user's eye information is dry eye syndrome and display a prompt message on the display page. In this way, the terminal device can accurately determine the user's eye information in combination with the user's eye features in the historical period, thereby improving the accuracy of determining the eye information.

[0090] The technical solution shown in the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar contents will not be repeatedly described in different embodiments.

[0091] Figure 2 This is a flowchart of an image processing method provided in an embodiment of the present application. Figure 2 , the method may include:

[0092] S201. Acquire multiple eye images of a user collected at each collection moment in a historical period.

[0093] The execution subject of the embodiment of the present application may be a terminal device, or an image processing device set in the terminal device. Optionally, the image processing device may be implemented by software, or by a combination of software and hardware.

[0094] The eye image may be an image including the user's eyes. For example, the eye image may be a facial image of the user, an eye image of the user, and the like. Optionally, the terminal device may obtain multiple eye images of the user through a camera device. For example, a front camera is usually provided on the screen end of a mobile phone, and when the user uses the mobile phone, the front camera may capture the user's eye image in real time. Optionally, when the user uses the terminal device, the camera of the terminal device may obtain a video stream including the user's face, and then determine multiple eye images of the user in the video stream. For example, the camera of a mobile phone may capture a 10-second video stream including the user's face, and then capture a frame of image in the video stream as the user's eye image. For example, the mobile phone camera may determine the highest definition frame of image in the video stream including the user's eyes, and determine the highest definition frame of image as the user's eye image.

[0095] The acquisition time may be the time when the terminal device acquires the eye image, and each eye image has a corresponding acquisition time. For example, if the terminal device acquires an eye image of a user at 8 a.m., the acquisition time corresponding to the eye image is 8 a.m. Optionally, the terminal device may acquire the user's eye image multiple times within a historical period, and therefore, the historical period may include multiple acquisition moments. For example, if the terminal device acquires 3,000 eye images within the past week, then the past week includes 3,000 acquisition moments.

[0096] Next, combine Figure 3 , the process of collecting multiple eye images of a user by a terminal device is explained.

[0097] Figure 3 This is a schematic diagram of a process for collecting eye images provided in an embodiment of the present application. Figure 3 , including a terminal device. In the shooting page of the terminal device, the camera device of the terminal device can collect a video stream including the user's face, and display the video stream including the user's face in the shooting page of the terminal device. The terminal device can determine a frame of image including the user's eyes in the video stream as the user's eye image.

[0098] S202: Determine multiple types of eye features corresponding to each acquisition moment according to the multiple eye images.

[0099] The eye features may be features of the user's eyes. For example, the eye features may be features of the user's eyelids, eyeballs, pupils, etc. For example, the eye features may be the position of the upper eyelids, the distance between the pupils, the area of ​​the eyes opened, etc.

[0100] The eye features include first-type eye features and second-type eye features, wherein the first-type eye features include upper eyelid positions and eye opening areas, and the second-type eye features include distance between pupils and eye shapes.

[0101] The position of the upper eyelid is the position of the upper eyelid of the user in the eye. For example, the position of the upper eyelid of the user can be in the middle of the eye, 0.5 cm above the eye, etc. The area of ​​the open eyes can be the area of ​​the eyeball when the user's eyes are open. For example, the area of ​​the open eyes of the user can be 1 square centimeter, 2 square centimeters, etc. The distance between the pupils is the distance between the positions of the pupils of the two eyes of the user. For example, the distance between the pupils can be 3 cm, 4 cm, etc.

[0102] The multiple eye features corresponding to each acquisition moment can be determined according to the following feasible implementation method: multiple eye images are processed by a preset model to obtain multiple eye features corresponding to each acquisition moment. The preset model is learned from multiple groups of samples, and each group of samples includes a sample eye image and a sample eye feature. The multiple groups of samples can be pre-labeled samples. For example, for sample eye image 1, sample eye feature 1 corresponding to sample eye image 1 is obtained to obtain a group of samples, and the group of samples includes eye image 1 and eye feature 1. In this way, multiple groups of samples can be obtained. For example, the multiple groups of samples can be shown in Table 1:

[0103] Table 1

[0104] Multiple groups of samples Sample eye image Sample eye features The first set of samples Sample eye image 1 Sample Eye Features 1 The second set of samples Sample eye image 2 Sample Eye Features 2 The third group of samples Sample eye image 3 Sample eye features 3 …… …… ……

[0105] It should be noted that Table 1 is only an example of multiple groups of samples and is not a limitation on the multiple groups of samples.

[0106] For example, assuming that the eye image input into the preset model is sample eye image 1, the eye feature corresponding to the eye image is sample eye feature 1; assuming that the eye image input into the preset model is sample eye image 2, the eye feature corresponding to the eye image is sample eye feature 2; assuming that the eye image input into the preset model is sample eye image 3, the eye feature corresponding to the eye image is sample eye feature 3.

[0107] Optionally, the same eye image may correspond to one eye feature or multiple eye features. For example, the eye image may include the first type of eye features, or the eye image may include the first type of eye features and the second type of eye features at the same time.

[0108] S203: Determine the user's eye information according to multiple types of eye features corresponding to each acquisition moment.

[0109] The user's eye information may be health information of the user's eyes. For example, the user's eye health information may include whether the user has strabismus, amblyopia, dry eyes, etc.

[0110] The user's eye information can be determined according to the following feasible implementation method: determine at least one first type of eye feature and at least one second type of eye feature from multiple types of eye features, and for any first type of eye feature, determine the change information of the first type of eye feature within a historical period according to the collection time corresponding to the first type of eye feature, and determine the eye information according to the change information of each first type of eye feature within a historical period and / or the at least one second type of eye feature. The first type of eye features include the upper eyelid position and the eye opening area, and the second type of eye features include the distance between the pupils and the eye shape. For example, the terminal device can determine the eye feature of the upper eyelid position from the first type of eye features, and determine the eye feature of the distance between the pupils from the second type of eye features.

[0111] The change information is used to indicate the relationship between the first type of eye feature and time. Optionally, the change information of the first type of eye feature within the historical period can be determined according to the following feasible implementation method: according to the order of the collection moments corresponding to the first type of eye feature, determine the waveform graph corresponding to the first type of eye feature. Among them, the horizontal axis of the waveform graph is time, and the vertical axis is the first type of eye feature. The waveform graph is determined as the change information of the first type of eye feature. For example, if the terminal device determines the upper eyelid positions of 300 users and 300 collection moments corresponding to the upper eyelid positions, the terminal device can obtain the waveform graph corresponding to the upper eyelid position according to the time sequence of the collection moments and the upper eyelid position, the horizontal axis of the waveform graph is the collection moment, and the vertical axis of the waveform graph is the upper eyelid position.

[0112] Optionally, when determining the waveform graph corresponding to the first type of eye feature, the first type of eye feature may be normalized so that the value range of the first type of eye feature is between 0 and 1.

[0113] Next, combine Figure 4A-4B , the waveform diagram corresponding to the first type of eye features is explained.

[0114] Figure 4A This is a waveform diagram corresponding to a first type of eye feature provided in an embodiment of the present application. The first type of eye feature is the upper eyelid position, see Figure 4A, the horizontal axis of the waveform is time, and the vertical axis of the waveform is the upper eyelid position. The value range of the upper eyelid position is 0-1. In the historical period, the upper eyelid position changes with time. The higher the upper eyelid position, the lower the curve in the waveform, and the lower the upper eyelid position, the higher the curve in the waveform. The waveform corresponding to the upper eyelid position can accurately obtain the change information of the upper eyelid position in the historical period.

[0115] Figure 4B This is another waveform diagram corresponding to the first type of eye feature provided in the embodiment of the present application. The first type of eye feature is the eye opening area, see Figure 4B The horizontal axis of the waveform is time, and the vertical axis is the eye open area. The value range of the eye open area is 0-1. In the historical period, the eye open area changes with time. The larger the eye open area, the higher the curve in the waveform, and the smaller the eye open area, the lower the curve in the waveform. The waveform corresponding to the eye open area can accurately obtain the change information of the eye open area in the historical period.

[0116] Optionally, the eye information may be determined according to the following feasible implementation: the degree of eye fatigue of the user's eyes in the historical period is determined according to the change information of the upper eyelid position and the change information of the eye opening area. The degree of eye fatigue is used to indicate the degree of eye fatigue of the user. For example, if the degree of eye fatigue of the user's eyes in the historical period is large, the user's current eye fatigue is high; if the degree of eye fatigue of the user's eyes in the historical period is small, the user's current eye fatigue is low.

[0117] Optionally, the degree of eye fatigue of the user's eyes in the historical period can be determined according to the following feasible implementation method: according to the change information of the upper eyelid position, the average position of the upper eyelid is determined. Wherein, the average position of the upper eyelid is the average position of the upper eyelid of the user's eyes in the historical period. For example, when the terminal device obtains the waveform of the upper eyelid position in the historical period, the terminal device can determine the average position of the upper eyelid of the user in the historical period according to the waveform corresponding to the upper eyelid position. Optionally, since the upper eyelid position is normalized, the value range of the average position of the upper eyelid is between 0 and 1. For example, if the average position of the upper eyelid in the historical period is the middle of the eye, the average position of the upper eyelid is 0.5. According to the change information of the eye opening area, the average eye opening area is determined. Wherein, the average eye opening area is the average open area of ​​the user's eyes in the historical period. For example, when the terminal device obtains the waveform of the eye opening area in the historical period, the terminal device can determine the average open area of ​​the user's eyes in the historical period according to the waveform corresponding to the eye opening area. Optionally, in actual application, the average eye open area is used to indicate the user's squinting degree. For example, when the user's eyes exert more force, the user's squinting degree is higher and the eye open area is smaller. When the user's eyes exert less force, the user's squinting degree is lower and the eye open area is larger. Optionally, since the eye open area is normalized, the value range of the average eye open area is also between 0 and 1.

[0118] Determine the degree of eye fatigue based on the average position and the average open area. For example, if the average position of the upper eyelids of the user's eyes in the historical period is small and the average open area is small, then it is determined that the user's degree of eye fatigue is high; if the average position of the upper eyelids of the user's eyes in the historical period is large and the average open area is large, then it is determined that the user's degree of eye fatigue is low. Optionally, the user's degree of eye fatigue can be determined by a trained model. For example, the average position and average open area of ​​the upper eyelid are input into the trained model, and the model outputs the corresponding degree of eye fatigue.

[0119] The eye information is determined according to the degree of eye fatigue and / or the second type of eye features. For example, if the user's eye fatigue degree is high, the user's eye information can be determined to be dry eyes; if the distance between the pupils of the user's eyes is large, the user's eye information is determined to be strabismus.

[0120] The embodiment of the present application provides an image processing method, which obtains multiple eye images of a user collected at each collection moment in a historical period, processes the multiple eye images through a preset model, obtains multiple types of eye features corresponding to each collection moment, determines at least one first type of eye feature and at least one second type of eye feature from the multiple types of eye features, determines the change information of the first type of eye feature in the historical period according to the collection moment corresponding to the first type of eye feature for any first type of eye feature, and determines the eye information according to the change information of each first type of eye feature in the historical period and / or the at least one second type of eye feature. In this way, the terminal device can accurately determine the multiple eye features according to the multiple eye images of the user, and because the first type of eye features and the second type of eye features can experience the user's eye information from multiple dimensions, therefore, the user's eye information can be determined timely and accurately by combining the first type of eye features and the second type of eye features, thereby improving the accuracy of determining the user's eye information.

[0121] exist Figure 2 Based on the embodiment shown below, combined with Figure 5 , the process of the above image processing method is described in detail.

[0122] Figure 5 This is a flowchart of another image processing method provided in an embodiment of the present application. Figure 5 , the method comprising:

[0123] S501. Acquire multiple eye images of a user collected at each collection moment in a historical period.

[0124] It should be noted that the execution process of step S501 can refer to step S201, and the embodiment of the present application will not be repeated here.

[0125] S502: Determine multiple types of eye features corresponding to each acquisition moment according to the multiple eye images.

[0126] It should be noted that the execution process of step S502 can refer to step S202, and the present embodiment will not be repeated here.

[0127] S503: Determine eye information of the user according to multiple types of eye features corresponding to each collection moment.

[0128] Optionally, the user's eye fatigue degree in the historical period is determined based on the change information of the upper eyelid position and the change information of the eye opening area, and the user's eye information is determined based on the eye fatigue degree and / or the second type of eye features. Optionally, after the terminal device determines the user's eye information, the user's eye information can be displayed on the display screen.

[0129] Optionally, there are four situations when determining the user's eye information:

[0130] Case 1: Determine the user's eye information based on the degree of eye fatigue.

[0131] When determining the user's eye information based on the degree of eye fatigue, if the degree of eye fatigue is greater than or equal to the first threshold, the eye information is determined to be dry eyes. For example, if the degree of eye fatigue of the user's eyes is greater than or equal to the first threshold, it means that the user's eyes have a high degree of fatigue in the historical period, and therefore, it can be determined that the user's eyes have dry eyes. Because when the eye fatigue is high over a period of time, the upper eyelid position of the eye is lower and the eye opening area is smaller, therefore, the user's eye fatigue degree can be accurately determined through the upper eyelid position and the eye opening area, thereby improving the accuracy of determining the eye information.

[0132] Case 2: Eye information is determined based on the user's eye shape.

[0133] When determining the user's eye information based on the eye shape, if the similarity of the shapes of the two eyes of the user is less than the second threshold, the eye information is determined to be conjunctivitis. For example, the terminal device can obtain the shapes of the left eye and the right eye of the user. If the shapes of the left eye and the right eye of the user are significantly different, it can be determined that the user's eyes have conjunctivitis or keratitis. Since the shape of the eyes changes when the eyes suffer from conjunctivitis or keratitis, the terminal device can accurately determine whether the eyes suffer from conjunctivitis or keratitis by collecting the shapes of the user's eyes and comparing the similarity of the shapes of the two eyes, thereby improving the accuracy of determining the eye information.

[0134] Case 3: Determine eye information based on the distance between pupils.

[0135] When determining the eye information according to the distance between the pupils of the user's eyes, if the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, the eye information is determined to be strabismus. Among them, the third threshold is greater than the fourth threshold. Optionally, before determining the eye information by the distance between the pupils, the terminal device can determine the standard pupil distance of the user, and then determine the third threshold and the fourth threshold by the standard pupil distance. For example, if the terminal device collects the user's standard pupil distance of 4 cm-6 cm, the terminal device determines the third threshold to be 6 cm and the fourth threshold to be 4 cm. For example, in a historical period, if the number of times the distance between the pupils of the user's eyes is greater than the third threshold is 1, it can be determined that the user's eyes are strabismus in the historical period. Since eye strabismus occurs when the user is unconscious, the terminal device can accurately determine whether the user's eyes are strabismus through the distance between the pupils, thereby improving the accuracy of determining the eye information.

[0136] Case 4: Determine the user's eye information based on the degree of eye fatigue and the distance between pupils.

[0137] Optionally, the user's eye information can be determined according to the following feasible implementation method: if the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, then the user's eye information is determined to be amblyopia. For example, in the past week, the user's eye fatigue level is high, and the user's eyes are squinting, then the user's eye information can be determined to be amblyopia. Optionally, when it is determined that the user's eye information is amblyopia, the user's vision can be further obtained. If the user's eyes are myopic, and the degree of eye fatigue is high, and the eyes are squinting, then the user's eye information can be determined to be amblyopia.

[0138] When the user has amblyopia, the user uses the eyes more strenuously, the eye exerts greater force, and the eyeballs shift unconsciously. Therefore, if the average position of the user's upper eyelids in the historical period is low and the eye opening area is small, it means that the user has used the eyes more strenuously in the historical period. If the user's eyes are squinting in the historical period, it can be determined that the user's eye information in the historical period is amblyopia, thereby improving the accuracy of determining eye information.

[0139] Optionally, in actual application, the user's eye information can be determined by a pre-trained first model, wherein the pre-trained first model includes the eye feature extraction function of the preset model in the embodiment of the present application. The first model is obtained by training multiple groups of samples, wherein each group of samples includes a sample eye image and sample eye information. For example, for sample eye image 1, sample eye information 1 corresponding to sample eye image 1 can be obtained, and then a group of samples can be obtained, and multiple groups of samples can be obtained by this method. Optionally, the first model can be set in a terminal device so that the terminal device can determine the user's eye information based on the user's eye image. For example, a mobile phone can capture a user's face image through a front camera and transmit the user's face image to the first model. When the face image is input into the first model, the first model can extract the eye features in the face image and output the user's eye information based on the eye features. In this way, the user's eye information can be obtained without uploading the eye image to the server, thereby improving the efficiency of determining the user's eye information.

[0140] S504: If the eye information is dry eyes, conjunctivitis, strabismus or amblyopia, a prompt message is generated.

[0141] Optionally, the prompt information includes: doctor information and object recommendation information. The doctor information may be information about doctors who treat eye diseases, and the object recommendation information may be information about medicines corresponding to the eye information. For example, when the user's eye information is dry eye, the terminal device may recommend information about ophthalmologists in the user's city and information about medicines needed to treat strabismus to the user. In this way, when the user's eyes are tired, the terminal device may generate prompt information in a timely manner, so that the user can relieve eye fatigue in a timely manner.

[0142] Next, combine Figure 6 , the process of generating prompt information by the terminal device is explained.

[0143] Figure 6 A schematic diagram of a process for generating prompt information provided in an embodiment of the present application. Figure 6 , including: a terminal device. The terminal device determines that the user's eye information is amblyopia based on multiple types of eye features of the user in a historical period. The terminal device generates prompt information, wherein the prompt information includes a photo of a doctor who treats amblyopia, the doctor's name, the doctor's department, and the doctor's title. When the user clicks the appointment button, the terminal device can jump to the corresponding doctor's online registration page, so that the user can directly make an appointment with the corresponding doctor according to the prompt information, reducing the complexity of the operation and thereby improving the user experience.

[0144] The embodiment of the present application provides an image processing method, which obtains multiple eye images of a user collected at each collection moment in a historical period, determines multiple types of eye features corresponding to each collection moment based on the multiple eye images, and determines the eye information of the user based on the multiple types of eye features corresponding to each collection moment, and generates prompt information if the eye information is dry eyes, conjunctivitis, strabismus or amblyopia. In the above method, the terminal device can accurately determine multiple eye features based on the multiple eye images of the user, and because the first type of eye features and the second type of eye features can experience the user's eye information from multiple dimensions, therefore, the user's eye information can be determined timely and accurately in combination with the first type of eye features and the second type of eye features, and after determining the eye information, the terminal device can generate prompt information to improve the user's experience, thereby improving the accuracy of determining the user's eye information.

[0145] Based on any of the above embodiments, after determining multiple types of eye features corresponding to each acquisition moment, the present application also includes another method for determining eye information. Figure 7 This method of determining eye information is described.

[0146] Figure 7 A flowchart of another method for determining eye information provided in an embodiment of the present application. Figure 7 , the method comprising:

[0147] S701. Obtain eye features collected at least two target moments.

[0148] The similarity between the environmental information of at least two target moments is greater than or equal to a first preset threshold. The environmental information includes ambient light information and screen display information of the user viewing the terminal device. Optionally, the ambient light information may be the lighting brightness and color temperature of the lighting device. For example, in actual application, the environment in which the user views the terminal device includes a lighting device, and the ambient light information is the brightness and color temperature of the lighting device. The screen display information of the terminal device may include: the font displayed on the screen, the brightness of the screen, the color and color temperature of the screen background. For example, the screen display information of the terminal device may be: the displayed font is Songti No. 14, the brightness of the screen is medium brightness, and the color of the screen background is green light information. For example, the environmental information of moment A is environmental information A, and the environmental information of moment B is environmental information B. If the similarity between environmental information A and environmental information B is greater than or equal to the first preset threshold, moment A and moment B may be determined as two target moments.

[0149] When the terminal device determines at least two target moments, the terminal device may obtain the eye features collected at the target moments. For example, if the eye features collected by the terminal device at moment A are eye features A, and the eye features collected by the terminal device at moment B are eye features B, if moment A and moment B are target moments, the terminal device obtains eye features A and eye features B.

[0150] S702: Determine eye information of the user based on eye features collected at least two target moments.

[0151] Optionally, the user's eye information may be determined according to the following feasible implementation method: if the similarity between the eye features at at least two target moments is less than or equal to a second preset threshold, the user's eye information is determined to be a change in vision. For example, if the user's eye features collected by the terminal device in the environmental information A one month ago are that the upper eyelid position is high and the eye opening area is large, and the user's eye features collected by the terminal device in the environmental information A currently are that the upper eyelid position is low and the eye opening area is small, then the user's eye information may be determined to be a decrease in vision.

[0152] The embodiment of the present application provides a method for determining eye information, obtaining eye features collected at at least two target moments, the similarity between the environmental information at at least two target moments is greater than or equal to a first preset threshold, and if the similarity between the eye features at at least two target moments is less than or equal to a second preset threshold, the user's eye information is determined to be a vision change. In this way, by comparing the eye features of the user at different times with the same environmental information, it is possible to accurately determine whether the user's eye vision has changed, thereby improving the accuracy of determining the eye information.

[0153] Figure 8 This is a schematic diagram of the structure of the image processing device provided in the embodiment of the present application. Figure 8 The image processing device 10 includes a first acquisition module 11, a first determination module 12 and a second determination module 13, wherein:

[0154] The first acquisition module 11 is used to acquire multiple eye images of the user acquired at each acquisition moment in a historical period;

[0155] The first determining module 12 is used to determine multiple types of eye features corresponding to each acquisition moment according to the multiple eye images;

[0156] The second determining module 13 is used to determine the user's eye information according to multiple types of eye features corresponding to each acquisition moment.

[0157] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0158] determining at least one eye feature of a first type and at least one eye feature of a second type among the plurality of types of eye features;

[0159] For any one of the first-type eye features, determining, according to the collection time corresponding to the first-type eye feature, change information of the first-type eye feature within the historical period;

[0160] The eye information is determined according to the change information of each first type of eye feature in the historical period and / or the at least one second type of eye feature.

[0161] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0162] determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area;

[0163] The eye information is determined according to the eye fatigue degree and / or the second type of eye features.

[0164] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0165] If the eye fatigue degree is greater than or equal to a first threshold, determining that the eye information is dry eyes;

[0166] If the similarity between the two eye shapes of the user is less than a second threshold, determining that the eye information is conjunctivitis;

[0167] If the distance between the pupils is greater than a third threshold, or the distance between the pupils is less than a fourth threshold, it is determined that the eye information is strabismus, and the third threshold is greater than the fourth threshold;

[0168] If the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, it is determined that the eye information is amblyopia.

[0169] In a possible implementation manner, the second determining module 13 is specifically configured to:

[0170] Determining the average position of the upper eyelid according to the change information of the upper eyelid position;

[0171] Determining an average open area of ​​the eyes according to the change information of the open area of ​​the eyes;

[0172] The degree of eye fatigue is determined according to the average position and the average open area.

[0173] In a possible implementation manner, the second determining module is specifically configured to:

[0174] Determine, according to the order of the collection moments corresponding to the eye features of the first type, a waveform graph corresponding to the eye features of the first type, wherein the horizontal axis of the waveform graph is time and the vertical axis is the eye features of the first type;

[0175] The waveform graph is determined as the change information of the first type of eye feature.

[0176] In a possible implementation manner, the first determining module 12 is specifically configured to:

[0177] Processing the multiple eye images by using a preset model to obtain multiple eye features corresponding to each acquisition moment;

[0178] The preset model is obtained by learning multiple groups of samples, and each group of samples includes sample eye images and sample eye features.

[0179] The image processing device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0180] Fig. 9 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application. Figure 8 Based on the examples shown, see Fig. 9 The image processing device 10 further includes a generating module 14, wherein the generating module 14 is used to:

[0181] If the eye information is dry eyes, conjunctivitis, strabismus or amblyopia, prompt information is generated, and the prompt information includes: doctor information and object recommendation information.

[0182] In another possible implementation, the image processing device 10 further includes a second acquisition module 15, and the second acquisition module 15 is used to:

[0183] Acquire eye features collected at at least two target moments, where the similarity between environmental information at the at least two target moments is greater than or equal to a first preset threshold, and the environmental information includes ambient light information and screen display information of a terminal device viewed by the user;

[0184] If the similarity between the eye features at the at least two target moments is less than or equal to a second preset threshold, it is determined that the eye information of the user is a vision change.

[0185] The image processing device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0186] Fig.10 Schematic diagram of the hardware structure of the image processing device provided for this application. Fig.10 The image processing device 20 may include: a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can communicate; illustratively, the processor 21 and the memory 22 communicate via a communication bus 23, the memory 22 is used to store program instructions, and the processor 21 is used to call the program instructions in the memory to execute the image processing method shown in any of the above method embodiments.

[0187] Optionally, the image processing device 20 may further include a communication interface, which may include a transmitter and / or a receiver.

[0188] Optionally, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the present application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0189] The present application provides a readable storage medium, on which a computer program is stored; the computer program is used to implement the image processing method described in any of the above embodiments.

[0190] An embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, a computer executes the above-mentioned image processing method.

[0191] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned memory (storage medium) includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

[0192] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0196] In the present application, the term "include" and its variations may refer to non-restrictive inclusion; the term "or" and its variations may refer to "and / or". The terms "first", "second", etc. in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In the present application, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

Claims

1. An image processing method, characterized in that: include: Acquire multiple eye images of the user collected at each collection time in the historical period; Determining, according to the plurality of eye images, a plurality of types of eye features corresponding to each acquisition moment; Determining at least one first type of eye feature and at least one second type of eye feature from among the plurality of types of eye features; the first type of eye feature comprises an upper eyelid position and an eye opening area, and the second type of eye feature comprises a distance between pupils and an eye shape; For any one of the first-type eye features, determining, according to the collection time corresponding to the first-type eye feature, change information of the first-type eye feature within the historical period; determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area; determining eye information of the user according to the eye fatigue degree and / or the at least one second type of eye feature; The determining the eye information of the user according to the eye fatigue degree and / or the at least one second type of eye feature includes: If the eye fatigue degree is greater than or equal to a first threshold, determining that the eye information is dry eyes; If the similarity between the two eye shapes of the user is less than a second threshold, determining that the eye information is conjunctivitis; If the distance between the pupils is greater than a third threshold, or the distance between the pupils is less than a fourth threshold, it is determined that the eye information is strabismus, and the third threshold is greater than the fourth threshold; If the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, it is determined that the eye information is amblyopia.

2. The method according to claim 1, characterized in that Determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area includes: Determining the average position of the upper eyelid according to the change information of the upper eyelid position; Determining an average open area of ​​the eyes according to the change information of the open area of ​​the eyes; The degree of eye fatigue is determined according to the average position and the average open area.

3. The method according to claim 1 or 2, characterized in that: For any eye feature of the first type, determining change information of the eye feature of the first type within the historical period according to the collection time corresponding to the eye feature of the first type, includes: Determine, according to the order of the collection moments corresponding to the eye features of the first type, a waveform graph corresponding to the eye features of the first type, wherein the horizontal axis of the waveform graph is time and the vertical axis is the eye features of the first type; The waveform diagram is determined as the change information of the first type of eye feature.

4. The method according to claim 1 or 2, characterized in that: Determining a plurality of eye features corresponding to each acquisition moment according to the plurality of eye images includes: Processing the multiple eye images by using a preset model to obtain multiple eye features corresponding to each acquisition moment; The preset model is obtained by learning multiple groups of samples, and each group of samples includes sample eye images and sample eye features.

5. The method according to claim 1 or 2, characterized in that: After determining the eye information of the user, the method further includes: If the eye information is dry eyes, conjunctivitis, strabismus or amblyopia, prompt information is generated, and the prompt information includes: doctor information and object recommendation information.

6. The method according to claim 1 or 2, characterized in that: After determining multiple types of eye features corresponding to each acquisition moment, the method further includes: Acquire eye features collected at at least two target moments, where the similarity between environmental information at the at least two target moments is greater than or equal to a first preset threshold, and the environmental information includes ambient light information and screen display information of a terminal device viewed by the user; If the similarity between the eye features at the at least two target moments is less than or equal to a second preset threshold, it is determined that the eye information of the user is a vision change.

7. An image processing device, characterized in that: It includes a first acquisition module, a first determination module and a second determination module, wherein: The first acquisition module is used to acquire multiple eye images of the user acquired at each acquisition time within a historical period; The first determination module is used to determine multiple types of eye features corresponding to each acquisition moment according to the multiple eye images; The second determination module is used to determine at least one first type of eye feature and at least one second type of eye feature from the multiple types of eye features; the first type of eye feature includes an upper eyelid position and an eye opening area, and the second type of eye feature includes an inter-pupillary distance and an eye shape; For any one of the first-type eye features, determining, according to the collection time corresponding to the first-type eye feature, change information of the first-type eye feature within the historical period; determining the degree of eye fatigue of the user's eyes in the historical period according to the change information of the upper eyelid position and the change information of the eye opening area; determining eye information of the user according to the eye fatigue degree and / or the at least one second type of eye feature; If the eye fatigue degree is greater than or equal to a first threshold, determining that the eye information is dry eyes; If the similarity between the two eye shapes of the user is less than a second threshold, determining that the eye information is conjunctivitis; If the distance between the pupils is greater than a third threshold, or the distance between the pupils is less than a fourth threshold, it is determined that the eye information is strabismus, and the third threshold is greater than the fourth threshold; If the degree of eye fatigue is greater than or equal to the first threshold, and the distance between the pupils is greater than the third threshold, or the distance between the pupils is less than the fourth threshold, it is determined that the eye information is amblyopia.

8. An image processing device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images