Eye use monitoring method and device

Abstract

The invention discloses an eye use monitoring method and device, and belongs to the field of computer image processing and AI video processing. The method comprises: acquiring a face image; extracting an eye image from the face image; acquiring an eye using condition according to the eye image; and when the eye using condition reaches a threshold value, warning information is sent out. The head image of a computer user is collected in real time through the camera, image data are processed in real time, and whether warning needs to be given or not is judged according to a calculation result. According to the method, specific hardware does not need to be bound, the scheme can be deployed as long as a computer is provided with a camera, and the method has the characteristics of quick and simple deployment and low cost; the distance between the human eyes and the displayer can be accurately measured, the time when the eyes watch the displayer is reasonably counted, warning is conducted in time, and eye health of a user is ensured.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer image processing and AI video processing, and in particular to an eye monitoring method and device. Background Art

[0002] The widespread use of electronic products (such as computers, tablets, mobile phones, etc.) can easily lead to eye health problems. Currently, eye monitoring of electronic products mostly requires specific hardware support. These hardware have eye monitoring functions, but they cannot be embedded in electronic products, which makes electronic product eye monitoring inconvenient and difficult to promote. Summary of the invention

[0003] The present invention provides an eye monitoring method and device, which can solve the problem that the eye monitoring of electronic products in the related art depends on specific hardware. The technical solution is as follows:

[0004] On the one hand, an eye monitoring method is provided, the method comprising: acquiring a facial image of a user; extracting an eye image from the facial image; acquiring eye usage conditions based on the eye image; and issuing a warning message when the eye usage conditions reach a threshold.

[0005] Optionally, the eye usage conditions include the duration of eye usage and the distance from the eyes to the display.

[0006] Optionally, the method for obtaining the eye usage time includes: judging whether the eyes are looking at the display according to the eye image, and calculating the eye usage time according to the judgment result.

[0007] Optionally, the method for acquiring the distance from the eye to the display includes: acquiring the distance from the eye to the display according to the focal length of a camera used to capture the facial image, the interocular distance of the user, and the interocular distance in the eye image.

[0008] Optionally, the method for acquiring the focal length of the camera includes: reading configuration information of the camera to acquire the focal length of the camera.

[0009] Optionally, the method for acquiring the focal length of the camera includes: acquiring an image of a target object, i.e., a target image, through the camera; and calculating the focal length of the camera according to the size of the target object, the size of the target object in the target image, and the distance between the target object and the camera.

[0010] On the other hand, an eye monitoring device is provided, which includes: a first image processing unit, suitable for obtaining a facial image of a user; a second image processing unit, suitable for extracting an eye image from the facial image; a data processing unit, suitable for obtaining eye usage conditions based on the eye image; and a warning unit, suitable for issuing a warning message when the eye usage condition reaches a threshold value.

[0011] On the other hand, a computer device is provided, which includes a memory and a processor, wherein the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the above-mentioned eye monitoring method.

[0012] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned eye monitoring method are implemented.

[0013] On the other hand, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned eye monitoring method when executed by a processor.

[0014] The technical solution provided by the present invention collects the facial image of the computer user (ie, user) in real time through a camera, processes the image data in real time, obtains the eye usage situation according to the processing result, and determines whether a warning is needed.

[0015] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0016] (1) This solution does not require binding to specific hardware. As long as the computer is equipped with a camera, the solution can be deployed. It has the characteristics of fast, simple and low-cost deployment.

[0017] (2) It can accurately measure the distance between the human eye and the display, reasonably calculate the time the eyes spend looking at the display, and issue timely warnings to ensure the user's eye health. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 is a flow chart of an eye monitoring method provided by an embodiment of the present invention;

[0020] Figure 21 is a schematic diagram of the calculation principle of the distance between the user's eyes and the display provided by an embodiment of the present invention;

[0021] Figure 3 is a flow chart of a method for obtaining camera focal length provided by an embodiment of the present invention;

[0022] Figure 4 is a schematic diagram of a computer provided by an embodiment of the present invention;

[0023] Figure 5 is a schematic diagram of an AI software architecture provided by an embodiment of the present invention;

[0024] Figure 6 is a structural diagram of an eye monitoring device provided by an embodiment of the present invention;

[0025] Figure 7 It is a hardware architecture diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] As mentioned above, the existing technology relies on specific external hardware devices when performing eye monitoring on electronic products, so the implementation of eye monitoring on electronic products is not convenient enough.

[0028] Based on this, an embodiment of the present invention provides an electronic product eye monitoring solution that does not require binding to specific hardware. The solution is used for electronic products with cameras to monitor and detect the behavior of people (i.e., users) using electronic products, including the distance between the human eyes and the screen and the length of time staring at the screen. Once the distance between the human eyes and the screen is too close or the eyes are in use for too long, timely reminders are given to ensure the user's eye health. This solution is especially important for teenagers.

[0029] The specific implementation of the above concept is described below.

[0030] Please refer to Figure 1 , an embodiment of the present invention provides an eye monitoring method, the method comprising:

[0031] Step 101, obtaining a user's facial image;

[0032] Step 102, extracting an eye image from the facial image;

[0033] Step 103, obtaining eye usage conditions according to the eye image;

[0034] Step 104, when the eye usage reaches a threshold, a warning message is issued.

[0035] In an embodiment of the present invention, a camera is used to obtain a facial image of a user. The camera may be a built-in camera of an electronic product or an external camera of an electronic product. For an external camera, the direction of its lens should be the same as the direction of the electronic product display to ensure that the user is within the camera's field of view when using the electronic product. In addition, the camera lens should be in the same plane as the electronic product display, or the distance between the plane where the camera lens is located and the plane where the electronic product display is located is small enough (e.g., within 10 cm).

[0036] Described below Figure 1 How the various steps are performed.

[0037] First, for step 101, the camera collects images in real time and outputs a video stream. Step 101 processes each image collected by the camera, extracts each target in the image, and classifies each extracted target. When a target is determined to be a face, a face image, that is, a facial image of the user, is extracted. Next, enter step 102.

[0038] Then, for step 102, an eye image is extracted from the facial image. The purpose of this step is to detect key points of the eye, mainly including key points of the eye such as eye contour and pupil, and mark the detected key points. Next, enter step 103.

[0039] Then, for step 103 and step 104, the eye usage condition is obtained according to the eye image, and a warning message is issued when the eye usage condition reaches a threshold.

[0040] In the embodiment of the present application, the eye usage mainly refers to the time the eyes look at the electronic product display and the distance between the eyes and the electronic product display.

[0041] The state recognition is performed based on the eye image extracted in the previous step, and the states mainly include looking at the display and not looking at the display.

[0042] The time that the user's eyes are fixed on the electronic product display can be determined by performing state recognition on the eye image.

[0043] In one implementation, the state of each eye image extracted from the video stream is identified, and the state identification result is recorded. The recorded items include the shooting time of the original image of the video stream corresponding to the eye image, and the state recognition result of the eye image. The timing starts from the appearance of the first image in the video stream in which the user is looking at the display, so as to monitor the time the user continues to look at the display. During the monitoring process, if the state recognition result of the eye image changes to not looking at the display at a certain moment, this moment is recorded as t1. After t1, when the state recognition result of the eye image changes to looking at the display at a certain moment, this moment is recorded as t2. The time interval δ between t1 and t2 is calculated, δ=t2-t1. If δ is less than the preset first threshold δ th , then ignore this time interval and continue timing. If δ is equal to or exceeds the preset first threshold δ th , then the timing starts again from t2. During the monitoring process, if the time Δ for which the user continues to stare at the display reaches a preset second threshold Δ th , a warning message is issued to the user.

[0044] For example, setting the first threshold δ th For 2 minutes, the second threshold Δ th 1 hour. After the electronic device is turned on, the camera starts to collect images and forms a video stream output. The moment when the first image of the state of looking at the display appears in the video stream is recorded as t0, and the timing starts from t0. Assume that the user's state from t0 to the previous moment t1 has been looking at the display, and at t1 the user turns his eyes to the keyboard, and at t2 he turns his eyes back to the display. The time interval δ between t1 and t2 is calculated to be 10 seconds, which is less than the first threshold δ th , then ignore this time interval δ, that is, it is considered that the user has been looking at the display from time t1 to time t2. Assume that the user leaves the electronic device at time t3, that is, the facial image cannot be extracted from the image captured by the camera at time t3, and the user returns to the display to continue using the electronic device at time t4. The calculated time interval δ between t3 and t4 is 11 minutes, which is greater than the first threshold δ th , it is considered that the user's eyes have been fully relaxed during the period from time t3 to time t4, and then the timing will be restarted with time t4 as the new starting point.

[0045] For another example, setting the first threshold δ th For 2 minutes, the second threshold Δ th1 hour. After the electronic device is turned on, the camera starts to collect images and forms a video stream output. The moment when the first image of the state of looking at the display appears in the video stream is recorded as t0, and the timing starts from t0. Assume that the user's state from t0 to the previous moment t1 has been looking at the display, and at t1 the user turns his eyes to the keyboard, and at t2 he turns his eyes back to the display. The time interval δ between t1 and t2 is calculated to be 10 seconds, which is less than the first threshold δ th , then ignore this time interval δ, that is, it is considered that the user has been looking at the display from time t1 to time t2. After time t2, the user's eyes continue to look at the display until time t3. The time difference Δ between t0 and t3 is calculated to be 1 hour, reaching the preset second threshold Δ th , a warning is immediately issued to the user.

[0046] In another implementation, while monitoring the eye usage, the user is identified and marked to monitor the eye usage of different users separately. The timing starts from the appearance of the first image in the video stream in which the user is looking at the display, and the features of the user who is looking at the display identified in the image are collected. At the same time, an ID is assigned to the user and recorded. During the monitoring process, when a user leaves the display for a period of time, it may be another user who returns to the display. In step 103, if the facial image cannot be extracted from the video stream captured by the camera within a period of time, that is, it is monitored that the user has left the electronic device during this period of time, when the user's face is monitored again to appear in the camera's field of view, the features of the user who is looking at the display identified in the image at the current moment are collected, and it is determined based on the collected features whether the current user represents the same user as a certain ID that has been recorded. If so, it is determined based on the time when the user last left the electronic device whether the length of time the user has left the electronic device reaches the first threshold δ. th If the current user does not belong to the user whose ID has been recorded, a new ID is assigned to the current user, and the ID is recorded, and then the current user is monitored. The specific monitoring method refers to the previous implementation method.

[0047] The distance from the eye to the electronic product display can be calculated based on the principle of single-camera ranging. Single-camera ranging is a method of obtaining object depth information using a target image obtained by a camera under the condition of known object information. Single-camera ranging is based on the pinhole imaging model (also known as the linear camera model) and uses the principle of similar triangles to calculate the distance from the camera to a known object or target.

[0048] Figure 2 The figure shows the principle of single camera ranging. Figure 2As shown, A is the pixel plane, that is, the plane where the image of the target object is located, and B is the plane where the camera lens is located. Assuming that the focal length of the camera is F, the width (or length) of the target object is W, the target object is placed in front of the camera and the distance between the camera and the camera is D. The target object is photographed with the camera, and the pixel width (or length) of the target object in the pixel plane is measured to be P, that is, the width of the target object in the photo is P. Then, according to the principle of similar triangles, the distance D between the target object and the camera is calculated as:

[0049] D=(F·W) / P

[0050] In the embodiment of the present application, when calculating the distance from the user's eyes to the display, the target object is the user's eyes, and the width of the target object is the pupil distance between the user's eyes. The statistical average of the human pupil distance is used here. It is known that the average pupil distance of Chinese men is 66mm, the average pupil distance of women is 61mm, and the average pupil distance of children is 52mm.

[0051] In one implementation, the camera focal length can be obtained by reading the configuration information of the hardware device. If the configuration information of the hardware device does not include the camera focal length, another implementation can be used to try to obtain it.

[0052] In another implementation, according to Figure 2 The focal length is calculated by the principle shown in the figure. The focal length is obtained by Figure 3 As shown, the specific steps include: step 301, prompting the user to place an object in front of the camera, the object is the target object, and measuring the width of the target object and the distance D from the target object to the camera. The target object can be, for example, an A4 paper; step 302, controlling the camera to take a picture of the target object to obtain an image of the target object, that is, a target image; step 303, processing the target image to obtain a pixel width P of the target object in the target image; step 304, calculating the focal length of the camera according to the formula D=(F·W) / P.

[0053] After obtaining the focal length of the camera, the distance from the user's eyes to the camera can be monitored using steps 101 to 103. It should be noted that for a built-in camera, it can be considered that the camera and the display are in the same plane, while the size of an external camera is larger. Usually, after the camera is installed, there will be a certain distance between the plane where its lens is located and the plane where the display is located. Therefore, during the monitoring process, the distance from the camera to the display should be considered based on the calculated distance D from the eyes to the camera. For example, the plane where the display is located is taken as the reference plane, and the display is facing the positive direction, that is, the user is in front of the display, and the computer prompts the user to input the camera location in front of / behind the display, and the distance d from the camera to the plane where the display is located. If the camera is in front of the display, then the distance from the user's eyes to the display is D+d, and if the camera is behind the display, then the distance from the user's eyes to the display is Dd.

[0054] For each image in the state of looking at the display, it is necessary to calculate the distance between the user's eyes and the display, and judge this distance. If the distance between the user's eyes and the display is equal to or less than the preset third threshold D th , a warning message is immediately sent to the user.

[0055] The warning message may be a pop-up warning message on the screen to warn the user that the eyes are too close to the screen / the time spent staring at the screen is too long, etc., and a sound prompt may also be output at the same time to warn the user.

[0056] The eye monitoring method of the embodiment of the present application can provide the user with an eye monitoring function in the form of AI software. Electronic products are the carriers of AI software. AI software is installed and runs in the operating system of electronic products. Electronic products need to be equipped with cameras. The cameras can be built-in cameras or external cameras. Figure 4 As shown in the figure, the camera is an external camera, and the AI ​​software is installed on the computer. The AI ​​software automatically starts running when the computer is turned on. When the focal length of the camera is unknown, the AI ​​software will obtain the focal length of the camera according to steps 301 to 304. If it is detected that the camera and the computer transmit signals through the USB interface, it means that the camera is an external camera. At this time, the user needs to be prompted to enter the distance between the camera in front of / behind the monitor and the distance between the camera lens and the plane where the monitor is located.

[0057] like Figure 5As shown in FIG. 1 , the above AI software architecture mainly includes four modules: face recognition (face detection), eye key point recognition (facial landmark), state recognition (eye status) and data processing (post processing). Specifically, the open source multimedia framework gstreamer technology can be used to build a complete video stream processing channel Gstreamerpipeline. At the video source end, configure the output format and frame rate of the camera; at the video processing end, embed the face detection module, facial landmark module, eye status module, and post processing module in the gstreamerpipeline in sequence. Among them, the face detection, facial landmark, and eye status three AI modules are used for face recognition, eye key point extraction, and eye state recognition respectively. The post processing module receives the output information of the above three AI modules, calculates the distance between the eyes and the screen and the length of time the eyes stare at the screen, and compares it with the preset threshold value, and decides whether to perform warning processing according to the comparison result. The time the eyes stare at the screen is completed by the eye status and post processing modules. The eye status module determines whether the eyes are staring at the screen, and the post processing module calculates the length of time the eyes stare at the screen according to the output of the eye status module. When the distance between the human eye and the screen is too close, or the time the human eye continues to stare at the screen exceeds the threshold, the AI ​​software will give timely reminders, including displaying warning messages on the screen, or making sound warnings at the same time, so as to remind the user to adjust their sitting posture, prevent the eyes from being too close to the screen, and rest the eyes properly to use the computer in a healthy way.

[0058] Please refer to Figure 6 An embodiment of the present invention provides an eye monitoring device, which includes a first image processing unit 601, a second image processing unit 602, a data processing unit 603, and a warning unit 604.

[0059] The first image processing unit 601 is adapted to obtain a facial image of a user.

[0060] The second image processing unit 602 is adapted to extract an eye image from the facial image.

[0061] The data processing unit 603 is adapted to obtain eye usage conditions according to the eye image.

[0062] The warning unit 604 is adapted to issue a warning message when the eye usage condition reaches a threshold.

[0063] As a preferred embodiment of the present application, the eye usage condition includes the eye usage time and the distance from the eyes to the display.

[0064] As a preferred embodiment of the present application, the method for obtaining the eye use time includes:

[0065] It is determined whether the eyes are looking at the display according to the eye image, and the eye use time is calculated according to the determination result.

[0066] As a preferred embodiment of the present application, the method for obtaining the distance from the eye to the display includes:

[0067] The distance from the eye to the display is acquired according to the focal length of the camera used to collect the facial image, the interocular distance of the user, and the interocular distance in the eye image.

[0068] As a preferred embodiment of the present application, the method for acquiring the focal length of the camera includes: reading configuration information of the camera to acquire the focal length of the camera.

[0069] As a preferred embodiment of the present application, the method for obtaining the focal length of the camera includes:

[0070] Acquire an image of a target object, i.e., a target image, through the camera;

[0071] The focal length of the camera is calculated according to the size of the target object, the size of the target object in the target image, and the distance between the target object and the camera.

[0072] It should be noted that the eye monitoring device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the eye monitoring device provided in the above embodiment and the eye monitoring method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0073] The embodiment of the present application also provides a computer device, please refer to Figure 7 The computer device includes a processor and a memory, in which at least one instruction, at least one program, a code set or an instruction set is stored, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the eye monitoring method provided by the above-mentioned method embodiments.

[0074] An embodiment of the present application also provides a computer-readable storage medium, on which is stored at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the eye monitoring method provided by the above-mentioned method embodiments.

[0075] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the eye monitoring method described in any of the above embodiments.

[0076] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0077] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0078] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0079] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for monitoring eye use, characterized in that: The method comprises: Get the user's facial image; extracting an eye image from the facial image; Acquiring eye usage conditions according to the eye image; and When the eye usage situation reaches a threshold, a warning message is issued.

2. The method according to claim 1, characterized in that The eye usage conditions include the length of time the eyes are used and the distance from the eyes to the display.

3. The method according to claim 2, characterized in that The method for obtaining the eye use time comprises: It is determined whether the eyes are looking at the display according to the eye image, and the eye use time is calculated according to the determination result.

4. The method according to claim 2, characterized in that The method for obtaining the distance from the eye to the display includes: The distance from the eye to the display is acquired according to the focal length of the camera used to collect the facial image, the interocular distance of the user, and the interocular distance in the eye image.

5. The method according to claim 4, characterized in that The method for acquiring the focal length of the camera includes: Read the configuration information of the camera to obtain the focal length of the camera.

6. The method according to claim 4, characterized in that The method for acquiring the focal length of the camera includes: Acquire an image of a target object, i.e., a target image, through the camera; The focal length of the camera is calculated according to the size of the target object, the size of the target object in the target image, and the distance between the target object and the camera.

7. An eye monitoring device, characterized in that: The device comprises: A first image processing unit, adapted to obtain a facial image of a user; a second image processing unit, adapted to extract an eye image from the facial image; a data processing unit, adapted to obtain eye conditions according to the eye image; and The warning unit is adapted to issue a warning message when the eye usage condition reaches a threshold value.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images