A method and system for detecting changes in facial expression

Abstract

The invention relates to a method, implemented by a processor unit (30) for detecting changes in the facial image received from an image capture unit (20) that provides for capturing at least one facial image of at least one user, comprising the step of capturing a first facial image of the user from the image capture unit (20) for a first reference period, a second facial image for a period of time during which a first sound is played from an audio unit (40).

Description

[0001] A METHOD AND SYSTEM FOR DETECTING CHANGES IN FACIAL EXPRESSION

[0002] TECHNICAL FIELD

[0003] The invention relates to a method and system for detecting changes in a facial image of at least one person.

[0004] PRIOR ART

[0005] In a neurosurgery clinic, evaluating the facial expression movements and blinking of Parkinson’s patients before and after surgery is of critical importance in determining the effectiveness of the treatment process. Similarly, monitoring the changes and movements of patients’ facial expressions in fields such as psychiatry, neurology, and plastic surgery plays a vital role in the diagnosis of diseases and in the follow-up of treatment processes. Monitoring mimic movements enables the detection of changes in the patient's facial expressions and provides an objective evaluation of the responses to treatment.

[0006] In current clinical practices, the evaluation of facial expression changes and the number of blinks is generally performed by doctors and specialists. In the evaluation process, specialists observe the changes in the patient’s facial expressions and evaluate these data by associating them with the treatment process. However, since this method largely depends on the personal experience and evaluation abilities of the specialists, it results in a subjective evaluation of the patient. This reduces the objectivity and accuracy of the obtained results. In particular, the inability to ensure evaluation consistency among different specialists leads to a weakening of the clinical reliability of the obtained data.

[0007] All these may lead to differences in assessment between specialists and different interpretations of the same patient by different specialists. This situation may cause inconsistencies in treatment processes and potentially erroneous clinical decisions. In addition, there is a risk that facial changes, especially subtle and small-scale ones, may be overlooked. Therefore, failure to reliably detect such subtle changes leads to deficiencies in both diagnosis and treatment processes.

[0008] Application number CN1 10991348A discloses a system for detecting microexpressions on a human face. The system enables the detection of micro-expressions by utilising the optical flow field between facial images. Patent application CN1 10991348A does not mention a system for detecting the number of blinks and the rate of change of facial expressions.

[0009] All the problems mentioned above have made it necessary to make an innovation in the relevant technical field as a result.

[0010] BRIEF DESCRIPTION OF THE INVENTION

[0011] The present invention relates to a method and system to eliminate the above- mentioned disadvantages and bring new advantages to the relevant technical field.

[0012] An object of the invention is to provide a method and system that enables the objective detection of changes occurring in facial expressions.

[0013] Another object of the invention is to provide a method and system that enables the diagnosis and follow-up processes of Neurosurgery, Psychiatry, Neurology and Plastic surgery patients to be performed accurately by detecting the changes in facial expression.

[0014] To achieve all the objects mentioned above and that will emerge from the following detailed description, the invention is a method, implemented by a processor unit for detecting changes in the facial image received from an image capture unit that provides for capturing at least one facial image of at least one user, comprising the step of capturing a first facial image of the user from the image capture unit for a first reference period, a second facial image for a period of time during which a first sound is played from an audio unit. Accordingly, it comprises the steps of receiving a mode input from a user interface regarding selection of at least one of a first operating mode, a second operating mode and a third operating mode, in the case where the mode input relates to the first operating mode, applying the first facial image as input to a first mathematical model that gives the location information of the eye in the facial image as output when it receives a facial image trained with the human face and the location of the eye on the human face as input, and obtaining the location information of the eyes in the first facial image as output, marking of the eye region using the eye location information, detecting the longest horizontal measurement of the eye region measured along the horizontal plane and the longest vertical measurement of the eye region measured along the vertical axis from the marked image, comparing the value obtained by the ratio of the vertical measurement to the horizontal measurement with a reference value, detecting of the user's blinking in case the value is found to be less than the reference value and determination of the total number of blinks detected during the first facial image to be presented to the user, in the case where mode input relates to the second operating mode, determining a first axis passing through a first reference point on a marked image frame, and a second axis intersecting the first axis at right angles to the first reference point, determining the motion of the first reference facial points in the marked image frame with respect to the first axis and the second axis, and obtaining a first distance change data to be presented to the user, in the case where mode input relates to the third operating mode, separating of the second facial image into at least one image frame, detecting the difference between a first time at which the first image frame of the second facial image is acquired and a second time at which the last image frame is acquired; determining the mimic change speed data stored in a memory unit corresponding to the difference, to be presented to the user.

[0015] A possible embodiment of the invention is characterized comprising the steps of; after the step of capturing a first facial image of the user from the image capture unit for a first reference period, a second facial image for a period of time during which a first sound is played from an sound unit, separating the first facial image into at least one first image frame, applying at least one first image frame in a first facial image as input to a second mathematical model that gives the first facial information in the first facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtaining the first facial information in the first facial image as output, changing the colour of the image to a format by marking the first facial in the first image frame applying the first facial image frame in a first facial image as input to a third mathematical model that gives the first reference facial points in a first facial image as output when it receives a facial image frame trained with facial image frames as input, and obtaining the first reference facial points in the first facial image as output, obtaining the first marked image frame by marking the first facial in the first image frame of the first facial image according to the first reference facial points steps in the process.

[0016] Another possible embodiment of the invention is characterized comprising the steps of after the step of capturing a first facial image of the user from the image capture unit for a first reference period, a second facial image for a period of time during which a first sound is played from an audio unit, separating the second facial image into at least one second image frame, applying at least one second image frame in a second facial image as input to a second mathematical model that gives the second facial information in the second facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtaining the second facial information in the second facial image as output, changing the colour of the image to the first format by marking the second facial in the second image frame, applying the second facial image frame as input to a third mathematical model that gives the reference facial points on the face as output when it receives second reference facial points on the second facial image trained with facial image frame as input, and obtaining the second reference facial points in the first facial image as output, obtaining the second marked image frame by marking the second face in the second image frame of the second facial image according to the second reference facial points steps in the process.

[0017] Also the invention is a system comprising an image capture unit for capturing at least one facial image of at least one user, a processor unit for detecting changes in the facial image captured from said image capture unit. Accordingly, the processor unit is configured to perform the steps of a method as in any one of the preceding claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a drawing illustrating schematic view of the working scenario of the invention.

[0019] DETAILED DESCRIPTION OF THE INVENTION

[0020] In this detailed description, the subject matter is explained with references to examples without forming any restrictive effect only in order to make the subject more understandable.

[0021] As shown in Figure 1 , the invention relates to a method and system (10) for detecting changes in a facial image of at least one user. The system (10) comprises an image capture unit (20) for capturing at least one facial image of the user. In a possible embodiment of the invention, the image capture unit (20) may be a mobile device camera. In a possible embodiment of the invention, the user may be a patient of neurosurgery, psychiatry, neurology, plastic surgery, etc.

[0022] The system (10) comprises a processor unit (30) for detecting changes in the facial image received from the image capture unit (20). In a possible embodiment of the invention, the processor unit (30) may be a microprocessor.

[0023] The processor unit (30) enables a first facial image of the user to be taken from the image capture unit (20) during a first reference period, and a second facial image of the user to be taken from the image capture unit (20) during a playback time of a first sound played from an sound unit (40). In a possible embodiment of the invention, the facial image may be a facial video image of the user. In a possible embodiment of the invention, the first reference period may be one minute. In this way, a one minute video image of the user's face is taken.

[0024] In one possible embodiment of the invention, the sound unit (40) may be a speaker. In another possible embodiment of the invention, the first sound may be a sound file of laughter played intermittently at regular intervals. In a possible embodiment of the invention, the first sound may be a 30-second sound file. For example, the first sound may include silence for the first 5 seconds, a laughter sound from the 5th second to the 10th second, silence again from the 10th second to the 15th second, a laughter sound again from the 15th second to the 20th second, silence again from the 20th second to the 25th second, and a laughter sound again from the 25th second to the 30th second. In this way, the user is enabled to laugh during the intervals when the laughter sound is played.

[0025] The system (10) comprises a user interface (50) for inputting user information. In a possible embodiment of the invention, the user information may include information such as name, age, gender, etc. In a possible embodiment of the invention, the user interface (50) may be a mobile application, a website, etc., which is accessed via a mobile device such as a computer, tablet, mobile phone, etc.

[0026] The processor unit (30), in the case where the mode input relates to the first operating mode, apply the first facial image as input to a first mathematical model that gives the location information of the eye in the facial image as output when it receives a facial image trained with the human face and the location of the eye on the human face as input, and obtain the location information of the eyes in the first facial image as output. In a possible embodiment of the invention, the first mathematical model is an artificial intelligence model.

[0027] The processor unit (30) enables the marking of the eye region based on the location information of the eye and the detection of the longest horizontal measurement of the eye region measured along the horizontal plane and the longest vertical measurement of the eye region measured along the vertical axis from the marked image. In a possible embodiment of the invention, the vertical measurement corresponds to the measurement of the user's palpebral fissure. In another possible embodiment of the invention, the horizontal measurement corresponds to the measurement of the distance between the inner and outer corners of the eyes (medial and lateral canthus). The processor unit (30) enables the comparison of the value obtained by the ratio of the vertical measurement to the horizontal measurement with a reference value and in the case where it is determined that the mentioned value is less than the reference value, the detection that the user has blinked, and the determination of the total number of blinks detected throughout the first facial image to be presented to the user. In a possible embodiment of the invention, the reference value may be 0.25. Accordingly, if the value obtained by the ratio of the vertical measurement to the horizontal measurement is less than 0.25, it can be determined that the user has blinked. The user is determined to have blinked only once until the value obtained by the ratio of the vertical measurement to the horizontal measurement becomes greater than 0.25.

[0028] In a possible embodiment of the invention, after acquiring a first facial image of the user during a first reference period from an image capture unit (20) and a second facial image during the playback time of a first sound played from a sound unit (40), the processor unit (30) enables the first facial image to be divided into at least one first image frame.

[0029] The processor unit (30) enables applying at least one first image frame in a first facial image as input to a second mathematical model that gives the first face information in the first facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtain the first face information in the first facial image as output. In a possible embodiment of the invention, the second mathematical model is an artificial intelligence model.

[0030] The processing unit (30) enables the colour of the image to be change to a format by marking the first face in the first image frame. In a possible embodiment of the invention, the format may be a grey scale (black and white) format.

[0031] The processor unit (30) provide to applying the first facial image frame in a first facial image as input to a third mathematical model that gives the reference facial points in a first facial image as output when it receives a facial image frame trained with facial image frames as input, and obtaining the reference facial points in the first facial image as output. In a possible embodiment of the invention, the third mathematical model may be a face key detector (Dlib) model. The facial key detector, as is well known in the art, enables the detection of certain key points (characteristic points) on a human face in an image or video. In a possible embodiment of the invention, the key points may be points representing the structure of the face, such as the corners of the eyes, the tip of the nose, the corners of the lips, etc., located in certain areas of the face.

[0032] In a possible embodiment of the invention, the first reference facial points comprise 68 key points on the face. The processor unit (30) enables the coordinates of the 68 key points to be determined and stored in a memory unit. In a possible embodiment of the invention, the memory unit may comprise a combination of memories for storing data permanently and for storing data temporarily when required.

[0033] The processor unit (30) enables the first marked image frame to be obtained by marking the first face in the first image frame of the first facial image according to the first reference facial points.

[0034] In a possible embodiment of the invention, after acquiring a first facial image of the user during a first reference period from an image capture unit (20) and a second facial image during the playback time of a first sound played from a sound unit (40), the processor unit (30) enables the second facial image to be divided into at least one second image frame.

[0035] The processor unit (30) ensures that apply at least one second image frame in a second facial image as input to a second mathematical model that gives the second face information in the second facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtain the second face information in the second facial image as output.

[0036] The processor unit (30) enables the colour of the image to be changed to the first format by marking the second face in the second image frame. In a possible embodiment of the invention, said first format may be a grey scale (black and white) format.

[0037] The processor unit (30) provides ensures that applying the second face image frame as input to a second mathematical model that gives the reference facial points on the face as output when it receives second reference facial points on the second facial image trained with face image frame as input, and obtaining the second reference facial points in the first face image as output.

[0038] The processor unit (30) enables the second marked image frame to be obtained by marking the second face in the second image frame of the second facial image according to the second reference facial points.

[0039] In a possible embodiment of the invention, the second operating mode may be a mode of operation for detecting the mimic distance change of the user.

[0040] In a possible embodiment of the invention, the processor unit (30) for detecting the mimic distance change comprises determining a first axis passing through a first reference point on a marked image frame and a second axis intersecting the first axis at right angles to the first reference point. In a possible embodiment of the invention, the marked image frame may be a second marked image frame.

[0041] In a possible embodiment of the invention, the first reference point may be the outer corners (lateral canthus) of both eyes of the user. In a possible embodiment of the invention, the first axis is an x-axis. Accordingly, the x-axis may be a line passing through the lateral canthus of both eyes of the user. In a possible embodiment of the invention, the second axis is a y-axis. In another possible embodiment of the invention, the y-axis may be a line passing through the root (radix) and tip of the nose of the user. In this way, the effect of the user's head movements on the coordinates is minimised.

[0042] The processor unit (30) determines the movement of the first reference facial points in the marked image frame with respect to the first axis and the second axis, and obtains a first distance change data to be presented to the user.

[0043] In a possible embodiment of the invention, the first reference facial points comprise points associated with a smile of the user. For example, the first reference facial points may be the corner of the right mouth (right mouth commissure), the corner of the left mouth (left mouth commissure), the inner part (medial side) of the right eyebrow, the inner part (medial side) of the left eyebrow. In a possible embodiment of the invention, the movements of these facial points with respect to the first axis and the second axis are determined.

[0044] In another possible embodiment of the invention, the first reference points may be the middle and upper point of the right eye, the middle and upper point of the left eye, the middle and lower point of the right eye, the middle and lower point of the left eye. In a possible embodiment of the invention, the movements of these facial points relative to the second axis are determined.

[0045] In a possible embodiment of the invention, the processor unit (30) may enable the first distance change data to be presented to the user as a graphical table and / or excel file. In another possible embodiment of the invention, the excel file may comprise mean, median, standard deviation, minimum and maximum values of the vertical and horizontal movements of the determined facial points in the image frames. In a possible embodiment of the invention, the first distance change data may be determined by the difference between the minimum and maximum values.

[0046] In a possible embodiment of the invention, the third operating mode may be an operating mode that enables the determination of the user's facial expression change speed. The processor unit (30) enables the division of the second facial image into at least one image frame and the detection of the difference between a first time at which the first image frame of the second facial image is acquired and a second time at which the last image frame is acquired.

[0047] In a possible embodiment of the invention, the first time at which the first image frame in the second facial image is taken may be the time at which the user starts laughing. In another possible embodiment of the invention, the second time when the last image frame is taken may be the time when the user has finished laughing.

[0048] The processor unit (30) enables the determination of the facial expression change speed data stored in the memory unit corresponding to the difference, to be presented to the user. In a possible embodiment of the invention, the facial expression change speed data may be the speed of the facial expression change in seconds. In a possible embodiment of the invention, the number of blinks, mimic distance change and mimic change rate can be monitored in real time. In a possible embodiment of the invention, facial images can be displayed via the user interface (50).

[0049] An example operating scenario of the invention is given below;

[0050] In order to enable the detection of changes in a patient's facial image, the user may be seated on a chair in a clinical room with a plain white background. A phone is placed on a tripod at a distance of approximately 75 cm from the patient to record a video of the patient's face.

[0051] A video recording of the patient is taken for one minute to determine the number of blinks. This one-minute video contains the patient's natural facial expression. Then, in order to determine the rate of mimic change and mimic distance change, a 30-second sound file is played from the speaker and a video recording of the patient laughing at certain intervals is taken. The sound file may include the sound of a person laughing at specific intervals. In this way, a video recording of the patient is taken to analyze the muscle movements on the patient’s face, particularly the movements of the muscles around the mouth and eyes.

[0052] After the end of the video recording, one of the operating modes can be selected and the patient's blink count, gesture change rate or gesture distance changes can be detected. For example, when the blink rate of the patient is to be determined, the blink rate of the patient is calculated from the images obtained from the one-minute video recording. In this way, it can be evaluated whether the patient's blink rate is within a normal range.

[0053] The mode can be selected to determine the patient's mimic change rate and mimic distance change. Accordingly, when the patient's mimic distance change is selected, the distance travelled by the corners of the mouth along the horizontal and vertical axes during the patient's laughter is measured. For example, it can be calculated how much the distance between the right and left mouth corners changes during laughter. These distance changes allow the patient's laughing intensity and muscle movements to be measured.

[0054] As another option, when the calculation of the patient's facial expression change is selected, how quickly the patient's facial expressions change during laughing is determined. The speed of muscle movements, which begin with the upward movement of the mouth corners during laughing and return to the initial position when the laughter ends, can be measured in seconds. This speed may indicate how flexible and responsive the patient's facial muscles are.

[0055] The data obtained for the number of blinks, rate of facial expression change and facial expression distance change are presented to the users through the user interface (50). The data can be presented graphically or in excel format, and it can be shown in detail how fast the changes in the patient's facial expression occur and how large these changes are. In this way, it is possible to objectively evaluate the changes in the face of normal pressure hydrocephalus, psychiatry, neurology or plastic surgery patients and to plan the treatment process correctly accordingly.

[0056] The scope of protection of the invention is specified in the attached claims and cannot be limited to those explained for sampling purposes in this detailed description. It is evident that a person skilled in the art may exhibit similar embodiments in light of the above-mentioned facts without drifting apart from the main theme of the invention.

[0057] REFERENCE NUMBERS GIVEN IN THE FIGURE

[0058] 10 System

[0059] 20 Image capture unit 30 Processor units

[0060] 40 Sound units

[0061] 50 User interface

Claims

CLAIMS1 . A method implemented by a processor unit (30) for detecting changes in the facial image received from an image capture unit (20) that provides for capturing at least one facial image of at least one user, comprising the step of- capturing a first facial image of the user from the image capture unit (20) for a first reference period, a second facial image for a period of time during which a first sound is played from an audio unit (40) characterized in that it comprises the step of;- receiving a mode input from a user interface (50) regarding selection of at least one of a first operating mode, a second operating mode and a third operating mode, in the case where the mode input relates to the first operating mode,- applying the first facial image as input to a first mathematical model that gives the location information of the eye in the facial image as output when it receives a facial image trained with the human face and the location of the eye on the human face as input, and obtaining the location information of the eyes in the first facial image as output,- marking of the eye region using the eye location information,- detecting the longest horizontal measurement of the eye region measured along the horizontal plane and the longest vertical measurement of the eye region measured along the vertical axis from the marked image,- comparing the value obtained by the ratio of the vertical measurement to the horizontal measurement with a reference value,- detecting of the user's blinking in case the value is found to be less than the reference value and determination of the total number of blinks detected during the first facial image to be presented to the user, in the case where mode input relates to the second operating mode,- determining a first axis passing through a first reference point on a marked image frame, and a second axis intersecting the first axis at right angles to the first reference point,- determining the motion of the first reference facial points in the marked image frame with respect to the first axis and the second axis, and obtaining a first distance change data to be presented to the user, in the case where mode input relates to the third operating mode,- separating of the second facial image into at least one image frame,- detecting the difference between a first time at which the first image frame of the second facial image is acquired and a second time at which the last image frame is acquired;- determining the mimic change speed data stored in a memory unit corresponding to the difference, to be presented to the user.

2. A method according to claim 1 , characterised in that, after the step of capturing a first facial image of the user from the image capture unit (20) for a first reference period, a second facial image for a period of time during which a first sound is played from an sound unit (40),- separating the first facial image into at least one first image frame,- applying at least one first image frame in a first facial image as input to a second mathematical model that gives the first facial information in the first facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtaining the first facial information in the first facial image as output,- changing the colour of the image to a format by marking the first facial in the first image frame- applying the first facial image frame in a first facial image as input to a third mathematical model that gives the reference facial points in a first facial image as output when it receives a facial image frame trained with facial image frames as input, and obtaining the first reference facial points in the first facial image as output,- obtaining the first marked image frame by marking the first face in the first image frame of the first facial image according to the first reference facial points, steps in the process.

3. A method according to claim 1 , characterised in that, after the step of capturing a first facial image of the user from the image capture unit (20) for a first reference period, a second facial image for a period of time during which a first sound is played from an audio unit (40),- separating the second facial image into at least one second image frame,- applying at least one second image frame in a second facial image as input to a second mathematical model that gives the second facial information in the second facial image as output when it receives a facial image frame trained with recognising a face in a facial image frame as input, and obtaining the second facial information in the second facial image as output,- changing the colour of the image to the first format by marking the second face in the second image frame,- applying the second facial image frame as input to a third mathematical model that gives the reference facial points on the face as output when it receives second reference facial points on the second facial image trained with facial image frame as input, and obtaining the second reference facial points in the first facial image as output,- obtaining the second marked image frame by marking the second facial in the second image frame of the second facial image according to the second reference facial points, steps in the process.

4. A system (10) comprising an image capture unit (20) for capturing at least one facial image of at least one user, a processor unit (30) for detecting changes in the facial image captured from said image capture unit (20), characterised in that the processor unit (30) is configured to perform the steps of a method as in any one of the preceding claims.

Images