Auxiliary analysis method and device for patients with mental illness based on facial expression recognition

Abstract

The present invention relates to the field of image processing technology, and specifically to an auxiliary analysis method and device for mental illness patients based on facial expression recognition, including: collecting a number of facial images when a patient communicates with a doctor; obtaining a number of suspected chuan-shaped pattern areas in the glabellar area and a sequence of suspected chuan-shaped pattern areas; obtaining the pupil area of ​​each facial image, obtaining a number of facial expression change moments; obtaining the possibility that the patient's expression is angry at each facial expression change moment; obtaining a number of anger stages of the patient; obtaining the time point when the doctor mentions the patient's keywords each time, obtaining the anger starting stage of each anger stage of the patient, and then obtaining the patient's ability to control his own emotions during the conversation; and obtaining the severity of the patient's mental problems. The present invention assists doctors in judging the patient's mental state and improves the diagnosis and treatment effect by analyzing the patient's facial changes.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to an auxiliary analysis method and device for mental illness patients based on facial expression recognition. Background Art

[0002] Mental illness is a complex and difficult field. Due to the particularity of mental illness, it is impossible to judge whether a patient has a certain disease by analyzing pathological factors like other diseases. However, mental illness is a disease that cannot be ignored, and there is a great demand for its related diagnosis and treatment. Some auxiliary means can be used to observe whether a patient has the manifestations of mental illness to help doctors make auxiliary analysis and judgment.

[0003] For mental illness patients, they may think that they do not have mental illness themselves, and doctors can only use their professional knowledge and skills to evaluate and diagnose. The evaluation and diagnosis are both based on the analysis and screening of the patient's behaviors, expressions and other manifestations. Since mental illness patients have many symptom manifestations, usually the possibility of a patient having schizophrenia is analyzed through the angry facial expression of the patient. Summary of the Invention

[0004] The present invention provides an auxiliary analysis method and device for mental illness patients based on facial expression recognition to solve the existing problems.

[0005] The present invention proposes an auxiliary analysis method for mental illness patients based on facial expression recognition, and the method includes the following steps:

[0006] Collect a plurality of face images of a patient during communication with a doctor;

[0007] For each face image, respectively construct a plane rectangular coordinate system, and divide to obtain the regions where each part of the face image is located; according to the coordinates of the pixel points in the eyebrow region of the face image, obtain the glabella region of each face image; through threshold segmentation of the glabella region, obtain several suspected frown lines regions and their suspected frown lines region sequences in the glabella region; according to the vertical coordinate difference between adjacent pixel points and the angle difference between the line connecting the pixel points and the vertical direction in each suspected frown lines region sequence, obtain the possibility of the existence of frown lines in the face image of the patient at each moment;

[0008] Obtain the pupil region of each face image, and according to the change in pupil area between two adjacent face images, obtain several facial expression change moments; according to the possibility of the existence of frown lines in the face image at the facial expression change moment, and the difference in pupil area from the face image at the previous moment, obtain the possibility that the patient's expression is angry at each facial expression change moment;

[0009] Based on the likelihood that the patient's expression is angry at the moment of facial expression change, several anger stages of the patient are obtained; the time points when the doctor mentions the patient's keywords each time are acquired, and in combination with the several anger stages of the patient, the starting stage of anger for each anger stage of the patient is obtained, thereby obtaining the patient's ability to control their own emotions during the conversation.

[0010] Based on the patient's ability to control their own emotions during the conversation, the severity of the patient's mental problems is obtained to assist the doctor in judging the patient's mental state.

[0011] Furthermore, the specific steps for obtaining the glabella region of each face image are as follows:

[0012] The horizontal line obtained by adding 5 to the ordinate of the pixel point with the largest ordinate in the middle of the eyebrow region is used as the top edge of the glabella region;

[0013] The horizontal line obtained by subtracting 5 from the ordinate of the pixel point with the smallest ordinate in the middle of the eyebrow region is used as the bottom edge of the glabella region;

[0014] The vertical line corresponding to the abscissa of the pixel point with the largest abscissa in the middle of the left eyebrow region is used as the left edge of the glabella region;

[0015] The vertical line corresponding to the abscissa of the pixel point with the smallest abscissa in the middle of the right eyebrow region is used as the right edge of the glabella region;

[0016] The closed region constructed by each edge is denoted as the glabella region.

[0017] Furthermore, the specific steps for obtaining several suspected frown lines regions and their suspected frown lines region sequences in the glabella region by performing threshold segmentation on the glabella region are as follows:

[0018] Perform Otsu threshold segmentation on the glabella region of any face image to obtain the number of pixel points with a gray value of 0 in the glabella region ;

[0019] When the number of pixel points is less than or equal to the first threshold, there is no frown line manifestation;

[0020] When the number of pixel points is greater than the first threshold, all pixel points with a gray value of 0 in the glabella region are divided by the abscissa of each pixel point, and pixel points with a difference in abscissa between (-2, 2) are divided into the same region to obtain several suspected frown lines regions in the glabella region;

[0021] Arrange the pixel points in any suspected frown lines region in the order of first from left to right and then from bottom to top in the coordinate system to obtain the suspected frown lines region sequence of the suspected frown lines region.

[0022] Further, obtaining the possibility of the presence of the frown lines in the face image of the patient at each moment according to the vertical coordinate difference between adjacent pixel points and the angle difference between the line connecting the pixel points and the vertical direction in each suspected frown line area sequence includes the following specific steps:

[0023] ;

[0024] wherein, represents the possibility of the presence of the frown lines in the face image of the patient at the th moment; represents the number of suspected frown line area sequences in the face image at the th moment; represents the number of elements in the th suspected frown line area sequence in the face image at the th moment; represents the vertical coordinate of the th element in the th suspected frown line area sequence in the face image at the th moment; represents the vertical coordinate of the th element in the th suspected frown line area sequence in the face image at the th moment; represents the angle between the line connecting the th element and the next element in the th suspected frown line area sequence in the face image at the th moment and the Y-axis of the coordinate system; represents the angle between the line connecting the th element and the next element in the th suspected frown line area sequence in the face image at the th moment and the Y-axis of the coordinate system; represents the absolute value function.

[0025] Further, obtaining the pupil area of each face image and obtaining several facial expression change moments according to the change in the pupil area between two adjacent face images includes the following specific steps:

[0026] Performing edge detection on the face image at any moment to obtain several edges; performing Hough circle transformation on all the edges to obtain circular pupil areas, and obtaining the area of each pupil area; recording the largest area among the two pupil areas in the face image at this moment as the pupil area in the face image at this moment;

[0027] The ratio of the facial image at this moment to the pupil area in the facial image at the previous moment is denoted as the degree of change in pupil area at this moment;

[0028] The moment when the degree of change in pupil area is greater than the pupil change threshold is denoted as the facial expression change moment.

[0029] Furthermore, obtaining the possibility that the patient's expression is angry at each facial expression change moment based on the possibility of the presence of vertical frown lines in the facial image at the facial expression change moment and the difference in pupil area from the facial image at the previous moment includes the following specific steps:

[0030] ;

[0031] Among them, represents the possibility that the patient's expression is angry at the th moment, and the th moment is the facial expression change moment; represents the possibility of the presence of vertical frown lines in the facial image of the patient at the th moment; represents the pupil area in the facial image of the patient at the th moment; represents the pupil area in the facial image of the patient at the th moment.

[0032] Furthermore, obtaining several anger stages of the patient based on the possibility that the patient's expression is angry at the facial expression change moment includes the following specific steps:

[0033] If the possibility that the patient's expression is angry at any facial expression change moment is a positive number, the patient's facial expression at this facial expression change moment changes from normal to angry; if the possibility that the patient's expression is angry at this facial expression change moment is a negative number, the patient's facial expression at this facial expression change moment changes from angry to normal; mark the time points when the patient's facial expression changes from normal to angry and from angry to normal respectively.

[0034] Taking each time point when the patient's facial expression changes from normal to angry as the starting point and the first time point when the patient's facial expression changes from angry to normal after the moment when the patient's facial expression changes from normal to angry as the end point, obtain several time periods during which the patient shows anger, denoted as several anger stages of the patient.

[0035] Furthermore, obtaining the anger starting stage of each anger stage of the patient by combining the time points when the doctor mentions the patient's keywords each time with the patient's several anger stages includes the following specific steps:

[0036] The time point at which the doctor mentions the patient's keywords each time is obtained, and the time period between the time point when the patient's facial expression changes from normal to angry and the time point when the doctor last mentioned the patient's keywords before that is recorded as the anger starting stage of the anger stage corresponding to the patient's facial expression changing from normal to angry.

[0037] Furthermore, the step of obtaining the patient's ability to control his or her own emotions during the conversation includes the following specific steps:

[0038] ;

[0039] in, It indicates the patient's ability to control his or her emotions during the conversation; Indicates the number of times the patient's facial expression changes from normal to angry during the conversation; Indicates the number of times the doctor mentioned the patient's keyword; Indicates that the patient The duration of the anger phase; Indicates the patient’s The duration of the anger onset phase of the anger phase; Indicates the patient’s The start time of the anger phase; Indicates the patient’s The start time of the anger phase; Represents the sigmoid function.

[0040] The present invention also proposes a device for assisting analysis of patients with mental illness based on facial expression recognition, the device comprising:

[0041] A patient facial image acquisition module and a patient mental condition analysis module; the patient facial image acquisition module is used to acquire facial images of patients when they are talking with doctors, and the patient mental condition analysis module obtains the severity of the patient's mental problems by calling a computer program to implement the steps of an auxiliary analysis method for mental illness patients based on facial expression recognition.

[0042] The beneficial effects of the technical solution of the present invention are:

[0043] By collecting multiple images, the facial expression changes of the patient during the communication process can be comprehensively recorded, improving the accuracy of analysis; for each face image, a plane rectangular coordinate system is respectively constructed, and the regions where each part of the face image is located are segmented to accurately locate facial features, providing high-resolution facial feature data; by obtaining the glabella region of each face image and focusing on the key expression region, it helps to analyze the presence and changes of the frown lines, which are closely related to the mental state; by performing threshold segmentation on the glabella region, several suspected frown line regions and their suspected frown line region sequences in the glabella region are obtained, and according to the vertical coordinate difference between adjacent pixel points and the angle difference between the connecting lines of pixel points and the vertical direction in the suspected frown line region sequence, the changes of the frown lines are quantified, and then the possibility of the presence of frown lines in the face images of the patient at each moment is obtained; by quantifying the changes in the pupil size in adjacent face images, the facial expression change moment is quantified, and combined with the possibility of the presence of frown lines in the face image at the facial expression change moment, the anger emotion is comprehensively evaluated; then the anger stage of the patient is divided, and the time points of the keywords mentioned by the doctor about the patient each time are obtained. Combining several anger stages of the patient, the emotion and the communication content are associated to find out the specific factors causing the patient's emotional fluctuations, and more targeted diagnosis and treatment suggestions are provided; according to the patient's ability to control their own emotions shown during the conversation, the emotion control ability is evaluated, the severity of the mental problems of the patient is obtained, and the doctor's judgment of the patient's mental state is assisted, improving the efficiency of the doctor's diagnosis of the patient's mental illness through facial expression recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0045] Figure 1 It is a flowchart of the steps of the method for assisting the analysis of mental illness patients based on facial expression recognition according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the intended invention purpose, the following, in combination with the drawings and preferred embodiments, will describe in detail the specific implementation manners, structures, features and effects of the method and device for assisting the analysis of mental illness patients based on facial expression recognition according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0047] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0048] The specific scheme of the method and device for auxiliary analysis of patients with mental illness based on facial expression recognition provided by the present invention is described in detail below with reference to the accompanying drawings.

[0049] See also Figure 1 , which shows a flowchart of a method for assisting analysis of mental illness patients based on facial expression recognition provided by an embodiment of the present invention, the method comprising the following steps:

[0050] Step S001: Collect several facial images of the patient communicating with the doctor.

[0051] The purpose of this embodiment is to judge the patient's ability to control his or her emotions by analyzing the patient's facial expression during the conversation between the doctor and the patient, thereby providing the doctor with an auxiliary means to help the doctor judge the patient's specific condition. Therefore, it is necessary to use a camera to collect facial images of the patient at continuous moments during the entire conversation.

[0052] Specifically, the camera is placed on the wall behind the doctor, and is positioned so as to capture all of the patient's facial information. In this embodiment, the patient's facial image collection interval is set to 1 second, and the collection time is the entire process of communication between the doctor and the patient, thereby obtaining several facial images of the patient while talking with the doctor.

[0053] Step S002: construct a plane rectangular coordinate system for each face image, and segment the areas where various parts of the face image are located; obtain the glabellar area of ​​each face image according to the coordinates of the pixel points in the eyebrow area of ​​the face image; obtain several suspected chuan character line areas and their suspected chuan character line area sequences in the glabellar area by performing threshold segmentation on the glabellar area; obtain the possibility of the presence of chuan character line in the patient's face image at each moment according to the difference in the vertical coordinates of adjacent pixels in each suspected chuan character line area sequence and the difference in the angle between the pixel points and the vertical direction.

[0054] It should be noted that there may be certain triggers for the onset of schizophrenia, and these triggers may be multifaceted. Before diagnosis, the patient's family should tell the doctor which specific words may make the patient emotionally unstable. The doctor can then analyze the patient's facial expressions during the conversation to determine whether the patient is angry at a certain moment, and further combine the patient's emotional changes under the time factor to determine whether the patient can control his or her emotional expression, thereby helping the doctor to determine whether the patient has the potential for mental illness after the communication.

[0055] It should be further explained that the doctor's diagnosis process is based on the analysis of the patient's facial expressions. When the doctor mentions certain specific words during the communication process with the patient, the patient may show more anger and unstable emotions; the subsequent analysis of whether the patient can control his emotions is based on the patient's anger expression. Therefore, it is necessary to judge whether there is anger in the patient's facial expression at each moment; by analyzing the patient's facial expression, when a person shows an angry state, it will be accompanied by a series of changes in facial features, such as changes in eyebrow shape, the appearance of forehead wrinkles, etc., which may indicate that the patient's expression state at this time is angry, but because some features may also appear during normal communication, specific analysis is required to distinguish the patient's angry state from the normal state.

[0056] Specifically, a plane rectangular coordinate system is constructed with the point in the lower left corner of each face image as the coordinate origin, and the coordinates of each point in the image in the plane rectangular coordinate system are obtained; and the areas where the left and right eyebrows, left and right eyes, and mouth are located in the face image are identified and segmented through a deep neural network.

[0057] The relevant content of deep neural network is as follows:

[0058] The deep neural network used in this embodiment is the DeepLabV3 neural network; the data set used is a face image data set.

[0059] The pixels that need to be segmented are divided into 6 categories, that is, the labeling process of the training set is: single-channel semantic label, the corresponding position pixel belongs to the background class is labeled as 0, the left eyebrow area is labeled as 1, the right eyebrow area is labeled as 2, the left eye area is labeled as 3, the right eye area is labeled as 4, and the mouth area is labeled as 5.

[0060] The task of the network is classification, so the loss function used is the cross entropy loss function.

[0061] The deep neural network is used to obtain the areas where the left and right eyebrows, left and right eyes, and mouth are located in the face image, thereby completing the segmentation of various areas in the face image.

[0062] It should be noted that when analyzing the facial expressions of patients, special attention should be paid to whether there is a frowning phenomenon. When patients show anger, they usually have similar wrinkles on their foreheads. The general distribution of wrinkles is between the two eyebrows on the face. Therefore, according to the coordinates of each pixel point in the eyebrow area, the glabella area can be obtained. Since the glabella will have two obvious lines in the middle of the two eyebrows, the glabella area is segmented by thresholding and analyzing the continuity and directionality of the pixels to determine whether there is a glabella.

[0063] Specifically, the vertical coordinate of the pixel point with the largest vertical coordinate in the eyebrow area (including the left eyebrow area and the right eyebrow area) plus the horizontal line corresponding to 5 is used as the top edge of the glabella area; the vertical coordinate of the pixel point with the smallest vertical coordinate in the eyebrow area minus the horizontal line corresponding to 5 is used as the bottom edge of the glabella area; the vertical line corresponding to the horizontal coordinate of the pixel point with the largest horizontal coordinate in the left eyebrow area is used as the left edge of the glabella area; the vertical line corresponding to the horizontal coordinate of the pixel point with the smallest horizontal coordinate in the right eyebrow area is used as the right edge of the glabella area; the closed area constructed by each edge is recorded as the glabella area; the glabella area in any face image is segmented by Otsu threshold to obtain the number of pixels with a grayscale value of 0 in the glabella area. ; The first threshold value preset in this embodiment is 10, which is used as an example for description. Other implementations may be set to other values; when the number of pixels is less than the first threshold value, that is, when When the number of pixels is greater than the first threshold, it means that there is no Sichuan pattern. When , it indicates that there may be Sichuan character lines. All the pixels with grayscale values ​​of 0 in the glabella area are divided by the horizontal coordinates between each pixel, and the pixels with the horizontal coordinate difference of the pixels being (-2, 2) are divided into the same area, and several suspected Sichuan character lines areas in the glabella area are obtained. The pixels in any suspected Sichuan character lines area are arranged in the order from left to right and then from bottom to top in the coordinate system to obtain the suspected Sichuan character lines area sequence of the suspected Sichuan character lines area; and then according to the vertical coordinate difference of adjacent pixels in each suspected Sichuan character lines area sequence and the angle difference between the connecting line between the pixels and the vertical direction, the possibility of the presence of Sichuan character lines in the patient's facial image at each moment is obtained. The specific calculation formula is:

[0064] ;

[0065] in, Indicates that the patient The possibility of the existence of Sichuan-shaped patterns in the face image at the moment; Indicates The number of suspected Sichuan character pattern region sequences in the face image at the moment; Indicates The number of elements in the th suspected Sichuan character pattern region sequence in the face image at the th moment (i.e., the number of pixel points in the th suspected Sichuan character pattern region in the face image at the th moment); Denote the ordinate of the th element in the th suspected Sichuan character pattern region sequence in the face image at the th moment; Denote the ordinate of the th element in the th suspected Sichuan character pattern region sequence in the face image at the th moment; Denote the angle between the line connecting the th element and the next element (the th element) in the th suspected Sichuan character pattern region sequence in the face image at the th moment and the Y-axis of the coordinate system; Denote the angle between the line connecting the th element and the next element (the th element) in the th suspected Sichuan character pattern region sequence in the face image at the th moment and the Y-axis of the coordinate system;

[0066] It should be noted that Denote the difference in ordinates between the th element and the next element in the th suspected Sichuan character pattern region sequence in the face image at the th moment, Denote the difference in the angles between the lines connecting adjacent two elements and the corresponding next element and the Y-axis of the coordinate system in the th suspected Sichuan character pattern region sequence in the face image at the th moment; By combining the difference in ordinates and the difference in angles, it is judged whether the continuous pixel points in the Sichuan character pattern region in the image are continuous and have the same direction. If the performance is more continuous and has the same direction, it means that the possibility of the patient having Sichuan character patterns in the face image at the th moment is higher.

[0067] Thus, the possibility of the patient having Sichuan character patterns in the face image at each moment is obtained.

[0068] Step S003: Obtain the pupil region of each face image. Based on the change in pupil area between two adjacent face images, obtain several facial expression change moments; based on the possibility of the presence of vertical frown lines in the face image at the facial expression change moment and the difference in pupil area from the face image at the previous moment, obtain the possibility that the patient's expression is angry at each facial expression change moment.

[0069] It should be noted that it is not sufficient to only use the vertical frown lines to illustrate the rigor of the patient showing anger. It is also necessary to further analyze in combination with the performance of the patient's eye pupils. When the patient shows an angry state, their eye pupils will dilate to show strong emotions and concentration. Therefore, it is necessary to analyze the patient's pupil performance over continuous time. Since the color of the eye pupils is black and the sclera area is white, the edge performance of the pupils can be obtained by performing edge detection on the image, and the diameter of the pupil region can be obtained through the Hough circle transform, thereby obtaining the patient's facial expression at each moment.

[0070] Specifically, perform edge detection on the face image at any moment to obtain several edges; perform Hough circle transform on all the edges to obtain the circular pupil region, and obtain the area of each pupil region; it should be noted that obtaining the circular region through the Hough circle transform is an existing method, which will not be elaborated in this embodiment; record the largest area among the two pupil regions in the face image at this moment as the pupil area in the face image at this moment; record the ratio of the pupil area in the face image at this moment to the pupil area in the face image at the previous moment as the pupil area change degree at this moment; the preset pupil change threshold in this embodiment is 0.3, and this is used as an example for description. In other embodiments, it can be set to other values; record the moment when the pupil area change degree is greater than the pupil change threshold as the facial expression change moment; thus, based on the possibility of the presence of vertical frown lines in the face image at the facial expression change moment and the difference in pupil area from the face image at the previous moment, obtain the possibility that the patient's expression is angry at each facial expression change moment. The specific calculation formula is:

[0071] ;

[0072] Among them, represents the possibility that the patient's expression is angry at the th moment; represents the possibility of the presence of vertical frown lines in the face image of the patient at the th moment; represents the pupil area in the face image of the patient at the th moment; represents the pupil area in the face image of the patient at the th moment.

[0073] It should be noted that when calculating the patient's The expression at the moment is the possibility of anger is a positive number, indicating that the patient's facial expression has changed from normal to angry. When the possibility of the facial expression being angry at the moment is negative, it means that the patient's facial expression has changed from anger to normal.

[0074] Step S004: Based on the possibility that the patient's facial expression is angry at the moment of facial expression change, several anger stages of the patient are obtained; the time point when the doctor mentions the patient's keywords each time is obtained, and combined with the patient's several anger stages, the anger starting stage of each anger stage of the patient is obtained, and then the patient's ability to control his own emotions during the conversation is obtained.

[0075] It should be noted that by analyzing the patient's angry expression in combination with time nodes, we can determine whether the patient can control his emotions well. When the doctor and the patient talk about keywords (that is, the patient's family tells the doctor that the patient may become emotional when talking about certain issues), if the patient's facial expression shows anger, it means that the patient may not be able to control his emotions after hearing the keywords. We further analyze the patient's ability to control his emotions in combination with factors such as the time and number of times the patient cannot control his emotions, so as to assist the doctor in judging the patient's mental state.

[0076] Specifically, if the possibility that the patient's facial expression is angry at any moment of facial expression change is positive, the patient's facial expression at the moment of facial expression change is converted from normal to anger; if the possibility that the patient's facial expression is angry at the moment of facial expression change is negative, the patient's facial expression at the moment of facial expression change is converted from anger to normal; the time points when the patient's facial expression is converted from normal to anger and from anger to normal are marked respectively; the time point when the patient's facial expression is converted from normal to anger is taken as the starting point, and the time point when the patient's facial expression is converted from anger to normal for the first time after the patient's facial expression is converted from normal to anger is taken as the end point, and several time periods when the patient shows anger are obtained, which are recorded as several anger stages of the patient; the time point when the doctor mentions the patient's keywords each time is obtained, and the time period between any time when the patient's facial expression is converted from normal to anger and the time point when the doctor mentions the patient's keywords before is obtained, which is recorded as the anger starting stage of the anger stage corresponding to the conversion of the patient's facial expression from normal to anger; according to the possibility that the patient's expression is angry at the moment of facial expression change, the patient's several anger stages and their starting stages, and the time point when the doctor mentions the patient's keywords each time, the patient's ability to control his own emotions during the conversation is obtained, and the specific calculation formula is:

[0077] ;

[0078] in, It indicates the patient's ability to control his or her emotions during the conversation; Indicates the number of times the patient expressed anger during the conversation; Indicates the number of times the doctor mentioned the patient's keyword; The patient said that during the conversation The duration of the anger phase; Indicates the patient’s The duration of the anger onset phase of the anger phase; Indicates the patient’s The start time of the anger phase; Indicates the patient’s The start time of the anger phase; represents a sigmoid function, which is used for normalization processing in this embodiment.

[0079] It should be noted that This means that the fewer times the doctor mentions the patient's keywords, and the longer each anger stage of the patient during the conversation, the more strongly the patient's emotions are affected by the keywords, the greater the emotional fluctuations when hearing the keywords, and the worse the ability to control their emotions. At the same time, combined with the time difference between the patient hearing the keywords and the appearance of the angry expression (i.e. the anger starting stage of each anger stage) and the time difference between the two anger stages, the shorter the time difference between the patient hearing the keywords and the appearance of the angry expression, and the shorter the time difference between the two anger stages, the more the patient is unable to get rid of the influence of the keywords on his or her emotions, and the worse the ability to control his or her emotions.

[0080] At this point, the patient's ability to control his or her own emotions during the conversation was obtained.

[0081] Step S005: Based on the patient's ability to control his or her own emotions during the conversation, the severity of the patient's mental problem is obtained to assist the doctor in judging the patient's mental state.

[0082] Specifically, the preset emotion control threshold in this embodiment is 0.6, which is used as an example for description. Other implementation modes may be set to other values; when the patient's ability to control his or her own emotions during the conversation is greater than or equal to the emotion control threshold, it means that the patient may have fewer mental problems and the mental problems are not serious; when the patient's ability to control his or her own emotions during the conversation is less than the emotion control threshold, it means that the patient may have more mental problems and the patient's mental problems are serious; the severity of the patient's mental problems is used as a means to assist doctors in judging the patient's mental state.

[0083] It should be noted that the above methods are only used as a means to assist doctors in making judgments. If you want to further judge the patient's mental state, doctors need to make reasonable judgments based on their professional knowledge.

[0084] At this point, this embodiment is completed.

[0085] The present invention also proposes a device for assisting analysis of patients with mental illness based on facial expression recognition, the device comprising:

[0086] A patient facial image acquisition module and a patient mental condition analysis module; the patient facial image acquisition module is used to acquire facial images of patients when they are talking with doctors, and the patient mental condition analysis module obtains the severity of the patient's mental problems by calling a computer program to implement the steps of an auxiliary analysis method for mental illness patients based on facial expression recognition.

[0087] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for assisting analysis of patients with mental illness based on facial expression recognition, characterized in that: The method includes the following steps: Collect several face images of the patient during the communication with the doctor; Construct a rectangular coordinate system for each face image respectively, and segment the regions where each part of the face image is located; According to the coordinates of the pixel points in the eyebrow region of the face image, obtain the glabella region of each face image; By performing threshold segmentation on the glabella region, obtain several suspected frown lines regions and their suspected frown lines region sequences in the glabella region; According to the vertical coordinate difference between adjacent pixel points and the angle difference between the line connecting pixel points and the vertical direction in each suspected frown lines region sequence, obtain the possibility of the existence of frown lines in the face image of the patient at each moment; Obtain the pupil region of each face image, and according to the change in pupil area between adjacent two face images, obtain several facial expression change moments; According to the possibility of the existence of frown lines in the face image at the facial expression change moment, and the difference in pupil area from the previous moment's face image, obtain the possibility that the patient's expression is angry at each facial expression change moment; According to the possibility that the patient's expression is angry at the facial expression change moment, obtain several angry stages of the patient; Obtain the time points when the doctor mentions the patient's keywords each time, and combine with several angry stages of the patient to obtain the angry start stage of each angry stage of the patient, and further obtain the patient's ability to control their own emotions during the conversation; According to the patient's ability to control their own emotions during the conversation, obtain the severity of the patient's mental problems, and assist the doctor in judging the patient's mental state; The specific steps included in obtaining several suspected frown lines regions and their suspected frown lines region sequences in the glabella region by performing threshold segmentation on the glabella region are as follows: Perform Otsu threshold segmentation on the glabellar region in any face image to obtain the number of pixels with a gray value of 0 in the glabellar region. ; When the number of pixel points is less than or equal to the first threshold, there is no manifestation of frown lines; When the number of pixel points is greater than the first threshold, divide all pixel points with a gray value of 0 in the glabella region by the abscissa of each pixel point, and divide the pixel points with the difference between abscissas in (-2, 2) into the same region to obtain several suspected frown lines regions in the glabella region; Arrange the pixel points in any one of the suspected frown lines regions in the order of first from left to right and then from bottom to top in the coordinate system to obtain the suspected frown lines region sequence of this suspected frown lines region; The specific steps included in obtaining the possibility of the existence of frown lines in the face image of the patient at each moment according to the vertical coordinate difference between adjacent pixel points and the angle difference between the line connecting pixel points and the vertical direction in each suspected frown lines region sequence are as follows: ； in, Indicates that the patient The possibility of the existence of Sichuan-shaped patterns in the face image at the moment; Indicates The number of suspected Sichuan character pattern region sequences in the face image at the moment; Indicates The face image at the moment The number of elements in the sequence of the suspected Sichuan pattern area; Indicates The face image at the moment The suspected Sichuan pattern region sequence The ordinate of the element; Indicates The face image at the moment The suspected Sichuan pattern region sequence The ordinate of the element; Indicates The face image at the moment The suspected Sichuan pattern region sequence The angle between the line connecting the first element and the next element and the Y axis of the coordinate system; Indicates The face image at the moment The suspected Sichuan pattern region sequence The angle between the line connecting the first element and the next element and the Y axis of the coordinate system; represents the absolute value function; The specific steps included in obtaining several facial expression change moments by obtaining the pupil region of each face image and according to the change in pupil area between adjacent two face images are as follows: Perform edge detection on the face image at any moment to obtain several edges; Perform Hough circle transformation on all edges to obtain circular pupil regions, and obtain the area of each pupil region; Denote the largest area of the two pupil regions in the face image at this moment as the pupil area of the face image at this moment; The ratio of the pupil area in the face image at this moment to the pupil area in the face image at the previous moment is denoted as the degree of change in pupil area at this moment; The moment when the degree of change in pupil area is greater than the pupil change threshold is denoted as the facial expression change moment.

2. The method for assisting analysis of patients with mental illness based on facial expression recognition according to claim 1, characterized in that: The steps for obtaining the glabella region of each face image are as follows: The horizontal line corresponding to adding 5 to the ordinate of the pixel point with the largest ordinate in the eyebrow region is used as the top edge of the glabella region; The horizontal line corresponding to subtracting 5 from the ordinate of the pixel point with the smallest ordinate in the eyebrow region is used as the bottom edge of the glabella region; The vertical line corresponding to the abscissa of the pixel point with the largest abscissa in the left eyebrow region is used as the left edge of the glabella region; The vertical line corresponding to the abscissa of the pixel point with the smallest abscissa in the right eyebrow region is used as the right edge of the glabella region; The closed region constructed by each edge is denoted as the glabella region.

3. The method for assisting analysis of patients with mental illness based on facial expression recognition according to claim 1, characterized in that: The steps for obtaining the possibility that the patient's expression is angry at each facial expression change moment based on the possibility of the presence of a frown line in the face image at the facial expression change moment and the difference in pupil area from the face image at the previous moment are as follows: ； in, Indicates that the patient The expression at this moment is the possibility of anger, Moment is the moment when facial expression changes; Indicates that the patient The possibility of the existence of Sichuan-shaped patterns in the face image at the moment; Indicates that the patient Pupil area in the face image at the moment; Indicates that the patient The pupil area in the face image at that moment.

4. The method for assisting analysis of patients with mental illness based on facial expression recognition according to claim 1, characterized in that: The steps for obtaining several anger stages of the patient based on the possibility that the patient's expression is angry at the facial expression change moment are as follows: If the possibility that the patient's expression is angry at any facial expression change moment is a positive number, the patient's facial expression at this facial expression change moment changes from normal to angry; If the possibility that the patient's expression is angry at this facial expression change moment is a negative number, the patient's facial expression at this facial expression change moment changes from angry to normal; Mark the time points when the patient's facial expression changes from normal to angry and from angry to normal respectively; Taking each time point when the patient's facial expression changes from normal to angry as the starting point and the first time point when the patient's facial expression changes from angry to normal after this moment as the end point, several time periods during which the patient shows anger are obtained, and are denoted as several anger stages of the patient.

5. The method for assisting analysis of patients with mental illness based on facial expression recognition according to claim 4, characterized in that: The steps for obtaining the starting stage of anger for each anger stage of the patient by combining the time points when the doctor mentions the patient's keywords each time with the patient's several anger stages are as follows: Obtain the time points when the doctor mentions the patient's keywords each time, and obtain the time period between any time point when the patient's facial expression changes from normal to angry and the last time point when the doctor mentioned the patient's keywords that appeared before it, and denote it as the starting stage of anger for the anger stage corresponding to this time when the patient's facial expression changes from normal to angry.

6. The method for assisting analysis of patients with mental illness based on facial expression recognition according to claim 1, characterized in that: The steps for obtaining the patient's ability to control their own emotions shown during the conversation are as follows: ； in, It indicates the patient's ability to control his or her emotions during the conversation; Indicates the number of times the patient's facial expression changes from normal to angry during the conversation; Indicates the number of times the doctor mentioned the patient's keyword; The patient said that during the conversation The duration of the anger phase; Indicates the patient’s The duration of the anger onset phase of the anger phase; Indicates the patient’s The start time of the anger phase; Indicates the patient’s The start time of the anger phase; Represents the sigmoid function.

7. A device for assisting analysis of patients with mental illness based on facial expression recognition, characterized in that: The device includes: A patient facial image acquisition module and a patient mental condition analysis module; the patient facial image acquisition module is used to acquire facial images of patients when they are talking with doctors, and the patient mental condition analysis module obtains the severity of the patient's mental problems by calling a computer program to implement the steps of the mental illness patient auxiliary analysis method based on facial expression recognition as described in any one of claims 1-6.

Images