Early screening method and device for stroke hidden affective disorder of facial micro-expression

Abstract

The invention discloses a method and a related device for early screening of stroke hidden affective disorder based on facial micro-expression. The method comprises the following steps: acquiring facial video data of a stroke patient through a camera array; extracting time domain texture change features, spontaneous blinking frequency and blinking duration of each area of the face; an optical flow field of facial muscle movement is calculated, and optical flow statistical characteristics including optical flow amplitude and optical flow direction are obtained; respectively inputting the time domain texture change features, the spontaneous blinking frequency, the blinking duration and the optical flow statistical features into an independent Bi-LSTM network model, and outputting the latent affective disorder probability of the patient; and calculating the risk level of the patient occult affective disorder according to the occult affective disorder probability of the patient, and presenting a visual result to a doctor for auxiliary diagnosis. According to the method, the time domain texture change feature, the spontaneous blinking frequency, the blinking duration and the optical flow statistical feature are respectively input into the independent Bi-LSTM network model, so that the recognition precision of the latent affective disorder probability of the patient is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and medical diagnosis assistance technology, and specifically to a method and device for early screening of latent affective disorders in stroke patients using facial micro-expressions, as well as computing equipment. Background Art

[0002] Stroke screening aims to identify high-risk groups and early-stage patients early, providing opportunities for timely treatment and reducing disability and mortality rates. The diagnosis of stroke is a prerequisite for patients to receive effective treatment.

[0003] Stroke screening usually includes the following:

[0004] 1. Perform a neurological physical examination on the patient to assess the patient's state of consciousness, language function, motor function, sensory function, cranial nerve function, etc.

[0005] 2. Assess the patient's general health status and potential risk factors through blood routine, blood sugar, blood lipids, electrolytes, renal function, and coagulation function.

[0006] 3. Brain MRI (magnetic resonance imaging) can be used to show the location and size of cerebral infarction, the surrounding brain edema, and the presence of bleeding signs. Cerebral angiography (CTA / MRA / DSA) can be used to observe the condition of cerebral blood vessels to understand whether the blood vessels are narrowed, occluded, or deformed.

[0007] The above screening methods mainly focus on the acute symptoms and vascular conditions of stroke, and are insufficient for the early identification of stroke sequelae (such as affective disorders).

[0008] To solve the above problems, the present invention inputs the temporal texture change characteristics, spontaneous blinking frequency and blink duration, and optical flow statistical characteristics into an independent Bi-LSTM network model to improve the recognition accuracy of the patient's latent affective disorder probability. Summary of the Invention

[0009] In view of the above problems, the present invention provides a method and device, as well as a computing device, for early screening of latent affective disorders in stroke patients using facial micro-expressions.

[0010] According to one aspect of the present invention, a method for early screening of latent affective disorders in stroke patients using facial micro-expressions is provided, comprising:

[0011] Facial video data of a stroke patient is collected using a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, wherein the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side;

[0012] Extracting temporal texture change features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture change features include the area between the eyebrows, the corners of the mouth, and the corners of the eyes;

[0013] The optical flow field of facial muscle movement is calculated according to the dense optical flow method, and the optical flow statistical features including the optical flow amplitude and direction are obtained;

[0014] The temporal texture change features, the spontaneous blink frequency and blink duration, and the optical flow statistical features are respectively input into an independent Bi-LSTM network model to output the probability of the patient's latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical dataset including NIHSS scores and lesion location information of stroke patients;

[0015] The risk level of the patient's latent affective disorder is calculated based on the probability of the patient's latent affective disorder, and the visual results are presented to the doctor for auxiliary diagnosis. The visual results include a facial expression change heat map, an optical flow distribution map, and an emotional state probability value.

[0016] In an optional manner, extracting the temporal texture change features of each facial region according to the local binary pattern operator further includes:

[0017] The facial video data is divided into multiple time windows, each time window contains multiple frame images;

[0018] For each frame image in each time window, the local binary pattern features are calculated to obtain the local binary pattern feature matrix;

[0019] The time change rate of the feature matrix in each time window is calculated to obtain the time domain texture change features.

[0020] In an optional manner, the optimization formula for light source compensation of the adjustable light source system is:

[0021]

[0022] Wherein, Ω is the ROI area of ​​the hemiplegic side face; is the incident light direction vector; N(x,y) is the normal vector of each pixel; (x,y) is the coordinate of the pixel in the facial image; Z(x,y) is the patient's facial depth map obtained by structured light projection; (x0,y0) is the coordinate of the eye canthus on the lesion side; α is the spatial attenuation coefficient; are the elevation angle and azimuth angle of the light source respectively.

[0023] In an optional manner, the independent Bi-LSTM network model includes a dual-channel feature encoder, a dynamic weight generator, a fusion feature layer and a time-domain convolutional network;

[0024] Among them, the dual-channel feature encoder includes a visual channel encoder and a clinical channel encoder. The visual channel encoder uses 3D-CNN to extract spatiotemporal features, and the clinical channel encoder uses a Transformer network layer to encode NIHSS score, lesion volume and side.

[0025] In an optional manner, the expression change heat map uses Grad-CAM++ to back-propagate the hidden layer activations of Bi-LSTM to obtain an importance weight map of each facial region;

[0026] The optical flow distribution map is obtained by mapping the optical flow vector field into the HSV color space, wherein the amplitude is encoded as the brightness channel and the direction is encoded as the hue channel.

[0027] In an optional embodiment, the camera array further includes an infrared camera for improving the acquisition quality of facial video data in a low-light environment; wherein the infrared camera operates synchronously with the front camera and the side camera.

[0028] In an optional manner, the method further includes:

[0029] Circle an area of ​​the expression change heat map through a touch screen, automatically extract the spatiotemporal features of the area and associate them with a medical knowledge base to display key points for differential diagnosis;

[0030] By adjusting the emotion probability warning threshold, the false positive / false negative rate change curve under the current setting can be fed back instantly;

[0031] The patient's historical screening records are loaded synchronously through multi-dimensional comparative views, and the progression trend of affective disorders is displayed through the optical flow trajectory remapping method.

[0032] In an optional manner, calculating the optical flow field of facial muscle movement according to the dense optical flow method to obtain the optical flow statistical features including the optical flow amplitude and the optical flow direction further includes:

[0033] Performing face alignment according to key point coordinates, wherein the face alignment operation includes rotation, scaling and translation;

[0034] The dense optical flow algorithm is used to calculate the optical flow field between consecutive frames of the face to obtain a two-dimensional vector field, where each pixel of the two-dimensional vector field corresponds to a displacement vector (u, v), where u is the horizontal displacement and v is the vertical displacement;

[0035] For each pixel's displacement vector (u, v), calculate its optical flow magnitude and optical flow direction.

[0036] According to another aspect of the present invention, a device for early screening of latent affective disorders in stroke patients using facial micro-expressions is provided, comprising:

[0037] A facial video acquisition module is configured to collect facial video data of a stroke patient using a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, wherein the two side cameras are respectively arranged on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side;

[0038] A temporal texture feature extraction module is used to extract temporal texture change features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture change features include the area between the eyebrows, the corners of the mouth, and the corners of the eyes;

[0039] The optical flow field calculation module is used to calculate the optical flow field of facial muscle movement based on the dense optical flow method to obtain the optical flow statistical features including the optical flow amplitude and direction;

[0040] a feature input module for inputting the temporal texture change features, the spontaneous blink frequency and blink duration, and the optical flow statistical features into an independent Bi-LSTM network model, and outputting the probability of the patient having a latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical dataset including NIHSS scores and lesion location information of stroke patients;

[0041] The risk level calculation module is used to calculate the risk level of the patient's latent affective disorder based on the probability of the patient's latent affective disorder, and present the results to the doctor for auxiliary diagnosis in the form of visualization. The visualization results include a heat map of expression changes in the facial area, an optical flow distribution map, and an emotional state probability value.

[0042] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0043] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned method for early screening of latent affective disorders in stroke based on facial micro-expressions.

[0044] According to the solution provided by the present invention, facial video data of stroke patients is collected by a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, and the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side; the temporal texture change characteristics, spontaneous blinking frequency and blinking duration of each facial area are extracted based on the local binary pattern operator; the temporal texture change characteristics include the eyebrows, corners of the mouth and corners of the eyes; the optical flow field of facial muscle movement is calculated based on the dense optical flow method, and the optical flow amplitude and direction are obtained. Optical flow statistical features; the temporal texture change features, spontaneous blinking frequency and blinking duration, and the optical flow statistical features are respectively input into an independent Bi-LSTM network model to output the probability of the patient's latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical data set including the NIHSS score and lesion location information of stroke patients; the risk level of the patient's latent affective disorder is calculated based on the patient's latent affective disorder probability, and the visualization results are presented to the doctor for auxiliary diagnosis, the visualization results including a facial expression change heat map, an optical flow distribution map, and an emotional state probability value. The present invention inputs the temporal texture change features, spontaneous blinking frequency and blinking duration, and optical flow statistical features into an independent Bi-LSTM network model, thereby improving the recognition accuracy of the patient's latent affective disorder probability.

[0045] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0047] Figure 1 A schematic diagram showing a flow chart of a method for early screening of latent affective disorders in stroke patients using facial micro-expressions according to an embodiment of the present invention;

[0048] Figure 2 A schematic diagram showing the framework of a device for early screening of latent affective disorders in stroke using facial micro-expressions according to an embodiment of the present invention;

[0049] Figure 3 A schematic structural diagram of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0051] Figure 1 The flowchart of the method for early screening of latent affective disorder of stroke based on facial micro-expressions according to an embodiment of the present invention is shown. Figure 1 As shown, the following steps are included:

[0052] Step S101, collecting facial video data of a stroke patient through a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, and the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movements on the hemiplegic side.

[0053] In this embodiment, stroke patients often have asymmetrical facial expressions. Dual lateral cameras are used to capture the asymmetry of facial muscle movements on the hemiplegic side, improving the sensitivity of identifying latent post-stroke affective disorders. The frontal camera captures overall facial expressions, while the lateral camera captures subtle movements on the hemiplegic side, which can aid in the early detection of latent affective disorders.

[0054] In an optional manner, the optimization formula for light source compensation of the adjustable light source system is:

[0055]

[0056] Wherein, Ω is the ROI area of ​​the hemiplegic side face; is the incident light direction vector; N(x,y) is the normal vector of each pixel; (x,y) is the coordinate of the pixel in the facial image; Z(x,y) is the patient's facial depth map obtained by structured light projection; (x0,y0) is the coordinate of the eye canthus on the lesion side; α is the spatial attenuation coefficient; are the elevation angle and azimuth angle of the light source respectively.

[0057] In this embodiment, a facial depth map is obtained by structured light projection to calculate the normal vector of the pixel point, which more accurately reflects the three-dimensional structure of the facial surface. By adjusting the light source angle (pitch angle and azimuth angle), the lighting effect of the ROI area on the side of the hemiplegia face is maximized to further capture the facial micro-expressions on the side of the hemiplegia face. Among them, the normal vector of each pixel point is used to calculate the dot product between the incident light direction vector and the surface normal vector, thereby reducing the influence of shadows. The corner of the eye on the lesion side is the key area for emotional micro-expressions. The coordinates of the corner of the eye on the lesion side are given a smaller weight by the Gaussian attenuation factor, which makes the light source optimization pay more attention to the adjacent area of ​​the corner of the eye on the lesion side.

[0058] In an optional embodiment, the camera array further includes an infrared camera for improving the acquisition quality of facial video data in a low-light environment; wherein the infrared camera operates synchronously with the front camera and the side camera.

[0059] In this embodiment, microexpressions are inherently subtle and easily affected by light and shadow. Stroke patients may be screened in a variety of lighting conditions, especially in poorly lit clinics. The infrared camera, unrestricted by visible light, can clearly capture facial images in low-light environments, avoiding the loss of facial features caused by insufficient lighting. Furthermore, the infrared camera operates synchronously with other cameras (front and side), ensuring optimal data collection even in fluctuating lighting conditions, ensuring the reliability of the screening process.

[0060] Step S102 , extracting temporal texture variation features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture variation features include the area between the eyebrows, corners of the mouth, and corners of the eyes.

[0061] In this embodiment, the local binary pattern is a powerful texture descriptor that is insensitive to illumination changes and can effectively extract local texture information of various facial regions. Combined with time domain analysis and blinking features, it significantly improves the accuracy of emotion recognition.

[0062] Specifically, a pre-trained facial landmark detection model is used to divide facial images into the glabella, mouth corners, and eye corners. Regions of Interest (ROIs) are then extracted, and texture features are extracted from each region using a local binary pattern operator (either uniform LBP or rotation-invariant LBP). Blink events in videos are detected using an eye aspect ratio (EAR) blink detection algorithm. The spontaneous blink frequency (number of blinks per unit time) and blink duration (duration of each blink) are calculated, and the temporal texture variation features, spontaneous blink frequency, and blink duration are integrated into a single feature vector.

[0063] In an optional manner, extracting the temporal texture change features of each facial region according to the local binary pattern operator further includes:

[0064] The facial video data is divided into multiple time windows, each time window contains multiple frame images;

[0065] For each frame image in each time window, the local binary pattern features are calculated to obtain the local binary pattern feature matrix;

[0066] The time change rate of the feature matrix in each time window is calculated to obtain the time domain texture change features.

[0067] In this embodiment, facial video data is segmented into time windows and the time rate of change of local binary pattern features in each window is calculated, which can capture the evolution of micro-expressions over time and thus better distinguish normal facial movements from abnormal movement patterns related to emotional disorders.

[0068] Step S103 , calculating the optical flow field of facial muscle movement according to the dense optical flow method, and obtaining optical flow statistical features including optical flow amplitude and optical flow direction.

[0069] In this embodiment, latent affective disorders often manifest as subtle micro-expressions that are difficult to detect. The dense optical flow method is used to estimate the motion vector of each pixel in the image, so that very subtle facial muscle movements can be captured. Unlike static image features, the optical flow method provides information about the direction and amplitude of facial movement, which can distinguish different emotional states (for example, the optical flow patterns of smiling and frowning are very different). In addition, since stroke patients often suffer from hemiplegia, which leads to asymmetry in facial muscle movement, the optical flow method is used to capture the asymmetry and thus identify latent affective disorders. That is, the degree of asymmetry is quantified by comparing the difference in the optical flow field on both sides of the face, which is used as a basis for diagnosis.

[0070] In an optional manner, calculating the optical flow field of facial muscle movement according to the dense optical flow method to obtain the optical flow statistical features including the optical flow amplitude and the optical flow direction further includes:

[0071] Performing face alignment according to key point coordinates, wherein the face alignment operation includes rotation, scaling and translation;

[0072] The dense optical flow algorithm is used to calculate the optical flow field between consecutive frames of the face to obtain a two-dimensional vector field, where each pixel of the two-dimensional vector field corresponds to a displacement vector (u, v), where u is the horizontal displacement and v is the vertical displacement;

[0073] For each pixel's displacement vector (u, v), calculate its optical flow magnitude and optical flow direction.

[0074] In this embodiment, face alignment operations (rotation, scaling, and translation) eliminate the impact of head posture changes on optical flow calculations, allowing the optical flow results to more accurately reflect the movement of facial muscles. Describing the movement characteristics of facial muscles by optical flow amplitude and direction can more accurately determine the type and intensity of muscle movement, leading to more effective analysis of latent affective disorders. The amplitude of the optical flow reflects the intensity of the movement, while the direction of the optical flow reflects the direction of the movement.

[0075] Step S104, inputting the temporal texture change characteristics, spontaneous blinking frequency and blinking duration, and the optical flow statistical characteristics into an independent Bi-LSTM network model respectively, and outputting the probability of the patient's latent affective disorder; wherein, the independent Bi-LSTM network model is trained based on a clinical data set including the NIHSS score and lesion location information of stroke patients.

[0076] In this embodiment, the independent Bi-LSTM network model includes a dual-channel feature encoder, a dynamic weight generator, a fusion feature layer and a time-domain convolutional network;

[0077] Among them, the dual-channel feature encoder includes a visual channel encoder and a clinical channel encoder. The visual channel encoder uses 3D-CNN to extract spatiotemporal features, and the clinical channel encoder uses a Transformer network layer to encode NIHSS score, lesion volume and side.

[0078] Step S105, calculate the risk level of the patient's latent affective disorder based on the probability of the patient's latent affective disorder, and present the results to the doctor for auxiliary diagnosis in the form of visualization, which includes a facial expression change heat map, an optical flow distribution map, and an emotional state probability value.

[0079] In this embodiment, the expression change heat map uses Grad-CAM++ to backpropagate the hidden layer activations of the Bi-LSTM to obtain an importance weight map for each facial region. Grad-CAM++ can capture more salient regions, helping doctors better understand the key facial regions that the model focuses on. The optical flow distribution map is obtained by mapping the optical flow vector field to the HSV color space, where the amplitude is encoded as the brightness channel and the direction is encoded as the hue channel. By mapping the optical flow vector field to the HSV color space, the amplitude and direction of facial muscle movements can be displayed, allowing doctors to quickly identify subtle facial muscle movement patterns. By combining the expression change heat map and the optical flow distribution map, doctors can simultaneously observe which facial regions contribute most to the model's judgment of affective disorders (heat map) and the muscle movement patterns of these facial regions (optical flow), thereby providing a more comprehensive assessment of the patient's condition. For example, the heat map shows a bright red area near the corners of the patient's mouth, indicating that the model focuses on subtle changes in the corners of the mouth when determining whether the patient has a latent affective disorder, such as whether the mouth corners are symmetrical and whether there are subtle muscle twitches. The optical flow distribution map shows that on the hemiplegic side of the patient's face, the amplitude and direction of muscle movement are significantly smaller than on the non-hemiplegic side. In addition, the corners of the mouth highlighted by the heat map show weak but consistent muscle movement in the optical flow distribution map, indicating that the patient is trying to hide his emotions. Combining the expression change heat map and the optical flow distribution map, the doctor found that the corners of the mouth area, where the model focuses on, do have asymmetric muscle movement and that the patient may be trying to suppress his emotions. Combined with the patient's NIHSS score and lesion location information, it can be more certain whether the patient has a hidden affective disorder.

[0080] In an optional manner, the method further includes:

[0081] Circle an area of ​​the expression change heat map through a touch screen, automatically extract the spatiotemporal features of the area and associate them with a medical knowledge base to display key points for differential diagnosis;

[0082] By adjusting the emotion probability warning threshold, the false positive / false negative rate change curve under the current setting can be fed back instantly;

[0083] The patient's historical screening records are loaded synchronously through multi-dimensional comparative views, and the progression trend of affective disorders is displayed through the optical flow trajectory remapping method.

[0084] In this embodiment, doctors can focus on specific high-risk facial areas in the heat map. By automatically linking the medical knowledge base, doctors can then provide relevant diseases or pathologies in that area. This significantly improves diagnostic efficiency and accuracy, reduces diagnostic bias caused by human factors, and helps doctors better understand the reasons behind changes in facial expressions. Doctors can dynamically adjust the emotion probability warning threshold based on actual conditions and clinical experience and immediately observe the impact of the adjustment on false positive and false negative rates. This allows them to balance the diagnostic risks at different thresholds. For example, lowering the threshold in high-risk populations to reduce missed diagnoses, while raising the threshold in low-risk populations to reduce misdiagnoses.

[0085] For example, a doctor circles the area near the corners of a patient's mouth on a heat map, automatically extracts the spatiotemporal features of that area, and retrieves from the medical knowledge base whether facial expression changes in that area are associated with "pseudobulbar palsy" or "emotional incontinence." The doctor lowers the emotional probability warning threshold from 0.5 to 0.3, and the system immediately reports that the false positive rate has increased from 10% to 20%, while the false negative rate has decreased from 5% to 2%. The doctor can then weigh the pros and cons of this information and select an appropriate threshold. The doctor reviewed the patient's screening records over the past three months and found that the density of optical flow trajectories in the corners of his mouth had gradually increased, as had the probability value of his emotional state, indicating that the patient's emotional disorder was worsening.

[0086] According to the solution provided by the present invention, facial video data of stroke patients is collected by a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, and the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side; the temporal texture change characteristics, spontaneous blinking frequency and blinking duration of each facial area are extracted based on the local binary pattern operator; the temporal texture change characteristics include the eyebrows, corners of the mouth and corners of the eyes; the optical flow field of facial muscle movement is calculated based on the dense optical flow method, and the optical flow amplitude and direction are obtained. Optical flow statistical features; the temporal texture change features, spontaneous blinking frequency and blinking duration, and the optical flow statistical features are respectively input into an independent Bi-LSTM network model to output the probability of the patient's latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical data set including the NIHSS score and lesion location information of stroke patients; the risk level of the patient's latent affective disorder is calculated based on the patient's latent affective disorder probability, and the visualization results are presented to the doctor for auxiliary diagnosis, the visualization results including a facial expression change heat map, an optical flow distribution map, and an emotional state probability value. The present invention inputs the temporal texture change features, spontaneous blinking frequency and blinking duration, and optical flow statistical features into an independent Bi-LSTM network model, thereby improving the recognition accuracy of the patient's latent affective disorder probability.

[0087] Figure 2The schematic diagram of the framework of the device for early screening of latent affective disorder of stroke based on facial micro-expressions according to an embodiment of the present invention is shown. The device for early screening of latent affective disorder of stroke based on facial micro-expressions comprises:

[0088] A facial video acquisition module 210 is configured to acquire facial video data of a stroke patient using a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, the two side cameras being respectively disposed on either side of the patient's head to capture asymmetric features of facial muscle movement on the hemiplegic side;

[0089] A temporal texture feature extraction module 220 is configured to extract temporal texture variation features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture variation features include those between the eyebrows, at the corners of the mouth, and at the corners of the eyes;

[0090] An optical flow calculation module 230 is configured to calculate the optical flow field of facial muscle motion according to a dense optical flow method, and obtain optical flow statistical features including optical flow amplitude and optical flow direction;

[0091] a feature input module 240 for inputting the temporal texture change feature, the spontaneous blink frequency and blink duration, and the optical flow statistical feature into an independent Bi-LSTM network model, and outputting a probability of the patient having a latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical dataset including NIHSS scores and lesion location information of stroke patients;

[0092] The risk level calculation module 250 is used to calculate the risk level of the patient's latent affective disorder based on the probability of the patient's latent affective disorder, and present the results to the doctor for auxiliary diagnosis in the form of visualization. The visualization results include a heat map of expression changes in the facial area, an optical flow distribution map, and an emotional state probability value.

[0093] Figure 3 The schematic diagram of the structure of the computing device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0094] like Figure 3 As shown, the computing device may include: a processor 302 , a communications interface 304 , a memory 306 , and a communication bus 308 .

[0095] Processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308. Communication interface 304 is used to communicate with other devices, such as client devices or other server network elements. Processor 302 is used to execute program 310, which specifically performs the steps described in the aforementioned embodiment of the method for early screening of latent affective disorder in stroke patients using facial micro-expressions.

[0096] Specifically, the program 310 may include program codes, which include computer operation instructions.

[0097] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in a computing device may be of the same type, such as one or more CPUs, or may be of different types, such as one or more CPUs and one or more ASICs.

[0098] The memory 306 is used to store the program 310. The memory 306 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0099] According to the solution provided by the present invention, facial video data of stroke patients is collected by a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, and the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side; the temporal texture change characteristics, spontaneous blinking frequency and blinking duration of each facial area are extracted based on the local binary pattern operator; the temporal texture change characteristics include the eyebrows, corners of the mouth and corners of the eyes; the optical flow field of facial muscle movement is calculated based on the dense optical flow method, and the optical flow amplitude and direction are obtained. Optical flow statistical features; the temporal texture change features, spontaneous blinking frequency and blinking duration, and the optical flow statistical features are respectively input into an independent Bi-LSTM network model to output the probability of the patient's latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical data set including the NIHSS score and lesion location information of stroke patients; the risk level of the patient's latent affective disorder is calculated based on the patient's latent affective disorder probability, and the visualization results are presented to the doctor for auxiliary diagnosis, the visualization results including a facial expression change heat map, an optical flow distribution map, and an emotional state probability value. The present invention inputs the temporal texture change features, spontaneous blinking frequency and blinking duration, and optical flow statistical features into an independent Bi-LSTM network model, thereby improving the recognition accuracy of the patient's latent affective disorder probability.

[0100] Those skilled in the art will appreciate that modules in the devices of the embodiments may be adaptively modified and deployed in one or more devices different from the embodiments. Modules, units, or components in the embodiments may be combined into a single module, unit, or component, and furthermore, they may be divided into multiple submodules, subunits, or subcomponents. All features disclosed in this specification (including the accompanying claims, abstract, and drawings), as well as all processes or units of any method or device disclosed therein, may be combined in any combination, except where at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that provides the same, equivalent, or similar purpose. Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination. The present invention may be implemented using hardware comprising a number of different elements and using a suitably programmed computer. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. Unless otherwise specified, the steps in the above embodiments should not be understood as limiting the order of execution.

Claims

1. A method for early screening of latent affective disorders in stroke patients using facial micro-expressions, characterized in that: include: Facial video data of a stroke patient is collected using a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, wherein the two side cameras are respectively set on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side; Extracting temporal texture change features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture change features include the area between the eyebrows, the corners of the mouth, and the corners of the eyes; The optical flow field of facial muscle movement is calculated according to the dense optical flow method, and the optical flow statistical features including the optical flow amplitude and direction are obtained; The temporal texture change features, the spontaneous blink frequency and blink duration, and the optical flow statistical features are respectively input into an independent Bi-LSTM network model to output the probability of the patient's latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical dataset including NIHSS scores and lesion location information of stroke patients; The risk level of the patient's latent affective disorder is calculated based on the probability of the patient's latent affective disorder, and the visual results are presented to the doctor for auxiliary diagnosis. The visual results include a facial expression change heat map, an optical flow distribution map, and an emotional state probability value.

2. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: The extracting of temporal texture variation features of each facial region according to the local binary pattern operator further comprises: The facial video data is divided into multiple time windows, each time window contains multiple frame images; For each frame image in each time window, the local binary pattern features are calculated to obtain the local binary pattern feature matrix; The time change rate of the feature matrix in each time window is calculated to obtain the time domain texture change features.

3. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: The optimization formula for light source compensation of the adjustable light source system is: Among them, Ω is the ROI area of ​​the hemiplegic side face; is the incident light direction vector; N(x,y) is the normal vector of each pixel; (x,y) is the coordinate of the pixel in the facial image; Z(x,y) is the patient's facial depth map obtained by structured light projection; (x0,y0) is the coordinate of the eye corner on the lesion side; α is the spatial attenuation coefficient; θ, are the elevation angle and azimuth angle of the light source respectively.

4. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: The independent Bi-LSTM network model includes a dual-channel feature encoder, a dynamic weight generator, a fusion feature layer and a time-domain convolutional network; Among them, the dual-channel feature encoder includes a visual channel encoder and a clinical channel encoder. The visual channel encoder uses 3D-CNN to extract spatiotemporal features, and the clinical channel encoder uses a Transformer network layer to encode NIHSS score, lesion volume and side.

5. The method for early screening of latent affective disorder in stroke using facial micro-expressions according to claim 1, characterized in that: The expression change heat map uses Grad-CAM++ to back-propagate the hidden layer activation of Bi-LSTM to obtain the importance weight map of each facial region; The optical flow distribution map is obtained by mapping the optical flow vector field into the HSV color space, wherein the amplitude is encoded as the brightness channel and the direction is encoded as the hue channel.

6. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: The camera array also includes an infrared camera for improving the acquisition quality of facial video data in low-light environments; wherein the infrared camera works synchronously with the front camera and the side camera.

7. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: The method further comprises: Circle an area of ​​the expression change heat map through a touch screen, automatically extract the spatiotemporal features of the area and associate them with a medical knowledge base to display key points for differential diagnosis; By adjusting the emotion probability warning threshold, the false positive / false negative rate change curve under the current setting can be fed back instantly; The patient's historical screening records are loaded synchronously through multi-dimensional comparative views, and the progression trend of affective disorders is displayed through the optical flow trajectory remapping method.

8. The method for early screening of latent affective disorders in stroke using facial micro-expressions according to claim 1, characterized in that: Calculating the optical flow field of facial muscle movement according to the dense optical flow method to obtain the optical flow statistical features including the optical flow amplitude and the optical flow direction further includes: Performing face alignment according to key point coordinates, wherein the face alignment operation includes rotation, scaling and translation; The dense optical flow algorithm is used to calculate the optical flow field between consecutive frames of the face to obtain a two-dimensional vector field, where each pixel of the two-dimensional vector field corresponds to a displacement vector (u, v), where u is the horizontal displacement and v is the vertical displacement; For each pixel's displacement vector (u, v), calculate its optical flow magnitude and optical flow direction.

9. An early screening device for latent affective disorders in stroke patients based on facial micro-expressions, characterized in that: include: A facial video acquisition module is configured to collect facial video data of a stroke patient using a camera array equipped with an adjustable light source system, wherein the camera array includes a front camera and two side cameras, wherein the two side cameras are respectively arranged on both sides of the patient's head to capture the asymmetric characteristics of facial muscle movement on the hemiplegic side; A temporal texture feature extraction module is used to extract temporal texture change features, spontaneous blinking frequency, and blinking duration of each facial region based on a local binary pattern operator; the temporal texture change features include the area between the eyebrows, the corners of the mouth, and the corners of the eyes; The optical flow field calculation module is used to calculate the optical flow field of facial muscle movement based on the dense optical flow method to obtain the optical flow statistical features including the optical flow amplitude and direction; a feature input module for inputting the temporal texture change features, the spontaneous blink frequency and blink duration, and the optical flow statistical features into an independent Bi-LSTM network model, and outputting the probability of the patient having a latent affective disorder; wherein the independent Bi-LSTM network model is trained based on a clinical dataset including NIHSS scores and lesion location information of stroke patients; The risk level calculation module is used to calculate the risk level of the patient's latent affective disorder based on the probability of the patient's latent affective disorder, and present the results to the doctor for auxiliary diagnosis in the form of visualization. The visualization results include a heat map of expression changes in the facial area, an optical flow distribution map, and an emotional state probability value.

10. A computing device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned method for early screening of latent affective disorders in stroke based on facial micro-expressions.

Images