Physiological state risk detection method, device, equipment and program product

Abstract

The invention provides a physiological state risk detection method, device, equipment and program product, which can be applied to the technical field of computers. The method comprises the following steps: acquiring a facial expression image, an eyeball tracking image, a limb balance image and audio data; key point coordinate values of a plurality of facial key points in the facial expression image are processed based on a first random forest to obtain a facial result, and the facial result represents a facial expression attribute and a facial symmetry attribute; inputting an eyeball movement distance obtained from the eyeball tracking image into a second random forest, and outputting a pupil staring result; inputting the related coordinate values of the limb key points in the limb balance image into a third random forest, and outputting a limb balance state result; performing voice analysis on the audio data based on a convolutional neural network to obtain a voice definition result; and inputting the face result, the pupil staring result, the limb balance state result and the voice definition result into a fourth random forest, and outputting a stroke risk detection result.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a physiological state risk detection method, device, equipment and program product. Background Art

[0002] "Stroke" is a type of disease caused by impaired blood circulation in the brain, which can lead to local neurological dysfunction. Although stroke can cause great harm to our bodies, fortunately, stroke can be prevented and treated. For patients, stroke is not recognized until the stroke is more serious. The inventors found that although the symptoms of stroke may be well known, due to personal fluke mentality, cost, laziness and busy schedules, treatment will not be carried out until the stroke is more serious. Therefore, there is an urgent need for a solution that can detect stroke early. When the patient feels the corresponding symptoms of stroke in the body, the patient can be given a stroke warning, so that the patient can pay attention to the discomfort symptoms in the body, consult a doctor or seek medical treatment in time to avoid serious consequences. Summary of the invention

[0003] In view of the above problems, the present invention provides a physiological state risk detection method, device, equipment and program product.

[0004] According to a first aspect of the present invention, a physiological state risk detection method is provided, comprising: acquiring facial expression images, eye tracking images, limb balance images and audio data; processing key point coordinate values ​​of multiple facial key points in the facial expression image based on a first random forest to obtain facial results, wherein the facial results represent facial expression attributes and facial symmetry attributes; inputting the eye movement distance obtained from the eye tracking image into a second random forest, and outputting pupil gaze results; inputting the relevant coordinate values ​​of limb key points in the limb balance image into a third random forest, and outputting limb balance state results; performing speech analysis on the audio data based on a convolutional neural network to obtain speech clarity results; inputting the facial results, pupil gaze results, limb balance state results and speech clarity results into a fourth random forest, and outputting stroke risk detection results.

[0005] The second aspect of the present invention provides a training method for a detection model for detecting a physiological state, characterized in that the detection model includes a first random forest, a second random forest, a third random forest, a fourth random forest and a convolutional neural network; the training method includes: obtaining a sample facial expression image, a sample eye tracking image, a sample limb balance image, a sample audio data and a stroke risk label; processing the sample key point coordinate values ​​of multiple facial key points in the sample facial expression image based on the first random forest to obtain a sample facial result, wherein the facial result represents the facial expression attribute and the facial symmetry attribute; inputting the eye movement distance obtained from the sample eye tracking image into the second random forest, and outputting a sample pupil gaze result; inputting the relevant coordinate values ​​of the limb key points in the sample limb balance image into the third random forest, and outputting a sample limb balance state result; performing speech analysis on the sample audio data based on the convolutional neural network to obtain a sample speech clarity result; inputting the sample facial result, the sample pupil gaze result, the sample limb balance state result and the sample speech clarity result into the fourth random forest, and outputting a sample stroke risk detection result; training the detection model according to the sample stroke risk detection result and the stroke risk label to obtain a trained detection model.

[0006] The third aspect of the present invention provides a physiological state detection device, including: an acquisition module, used to acquire facial expression images, eye tracking images, limb balance images and audio data; a facial result acquisition module, used to process the key point coordinate values ​​of multiple facial key points in the facial expression image based on the first random forest to obtain facial results, wherein the facial results represent facial expression attributes and facial symmetry attributes; a pupil gaze result output module, used to input the eye movement distance obtained from the eye tracking image into the second random forest, and output the pupil gaze result; a limb balance state result output module, used to input the relevant coordinate values ​​of the limb key points in the limb balance image into the third random forest, and output the limb balance state result; a speech clarity result acquisition module, used to perform speech analysis on audio data based on a convolutional neural network to obtain a speech clarity result; a stroke risk detection result output module, used to input the facial results, pupil gaze results, limb balance state results and speech clarity results into the fourth random forest, and output the stroke risk detection result.

[0007] A fourth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above method.

[0008] The fifth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above method.

[0009] The sixth aspect of the present invention also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0010] According to a physiological state risk detection method, device, equipment and program product provided by the present invention, the key point coordinate values ​​of multiple facial key points in a facial expression image are processed based on a first random forest to obtain facial results; the eye movement distance obtained from the eye tracking image is input into the second random forest to output the pupil gaze result; the relevant coordinate values ​​of the limb key points in the limb balance image are input into the third random forest to output the limb balance state result; the audio data is subjected to speech analysis based on a convolutional neural network to obtain the speech clarity result; the facial results, pupil gaze results, limb balance state results and speech clarity results are input into the fourth random forest to output the stroke risk detection result, and the stroke risk is accurately assessed by analyzing the facial symmetry, limb balance, speech clarity and gaze performance of the tester, and timely reminders are given to stroke risk patients according to the stroke risk detection results to avoid serious consequences caused by ignorance of stroke symptoms. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0012] Figure 1 A flowchart of a physiological state risk detection method according to an embodiment of the present invention is shown;

[0013] Figure 2 A diagram showing the effect of acquiring facial key points according to an embodiment of the present invention is shown;

[0014] Figure 3 A schematic diagram of the random forest principle according to an embodiment of the present invention is shown;

[0015] Figure 4 A schematic diagram of the key point effect of a limb according to an embodiment of the present invention is shown;

[0016] Figure 5 A limb balance analysis flow chart according to an embodiment of the present invention is shown;

[0017] Figure 6 A comparison between a general neural network structure (left) and a residual network structure (right) according to an embodiment of the present invention is shown;

[0018] Figure 7 The ResNet18 network structure according to an embodiment of the present invention is shown;

[0019] Figure 8A flow chart of stroke risk assessment according to an embodiment of the present invention is shown;

[0020] Fig. 9 A flow chart of facial balance state analysis according to an embodiment of the present invention is shown;

[0021] Fig.10 A diagram showing the key points of pupil and eye positions according to an embodiment of the present invention is shown;

[0022] Fig.11 A flow chart of gaze function analysis according to an embodiment of the present invention is shown;

[0023] Fig.12 A flow chart of speech intelligibility analysis according to an embodiment of the present invention is shown;

[0024] Fig.13 shows speech data and corresponding spectrogram according to an embodiment of the present invention;

[0025] Fig.14 A flow chart of a method for training a detection model for detecting a physiological state according to an embodiment of the present invention is shown;

[0026] Fig.15 It shows a structural block diagram of a physiological state detection device according to an embodiment of the present invention;

[0027] Fig.16A A facial symmetry judgment page display diagram according to an embodiment of the present invention is shown;

[0028] Fig. 16B An eye tracking status page display diagram according to an embodiment of the present invention is shown;

[0029] Fig. 16C It shows a limb balance judgment page display diagram according to an embodiment of the present invention;

[0030] Fig.16D A speech clarity judgment page display diagram according to an embodiment of the present invention is shown;

[0031] Fig.17 A diagram showing the system start and result interface display according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] "Stroke" is also called cerebral infarction or cerebral apoplexy. Common symptoms include crooked mouth, poor speech, and limb incoordination. Because stroke occurs suddenly, has various symptoms, and changes rapidly, similar to the fickleness of wind, it is named "stroke" or "stroke". Although stroke can cause great harm to our body, fortunately, stroke can be prevented and treated. Through a healthy lifestyle, such as effective control of high blood pressure, emotions, not smoking, eating healthy food, actively exercising, maintaining a healthy weight, and controlling diabetes, stroke can be effectively prevented. If you unfortunately have a stroke. Treatment time is more important. For stroke or, timely treatment can minimize the impact of stroke on the individual's body, even avoid death, and improve clinical outcomes after treatment. For patients with ischemic stroke, if they can reach the hospital within 3.0 to 4.5 hours after the onset of symptoms, they can use drugs to dissolve the clot. If they reach the hospital within 6 to 24 hours, they can still be effectively treated with mechanical thrombotomy. If a stroke is not effectively treated for a long time, it will lead to serious consequences due to insufficient blood supply to brain cells, which may even be life-threatening. Therefore, the "golden three hours" are generally recommended for stroke patients, and every second counts.

[0037] There are many existing methods for early warning of stroke. One is to obtain the patient's first and second physiological signals through multiple sensors to determine the patient's stroke status. This method uses multiple sensors to analyze multiple signals such as heart rate, breathing, blood pressure, images, text, etc. to determine the stroke status. This method is cumbersome to use, and the contact measurement method also makes it difficult to measure at any time, and it is difficult to give a timely warning when the patient feels uncomfortable. Another method is to judge the emotional state by detecting the subject's body information, grip information of the left and right hands, facial information, and the pressure state of the left and right feet when standing. This method can achieve early warning of the stroke status, but it is also cumbersome to use and difficult to detect and warn in time. More stroke status predictions are achieved using methods such as EEG, EMG, and MRI. This requires professional equipment in hospitals and is also cumbersome for the general public. Even in hospitals with more professionals, timely diagnosis of stroke status can be achieved without too many complex equipment.

[0038] As current stroke early warning devices are unable to detect stroke status in a timely manner, the inventor hopes to find a simple and effective test that can provide timely warnings to patients who lack stroke knowledge, so that they can receive timely stroke treatment and avoid serious consequences.

[0039] In view of this, the embodiments of the present invention provide a physiological state risk detection method, device, equipment and program product. The method includes: obtaining facial expression images, eye tracking images, limb balance images and audio data; processing the key point coordinate values ​​of multiple facial key points in the facial expression image based on the first random forest to obtain facial results, wherein the facial results represent facial expression attributes and facial symmetry attributes; inputting the eye movement distance obtained from the eye tracking image into the second random forest, and outputting the pupil gaze result; inputting the relevant coordinate values ​​of the limb key points in the limb balance image into the third random forest, and outputting the limb balance state result; performing speech analysis on the audio data based on the convolutional neural network to obtain the speech clarity result; inputting the facial result, the pupil gaze result, the limb balance state result and the speech clarity result into the fourth random forest, and outputting the stroke risk detection result.

[0040] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0041] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0042] Figure 1 A flow chart of a physiological state risk detection method according to an embodiment of the present invention is shown.

[0043] like Figure 1 As shown, the method 100 includes operations S110 to S160.

[0044] In operation S110 , a facial expression image, an eye tracking image, a limb balance image, and audio data are acquired.

[0045] In operation S120, key point coordinate values ​​of a plurality of facial key points in the facial expression image are processed based on the first random forest to obtain a facial result.

[0046] In operation S130 , the eye movement distance obtained from the eye tracking image is input to the second random forest, and the pupil gaze result is output.

[0047] In operation S140 , the relevant coordinate values ​​of the limb key points in the limb balance image are input into the third random forest, and the limb balance state result is output.

[0048] In operation S150, speech analysis is performed on the audio data based on a convolutional neural network to obtain a speech clarity result.

[0049] In operation S160 , the face result, the pupil gaze result, the limb balance state result, and the speech clarity result are input into a fourth random forest, and a stroke risk detection result is output.

[0050] Optionally, the face, body, language and gaze of the tester can be analyzed by color video signal analysis, and facial expression images, eye tracking images, body balance images and audio data can be obtained from the captured video. However, this is not limited to this, and the present invention does not limit the channel for obtaining images.

[0051] Optionally, facial asymmetry caused by stroke is mainly due to facial muscle movement disorders caused by ischemic stroke and hemorrhagic stroke, that is, cerebral hemorrhage or blockage will affect the facial nerve function, making it difficult to effectively control the facial position muscles. The initial screening of stroke can be carried out by whether the face is symmetrical. When the stroke is shallow, when the stroke person has no expression or grinning, facial symmetry is difficult to capture well, that is, in the expressionless state, the facial muscles of the stroke person do not need to be controlled more, and from the appearance, it is still a normal state. When in the expression state, compared with the expressionless state, more muscles need to be controlled, and the facial nerve function is damaged, which makes it difficult to effectively control the facial muscles, resulting in facial asymmetry. Therefore, facial symmetry can be divided into two parts. First, it is judged whether there is an expression. When testing the tester, remind the tester to grin or smile, so the judgment of whether there is an expression is changed to whether there is a grinning or smiling action; secondly, the judgment of facial symmetry.

[0052] Optionally, the key point coordinate values ​​of multiple facial key points in the facial expression image can be processed based on the first random forest to obtain a facial result. The facial result represents the facial expression attribute and the facial symmetry attribute. In this way, two discrimination results can be obtained for each image, namely, whether an expression is made and whether it is symmetrical.

[0053] Optionally, the following results can be obtained by judging each image in a face-related video: the proportion of images with expressions in all images in a video, the facial symmetry results in the expression state, and the facial symmetry results in the non-expression state. These judgment results will be used as input features for the final stroke risk assessment. For example, there are 100 frames of images in a video, of which 60 frames have expressions, then the proportion of images with expressions is 0.6, there are 40 frames of facial symmetry in the expression state, then the facial symmetry results in the expression state are 2 / 3, there are 10 frames of facial symmetry in the non-expression state, then the facial symmetry results in the non-expression state are 1 / 4.

[0054] Figure 2 A diagram showing the effect of acquiring facial key points according to an embodiment of the present invention is shown.

[0055] like Figure 2 As shown in the figure, for the judgment of expression and symmetry in each facial expression image, it can be judged in the form of facial key points, and the complete image can be directly compared with the convolutional neural network. The key points can greatly reduce the amount of calculation and improve the operation speed. For facial expression and facial symmetry analysis, most of the facial pixels are irrelevant and may even interfere. Because the convolution block analyzes the image, it classifies it according to pixel values, differential movements, etc., which means that changes in the surrounding environment and habitual movements between different expressions will affect the judgment results. This phenomenon will not occur when analyzing facial key points. The correlation of the desired key points can be directly analyzed, and other irrelevant parts are directly eliminated. In addition, symmetry and some simple facial expressions can basically be regarded as the combination of key points. Therefore, facial expression and symmetry judgment through facial key points can achieve accurate judgment of the final result in a shorter time.

[0056] Figure 3 A schematic diagram of the random forest principle according to an embodiment of the present invention is shown.

[0057] like Figure 3 As shown in the figure, random forest is an ensemble learning method and an extension of decision tree. It can improve the accuracy of prediction and reduce the risk of overfitting by building multiple decision trees (decision tree A, decision tree B, decision tree C and decision tree D) on multiple subsets of the data set, and then combining the prediction results of these decision trees to produce the final prediction. This method is chosen for judgment because it can handle a large number of features and can automatically evaluate the importance of each feature. Even for missing or unbalanced data sets, random forests can usually give good results and random forests can be trained in parallel, with a relatively fast training speed.

[0058] Optionally, the eye movement distance obtained from the eye tracking image can be input into the second random forest to output the pupil gaze result.

[0059] Optionally, the gaze function analysis is mainly to determine whether the test subject can control the eye movement as required. In the present invention, the eye tracking estimation is designed by imitating the doctor, and the eye tracking route is designed. By determining whether the test subject can control the eye tracking target well, the gaze function can be judged. Eye-related videos can be obtained, and the eye movement distance obtained for each frame of eye tracking images in the video, and the obtained multiple eye movement distances are input as judgment features into the second random forest to output the pupil gaze result. It should be noted that the left eye and the right eye are judged separately, that is, the gaze function of each eye is judged separately.

[0060] Optionally, stroke is usually caused by blockage or rupture of brain blood vessels, which leads to interruption of blood supply to certain areas of the brain, thereby causing damage or death of nerve cells in certain areas, and some of these areas involve brain areas that control balance and coordinated movement, such as the cerebellum, cerebrum or brainstem, which will cause patients to have symptoms of limb imbalance. Similarly, limb balance is not a completely unchanging state. For most patients, a short-term balance state can be maintained. When the duration is too long, unbalanced behavior will slowly appear. This change is an unconscious and uncontrollable behavior for patients. Therefore, for some patients, when conducting initial screening of stroke patients, the arm balance state of the patient for 10 to 20 seconds will be judged. By reminding the tester to maintain arm balance for 20 seconds, and then analyzing each image during this period, the relevant coordinate values ​​of the limb key points in the limb balance image can be input into the third random forest to obtain the limb balance state result. The proportion of parallel arms in multiple images can be used as the limb balance state score of the tester.

[0061] Figure 4 A schematic diagram of limb key point effects according to an embodiment of the present invention is shown.

[0062] like Figure 4 As shown in the figure, the judgment of limb balance in the limb balance image can be realized through limb key points. Different from facial key points, limb key points are less and the distribution is clearer. The shoulders, elbows, wrists, hands, abdomen and nose tips can be selected as judgment points. Among them, the shoulder, periarticular joints, wrists, hands and other position key points are mainly used for limb balance judgment, while the abdomen and nose tip are used for reference. 21 key points in the limbs can be obtained, and the X and Y coordinates corresponding to the 21 key points are separated to obtain 42 data. These 42 key data (i.e., the relevant coordinate values ​​of the limb key points) can be input into the third random forest for classification and judgment to realize the judgment of limb balance.

[0063] Figure 5 A limb balance analysis flow chart according to an embodiment of the present invention is shown.

[0064] like Figure 5 As shown, limb balance video data can be obtained S510, and the limb balance video data includes multiple frames of limb balance images. For each frame of the limb balance image, limb key points are obtained S520, and the limb key points are screened S530 to screen out important key points, such as key points at positions such as shoulders, peripheral joints, wrists, and hands. The relevant coordinate values ​​of the obtained limb key points are input into the random forest algorithm S540, and the limb balance state (limb balance or limb imbalance) corresponding to the frame of limb balance image is output S550. Each frame of the limb balance video corresponds to a limb balance state, and the limb balance state results of all limb balance images are integrated to obtain a limb balance score S560.

[0065] Optionally, speech intelligibility is an important feature for stroke risk assessment. Stroke causes interruption of blood supply to brain regions, resulting in speech problems. Speech analysis can be performed on audio data based on convolutional neural networks to obtain speech intelligibility results.

[0066] Figure 6 A comparison between a basic convolutional neural network structure (left) and a residual neural network structure (right) according to an embodiment of the present invention is shown.

[0067] like Figure 6 As shown in the figure, in the traditional basic convolutional neural network structure (left), the network layer usually learns the mapping relationship between input and output, but as the network layer deepens, the problem of gradient disappearance will occur. The residual neural network structure (right) introduces residual blocks to learn the identity mapping and learn the difference between input and output instead of directly learning the mapping. This structure makes it easier for the network to learn the identity mapping and avoid the problem of overfitting the training data.

[0068] Figure 7 The ResNet18 network structure according to an embodiment of the present invention is shown.

[0069] like Figure 7 As shown in Figure 1, ResNet-18 is a deep convolutional neural network architecture that is part of the residual network family. Due to its relatively shallow structure, it is suitable for resource-constrained scenarios, but still provides powerful feature learning capabilities.

[0070] Optionally, facial results, pupil gaze results, limb balance status results and speech clarity results can be input into a fourth random forest to output stroke risk detection results.

[0071] Optionally, multiple valid features can be obtained by analyzing different indicators. First, for facial result analysis, the expression ratio, facial balance ratio in expression state, and facial balance ratio in non-expression state can be obtained. For gaze analysis, the left eye and right eye gaze scores can be obtained. For limb balance, the proportion of unbalanced states in multiple images can be obtained. Finally, for speech, multiple audio data of the tester can be obtained to obtain the corresponding multiple speech clarity results. All the obtained results can be used as data features and input into the random forest classifier for classification to obtain the final stroke risk detection results, thereby realizing end-to-end evaluation of stroke risk through video or image.

[0072] Optionally, the present invention is based on the FAST theory. FAST is a simple and effective method for stroke detection. It can quickly determine whether it is a stroke by identifying stroke-related symptoms. It is a widely used stroke identification method. "F" represents the face. By asking the patient to smile, it is determined whether there is a drooping or crooked face. The smile is because the facial asymmetry can be better observed in the expression state. "A" represents the arm. The patient's arm is kept straight for a period of time to observe whether one arm is drooping or unable to be raised. "S" represents speech. By analyzing the clarity of the patient's speech, it is determined whether there is a state of unclear speech. "T" represents time. When one of the above symptoms occurs, it is necessary to seek medical treatment in time to avoid delaying the best treatment time. The present invention is based on the FAST theory, and refers to many theoretical foundations of stroke detection, and refers to the judgment of stroke symptoms in the stroke scale, and adds the gaze function judgment. Finally, by comprehensively analyzing the degree of symmetry, eye horizontal gaze, speech clarity and limb balance in the facial smile state, the patient's stroke risk is comprehensively judged, and timely warning is given to stroke patients so that patients can receive timely treatment. By collecting patients' smiling facial videos, eye tracking effect videos, voice data, and limb balance videos, and analyzing the risk level of stroke in different information data, the test subject is ultimately given a stroke risk score. This can provide early warnings to patients who are more likely to have a stroke, thereby achieving early diagnosis and warning of stroke.

[0073] Figure 8 A flow chart of stroke risk assessment according to an embodiment of the present invention is shown.

[0074] like Figure 8As shown, the face can be tested S810, the facial data in the facial expression image can be obtained S811, the face can be operated accordingly, and the facial result S812 can be determined. The eyes can be tested S820, the eye data in the eye tracking image can be obtained S821, the eyes can be operated accordingly, and the pupil gaze result S822 can be determined. The voice can be tested S830, the audio data can be collected S831, the voice data can be operated using the corresponding method, and the voice clarity result S832 can be determined. The limbs can be tested S840, the limb data in the limb balance image can be obtained S841, and the balance of the limbs can be tested using the corresponding method of limb balance, so as to determine the limb balance state result S842. The determined facial result S812, pupil gaze result S822, limb balance state result S832 and voice clarity result S842 are input into the fourth random forest S850, so as to obtain the stroke risk result S860. Among them, the stroke risk results obtained cannot directly lead to the diagnosis of the disease or health status, but can only be used as an intermediate result.

[0075] According to a physiological state risk detection method provided by the present invention, the key point coordinate values ​​of multiple facial key points in a facial expression image are processed based on a first random forest to obtain a facial result; the eye movement distance obtained from the eye tracking image is input into the second random forest to output a pupil gaze result; the relevant coordinate values ​​of the limb key points in the limb balance image are input into the third random forest to output a limb balance state result; based on a convolutional neural network, a speech analysis is performed on the audio data to obtain a speech clarity result; the facial result, the pupil gaze result, the limb balance state result and the speech clarity result are input into a fourth random forest to output a stroke risk detection result, and an accurate assessment of the stroke risk is achieved by analyzing the facial symmetry, limb balance, speech clarity and gaze performance of the tester, and timely reminders are given to stroke risk patients according to the stroke risk detection results to avoid serious consequences caused by ignorance of stroke symptoms.

[0076] Optionally, obtaining facial results based on key point coordinate values ​​of multiple facial key points in the facial expression image processed by the first random forest includes: normalizing the facial coordinate values ​​of the multiple facial key points to obtain normalized facial coordinate values; inputting the normalized facial coordinate values ​​into the first random forest to perform facial expression and facial symmetry recognition to obtain facial expression results and facial symmetry results, the facial expression results indicating whether there is an expression, and the facial symmetry results indicating whether the face is symmetrical; obtaining facial results based on the facial expression results and the facial symmetry results.

[0077] Optionally, the facial coordinate values ​​of multiple facial key points can be normalized to obtain normalized facial coordinate values. After acquiring the facial expression image, 468 facial key points can be first acquired, each key point includes an X coordinate and a Y coordinate, and the X and Y coordinates are stored separately, so that a total of 936 data are obtained for a facial expression image, and then the acquired data is normalized. Data normalization can highlight the direct relative relationship of the data, remove the interference caused by data acquisition, and input the normalized facial coordinate values ​​into the first random forest for facial expression and facial symmetry recognition, thereby obtaining facial expression results and facial symmetry results. The facial expression results indicate whether there is an expression, and the facial symmetry results indicate whether the face is symmetrical; according to the facial expression results and facial symmetry results, the facial results are obtained.

[0078] Optionally, the first random forest includes a first random forest layer and a second random forest layer; wherein, inputting the normalized facial coordinate values ​​into the first random forest for facial expression and facial symmetry recognition to obtain facial expression results and facial symmetry results includes: inputting the normalized facial coordinate values ​​into the first random forest layer for facial expression recognition to obtain facial expression results; inputting the normalized facial coordinate values ​​into the second random forest layer for facial symmetry recognition to obtain facial symmetry results.

[0079] Optionally, the normalized facial coordinate values ​​may be input into a first random forest layer for facial expression recognition to obtain a facial expression result; the normalized facial coordinate values ​​may be input into a second random forest layer for facial symmetry recognition to obtain a facial symmetry result.

[0080] Fig. 9 A flow chart of facial balance state analysis according to an embodiment of the present invention is shown.

[0081] like Fig. 9 As shown, firstly, a face-related video is obtained S910, and facial key points in each frame of facial expression image in the face-related video are obtained S920. Based on the facial key points obtained by the first random forest layer processing, a facial expression result S940 is determined. Based on the facial key points obtained by the second random forest layer processing, a facial symmetry result S930 is determined. The obtained facial expression results and facial symmetry results are collected to obtain the facial symmetry under different facial actions S950. Finally, the results corresponding to each frame of facial expression image in the face-related video are summarized to obtain a facial result 960.

[0082] Optionally, the eye movement distance is obtained in the following manner: determine the pupil key point coordinates based on the eye tracking image, wherein the pupil key point coordinates are determined by the center position coordinates related to the pupil position and the adjacent key point coordinates adjacent to the center position coordinates; determine the left position key point coordinates and the right position key point coordinates of each eye; subtract the pupil key point coordinates from the left position key point coordinates to obtain a left distance value; subtract the right position key point coordinates from the pupil key point coordinates to obtain a right distance value; subtract the left distance value from the right distance value to obtain the eye movement distance.

[0083] Fig.10 A diagram showing the pupil and eye position key point effects according to an embodiment of the present invention is shown.

[0084] like Fig.10 As shown, for each eye in each eye tracking image, the pupil position is first obtained, and the pupil key point coordinates can be determined by the center position coordinates related to the pupil position and the coordinates of the adjacent key points adjacent to the center position coordinates. Each pupil can include five key points, mainly the center position point and four key points of top, bottom, left and right, and then the center point of these five key points is obtained as the pupil key point.

[0085] Get the key points around the eyes and divide them into left area key points and right area key points according to the point distribution. Get the center positions of the two area key points as the left position key point and the right position key point of an eye, so as to determine the coordinates of the left position key point and the right position key point of each eye.

[0086] Optionally, the left distance value may be obtained by subtracting the pupil key point coordinates from the left position key point coordinates, and the right distance value may be obtained by subtracting the pupil key point coordinates from the right position key point coordinates. The left distance value and the right distance value may be subtracted to obtain the eye movement distance. The eye movement distance may be used as a characteristic value for determining the gaze function in an image.

[0087] Optionally, the direction of eye movement can be added to the judgment feature by obtaining the eye movement distance. Compared with directly calculating the distance between the pupil key point and the eye center key point, calculating the distance between the pupil and the center of the left and right areas and subtracting them can add the direction factor on the original basis. In this way, the pupil movement trajectory can be better analyzed based on the distance change analysis, thereby increasing the judgment accuracy.

[0088] Fig.11 A gaze function analysis flow chart according to an embodiment of the present invention is shown.

[0089] like Fig.11As shown, firstly, the face-related video is obtained S1110, the facial key points in each frame of the facial expression image in the face-related video are obtained S1120, and then the center point of the left area of ​​the eye S1150, the center point of the right area of ​​the eye S1130 and the center point of the pupil S1140 are obtained respectively. Then, the position of the pupil relative to the eye is calculated based on these three data S1160, and finally the pupil movement data (i.e., the eye movement distance of each frame in the face-related video) is obtained S1170, which is input into the second random forest to obtain the pupil gaze result S1180.

[0090] Optionally, preprocessing the audio data includes: trimming silent parts in the audio data to obtain trimmed audio data; performing noise reduction processing on the trimmed audio data using a Wiener filter algorithm to obtain audio noise reduction data; trimming the length of the audio noise reduction data based on a preset length to obtain first audio data; and processing the first audio data based on a Mel spectrum to obtain preprocessed audio data.

[0091] Fig.12 A flow chart of speech intelligibility analysis according to an embodiment of the present invention is shown.

[0092] Fig.13 Speech data and corresponding spectrograms according to an embodiment of the present invention are shown.

[0093] like Fig.12 As shown, first input audio data S1210, first trim the silent part of the audio S1220, then perform the Wiener filter algorithm on the trimmed audio data to reduce the noise in the voice data and obtain audio noise reduction data, and the length of the audio noise reduction data can be trimmed based on a preset length S1230 to unify the lengths of multiple audio data. Finally, the Mel spectrum can be calculated for the first audio data S1240 and converted into a single-channel image form to obtain preprocessed audio data. Perform speech analysis on the audio data based on a convolutional neural network to obtain a speech clarity result S1250.

[0094] There are two methods for trimming the length of the audio noise reduction data. One method is to only retain the audio data of the preset length if the length of the audio noise reduction data is greater than the preset length. The other method is to Fig.13 As shown, if the length of the audio noise reduction data is less than the preset length, enough zeros can be added to the end of the audio noise reduction data until its length is the same as the preset length. The figure above shows the voice data after adding enough zeros, and the figure below shows the spectrum graph corresponding to the voice data.

[0095] Fig.14 A flow chart of a method for training a detection model for detecting a physiological state according to an embodiment of the present invention is shown.

[0096] like Fig.14 As shown, the method 1400 includes operations S1410 to S1470.

[0097] In operation S1410 , sample facial expression images, sample eye tracking images, sample limb balance images, sample audio data, and stroke risk labels are acquired.

[0098] In operation S1420, sample key point coordinate values ​​of a plurality of facial key points in the sample facial expression image are processed based on the first random forest to obtain a sample facial result.

[0099] In operation S1430, the eye movement distance obtained from the sample eye tracking image is input to the second random forest, and a sample pupil gaze result is output.

[0100] In operation S1440, the relevant coordinate values ​​of the limb key points in the sample limb balance image are input into the third random forest, and the sample limb balance state result is output.

[0101] In operation S1450, speech analysis is performed on the sample audio data based on a convolutional neural network to obtain a sample speech clarity result.

[0102] In operation S1460, the sample face result, the sample pupil gaze result, the sample limb balance state result, and the sample speech clarity result are input into the fourth random forest, and the sample stroke risk detection result is output.

[0103] In operation S1470, a detection model is trained according to the sample stroke risk detection result and the stroke risk label to obtain a trained detection model.

[0104] Optionally, the sample facial expression images, sample eye tracking images and sample limb balance images can be obtained from a video or by taking multiple photos of the tester. Here, there is no limitation on the method of obtaining the sample facial expression images, sample eye tracking images and sample limb balance images, and multiple audio data with the same content of the tester can be obtained.

[0105] Optionally, the sample key point coordinate values ​​of multiple facial key points in the sample facial expression image can be processed based on the first random forest to obtain a sample facial result, and the sample facial result may include a facial symmetry ratio, a facial expression ratio, etc.

[0106] Optionally, the eye movement distance obtained from the sample eye tracking image can be input into the second random forest to output a sample pupil gaze result, wherein the sample through-hole gaze result represents the gaze function of the eye, and the gaze function judgment score of each eye can be used as the final sample through-hole gaze result.

[0107] Optionally, the coordinate values ​​of the key points of the limbs in the sample limb balance image can be input into the third random forest to output the sample limb balance state result. The sample limb balance state result includes the proportion of the image in the limb unbalanced state, which can be used as the limb balance state score.

[0108] Optionally, a speech analysis may be performed on the sample audio data based on a convolutional neural network to obtain a sample speech clarity result. Obtaining the sample speech clarity results corresponding to multiple audio data may make the results more accurate.

[0109] Optionally, sample facial results, sample pupil gaze results, sample limb balance state results and sample speech clarity results may be input into a fourth random forest to output sample stroke risk detection results.

[0110] Optionally, a detection model may be trained according to the sample stroke risk detection results and the stroke risk labels to obtain a trained detection model.

[0111] Optionally, training a detection model according to sample stroke risk detection results and stroke risk labels to obtain a trained detection model includes: processing sample stroke risk detection results and stroke risk labels according to a loss function to obtain a loss value; and training the detection model according to the loss value to obtain a trained detection model.

[0112] Optionally, the sample stroke risk detection results and stroke risk labels can be processed according to a loss function to obtain a loss value; then the loss value can be used to train a detection model to obtain a trained detection model.

[0113] Based on the above physiological state risk detection method, the present invention also provides a detection device. Fig.15 The device is described in detail.

[0114] Fig.15 A structural block diagram of a physiological state detection device according to an embodiment of the present invention is shown.

[0115] like Fig.15 As shown, the detection device 1500 of this embodiment includes an acquisition module 1510, a facial result acquisition module 1520, a pupil gaze result module 1530, a limb balance state result module 1540, a speech clarity result module 1550 and a stroke risk detection result module 1560.

[0116] The acquisition module 1510 is used to acquire facial expression images, eye tracking images, limb balance images and audio data.

[0117] The facial result obtaining module 1520 is used to obtain a facial result by processing the key point coordinate values ​​of multiple facial key points in the facial expression image based on the first random forest, wherein the facial result represents the facial expression attribute and the facial symmetry attribute.

[0118] The pupil gaze result module 1530 is used to input the eye movement distance obtained from the eye tracking image into the second random forest and output the pupil gaze result.

[0119] The limb balance state result module 1540 is used to input the relevant coordinate values ​​of the limb key points in the limb balance image into the third random forest and output the limb balance state result.

[0120] The speech clarity result module 1550 is used to perform speech analysis on the audio data based on a convolutional neural network to obtain a speech clarity result.

[0121] The stroke risk detection result module 1560 is used to input the facial results, pupil gaze results, limb balance state results and speech clarity results into the fourth random forest, and output the stroke risk detection results.

[0122] According to a physiological state risk detection method provided by the present invention, the key point coordinate values ​​of multiple facial key points in a facial expression image are processed based on a first random forest to obtain a facial result; the eye movement distance obtained from the eye tracking image is input into the second random forest to output a pupil gaze result; the relevant coordinate values ​​of the limb key points in the limb balance image are input into the third random forest to output a limb balance state result; based on a convolutional neural network, a speech analysis is performed on the audio data to obtain a speech clarity result; the facial result, the pupil gaze result, the limb balance state result and the speech clarity result are input into a fourth random forest to output a stroke risk detection result, and an accurate assessment of the stroke risk is achieved by analyzing the facial symmetry, limb balance, speech clarity and gaze performance of the tester, and timely reminders are given to stroke risk patients according to the stroke risk detection results to avoid serious consequences caused by ignorance of stroke symptoms.

[0123] Optionally, the facial result obtaining module includes: a normalization unit, a facial expression symmetry unit and a facial result unit.

[0124] The normalization unit is used to normalize the facial coordinate values ​​of multiple facial key points to obtain normalized facial coordinate values.

[0125] The facial expression symmetry unit is used to input the normalized facial coordinate values ​​into the first random forest for facial expression and facial symmetry recognition to obtain a facial expression result and a facial symmetry result. The facial expression result indicates whether there is an expression, and the facial symmetry result indicates whether the face is symmetrical.

[0126] The facial result unit is used to obtain the facial result according to the facial expression result and the facial symmetry result.

[0127] Optionally, the facial expression symmetry unit includes: a facial expression result subunit and a facial symmetry result subunit.

[0128] The facial expression result subunit is used to input the normalized facial coordinate values ​​into the first random forest layer for facial expression recognition to obtain facial expression results.

[0129] The facial symmetry result subunit is used to input the normalized facial coordinate values ​​into the second random forest layer for facial symmetry recognition to obtain a facial symmetry result.

[0130] Optionally, the eye movement distance is obtained in the following manner: determine the pupil key point coordinates based on the eye tracking image, wherein the pupil key point coordinates are determined by the center position coordinates related to the pupil position and the adjacent key point coordinates adjacent to the center position coordinates; determine the left position key point coordinates and the right position key point coordinates of each eye; subtract the pupil key point coordinates from the left position key point coordinates to obtain a left distance value; subtract the right position key point coordinates from the pupil key point coordinates to obtain a right distance value; subtract the left distance value from the right distance value to obtain the eye movement distance.

[0131] Optionally, the speech clarity result module includes: a first pruning unit, a noise reduction unit, a second pruning unit and a Mel spectrum unit.

[0132] The first trimming unit is used to trim the silent part in the audio data to obtain trimmed audio data.

[0133] The denoising unit is used to perform denoising processing on the pruned audio data using a Wiener filtering algorithm to obtain audio denoising data.

[0134] The second trimming unit is used to trim the length of the audio noise reduction data based on a preset length to obtain the first audio data.

[0135] The mel spectrum unit is used to process the first audio data based on the mel spectrum to obtain preprocessed audio data.

[0136] Fig.16A A facial symmetry judgment page display diagram according to an embodiment of the present invention is shown.

[0137] Fig. 16B A diagram showing an eye tracking status page according to an embodiment of the present invention is shown.

[0138] Fig. 16C A limb balance judgment page display diagram according to an embodiment of the present invention is shown.

[0139] Fig.16D A display diagram of a speech clarity judgment page according to an embodiment of the present invention is shown. Fig.17 A diagram showing the system start and result interface display according to an embodiment of the present invention is shown.

[0140] The physiological state risk detection provided by the present invention includes four parts. The physiological state risk detection system has four parts of page display diagrams with different functions, such as Fig.16A , which is a page display for judging facial symmetry. Fig. 16B This is the eye tracking status page display. Fig. 16C This is the display diagram of the limb balance judgment page. Fig.16D This is a picture showing the speech clarity judgment page. Fig.17 As shown, these four parts can be tested separately, either manually or automatically, and finally the facial results, pupil gaze results, limb balance state results and speech clarity results are obtained respectively. Specifically, Fig.17 The detailed result page will be used to judge the test taker based on the results and obtain the stroke risk test results.

[0141] The training device of the detection model for detecting physiological states of this embodiment includes a sample acquisition module, a sample facial result acquisition module, a sample pupil gaze result module, a sample limb balance state result module, a sample speech clarity result module, a sample stroke risk detection result module and a training module.

[0142] A sample acquisition module is used to acquire a sample facial expression image, a sample eye tracking image, a sample limb balance image, a sample audio data and a stroke risk label. A sample facial result acquisition module is used to process the sample key point coordinate values ​​of multiple facial key points in the sample facial expression image based on the first random forest to obtain a sample facial result, wherein the facial result represents the facial expression attribute and the facial symmetry attribute. A sample pupil gaze result module is used to input the eye movement distance obtained from the sample eye tracking image into the second random forest and output a sample pupil gaze result. A sample limb balance state result module is used to input the relevant coordinate values ​​of the limb key points in the sample limb balance image into the third random forest and output a sample limb balance state result. A sample speech clarity result module is used to perform speech analysis on the sample audio data based on the convolutional neural network to obtain a sample speech clarity result. A sample stroke risk detection result module is used to input the sample facial result, the sample pupil gaze result, the sample limb balance state result and the sample speech clarity result into the fourth random forest and output a sample stroke risk detection result. The training module is used to train the detection model according to the sample stroke risk detection result and the stroke risk label to obtain a trained detection model.

[0143] Optionally, the training device for the detection model for detecting physiological state of this embodiment includes: a processing module and a model training module. The processing module is used to process the sample stroke risk detection result and the stroke risk label according to the loss function to obtain the loss value. The model training module is used to train the detection model according to the loss value to obtain the trained detection model.

[0144] Optionally, any multiple modules among the modules, submodules, units, and subunits can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the modules, submodules, units, and subunits can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0145] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teaching of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0146] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A physiological state risk detection method, characterized in that: The detection method is performed by an electronic device, and the method includes: Acquire facial expression images, eye tracking images, limb balance images, and audio data; Processing the key point coordinate values ​​of a plurality of facial key points in the facial expression image based on the first random forest to obtain a facial result, wherein the facial result represents a facial expression attribute and a facial symmetry attribute; Inputting the eye movement distance obtained from the eye tracking image into the second random forest, and outputting the pupil gaze result; Inputting relevant coordinate values ​​of limb key points in the limb balance image into a third random forest, and outputting a limb balance state result; Performing speech analysis on the audio data based on a convolutional neural network to obtain a speech clarity result; The facial result, the pupil gaze result, the limb balance state result and the speech clarity result are input into a fourth random forest to output a stroke risk detection result.

2. The method according to claim 1, characterized in that The facial result obtained by processing the key point coordinate values ​​of the plurality of facial key points in the facial expression image based on the first random forest comprises: Normalizing the facial coordinate values ​​of the multiple facial key points to obtain normalized facial coordinate values; Inputting the normalized facial coordinate values ​​into the first random forest to perform facial expression and facial symmetry recognition, and obtaining a facial expression result and a facial symmetry result, wherein the facial expression result indicates whether there is an expression, and the facial symmetry result indicates whether the face is symmetrical; The facial result is obtained according to the facial expression result and the facial symmetry result.

3. The method according to claim 2, characterized in that The first random forest includes a first random forest layer and a second random forest layer; The step of inputting the normalized facial coordinate values ​​into the first random forest to perform facial expression and facial symmetry recognition to obtain facial expression results and facial symmetry results includes: Inputting the normalized facial coordinate values ​​into the first random forest layer to perform facial expression recognition to obtain the facial expression result; The normalized facial coordinate values ​​are input into the second random forest layer for facial symmetry recognition to obtain the facial symmetry result.

4. The method according to claim 1, characterized in that: The eye movement distance is obtained by: Determine pupil key point coordinates according to the eye tracking image, wherein the pupil key point coordinates are determined by a center position coordinate related to a pupil position and a neighboring key point coordinate adjacent to the center position coordinate; Determine the left position key point coordinates and the right position key point coordinates of each eye; Subtract the pupil key point coordinates from the left position key point coordinates to obtain a left distance value; Subtract the right position key point coordinate from the pupil key point coordinate to obtain a right distance value; The left distance value is subtracted from the right distance value to obtain the eye movement distance.

5. The method according to claim 1, characterized in that Preprocessing the audio data includes: Trimming the silent part in the audio data to obtain trimmed audio data; Performing noise reduction processing on the trimmed audio data using a Wiener filter algorithm to obtain audio noise reduction data; Trimming the length of the audio noise reduction data based on a preset length to obtain first audio data; The first audio data is processed based on the Mel spectrum to obtain preprocessed audio data.

6. A method for training a detection model for detecting physiological state risks, characterized in that: The detection model includes a first random forest, a second random forest, a third random forest, a fourth random forest and a convolutional neural network; the training method includes: Obtain sample facial expression images, sample eye tracking images, sample limb balance images, sample audio data, and stroke risk labels; Processing the sample key point coordinate values ​​of a plurality of facial key points in the sample facial expression image based on the first random forest to obtain a sample facial result, wherein the facial result represents a facial expression attribute and a facial symmetry attribute; Inputting the eye movement distance obtained from the sample eye tracking image into the second random forest, and outputting the sample pupil gaze result; Inputting the relevant coordinate values ​​of the limb key points in the sample limb balance image into the third random forest, and outputting the sample limb balance state result; Performing speech analysis on the sample audio data based on the convolutional neural network to obtain a sample speech clarity result; Input the sample face result, the sample pupil gaze result, the sample limb balance state result and the sample speech clarity result into the fourth random forest, and output a sample stroke risk detection result; A detection model is trained according to the sample stroke risk detection result and the stroke risk label to obtain a trained detection model.

7. The method according to claim 6, characterized in that The step of training the detection model according to the sample stroke risk detection result and the stroke risk label to obtain the trained detection model comprises: Processing the sample stroke risk detection result and the stroke risk label according to a loss function to obtain a loss value; The detection model is trained according to the loss value to obtain a trained detection model.

8. A physiological state detection device, characterized in that: The device comprises: An acquisition module, used for acquiring facial expression images, eye tracking images, limb balance images and audio data; A facial result obtaining module, used for processing the key point coordinate values ​​of a plurality of facial key points in the facial expression image based on a first random forest to obtain a facial result, wherein the facial result represents a facial expression attribute and a facial symmetry attribute; A pupil gaze result module, used for inputting the eye movement distance obtained from the eye tracking image into the second random forest, and outputting the pupil gaze result; A limb balance state result module, used for inputting relevant coordinate values ​​of limb key points in the limb balance image into a third random forest, and outputting a limb balance state result; A speech clarity result module, used for performing speech analysis on the audio data based on a convolutional neural network to obtain a speech clarity result; The stroke risk detection result module is used to input the facial result, the pupil gaze result, the limb balance state result and the speech clarity result into the fourth random forest, and output the stroke risk detection result.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images