Stroke early warning system based on voice, image, motion data and electrocardiogram data

Abstract

The invention provides an apoplexy early warning system based on voice, images, motion data and electrocardiogram data. The apoplexy early warning system comprises a language barrier judgment module used for analyzing user voice definition and fluency based on user voice; the ataxia judgment module is used for analyzing the accuracy and continuity of the finger and nose actions of the user based on the video stream; the mouth inclination judgment module is used for analyzing the symmetry condition of the mouth relative to the central axis of the face based on the picture to obtain a mouth inclination judgment conclusion; the limb weakness judgment module is used for capturing action data in real time based on a built-in sensor and analyzing the action data; the electrocardio analysis module is used for analyzing the user electrocardio signals based on a neural network model to obtain an atrial fibrillation judgment conclusion; and the stroke risk comprehensive evaluation module is used for comprehensively evaluating the stroke risk score of the user based on a language barrier judgment conclusion, an ataxia judgment conclusion, a mouth inclination judgment conclusion, a limb weakness judgment conclusion and an atrial fibrillation judgment conclusion, and realizing risk layering.

Description

Technical Field

[0001] The present invention relates to the field of medical detection technology, and in particular, to a stroke warning system based on voice, image, motion data and electrocardiogram data, and more particularly, to a stroke warning recognition model integrating voice, image, motion data and electrocardiogram data. Background Art

[0002] At present, stroke is mainly determined by clinicians who make a comprehensive diagnosis of the patient's symptoms based on the NIHSS score. This diagnostic method is somewhat subjective and inefficient. In addition, although conventional clinician screening can record the patient's behavior patterns, there are delays in data processing and real-time feedback, making it difficult to assist patients in self-examination of stroke symptoms, causing patients to miss the "golden time" for treatment.

[0003] Patent document CN113936799A (application number: 202111193461.1) discloses a method for rapid stroke identification combining TOPSIS and artificial intelligence. First, the user's arms are raised horizontally to determine whether there is a unilateral muscle strength decrease, and the muscle strength difference is graded by the angle between the arms; then the degree of slurred speech of the user is analyzed by artificial intelligence speech recognition and combined with the error rate calculation method; then the position coordinates of the facial feature points are detected by the artificial intelligence facial feature point recognition model, and the degree of facial droop is calculated by the TOPSIS method. Finally, the TOPSIS method is used to comprehensively classify the muscle strength difference, the degree of slurred speech, and the degree of facial droop to quickly identify the risk of stroke. Although it can analyze the facial, voice, limb and other conditions of stroke patients, it lacks balance function tests such as ataxia and electrocardiogram analysis conclusions. The recognition method has certain limitations and lacks the function of predicting the onset of stroke. Summary of the invention

[0004] In view of the defects in the prior art, the object of the present invention is to provide a stroke early warning system based on voice, image, motion data and electrocardiogram data.

[0005] According to the present invention, a stroke early warning system based on voice, image, motion data and electrocardiogram data is provided, comprising:

[0006] A language barrier judgment module is used to analyze the clarity and fluency of the user's voice based on the user's voice to obtain a language barrier judgment conclusion;

[0007] The ataxia judgment module is used to analyze the accuracy and consistency of the user's finger-to-nose action based on the video stream to obtain the ataxia judgment conclusion;

[0008] The mouth corner deflection judgment module is used to analyze the symmetry of the mouth corners about the facial central axis based on the image and obtain the conclusion of the mouth corner deflection judgment;

[0009] The limb fatigue judgment module is used to capture motion data in real time based on built-in sensors, and analyze the motion data to obtain a limb fatigue judgment conclusion;

[0010] ECG analysis module, used to analyze the user's ECG signal based on a neural network model to obtain atrial fibrillation judgment conclusion;

[0011] The stroke risk comprehensive assessment module is used to comprehensively assess the user's stroke risk score and achieve risk stratification based on the language disorder judgment conclusions, ataxia judgment conclusions, mouth corner deviation judgment conclusions, limb weakness judgment conclusions and atrial fibrillation judgment conclusions.

[0012] Preferably, the language barrier judgment module includes: analyzing the clarity and fluency of the user's voice based on the user's voice, when the clarity and / or fluency does not meet the preset requirements, there is a language barrier; when the clarity and fluency meet the preset requirements, there is no language barrier.

[0013] Preferably, the language barrier determination module includes:

[0014] Obtain the user's recorded voice and upload it to the cloud platform for analysis. Use speech recognition technology to extract the text content in the user's recorded voice and compare it with the corresponding text when recording the voice to obtain the error rate. According to the error rate, the clarity of the user's voice is judged.

[0015] The determining of the clarity of the user's speech according to the word error rate includes: when the word error rate is higher than a preset value, the user's speech is considered unclear; otherwise, the user's speech is considered clear.

[0016] Preferably, the language barrier determination module includes:

[0017] A fluency judgment model is constructed based on a convolutional neural network, and the constructed fluency judgment model is trained to obtain a trained fluency judgment model; audio information to be tested is obtained, and the Mel-frequency cepstral coefficients of the audio information to be tested are extracted, and the extracted Mel-frequency cepstral coefficients are input into the trained fluency judgment model to determine whether the user's voice is fluent.

[0018] Preferably, the ataxia judgment module includes:

[0019] Videos of users completing the finger-to-nose action with their left and right hands are obtained, and the obtained videos are uploaded to the cloud platform. The preset key points of the face and hand in each frame of the video are located using a deep learning network, and the relative position relationship of the preset key points of the face and hand and the change of the position relationship of the preset key points of the face and hand in adjacent time periods are analyzed; when the position relationship of the preset key points of the face and hand remains within the preset distance range, and the change speed of the position relationship of the preset key points of the face and hand is less than the preset value, it is considered that no ataxia has occurred; otherwise, it is considered that ataxia has occurred.

[0020] Preferably, the mouth corner deflection judgment module comprises:

[0021] Obtain pictures of users taking standardized facial selfies with their left and right hands respectively, and upload the pictures to the cloud platform. Use the deep learning network to locate the preset key points in the pictures, obtain the facial central axis and the corners of the mouth based on the preset facial key points, and determine whether the corners of the mouth are crooked based on the facial central axis.

[0022] Preferably, the limb fatigue judgment module includes:

[0023] Use built-in sensors to capture the user's motion data in real time when completing standard preset movements, analyze the movement amplitude, direction change, movement duration and speed based on the obtained motion data, and judge whether there is limb weakness based on the analyzed movement amplitude, direction change, movement duration and speed according to the set threshold.

[0024] Preferably, the ECG analysis module includes: acquiring the user's ECG signal, and using a deep neural network to analyze the acquired user's ECG signal to obtain heart rate variability and ECG waveform characteristics, and using a ResNet network to determine whether atrial fibrillation exists based on the obtained heart rate variability and ECG waveform characteristics.

[0025] Preferably, the comprehensive stroke risk assessment module includes: based on the language disorder judgment conclusion, ataxia judgment conclusion, mouth corner deviation judgment conclusion, limb weakness judgment conclusion and atrial fibrillation judgment conclusion, corresponding scores are obtained according to relevant indicators of a preset stroke assessment scale, and the corresponding scores obtained are added together to obtain the user's stroke risk score to achieve risk stratification.

[0026] Preferably, the preset stroke assessment scale includes: NIHSS scoring method, FAST scoring method or CHADS scoring table.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The present invention utilizes the cloud platform architecture to deploy complex deep learning models on the cloud, avoiding the high requirements of the user end on hardware performance. By uploading videos and data to the cloud for real-time analysis, users do not need to rely on high-performance equipment to achieve real-time ataxia detection and judgment;

[0029] 2. The present invention can complete the collection of finger-to-nose movements and mouth corner deviation pictures only through the front camera of the smart device, without the need for special sensors or wearable devices, thereby reducing the hardware requirements of the system and increasing applicability and ease of use;

[0030] 3. Through the combined use of ataxia and electrocardiogram analysis, a comprehensive stroke warning effect is achieved, and a multi-dimensional assessment of the patient's health status is realized. Through this combined technology, potential stroke risks can be discovered earlier and effective warnings can be provided.

[0031] 4. The present invention realizes a comprehensive analysis of language disorders, ataxia, crooked mouth corners, limb weakness and electrocardiogram signals, thereby providing an efficient, objective and timely stroke early warning system, significantly improving the ability of early identification and intervention;

[0032] 5. The present invention extracts precise motion and facial key points and performs temporal analysis, which requires not only high-precision computer vision algorithms, but also dynamic tracking of time series, especially identifying subtle changes in movements between the hands and the face and performing accurate analysis; the present invention extracts the relative position of the hands and the face through an open source posture estimation algorithm framework, and ultimately determines whether the user's fingertips can accurately point to the tip of the nose, thereby drawing a conclusion of ataxia. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0034] Figure 1 Schematic diagram of a stroke warning system based on voice, image, motion data and electrocardiogram data. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0036] Example 1

[0037] According to the present invention, a stroke early warning system based on voice, image, motion data and electrocardiogram data is provided. Figure 1 As shown, including:

[0038] A language barrier judgment module is used to analyze the clarity and fluency of the user's voice based on the user's voice to obtain a language barrier judgment conclusion;

[0039] The ataxia judgment module is used to analyze the accuracy and consistency of the user's finger-to-nose action based on the video stream to obtain the ataxia judgment conclusion;

[0040] The mouth corner deflection judgment module is used to analyze the symmetry of the mouth corners about the facial central axis based on the image and obtain the conclusion of the mouth corner deflection judgment;

[0041] The limb fatigue judgment module is used to capture motion data in real time based on built-in sensors, and analyze the motion data to obtain a limb fatigue judgment conclusion;

[0042] ECG analysis module, used to analyze the user's ECG signal based on a neural network model to obtain atrial fibrillation judgment conclusion;

[0043] The stroke risk comprehensive assessment module is used to comprehensively assess the user's stroke risk score and achieve risk stratification based on the language disorder judgment conclusions, ataxia judgment conclusions, mouth corner deviation judgment conclusions, limb weakness judgment conclusions and atrial fibrillation judgment conclusions.

[0044] Specifically, the language barrier judgment module includes: analyzing the clarity and fluency of the user's voice based on the user's voice. When the clarity and / or fluency do not meet the preset requirements, there is a language barrier; when the clarity and fluency meet the preset requirements, there is no language barrier.

[0045] Analyze the clarity of user speech based on user speech, including: give a specific sentence, ask the user to read the sentence and complete the recording; upload the speech to the cloud platform for analysis, extract the text content in the user speech through speech recognition technology and compare it with the given sentence, and judge whether the user speech is clear by the word error rate. When the word error rate is greater than the preset value, it is considered unclear; otherwise, it is considered clear;

[0046] Analyzing the fluency of the user's speech based on the user's speech, including: analyzing the fluency of the speech through a language disorder classification model;

[0047] Among them, the speech disorder classification model includes a convolutional neural network composed of multiple one-dimensional convolutional layers, batch normalization layers, and maximum pooling layers; the user's voice is subjected to fast Fourier transform, and its Mel-frequency cepstral coefficient is calculated and used as the input of the speech disorder classification model to determine whether the user's voice is fluent.

[0048] Specifically, the ataxia judgment module includes: guiding the user to complete the finger-to-nose action with the left and right hands respectively, processing the video stream uploaded to the cloud platform in frames, extracting each frame for separate analysis, locating the face and hand key points in each frame, analyzing the relative position relationship between the two, and the change in the position relationship between the two in adjacent time periods, and judging the accuracy and continuity of the user's finger-to-nose action. When the position relationship between the preset key points of the face and the hand remains within the preset distance range, and the change speed of the position relationship between the preset key points of the face and the hand is less than the preset value, it is considered that ataxia has not occurred; otherwise, it is considered that ataxia has occurred.

[0049] More specifically, an open source posture estimation algorithm framework is used to extract key points from each 2D frame and output the two-dimensional coordinates (x, y) of these key points in the image and the hand displacement s. hand .

[0050] Analyze the relative position relationship between the face and the hand and the relationship between the distance changes over time in adjacent time periods. Through geometric calculation and time series analysis, the relative position relationship of key points is converted into the corresponding motion pattern to form a time series function, the Euclidean distance

[0051] Hand movement speed

[0052] Among them, x face (t) represents the x-axis coordinate of the facial key point at time t;

[0053] y face (t) represents the y-axis coordinate of the facial key point at time t;

[0054] x hand (t) represents the x-axis coordinate of the key point of the hand at time t;

[0055] y hand (t) represents the y-axis coordinate of the key point of the hand at time t;

[0056] t represents time.

[0057] The accuracy and consistency of judging the user's finger-to-nose action include:

[0058] In the accuracy judgment, if the Euclidean distance D(t) between the hand and the nose tip is less than 10 cm, and the finger can accurately touch the nose tip, and the position changes smoothly during the entire action, then it is determined that the accuracy requirement is met;

[0059] If the Euclidean distance D(t) between the hand and the nose tip is greater than or equal to 10 cm, or the finger cannot accurately touch the nose tip, or the position change is not stable during the entire action, it is determined that the accuracy requirement is not met.

[0060] In the continuity judgment, if the hand movement speed v is within 0.5 m / s and there is no sudden pause or rapid retreat, it means that the movement is smooth and it is determined that the continuity requirement is met;

[0061] If the hand movement speed v is greater than or equal to 0.5 m / s, or the fingers fail to touch the tip of the nose and there is obvious tremor or pause in the hand movement, it indicates that the movement is not smooth and is judged to not meet the continuity requirement.

[0062] Specifically, the mouth corner deflection judgment module includes:

[0063] Instruct users to take facial selfies with their left and right hands respectively, and upload the pictures to the cloud platform. Use deep learning algorithms to locate key facial points in the pictures, depict the facial central axis and mouth corners based on the facial key point positioning, analyze the symmetry of the mouth corners about the facial central axis to get the conclusion of the crooked mouth corners. If the mouth corners are symmetrical about the facial central axis, then the mouth corners are not crooked; otherwise, the mouth corners are crooked.

[0064] Specifically, the limb weakness judgment module includes: guiding the user to perform a standard hand-raising action, using built-in sensors (accelerometer and gyroscope) to capture action data in real time, obtaining the movement amplitude and direction change, the duration and speed of the movement, and judging whether there is limb weakness based on the analyzed movement amplitude, direction change, duration and speed according to the set threshold.

[0065] More specifically, the captured raw motion data is preprocessed, including: filtering noise and smoothing, extracting time series feature data, including: the start and end time of the motion, the maximum acceleration and the average acceleration; processing the time series feature data to obtain the motion amplitude, direction change, duration and speed of the motion; setting a threshold and judging limb fatigue to obtain a judgment conclusion;

[0066] The processing of the time series feature data, obtaining the movement amplitude, direction change, duration and speed of the movement, setting a threshold and judging limb fatigue, and obtaining a conclusion include:

[0067] After filtering the acceleration in the original motion data collected by the sensor, the filtered acceleration Integrate to get the speed

[0068] In this discrete data Perform a second integration to obtain displacement data Calculate the standard deviation of acceleration σ:

[0069] Where N represents the total number of data points considered when calculating the average;

[0070] a represents the original acceleration;

[0071] t represents the sampling time;

[0072] i represents the index variable of the i-th data point to be summed;

[0073] v0 represents the initial velocity;

[0074] t′ represents the middle moment of the integration;

[0075] t k represents the kth sampling moment;

[0076] v(t k ) represents the speed at the kth sampling moment;

[0077] s(t k ) represents the displacement at the kth sampling moment;

[0078] It represents the average acceleration of the whole process.

[0079] The acceleration standard deviation threshold is set to 0.2. If the acceleration standard deviation is higher than the threshold, it is judged that the arm shakes violently during the arm raising and there is a lack of muscle strength. If the acceleration standard deviation is less than or equal to the threshold, it is judged that the arm remains stable during the arm raising and there is no lack of muscle strength.

[0080] The displacement threshold is set to 3. If the displacement is less than the threshold, it is determined that the arm has not fallen and there is no limb weakness. If the displacement is greater than or equal to the threshold, it is determined that the arm has fallen more seriously and there is limb weakness.

[0081] Specifically, the ECG analysis module includes: training a neural network model to analyze the user's ECG signal to obtain multiple features of heart rate variability and ECG waveform, and performing atrial fibrillation judgment based on multiple features such as heart rate variability and ECG waveform through a ResNet network.

[0082] Specifically, the stroke risk comprehensive assessment module includes: comprehensively analyzing the input language disorder judgment conclusions, ataxia judgment conclusions, mouth corner deviation judgment conclusions, limb weakness judgment conclusions, and atrial fibrillation judgment conclusions, and performing a comprehensive assessment based on a formulated scoring table to obtain the user's stroke risk score and achieve risk stratification.

[0083] In the stroke risk comprehensive assessment module, relevant indicators of common stroke assessment scales (such as NIHSS (National Institutes of Health Stroke Scale) and FAST scoring method) are combined to perform weighted scoring and comprehensive analysis on the input judgment conclusions, and the atrial fibrillation judgment conclusion assessment refers to the CHADS stroke scoring table (or CHADS2 score), which is a commonly used scale for assessing the risk of stroke in patients with non-valvular atrial fibrillation.

[0084] The present invention also provides a stroke warning system based on voice, image, motion data and electrocardiogram data. The stroke warning system based on voice, image, motion data and electrocardiogram data can be realized by executing the process steps of the stroke warning method based on voice, image, motion data and electrocardiogram data, that is, those skilled in the art can understand the stroke warning method based on voice, image, motion data and electrocardiogram data as a preferred implementation of the stroke warning system based on voice, image, motion data and electrocardiogram data.

[0085] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0086] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A stroke early warning system based on voice, image, motion data and electrocardiogram data, characterized in that: include: A language barrier judgment module is used to analyze the clarity and fluency of the user's voice based on the user's voice to obtain a language barrier judgment conclusion; The ataxia judgment module is used to analyze the accuracy and consistency of the user's finger-to-nose action based on the video stream to obtain the ataxia judgment conclusion; The mouth corner deflection judgment module is used to analyze the symmetry of the mouth corners about the facial central axis based on the image and obtain the conclusion of the mouth corner deflection judgment; The limb fatigue judgment module is used to capture motion data in real time based on built-in sensors, and analyze the motion data to obtain a limb fatigue judgment conclusion; ECG analysis module, used to analyze the user's ECG signal based on a neural network model to obtain atrial fibrillation judgment conclusion; The stroke risk comprehensive assessment module is used to comprehensively assess the user's stroke risk score and achieve risk stratification based on the language disorder judgment conclusions, ataxia judgment conclusions, mouth corner deviation judgment conclusions, limb weakness judgment conclusions and atrial fibrillation judgment conclusions.

2. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The language barrier judgment module includes: analyzing the clarity and fluency of the user's voice based on the user's voice. When the clarity and / or fluency do not meet the preset requirements, there is a language barrier; when the clarity and fluency meet the preset requirements, there is no language barrier.

3. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 2, characterized in that: The language barrier determination module comprises: Obtain the user's recorded voice and upload it to the cloud platform for analysis. Use speech recognition technology to extract the text content in the user's recorded voice and compare it with the corresponding text when recording the voice to obtain the error rate. According to the error rate, the clarity of the user's voice is judged. The determining of the clarity of the user's speech according to the word error rate includes: when the word error rate is higher than a preset value, the user's speech is considered unclear; otherwise, the user's speech is considered clear.

4. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 2, characterized in that: The language barrier determination module comprises: A speech disorder classification model is constructed based on a convolutional neural network, and the constructed speech disorder classification model is trained to obtain a trained speech disorder classification model; audio information to be tested is obtained, and the Mel-frequency cepstral coefficients of the audio information to be tested are extracted, and the extracted Mel-frequency cepstral coefficients are input into the trained speech disorder classification model to determine whether the user's speech is fluent.

5. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The ataxia judgment module comprises: Videos of users completing the finger-to-nose action with their left and right hands are obtained, and the obtained videos are uploaded to the cloud platform. The preset key points of the face and hand in each frame of the video are located using a deep learning network, and the relative position relationship of the preset key points of the face and hand and the change of the position relationship of the preset key points of the face and hand in adjacent time periods are analyzed; when the position relationship of the preset key points of the face and hand remains within the preset distance range, and the change speed of the position relationship of the preset key points of the face and hand is less than the preset value, it is considered that no ataxia has occurred; otherwise, it is considered that ataxia has occurred.

6. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The mouth corner deflection judgment module comprises: Obtain pictures of users taking standardized facial selfies with their left and right hands respectively, and upload the pictures to the cloud platform. Use the deep learning network to locate the preset key points in the pictures, obtain the facial central axis and the corners of the mouth based on the preset facial key points, and determine whether the corners of the mouth are crooked based on the facial central axis.

7. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The limb fatigue judgment module comprises: Use built-in sensors to capture the user's motion data in real time when completing standard preset movements, analyze the movement amplitude, direction change, movement duration and speed based on the obtained motion data, and judge whether there is limb weakness based on the analyzed movement amplitude, direction change, movement duration and speed according to the set threshold.

8. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The ECG analysis module includes: obtaining the user's ECG signal, and using a deep neural network to analyze the obtained user's ECG signal to obtain heart rate variability and ECG waveform characteristics, and using a ResNet network to determine whether atrial fibrillation exists based on the obtained heart rate variability and ECG waveform characteristics.

9. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The comprehensive stroke risk assessment module includes: based on the language disorder judgment conclusion, ataxia judgment conclusion, mouth corner deviation judgment conclusion, limb weakness judgment conclusion and atrial fibrillation judgment conclusion, corresponding scores are obtained according to relevant indicators of the preset stroke assessment scale, and the corresponding scores obtained are added together to obtain the user's stroke risk score to achieve risk stratification.

10. The stroke early warning system based on voice, image, motion data and electrocardiogram data according to claim 1, characterized in that: The preset stroke assessment scale includes: NIHSS scoring method, FAST scoring method or CHADS scoring table.

Images