Model training method and device for assisting in determining heart disease categories

Abstract

The present invention provides a model training method and device for assisting in determining the category of heart diseases, which can be applied to the technical fields of millimeter-wave radar and non-contact heart disease detection. The method includes: obtaining an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, where the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; respectively extracting features from the electrocardiogram data set and the millimeter-wave radar data set to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period; according to the electrocardiogram features and millimeter-wave radar features at multiple moments, semantically matching the sample heartbeat information with the electrocardiogram data set to obtain aligned target electrocardiogram features and target millimeter-wave radar features; and adjusting an initial classification model trained using electrocardiogram features according to the target electrocardiogram features and target millimeter-wave radar features to obtain a target classification model.

Description

Technical Field

[0001] The present invention relates to the technical fields of millimeter-wave radar technology and non-contact heart disease detection technology, and more specifically, to a model training method and device for assisting in determining the category of heart diseases. Background Art

[0002] In recent years, millimeter-wave radar technology with high resolution and high precision can not only perform non-contact action recognition and monitor vital signs such as heart rate and respiration, but also achieve non-contact electrocardiogram (ECG) monitoring, which is of great significance for the early auxiliary diagnosis and monitoring of heart diseases.

[0003] In the process of implementing the inventive concept of the present invention, it is found through research that the lack of high-quality millimeter-wave radar data labels in the related art restricts the progress of millimeter-wave radar technology in the field of non-contact heart disease detection. Summary of the Invention

[0004] In view of this, the present invention provides a model training method and device for assisting in determining the category of heart diseases.

[0005] One aspect of the present invention provides a model training method for assisting in determining the category of heart diseases, including: obtaining an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, where the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; respectively performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period; according to the electrocardiogram features and millimeter-wave radar features at multiple moments, semantically matching the sample heartbeat information with the electrocardiogram data set to obtain aligned target electrocardiogram features and target millimeter-wave radar features; and adjusting an initial classification model trained using electrocardiogram features according to the target electrocardiogram features and target millimeter-wave radar features to obtain a target classification model, where the target classification model is used to determine the category of heart diseases.

[0006] According to an embodiment of the present invention, respectively performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period includes: using a deep learning model to perform feature extraction on the electrocardiogram data set to obtain first initial features; performing spatial discretization processing on the first initial features to obtain electrocardiogram features at multiple moments within the predetermined time period; using a deep learning model to perform feature extraction on the millimeter-wave radar data set to obtain second initial features; and performing spatial discretization processing on the second initial features to obtain millimeter-wave features at multiple moments within the predetermined time period.

[0007] According to an embodiment of the present invention, the second initial feature includes a spatial feature and a temporal feature. By using a deep learning model, feature extraction is performed on a millimeter-wave radar data set to obtain the second initial feature, including: using a three-dimensional convolutional network of the deep learning model to perform feature extraction on the millimeter-wave radar data set in the spatial domain to obtain spatial features for each of multiple moments, where the spatial features characterize the variation law of the millimeter-wave radar data set in three-dimensional space; using a one-dimensional convolutional network of the deep learning model to perform feature extraction on the millimeter-wave radar data set in the temporal domain to obtain temporal features corresponding to different time scales, where the temporal features characterize the variation law of the millimeter-wave radar data set at different time scales; and obtaining the second initial feature according to the spatial features and the temporal features.

[0008] According to an embodiment of the present invention, obtaining the second initial feature according to the spatial features and the temporal features includes: using an attention model corresponding to the one-dimensional convolutional network to perform scale fusion on the temporal features corresponding to different time scales to obtain temporal sequence data corresponding to the temporal features; and obtaining the second initial feature according to the spatial features and the temporal sequence data corresponding to the temporal features.

[0009] According to an embodiment of the present invention, the initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: constructing the classifier sub-model according to the correspondence between electrocardiogram features and heart disease categories; using the decoder sub-model to perform feature reconstruction on the electrocardiogram features to obtain reconstructed electrocardiogram data; and using the reconstructed electrocardiogram data to iteratively optimize the classifier sub-model until the loss function of the classifier sub-model meets a predetermined threshold to obtain the initial classification model.

[0010] According to an embodiment of the present invention, semantic matching is performed between the sample heartbeat information and the electrocardiogram data set according to the electrocardiogram features and millimeter-wave radar features for each of multiple moments to obtain aligned target electrocardiogram features and target millimeter-wave radar features, including: for each moment, aligning the gradient corresponding to the millimeter-wave radar feature with the gradient of the electrocardiogram feature so that the sample heartbeat information in the millimeter-wave radar data set is semantically matched with the electrocardiogram data set to obtain the aligned target electrocardiogram features and target millimeter-wave radar features.

[0011] According to an embodiment of the present invention, obtaining an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period includes: obtaining an electrocardiogram data set of the sample object within the predetermined time period; obtaining radar signals corresponding to each spatial position point in the target space of the sample object within the predetermined time period; performing second-order difference processing on the radar signals corresponding to each spatial position point to obtain second-order radar signals; respectively extracting real part data, imaginary part data, and phase data from each second-order radar signal; and obtaining a millimeter-wave radar data set according to the real part data, imaginary part data, and phase data of each second-order radar signal.

[0012] Another aspect of the present invention provides a method for assisting in determining a heart disease category, including: collecting millimeter-wave radar signals corresponding to a target object, where the millimeter-wave radar signals include heartbeat information; inputting the millimeter-wave radar signals into a target classification model, and outputting a heart disease category corresponding to the heartbeat information, where the target classification model is trained by using the above model training method.

[0013] Another aspect of the present invention provides a model training device for assisting in determining a heart disease category, including: an acquisition module for acquiring an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, where the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; an extraction module for respectively performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period; a obtaining module for semantically matching the sample heartbeat information with the electrocardiogram data set according to the electrocardiogram features and millimeter-wave radar features at multiple moments to obtain aligned target electrocardiogram features and target millimeter-wave radar features; and an adjustment module for adjusting an initial classification model trained by using electrocardiogram features according to the target electrocardiogram features and target millimeter-wave radar features to obtain a target classification model, where the target classification model is used to determine the heart disease category.

[0014] Another aspect of the present invention provides a device for assisting in determining a heart disease category, including: a collection module for collecting millimeter-wave radar signals corresponding to a target object, where the millimeter-wave radar signals include heartbeat information; and an output module for inputting the millimeter-wave radar signals into a target classification model and outputting a heart disease category corresponding to the heartbeat information, where the target classification model is trained by using the above model training method.

[0015] According to an embodiment of the present invention, by extracting features from the electrocardiogram (ECG) dataset and millimeter-wave radar dataset of the collected sample objects, an initial classification model is trained using the obtained ECG features. According to the ECG features and millimeter-wave radar features at multiple moments within a predetermined period, the sample heartbeat information is semantically matched with the ECG dataset, and relatively aligned target ECG features and target millimeter-wave radar features can be obtained, while redundant information is removed. Then, according to the target ECG features and target millimeter-wave radar features, the initial classification model is adjusted, so that the initial classification model trained with ECG features can identify millimeter-wave radar data and assist in determining the category of heart diseases. Since ECG features are used instead of millimeter-wave radar features when training the initial classification model, zero-shot learning for millimeter-wave radar heart disease classification is achieved, avoiding the dependence of heart disease classification on millimeter-wave radar data and disease labels, and using ECG data to make up for the deficiencies of millimeter-wave data, enabling the millimeter-wave radar to detect and classify more heart diseases, further improving the accuracy of disease classification, expanding the application scope of millimeter-wave radar technology in the field of heart disease monitoring, and providing new possibilities for contactless heart health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Through the following description of the embodiments of the present invention with reference to the accompanying drawings, the above and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0017] Figure 1 The flowchart of a model training method for assisting in determining the category of heart diseases according to an embodiment of the present invention is shown.

[0018] Figure 2 The schematic diagram of feature extraction for the millimeter-wave radar dataset according to an embodiment of the present invention is shown.

[0019] Figure 3 The schematic diagram of the training process of the target classification model according to an embodiment of the present invention is shown.

[0020] Figure 4 The flowchart of a method for assisting in determining the category of heart diseases according to an embodiment of the present invention is shown.

[0021] Figure 5 The block diagram of a model training device for assisting in determining the category of heart diseases according to an embodiment of the present invention is shown.

[0022] Figure 6 The block diagram of a device for assisting in determining the category of heart diseases according to an embodiment of the present invention is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In the following detailed description, for the sake of explanation, numerous specific details are set forth to provide a comprehensive understanding of the embodiments of the present invention. However, it is obvious that one or more embodiments can also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessarily confusing the concepts of the present invention.

[0024] The terms used herein are merely for describing specific embodiments and are not intended to limit the present invention. The terms "including", "comprising", etc. used herein indicate the presence of the described features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] In the case of using expressions such as "at least one of A, B, and C, etc.", generally, it should be interpreted according to the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having only A, only B, only C, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0027] In the embodiments of the present invention, in terms of the collection, update, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the involved data (for example, including but not limited to user personal information), they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and to maintain the security of user personal information and network security.

[0028] In the embodiments of the present invention, before obtaining or collecting user personal information, the authorization or consent of the user is obtained.

[0029] In recent years, millimeter-wave radar technology has been widely applied in commercial fields such as autonomous driving and human perception. Especially in the field of human perception, high-precision millimeter-wave radar technology has demonstrated its unique advantages.

[0030] In the field of contactless heart disease detection, the application of deep learning technology has promoted the development of millimeter-wave radar technology. However, in the training of supervised learning models, the model relies on a large-scale millimeter-wave radar dataset, that is, the dataset needs to meet both a sufficient number and high-quality labels, and often requires the in-depth participation of medical experts. Therefore, there are technical barriers such as high difficulty, large workload, and high labor cost in the collection of the dataset. Especially for rare heart diseases, the collection and annotation of relevant millimeter-wave radar data are even more difficult. At the same time, only a small amount of cardiac effective information exists in millimeter-wave radar data, while a large amount is filled with other redundant information such as environmental noise and respiratory information.

[0031] Based on this, due to the lack of high-quality labeled millimeter-wave radar data covering various heart disease types, the wide deployment of millimeter-wave radar technology in the practical application of contactless heart disease detection is restricted. In sharp contrast to the scarce millimeter-wave radar data used to train heart disease recognition models is the abundant and high-quality electrocardiogram (ECG) data, which cover a variety of rare heart diseases.

[0032] In view of this, an embodiment of the present invention provides a model training method for assisting in determining the category of heart diseases, including: obtaining an electrocardiogram dataset and a millimeter-wave radar dataset of a sample object within a predetermined time period, where the millimeter-wave radar dataset includes sample heartbeat information and sample redundant information; respectively extracting features from the electrocardiogram dataset and the millimeter-wave radar dataset to obtain electrocardiogram features and millimeter-wave radar features at each of multiple moments within the predetermined time period; according to the electrocardiogram features and millimeter-wave radar features at each of the multiple moments, semantically matching the sample heartbeat information with the electrocardiogram dataset to obtain aligned target electrocardiogram features and target millimeter-wave radar features; and adjusting an initial classification model trained using electrocardiogram features according to the target electrocardiogram features and target millimeter-wave radar features to obtain a target classification model, where the target classification model is used to determine the category of heart diseases.

[0033] Figure 1 The flowchart of the model training method for assisting in determining the category of heart diseases according to an embodiment of the present invention is shown.

[0034] As Figure 1 shown, the method 100 includes operations S110 to S140.

[0035] In operation S110, an electrocardiogram dataset and a millimeter-wave radar dataset of a sample object within a predetermined time period are obtained.

[0036] In operation S120 , feature extraction is performed on the electrocardiogram dataset and the millimeter-wave radar dataset to obtain electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined period of time.

[0037] In operation S130, semantic matching is performed between the sample heartbeat information and the electrocardiogram dataset according to respective electrocardiogram features and millimeter-wave radar features at multiple moments, so as to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0038] In operation S140, the initial classification model trained using the electrocardiogram features is adjusted according to the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model.

[0039] According to an embodiment of the present invention, the millimeter wave radar data set includes sample heartbeat information and sample redundant information, and the sample redundant information includes environmental noise, breathing information, motion information, and the like.

[0040] According to an embodiment of the present invention, wearable devices, medical monitoring systems, and other sensors may be used to collect electrocardiogram datasets and millimeter-wave radar datasets of sample subjects.

[0041] According to an embodiment of the present invention, although the data contained in the electrocardiogram dataset is intuitive, it has certain complexity and subtle changes. Therefore, a more advanced deep learning model can be used to fully explore the electrocardiogram features. For example, a 1DResNet (one-dimensional residual network) model is used to extract features from the electrocardiogram data. The 1D ResNet model is very effective in processing one-dimensional sequence electrocardiogram data.

[0042] According to an embodiment of the present invention, electrocardiogram features are used to effectively supplement millimeter-wave radar features. It is necessary to ensure that the sample heartbeat information in the millimeter-wave radar data set is semantically consistent with the electrocardiogram data set. Therefore, feature alignment can be used to eliminate redundant information in the millimeter-wave radar data set to avoid redundant information interfering with the electrocardiogram features, so that the initial classification model can identify the millimeter-wave radar features and then determine the category of heart disease.

[0043] For example, heart disease categories may include atrial fibrillation (AF), sinus bradycardia (SB), sinus tachycardia (STach), T wave abnormality (TAb), right bundle branch block (RBBB), left bundle branch block (LBBB), complete right bundle branch block (CRBBB), complete left bundle branch block (CLBBB), etc.

[0044] In order to better demonstrate the classification effect of the target classification model obtained by the model training method for assisting in determining the category of heart disease according to an embodiment of the present invention on heart disease, the target classification model will be verified as follows.

[0045] According to an embodiment of the present invention, an electrocardiogram dataset in the related art can be used as auxiliary data for millimeter-wave radar data and labels. All electrocardiogram data in the electrocardiogram dataset need to be first normalized from 0 to 1 and resampled to 100 Hz to match the sampling rate of the millimeter-wave radar dataset.

[0046] According to an embodiment of the present invention, the F1 score takes into account both the precision and recall of the model and is commonly used to comprehensively evaluate the performance of the model. The F1 scores for different heart disease classifications are shown in Table 1 below.

[0047] Table 1

[0048]

[0049] Among them, group P represents a classification model that has both millimeter-wave radar data and labels and uses electrocardiogram data as auxiliary data, and group Q represents a classification model that uses millimeter-wave radar data and labels for supervised learning.

[0050] According to an embodiment of the present invention, group P significantly improves the F1 score of millimeter-wave radar data in the classification of various heart diseases by integrating external electrocardiogram data, with the improvement range from 1.99% to 68.08%. And as the complexity of the disease increases and the sample size decreases, the enhancement effect of electrocardiogram data on the F1 score is particularly obvious. For example, in the classification of LBBB with a small sample size, the F1 score is significantly improved from 15.38% to 83.46%. By comparison, it can be seen that the classification model trained by using electrocardiogram data to assist millimeter-wave radar data and labels is superior to the supervised learning heart disease classification model that only relies on millimeter-wave radar data and labels in the related art.

[0051] According to an embodiment of the present invention, in the case of only having millimeter-wave radar data without labels, that is, in the case of transductive zero-shot learning, the same disease classification in Table 1 above and the same external electrocardiogram dataset in the related art can be used to evaluate the F1 score of the classification model trained by using unlabeled millimeter-wave radar data.

[0052] According to an embodiment of the present invention, group M is in the case of only having millimeter-wave radar data without labels. The comparison of the F1 scores of group M and group Q is shown in Table 2 below.

[0053] Table 2

[0054]

[0055] According to the experimental results shown in Table 2, for diseases with a relatively large sample size such as atrial fibrillation (AF), sinus bradycardia (SB), and sinus tachycardia (STach), the M group can achieve performance close to that of the supervised learning Q group. For diseases with a relatively small sample size, such as atrial tachycardia (TAb) and left bundle branch block (LBBB), the F1 score of the M group even exceeds that of the Q group. Therefore, without relying on experienced medical experts to accurately label the data, the above model can still achieve accurate classification of different heart diseases.

[0056] According to an embodiment of the present invention, for the target classification model obtained by using the model training method for assisting in determining the category of heart diseases according to the embodiment of the present invention, that is, the target classification model not trained with millimeter-wave radar data, in the case of lacking millimeter-wave radar data of SB or STach diseases, the F1 score for SB can still reach 0.8938, and the F1 score for STach can reach 0.5805.

[0057] It can be seen that the target classification model obtained by zero-shot learning of unlabeled millimeter-wave radar data uses electrocardiogram data as a medium, and can still accurately judge the category of heart diseases, and can also identify disease types that have never been exposed to millimeter-wave radar technology. Therefore, it effectively expands the development of millimeter-wave radar technology in the field of non-contact heart disease monitoring.

[0058] According to an embodiment of the present invention, by extracting features from the electrocardiogram data set and millimeter-wave radar data set of the collected sample objects, an initial classification model is trained using the obtained electrocardiogram features. According to the electrocardiogram features and millimeter-wave radar features at multiple moments within a predetermined time period, the sample heartbeat information is semantically matched with the electrocardiogram data set, and relatively aligned target electrocardiogram features and target millimeter-wave radar features can be obtained, while redundant information is removed. And according to the target electrocardiogram features and target millimeter-wave radar features, the initial classification model is adjusted, so that the initial classification model trained with electrocardiogram features can identify millimeter-wave radar data and assist in determining the category of heart diseases. Since electrocardiogram features are used instead of millimeter-wave radar features when training the initial classification model, zero-shot learning of millimeter-wave radar heart disease classification is realized, breaking the dependence of heart disease classification on millimeter-wave radar data and disease labels, and using electrocardiogram data to make up for the deficiency of millimeter-wave data, so that millimeter-wave radar can detect and classify more heart diseases, further improving the accuracy of disease classification, broadening the application scope of millimeter-wave radar technology in the field of heart disease monitoring, and providing new possibilities for non-contact heart health monitoring.

[0059] According to an embodiment of the present invention, obtaining an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period includes: obtaining an electrocardiogram data set of the sample object within the predetermined time period; obtaining radar signals corresponding to each spatial position point in the target space of the sample object within the predetermined time period; performing second-order difference processing on the radar signals corresponding to each spatial position point to obtain second-order radar signals; respectively extracting real part data, imaginary part data, and phase data from each second-order radar signal; and obtaining a millimeter-wave radar data set according to the real part data, imaginary part data, and phase data of each second-order radar signal.

[0060] According to an embodiment of the present invention, a two-dimensional antenna array and Frequency Modulated Continuous Wave (FMCW) can be used to obtain radar signals corresponding to each spatial position point in the target space, and the 3D beamforming process of the radar signals can be expressed by the following formula (1).

[0061] (1)

[0062] Where n is the number of virtual channels formed by the transmit-receive antenna pairs, n ∈ N, t is the time within the predetermined time period, t ∈ T, k is the frequency modulation slope of the frequency modulated continuous wave, λ is the wavelength, r(x, y, z, n) is the round-trip distance from the spatial position point (x, y, z) to the transmit-receive antenna pair, S(x, y, z, t) is the radar signal corresponding to the spatial position point (x, y, z), j is the imaginary unit, and c is the speed of light.

[0063] According to an embodiment of the present invention, in order to focus on the radar signal changes caused by heart movement, it is necessary to shield the movements other than the heartbeat. Therefore, the second-order difference processing can be performed on the radar signals corresponding to each spatial position point by using the frequency difference between the heartbeat and other movements, and the process of the second-order difference processing can be expressed by the following formula (2).

[0064] (2)

[0065] Where S t ’’ represents the second derivative of the radar signal at the t-th moment, S t-3 represents the radar signal at the (t - 3)-th moment, S t-2 represents the radar signal at the (t - 2)-th moment, S t-1 represents the radar signal at the (t - 1)-th moment, S t represents the radar signal at the t-th moment, S t+3 represents the radar signal at the (t + 3)-th moment, S t+2 represents the radar signal at the (t + 2)-th moment, S t+1 represents the radar signal at the (t + 1)-th moment, and h represents the frame period of the predetermined time period.

[0066] According to an embodiment of the present invention, since the radar signal is in complex form and it is difficult to process complex numbers in subsequent operations such as feature extraction of radar data, real part data, imaginary part data, and phase data are extracted from each second-order radar signal. Among them, the phase data can provide time delay or offset information of the radar signal, and a millimeter-wave radar data set including cardiac motion information is obtained based on the real part data, imaginary part data, and phase data of each second-order radar signal respectively.

[0067] According to an embodiment of the present invention, the initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: a classifier sub-model is constructed according to the correspondence between electrocardiogram features and cardiac disease categories; the decoder sub-model is used to reconstruct the electrocardiogram features to obtain reconstructed electrocardiogram data; the reconstructed electrocardiogram data is used to iteratively optimize the classifier sub-model until the loss function of the classifier sub-model meets a predetermined threshold, and the initial classification model is obtained.

[0068] According to an embodiment of the present invention, the classifier sub-model includes a 1D convolutional layer and a fully connected layer. The 1D convolutional layer can be used to extract time series features in the electrocardiogram features, and the fully connected layer integrates the features extracted by the 1D convolutional layer and maps the correspondence between the electrocardiogram features and cardiac disease categories.

[0069] According to an embodiment of the present invention, the decoder sub-model may include multiple transposed convolutional layers, which can reconstruct the electrocardiogram data according to the extracted electrocardiogram features to ensure that the obtained initial classification model can accurately capture and restore the key information of the electrocardiogram data.

[0070] According to an embodiment of the present invention, the loss function may include a classification loss function and a reconstruction loss function. The decoder sub-model is used to optimize and iterate the classifier sub-model to verify whether the loss function of the classifier sub-model meets a predetermined threshold, and then an initial classification model with high separability and strong generalization ability is obtained.

[0071] According to an embodiment of the present invention, using a deep learning model, the extracted electrocardiogram features are reconstructed through the transposed convolutional layers of the decoder sub-model, and the classifier sub-model is further verified to obtain an initial classification model with better performance, providing a certain basis for subsequent determination of cardiac disease categories.

[0072] According to an embodiment of the present invention, feature extraction is respectively performed on the electrocardiogram dataset and the millimeter-wave radar dataset to obtain the electrocardiogram features and millimeter-wave radar features at multiple moments within a predetermined period, including: using a deep learning model to perform feature extraction on the electrocardiogram dataset to obtain a first initial feature; performing spatial discretization processing on the first initial feature to obtain the electrocardiogram features at multiple moments within a predetermined period; using a deep learning model to perform feature extraction on the millimeter-wave radar dataset to obtain a second initial feature; performing spatial discretization processing on the second initial feature to obtain the millimeter-wave features at multiple moments within a predetermined period.

[0073] According to an embodiment of the present invention, the process of using the finite scalar quantization technique (FSQ) to perform spatial discretization processing on the first initial feature to obtain the electrocardiogram features at multiple moments within a predetermined period can be expressed as the following formula (3).

[0074] (3)

[0075] where F e represents the electrocardiogram feature, f e represents the first initial feature, sg() represents the stop gradient, and round() represents rounding.

[0076] According to an embodiment of the present invention, the second initial feature includes spatial features and temporal features. Using a deep learning model to perform feature extraction on the millimeter-wave radar dataset to obtain the second initial feature includes: using the three-dimensional convolutional network of the deep learning model to perform feature extraction in the spatial domain on the millimeter-wave radar dataset to obtain the spatial features at multiple moments, where the spatial features characterize the variation law of the millimeter-wave radar dataset in the three-dimensional space; using the one-dimensional convolutional network of the deep learning model to perform feature extraction in the temporal domain on the millimeter-wave radar dataset to obtain the temporal features corresponding to different time scales, where the temporal features characterize the variation law of the millimeter-wave radar dataset at different time scales; obtaining the second initial feature according to the spatial features and the temporal features.

[0077] According to an embodiment of the present invention, the three-dimensional convolutional network of the deep learning model is used to extract the millimeter-wave radar spatial features at each moment, so as to effectively capture the variation law of the radar signal in the three-dimensional space. After obtaining the spatial features, the one-dimensional convolutional network of the deep learning model is used to analyze the radar signal from different time scales, so as to more comprehensively understand the dynamic characteristics of cardiac motion.

[0078] According to an embodiment of the present invention, the continuous second initial features can be compressed into a discrete feature space by using the finite scalar quantization (FSQ) technique, so as to effectively extract and utilize the cardiac information in the millimeter-wave radar dataset, providing important support for subsequent feature alignment and disease classification.

[0079] According to an embodiment of the present invention, the electrocardiogram dataset and the millimeter-wave radar dataset are subjected to feature extraction through a deep learning model and spatial discretization processing, obtaining electrocardiogram features and millimeter-wave radar features including cardiac information, thereby significantly improving the accuracy and reliability of cardiac disease classification and providing support for cardiac health monitoring and disease diagnosis.

[0080] According to an embodiment of the present invention, the second initial features are obtained according to spatial features and temporal features, including: using an attention model corresponding to a one-dimensional convolutional network to perform scale fusion on temporal features corresponding to different time scales to obtain temporal data corresponding to the temporal features; obtaining the second initial features according to the spatial features and the temporal data corresponding to the temporal features.

[0081] According to an embodiment of the present invention, in order to convert disordered temporal features into high-level temporal semantic features, the important cardiac-related information can be extracted from the temporal features corresponding to different time scales through the Transformer model based on the self-attention mechanism in the deep learning model, and then the temporal features of different scales are fused to obtain temporal data.

[0082] Figure 2 The schematic diagram of feature extraction for the millimeter-wave radar dataset according to an embodiment of the present invention is shown.

[0083] As Figure 2 shown, in this embodiment 200, the millimeter-wave radar dataset 210 is input into the three-dimensional convolutional network 220 to obtain spatial features 230. The millimeter-wave radar dataset 210 is continuously input into the one-dimensional convolutional network 240 to obtain temporal features 250. The temporal features 250 are input into the Transformer model 260 to capture the semantic information of the temporal features 250 to obtain temporal semantic features 270. The temporal semantic features are fused according to the scale fusion module 280, and the fused temporal features and spatial features are integrated to obtain the second initial features 290. Among them, the temporal features 250 can be dual-scale temporal features. For example, the total duration can be divided into 8 time periods or 4 time periods.

[0084] According to an embodiment of the present invention, based on the electrocardiogram (ECG) features and millimeter-wave radar features at respective multiple moments, semantic matching is performed between the sample heartbeat information and the ECG dataset to obtain aligned target ECG features and target millimeter-wave radar features, including: for each moment, aligning the gradient corresponding to the millimeter-wave radar features with the gradient of the ECG features, so that the sample heartbeat information in the millimeter-wave radar dataset is semantically matched with the ECG dataset, and the aligned target ECG features and target millimeter-wave radar features are obtained.

[0085] According to an embodiment of the present invention, for each moment, the gradient corresponding to the millimeter-wave radar features is unidirectionally aligned with the gradient of the ECG features, so that the sample heartbeat information in the millimeter-wave radar dataset is semantically matched with the ECG dataset, and at the same time, sample redundant information can be eliminated. The process of feature alignment can be expressed by the following formula (4).

[0086] (4)

[0087] where loss align represents the alignment loss function, MSE() represents the mean square error, and F r represents the millimeter-wave radar features.

[0088] According to an embodiment of the present invention, the ECG features and the millimeter-wave radar features are unidirectionally aligned by using the stop-gradient technique, so that the sample heartbeat information in the millimeter-wave radar dataset is semantically matched with the ECG dataset, and the redundant information irrelevant to the sample heartbeat information is eliminated, avoiding unnecessary interference with the heartbeat information.

[0089] Figure 3 Fig. shows a schematic diagram of the training process of the target classification model according to an embodiment of the present invention.

[0090] As Figure 3 shown, the ECG dataset of the collected sample object is input into the ECG encoder for feature extraction to obtain the first initial features. The first initial features are subjected to spatial discretization processing by a quantizer to obtain the ECG features. The initial classification model includes a decoder sub-model for reconstructing the ECG data and a classifier sub-model for classification. The external ECG dataset is used to verify the accuracy of the model. The millimeter-wave radar dataset in the same time period as the ECG dataset is first subjected to signal preprocessing and then enters the radar encoder for feature extraction to obtain the second initial features. The second initial features are subjected to spatial discretization processing by a quantizer to obtain the millimeter-wave radar features. The gradient corresponding to the millimeter-wave radar features is aligned with the gradient of the ECG features, and the target classification model is obtained after completing the model training.

[0091] Figure 4The flowchart of a method for assisting in determining a heart disease category according to an embodiment of the present invention is shown.

[0092] As Figure 4 shown, the method 400 includes operations S410 to S420.

[0093] In operation S410, millimeter-wave radar signals corresponding to a target object are collected.

[0094] In operation S420, the millimeter-wave radar signals are input into a target classification model, and a heart disease category corresponding to heartbeat information is output.

[0095] According to an embodiment of the present invention, the target classification model obtained through the above model training method can identify millimeter-wave radar signals and output corresponding heart disease categories, so that it can be applied to the application scenario of contactless heart monitoring.

[0096] Figure 5 The block diagram of a model training device for assisting in determining a heart disease category according to an embodiment of the present invention is shown.

[0097] As Figure 5 shown, the device 500 includes an acquisition module 510, an extraction module 520, a obtaining module 530, and an adjustment module 540.

[0098] The acquisition module 510 is configured to acquire an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information.

[0099] The extraction module 520 is configured to respectively extract features from the electrocardiogram data set and the millimeter-wave radar data set to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within a predetermined time period.

[0100] The obtaining module 530 is configured to semantically match the heartbeat information with the electrocardiogram data set according to the electrocardiogram features and millimeter-wave radar features at multiple moments respectively, to obtain aligned target electrocardiogram features and target millimeter-wave radar features.

[0101] The adjustment module 540 is configured to adjust an initial classification model trained using electrocardiogram features according to the target electrocardiogram features and target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine a heart disease category.

[0102] According to an embodiment of the present invention, the acquisition module 510 includes a first acquisition sub-module, a second acquisition sub-module, a first obtaining sub-module, an extraction sub-module, and a second obtaining sub-module.

[0103] The first acquisition sub-module is used to acquire the electrocardiogram data set of the sample object within a predetermined time period.

[0104] The second acquisition sub-module is used to acquire the radar signals corresponding to each spatial position point in the target space of the sample object within a predetermined time period.

[0105] The first obtaining sub-module is used to perform second-order difference processing on the radar signals corresponding to each spatial position point to obtain second-order radar signals.

[0106] The extraction sub-module is used to extract real part data, imaginary part data, and phase data from each second-order radar signal respectively.

[0107] The second obtaining sub-module is used to obtain a millimeter-wave radar data set according to the real part data, imaginary part data, and phase data of each second-order radar signal.

[0108] According to an embodiment of the present invention, the extraction module 520 includes a first obtaining sub-module, a second obtaining sub-module, a third obtaining sub-module, and a fourth obtaining sub-module.

[0109] The first obtaining sub-module is used to extract features from the electrocardiogram data set by using a deep learning model to obtain first initial features.

[0110] The second obtaining sub-module is used to perform spatial discretization processing on the first initial features to obtain electrocardiogram features at each of multiple moments within a predetermined time period.

[0111] The third obtaining sub-module is used to extract features from the millimeter-wave radar data set by using a deep learning model to obtain second initial features.

[0112] The fourth obtaining sub-module is used to perform spatial discretization processing on the second initial features to obtain millimeter-wave features at each of multiple moments within a predetermined time period.

[0113] According to an embodiment of the present invention, the third obtaining sub-module includes a first obtaining unit, a second obtaining unit, and a third obtaining unit.

[0114] The first obtaining unit is used to extract spatial domain features from the millimeter-wave radar data set by using the three-dimensional convolutional network of the deep learning model to obtain spatial features at each of multiple moments, and the spatial features characterize the variation law of the millimeter-wave radar data set in the three-dimensional space.

[0115] The second obtaining unit is used to extract time domain features from the millimeter-wave radar data set by using the one-dimensional convolutional network of the deep learning model to obtain time features corresponding to different time scales, and the time features characterize the variation law of the millimeter-wave radar data set at different time scales.

[0116] A third obtaining unit, configured to obtain a second initial feature according to a spatial feature and a temporal feature.

[0117] According to an embodiment of the present invention, the third obtaining unit includes a fusion subunit and an obtaining subunit.

[0118] The fusion subunit is configured to perform scale fusion on temporal features corresponding to different time scales by using an attention model corresponding to a one-dimensional convolutional network, so as to obtain temporal data corresponding to the temporal features.

[0119] The obtaining subunit is configured to obtain a second initial feature according to the spatial feature and the temporal data corresponding to the temporal feature.

[0120] According to an embodiment of the present invention, the obtaining module 530 includes an alignment sub-module.

[0121] The alignment sub-module is configured to, for each moment, align the gradient corresponding to the millimeter-wave radar feature with the gradient of the electrocardiogram feature, so that the heartbeat information in the millimeter-wave radar dataset is semantically matched with the electrocardiogram dataset, and obtain the aligned target electrocardiogram feature and target millimeter-wave radar feature.

[0122] Figure 6 The block diagram of a device for assisting in determining a heart disease category according to an embodiment of the present invention is shown.

[0123] As Figure 6 shown, the device 600 includes an acquisition module 610 and an output module 620.

[0124] The acquisition module 610 is configured to acquire a millimeter-wave radar signal corresponding to a target object. Wherein, the millimeter-wave radar signal includes heartbeat information.

[0125] The output module 620 is configured to input the millimeter-wave radar signal into a target classification model and output a heart disease category corresponding to the heartbeat information.

[0126] Any number of modules, sub-modules, units, and sub-units according to embodiments of the present invention, or at least some functions of any number of them, can be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system on a chip, system on a substrate, system in a package, application-specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, or in any one of the three implementation modes of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, one or more of the modules, sub-modules, units, and sub-units according to embodiments of the present invention can be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0127] For example, any number of the acquisition module 510, extraction module 520, obtaining module 530, and adjustment module 540 can be combined and implemented in one module / unit / sub-unit, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least some functions of one or more of these modules / units / sub-units can be combined with at least some functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to embodiments of the present invention, at least one of the acquisition module 510, extraction module 520, obtaining module 530, and adjustment module 540 can be at least partially implemented as a hardware circuit, such as a field-programmable gate array (FPGA), programmable logic array (PLA), system on a chip, system on a substrate, system in a package, application-specific integrated circuit (ASIC), or can be implemented by any other reasonable way of integrating or packaging circuits, or in any one of the three implementation modes of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 510, extraction module 520, obtaining module 530, and adjustment module 540 can be at least partially implemented as a computer program module, and when the computer program module is run, it can execute the corresponding functions.

[0128] For example, any number of the acquisition module 610 and the output module 620 may be combined and implemented in one module / unit / sub-unit, or any one of the modules / units / sub-units may be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units may be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 610 and the output module 620 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging circuits, etc., implemented by hardware or firmware, or implemented in any one of the three implementation manners of software, hardware, and firmware, or in an appropriate combination of any several of them. Alternatively, at least one of the acquisition module 610 and the output module 620 may be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding functions may be executed.

[0129] It should be noted that the part of the model training device for assisting in determining the category of heart diseases in the embodiments of the present invention corresponds to the part of the model training method for assisting in determining the category of heart diseases in the embodiments of the present invention. For the description of the model training device part, please refer to the model training method part specifically, and details will not be repeated here.

[0130] It should be noted that the part of the device for assisting in determining the category of heart diseases in the embodiments of the present invention corresponds to the part of the method for assisting in determining the category of heart diseases in the embodiments of the present invention. For the description of the device part, please refer to the method part specifically, and details will not be repeated here.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions. Those skilled in the art can understand that the features described in various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0132] The embodiments of the present invention have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although the embodiments have been described separately above, this does not mean that the measures in each embodiment cannot be used advantageously in combination. Without departing from the scope of the present invention, those skilled in the art can make various substitutions and modifications, and all such substitutions and modifications should fall within the scope of the present invention.

Claims

1. A model training method for assisting in determining the category of heart disease, characterized in that: include: Acquire an electrocardiogram data set and a millimeter wave radar data set of a sample object within a predetermined period of time, wherein the millimeter wave radar data set includes sample heartbeat information and sample redundant information; Performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively to obtain electrocardiogram features and millimeter-wave radar features at respective times within the predetermined time period; According to the respective electrocardiogram features and millimeter-wave radar features at the multiple moments, semantically matching the sample heartbeat information with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features, including: for each moment, unidirectionally aligning the gradient corresponding to the millimeter-wave radar feature with the gradient of the electrocardiogram feature so that the sample heartbeat information in the millimeter-wave radar data set and the electrocardiogram data set are semantically matched, and redundant information in the millimeter-wave radar data set is eliminated to avoid the redundant information from interfering with the electrocardiogram features, so as to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; According to the target electrocardiogram features and the target millimeter-wave radar features, the initial classification model trained using the electrocardiogram features is adjusted to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease, and the category of heart disease includes atrial fibrillation (AF), sinus bradycardia (SB), sinus tachycardia (STach), T wave abnormality (TAb), right bundle branch block (RBBB), left bundle branch block (LBBB), complete right bundle branch block (CRBBB), complete left bundle branch block (CLBBB), etc.

2. The method according to claim 1, characterized in that: The extracting features from the electrocardiogram dataset and the millimeter-wave radar dataset respectively to obtain the electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period includes: Using a deep learning model, extracting features from the electrocardiogram dataset to obtain a first initial feature; Performing spatial discretization processing on the first initial feature to obtain electrocardiogram features of each of multiple moments within the predetermined time period; Using the deep learning model, extracting features from the millimeter wave radar data set to obtain a second initial feature; The second initial feature is spatially discretized to obtain millimeter wave features at multiple moments in the predetermined time period.

3. The method according to claim 2, characterized in that The second initial feature includes a spatial feature and a temporal feature, and the use of the deep learning model to extract features from the millimeter wave radar data set to obtain the second initial feature includes: Using the three-dimensional convolutional network of the deep learning model, extracting features of the millimeter-wave radar data set in the spatial domain to obtain spatial features of each of the multiple moments, wherein the spatial features represent a change pattern of the millimeter-wave radar data set in the three-dimensional space; Using the one-dimensional convolutional network of the deep learning model, extracting features in the time domain of the millimeter-wave radar data set to obtain time features corresponding to different time scales, wherein the time features represent the change rules of the millimeter-wave radar data set at different time scales; The second initial feature is obtained according to the spatial feature and the temporal feature.

4. The method according to claim 3, characterized in that The obtaining the second initial feature according to the spatial feature and the temporal feature includes: Using the attention model corresponding to the one-dimensional convolutional network, scale fusion is performed on the time features corresponding to different time scales to obtain time series data corresponding to the time features; The second initial feature is obtained according to the spatial feature and the time series data corresponding to the time feature.

5. The method according to claim 1, characterized in that: The initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: constructing the classifier sub-model according to the correspondence between the electrocardiogram features and the categories of heart diseases; Using the decoder sub-model, reconstructing the electrocardiogram features to obtain reconstructed electrocardiogram data; The classifier sub-model is iteratively optimized using the reconstructed electrocardiogram data until the loss function of the classifier sub-model meets a predetermined threshold, thereby obtaining the initial classification model.

6. The method according to claim 1, characterized in that The step of acquiring an electrocardiogram dataset and a millimeter wave radar dataset of a sample object within a predetermined period of time includes: Acquire the electrocardiogram data set of the sample object within the predetermined time period; Acquire radar signals corresponding to the sample object and each spatial position point in the target space within the predetermined time period; Performing second-order difference processing on the radar signal corresponding to each of the spatial position points to obtain a second-order radar signal; Extracting real data, imaginary data and phase data from each of the second-order radar signals respectively; The millimeter-wave radar data set is obtained according to the real data, the imaginary data and the phase data of each of the second-order radar signals.

7. A method for assisting in determining the type of heart disease, characterized in that: include: Collecting a millimeter-wave radar signal corresponding to the target object, wherein the millimeter-wave radar signal includes heartbeat information; The millimeter-wave radar signal is input into a target classification model, and a heart disease category corresponding to the heartbeat information is output, wherein the target classification model is trained using the method of any one of claims 1 to 6.

8. A model training device for assisting in determining the type of heart disease, characterized in that: include: An acquisition module, used to acquire an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined period of time, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; An extraction module, used to perform feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively, to obtain electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period; A obtaining module, used for semantically matching the sample heartbeat information with the electrocardiogram data set according to the respective electrocardiogram features and millimeter-wave radar features at the multiple moments, to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features, wherein the obtaining module includes: an alignment submodule, used for unidirectionally aligning the gradient corresponding to the millimeter-wave radar feature with the gradient of the electrocardiogram feature at each moment, so that the sample heartbeat information in the millimeter-wave radar data set and the electrocardiogram data set are semantically matched, and redundant information in the millimeter-wave radar data set is eliminated to avoid the redundant information from interfering with the electrocardiogram features, so as to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; An adjustment module is used to adjust the initial classification model trained using the electrocardiogram features according to the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease, and the category of heart disease includes atrial fibrillation (AF), sinus bradycardia (SB), sinus tachycardia (STach), T wave abnormality (TAb), right bundle branch block (RBBB), left bundle branch block (LBBB), complete right bundle branch block (CRBBB), complete left bundle branch block (CLBBB), etc.

9. A device for assisting in determining the type of heart disease, characterized in that: include: A collection module, used to collect a millimeter-wave radar signal corresponding to a target object, wherein the millimeter-wave radar signal includes heartbeat information; An output module is used to input the millimeter wave radar signal into a target classification model and output a heart disease category corresponding to the heartbeat information, wherein the target classification model is trained using the method of any one of claims 1 to 6.

Images