Model training method and device for assisting in determining heart disease category

Abstract

The invention provides a model training method and device for assisting in determining heart disease categories, which can be applied to the technical field of millimeter wave radar and the technical field of non-contact heart disease detection. The method comprises the following steps: acquiring an electrocardiogram data set and a millimeter-wave radar data set of a sample object in a preset time period, wherein the millimeter-wave radar data set comprises sample heartbeat information and sample redundant information; respectively carrying out feature extraction on the electrocardiogram data set and the millimeter wave radar data set to obtain respective electrocardiogram features and millimeter wave radar features at a plurality of moments in a predetermined time period; performing semantic matching on the sample heartbeat information and the electrocardiogram data set according to the electrocardiogram features and the millimeter wave radar features of the multiple moments to obtain aligned target electrocardiogram features and target millimeter wave radar features; and according to the target electrocardiogram features and the target millimeter wave radar features, adjusting an initial classification model obtained by training the electrocardiogram features to obtain a target classification model.

Description

Technical Field

[0001] The present invention relates to the technical field of millimeter wave radar and the technical field of contactless heart disease detection, and more specifically, to a model training method and device for assisting in determining the category of heart disease. Background Art

[0002] In recent years, millimeter-wave radar technology with high resolution and high precision can not only perform contactless motion recognition and monitor vital signs such as heart rate and breathing, but also realize contactless electrocardiogram (ECG) monitoring, which is of great significance for early auxiliary diagnosis and monitoring of heart disease.

[0003] In the process of realizing the concept of the present invention, it was found through research that the progress of millimeter-wave radar technology in the field of contactless heart disease detection is restricted due to the lack of high-quality millimeter-wave radar data labels in the related technology. Summary of the invention

[0004] In view of this, the present invention provides a model training method and device for assisting in determining the category of heart disease.

[0005] One aspect of the present invention provides a model training method for assisting in determining the category of heart disease, comprising: obtaining an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively to obtain the electrocardiogram features and millimeter-wave radar features of each of the multiple moments within the predetermined time period; semantically matching the sample heartbeat information with the electrocardiogram data set based on the electrocardiogram features and millimeter-wave radar features of each of the multiple moments to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; adjusting the initial classification model trained using the electrocardiogram features based on the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

[0006] According to an embodiment of the present invention, feature extraction is performed on an electrocardiogram dataset and a millimeter-wave radar dataset respectively to obtain electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined time period, including: using a deep learning model to extract features from the electrocardiogram dataset to obtain a first initial feature; performing spatial discretization processing on the first initial feature to obtain the electrocardiogram features at multiple moments in the predetermined time period; using a deep learning model to extract features from the millimeter-wave radar dataset to obtain a second initial feature; performing spatial discretization processing on the second initial feature to obtain the millimeter-wave features at multiple moments in the predetermined time period.

[0007] According to an embodiment of the present invention, the second initial feature includes a spatial feature and a temporal feature, and a deep learning model is used to perform feature extraction on a millimeter-wave radar data set to obtain the second initial feature, including: using a three-dimensional convolutional network of a deep learning model to perform feature extraction on the millimeter-wave radar data set in the spatial domain to obtain spatial features of each of multiple moments, wherein the spatial features characterize the change pattern of the millimeter-wave radar data set in three-dimensional space; using a one-dimensional convolutional network of a deep learning model to perform feature extraction on the millimeter-wave radar data set in the temporal domain to obtain time features corresponding to different time scales, wherein the time features characterize the change pattern of the millimeter-wave radar data set at different time scales; and obtaining the second initial feature based on the spatial feature and the temporal feature.

[0008] According to an embodiment of the present invention, a second initial feature is obtained based on spatial features and temporal features, including: using an attention model corresponding to a one-dimensional convolutional network to scale-fuse temporal features corresponding to different time scales to obtain time series data corresponding to the time features; and obtaining a second initial feature based on the spatial features and the time series data corresponding to the time features.

[0009] According to an embodiment of the present invention, the initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: constructing a classifier sub-model according to the correspondence between electrocardiogram features and heart disease categories; using the decoder sub-model, reconstructing the electrocardiogram features to obtain reconstructed electrocardiogram data; using the reconstructed electrocardiogram data, iteratively optimizing the classifier sub-model until the loss function of the classifier sub-model meets a predetermined threshold, thereby obtaining the initial classification model.

[0010] According to an embodiment of the present invention, based on the respective electrocardiogram features and millimeter-wave radar features at multiple moments, the sample heartbeat information is semantically matched with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features, including: for each moment, the gradient corresponding to the millimeter-wave radar feature is aligned with the gradient of the electrocardiogram feature, so that the sample heartbeat information in the millimeter-wave radar data set is semantically matched with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0011] According to an embodiment of the present invention, an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period are obtained, including: obtaining an electrocardiogram data set of the sample object within the predetermined time period; obtaining a radar signal corresponding to each spatial position point of the sample object in the predetermined time period in the target space; performing second-order difference processing on the radar signal corresponding to each spatial position point to obtain a second-order radar signal; respectively extracting real data, imaginary data and phase data from each second-order radar signal; and obtaining a millimeter-wave radar data set according to the real data, imaginary data and phase data of each second-order radar signal.

[0012] Another aspect of the present invention provides a method for assisting in determining the category of heart disease, comprising: collecting a millimeter-wave radar signal corresponding to a target object, wherein the millimeter-wave radar signal includes heartbeat information; inputting the millimeter-wave radar signal into a target classification model, and outputting the category of heart disease corresponding to the heartbeat information, wherein the target classification model is trained using the above-mentioned model training method.

[0013] Another aspect of the present invention provides a model training device for assisting in determining the category of heart disease, including: an acquisition module, used to acquire an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; an extraction module, used to perform feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively, to obtain the electrocardiogram features and millimeter-wave radar features of each of the multiple moments within the predetermined time period; an acquisition module, used to semantically match the sample heartbeat information with the electrocardiogram data set according to the electrocardiogram features and millimeter-wave radar features of each of the multiple moments, to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; an adjustment module, used to adjust the initial classification model trained using the electrocardiogram features according to the target electrocardiogram features and the target millimeter-wave radar features, to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

[0014] Another aspect of the present invention provides a device for assisting in determining the category of heart disease, comprising: an acquisition module for acquiring millimeter-wave radar signals corresponding to a target object, wherein the millimeter-wave radar signals include heartbeat information; an output module for inputting the millimeter-wave radar signals into a target classification model, and outputting the category of heart disease corresponding to the heartbeat information, wherein the target classification model is trained using the above-mentioned model training method.

[0015] According to an embodiment of the present invention, feature extraction is performed on the collected electrocardiogram data set and millimeter-wave radar data set of the sample object, and an initial classification model is obtained by training the obtained electrocardiogram features. According to the respective electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined period, the sample heartbeat information is semantically matched with the electrocardiogram data set, so that relatively aligned target electrocardiogram features and target millimeter-wave radar features can be obtained, and redundant information is eliminated at the same time. According to the target electrocardiogram features and the target millimeter-wave radar features, the initial classification model is adjusted so that the initial classification model trained with the electrocardiogram features can recognize the millimeter-wave radar data and assist in determining the category of heart disease. Since the electrocardiogram features are used instead of the millimeter-wave radar features when training the initial classification model, zero-sample learning of millimeter-wave radar heart disease classification is achieved, and the dependence of heart disease classification on millimeter-wave radar data and disease labels is avoided. The electrocardiogram data is used to make up for the deficiency of millimeter-wave data, so that the millimeter-wave radar can detect and classify more heart diseases, further improve the accuracy of disease classification, broaden the application scope of millimeter-wave radar technology in the field of heart disease monitoring, and provide new possibilities for contactless heart health monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0017] Figure 1 A flow chart of a model training method for assisting in determining a category of heart disease according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram of extracting features from a millimeter-wave radar data set according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of a target classification model training process according to an embodiment of the present invention is shown.

[0020] Figure 4 A flow chart of a method for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown.

[0021] Figure 5 A block diagram of a model training device for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown.

[0022] Figure 6 A block diagram of an apparatus for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage and other aspects of the data involved (for example, including but not limited to user personal information) are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0028] In the embodiment of the present invention, before obtaining or collecting the user's personal information, the user's authorization or consent is obtained.

[0029] In recent years, millimeter-wave radar technology has been widely used in commercial fields such as autonomous driving and human perception. Especially in the field of human perception, high-precision millimeter-wave radar technology has demonstrated its unique advantages.

[0030] In the field of contactless heart disease detection, the application of deep learning technology has promoted the development of millimeter-wave radar technology. However, in the training of supervised learning models, the model relies on large-scale millimeter-wave radar data sets, that is, the data sets must meet both sufficient quantity and high-quality labels, and often require the in-depth participation of medical experts. Therefore, there are technical barriers to the collection of data sets, such as high difficulty, large workload, and high labor costs. Especially for rare heart diseases, the collection and labeling of related millimeter-wave radar data is more difficult. At the same time, there is only a small amount of effective heart information in the millimeter-wave radar data, and a large amount of other redundant information such as environmental noise and respiratory information is filled.

[0031] Based on this, the lack of high-quality labeled mmWave radar data covering multiple types of heart diseases has limited the widespread deployment of mmWave radar technology in practical applications in the field of contactless heart disease detection. In stark contrast to the scarce mmWave radar data for training heart disease recognition models is the abundant and high-quality electrocardiogram (ECG) data that covers a variety of rare heart diseases.

[0032] In view of this, an embodiment of the present invention provides a model training method for assisting in determining the category of heart disease, including: obtaining an electrocardiogram dataset and a millimeter-wave radar dataset of a sample object within a predetermined time period, wherein the millimeter-wave radar dataset includes sample heartbeat information and sample redundant information; performing feature extraction on the electrocardiogram dataset and the millimeter-wave radar dataset respectively to obtain the electrocardiogram features and millimeter-wave radar features of each of the multiple moments within the predetermined time period; semantically matching the sample heartbeat information with the electrocardiogram dataset based on the electrocardiogram features and millimeter-wave radar features of each of the multiple moments to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; adjusting the initial classification model trained using the electrocardiogram features based on the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

[0033] Figure 1 A flow chart of a model training method for assisting in determining a category of heart disease according to an embodiment of the present invention is shown.

[0034] like Figure 1 As shown, the method 100 includes operations S110 to S140.

[0035] In operation S110 , an electrocardiogram dataset and a millimeter wave radar dataset of a sample object within a predetermined period of time are acquired.

[0036] In operation S120 , feature extraction is performed on the electrocardiogram dataset and the millimeter-wave radar dataset to obtain electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined period of time.

[0037] In operation S130, semantic matching is performed between the sample heartbeat information and the electrocardiogram dataset according to respective electrocardiogram features and millimeter-wave radar features at multiple moments, so as to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0038] In operation S140, the initial classification model trained using the electrocardiogram features is adjusted according to the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model.

[0039] According to an embodiment of the present invention, the millimeter wave radar data set includes sample heartbeat information and sample redundant information, and the sample redundant information includes environmental noise, breathing information, motion information, and the like.

[0040] According to an embodiment of the present invention, wearable devices, medical monitoring systems, and other sensors may be used to collect electrocardiogram datasets and millimeter-wave radar datasets of sample subjects.

[0041] According to an embodiment of the present invention, although the data contained in the electrocardiogram dataset is intuitive, it has certain complexity and subtle changes. Therefore, a more advanced deep learning model can be used to fully explore the electrocardiogram features. For example, a 1DResNet (one-dimensional residual network) model is used to extract features from the electrocardiogram data. The 1D ResNet model is very effective in processing one-dimensional sequence electrocardiogram data.

[0042] According to an embodiment of the present invention, electrocardiogram features are used to effectively supplement millimeter-wave radar features. It is necessary to ensure that the sample heartbeat information in the millimeter-wave radar data set is semantically consistent with the electrocardiogram data set. Therefore, feature alignment can be used to eliminate redundant information in the millimeter-wave radar data set to avoid redundant information interfering with the electrocardiogram features, so that the initial classification model can identify the millimeter-wave radar features and then determine the category of heart disease.

[0043] For example, heart disease categories may include atrial fibrillation (AF), sinus bradycardia (SB), sinus tachycardia (STach), T wave abnormality (TAb), right bundle branch block (RBBB), left bundle branch block (LBBB), complete right bundle branch block (CRBBB), complete left bundle branch block (CLBBB), etc.

[0044] In order to better demonstrate the classification effect of the target classification model obtained by the model training method for assisting in determining the category of heart disease according to an embodiment of the present invention on heart disease, the target classification model will be verified as follows.

[0045] According to an embodiment of the present invention, an electrocardiogram data set in the related art can be used as auxiliary data for millimeter wave radar data and labels. Among them, all electrocardiogram data in the electrocardiogram data set must first be normalized from 0 to 1 and resampled to 100 Hz to match the sampling rate of the millimeter wave radar data set.

[0046] According to an embodiment of the present invention, the F1 score takes into account both the precision and recall of the model and is often used to comprehensively evaluate the performance of the model. The F1 scores for different heart disease classifications are shown in Table 1 below.

[0047] Table 1

[0048]

[0049] Among them, Group P represents a classification model that has both millimeter-wave radar data and labels and uses electrocardiogram data as auxiliary data, and Group Q represents a classification model that uses millimeter-wave radar data and labels for supervised learning.

[0050] According to an embodiment of the present invention, Group P significantly improved the F1 score of millimeter-wave radar data in the classification of various heart diseases by integrating external electrocardiogram data, with the improvement ranging from 1.99% to 68.08%. And as the complexity of the disease increases and the number of samples decreases, the enhancement effect of electrocardiogram data on the F1 score is particularly obvious. For example, for the LBBB classification with a small sample size, the F1 score increased significantly from 15.38% to 83.46%. By comparison, the classification model obtained by training with electrocardiogram data to assist millimeter-wave radar data and labels is superior to the supervised learning heart disease classification model in the related art that only relies on millimeter-wave radar data and labels.

[0051] According to an embodiment of the present invention, when only millimeter-wave radar data is available without labels, that is, in the case of direct zero-shot learning, the same disease classification in Table 1 and the same external electrocardiogram dataset in the related technology can be used to evaluate the F1 score of the classification model trained using the unlabeled millimeter-wave radar data.

[0052] According to an embodiment of the present invention, when group M only has millimeter-wave radar data without labels, the F1 score of group M is compared with the F1 score of group Q as shown in Table 2 below.

[0053] Table 2

[0054]

[0055] According to the experimental results shown in Table 2, for diseases with large sample sizes such as atrial fibrillation (AF), sinus bradycardia (SB), and sinus tachycardia (STach), the M group can achieve performance close to that of the supervised learning Q group. For diseases with small sample sizes, such as atrial tachycardia (TAb) and left bundle branch block (LBBB), the F1 score of the M group even surpassed the F1 score of the Q group. Therefore, without relying on experienced medical experts to accurately label the data, the above model can still achieve accurate classification of different heart diseases.

[0056] According to an embodiment of the present invention, for the target classification model obtained by using the model training method for assisting in determining the category of heart disease in an embodiment of the present invention, that is, the target classification model obtained without using millimeter-wave radar data for training, in the absence of millimeter-wave radar data of SB or STach diseases, the F1 score for SB can still reach 0.8938, and the F1 score for STach can reach 0.5805.

[0057] It can be seen that the target classification model obtained by zero-sample learning of unlabeled millimeter-wave radar data can still accurately determine the category of heart disease using electrocardiogram data as a medium, and can also identify disease types that millimeter-wave radar technology has never encountered. Therefore, it effectively expands the development of millimeter-wave radar technology in the field of contactless heart disease monitoring.

[0058] According to an embodiment of the present invention, feature extraction is performed on the collected electrocardiogram data set and millimeter-wave radar data set of the sample object, and an initial classification model is obtained by training the obtained electrocardiogram features. According to the respective electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined period, the sample heartbeat information is semantically matched with the electrocardiogram data set, so that relatively aligned target electrocardiogram features and target millimeter-wave radar features can be obtained, and redundant information is eliminated at the same time. According to the target electrocardiogram features and the target millimeter-wave radar features, the initial classification model is adjusted so that the initial classification model trained with the electrocardiogram features can recognize the millimeter-wave radar data and assist in determining the category of heart disease. Since the electrocardiogram features are used instead of the millimeter-wave radar features when training the initial classification model, zero-sample learning of millimeter-wave radar heart disease classification is achieved, breaking the dependence of heart disease classification on millimeter-wave radar data and disease labels, using electrocardiogram data to make up for the deficiency of millimeter-wave data, so that millimeter-wave radar can detect and classify more heart diseases, further improving the accuracy of disease classification, broadening the application scope of millimeter-wave radar technology in the field of heart disease monitoring, and providing new possibilities for contactless heart health monitoring.

[0059] According to an embodiment of the present invention, an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined time period are obtained, including: obtaining an electrocardiogram data set of the sample object within the predetermined time period; obtaining a radar signal corresponding to each spatial position point of the sample object in the target space within the predetermined time period; performing second-order difference processing on the radar signal corresponding to each spatial position point to obtain a second-order radar signal; respectively extracting real data, imaginary data and phase data from each second-order radar signal; and obtaining a millimeter-wave radar data set according to the real data, imaginary data and phase data of each second-order radar signal.

[0060] According to an embodiment of the present invention, a radar signal corresponding to each spatial position point in the target space can be obtained by using a two-dimensional antenna array and a frequency modulated continuous wave (Frequency Modulated Continuous Wave, FMCW). The 3D beamforming process of the radar signal can be expressed as the following formula (1).

[0061] (1)

[0062] Wherein, n is the number of virtual channels formed by the transmit-receive antenna pair, n∈N, t is the time within the predetermined time period, t∈T, k is the frequency modulation slope of the frequency modulated continuous wave, λ is the wavelength, r(x,y,z,n) is the round-trip distance from the spatial position point (x,y,z) to the transmit-receive antenna pair, S(x,y,z,t) is the radar signal corresponding to the spatial position point (x,y,z), j is the imaginary unit, and c is the speed of light.

[0063] According to an embodiment of the present invention, in order to focus on the radar signal changes caused by heart movement, it is necessary to shield movements other than heartbeats. Therefore, the frequency difference between heartbeats and other movements can be used to perform second-order difference processing on the radar signal corresponding to each spatial position point. The second-order difference processing process can be expressed as the following formula (2).

[0064] (2)

[0065] Among them, S t '' represents the second-order derivative of the radar signal at time t, S t-3 represents the radar signal at time t-3, S t-2 represents the radar signal at time t-2, S t-1 represents the radar signal at time t-1, S t represents the radar signal at time t, S t+3 represents the radar signal at time t+3, S t+2 represents the radar signal at time t+2, S t+1 represents the radar signal at time t+1, and h represents the frame period of a predetermined period of time.

[0066] According to an embodiment of the present invention, since the radar signal is in complex form and subsequent operations such as feature extraction on the radar data are difficult to process complex numbers, real data, imaginary data and phase data are extracted from each second-order radar signal, wherein the phase data can provide time delay or offset information of the radar signal, and based on the real data, imaginary data and phase data of each second-order radar signal, a millimeter-wave radar data set including heart motion information is obtained.

[0067] According to an embodiment of the present invention, the initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: constructing a classifier sub-model according to the correspondence between electrocardiogram features and heart disease categories; using the decoder sub-model, reconstructing the electrocardiogram features to obtain reconstructed electrocardiogram data; using the reconstructed electrocardiogram data, iteratively optimizing the classifier sub-model until the loss function of the classifier sub-model meets a predetermined threshold, thereby obtaining the initial classification model.

[0068] According to an embodiment of the present invention, the classifier sub-model includes a 1D convolutional layer and a fully connected layer. The 1D convolutional layer can be used to extract time series features in the electrocardiogram features. The fully connected layer integrates the features extracted by the 1D convolutional layer and maps the correspondence between the electrocardiogram features and the heart disease categories.

[0069] According to an embodiment of the present invention, the decoder sub-model may include multiple deconvolution layers, which can reconstruct the ECG data based on the extracted ECG features, ensuring that the obtained initial classification model can accurately capture and restore the key information of the ECG data.

[0070] According to an embodiment of the present invention, the loss function may include a classification loss function and a reconstruction loss function. The decoder sub-model is used to optimize and iterate the classifier sub-model to verify whether the loss function of the classifier sub-model meets a predetermined threshold, thereby obtaining an initial classification model with high separability and strong generalization ability.

[0071] According to an embodiment of the present invention, a deep learning model is utilized to reconstruct the extracted electrocardiogram features through the deconvolution layer of the decoder sub-model, and the classifier sub-model is further verified to obtain an initial classification model with better performance, thereby providing a certain basis for the subsequent determination of the category of heart disease.

[0072] According to an embodiment of the present invention, feature extraction is performed on an electrocardiogram dataset and a millimeter-wave radar dataset respectively to obtain electrocardiogram features and millimeter-wave radar features at multiple moments in a predetermined time period, including: using a deep learning model to extract features from the electrocardiogram dataset to obtain a first initial feature; performing spatial discretization processing on the first initial feature to obtain the electrocardiogram features at multiple moments in the predetermined time period; using a deep learning model to extract features from the millimeter-wave radar dataset to obtain a second initial feature; performing spatial discretization processing on the second initial feature to obtain the millimeter-wave features at multiple moments in the predetermined time period.

[0073] According to an embodiment of the present invention, the process of performing spatial discretization processing on the first initial feature using finite scalar quantization (FSQ) technology to obtain electrocardiogram features at multiple moments within a predetermined time period can be expressed as the following formula (3).

[0074] (3)

[0075] Among them, F e represents the electrocardiogram characteristics, f e represents the first initial feature, sg() represents stop gradient, and round() represents rounding.

[0076] According to an embodiment of the present invention, the second initial feature includes a spatial feature and a temporal feature, and a deep learning model is used to perform feature extraction on a millimeter-wave radar data set to obtain the second initial feature, including: using a three-dimensional convolutional network of a deep learning model to perform feature extraction on the millimeter-wave radar data set in the spatial domain to obtain spatial features of each of multiple moments, wherein the spatial features characterize the change pattern of the millimeter-wave radar data set in three-dimensional space; using a one-dimensional convolutional network of a deep learning model to perform feature extraction on the millimeter-wave radar data set in the temporal domain to obtain time features corresponding to different time scales, wherein the time features characterize the change pattern of the millimeter-wave radar data set at different time scales; and obtaining the second initial feature based on the spatial feature and the temporal feature.

[0077] According to an embodiment of the present invention, a three-dimensional convolutional network of a deep learning model is used to extract the spatial features of the millimeter-wave radar at each moment, thereby effectively capturing the changing pattern of the radar signal in three-dimensional space. After obtaining the spatial features, a one-dimensional convolutional network of a deep learning model is used to analyze the radar signal from different time scales, thereby more comprehensively understanding the dynamic characteristics of cardiac motion.

[0078] According to an embodiment of the present invention, the finite scalar quantization (FSQ) technology can be used to compress the continuous second initial features into a discrete feature space, thereby effectively extracting and utilizing the cardiac information in the millimeter wave radar dataset, providing important support for subsequent feature alignment and disease classification.

[0079] According to an embodiment of the present invention, feature extraction is performed on an electrocardiogram dataset and a millimeter-wave radar dataset through a deep learning model, and spatial discretization is performed to obtain electrocardiogram features and millimeter-wave radar features including heart information, thereby significantly improving the accuracy and reliability of heart disease classification and providing support for heart health monitoring and disease diagnosis.

[0080] According to an embodiment of the present invention, a second initial feature is obtained based on spatial features and temporal features, including: using an attention model corresponding to a one-dimensional convolutional network to scale-fuse temporal features corresponding to different time scales to obtain time series data corresponding to the time features; and obtaining a second initial feature based on the spatial features and the time series data corresponding to the time features.

[0081] According to an embodiment of the present invention, in order to transform disordered time series features into high-level time semantic features, a Transformer model based on a self-attention mechanism in a deep learning model can be used to extract important heart-related information from time features corresponding to different time scales, and then the time features of different scales can be fused to obtain time series data.

[0082] Figure 2 A schematic diagram of extracting features from a millimeter-wave radar data set according to an embodiment of the present invention is shown.

[0083] like Figure 2 As shown, in this embodiment 200, the millimeter wave radar data set 210 is input into the three-dimensional convolution network 220 to obtain the spatial feature 230, and the millimeter wave radar data set 210 is further input into the one-dimensional convolution network 240 to obtain the time feature 250, and the time feature 250 is input into the Transformer model 260 to capture the semantic information of the time feature 250 to obtain the time semantic feature 270, and the time semantic feature is fused according to the scale fusion module 280, and the fused time feature and space feature are integrated to obtain the second initial feature 290. Among them, the time feature 250 can be a dual-scale time feature, for example, the total time length can be divided into 8 time periods or 4 time periods.

[0084] According to an embodiment of the present invention, based on the respective electrocardiogram features and millimeter-wave radar features at multiple moments, the sample heartbeat information is semantically matched with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features, including: for each moment, the gradient corresponding to the millimeter-wave radar feature is aligned with the gradient of the electrocardiogram feature, so that the sample heartbeat information in the millimeter-wave radar data set is semantically matched with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0085] According to an embodiment of the present invention, at each moment, the gradient corresponding to the millimeter-wave radar feature is aligned unidirectionally with the gradient of the electrocardiogram feature, so that the sample heartbeat information in the millimeter-wave radar data set and the electrocardiogram data set can achieve semantic matching, and the sample redundant information can also be eliminated. The feature alignment process can be expressed as the following formula (4).

[0086] (4)

[0087] Among them, loss align represents the alignment loss function, MSE() represents the mean square error, F r Represents the millimeter wave radar feature.

[0088] According to an embodiment of the present invention, the electrocardiogram features and the millimeter-wave radar features are unidirectionally aligned through the stopping gradient technology, so that the sample heartbeat information in the millimeter-wave radar data set is semantically matched with the electrocardiogram data set, and redundant information irrelevant to the sample heartbeat information is eliminated to avoid unnecessary interference to the heartbeat information.

[0089] Figure 3 A schematic diagram of a target classification model training process according to an embodiment of the present invention is shown.

[0090] like Figure 3 As shown, the collected ECG data set of the sample object is input into the ECG encoder for feature extraction to obtain the first initial feature, and the first initial feature is processed by the quantizer for spatial discretization to obtain the ECG feature. The initial classification model includes a decoder sub-model for reconstructing the ECG data and a classifier sub-model for classification. The external ECG data set is used to verify the accuracy of the model. The millimeter-wave radar data set in the same period as the ECG data set is first subjected to signal preprocessing and then enters the radar encoder for feature extraction to obtain the second initial feature. The second initial feature is processed by the quantizer for spatial discretization to obtain the millimeter-wave radar feature, and the gradient corresponding to the millimeter-wave radar feature is aligned with the gradient of the ECG feature to complete the model training and obtain the target classification model.

[0091] Figure 4A flow chart of a method for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown.

[0092] like Figure 4 As shown, the method 400 includes operations S410-S420.

[0093] In operation S410, a millimeter-wave radar signal corresponding to a target object is collected.

[0094] In operation S420, the millimeter wave radar signal is input to a target classification model, and a heart disease category corresponding to the heartbeat information is output.

[0095] According to an embodiment of the present invention, the target classification model obtained by the above-mentioned model training method can identify millimeter-wave radar signals and output corresponding heart disease categories, so that it can be applied to application scenarios of contactless heart monitoring.

[0096] Figure 5 A block diagram of a model training device for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown.

[0097] like Figure 5 As shown, the apparatus 500 includes an acquisition module 510 , an extraction module 520 , a obtaining module 530 and an adjustment module 540 .

[0098] The acquisition module 510 is used to acquire an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined period of time, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information.

[0099] The extraction module 520 is used to perform feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively, and obtain the electrocardiogram features and millimeter-wave radar features of each of the multiple time points within a predetermined time period.

[0100] The obtaining module 530 is used to semantically match the heartbeat information with the electrocardiogram data set according to the respective electrocardiogram features and millimeter-wave radar features at multiple moments, and obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0101] The adjustment module 540 is used to adjust the initial classification model trained using the electrocardiogram features according to the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

[0102] According to an embodiment of the present invention, the acquisition module 510 includes a first acquisition submodule, a second acquisition submodule, a first obtaining submodule, an extraction submodule and a second obtaining submodule.

[0103] The first acquisition submodule is used to acquire an electrocardiogram data set of a sample object within a predetermined period of time.

[0104] The second acquisition submodule is used to acquire the radar signal corresponding to the sample object and each spatial position point in the target space within a predetermined time period.

[0105] The first obtaining submodule is used to perform second-order difference processing on the radar signal corresponding to each spatial position point to obtain a second-order radar signal.

[0106] The extraction submodule is used to extract real data, imaginary data and phase data from each second-order radar signal respectively.

[0107] The second obtaining submodule is used to obtain a millimeter wave radar data set according to the real data, imaginary data and phase data of each second-order radar signal.

[0108] According to an embodiment of the present invention, the extraction module 520 includes a first obtaining submodule, a second obtaining submodule, a third obtaining submodule and a fourth obtaining submodule.

[0109] The first obtaining submodule is used to extract features from the electrocardiogram data set using a deep learning model to obtain a first initial feature.

[0110] The second obtaining submodule is used to perform spatial discretization processing on the first initial feature to obtain the electrocardiogram features of each of multiple moments within a predetermined time period.

[0111] The third submodule is used to extract features from the millimeter wave radar data set using a deep learning model to obtain a second initial feature.

[0112] The fourth obtaining submodule is used to perform spatial discretization processing on the second initial feature to obtain the millimeter wave features of each of the multiple moments in the predetermined time period.

[0113] According to an embodiment of the present invention, the third obtaining submodule includes a first obtaining unit, a second obtaining unit and a third obtaining unit.

[0114] The first obtaining unit is used to use the three-dimensional convolutional network of the deep learning model to extract the features of the millimeter-wave radar data set in the spatial domain, and obtain the spatial features of each of the multiple moments. The spatial features characterize the change law of the millimeter-wave radar data set in the three-dimensional space.

[0115] The second obtaining unit is used to use the one-dimensional convolutional network of the deep learning model to extract the time domain features of the millimeter wave radar data set to obtain time features corresponding to different time scales. The time features characterize the change rules of the millimeter wave radar data set at different time scales.

[0116] The third obtaining unit is used to obtain the second initial feature according to the spatial feature and the temporal feature.

[0117] According to an embodiment of the present invention, the third obtaining unit includes a fusion subunit and an obtaining subunit.

[0118] The fusion subunit is used to use the attention model corresponding to the one-dimensional convolutional network to scale-fuse the time features corresponding to different time scales to obtain time series data corresponding to the time features.

[0119] The subunit is obtained, and is used to obtain the second initial feature according to the spatial feature and the time series data corresponding to the time feature.

[0120] According to an embodiment of the present invention, the obtaining module 530 includes an alignment submodule.

[0121] The alignment submodule is used to align the gradient corresponding to the millimeter-wave radar feature with the gradient of the electrocardiogram feature at each moment, so that the heartbeat information in the millimeter-wave radar data set and the electrocardiogram data set can achieve semantic matching, and obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

[0122] Figure 6 A block diagram of an apparatus for assisting in determining a category of a heart disease according to an embodiment of the present invention is shown.

[0123] like Figure 6 As shown, the device 600 includes a collection module 610 and an output module 620 .

[0124] The acquisition module 610 is used to acquire the millimeter wave radar signal corresponding to the target object, wherein the millimeter wave radar signal includes heartbeat information.

[0125] The output module 620 is used to input the millimeter wave radar signal into the target classification model and output the heart disease category corresponding to the heartbeat information.

[0126] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a proper combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be executed.

[0127] For example, any multiple of the acquisition module 510, the extraction module 520, the obtaining module 530, and the adjustment module 540 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 510, the extraction module 520, the obtaining module 530, and the adjustment module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 510 , the extraction module 520 , the obtaining module 530 and the adjustment module 540 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0128] For example, any multiple of the acquisition module 610 and the output module 620 can be combined in one module / unit / sub-unit for implementation, or any one of the modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functions of one or more of these modules / units / sub-units can be combined with at least part of the functions of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present invention, at least one of the acquisition module 610 and the output module 620 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 610 and the output module 620 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0129] It should be noted that the model training device part used to assist in determining the category of heart disease in the embodiment of the present invention corresponds to the model training method part used to assist in determining the category of heart disease in the embodiment of the present invention. The description of the model training device part specifically refers to the model training method part and will not be repeated here.

[0130] It should be noted that the device part for assisting in determining the category of heart disease in the embodiment of the present invention corresponds to the method part for assisting in determining the category of heart disease in the embodiment of the present invention. The description of the device part specifically refers to the method part and will not be repeated here.

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of the present invention.

[0132] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A model training method for assisting in determining the category of heart disease, characterized in that: include: Acquire an electrocardiogram data set and a millimeter wave radar data set of a sample object within a predetermined period of time, wherein the millimeter wave radar data set includes sample heartbeat information and sample redundant information; Performing feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively to obtain electrocardiogram features and millimeter-wave radar features at respective times within the predetermined time period; According to the respective electrocardiogram features and millimeter-wave radar features at the multiple moments, semantically matching the sample heartbeat information with the electrocardiogram data set to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; According to the target electrocardiogram features and the target millimeter-wave radar features, an initial classification model trained using the electrocardiogram features is adjusted to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

2. The method according to claim 1, characterized in that The extracting features from the electrocardiogram dataset and the millimeter-wave radar dataset respectively to obtain the electrocardiogram features and millimeter-wave radar features at multiple moments within the predetermined time period includes: Using a deep learning model, extracting features from the electrocardiogram dataset to obtain a first initial feature; Performing spatial discretization processing on the first initial feature to obtain electrocardiogram features of each of multiple moments within the predetermined time period; Using the deep learning model, extracting features from the millimeter wave radar data set to obtain a second initial feature; The second initial feature is spatially discretized to obtain millimeter wave features at multiple moments in the predetermined time period.

3. The method according to claim 2, characterized in that The second initial feature includes a spatial feature and a temporal feature, and the use of the deep learning model to extract features from the millimeter wave radar data set to obtain the second initial feature includes: Using the three-dimensional convolutional network of the deep learning model, extracting features of the millimeter-wave radar data set in the spatial domain to obtain spatial features of each of the multiple moments, wherein the spatial features represent a change pattern of the millimeter-wave radar data set in the three-dimensional space; Using the one-dimensional convolutional network of the deep learning model, extracting features in the time domain of the millimeter-wave radar data set to obtain time features corresponding to different time scales, wherein the time features represent the change rules of the millimeter-wave radar data set at different time scales; The second initial feature is obtained according to the spatial feature and the temporal feature.

4. The method according to claim 3, characterized in that The obtaining the second initial feature according to the spatial feature and the temporal feature includes: Using the attention model corresponding to the one-dimensional convolutional network, scale fusion is performed on the time features corresponding to different time scales to obtain time series data corresponding to the time features; The second initial feature is obtained according to the spatial feature and the time series data corresponding to the time feature.

5. The method according to claim 1, characterized in that The initial classification model includes a classifier sub-model and a decoder sub-model, and the initial classification model is trained according to the following steps: constructing the classifier sub-model according to the correspondence between the electrocardiogram features and the categories of heart diseases; Using the decoder sub-model, reconstructing the electrocardiogram features to obtain reconstructed electrocardiogram data; The classifier sub-model is iteratively optimized using the reconstructed electrocardiogram data until the loss function of the classifier sub-model meets a predetermined threshold, thereby obtaining the initial classification model.

6. The method according to claim 1, characterized in that The semantic matching of the sample heartbeat information with the electrocardiogram data set according to the respective electrocardiogram features and millimeter-wave radar features at the multiple moments to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features includes: For each moment, the gradient corresponding to the millimeter-wave radar feature is aligned with the gradient of the electrocardiogram feature, so that the sample heartbeat information in the millimeter-wave radar data set and the electrocardiogram data set are semantically matched to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features.

7. The method according to claim 1, characterized in that The step of acquiring an electrocardiogram dataset and a millimeter wave radar dataset of a sample object within a predetermined period of time includes: Acquire the electrocardiogram data set of the sample object within the predetermined time period; Acquire radar signals corresponding to the sample object and each spatial position point in the target space within the predetermined time period; Performing second-order difference processing on the radar signal corresponding to each of the spatial position points to obtain a second-order radar signal; Extracting real data, imaginary data and phase data from each of the second-order radar signals respectively; The millimeter-wave radar data set is obtained according to the real data, the imaginary data and the phase data of each of the second-order radar signals.

8. A method for assisting in determining the type of heart disease, characterized in that: include: Collecting a millimeter-wave radar signal corresponding to the target object, wherein the millimeter-wave radar signal includes heartbeat information; The millimeter-wave radar signal is input into a target classification model, and a heart disease category corresponding to the heartbeat information is output, wherein the target classification model is trained using the method of any one of claims 1 to 7.

9. A model training device for assisting in determining the type of heart disease, characterized in that: include: An acquisition module, used to acquire an electrocardiogram data set and a millimeter-wave radar data set of a sample object within a predetermined period of time, wherein the millimeter-wave radar data set includes sample heartbeat information and sample redundant information; An extraction module, used to perform feature extraction on the electrocardiogram data set and the millimeter-wave radar data set respectively, to obtain electrocardiogram features and millimeter-wave radar features at respective times within the predetermined time period; An obtaining module, used for semantically matching the sample heartbeat information with the electrocardiogram data set according to the respective electrocardiogram features and millimeter-wave radar features at the multiple moments, to obtain relatively aligned target electrocardiogram features and target millimeter-wave radar features; The adjustment module is used to adjust the initial classification model trained using the electrocardiogram features according to the target electrocardiogram features and the target millimeter-wave radar features to obtain a target classification model, wherein the target classification model is used to determine the category of heart disease.

10. A device for assisting in determining the type of heart disease, characterized in that: include: A collection module, used to collect a millimeter-wave radar signal corresponding to a target object, wherein the millimeter-wave radar signal includes heartbeat information; An output module is used to input the millimeter wave radar signal into a target classification model and output a heart disease category corresponding to the heartbeat information, wherein the target classification model is trained using the method of any one of claims 1 to 7.

Images