Medical data processing method and device and electronic equipment

Abstract

The invention discloses a medical data processing method and apparatus, and an electronic device. The method comprises the steps of constructing a plurality of single-mode prediction models and a plurality of single-mode training data sets in one-to-one correspondence with the single-mode prediction models; based on the single-mode training data set, training a single-mode feature extractor in the corresponding single-mode prediction model; constructing a multi-modal prediction model and a multi-modal training data set; and based on the multi-modal training data set, training a multi-modal classifier in the multi-modal prediction model to obtain a trained multi-modal prediction model. The trained multi-modal prediction model can perform accurate feature representation on the medical data of the multiple modals, and on the basis, the risk of the coronary heart disease can be accurately predicted based on the medical data of the single modal or the medical data of the multiple modals through the multi-modal prediction model.

Description

Technical Field

[0001] The present application relates to the field of medical data processing technology, and in particular to a medical data processing method, device and electronic equipment. Background Art

[0002] The full name of coronary heart disease is coronary atherosclerotic heart disease, which is a type of ischemic heart disease. Coronary heart disease usually takes about twenty years to develop. The early stage of hardening is small in scope and does not cause obvious vascular stenosis, making it difficult to detect. If the patient does not receive clear anti-atherosclerotic treatment in the early stages, it is basically irreversible after formation, and once the disease occurs, the mortality rate is high. Therefore, if the risk of coronary heart disease can be predicted and intervention can be carried out in advance, it will be beneficial to delay the formation of atherosclerosis and help reduce the mortality rate of coronary heart disease.

[0003] Medical data is private data, especially personal identity information in medical data, which is usually not disclosed to the public. Only single medical data can be obtained from public databases, and these medical data usually do not contain the patient's identity information, and different types of medical data cannot usually be associated with each other. For example, medical data such as electrocardiograms and cardiac ultrasound images can be obtained from public databases, but due to personal privacy protection, electrocardiograms, cardiac ultrasound and other medical data do not contain patient identity information. The number of medical samples in a single medical institution is small, and it is usually difficult to meet the training requirements of machine learning models. On this basis, conventional coronary heart disease risk prediction models usually predict patients' coronary heart disease risks based on single-modal medical data, resulting in poor accuracy of prediction results. Summary of the invention

[0004] In view of the above problems existing in the prior art, the present application provides a medical data processing method, device and electronic device.

[0005] The first aspect of the present application provides a medical data processing method, comprising:

[0006] Constructing multiple unimodal prediction models and multiple unimodal training data sets corresponding to the unimodal prediction models one by one; wherein the unimodal prediction model includes a unimodal feature extractor and a unimodal classifier; each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different;

[0007] Based on the unimodal training data set, the unimodal feature extractor in the corresponding unimodal prediction model is trained; the unimodal feature extractor is used to extract features from medical data to obtain a unimodal feature vector;

[0008] Constructing a multimodal prediction model and a multimodal training data set; wherein the multimodal prediction model includes a plurality of unimodal feature extractors trained in the unimodal prediction model, a feature fusion module for fusing unimodal feature vectors output by the plurality of unimodal feature extractors into multimodal feature vectors, and a multimodal classifier for predicting the risk of coronary heart disease based on the multimodal feature vectors; the multimodal training data set includes medical data of multiple modalities of sample objects;

[0009] Based on the multimodal training data set, the multimodal classifier in the multimodal prediction model is trained to obtain a trained multimodal prediction model.

[0010] In some embodiments, based on the unimodal training data set, training the unimodal feature extractor in the corresponding unimodal prediction model includes:

[0011] Extracting features from the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector;

[0012] Obtaining a unimodal prediction result based on the unimodal feature vector by the unimodal classifier;

[0013] The unimodal feature extractor is iteratively trained based on the unimodal label information and unimodal prediction results corresponding to the medical data through a loss function.

[0014] In some embodiments, the plurality of unimodal prediction models include an electrocardiogram prediction model, the plurality of unimodal training data sets include an electrocardiogram training data set, and the electrocardiogram training data set includes multi-lead electrocardiogram data of a sample subject;

[0015] The medical data in the unimodal training data set is subjected to feature extraction by the unimodal feature extractor to obtain a unimodal feature vector, including:

[0016] The electrocardiogram feature extractor in the electrocardiogram prediction model extracts features of the electrocardiogram data of each lead to obtain single-lead feature information;

[0017] Extracting features from the electrocardiogram data of multiple leads by the electrocardiogram feature extractor to obtain multi-lead feature information;

[0018] Performing feature fusion on the plurality of single-lead feature information and the multi-lead feature information to obtain electrocardiogram fusion feature information;

[0019] Feature extraction is performed on the electrocardiogram fusion feature information to obtain an electrocardiogram feature vector.

[0020] In some embodiments, the electrocardiogram feature extractor performs feature extraction on the electrocardiogram data of each lead to obtain single-lead feature information, including:

[0021] Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle;

[0022] Performing feature encoding on each of the electrocardiogram segments respectively to obtain local feature information of a single lead;

[0023] Perform feature encoding on the electrocardiogram data of each lead to obtain all feature information of a single lead;

[0024] Based on the cross attention mechanism, feature fusion is performed on the single-lead local feature information and the single-lead global feature information to obtain the single-lead feature information.

[0025] In some embodiments, synchronizing electrocardiogram data of multiple leads to perform feature extraction by the electrocardiogram feature extractor to obtain multi-lead feature information includes:

[0026] Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle;

[0027] Perform feature encoding and feature fusion on multiple ECG segments of the same cardiac cycle to obtain multi-lead local feature information;

[0028] Feature encoding is performed on the electrocardiogram data of multiple leads respectively to obtain multi-lead global feature information.

[0029] In some embodiments, the plurality of single-modality prediction models include an echocardiography prediction model, and the echocardiography prediction model includes an echocardiography feature extractor; the plurality of single-modality training data sets include an echocardiography training data set, and the echocardiography training data set includes multi-view echocardiography data of a sample subject;

[0030] The medical data in the unimodal training data set is subjected to feature extraction by the unimodal feature extractor to obtain a unimodal feature vector, including:

[0031] The ultrasound cardiogram feature extractor performs feature encoding on the ultrasound cardiogram data of each viewing angle respectively to obtain feature information of multiple single viewing angles;

[0032] The feature information of multiple single-views is fused to obtain the echocardiogram feature vector.

[0033] In some embodiments, the multimodal training data set includes electrocardiogram data, echocardiogram data, and personal health records of the sample subject, as well as multimodal label information for identifying the risk of coronary heart disease of the sample subject; the multimodal prediction model includes an electrocardiogram feature extractor, an echocardiogram feature extractor, and a health record feature extractor;

[0034] Based on the multimodal training data set, the multimodal classifier in the multimodal prediction model is trained to obtain the trained multimodal prediction model, including:

[0035] Extracting features from the electrocardiogram data by using the electrocardiogram feature extractor to obtain an electrocardiogram feature vector;

[0036] Extracting features from the echocardiogram data using an echocardiogram feature extractor to obtain an echocardiogram feature vector;

[0037] Extracting features from the personal health record using a health record feature extractor to obtain a health record feature vector;

[0038] Performing feature splicing on the electrocardiogram feature vector and the echocardiogram feature vector, and performing feature splicing on the spliced ​​feature vector and the health record feature vector to obtain a multimodal feature vector;

[0039] Inputting the multimodal feature vector into the multimodal classifier to obtain a multimodal prediction result output by the multimodal classifier;

[0040] Based on the multimodal label information and the multimodal prediction result, the multimodal classifier is iteratively trained to obtain a trained multimodal prediction model.

[0041] In some embodiments, the method further comprises:

[0042] A risk prediction result capable of characterizing the coronary heart disease risk of the prediction object is obtained through a multimodal prediction model based on single-modality medical data or multiple-modality medical data of the prediction object.

[0043] A second aspect of the present application provides a medical data processing device, comprising:

[0044] A first construction module is used to construct a plurality of unimodal prediction models and a plurality of unimodal training data sets corresponding to the unimodal prediction models one by one; wherein the unimodal prediction models include a unimodal feature extractor and a unimodal classifier; each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different;

[0045] A first training module, used for training a unimodal feature extractor in the corresponding unimodal prediction model based on the unimodal training data set; the unimodal feature extractor is used for extracting features from medical data to obtain a unimodal feature vector;

[0046] A second construction module is used to construct a multimodal prediction model and a multimodal training data set; wherein the multimodal prediction model includes a plurality of unimodal feature extractors trained in the unimodal prediction models, a feature fusion module for fusing unimodal feature vectors output by the plurality of unimodal feature extractors into multimodal feature vectors, and a multimodal classifier for predicting the risk of coronary heart disease based on the multimodal feature vectors; the multimodal training data set contains medical data of multiple modalities of sample objects;

[0047] The second training module is used to train the multimodal classifier in the multimodal prediction model based on the multimodal training data set to obtain the trained multimodal prediction model.

[0048] A third aspect of the present application provides an electronic device, comprising at least a memory and a processor, wherein the memory stores an application program, and the processor implements the above-mentioned method when executing the application program on the memory.

[0049] The medical data processing method of the embodiment of the present application, since the unimodal training data set only contains the medical data of a single modality of the sample object, and does not require the sample objects of the medical data in different unimodal training data sets to be related to each other, can obtain the medical data of a single modality from the public data set to construct the unimodal training data set, so that the training data in the unimodal training data set can have a higher data volume, thereby ensuring the training effect of the unimodal feature extractor, so that the unimodal feature vector output by the unimodal feature extractor can accurately represent the typical characteristics of the medical data. When the unimodal feature extractor has been trained, the multimodal classifier is trained by the limited private multimodal medical data in the multimodal training data set, which can still ensure that the multimodal classifier forms a good fitting effect. The obtained multimodal prediction model can accurately represent the features of medical data of multiple modalities. On this basis, the multimodal prediction model can accurately predict the risk of coronary heart disease based on medical data of a single modality or medical data of multiple modalities. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a medical data processing method according to an embodiment of the present application;

[0051] Figure 2 This is a flowchart of step S120 in the medical data processing method of an embodiment of the present application;

[0052] Figure 3 Data flow diagram for the ECG feature extractor;

[0053] Figure 4 The data flow diagram for the echocardiogram feature extractor;

[0054] Figure 5 Data flow diagram for multimodal prediction model;

[0055] Figure 6 This is a structural block diagram of a medical data processing device according to an embodiment of the present application;

[0056] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0058] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but only as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0059] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0060] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0061] It should also be understood that although the present application has been described with reference to some specific examples, those skilled in the art will be able to readily implement many other equivalent forms of the present application.

[0062] The above and other aspects, features and advantages of the present application will become more apparent in view of the following detailed description when taken in conjunction with the accompanying drawings.

[0063] Specific embodiments of the present application are described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments applied for are merely examples of the present application, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present application. Therefore, the specific structural and functional details applied for herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present application in a variety of ways with substantially any suitable detailed structure.

[0064] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present application.

[0065] The present application embodiment provides a medical data processing method, Figure 1 This is a flowchart of the medical data processing method of the present application embodiment, see Figure 1 As shown, the medical data processing method of the embodiment of the present application may specifically include the following steps.

[0066] S110, constructing a plurality of unimodal prediction models and a plurality of unimodal training data sets corresponding one-to-one to the unimodal prediction models.

[0067] The unimodal prediction model includes a unimodal feature extractor and a unimodal classifier. The unimodal feature extractor is used to extract features from medical data and obtain a unimodal feature vector. The unimodal classifier is used to output a unimodal prediction result based on the unimodal feature vector, and the unimodal prediction result can characterize the coronary heart disease risk of the sample object. It should be noted that the coronary heart disease risk characterizes the possibility that the sample object or the predicted object will develop coronary heart disease in the future, so as to provide a reference for early intervention behavior. The coronary heart disease risk does not characterize whether the sample object or the predicted object currently suffers from coronary heart disease.

[0068] Optionally, multiple single-modality prediction models may be constructed one-to-one for multiple modal medical data. For example, a first single-modality prediction model may be constructed for the first modal medical data, and the first single-modality prediction model may be used to predict the risk of coronary heart disease based on the first modal medical data. A second modality prediction model may also be constructed for the second modal medical data, and the second modality prediction model may be used to predict the risk of coronary heart disease based on the second modal medical data.

[0069] Each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different. Optionally, the unimodal training data sets may also include unimodal label information, and the unimodal label information is used to identify the coronary heart disease risk of the sample object. The unimodal training data set includes medical data of a single modality of a sample object, which means that the unimodal training data set only contains a certain type of medical data of the sample object. The different modalities of medical data in different unimodal training data sets should be understood as different types of medical data in different unimodal training data sets.

[0070] For example, for the first unimodal prediction model, a first modality training data set may be constructed based on the first modality medical data of the sample object and the first unimodal label information corresponding to the first modality medical data. For the second unimodal prediction model, a second modality training data set may be constructed based on the second modality medical data of the sample object and the second unimodal label information corresponding to the second modality medical data.

[0071] Since the unimodal training dataset only contains the medical data of a single modality of the sample objects, and does not require the sample objects of the medical data in different unimodal training datasets to be associated with each other, the medical data of a single modality can be obtained from the public dataset to construct the unimodal training dataset, so that the training data in the unimodal training dataset can have a higher data volume.

[0072] S120: Based on the unimodal training data set, training the unimodal feature extractor in the corresponding unimodal prediction model.

[0073] Optionally, the first feature extractor in the first unimodal prediction model can be trained based on the first modal training data set, and when the output result of the first unimodal prediction model meets the first target condition, it can be determined that the training of the first unimodal prediction model is completed. The second feature extractor in the second unimodal prediction model can be trained based on the second modal training data set, and when the output result of the second unimodal prediction model meets the second target condition, it can be determined that the training of the second unimodal prediction model is completed. And so on, until the unimodal feature extractors in multiple unimodal prediction models are trained, the parameters in the multiple unimodal feature extractors can be solidified respectively.

[0074] By training the unimodal feature extractor in the unimodal prediction model with a unimodal training data set, the unimodal feature extractor can be trained to have the feature representation capability for the corresponding medical data, so that the unimodal feature extractor can output a feature vector that can represent the typical characteristics of the medical data based on the medical data.

[0075] S130, constructing a multimodal prediction model and a multimodal training dataset.

[0076] The multimodal training data set includes medical data of multiple modalities of the sample object, and multimodal label information for identifying the risk of coronary heart disease of the sample object. It is understandable that all or part of the sample objects in the multimodal training data set should have medical data of multiple modalities. For example, private medical data of multiple modalities of patients or physical examination personnel can be obtained from medical institutions such as hospitals and physical examination centers, and a multimodal training data set can be constructed based on these private medical data.

[0077] It should be noted here that the acquisition, storage and application of private medical data in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals. Specifically, the personal privacy of patients or medical examinees can be protected by processing the relevant private medical data in the privacy module, desensitizing the private medical data, and refusing to display or transmit the relevant private medical data.

[0078] The multimodal prediction model includes multiple unimodal feature extractors, a feature fusion module and a multimodal classifier. The multiple unimodal feature extractors are unimodal feature extractors trained in multiple unimodal prediction models. The feature fusion module is used to fuse the unimodal feature vectors output by the multiple unimodal feature extractors into a multimodal feature vector. The multimodal classifier is used to predict the risk of coronary heart disease based on the multimodal feature vector to obtain a multimodal prediction result.

[0079] S140, based on the multimodal training data set, training the multimodal classifier in the multimodal prediction model to obtain a trained multimodal prediction model.

[0080] Optionally, the medical data of multiple modalities of the sample object can be respectively input into the corresponding unimodal feature extractors, and the corresponding medical data can be respectively extracted by each unimodal feature extractor to obtain multiple unimodal feature vectors. The multiple unimodal feature vectors are fused into a multimodal feature vector by the feature fusion module, and the multimodal feature vector is input into the multimodal classifier, and the multimodal classifier outputs a multimodal prediction result based on the multimodal feature vector. The multimodal classifier can be iteratively trained based on the multimodal prediction result and the multimodal label information by a multimodal loss function until the output result of the multimodal separator meets the end condition of the iterative training, and the parameters of the multimodal classifier are solidified to complete the training of the entire multimodal prediction model.

[0081] The medical data processing method of the embodiment of the present application, since the unimodal training data set only contains the medical data of a single modality of the sample object, and does not require the sample objects of the medical data in different unimodal training data sets to be related to each other, can obtain the medical data of a single modality from the public data set to construct the unimodal training data set, so that the training data in the unimodal training data set can have a higher data volume, thereby ensuring the training effect of the unimodal feature extractor, so that the unimodal feature vector output by the unimodal feature extractor can accurately represent the typical characteristics of the medical data. When the unimodal feature extractor has been trained, the multimodal classifier is trained by the limited private multimodal medical data in the multimodal training data set, which can still ensure that the multimodal classifier forms a good fitting effect. The obtained multimodal prediction model can accurately represent the features of medical data of multiple modalities. On this basis, the multimodal prediction model can accurately predict the risk of coronary heart disease based on medical data of a single modality or medical data of multiple modalities.

[0082] Cooperate Figure 2 As shown, in some embodiments, step S120, based on the unimodal training data set, training the unimodal feature extractor in the corresponding unimodal prediction model may include the following steps.

[0083] S121, performing feature extraction on the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector.

[0084] S122: Obtain a unimodal prediction result based on the unimodal feature vector through the unimodal classifier.

[0085] S123, iteratively training the unimodal feature extractor based on the unimodal label information and the unimodal prediction result through a loss function.

[0086] Optionally, the loss function can be expressed by the following formula.

[0087]

[0088] Wherein, H represents the loss function; P represents the probability of being predicted as a label category; Q represents the predicted label category; i represents the i-th medical data in the unimodal training data set. Under the guidance of the loss function, it is conducive to the rapid fitting of the unimodal prediction model.

[0089] Cooperate Figure 3As shown, in some embodiments, the plurality of unimodal prediction models include an electrocardiogram prediction model, and the electrocardiogram prediction model may include an electrocardiogram feature extractor. The plurality of unimodal training data sets include an electrocardiogram training data set, and the electrocardiogram training data set includes multi-lead electrocardiogram data of a sample subject.

[0090] Step S121, extracting features from the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector, may include the following steps.

[0091] The electrocardiogram feature extractor is used to extract features from the electrocardiogram data of each lead to obtain single-lead feature information.

[0092] The electrocardiogram feature extractor is used to extract features from electrocardiogram data of multiple leads to obtain multi-lead feature information.

[0093] Feature fusion is performed on the plurality of single-lead feature information and the multi-lead feature information to obtain electrocardiogram fusion feature information.

[0094] Feature extraction is performed on the electrocardiogram fusion feature information to obtain an electrocardiogram feature vector.

[0095] Optionally, the multi-lead ECG data includes but is not limited to twelve-lead ECG data. For example, twelve-lead ECG data of 500 Hz for 10 seconds may be used. Optionally, the ECG feature extractor may include an encoder, a feature fusion unit, and a transformer. The encoder may be used to perform feature extraction on the twelve-lead ECG data to obtain twelve single-lead feature information. The encoder may also be used to synchronously perform feature extraction on the twelve-lead ECG data to obtain multi-lead feature information. The feature fusion unit may be used to perform feature fusion on the twelve single-lead feature information and the multi-lead feature information to obtain ECG fusion feature information. The ECG fusion feature information may be feature extracted by a transformer to obtain an ECG feature vector. Feature extraction is performed from two dimensions, namely, single-lead ECG data and multiple-lead ECG data, which is conducive to comprehensive acquisition of feature information in the ECG data.

[0096] Cooperate Figure 3 As shown, in some embodiments, the electrocardiogram data of each lead is extracted with the electrocardiogram feature extractor to obtain single-lead feature information, which may include the following steps.

[0097] The ECG data of each lead is segmented into ECG segments corresponding to the cardiac cycle.

[0098] Feature encoding is performed on each of the electrocardiogram segments to obtain local feature information of a single lead.

[0099] Perform feature encoding on the electrocardiogram data of each lead to obtain all feature information of a single lead.

[0100] Based on the cross attention mechanism, feature fusion is performed on the single-lead local feature information and the single-lead global feature information to obtain the single-lead feature information.

[0101] Optionally, the electrocardiogram data of a single lead may be segmented into electrocardiogram segments corresponding to the cardiac cycle based on a peak detection algorithm. For example, the electrocardiogram data of a single lead may be segmented into electrocardiogram segments including a P wave, a QRS wave, and a T wave using an R peak detection (Beat Segmentation) algorithm.

[0102] Optionally, the electrocardiogram feature extractor may include a first encoder, a second encoder, a first feature fusion unit, and a second feature fusion unit. The first encoder may be used to perform feature encoding on a single electrocardiogram segment to obtain single-lead local feature information. The second encoder may be used to perform feature encoding on each single-lead electrocardiogram data to obtain single-lead global feature information. The first feature fusion unit may perform feature fusion on the single-lead local feature information and the single-lead global feature information based on a cross attention mechanism to obtain single-lead feature information. The second feature fusion unit may perform feature fusion on the single-lead feature information and the multi-lead feature information to obtain an electrocardiogram feature vector. In this way, feature extraction can be performed from the local and global aspects of the electrocardiogram data of a single lead respectively, and both coarse-grained feature information and fine-grained feature information can be obtained, which is conducive to improving the feature representation capability of the Xindian feature extractor.

[0103] Cooperate Figure 3 As shown, in some embodiments, synchronizing electrocardiogram data of multiple leads to perform feature extraction through the electrocardiogram feature extractor to obtain multi-lead feature information may include the following steps.

[0104] The electrocardiogram data of each lead is segmented into electrocardiogram segments corresponding to the cardiac cycle.

[0105] Feature encoding and feature fusion are performed on multiple ECG segments of the same cardiac cycle to obtain multi-lead local feature information.

[0106] Feature encoding is performed on the electrocardiogram data of multiple leads respectively to obtain multi-lead global feature information.

[0107] Optionally, the electrocardiogram feature extractor may further include a third encoder and a fourth encoder, and the third encoder may be used to perform feature encoding on multiple electrocardiogram segments of the same cardiac cycle, and feature fusion is performed on the feature information after feature encoding to obtain multi-lead local feature information. For example, the third encoder may be used to perform feature encoding on twelve electrocardiogram segments of the same cardiac cycle in twelve-lead electrocardiogram data, and feature fusion is performed on the feature information after twelve feature encoding to obtain multi-lead local feature information.

[0108] Optionally, the fourth encoder may be used to perform feature encoding on the electrocardiogram data of multiple leads respectively to obtain multi-lead global feature information. Then, the single-lead feature information, multi-lead local feature information and multi-lead global feature information may be feature fused by, for example, a second feature fusion unit to obtain the electrocardiogram feature vector.

[0109] Cooperate Figure 4 As shown, in some embodiments, the plurality of single-modality prediction models include an echocardiography prediction model, and the echocardiography prediction model includes an echocardiography feature extractor. The plurality of single-modality training data sets include an echocardiography training data set, and the echocardiography training data set includes multi-view echocardiography data of a sample subject.

[0110] Step S121, extracting features from the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector, may include the following steps.

[0111] The ultrasound cardiogram feature extractor performs feature encoding on the ultrasound cardiogram data of each viewing angle respectively to obtain feature information of multiple single viewing angles;

[0112] The feature information of multiple single-views is fused to obtain the echocardiogram feature vector.

[0113] Optionally, the multi-view echocardiographic data may include four-view echocardiographic data, such as Figure 4 As shown. Optionally, the echocardiogram feature extractor may include an initial residual encoder (Inception-ResNet) and a third feature fusion unit. The initial residual encoder can be used to perform feature encoding on the echocardiogram data of each perspective respectively to obtain multiple single-perspective feature information. The third feature fusion unit can perform feature fusion on multiple single-perspective feature information to obtain an echocardiogram feature vector. The initial residual encoder has multiple convolutional layers and residual connections, and can process features of different scales, positions and directions in the echocardiogram video. In addition, the initial residual encoder also has a small number of parameters and a low computational complexity, which is conducive to processing large-scale echocardiogram video data.

[0114] Cooperate Figure 5 As shown, in some embodiments, the multimodal training data set includes electrocardiogram data, echocardiogram data and personal health records of sample subjects, as well as multimodal label information for identifying the risk of coronary heart disease of the sample subjects. The multimodal prediction model includes an electrocardiogram feature extractor, an echocardiogram feature extractor and a health record feature extractor.

[0115] Step S140, based on the multimodal training data set, training the multimodal classifier in the multimodal prediction model to obtain a trained multimodal prediction model, may include the following steps.

[0116] S141, extracting features from the electrocardiogram data by the electrocardiogram feature extractor to obtain an electrocardiogram feature vector.

[0117] S142, extracting features from the echocardiogram data using an echocardiogram feature extractor to obtain an echocardiogram feature vector.

[0118] S143, extracting features from the personal health record using a health record feature extractor to obtain a health record feature vector.

[0119] S144, feature splicing the electrocardiogram feature vector and the echocardiogram feature vector, and feature splicing the spliced ​​feature vector with the health record feature vector to obtain a multimodal feature vector.

[0120] S145, inputting the multimodal feature vector into the multimodal classifier to obtain a multimodal prediction result output by the multimodal classifier.

[0121] S146: Based on the multimodal label information and the multimodal prediction result, iteratively train the multimodal classifier to obtain a trained multimodal prediction model.

[0122] Optionally, the personal health record may include, but is not limited to, information such as age, gender, smoking history, drinking history, total cholesterol, diabetes, hypertension, chest pain type, serum cholesterol, body mass index (BMI), maximum heart rate, etc. Optionally, the health record feature extractor may use a transformer-based bidirectional encoder (BERT-base-chinese).

[0123] Optionally, the multimodal classifier may be formed by a multilayer perceptron. When a multimodal feature vector is obtained, the multimodal feature vector may be input into the multilayer perceptron, and the risk of coronary heart disease of the sample object may be predicted by the multilayer perceptron.

[0124] It should be noted that, although the above embodiments use personal health records, electrocardiogram data and echocardiogram data as examples to illustrate the medical data processing method of the present application, it should not be understood that the medical data is limited to personal health records, electrocardiogram data and echocardiogram data, and the medical data may also include but is not limited to coronary angiography test results, electronic computed tomography (CT) images, cardiac radionuclide imaging images, coronary artery CTA test results, cardiac magnetic resonance imaging images, etc.

[0125] In some embodiments, the method may further include: step S140, obtaining a risk prediction result capable of characterizing the risk of coronary heart disease of the predicted object based on single-modality medical data or multiple-modality medical data of the predicted object through a multimodal prediction model.

[0126] When obtaining medical data of a single modality of the prediction object, the medical data can be feature extracted by a single modality feature extractor corresponding to the single modality medical data in the multimodal prediction model to obtain a single modality feature vector. Subsequently, the single modality feature vector can be input into a multimodal classifier, and a risk prediction result can be obtained based on the single modality feature vector by the multimodal classifier.

[0127] When multiple modal medical data of the predicted object are obtained, the medical data of multiple modalities can be respectively input into the corresponding unimodal feature extractors, and the unimodal feature vectors are respectively output by each unimodal feature extractor. The feature fusion module is used to perform feature fusion on multiple unimodal feature vectors to obtain a multimodal feature vector. Then, a risk prediction result is obtained based on the multimodal feature vector through a multimodal classifier. The multimodal prediction model can accurately predict the risk of coronary heart disease based on single modal medical data or multiple modal medical data.

[0128] See also Figure 6 As shown, the embodiment of the present application also provides a medical data processing device, including:

[0129] The first construction module 201 is used to construct multiple unimodal prediction models and multiple unimodal training data sets corresponding to the modal prediction models one by one; wherein the unimodal prediction models include a unimodal feature extractor and a unimodal classifier; each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different;

[0130] The first training module 202 is used to train the unimodal feature extractor in the corresponding unimodal prediction model based on the unimodal training data set; the unimodal feature extractor is used to extract features from medical data to obtain a unimodal feature vector;

[0131] The second construction module 203 is used to construct a multimodal prediction model and a multimodal training data set; wherein the multimodal prediction model includes a plurality of unimodal feature extractors trained in the unimodal prediction models, a feature fusion module for fusing unimodal feature vectors output by the plurality of unimodal feature extractors into multimodal feature vectors, and a multimodal classifier for predicting the risk of coronary heart disease based on the multimodal feature vectors; the multimodal training data set includes medical data of multiple modalities of sample objects;

[0132] The second training module 204 is used to train the multimodal classifier in the multimodal prediction model based on the multimodal training data set to obtain a trained multimodal prediction model.

[0133] In some embodiments, the first training module 202 is specifically used to:

[0134] Extracting features from the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector;

[0135] Obtaining a unimodal prediction result based on the unimodal feature vector by the unimodal classifier;

[0136] The unimodal feature extractor is iteratively trained based on the unimodal label information and unimodal prediction results corresponding to the medical data through a loss function.

[0137] In some embodiments, the plurality of unimodal prediction models include an electrocardiogram prediction model, the plurality of unimodal training data sets include an electrocardiogram training data set, and the electrocardiogram training data set includes multi-lead electrocardiogram data of a sample object; the first training module 202 is specifically used for:

[0138] The electrocardiogram feature extractor in the electrocardiogram prediction model extracts features of the electrocardiogram data of each lead to obtain single-lead feature information;

[0139] Extracting features from the electrocardiogram data of multiple leads by the electrocardiogram feature extractor to obtain multi-lead feature information;

[0140] Performing feature fusion on the plurality of single-lead feature information and the multi-lead feature information to obtain electrocardiogram fusion feature information;

[0141] Feature extraction is performed on the electrocardiogram fusion feature information to obtain an electrocardiogram feature vector.

[0142] In some embodiments, the first training module 202 is specifically used to:

[0143] Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle;

[0144] Performing feature encoding on each of the electrocardiogram segments respectively to obtain local feature information of a single lead;

[0145] Perform feature encoding on the electrocardiogram data of each lead to obtain all feature information of a single lead;

[0146] Based on the cross attention mechanism, feature fusion is performed on the single-lead local feature information and the single-lead global feature information to obtain the single-lead feature information.

[0147] In some embodiments, the first training module 202 is specifically used to:

[0148] Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle;

[0149] Perform feature encoding and feature fusion on multiple ECG segments of the same cardiac cycle to obtain multi-lead local feature information;

[0150] Feature encoding is performed on the electrocardiogram data of multiple leads respectively to obtain multi-lead global feature information.

[0151] In some embodiments, the plurality of single-modality prediction models include an echocardiography prediction model, and the echocardiography prediction model includes an echocardiography feature extractor; the plurality of single-modality training data sets include an echocardiography training data set, and the echocardiography training data set includes multi-view echocardiography data of a sample object; the first training module 202 is specifically used for:

[0152] The ultrasound cardiogram feature extractor performs feature encoding on the ultrasound cardiogram data of each viewing angle respectively to obtain feature information of multiple single viewing angles;

[0153] The feature information of multiple single-views is fused to obtain the echocardiogram feature vector.

[0154] In some embodiments, the multimodal training data set includes electrocardiogram data, echocardiogram data and personal health records of the sample subject, and multimodal label information for identifying the risk of coronary heart disease of the sample subject; the multimodal prediction model includes an electrocardiogram feature extractor, an echocardiogram feature extractor and a health record feature extractor; the second training module 204 is specifically used to:

[0155] Extracting features from the electrocardiogram data by using the electrocardiogram feature extractor to obtain an electrocardiogram feature vector;

[0156] Extracting features from the echocardiogram data using an echocardiogram feature extractor to obtain an echocardiogram feature vector;

[0157] Extracting features from the personal health record using a health record feature extractor to obtain a health record feature vector;

[0158] Performing feature splicing on the electrocardiogram feature vector and the echocardiogram feature vector, and performing feature splicing on the spliced ​​feature vector and the health record feature vector to obtain a multimodal feature vector;

[0159] Inputting the multimodal feature vector into the multimodal classifier to obtain a multimodal prediction result output by the multimodal classifier;

[0160] Based on the multimodal label information and the multimodal prediction result, the multimodal classifier is iteratively trained to obtain a trained multimodal prediction model.

[0161] In some embodiments, the device further comprises a risk prediction module, wherein the risk prediction module is specifically configured to:

[0162] A risk prediction result capable of characterizing the coronary heart disease risk of the prediction object is obtained through a multimodal prediction model based on single-modality medical data or multiple-modality medical data of the prediction object.

[0163] See also Figure 7 As shown, an embodiment of the present application further provides an electronic device, comprising at least a memory 301 and a processor 302, wherein the memory 301 stores an application program, and the processor 302 implements the method described in any of the above embodiments when executing the application program on the memory 301.

[0164] An embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions in the computer-readable storage medium are executed, the method described in any of the above embodiments is implemented.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0166] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.

[0167] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0168] The above-mentioned readable storage medium can be a disk, an optical disk, a DVD, a USB, a read-only memory (ROM) or a random access memory (RAM), etc. This application does not limit the specific form of the storage medium.

[0169] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A medical data processing method, characterized in that: include: Constructing multiple unimodal prediction models and multiple unimodal training data sets corresponding to the unimodal prediction models one by one; wherein the unimodal prediction model includes a unimodal feature extractor and a unimodal classifier; each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different; Based on the unimodal training data set, the unimodal feature extractor in the corresponding unimodal prediction model is trained; the unimodal feature extractor is used to extract features from medical data to obtain a unimodal feature vector; Constructing a multimodal prediction model and a multimodal training data set; wherein the multimodal prediction model includes a plurality of unimodal feature extractors trained in the unimodal prediction model, a feature fusion module for fusing unimodal feature vectors output by the plurality of unimodal feature extractors into multimodal feature vectors, and a multimodal classifier for predicting the risk of coronary heart disease based on the multimodal feature vectors; the multimodal training data set includes medical data of multiple modalities of sample objects; Based on the multimodal training data set, the multimodal classifier in the multimodal prediction model is trained to obtain a trained multimodal prediction model.

2. The method according to claim 1, characterized in that: Based on the unimodal training data set, training a unimodal feature extractor in the corresponding unimodal prediction model includes: Performing feature extraction on the medical data in the unimodal training data set by the unimodal feature extractor to obtain a unimodal feature vector; Obtaining a unimodal prediction result based on the unimodal feature vector by the unimodal classifier; The unimodal feature extractor is iteratively trained based on the unimodal label information and unimodal prediction results corresponding to the medical data through a loss function.

3. The method according to claim 2, characterized in that The plurality of single-modality prediction models include an electrocardiogram prediction model, the plurality of single-modality training data sets include an electrocardiogram training data set, and the electrocardiogram training data set includes multi-lead electrocardiogram data of a sample object; The medical data in the unimodal training data set is subjected to feature extraction by the unimodal feature extractor to obtain a unimodal feature vector, including: The electrocardiogram feature extractor in the electrocardiogram prediction model extracts features of the electrocardiogram data of each lead to obtain single-lead feature information; Extracting features from the electrocardiogram data of multiple leads by the electrocardiogram feature extractor to obtain multi-lead feature information; Performing feature fusion on the plurality of single-lead feature information and the multi-lead feature information to obtain electrocardiogram fusion feature information; Feature extraction is performed on the electrocardiogram fusion feature information to obtain an electrocardiogram feature vector.

4. The method according to claim 3, characterized in that The electrocardiogram feature extractor is used to extract features from the electrocardiogram data of each lead to obtain single-lead feature information, including: Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle; Performing feature encoding on each of the electrocardiogram segments respectively to obtain local feature information of a single lead; Perform feature encoding on the electrocardiogram data of each lead to obtain all feature information of a single lead; Based on the cross attention mechanism, feature fusion is performed on the single-lead local feature information and the single-lead global feature information to obtain the single-lead feature information.

5. The method according to claim 3, characterized in that: The electrocardiogram feature extractor synchronizes electrocardiogram data of multiple leads to perform feature extraction and obtain multi-lead feature information, including: Segmenting the electrocardiogram data of each lead into electrocardiogram segments corresponding to the cardiac cycle; Perform feature encoding and feature fusion on multiple ECG segments of the same cardiac cycle to obtain multi-lead local feature information; Feature encoding is performed on the electrocardiogram data of multiple leads respectively to obtain multi-lead global feature information.

6. The method according to claim 2, characterized in that The plurality of single-modality prediction models include an echocardiography prediction model, and the echocardiography prediction model includes an echocardiography feature extractor; the plurality of single-modality training data sets include an echocardiography training data set, and the echocardiography training data set includes multi-view echocardiography data of a sample object; The medical data in the unimodal training data set is subjected to feature extraction by the unimodal feature extractor to obtain a unimodal feature vector, including: The ultrasound cardiogram feature extractor performs feature encoding on the ultrasound cardiogram data of each viewing angle respectively to obtain a plurality of single viewing angle feature information; The feature information of multiple single-views is fused to obtain the echocardiogram feature vector.

7. The method according to claim 1, characterized in that The multimodal training data set includes electrocardiogram data, echocardiogram data and personal health records of the sample subject, and multimodal label information for identifying the coronary heart disease risk of the sample subject; the multimodal prediction model includes an electrocardiogram feature extractor, an echocardiogram feature extractor and a health record feature extractor; Based on the multimodal training data set, the multimodal classifier in the multimodal prediction model is trained to obtain the trained multimodal prediction model, including: Performing feature extraction on the electrocardiogram data by the electrocardiogram feature extractor to obtain an electrocardiogram feature vector; Extracting features from the echocardiogram data using an echocardiogram feature extractor to obtain an echocardiogram feature vector; Extracting features from the personal health record using a health record feature extractor to obtain a health record feature vector; Performing feature splicing on the electrocardiogram feature vector and the echocardiogram feature vector, and performing feature splicing on the spliced ​​feature vector and the health record feature vector to obtain a multimodal feature vector; Inputting the multimodal feature vector into the multimodal classifier to obtain a multimodal prediction result output by the multimodal classifier; Based on the multimodal label information and the multimodal prediction result, the multimodal classifier is iteratively trained to obtain a trained multimodal prediction model.

8. The method according to claim 1, characterized in that The method further comprises: A risk prediction result capable of characterizing the coronary heart disease risk of the prediction object is obtained through a multimodal prediction model based on single-modality medical data or multiple-modality medical data of the prediction object.

9. A medical data processing device, characterized in that: include: A first construction module is used to construct a plurality of unimodal prediction models and a plurality of unimodal training data sets corresponding to the unimodal prediction models one by one; wherein the unimodal prediction models include a unimodal feature extractor and a unimodal classifier; each of the unimodal training data sets includes medical data of a single modality of a sample object, and the medical data modalities in different unimodal training data sets are different; A first training module, used for training a unimodal feature extractor in the corresponding unimodal prediction model based on the unimodal training data set; the unimodal feature extractor is used for extracting features from medical data to obtain a unimodal feature vector; A second construction module is used to construct a multimodal prediction model and a multimodal training data set; wherein the multimodal prediction model includes a plurality of unimodal feature extractors trained in the unimodal prediction models, a feature fusion module for fusing unimodal feature vectors output by the plurality of unimodal feature extractors into multimodal feature vectors, and a multimodal classifier for predicting the risk of coronary heart disease based on the multimodal feature vectors; the multimodal training data set contains medical data of multiple modalities of sample objects; The second training module is used to train the multimodal classifier in the multimodal prediction model based on the multimodal training data set to obtain the trained multimodal prediction model.

10. An electronic device comprising at least a memory and a processor, characterized in that: An application program is stored in the memory, and the processor implements the method according to any one of claims 1 to 8 when executing the application program in the memory.

Images