An ehr data analysis method and system based on augmented discriminative information

Abstract

The application discloses an EHR data analysis method and system based on augmented discriminative information and belongs to the technical field of artificial intelligence and digital medical treatment.The system comprises a computer processor, a memory and a graphic processor; a multi-modal EHR data storage unit; a multi-modal EHR data preprocessing unit; a multi-modal contrast learning deep network model training unit; a multi-modal contrast learning deep network model storage unit and a downstream task unit.Through the above system, the application firstly constructs a multi-modal EHR data set with large data volume and rich data content, and secondly designs an EHR data analysis method based on augmented discriminative information which can effectively utilize unlabelled EHR image data and EHR text data.The method can extract a feature representation vector containing more discriminative information from EHR data, thereby improving the utilization rate of medical data and the accuracy of downstream medical tasks by using the feature representation vector.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and digital medical technology, and in particular to an EHR data analysis method and system based on augmented discriminant information. Background Art

[0002] With the continuous integration and mutual promotion of technologies such as artificial intelligence, the Internet of Things, big data, and the fifth generation of mobile communications, the global medical industry has developed greatly towards electronicization, informatization, and intelligence, resulting in a variety of new medical forms such as Internet medicine, mobile medicine, and smart medicine. These new forms of electronic medical industries generate a large number of electronic health records (EHRs) every day. EHRs record patients' personal information and medical diagnosis results in the form of electronic data, such as identity information, height, weight, medical history, medication and allergy history, diagnosis, clinical records, vital signs, etc. Compared with traditional medical data, EHR data has the advantages of more comprehensive longitudinal medical diagnosis information, stronger data sharing across medical institutions, and higher efficiency in disease diagnosis and patient management. However, due to the following two reasons, EHR data cannot be fully applied through supervised tasks: (1) EHR data lacks high-quality manual annotation; (2) rule-based label extraction methods can only be applied to specific electronic medical database information and cannot be accurately applied to any form of electronic medical database information. This makes it difficult for researchers to directly and effectively use EHR data for in-depth pathological research or to implement intelligent diagnostic systems. Therefore, researchers began to explore how to use unsupervised learning methods or self-supervised learning methods to conduct deeper analysis and utilization of EHR data.

[0003] The EHR data is mainly composed of image data information and text data information without artificial annotation. In the field of self-supervised learning, the contrastive learning method can effectively utilize these unlabeled data to train a deep network model (pre-trained model) for extracting sample features with a simple pre-task (instance-level contrastive task). Specifically, the instance-level contrastive task is to regard the current sample and its augmented positive example as the same "class", and regard the current sample and other samples and the augmentations of other samples as different "classes". By training the contrastive learning deep network model to make the same "class" similar and the different "classes" far away, the contrastive learning deep network model can extract discriminative feature information. After obtaining the contrastive learning deep network model, we can apply the feature information extracted by the model to downstream medical tasks, such as classifying the EHR images of patients to realize computer preliminary diagnosis of patients, or even fully realize intelligent medical treatment; segmenting or target detecting the EHR images of patients to accurately locate the lesion area of patients, greatly reducing the diagnosis difficulty of doctors and improving the diagnosis efficiency and quality of doctors; predicting the clinical results of patients through the EHR images and text data information of patients to find other diseases that have not yet been found or some diseases that may occur in the future.

[0004] Recently, methods using contrastive learning methods for feature extraction from EHR data include: Chaitanya et al. proposed a new strategy for contrastive learning using the structural similarity of volumetric EHR images in "Contrastive learning of global and local features for medical images segmentation with limited annotations", using local contrast loss to learn unique local feature representations of EHR images, significantly improving the performance of contrastive learning methods in specific pixel-level image segmentation tasks; Yan et al. proposed a weakly supervised contrastive learning framework for generating chest X-ray reports in "Weakly supervised contrastive learning for chest X-ray report generation", assigning more weight to semantically similar target reports (hard negative samples) during the model training process, ultimately producing better results in clinical accuracy and text generation indicators; Zhang et al. proposed the ConVIRT method in "Contrastive learning of medical visual representations from paired images and text", using paired information consisting of EHR image data and EHR text data for bidirectional contrastive learning, making contrastive learning methods applicable to multimodal EHR data information.

[0005] The patent with publication number CN113673613A discloses a multi-modal data feature expression method, device and medium based on contrast learning. The method includes: obtaining a multi-modal data feature expression method sample, and preprocessing the multi-modal data feature expression method sample to obtain a feature vector combination; replacing at least one feature vector in the feature vector combination to obtain a contrast feature vector combination; inputting the feature vector combination and the contrast feature vector combination into a multi-layer encoding and decoding model respectively to obtain positive and negative feature expressions of the multi-modal data feature expression method sample; training parameters of the multi-layer encoding and decoding model according to the positive and negative feature expressions of the multi-modal data feature expression method sample to obtain a multi-modal data feature expression method processing model; obtaining a multi-modal data feature expression method to be processed, inputting the multi-modal data feature expression method to be processed into the multi-modal data feature expression method processing model, and obtaining a feature expression of the multi-modal data feature expression method to be processed, thereby improving sample utilization and feature expression accuracy. The patent with publication number CN114359656A proposes a melanoma image recognition method based on self-supervised contrast learning, including the following steps: step 1, obtaining labeled medical image data and unlabeled medical image data; step 2, inputting the unlabeled medical image data into a self-supervised contrast learning network structure to train network parameters; step 3, inputting the labeled medical image data into a melanoma image recognition method network model to train multiple network models; step 4, calculating whether each labeled medical image data corresponds to a melanoma image recognition method through the multiple network models. The patent with publication number CN115019923A provides an electronic medical record data pre-training method based on contrast learning, including the following steps: establishing a medical record sample database using different categories and different attributes for each electronic medical record data pre-training method; pre-processing the data in the database using data classification and data normalization; sampling positive and negative sample data for each data based on attribute distance; training all data and their corresponding positive and negative sample data one by one to obtain feature vectors and predicted values for each data; then calculating a loss function based on the feature vectors and predicted values of all data, and updating the weight matrix of the neural network model based on the loss function; determining whether the training of the neural network model has reached a stopping condition, if so, the training is ended and the feature vectors and predicted values of each data are output; if not, the data is trained for a new round until the stopping condition of the neural network model training is reached.

[0006] In summary, contrastive learning methods can be used to analyze and process EHR data. The analysis process can be roughly divided into two steps: (1) using the EHR data training set to train a deep network model with feature extraction capabilities (pre-training model); (2) using the deep network model in the first step to extract feature representation vectors of samples in the downstream task dataset, and using this feature representation vector to perform specific medical tasks, such as EHR image classification, EHR image segmentation, EHR image object detection, and clinical outcome prediction.

[0007] Furthermore, the contrastive learning methods for obtaining medical pre-training models can be divided into single-modal contrastive learning methods for EHR image data or EHR text data, and multimodal contrastive learning methods that use both EHR image data and EHR text data. Among them, the single-modal contrastive learning method using a single form of EHR data cannot fully utilize the massive amount of unlabeled EHR data and has its own limitations. In the multimodal medical contrastive learning method, there is often a problem that the feature vector cannot provide sufficient discriminant information during the training of the multimodal contrastive learning deep network model. For example, the case images and case text descriptions of the same part in the EHR data may be very similar, which results in the feature representation vector extracted by the model being unable to provide sufficient discriminant information for downstream tasks to distinguish between the two similar diseases. Therefore, the key issues to improve the utilization of EHR data and enhance the accuracy of tasks such as EHR image classification, EHR image segmentation, EHR image target detection, and clinical outcome prediction are: (1) how to design an EHR data analysis method and system using multimodal EHR data; (2) how to make the feature representation vector of EHR data have more sufficient discriminant information. Summary of the Invention

[0008] The purpose of this invention is to provide an EHR data analysis method and system based on augmented discriminant information. This method and system can obtain feature representation vectors of EHR data that contain more discriminant information, thereby improving the sample utilization rate of EHR data and enhancing the accuracy of downstream medical tasks such as EHR image classification, EHR image segmentation, EHR image object detection, and clinical outcome prediction.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] The present invention provides an EHR data analysis method and system based on augmented discriminant information. The method includes three main links, namely, a link for constructing and processing a multimodal EHR data set, a link for constructing a multimodal comparative learning deep network model, initializing, training and saving the optimal model, and a link for performing downstream tasks. Among them, the link for constructing, initializing, training and saving the optimal model of a multimodal comparative learning deep network model generally includes constructing and initializing the model, extracting sample feature vectors, reconstructing sample feature vectors, calculating the loss of each module and the total loss, training the model, and saving the optimal model. The main parameters of the present invention include: temperature hyperparameters, batch size, optimizer parameters, and the number of training rounds of the model, wherein the temperature hyperparameter is used to adjust the similarity measure between samples during the comparative learning process; the batch size is used to control the number of samples sampled simultaneously during the training model process; the optimizer parameters include optimizer type, initial learning rate, and learning rate decay method, which are used to set the optimizer in the training model process; the number of training rounds of the model is used to set the end condition of model training. The method comprises the following steps:

[0011] S1. Construct a multimodal EHR dataset and extract paired image and text data samples from the multimodal EHR dataset to form initial sample pairs;

[0012] S2, performing augmentation preprocessing on the image and text data samples in the initial sample pair in step S1 to obtain image augmented samples and text augmented samples;

[0013] S3. Build and initialize each network module in the multimodal contrastive learning deep network model;

[0014] S4, using the image augmented samples and text augmented samples obtained in step S2 to train the multimodal contrastive learning deep network model in step S3 through the Adam optimizer and multiple rounds of small batch iterative optimization, and saving the optimal multimodal contrastive learning deep network model;

[0015] S5. Use the optimal multimodal contrastive learning deep network model in step S4 to extract the sample feature representation vector of the downstream task dataset, and apply the feature representation vector to downstream medical tasks, such as EHR image classification, EHR image segmentation, EHR image target detection, and clinical outcome prediction.

[0016] Furthermore, the step S1 specifically includes the following steps:

[0017] S11, in the EHR data analysis method based on augmented discriminative information, a multi-modal contrast learning deep network model is designed to extract EHR data sample features. In order to more fully train the multi-modal contrast learning deep network model, the present application first collects EHR image data and EHR text data, and preliminarily constructs a multi-modal EHR data set with large data volume and rich data content. The EHR text information corresponding to the EHR image in the data set includes text reports, case descriptions, plans, medical histories, diagnosis results, etc.

[0018] S12, the preliminarily constructed multi-modal EHR data set is cleaned. Positive and clear X-ray images are selected from the constructed data set, and text data paired with the images, such as text reports, medical notes, and case descriptions, are extracted. The selected EHR image data and corresponding EHR text data are used as initial sample pairs in the model training process, denoted as Where n is the total number of samples, represents the i-th image and text sample pair, represents the i-th image sample, represents the i-th text sample.

[0019] Further, the step S2 specifically comprises the following steps:

[0020] S21, since the multi-modal EHR data set has a large data volume, the method adopts the method of training model parameters with small batches of data commonly used in deep learning. Specifically, the batch size in the training model process is set to b according to the performance of the graphics card or server. Thus, the data set is divided into s mini-batch data to train the network model in steps, where s=n / b, i.e. each round of training is divided into s steps of iteration, and b pairs of EHR image and text augmented samples are used in each training process.

[0021] S22, the small batch image samples x v in step S21 are preprocessed by image augmentation to obtain image augmented samples The purpose of image augmentation preprocessing is to increase the difference between the initial EHR image samples, so that the multi-modal contrast learning deep network model can learn the essential features of the case between the images with differences; and to convert the image data into tensor data that can be input into the model. The image augmentation preprocessing operations include random cropping, scaling, horizontal mirroring, random color jittering, random grayscale transformation, data format conversion, etc.

[0022] S23, the small batch text samples x u in step S21 are preprocessed by text augmentation to obtain text augmented samples The goal of text augmentation preprocessing is to filter out text content containing important information and convert text data into tensor data that can be input into the model. This includes operations such as tagging text samples, retaining content with a set theme, truncating or expanding text samples, and converting data formats.

[0023] Furthermore, the multimodal contrastive learning deep network model in step S3 includes six network modules, namely, an image backbone network, a text backbone network, an image projection head network, a text projection head network, an image feature amplifier network, and a text feature amplifier network. Step S3 includes the following steps:

[0024] S31. Build and initialize the image backbone network f v (·)(Image encoder network). Specifically, the image backbone network f v The main body of (·) is the ResNet-50 model, commonly used in the field of computer vision. The difference is that the last layer of the ResNet-50 model is a linear neural network used for classification. In this method, the linear classification layer is replaced with an identity mapping neural network. Furthermore, this method selects an untrained ResNet-50 model and randomly initializes its parameters after model construction.

[0025] S32. Build and initialize the text backbone network f u (·)(Text Encoder Network). Specifically, the text backbone network f u (·) is the standard BERT model in the field of natural language processing. This method selects an untrained BERT model and randomly initializes its parameters after building the model.

[0026] S33. Build and initialize the image projection head network (Multi-layer Perceptron Network). Specifically, the image projection head network It is a multi-layer nonlinear perceptron network model, including linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the image projection head network;

[0027] S34. Build and initialize the text projection head network (Multi-layer Perceptron Network). Specifically, the text projection head network It is a multi-layer nonlinear perceptron network model, including linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the text projection head network;

[0028] S35. Build and initialize the image feature amplifier network (Multi-layer Perceptron Network). Specifically, the Image Feature Amplifier Network It is a multi-layer nonlinear perceptron network model, including linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and is the trainable parameter in the image feature amplifier network;

[0029] S36. Build and initialize the text feature amplifier network (Multi-layer Perceptron Network). Specifically, the Text Feature Amplifier Network It is a multi-layer nonlinear perceptron network model, including linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the text feature amplifier network.

[0030] S37: The multimodal contrastive learning deep network model uses the Adam optimizer. Initializing the Adam optimizer involves setting the initial learning rate and weight decay rate. Also, setting a stopping condition for the model training process. In this method, a fixed number of training rounds is used, and model training stops after reaching this number of training rounds.

[0031] Furthermore, in step S4, the image and text augmented samples in step S2 are used to train the multimodal contrastive learning deep network model in step S3. Taking single-step training as an example, the following specific steps are included:

[0032] S41. Augment samples of small batch images Text augmentation samples Input into the image backbone network f v (·) with the text backbone network f u (·) to obtain the sample feature representation Where b is the batch size, represents the feature representation of the i-th pair of augmented samples, represents the feature representation of the i-th image augmentation sample, Represents the feature representation of the i-th text augmentation sample.

[0033] S42, characterize the small batch image sample features h v, text sample feature representation h u Input to the image projection head network respectively Header network with text projection In the example, we get the sample feature embedding Where b is the batch size, represents the feature embedding of the i-th pair of samples, represents the feature embedding of the i-th image sample, represents the feature embedding of the i-th text sample.

[0034] Specifically, to enable the model to learn more abstract information from the samples, contrastive learning methods often project the learned high-dimensional feature representations into a low-dimensional feature embedding space through a projection head network. For example, in a facial image dataset, in a higher-dimensional space, the feature representations mostly contain information about features such as image color, image boundaries, and image texture; in a lower-dimensional embedding space, the feature embedding vectors mostly contain abstract information such as facial features, facial expressions, skin color, and hair color. Similarly, in this method, to enable the multimodal contrastive learning deep network model to learn more abstract information from the samples, we use a projection head network to project the feature representations into a relatively low-dimensional space and calculate the bidirectional contrast loss.

[0035] S43. Generate small batches of random Gaussian noise vectors Where b is the batch size, e i Represents the i-th Gaussian noise vector. With a single random Gaussian noise vector e i For example, e i It is divided into two parts, upper and lower, and is recorded as Reorganization and Generate image reconstruction feature embedding Reorganization and Generate text reconstruction feature embedding

[0036] S44. Embedding features of small batch images Text reconstruction feature embedding Input into the image feature amplifier network respectively and text feature amplifier network In the above example, we can get the reconstructed sample feature representation Where b is the batch size, represents the feature representation of the i-th pair of reconstructed samples, represents the feature representation of the i-th image reconstruction, Represents the i-th text reconstruction feature representation.

[0037] S45. Use small batch image sample feature embedding z v , text sample feature embedding z u Calculate the two-way contrast loss, and the loss definition is as shown in formula (1) and (2):

[0038]

[0039]

[0040] in Indicates the contrast loss of image samples as anchor points and text samples as positive and negative examples; It represents the contrast loss of text samples as anchor points and image samples as positive and negative examples; τ is the temperature hyperparameter.

[0041] Reconstructing feature representations using mini-batches of images v , text reconstruction feature representation u Calculate the bidirectional augmented information loss. In the present invention, we assume that the newly generated reconstructed sample feature representation R obeys the T distribution. The heavier tail of the T distribution can make the similarity between most negative examples and the anchor point lower than the average similarity, that is, the negative examples that are very similar to the anchor point play a more important role in the model training process. The starting point of this assumption is: during the model training process, negative examples close to the anchor point are very likely to be false negative examples (the same semantic class as the anchor point). Excluding false negative examples of the same semantic class will potentially cause the model's "semantic class ambiguity" problem, thereby resulting in poor generalization of the multimodal contrastive learning deep network model in downstream tasks. The specific definition of bidirectional augmented information loss is shown in Equations (3) and (4):

[0042]

[0043]

[0044] in The image reconstruction feature representation is the anchor point, and the text reconstruction feature representation is the augmented information loss of positive and negative examples; It represents the text reconstruction feature representation as the anchor point and the image reconstruction feature representation as the augmented information loss of positive and negative examples.

[0045] Calculate the total training loss for a single step To train network parameters, the total training loss The definition is as follows:

[0046]

[0047] where w is an increasing coefficient. This is because in the early stage of training, the reconstructed sample feature representation R contains task-irrelevant information or noise, so w is set to constrain the misleading of bidirectional augmented information loss in the early stage of model training.

[0048] S46, repeat steps S41-S45 to train the multi-modal contrast learning deep network model to the end, save the optimal multi-modal contrast learning deep network model backbone network part (image backbone network f v (·), text backbone network f u (·)).

[0049] Further, the step S5 specifically comprises the following steps:

[0050] S51, perform the augmentation preprocessing operation of step S2 on the data set in the downstream task to obtain image and text augmented samples of the downstream task data set.

[0051] S52, construct a downstream task module network g t (·) according to the downstream task, such as constructing and initializing a linear classification layer neural network in the EHR image classification task, which can be represented as where is the linear layer parameter.

[0052] S53, use the optimal multi-modal contrast learning deep network model saved in step S46 to perform the feature extraction operation of step S41 on the image and text augmented samples in the downstream task data set in step S51, to obtain image and text augmented sample feature representations of the downstream task data set.

[0053] S54, input the image and text augmented sample feature representations of the downstream task data set in step S53 to the module g t (·) of the downstream task in step S52. Taking the EHR image classification task as an example, input the image augmented sample feature representation to the linear classification layer g t (·) to obtain the classification result. The EHR image segmentation, EHR image object detection, and clinical outcome prediction downstream tasks are the same, and the steps S51-S54 are performed to obtain the required results. The difference between different downstream tasks lies in the difference in the module network g t (·) of the downstream task in step S52.

[0054] The present invention also provides an EHR data analysis system based on augmented discriminant information, which is used to implement the above-mentioned EHR data analysis method based on augmented discriminant information, including a computer processor, memory and graphics processor; a multimodal EHR data storage unit; a multimodal EHR data preprocessing unit; a multimodal contrastive learning deep network model training unit; a multimodal contrastive learning deep network model storage unit and a downstream task unit.

[0055] Furthermore, the multimodal EHR data storage unit stores the multimodal EHR dataset and downstream medical task dataset constructed in step S1, and loads them into the computer memory; the multimodal EHR data preprocessing unit extracts batch-sized multimodal EHR samples from the memory in steps, executes step S2 to perform image and text augmentation preprocessing to obtain EHR image augmented samples and EHR text augmented samples, and loads them into the graphics processor; the multimodal contrast learning deep network model training unit uses the EHR image augmented samples and EHR text augmented samples in the graphics processor to execute steps S3-S4 to obtain the optimal multimodal contrast learning deep network model and store it in the multimodal contrast learning deep network model storage unit; based on the optimal model stored in the multimodal contrast learning deep network model storage unit, the downstream task unit executes step S5 in the graphics processor to obtain the required results of the downstream medical task; the specific data processing and calculation work in all units is completed by the computer processor and graphics processor.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] 1. The EHR data analysis method based on augmented discriminant information designed in the present invention can effectively improve the utilization rate of EHR multimodal data and the quality of EHR multimodal data feature representation vectors. Specifically, this method uses unlabeled EHR image data and EHR text data to train the designed multimodal contrastive learning deep network model. The use of naturally existing paired EHR image and text data as self-supervisory signals can effectively improve the utilization rate of EHR data. At the same time, image and text data can constrain each other to improve the accuracy of feature representation information.

[0058] 2、The EHR data analysis method based on augmented discriminative information designed by the application can effectively add discriminative information to the feature representation vector of the EHR multi-modal data. Specifically, the application designs a feature embedding reconstruction method for supplementing discriminative information and a feature amplifier module. Compared with the traditional contrast learning method using multi-modal EHR data, the method proposed by the application can make the feature representation vector extracted by the model contain more high-quality discriminative information. In addition, the network structure parameter of the newly added feature amplifier module is small, and it can add more distribution information to the feature representation vector while increasing a small amount of additional calculation, which significantly improves the quality of the discriminative information in the feature representation vector.

[0059] 3、The EHR data analysis method and system based on augmented discriminative information designed by the application can effectively process downstream medical tasks, use sample feature representation vectors containing more discriminative information, and effectively improve the accuracy of EHR image classification, EHR image segmentation, EHR image target detection and clinical result prediction. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The multi-modal contrast learning deep network model structure diagram in the EHR data analysis method based on augmented discriminative information.

[0061] Figure 2 The downstream task module structure diagram in the EHR data analysis method based on augmented discriminative information.

[0062] Figure 3 The computer-implemented system structure diagram of the EHR data analysis method based on augmented discriminative information.

[0063] Figure 4 The flowchart of the EHR data analysis method based on augmented discriminative information. DETAILED DESCRIPTION

[0064] The EHR data analysis method based on augmented discriminative information is implemented by a computer program, Figure 3 The computer-implemented system structure diagram is shown, wherein the "graphics processor" only represents the type of computing resources, which can be a separate graphics card supporting GPU computing, or a local server, cloud server, etc. supporting GPU computing. The following will be described in detail Figure 1 、 2 The model structure diagram and the method flowchart in Figure 4 The technical solutions in the embodiments of the application are specifically and in detail explained. The implementation mainly includes the following key contents:

[0065] S1, construct a multi-modal EHR dataset, extract paired image, text data samples in the multi-modal EHR dataset to form an initial sample pair, and the specific steps include the following steps:

[0066] S11, in order to fully train the multi-modal medical contrast learning deep network model, the present application preliminarily constructs a multi-modal EHR dataset. The new dataset includes MIMIC-CXR version 2 database, OpenI-IU public database, MedPix database. Specifically, the MIMIC-CXR version 2 database contains 377110 chest X-ray images and 227835 related radiology text reports, and about 217000 paired image and text pairs can be obtained; the OpenI-IU public database contains 3996 radiology reports and 8121 related chest X-ray images, and about 3640 paired image and text pairs can be selected; the MedPix database is an online public database, which contains about 12000 patient cases, 9000 topics and 59000 X-ray images and corresponding medical text information such as case description, scheme, medical history, diagnosis result and the like.

[0067] S12, clean the preliminarily constructed multi-modal EHR dataset. Select clear X-ray images from the constructed dataset, and extract paired text data such as text reports, medical notes and case descriptions. The selected image and text sample pairs are used as initial sample pairs in the model training process. The initial sample pair specifically includes 245600 EHR image and text sample pairs, denoted as where 245600 is the total amount of samples, represents the i-th image and text sample pair, represents the i-th image sample, represents the i-th text sample.

[0068] S2, the image and text samples of the initial sample pair in step S1 are augmented and preprocessed to obtain image augmented samples and text augmented samples, and the specific steps include the following steps:

[0069] S21, since the data volume of the multi-modal EHR dataset is large, the present application adopts the method of training model parameters with small batch data commonly used in deep learning method. In the present application, the batch size in the model training process is set to 128, wherein the batch size is a hyperparameter set according to the performance of the graphics card or server computer. Therefore, the dataset is divided into 1919 mini-batch data to train the model network in steps, that is, each training is divided into 1919 steps, and 128 EHR image and text samples are used in each training process.

[0070] S22, the small batch image sample xv Perform image augmentation preprocessing to obtain image augmentation samples Specifically, for each image sample, we first cropped an image block of random size and position from the original image, and then scaled the cropped image block to a standard image of size 256×256; performed a horizontal mirror operation with a parameter of 0.5, a random color jitter operation with a parameter of 0.8, and a random grayscale transformation operation with a parameter of 0.2 on the standard image; and finally converted the processed image into a tensor data with a dimension of [128, 3, 256, 256]. For use by the image encoder network.

[0071] S23, the small batch text sample x in step S21 u Perform text augmentation preprocessing to obtain text augmentation samples Specifically, for text samples, we first use the default English tagger in the CoreNLP library to tag the text samples, retaining the title (Caption), results (Findings), impression (Impression) and words that appear more than twice in the text; the processed text samples are truncated or expanded (randomly repeating words) to make their length 256, and finally the processed text is converted into tensor data with a dimension of [128, 256] For use by the text encoder network.

[0072] S3. Construct and initialize each module in the multimodal contrastive learning deep network model. Its overall structure is as shown in the attached figure. Figure 1 As shown, the specific steps include:

[0073] S31. Build and initialize the image backbone network f v (·)(Image encoder network). Specifically, the image backbone network f v The main body of (·) is the ResNet-50 model, commonly used in the field of computer vision. The difference is that the last layer of the ResNet-50 model is a linear neural network used for classification. In this method, the linear classification layer is replaced with an identity mapping neural network. Furthermore, this method selects an untrained ResNet-50 model and randomly initializes its parameters after model construction. The input of the ResNet-50 model is a mini-batch of image samples of dimension [128, 3, 256, 256], where 128 is the batch size, 3 is the image color channel, and [256, 256] is the image size. Its output is an image feature representation tensor of dimension [128, 2048].

[0074] S32. Build and initialize the text backbone network f u (·)(Text Encoder Network). Specifically, the text backbone network fu (·) is a standard BERT model in natural language processing. This method uses an untrained BERT model and randomly initializes its parameters after model construction. The BERT model input is a mini-batch of text samples of dimension [128, 256], and the output is a text feature representation tensor of dimension [128, 256 × 768], where 768 represents the vector dimension of each token in the output tensor.

[0075] S33. Build and initialize the image projection head network (Multi-layer Perceptron Network). Specifically, the image projection head network The multi-layer nonlinear perceptron network, i.e. linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, can be expressed as in and are the trainable parameters in the image projection head network, and the input dimension, intermediate hidden layer dimension, and output dimension are set to [2048, 2048, 128].

[0076] S34. Build and initialize the text projection head network (Multi-layer Perceptron Network). Specifically, the text projection head network It is a multi-layer nonlinear perceptron network, namely linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the text projection head network, and the input dimension, intermediate hidden layer dimension, and output dimension are set to [256×768, 2048, 128].

[0077] S35. Build and initialize the image feature amplifier network (Multi-layer Perceptron Network). Specifically, the Image Feature Amplifier Network It is a multi-layer nonlinear perceptron network, namely linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the image feature amplifier network. The input dimension, intermediate hidden layer dimension, and output dimension are set to [128, 2048, 2048].

[0078] S36. Build and initialize the text feature amplifier network (Multi-layer Perceptron Network). Specifically, the Text Feature Amplifier Network It is a multi-layer nonlinear perceptron network, namely linear layer, BatchNorm normalization layer, ReLU nonlinear activation layer, linear layer and BatchNorm normalization layer, which can be expressed as in and are the trainable parameters in the text feature amplifier network. The input dimension, intermediate hidden layer dimension, and output dimension are set to [128, 2048, 2048].

[0079] S37, Multimodal Contrast Learning Deep Network Model uses Adam optimizer as a whole, and initializes Adam optimizer to set its initial learning rate to 1e -4 , the weight decay rate is set to 1e -6 . Set the number of model training rounds to 300 rounds, and stop training the model after reaching the number of training rounds.

[0080] S4. Use the image and text augmentation samples from step S2 to train the multimodal contrastive learning deep network model from step S3 for 300 rounds using the Adam optimizer and multiple rounds of mini-batch iterative optimization. Save the optimal multimodal contrastive learning deep network model. Each round of iterative training is divided into 1919 steps, and each step of training uses 128 pairs of image and text augmentation samples. Taking single-step training as an example, the specific steps include the following:

[0081] S41. Augment the small batch (128 pairs) of images Text augmentation samples Input into the image backbone network f v (·) with the text backbone network f u (·) to obtain the sample feature representation in represents the i-th pair of augmented sample feature representations, represents the feature representation of the i-th image augmentation sample, In the actual training process, the input small batch image sample tensor dimension is [128,3,256,256], and the input small batch text sample tensor dimension is [128,256]. They are respectively passed through the image backbone network f v (·) with the text backbone network f u (·) Extract feature information and obtain the image augmentation sample feature representation h with tensor dimension [128,2048] v , the text augmentation sample feature representation h with tensor dimension [128,256×768] u .

[0082] S42, represent the features of small batch (128 pairs) image samples h v , text sample feature representation hu Input to the image projection head network respectively Header network with text projection In the example, we get the sample feature embedding in represents the i-th pair of sample feature embeddings, represents the feature embedding of the i-th image sample, represents the feature embedding of the i-th text sample.

[0083] Specifically, to enable the model to learn more abstract information about the samples, contrastive learning methods often project the learned high-dimensional feature representations into a low-dimensional feature embedding space through a projection head network. For example, in a facial image dataset, in the higher-dimensional space, the feature representations contain mostly information about features such as color, image boundaries, and image texture; whereas in the lower-dimensional embedding space, the feature embedding vectors contain mostly abstract information such as facial features, facial expressions, skin color, and hair color.

[0084] Similarly, in this method, in order to enable the multimodal contrastive learning deep network model to learn more abstract information in the sample, we use the projection head network to project the feature representation into a relatively low-dimensional space to calculate the bidirectional contrast loss. Input image feature representation h v The tensor dimension is [128, 2048], and the output image feature embedding z v The tensor dimension is [128,128]; the text projection head network Input text feature representation h u The tensor dimension is [128, 256 × 768], and the output text feature embedding z u The tensor dimension is [128,128].

[0085] S43, as attached Figure 1 Generate a small batch (128) of random Gaussian noise vectors as shown in The tensor dimension is [128,128], where e i Represents the i-th Gaussian noise vector. With a single random Gaussian noise vector e i For example, e i It is divided into two parts, upper and lower, and is recorded as (e i The vector part with dimension [1:64] in (e i The vector part with dimension [65:128] in the . and Generate image reconstruction feature embedding That is and The vectors with dimensions [65:128] are partially concatenated to obtain

[128] -dimensional Reorganization and Generate text reconstruction feature embedding That is and The vectors with dimensions [1:64] are partially concatenated to obtain

[128] -dimensional

[0086] S44, embedding small batch (128 pairs) image reconstruction features Text reconstruction feature embedding Input into the image feature amplifier network respectively and text feature amplifier network In the above example, we can get the reconstructed sample feature representation in represents the i-th pair of reconstructed sample feature representations, represents the feature representation of the i-th image reconstruction, Represents the i-th text reconstruction feature representation. Among them, the image feature amplifier network Input image reconstruction feature embedding The tensor dimension is [128,128], and the output image reconstruction feature representation r v The tensor dimension is [128,2048]. Text Feature Amplifier Network Input text reconstruction feature embedding The tensor dimension is [128,128], and the output text reconstruction feature representation r u The tensor dimension is [128, 2048].

[0087] S45. Use small batches (128 pairs) of image sample features to embed z v , text sample feature embedding z u Calculate the two-way contrast loss, and the loss definition is as shown in formula (1) and (2):

[0088]

[0089]

[0090] in Indicates the contrast loss of image samples as anchor points and text samples as positive and negative examples; The text samples are anchor points, and the image samples are the contrast losses of positive and negative examples; τ is a temperature hyperparameter, which is set to 0.5 in this invention.

[0091] Reconstruct feature representation using small batches (128 pairs) of imagesv , text reconstruction feature representation u Calculate the bidirectional augmented information loss. In the present invention, we assume that the newly generated reconstructed sample feature representation R obeys the T distribution. The heavier tail of the T distribution can make the similarity between most negative examples and the anchor point lower than the average similarity, that is, the negative examples that are very similar to the anchor point play a more important role in the model training process. The starting point of this assumption is: during the model training process, negative examples close to the anchor point are very likely to be false negative examples (the same semantic class as the anchor point). Excluding false negative examples of the same semantic class will potentially cause the model's "semantic class ambiguity" problem, thereby resulting in poor generalization of the multimodal contrastive learning deep network model in downstream tasks. The specific definition of bidirectional augmented information loss is shown in Equations (3) and (4):

[0092]

[0093]

[0094] in The image reconstruction feature representation is the anchor point, and the text reconstruction feature representation is the augmented information loss of positive and negative examples; It represents the text reconstruction feature representation as the anchor point and the image reconstruction feature representation as the augmented information loss of positive and negative examples.

[0095] Calculate the total training loss for a single step To train network parameters, the total training loss The definition is as follows:

[0096]

[0097] Where w is a gradually increasing coefficient, which gradually increases to After 100 rounds of training, w is constant. This is because in the early stages of training, the reconstructed sample feature representation R contains information or noise that is irrelevant to the task, so it is necessary to set w to constrain the misleading effects that may be caused by the bidirectional augmentation information loss in the early stages of model training.

[0098] S46, repeat steps S41-S45, and train the multimodal contrastive learning deep network model for 300 rounds. At the end of the training, save the backbone network of the optimal multimodal contrastive learning deep network model (image backbone network f v (·), text backbone network f u (·)).

[0099] S5, as attached Figure 2As shown, the optimal multimodal contrastive learning deep network model in step S4 is used to extract the sample feature representation vector of the downstream task dataset, and the feature representation vector is applied to downstream medical tasks such as EHR image classification, EHR image segmentation, EHR image target detection, clinical outcome prediction, etc., specifically including the following steps:

[0100] S51 , performing augmentation preprocessing operations of steps S21 to S23 on the dataset in the downstream task to obtain image and text augmented samples of the downstream task dataset.

[0101] S52, construct downstream task module network g according to downstream tasks t (·), such as constructing and initializing a linear classification layer neural network in the EHR image classification task, which can be expressed as in are the linear layer parameters. Its input dimension is [128, 2048], and its output dimension is [128, c], where 128 is the batch size, 2048 is the sample feature representation size, and c is the number of sample categories.

[0102] S53. Use the optimal multimodal contrastive learning deep network model saved in step S46 to perform the feature extraction operation of step S41 on the image and text augmentation samples in the downstream task data set in step S51 to obtain feature representations of the image and text augmentation samples in the downstream task data set.

[0103] S54, input the image and text augmented sample feature representation of the downstream task data set in step S53 into the module g of the downstream task in step S52 t (·). Taking the EHR image classification task as an example, the image augmentation sample feature representation is input into the linear classification layer g t (·) to obtain the classification result. Similarly, downstream tasks such as EHR image segmentation, EHR image target detection, and clinical outcome prediction execute steps S51-S54 to obtain the required results. The difference in the implementation of different downstream tasks lies in the module network g of the downstream task in step S52. t (·) is different.

[0104] As attached Figure 3As shown, an EHR data analysis system based on augmented discriminant information includes a computer processor, memory and graphics processor; a multimodal EHR data storage unit; a multimodal EHR data preprocessing unit; a multimodal contrastive learning deep network model training unit; a multimodal contrastive learning deep network model storage unit and a downstream task unit. The multimodal EHR data storage unit stores the multimodal EHR data set and downstream medical task data set constructed in step S1, and loads them into the computer memory; the multimodal EHR data preprocessing unit extracts batch-sized multimodal EHR samples from the memory in steps, executes step S2 to perform image and text augmentation preprocessing to obtain small batches of EHR image augmented samples and EHR text augmented samples, and loads them into the graphics processor; the multimodal contrastive learning deep network model training unit uses the EHR image augmented samples and EHR text augmented samples in the graphics processor to execute steps S3-S4 to obtain the optimal multimodal contrastive learning deep network model and store it in the multimodal contrastive learning deep network model storage unit; based on the optimal model stored in the multimodal contrastive learning deep network model, the downstream task unit executes step S5 in the graphics processor to obtain the required results of the downstream medical tasks, such as EHR image classification results, EHR image segmentation results, EHR image target detection results, and clinical prediction results of EHR images and EHR texts; the specific data processing and calculation work in all units is completed by the computer processor and graphics processor.

[0105] Finally, the foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for analyzing EHR data based on augmented discriminant information, characterized in that: The following steps are involved: S1. Construct a multimodal EHR dataset and extract paired image and text data samples from the multimodal EHR dataset to form initial sample pairs; S2, performing augmentation preprocessing on the image and text data samples in the initial sample pair in step S1 to obtain image augmented samples and text augmented samples; S3. Build and initialize each network module in the multimodal contrastive learning deep network model; The step S3 specifically includes the following steps: S31. Build and initialize the image backbone network , that is, image encoder network; image backbone network The main body of the method is the ResNet-50 model, the difference is that the last layer of the linear neural network for classification in the ResNet-50 model is replaced by the identity mapping layer neural network; In addition, this method selects an untrained ResNet-50 model and randomly initializes its parameters after building the model; S32. Build and initialize the text backbone network , that is, text encoder network; text backbone network It is a Bert model; this method selects an untrained Bert model and randomly initializes its parameters after building the model; S33. Build and initialize the image projection head network , that is, multi-layer perceptron network; image projection head network It is a multi-layer nonlinear perceptron network model, including a linear layer, a BatchNorm normalization layer, a ReLU nonlinear activation layer, a linear layer and a BatchNorm normalization layer, expressed as ,in , and are the trainable parameters in the image projection head network; S34. Build and initialize the text projection head network , that is, multi-layer perceptron network; text projection head network It is a multi-layer nonlinear perceptron network model, including a linear layer, a BatchNorm normalization layer, a ReLU nonlinear activation layer, a linear layer and a BatchNorm normalization layer, expressed as ,in , and are the trainable parameters in the text projection head network; S35. Build and initialize the image feature amplifier network , that is, multi-layer perceptron network; image feature amplifier network It is a multi-layer nonlinear perceptron network model, including a linear layer, a BatchNorm normalization layer, a ReLU nonlinear activation layer, a linear layer and a BatchNorm normalization layer, expressed as ,in , and is the trainable parameter in the image feature amplifier network; S36. Build and initialize the text feature amplifier network , that is, multi-layer perceptron network; text feature amplifier network It is a multi-layer nonlinear perceptron network model, including a linear layer, a BatchNorm normalization layer, a ReLU nonlinear activation layer, a linear layer and a BatchNorm normalization layer, expressed as ,in , and It is the trainable parameter in the text feature amplifier network; S37, Multimodal Contrast Learning Deep Network Model uses Adam optimizer as a whole. Initializing Adam optimizer includes setting the initial learning rate and weight decay rate. At the same time, setting the number of training rounds of the model, and stopping the training model after reaching the number of training rounds. S4, using the image augmented samples and text augmented samples obtained in step S2 to train the multimodal contrastive learning deep network model in step S3 through the Adam optimizer and multiple rounds of small batch iterative optimization, and saving the optimal multimodal contrastive learning deep network model; The step S4 specifically includes the following steps: S41. Augment samples of small batch images , text augmentation samples Input into the image backbone network respectively With text backbone network In the above example, we get the sample feature representation ,in is the batch size, Indicates the Characterize the features of the augmented samples. Indicates the Image augmentation sample feature representation, Indicates the Feature representation of text augmentation samples; S42. Characterize the features of small batch image samples , text sample feature representation Input to the image projection head network respectively Header network with text projection In the example, we get the sample feature embedding ,in is the batch size, Indicates the Embedding sample features, Indicates the Image sample feature embedding, Indicates the Text sample feature embedding; S43. Generate small batches of random Gaussian noise vectors ,in is the batch size, Indicates the Gaussian noise vectors, with a single random Gaussian noise vector For example, It is divided into two parts, upper and lower, and is recorded as 、 ; Reorganization and Generate image reconstruction feature embedding , reorganization and Generate text reconstruction feature embedding ; S44. Embedding features of small batch images , text reconstruction feature embedding Input into the image feature amplifier network respectively and text feature amplifier network In the above example, we can get the reconstructed sample feature representation ,in is the batch size, Indicates the Characterize the reconstructed sample features. Indicates the Image reconstruction feature representation, Indicates the A text reconstruction feature representation; S45. Using small batches of image sample feature embedding , text sample feature embedding Calculate the two-way contrast loss, and the loss definition is as shown in formula (1) and (2): ， （1） ， （2） in Indicates the contrast loss of image samples as anchor points and text samples as positive and negative examples; The contrast loss represents text samples as anchor points and image samples as positive and negative examples; is the temperature hyperparameter; Reconstructing feature representations using mini-batches of images , text reconstruction feature representation Calculate the bidirectional augmentation information loss, which is defined as follows: ， （3） ， （4） in The image reconstruction feature representation is the anchor point, and the text reconstruction feature representation is the augmented information loss of positive and negative examples; The feature representation of text reconstruction is the anchor point, and the feature representation of image reconstruction is the augmented information loss of positive and negative examples; Calculate the total training loss for a single step To train network parameters, the total training loss The definition is as follows (5): ， （5） in is an increasing coefficient; S46, repeat steps S41-S45, train the multimodal contrastive learning deep network model until the end, and save the image backbone network in the optimal multimodal contrastive learning deep network model and text backbone network ; S5. Use the optimal multimodal contrastive learning deep network model in step S4 to extract the sample feature representation vector of the downstream task dataset, and apply the feature representation vector to the downstream medical task.

2. The EHR data analysis method based on augmented discriminant information according to claim 1, characterized in that: The step S1 specifically includes the following steps: S11. First, collect EHR image data and EHR text data to preliminarily build a multimodal EHR dataset with large data volume and rich data content; S12. Clean the preliminarily constructed multimodal EHR dataset, select frontal, clear X-ray images from the constructed dataset, and extract the text data of the image pairings. Use the selected EHR image data and the corresponding EHR text data as the initial sample pair in the model training process, denoted as , where n is the total sample size, Indicates the For image and text sample pairs, Indicates the image samples, Indicates the Text samples.

3. The EHR data analysis method based on augmented discriminant information according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21. The method of training model parameters with small batches of data commonly used in deep learning is adopted. The batch size in the training model process is set to b. The dataset is divided into s small batches of data to train the network model step by step, where s = n / b, that is, each round of training is divided into s steps of iteration, and b EHR image and text augmentation samples are used in each training step; S22, the small batch image samples in step S21 Perform image augmentation preprocessing to obtain image augmentation samples ; Among them, the image augmentation preprocessing operations include random cropping, scaling, horizontal mirroring, random color jittering, random grayscale transformation, and data format conversion operations; S23, the small batch text samples in step S21 Perform text augmentation preprocessing to obtain text augmentation samples ; Among them, the text augmentation preprocessing operations include text sample tagging, retaining the content of the set topic, truncating or expanding the text sample, and converting the data format.

4. The EHR data analysis method based on augmented discriminant information according to claim 1, characterized in that: The step S5 specifically includes the following steps: S51, performing the augmentation preprocessing operation of step S2 on the dataset in the downstream task to obtain image and text augmented samples of the downstream task dataset; S52. Build downstream task module network based on downstream tasks , in the EHR image classification task, a linear classification layer neural network is constructed and initialized, which can be expressed as ,in is the linear layer parameter; S53, using the optimal multimodal contrastive learning deep network model saved in step S46, performing the feature extraction operation of step S41 on the image and text augmentation samples in the downstream task dataset in step S51, to obtain feature representations of the image and text augmentation samples in the downstream task dataset; S54: Input the image and text augmented sample feature representations of the downstream task data set in step S53 into the downstream task module in step S52 ; In the EHR image classification task, the image augmentation sample feature representation is input into the linear classification layer The classification results are obtained; similarly, the downstream tasks of EHR image segmentation, EHR image target detection, and clinical outcome prediction are performed by executing steps S51-S54 to obtain the required results of the tasks. The difference in the implementation of different downstream tasks lies in the module network of the downstream tasks in step S52. There's something different.

5. An EHR data analysis system based on augmented discriminant information, characterized by: The system is used to implement the EHR data analysis method based on augmented discriminant information according to any one of claims 1 to 4, comprising a computer processor, a memory and a graphics processor; a multimodal EHR data storage unit; and a multimodal EHR data preprocessing unit; Multimodal contrastive learning deep network model training unit; multimodal contrastive learning deep network model storage unit and downstream task unit.

6. The EHR data analysis system based on augmented discriminant information according to claim 5, characterized in that: The multimodal EHR data storage unit stores the multimodal EHR data set and downstream medical task data set constructed in step S1, and loads them into the computer memory; the multimodal EHR data preprocessing unit extracts batch-sized multimodal EHR samples from the memory in steps, executes step S2 to perform image and text augmentation preprocessing to obtain EHR image augmentation samples and EHR text augmentation samples, and loads them into the graphics processor; the multimodal contrast learning deep network model training unit uses the EHR image augmentation samples and EHR text augmentation samples in the graphics processor to execute steps S3-S4 to obtain the optimal multimodal contrast learning deep network model and store it in the multimodal contrast learning deep network model storage unit; based on the optimal model stored in the multimodal contrast learning deep network model storage unit, the downstream task unit executes step S5 in the graphics processor to obtain the required results of the downstream medical task; the specific data processing and calculation work in all units is completed by the computer processor and the graphics processor.

Images