Method and system for grading placental implantation disorder based on multi-modal data

Abstract

The present application relates to the field of medical image analysis, and particularly relates to a placenta implantation disease grading method and system based on multi-modal data, the method steps comprising: S1: collecting clinical information, medical image information and pathological examination results of a patient to form sample data; S2: constructing a data set used by a training model based on the sample data, and training a placenta implantation disease grading diagnosis model; S3: using the trained placenta implantation disease grading diagnosis model to perform grading diagnosis on the patient. The present application solves the problem of coarse grading granularity and insufficient accuracy of placenta implantation disease in the prior art by fusing medical image features and clinical feature data to construct a deep learning model, and realizes accurate grading of placenta adhesion, implantation and penetration.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image analysis, in particular to a placenta accreta spectrum disorders grading method and system based on multi-modal data. BACKGROUND

[0002] Placenta accreta spectrum disorders (PAS) is a high-risk pregnancy complication, and its pathological feature is that the placental villi abnormally invade the uterine muscle layer or even the serous layer. According to the International Federation of Obstetrics and Gynecology guidelines, PAS is divided into three levels according to the depth of invasion: placenta accreta, placenta increta and placenta percreta, as shown in FIG. 1. The maternal and infant risks corresponding to different grades are significantly different, which directly affects the choice of delivery mode and perinatal management, etc., so accurate prenatal grading diagnosis is crucial to reduce the rate of hysterectomy, postpartum hemorrhage and other serious complications. Figure 1

[0003] At present, the clinical diagnosis mainly relies on the artificial comprehensive evaluation of obstetric medical image data (such as ultrasound, magnetic resonance imaging) and patient clinical risk factors (such as cesarean section history, uterine surgery history). The image physician needs to subjectively interpret the placental position, uterine muscle layer blood flow signal and other characteristics, and make grading judgments in combination with the patient's medical history. However, this method has significant limitations: the subjective interpretation of image features is easily influenced by the experience of the physician, and is prone to misjudgment, which delays the treatment of the patient.

[0004] In recent years, artificial intelligence technology has been gradually applied to the auxiliary diagnosis of PAS. For example, a model based on a convolutional neural network can distinguish between normal and abnormal placentas through medical images. However, the existing technology still has obvious defects: on the one hand, the existing model can only classify accreta, increta and percreta as "abnormal placenta", and cannot achieve accurate grading; on the other hand, the existing method only relies on single-modal image data and does not fuse patient clinical feature data, resulting in insufficient accuracy of the model in grading complex cases. The above limitations seriously restrict the clinical application value of artificial intelligence technology in PAS grading diagnosis. SUMMARY

[0005] The purpose of the present application is to provide a placenta accreta spectrum disorders (PAS) grading method and system based on multi-modal data, which fuses medical image features and clinical feature data to construct a deep learning model, solves the problem of coarse PAS grading granularity and insufficient accuracy in the prior art, and realizes accurate grading of placenta accreta, increta and percreta.

[0006] To achieve the above purpose, the present application proposes the following technical solutions:

[0007] In a first aspect, the present application provides a placenta accreta spectrum disorders grading method based on multi-modal data, the method steps comprising:​

[0008] S1: collecting clinical information, medical image information, and pathological examination results of a patient to form sample data;

[0009] S2: constructing a data set used by a training model based on the sample data, and training a placenta accreta spectrum disease grading diagnosis model;

[0010] S3: using the trained placenta accreta spectrum disease grading diagnosis model to perform grading diagnosis on the patient;

[0011] The placenta accreta spectrum disease grading diagnosis model comprises a medical image encoder, a placenta position classifier, a multi-modal feature combiner, and a multi-modal feature classifier.

[0012] The medical image encoder is configured to obtain information related to placenta accreta spectrum disease in a medical image, input the medical image of the patient into the medical image encoder, and output a medical image representation vector of the patient.

[0013] The placenta position classifier is configured to determine the position of the placenta of the patient according to the medical image representation vector, and output an imaging feature vector of the patient; the imaging feature vector comprises the probability of non-placenta previa, the probability of complete placenta previa, and the probability of placenta in a moderate risk position.

[0014] The multi-modal feature combiner is configured to concatenate the imaging feature vector and a clinical feature vector of the patient, and output a multi-modal feature vector; the clinical feature vector is a vector composed of N elements, each element corresponding to a clinical feature.

[0015] The multi-modal feature classifier is configured to determine the grade of placenta accreta spectrum disease of the patient according to the multi-modal feature vector.

[0016] Preferably, the clinical information comprises age, weight, height, pregnancy history, cesarean section history, and artificial abortion history; the medical image information comprises ultrasound and MRI images and imaging examination results of the MRI images; and the pathological examination results are obtained by performing pathological examination on the placental villi, uterine decidua, and muscle layer tissue in the placenta attachment area after surgery.

[0017] Preferably, step S1 further comprises screening the clinical information using a chi-square test to obtain screened clinical features, and constructing sample data according to the screened clinical features.

[0018] Further, the method for constructing the data set used by the training model comprises:

[0019] converting the medical image information into a fixed size conforming to the model input, and performing image standardization processing;

[0020] The screened clinical features are binned to obtain a clinical feature vector;

[0021] The label of the training sample is calculated, including a placenta position label and a placenta implantation grade label.

[0022] Further, the placenta position label is , wherein is the probability of non-preplacenta, is the probability of complete preplacenta, is the probability of medium-risk position placenta, wherein the medium-risk position placenta includes low-set placenta, marginal preplacenta, and partial preplacenta;

[0023] The placenta implantation grade label is , wherein is the probability of normal placenta, is the probability of adherent placenta, is the probability of implanted placenta, is the probability of penetrating placenta.

[0024] Preferably, the splicing method of the multi-modal feature vector includes:

[0025] The imaging feature vector and the clinical feature vector are spliced into a first-order multi-modal feature vector;

[0026] The outer product of the imaging feature vector and the clinical feature vector is calculated to obtain a second-order multi-modal feature matrix;

[0027] The second-order multi-modal feature matrix is matrix-vectorized to obtain a second-order multi-modal feature vector;

[0028] The first-order multi-modal feature vector and the second-order multi-modal feature vector are spliced to obtain the multi-modal feature vector.

[0029] Preferably, the multi-modal feature classifier outputs a placenta implantation disease grade probability distribution of the patient, including the probability of normal placenta of the patient, the probability of placenta adhesion of the patient, the probability of placenta implantation of the patient, and the probability of placenta penetration of the patient, and the grade with the largest probability is taken as the diagnosis grade of the patient.

[0030] Preferably, the placenta implantation disease grading diagnosis model is trained by using a stochastic gradient descent algorithm, and the model training is completed when the loss does not change for 5 consecutive rounds or the training reaches a specified number of rounds.

[0031] Further, the model loss function is as follows:

[0032]

[0033] In the above formula, Indicates weight, , This indicates placental position classification loss. This indicates placental grading loss.

[0034] In a second aspect, the present invention provides a grading system for placental implantation disease based on multimodal data, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; the processor executes the computer program to implement the aforementioned grading method for placental implantation disease based on multimodal data.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention first trains a multimodal neural network model for the grading and diagnosis of placenta accreta, and then uses the trained model to grade patients. The proposed model uses the patient's medical imaging data and clinical characteristics to comprehensively diagnose the grade of placenta accreta, improving the accuracy and efficiency of clinical grading of placenta accreta and providing assistance to obstetricians in making accurate and efficient diagnoses. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating the grading of placental accreta disease in the background art.

[0038] Figure 2 This is a flowchart of the grading method for placental implantation disease in Specific Implementation Example 1;

[0039] Figure 3 This is a schematic diagram of the grading and diagnostic model for placental implantation disease in specific embodiment 1;

[0040] Figure 4 This is a flowchart illustrating the implementation of the placental implantation disease grading and diagnostic model in Specific Implementation Example 1;

[0041] Figure 5 This is a schematic diagram of the placental location classifier structure in specific embodiment 2;

[0042] Figure 6 This is a schematic diagram of the multimodal feature classifier structure in specific embodiment 2;

[0043] Figure 7 The images shown are MRI images of various sections of a patient with suspected placenta accreta in Specific Implementation Example 3, along with their corresponding placental annotations. Detailed Implementation

[0044] For better understanding of the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0045] Embodiment 1

[0046] Figure 2 The method provided by the present application is a placenta implantation disease grading method based on multi-modal data, comprising the following steps: collecting clinical information, medical image information and pathological examination results of a patient to form sample data; constructing a data set used by a training model based on the sample data, and training a placenta implantation disease grading diagnosis model; and using the trained placenta implantation disease grading diagnosis model to perform grading diagnosis on the patient. The specific contents of each step are as follows:

[0047] S1: Collecting clinical information, medical image information and pathological examination results of a patient to form sample data.

[0048] The clinical information includes age, weight, height, pregnancy history, cesarean section history and artificial abortion history; the medical image information includes ultrasound and MRI images and the imaging examination results of the MRI images; and the pathological examination results are obtained by performing pathological examination on the placental villi, uterine decidua and muscle layer tissue in the placenta attachment area after surgery, and are used as the gold standard for PAS grading.

[0049] The clinical information is screened using a chi-square test to determine the clinical characteristics included in the sample. In the chi-square test, feature screening is generally performed according to the significance level indicator p-value, and the critical threshold value of the p-value is set to 0.05 by default, or can be adjusted according to the number of patients collected.

[0050] S2: Constructing a data set used by a training model based on the sample data, and training a placenta implantation disease grading diagnosis model.

[0051] First, the medical image information and the screened clinical characteristics of each patient are preprocessed, wherein the medical image information needs to be preprocessed to a fixed size (320x320 pixels) conforming to the model input, and standardized to zero mean unit variance; the screened clinical characteristics need to be preprocessed by two-part box, for example: for the age feature, when the age is ≤35 years old, the value is 0, and when the age is >35 years old, the value is 1; for the number of previous cesarean sections feature, when the number of cesarean sections of the patient is 0, the value is 0, and when the number of cesarean sections of the patient is greater than 0, the value is 1; for the number of previous artificial abortions feature, when the number of artificial abortions of the patient is 0, the value is 0, and when the number of artificial abortions of the patient is greater than 0, the value is 1.

[0052] Secondly, the labels of the training samples are calculated, including placenta location labels and placenta implantation grade labels . Among them, the probability of being a non-placenta previa, the probability of being a complete placenta previa, the probability of being a moderate-risk placenta, the moderate-risk placenta including a low-lying placenta, a marginal placenta previa, and a partial placenta previa; the probability of being a normal placenta, the probability of being a adherent placenta, the probability of being an implantation placenta, the probability of being a penetrating placenta.

[0053] Then, the placenta implantation disease diagnosis model is trained by using the data set constructed above; the structure and implementation process of the diagnosis model are shown in Figure 3 , Figure 4 respectively, including a medical image encoder, a placenta position classifier, a multi-modal feature combiner, and a multi-modal feature classifier.

[0054] The medical image encoder is used to learn information closely related to the placenta implantation disease in the medical image, and the medical image of the patient is input into the medical image encoder to output a medical image representation vector of the patient; the structure of the medical image encoder includes but is not limited to a residual neural network, an encoder of a pre-trained medical image large model, etc.

[0055] The placenta position classifier is used to determine the position of the placenta of the patient according to the medical image representation vector output by the medical image encoder, and output an imaging feature vector of the patient; the imaging feature vector is a vector containing three elements , which correspond to the probability of being a non-placenta previa, the probability of being a complete placenta previa, and the probability of being a moderate-risk placenta, respectively, wherein the moderate-risk placenta includes a low-lying placenta, a marginal placenta previa, and a partial placenta previa; the structure of the placenta position classifier includes but is not limited to a feedforward fully connected neural network, a Transformer module, etc.

[0056] The multi-modal feature combiner is used to splice the imaging feature vector and the clinical feature vector of the patient; wherein the clinical feature vector is a vector composed of N elements, and each element corresponds to a clinical feature; first, the imaging feature vector and the clinical feature vector are spliced into a first-order multi-modal feature vector, and the first-order multi-modal feature vector after splicing is as shown in formula (1), where represents the imaging feature vector, represents the clinical feature vector, wherein (i is a subscript, i=1,2,…,N) is the value after two-way binning preprocessing of the clinical feature, , represents a real number set; then, a second-order multi-modal feature matrix is calculated by using formula (2) , then the matrix is vectorized (concatenated by row), and a second-order multimodal feature vector is obtained, which contains 3N elements; finally, the first-order multimodal feature vector and the second-order multimodal feature vector are concatenated to obtain the final multimodal feature vector.

[0057]

[0058]

[0059] The multimodal feature classifier is used to determine the grade of placental implantation disease of the patient according to the input multimodal feature vector, and the classifier outputs a vector containing 4 elements , element represents the probability that the patient's placenta is normal, represents the probability that the patient is placenta accreta, represents the probability that the patient is placenta percreta, represents the probability that the patient is placenta percreta; the multimodal feature classifier includes but is not limited to a simple feedforward neural network containing K hidden layers.

[0060] Finally, the model is trained by using the stochastic gradient descent algorithm, and the total loss function of the model is shown in formula (3); here represents the weight, , represents the placenta position classification loss, represents the placenta grading loss.

[0061] (3)

[0062] The placenta position classification loss is shown in formula (4), is the placenta position label of patient i, only one element is 1, indicating the placenta position, and the remaining elements are 0, is the placenta position category probability distribution of patient i output by the placenta position classifier proposed by the present application, and L represents the number of patient samples in the training set.

[0063] (4)

[0064] The placenta grading loss is shown in formula (5), is the placenta implantation grade label of patient i, only one element is 1, indicating the grade of placental implantation disease of the patient, and the remaining elements are 0, is the grading probability distribution of the information output by the placenta implantation disease grading diagnosis model proposed by the present application for patient i, and L represents the number of patient samples in the training set.

[0065] (5)

[0066] When the loss does not change for 5 consecutive rounds or the training reaches the specified round, the model training is completed.

[0067] S3: Using the trained placenta implantation disease grading diagnosis model to grade and diagnose the patient.

[0068] The medical image information and clinical features of a new patient are input into the placenta implantation disease grading diagnosis model, the medical image information and clinical features are preprocessed according to step S2, and the preprocessed data is input into the trained placenta implantation disease grading diagnosis model in step S3. The placenta implantation disease grading diagnosis model outputs the probability distribution of the patient's placenta implantation disease grade, and the most probable grade is taken as the patient's diagnosis grade.

[0069] Example 2

[0070] Take a patient's information as an example to illustrate how to construct a training sample. Assume that the clinical features selected in step S1 include age, number of previous cesarean sections, and number of previous artificial abortions. The patient's clinical features, medical image information, and pathological examination results are shown in Table 1:

[0071] Table 1 Information of a certain patient

[0072]

[0073] First, the clinical features are preprocessed to obtain the clinical feature vector .

[0074] Second, the medical image is preprocessed by adjusting the medical image pixels to 320x320. The specific method is to adjust the long side of the image to 320 pixels, and then symmetrically fill the short side with zero values. Then, the intensity of the medical image is standardized so that the mean of the image gray value is 0 and the variance is 1.

[0075] Finally, the label of the training sample is calculated. The label has two, namely the placenta position label and the placenta implantation grade label . According to the medical image examination report, the patient is complete placenta previa, so the placenta position label is . According to the pathological examination result, the patient is placenta implantation, so the placenta implantation grade label is .

[0076] Based on the above method, information of multiple patients is collected to construct training samples, which are combined into a training data set, and used to train the placenta implantation disease grading diagnosis model.

[0077] The information of the patient in Table 1 is taken as an example to illustrate how to train the placenta implantation disease grading diagnosis model. First, the preprocessed medical image in the training sample is input into the medical image encoder to extract the medical image representation vector. In specific implementation, the medical image encoder can include but is not limited to a residual neural network, for example, ResNet50. In this specific embodiment, the adopted encoder is ResNet50, which includes 1 initial convolutional layer, 4 residual stages, and 1 global average pooling layer, wherein the 4 residual stages respectively include 3, 4, 6, and 3 residual blocks. The encoder converts the image features into a vector with a length of 2048.

[0078] Secondly, the medical image representation vector is input into the placenta position classifier to obtain an imaging feature vector. In this specific embodiment, a multi-layer perceptron is adopted as the placenta position classifier, which includes an input layer, two hidden layers, and an output layer, as shown in Figure 5 The input layer has 2048 input neurons, the two hidden layers respectively include 512 and 128 neurons, the activation function adopts the ReLU function, the output layer includes 3 output neurons, the activation function adopts the Softmax function, and the 3 output neurons correspond to the probabilities of non-preplacenta, complete preplacenta, and moderate risk position placenta in turn.

[0079] Then, the imaging feature vector and the clinical feature vector are spliced to obtain a first-order multi-modal feature vector, and the splicing manner is shown in formula (1). The first-order multi-modal feature vector obtained by splicing includes 6 elements. The outer product of the imaging feature vector and the clinical feature vector is calculated to obtain a second-order multi-modal feature matrix, and the outer product manner is shown in formula (2). Then, the second-order multi-modal feature matrix is vectorized to obtain a second-order multi-modal feature vector, which includes 9 elements.

[0080] Finally, the first-order multi-modal feature vector and the second-order multi-modal feature vector are spliced to obtain a multi-modal feature vector. The multi-modal feature vector is input into the multi-modal feature classifier to obtain the placenta implantation disease grade probability distribution of the patient. In this specific embodiment, as shown in Figure 6 The multi-modal feature classifier is implemented by a multi-layer perceptron, which includes an input layer, two hidden layers, and an output layer. The input layer includes 15 neurons. The two hidden layers respectively include 256 neurons and 128 neurons, and each neuron adopts the ReLU function as the activation function. The output layer includes 4 output neurons, each of which adopts the Softmax function as the activation function, and calculates the probabilities of the patient belonging to placenta normal, placenta adhesion, placenta implantation, and placenta penetration.

[0081] In the training process, for each sample, the placenta position classification loss is calculated by formula (4), the placenta grading loss is calculated by formula (5), and the total loss is calculated by formula (3). In this specific embodiment, the loss is set to ; the joint loss is calculated by back propagation with respect to the model parameters using a stochastic gradient descent optimizer, and the network parameters are updated. When the loss does not decrease for 5 consecutive rounds or the training reaches the specified number of rounds, the model training is complete.

[0082] Embodiment 3

[0083] An example of a patient's information is used to illustrate how to use the trained placenta implantation disease grading diagnostic model to diagnose the patient.

[0084] For a new patient, first collect the patient's clinical characteristics and medical image information, such as age 36, number of previous cesarean sections 1, number of previous induced abortions 0, and a set of MRI images (such as Figure 7 ) of the patient. Then the patient's clinical characteristics are preprocessed according to step S2 to obtain the clinical feature vector [1, 1, 0]; the patient's MRI image is preprocessed to 320x320 pixels according to step S2 and standardized to obtain the standardized medical image. Finally, the clinical feature vector and the standardized medical image are input into the trained placenta implantation disease grading diagnostic model of step S3. The model outputs the probability of the patient belonging to the four grades, assuming it is [0.1, 0.1, 0.3, 0.5], and the grade with the highest probability is selected as the patient's grade. Here, the final grading result of the patient is placenta penetration.

[0085] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for grading placental accreta disease based on multi-modal data, characterized in that, The method steps comprise: S1: collecting clinical information, medical image information, and pathological examination results of a patient to form sample data; In step S1, the clinical information is screened using a chi-square test to obtain screened clinical features, and the sample data is constructed according to the screened clinical features; S2: constructing a data set used for training a model based on the sample data, and training a placental implantation disease grading diagnosis model; The method for constructing the data set used for training the model comprises: converting the medical image information into a fixed size conforming to the model input, and performing image standardization processing; performing two-bin preprocessing on the screened clinical features to obtain a clinical feature vector; calculating the labels of the training samples, including placenta position labels and placental implantation grade labels; The placental position label is wherein the probability that the placenta is not a placenta previa, the probability that the placenta is a complete placenta previa, the probability that the placenta is a moderate risk position placenta, wherein the moderate risk position placenta comprises a low-lying placenta, a marginal placenta previa, a partial placenta previa; The placental implantation grade label is wherein is the probability of a normal placenta, is the probability of a placenta accreta, is the probability of a placenta increta, is the probability of a placenta percreta; S3: using the trained placental implantation disease grading diagnosis model to perform grading diagnosis on the patient; The placental implantation disease grading diagnosis model comprises a medical image encoder, a placenta position classifier, a multi-modal feature combiner, and a multi-modal feature classifier. The medical image encoder is used to obtain information related to the placental implantation disease in the medical image, input the medical image of the patient into the medical image encoder, and output a medical image representation vector of the patient. The placenta position classifier is used to determine the position of the placenta of the patient according to the medical image representation vector, and output an imaging feature vector of the patient; the imaging feature vector comprises the probability of non-placenta previa, the probability of complete placenta previa, and the probability of placenta in a moderate risk position. The multi-modal feature combiner is used to splice the imaging feature vector and the clinical feature vector of the patient to output a multi-modal feature vector; the clinical feature vector is a vector composed of N elements, and each element corresponds to a clinical feature. The multi-modal feature classifier is used to determine the grade of the placental implantation disease of the patient according to the multi-modal feature vector.

2. The method of claim 1, wherein the method is based on multi-modal data. The clinical information comprises age, weight, height, pregnancy history, cesarean section history, and artificial abortion history; the medical image information comprises ultrasound and MRI images and imaging examination results of the MRI images; the pathological examination results are obtained by performing pathological examination on the placental villi, uterine decidua, and muscle layer tissue in the placenta attachment area after surgery.

3. The method of claim 1, wherein the method is based on multi-modal data. The splicing method of the multi-modal feature vector comprises: splicing the imaging feature vector and the clinical feature vector into a first-order multi-modal feature vector; calculating the outer product of the imaging feature vector and the clinical feature vector to obtain a second-order multi-modal feature matrix; performing matrix vectorization on the second-order multi-modal feature matrix to obtain a second-order multi-modal feature vector; splicing the first-order multi-modal feature vector and the second-order multi-modal feature vector to obtain the multi-modal feature vector.

4. The method of claim 1, wherein the method is based on multi-modal data. The multi-modal feature classifier outputs the probability distribution of the grade of the placental implantation disease of the patient, including the probability of normal placenta, the probability of placental adhesion, the probability of placental implantation, and the probability of placental penetration, and the grade with the maximum probability is taken as the diagnosis grade of the patient.

5. The method of claim 1, wherein the method is based on multi-modal data. The placenta implantation disease grading diagnosis model is trained by using a stochastic gradient descent algorithm, and the model training is completed when the loss does not change for 5 consecutive rounds or the training reaches a specified round.

6. The method of claim 5, wherein the method is based on multi-modal data. The model loss function is as follows: In the above formula, denotes the weight, , denotes the placenta position classification loss, denotes the placenta grading loss.

7. A placental accreta disease grading system based on multi-modal data, characterized in that, The computer program stored in the memory and executable on the processor; the processor executes the computer program to realize the placenta implantation disease grading method based on multi-modal data according to any one of claims 1 to 6.

Images