Placenta implantable disease grading diagnosis method based on multi-modal deep fusion model

Abstract

The invention relates to the field of image analysis, and discloses a placenta implantable disease grading diagnosis method based on a multi-modal deep fusion model. The method comprises the following steps: collecting patient information including basic information, ultrasonic images, imaging examination results and pathological examination results, and constructing a data set for training a placenta implantable disease grading diagnosis model according to the patient information; constructing and training a numerical feature encoder and an ultrasonic image encoder by using the data set; and constructing and training a multi-modal deep fusion model by using the data set, the trained numerical feature encoder and the trained ultrasonic image encoder. According to the method, the problem of insufficient accuracy of the single-mode model in the classification diagnosis task of the placenta implantable disease is solved, and the diagnosis performance is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image analysis, and in particular to a placenta accreta spectrum grading diagnosis method based on a multi-modal deep fusion model. BACKGROUND

[0002] Placenta accreta spectrum is a common complication of pregnancy. The placenta of a patient will adhere or invade the deep layer of the uterine wall in an abnormal way. When the invasion is severe, the placental villi will penetrate the myometrium, causing the placenta to not fall off naturally during delivery. This abnormal placenta needs to be manually stripped, accompanied by massive maternal hemorrhage shock, and even the risk of death. Preoperative staging of the placenta accreta spectrum of a patient is of great significance and value for doctors to develop a treatment process and prognosis for the patient. The international grading standard for placenta accreta spectrum is: normal placenta, adherent placenta, invasive placenta and penetrating placenta. Existing placenta accreta spectrum grading diagnosis is mainly based on obstetric ultrasound images by ultrasound doctors. For high-risk patients, a higher definition magnetic resonance imaging (MRI) is usually needed to grade the invasion of the patient, which is adherent, invasive, and penetrating. On the other hand, studies have shown that patient age, history of miscarriage, history of cesarean section, and uterine myomectomy are risk factors for placenta accreta spectrum.

[0003] With the rapid development of deep learning technology, multi-modal models have been gradually applied to the field of medical diagnosis. By combining medical image information and clinical information, multi-modal models can effectively improve the performance of medical diagnosis. Currently, there is no research on applying multi-modal models to the grading diagnosis of placenta accreta spectrum. In view of this situation, the present application proposes a placenta accreta spectrum grading diagnosis method based on a multi-modal deep fusion model. The multi-modal deep fusion model proposed by the present application has the following characteristics: first, the numerical feature encoder and the ultrasound image encoder are trained using the dataset; then the multi-modal diagnosis model is trained, and the parameters of the numerical feature encoder and the ultrasound image encoder are frozen during training, and only the parameters of the feature fusion encoder and the classification head are optimized; finally, the trained multi-modal model is used to grade the placenta accreta spectrum of a patient. The experimental results based on a single-center dataset show that compared with a single-modal model (only considering medical image information), the accuracy of the multi-modal model in the grading diagnosis of placenta accreta spectrum is significantly improved. SUMMARY

[0004] The application aims to provide a placental implantation disease grading diagnosis method based on a multi-modal deep fusion model. The method can make full use of the clinical information and ultrasound image information of the patient to perform prenatal grading diagnosis on the placental implantation degree of the patient. To achieve the above application purpose, the construction of the multi-modal deep fusion model for placental implantation disease grading diagnosis provided by the application includes three stages: the first stage is used to construct a multi-modal data set, including a training data set and a verification data set; the second stage uses the data set constructed in the first stage to train a numerical feature encoder and an ultrasound image encoder respectively; and the third stage freezes the parameters of the numerical feature encoder and the ultrasound image encoder, trains a multi-modal deep fusion model, and selects the model with the optimal performance on the verification set as the final diagnosis model. Figure 1 The structure diagram of the multi-modal deep fusion model for placental implantation disease grading diagnosis is shown.

[0005] The specific steps of the first stage are as follows:

[0006] Step 1: Collect the basic information, ultrasound image, imaging examination result and pathological examination result of the patient; the basic information includes but is not limited to: age, weight, height, number of pregnancies, number of this pregnancy, number of previous cesarean sections, number of previous artificial abortions, etc.; the ultrasound image is an abdominal ultrasound image of the patient after 28 weeks of pregnancy, which is intercepted by an ultrasonographer to ensure that the image contains the placenta and uterine muscle layer region; the imaging examination result contains the diagnosis result of the placental position; the pathological examination result is the result of sending the placental villi, uterine decidua and muscle layer tissue of the placental attachment area of the patient for pathological examination after surgery, which is used as the gold standard of the placental implantation grade of the patient in the application;

[0007] Step 2: Construct a data set for training a placental implantation disease grading diagnosis model, and the specific method is as follows: construct a sample for each patient, and the sample features include the numerical representation of the ultrasound image and the clinical information of the patient; the ultrasound image needs to be preprocessed to a fixed size M◊M pixel conforming to the model input, and the pixel value needs to be normalized; in the application, the clinical information included in the sample includes age, number of previous cesarean sections, number of previous artificial abortions, and placental position in the ultrasound examination result;

[0008] The calculation formula for age characteristics is as follows (1), the calculation formula for the number of previous cesarean sections is as follows (2), the calculation formula for the number of previous artificial abortions is as follows (3), and the calculation formula for the placental position characteristics is as follows (4); Here, the diagnosis results of placental position are divided into six categories: non-placenta previa, low-lying placenta, marginal placenta previa, partial placenta previa, complete placenta previa, and pernicious placenta previa; for example, the patient's numerical characteristics are (0, 1, 0, 4), which means that the patient is less than or equal to 35 years old, has undergone a cesarean section and has not undergone an abortion, and the placental position determined by ultrasound examination is complete placenta previa;

[0009] (1)

[0010] (2)

[0011] (3)

[0012] (4)

[0013] The sample labels are the results of pathological examination, which are divided into four levels: normal placenta, placenta accreta, placenta accreta, and placenta percreta. The preprocessing is done using formula (5):

[0014] (5)

[0015] After the dataset is constructed, it is divided into training set and validation set according to a certain ratio. The training set is used to train the model parameters, and the validation set is used for model selection.

[0016] The specific steps of the second phase are as follows:

[0017] Step 3: Use the dataset constructed in step 2 to train the numerical feature encoder. Only the numerical features and labels of the samples are used during the training process. The structure of the numerical feature encoder training model is shown in the attached figure. Figure 2 As shown, it includes a numerical feature encoder and a numerical feature classifier; the numerical feature encoder is used to learn information closely related to placenta accreta diseases in clinical data, the patient's numerical features are input to the numerical feature encoder, and the output is the patient's numerical feature representation vector; the structure of the numerical feature encoder can adopt but is not limited to a multi-layer perceptron; the classifier is used to predict the patient's placenta accreta grade based on the numerical feature representation vector output by the numerical feature encoder; the classifier outputs a vector containing 4 elements ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, Indicates the probability that the patient has placenta percreta; the classifier is implemented using a single-layer perceptron;

[0018] The model is trained using the stochastic gradient descent algorithm, and the model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the probability distribution of placenta accreta disease classification output by the numerical feature encoder training model proposed in the present invention for patient i, and L represents the number of patient samples in the training set;

[0019] (6)

[0020] When the loss on the validation set does not decrease for T consecutive rounds or the training reaches the specified round, the model training is completed. The model with the smallest loss on the validation set is selected, and the numerical feature encoder in the model is used as the numerical feature encoder in the multimodal model in step 5;

[0021] Step 4: Use the dataset constructed in step 2 to train the ultrasound image encoder. Only the ultrasound images and labels of the samples are used during the training process. The structure of the ultrasound image encoder training model is shown in the attached figure. Figure 3 As shown, it includes an ultrasound image encoder and a classifier; the ultrasound image encoder is used to learn information closely related to placenta accreta disease in ultrasound images, the patient's ultrasound image is input to the ultrasound image encoder, and the output is an image feature representation vector; the structure of the ultrasound image encoder can be implemented using but not limited to a convolutional neural network, a residual network, an encoder of a medical image basic model, etc.; the classifier is used to predict the grade of the patient's placenta accreta disease based on the image feature representation vector output by the ultrasound image encoder; the classifier outputs a vector containing 4 elements ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, Indicates the probability that the patient has placenta percreta; the classifier is implemented using a single-layer perceptron;

[0022] The model is trained using the stochastic gradient descent algorithm, and the model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the hierarchical probability distribution output by the ultrasound image encoder training model proposed in the present invention for patient i, and L represents the number of patient samples in the training set;

[0023] When the loss on the validation set does not decrease for T consecutive rounds or the training reaches the specified round, the model training is completed, and the model with the smallest loss on the validation set is selected, and the ultrasound image encoder in the model is used as the ultrasound image encoder in the multimodal model in step 5;

[0024] The specific steps of the third phase are as follows:

[0025] Step 5: Use the dataset constructed in step 2 to train a multimodal deep fusion model for the graded diagnosis of placenta accreta disease. The structure of the multimodal model is shown in the attached figure. Figure 1 As shown, it includes a numerical feature encoder, an ultrasound image encoder, a fusion feature encoder, and a fusion feature classifier; the numerical feature encoder and the ultrasound image encoder here are the numerical feature encoder trained in step 3 and the ultrasound image encoder trained in step 4, respectively. During the multimodal model training process, the model parameters of the numerical feature encoder and the ultrasound image encoder remain unchanged;

[0026] The numerical features and ultrasound images of the patients in the sample are input into the numerical feature encoder and the ultrasound image encoder respectively to obtain the numerical feature representation vector and the image representation vector; the two representation vectors are input into the fusion feature encoder to calculate the fusion feature representation vector after deep fusion; the structure of the fusion feature encoder is shown in the attached figure. Figure 4 As shown in Figure 2, it consists of 4 stacked multi-head self-attention modules. The number of self-attention heads in a single multi-head self-attention module is set to 4. The fused feature representation vector is input to the fused feature classifier, and the classifier outputs a vector containing 4 elements. ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, Indicates the probability that the patient has placenta percreta; the classifier is implemented using a single-layer perceptron;

[0027] The model is trained using the stochastic gradient descent algorithm, and the model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the hierarchical probability distribution output by the multimodal model proposed in this invention for patient i, and L represents the number of patient samples in the training set;

[0028] When the loss on the validation set does not decrease for T consecutive rounds or reaches the specified round of training, model training is completed, and the model with the smallest loss on the validation set is selected as the final deployment model for the graded diagnosis of placenta accreta disease;

[0029] Step 6: Preprocess a new patient's basic information, ultrasound images, and imaging examination results according to step 2 to obtain a numerical representation of the patient's clinical information and an ultrasound image that meets the model input size. Input the numerical representation and ultrasound image into the multimodal model. The multimodal model outputs the patient's placenta accreta disease grade probability distribution and selects the grade with the highest probability as the patient's diagnostic grade. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the implementation of the present invention or the existing technical solutions, the following briefly introduces the drawings required for use in the embodiments or the description of the existing technologies.

[0031] Figure 1 It is a structural diagram of the multimodal deep fusion model provided by the present invention;

[0032] Figure 2 Schematic diagram of the numerical feature encoder training model structure provided by the present invention;

[0033] Figure 3 1 is a schematic diagram of the structure of the ultrasound image encoder training model provided by the present invention;

[0034] Figure 4 Schematic diagram of the structure of the fusion feature encoder provided by the present invention;

[0035] Figure 5 This is a basic flow chart for constructing a hierarchical diagnostic model for placenta accreta disease provided by the present invention;

[0036] Figure 6 is a sample schematic diagram constructed based on patient information provided by the present invention; DETAILED DESCRIPTION

[0037] The following describes the specific embodiments of the present invention in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, such descriptions will be omitted here.

[0038] Attachment Figure 5 This is the process for building a multimodal model for the graded diagnosis of placenta accreta, as provided by the present invention. It consists of three phases: the first phase constructs the training and validation sets required for model training; the second phase trains the numerical feature encoder for extracting clinical numerical features and the ultrasound image encoder for extracting ultrasound image features; and the third phase performs end-to-end training on the multimodal model to obtain the complete parameters of the multimodal model for the graded diagnosis of placenta accreta.

[0039] The first stage steps are as follows:

[0040] Step 1: Collect the basic information of the patient, ultrasound image, imaging examination results and pathological examination results; the basic information includes but is not limited to: age, weight, height, number of pregnancies, number of this pregnancy, number of previous cesarean sections, number of previous artificial abortions, etc.; the ultrasound image is an abdominal ultrasound image of the patient after 28 weeks of pregnancy, which is intercepted by an ultrasonographer, ensuring that the image contains the placenta and myometrial region; the imaging examination results include the diagnosis results of the placental position; the pathological examination results are the results of pathological examination of the placental villi, uterine decidua and myometrial tissue in the placenta attachment area of the patient after surgery, which are used as the gold standard for the classification of placenta accreta in the present application;

[0041] Step 2: Construct a data set for training the placenta accreta disease classification model, the specific method is as follows: construct a sample for each patient, the sample features include the numerical representation of the patient's ultrasound image and clinical information (numerical features); the ultrasound image needs to be preprocessed to meet the fixed size of the model input, and the pixel value needs to be normalized; in the present application, the clinical information included in the sample includes age, number of previous cesarean sections, number of previous artificial abortions, and placental position in the ultrasound examination results, which are calculated by formulas (1), (2), (3), and (4), respectively; the label of the sample is calculated by formula (5).

[0042] Here, the information of a patient is taken as an example to illustrate how to construct a training sample, assuming that the patient's age, number of previous cesarean sections, number of previous artificial abortions, ultrasound image, imaging examination results and pathological examination results are as shown in the attached Figure 6

[0043] First, the clinical features are preprocessed, and according to formulas (1), (2), (3), and (4), the clinical features are obtained as ; Then, the medical image is preprocessed, the medical image pixels are adjusted to 320◊320, the specific method is to adjust the long side of the image to 320 pixels, and then symmetrically fill the short side with zero values; then, the intensity of the medical image is normalized, and the gray value of each pixel is divided by 255, so that all pixel gray values are standardized to the range of [0, 1]; finally, the label of the sample is calculated according to formula (5), according to the pathological examination results, the placenta accreta grade label is .

[0044] After the data set is constructed, the training set and the validation set are divided, the specific method is as follows: under the premise of ensuring the balance of data distribution, the stratified sampling strategy is adopted to divide the data set into training set and validation set in the ratio of 4:1, wherein the training set is used for training and optimization of model parameters, and the validation set is used for model selection;

[0045] ​Step 3: Train the numerical feature encoder using the dataset constructed in step 2, and only use the numerical features and labels of the samples in the training process;

[0046] First, the numerical features of the samples are input into the numerical feature encoder to calculate the numerical feature representation vector; in this embodiment, the encoder uses a single-layer perceptron, the input layer dimension is 4, the output layer dimension is 256, the activation function is ReLU, and the normalization processing is performed through the LayerNorm layer; then, the numerical feature representation vector is input into the numerical feature classifier to calculate the probability distribution of the degree of placental implantation of the patient; the numerical feature classifier uses a single-layer perceptron to realize, as shown in FIG. 6, which includes an input layer and an output layer, the input layer includes 256 neurons, and the output layer includes 4 output neurons, each output neuron uses a Softmax function as an activation function to calculate the probability of the patient belonging to a normal placenta, an adherent placenta, an implanted placenta, and a penetrating placenta, respectively. Figure 2

[0047] In the training process, for the training set samples, the loss is calculated by formula (6). In this specific embodiment, a stochastic gradient descent optimizer is used to calculate the gradient of the loss with respect to the model parameters through backpropagation, and then the network parameters are updated. When the loss on the validation set does not decrease for 20 consecutive rounds or the training reaches the specified number of rounds, the model training is completed. The encoder of the model with the smallest loss on the validation set is used as the numerical feature encoder of the multi-modal model in step 5;

[0048] Step 4: Train the ultrasound image encoder using the dataset constructed in step 2, and only use the ultrasound images and labels of the samples in the training process; first, input the ultrasound images of the samples into the ultrasound image encoder to obtain the image representation vector, in this specific embodiment, the ultrasound image encoder used is Resnet50, and the last layer is a fully connected layer with a dimension of 256. The image representation vector obtained by inputting the ultrasound image into the encoder is a one-dimensional vector with a length of 256; then, the image representation vector is input into the image feature classifier to calculate the probability distribution of the degree of placental implantation of the patient; in this specific embodiment, the image feature classifier uses a single-layer perceptron to realize, as shown in FIG. 7, which includes an input layer and an output layer, the input layer includes 256 neurons, and the output layer includes 4 output neurons, each output neuron uses a Softmax function as an activation function to calculate the probability of the patient belonging to a normal placenta, an adherent placenta, an implanted placenta, and a penetrating placenta, respectively. Figure 3 ​As shown, it includes an input layer and an output layer. The input layer contains 256 neurons, and the output layer contains 4 output neurons. Each output neuron uses the Softmax function as the activation function to calculate the probability of the patient belonging to a normal placenta, accreta placenta, accreta placenta, and percreta placenta. During the training process, the loss is calculated using formula (6) for the training set samples. In this specific embodiment, a stochastic gradient descent optimizer is used to calculate the gradient of the loss with respect to the model parameters through back propagation, thereby updating the network parameters. When the loss on the validation set does not decrease for 20 consecutive rounds or the training reaches the specified round, the model training is completed. The encoder of the model with the smallest loss on the validation set is used as the ultrasound image encoder for subsequent end-to-end training.

[0049] The specific steps of the third phase are as follows:

[0050] Step 5: Use the dataset constructed in step 2 to train a multimodal deep fusion model for the graded diagnosis of placenta accreta disease. The structure of the multimodal model is shown in the attached figure. Figure 1 As shown, it includes a numerical feature encoder, an ultrasound image encoder, a fusion feature encoder, and a fusion feature classifier; the numerical feature encoder and the ultrasound image encoder here are the numerical feature encoder trained in step 3 and the ultrasound image encoder trained in step 4, respectively. During the multimodal model training process, the model parameters of the numerical feature encoder and the ultrasound image encoder remain unchanged; the numerical features and ultrasound images of the patients in the sample are input into the numerical feature encoder and the ultrasound image encoder respectively to obtain the numerical feature representation vector and the image representation vector. In this specific embodiment, the dimensions of the two representation vectors are both 256; the two representation vectors are spliced ​​into a vector with a dimension of 512, which is input into the fusion feature encoder, and the fusion feature representation vector with a dimension of 512 is output; in this specific embodiment, the structure of the fusion feature encoder is as shown in the attached figure. Figure 4 As shown, it consists of 4 stacked multi-head self-attention modules, and the number of self-attention heads in a single multi-head self-attention module is set to 4; the fused feature representation vector is input into the fused feature classifier. In this specific embodiment, the fused feature classifier is implemented using a single-layer perceptron, including an input layer and an output layer. The input layer contains 512 neurons, and the output layer contains 4 output neurons. Each output neuron uses the Softmax function as an activation function to calculate the probability that the patient has a normal placenta, accreta placenta, accreta placenta, and percreta placenta respectively;

[0051] When training the multi-modal model, for the training set samples, the loss is calculated by formula (6). In the specific embodiment, the random gradient descent optimizer is used, the gradient of the loss with respect to the model parameters is calculated by back propagation, and then the network parameters are updated. When the loss on the validation set does not decrease for 20 consecutive rounds or the training reaches the specified round, the model training is completed; the model with the minimum loss on the validation set is used as the final deployment model;

[0052] Step 6: The ultrasound image, imaging examination result and basic information of the patient are preprocessed according to step 2 to obtain the numerical features and the ultrasound image conforming to the input of the multi-modal model of the patient, and the two are input into the multi-modal deep fusion model trained in step 5, the model outputs the probability distribution of the placenta implantation disease grade of the patient, assuming (0.5, 0.2, 0.2, 0.1), and the grade with the maximum probability is taken as the diagnosis grade of the patient, and here the final recognition result of the patient is normal placenta.

[0053] Although the above describes the specific embodiments of the present application in order to facilitate the understanding of the present application by those skilled in the art, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

Claims

1. A method for grading and diagnosing placenta accreta disease based on a multimodal deep fusion model, characterized in that: The steps include: The first stage is used to construct a multimodal dataset, which includes two steps: collecting patient information and building a dataset for training a graded diagnostic model for placenta accreta disease. The second stage uses the dataset constructed in the first stage to train the numerical feature encoder and the ultrasound image encoder respectively, which includes two steps: training the numerical feature encoder and training the ultrasound image encoder; The third stage uses the numerical feature encoder and ultrasound image encoder trained in the second stage to train the multimodal deep fusion model.

2. A method for grading placenta accreta disease based on a multimodal deep fusion model according to claim 1, characterized in that: The process of collecting patient information in the first stage includes: collecting the patient's basic information, ultrasound images, imaging examination results and pathological examination results; basic information includes but is not limited to: age, weight, height, number of pregnancies, number of pregnancies in this pregnancy, number of previous cesarean sections, number of previous induced abortions, etc.; the ultrasound image is an abdominal ultrasound image of the patient after 28 weeks of pregnancy, which is intercepted by the ultrasound doctor to ensure that the image contains the placenta and uterine myometrium area; the imaging examination results include the diagnosis of the placental position; the pathological examination results are the results of pathological examination of the placental villi, uterine decidua and myometrial tissue in the patient's placental attachment area after surgery, which are used as the gold standard for the patient's placenta implantation level in the present invention.

3. The method for grading placenta accreta disease based on a multimodal deep fusion model according to claim 1, characterized in that: The process of constructing a data set for training a graded diagnostic model for placenta accreta in the first stage includes: constructing a sample for each patient, wherein the sample features include the patient's ultrasound image and numerical representation of clinical information; the ultrasound image needs to be preprocessed to a fixed size of M×M pixels that meets the model input, and the pixel values ​​are normalized; in the present invention, the clinical information included in the sample includes age, number of previous cesarean sections, number of previous artificial abortions, and placental position in the ultrasound examination results; the calculation formula for age features is as follows (1), the calculation formula for number of previous cesarean sections features is as follows (2), the calculation formula for number of previous artificial abortions features is as follows (3), and the calculation formula for placental position features is as follows (4); here, the diagnostic results of placental position are divided into six categories: non-placenta previa, low-lying placenta, marginal placenta previa, partial placenta previa, complete placenta previa, and pernicious placenta previa; for example, the numerical features of the patient are (0, 1, 0, 4) The patient is 35 years old or younger, has undergone a cesarean section but not an abortion, and the ultrasound examination shows complete placenta previa; （1） （2） （3） （4） The sample labels are the results of pathological examination, which are divided into four levels: normal placenta, placenta accreta, placenta accreta, and placenta percreta. The preprocessing is done using formula (5): （5） After the dataset is constructed, it is divided into training set and validation set according to a certain ratio. The training set is used to train model parameters, and the validation set is used for model selection.

4. The method for grading placenta accreta disease based on a multimodal deep fusion model according to claim 1, characterized in that: The process of training the numerical feature encoder in the second stage includes: using the data set constructed in the first stage to train the numerical feature encoder, and only using the numerical features and labels of the samples during the training process; the structure of the numerical feature encoder training model includes a numerical feature encoder and a numerical feature classifier; the numerical feature encoder is used to learn information closely related to placenta accreta diseases in clinical data, the patient's numerical features are input into the numerical feature encoder, and the output is the patient's numerical feature representation vector; the structure of the numerical feature encoder can adopt but is not limited to a multi-layer perceptron; the classifier is used to predict the patient's placenta accreta grade based on the numerical feature representation vector output by the numerical feature encoder; the classifier outputs a vector containing 4 elements ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, represents the probability that the patient has placenta percreta; The classifier is implemented using a single-layer perceptron; The model is trained using the stochastic gradient descent algorithm. The model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the probability distribution of placenta accreta disease classification output by the numerical feature encoder training model proposed in the present invention for patient i, and L represents the number of patient samples in the training set; （6） When the loss on the validation set does not decrease for T consecutive rounds or the training reaches the specified round, the model training is completed. The model with the smallest loss on the validation set is selected, and the numerical feature encoder in the model is used as the numerical feature encoder in the multimodal model in the third stage.

5. The method for grading placenta accreta disease based on a multimodal deep fusion model according to claim 1, characterized in that: The process of training the ultrasound image encoder in the second stage includes: using the data set constructed in the first stage to train the ultrasound image encoder, and only using the ultrasound images and labels of the samples during the training process; the structure of the ultrasound image encoder training model includes an ultrasound image encoder and a classifier; the ultrasound image encoder is used to learn information closely related to placenta accreta disease in the ultrasound image, and the patient's ultrasound image is input to the ultrasound image encoder, and the output is an image feature representation vector; the structure of the ultrasound image encoder can be implemented using but not limited to convolutional neural networks, residual networks, encoders of medical image basic models, and other structures; the classifier is used to predict the level of the patient's placenta accreta disease based on the image feature representation vector output by the ultrasound image encoder; the classifier outputs a vector containing 4 elements ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, represents the probability that the patient has placenta percreta; The classifier is implemented using a single-layer perceptron; The model is trained using the stochastic gradient descent algorithm, and the model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the hierarchical probability distribution output by the ultrasound image encoder training model proposed in the present invention for patient i, and L represents the number of patient samples in the training set; When the loss on the validation set does not decrease for T consecutive rounds or the training reaches the specified round, the model training is completed. The model with the smallest loss on the validation set is selected, and the ultrasound image encoder in the model is used as the ultrasound image encoder in the multimodal model in the third stage.

6. The method for grading placenta accreta disease based on a multimodal deep fusion model according to claim 1, characterized in that: The process of training the multimodal deep fusion model in the third stage includes: using the dataset constructed in the first stage to train the multimodal deep fusion model for the graded diagnosis of placenta accreta disease. The structure of the multimodal model includes a numerical feature encoder, an ultrasound image encoder, a fusion feature encoder, and a fusion feature classifier. The numerical feature encoder and ultrasound image encoder here are respectively the numerical feature encoder and ultrasound image encoder trained in the second stage. During the multimodal model training process, the model parameters of the numerical feature encoder and ultrasound image encoder remain unchanged. The numerical features and ultrasound images of the patients in the sample are input into the numerical feature encoder and the ultrasound image encoder respectively to obtain the numerical feature representation vector and the image representation vector; the two representation vectors are input into the fusion feature encoder to calculate the fusion feature representation vector after deep fusion; The structure of the fusion feature encoder is shown in Figure 4. It consists of 4 stacked multi-head self-attention modules. The number of self-attention heads in a single multi-head self-attention module is set to 4. The fusion feature representation vector is input to the fusion feature classifier, and the classifier outputs a vector containing 4 elements. ,element Indicates the probability that the patient's placenta is normal, Indicates the probability that the patient has placenta accreta, represents the probability that the patient has placenta accreta, represents the probability that the patient has placenta percreta; The classifier is implemented using a single-layer perceptron; The model is trained using the stochastic gradient descent algorithm. The model loss is calculated as shown in formula (6): is the hierarchical diagnostic label of patient i, is the hierarchical probability distribution output by the multimodal model proposed in this invention for patient i, and L represents the number of patient samples in the training set; When the loss on the validation set does not decrease for T consecutive rounds or reaches the specified round of training, model training is completed, and the model with the smallest loss on the validation set is selected as the final deployment model for the graded diagnosis of placenta accreta.

Images