Delivery time prediction method and system combining ultrasonic image and clinical information

Abstract

The invention discloses a delivery time prediction method and system in combination with an ultrasonic image and clinical information, and relates to the field of medical image analysis and artificial intelligence, and the method comprises the steps: automatically extracting image features related to natural delivery time from an antenatal care ultrasonic image of a pregnant woman in the late pregnancy period; performing characterization processing on the clinical information of the pregnant woman to obtain clinical features related to natural delivery time; through a multi-modal fusion strategy, carrying out fusion processing on the image features and the clinical features to obtain fusion features with better delivery time predictability; and predicting the natural childbirth time of the pregnant woman based on the fusion features to obtain a natural childbirth time prediction result of the pregnant woman. Compared with a traditional expected date-of-confinement prediction method, the method combines the automatically extracted ultrasonic features with the clinical features, can more accurately predict the occurrence time of natural delivery, is beneficial to resource allocation of medical institutions, and is beneficial to alleviation of anxiety mood of pregnant women.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image analysis and artificial intelligence, and in particular to a delivery time prediction method and system combining ultrasound images and clinical information. BACKGROUND

[0002] Accurate prediction of delivery time has multiple key values: in clinical diagnosis and treatment, it can help to identify high-risk pregnancies as early as possible and start intervention measures in time, thereby reducing the risk of premature birth and related maternal and infant complications; for medical institutions, accurate prediction of delivery time can optimize obstetric resource allocation, including reasonable scheduling of hospital beds, medical equipment and medical staff, and improve the preparation efficiency for emergency situations such as premature birth and emergency cesarean section; for pregnant women, the uncertainty of delivery time often causes anxiety and psychological stress, and reliable prediction technology can help them plan hospital arrangements scientifically, thereby enhancing their psychological safety and improving the delivery experience.

[0003] The prior art has obvious limitations in predicting delivery time:

[0004] In traditional methods, the due date is calculated according to the standard period of 40 weeks (280 days) based on the gestational age determined in the early stage of pregnancy, and the determination of gestational age needs to combine artificially extracted ultrasound features and the last menstrual period. The accuracy of this method is significantly affected by individual differences, such as fluctuations in the menstrual cycle of pregnant women and individual differences in fetal growth patterns, which can cause a large deviation between the predicted result and the actual delivery time. In addition, traditional ultrasound analysis relies on artificial extraction of features including crown-rump length (CRL), biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), and femur length (FL), which is not only inefficient but also subject to the influence of the operator's experience and is highly subjective.

[0005] Some other methods attempt to predict delivery time through indicators such as cervical length and placental maturity. However, cervical length has some reference value for predicting premature birth, but its accuracy for predicting full-term delivery is limited; the accuracy of placental maturity in predicting delivery time is also limited, making it difficult to accurately associate delivery time.

[0006] In summary, there is still a lack of a technology that can accurately predict the natural delivery time (i.e., the time of normal delivery). SUMMARY

[0007] The present application provides a delivery time prediction method and system combining ultrasound images and clinical information to accurately predict the natural delivery time.

[0008] The application provides a delivery time prediction method combining ultrasound images and clinical information, comprising: automatically extracting image features related to natural delivery time from a pregnant woman's third trimester ultrasound image; processing the clinical information of the pregnant woman to obtain clinical features related to natural delivery time; fusing the image features and the clinical features through a multi-modal fusion strategy to obtain fusion features with better delivery time prediction ability; and predicting the natural delivery time of the pregnant woman based on the fusion features to obtain the natural delivery time prediction result of the pregnant woman.

[0009] Preferably, the automatic extraction of image features related to natural delivery time from the pregnant woman's third trimester ultrasound image comprises: using an image feature extraction network to automatically extract high-dimensional image features from the pregnant woman's third trimester ultrasound image, and determining the extracted high-dimensional image features as the image features related to natural delivery time; wherein the image feature extraction network adopts at least one of a convolutional neural network, a residual network, a dense connection network, and a hybrid neural network combining convolution operation and self-attention mechanism.

[0010] Preferably, the clinical information of the pregnant woman includes numerical clinical information, and correspondingly, the processing of the clinical information of the pregnant woman to obtain clinical features related to natural delivery time comprises: directly taking the numerical clinical information after standardization and normalization processing as the clinical features, or using a fully connected neural network or a self-attention mechanism to feature encode and embed the numerical clinical information after standardization and normalization processing to obtain the clinical features of the numerical clinical information.

[0011] Preferably, the clinical information of the pregnant woman also includes categorical clinical information, and correspondingly, the processing of the clinical information of the pregnant woman to obtain clinical features related to natural delivery time comprises: using an embedding layer to map the categorical clinical information into a dense vector, and determining the dense vector as the clinical features of the categorical clinical information.

[0012] Preferably, the fusion processing of the image features and the clinical features through a multi-modal fusion strategy to obtain fusion features with better delivery time prediction ability comprises: directly splicing the image features and the clinical features to obtain the fusion features; or, after directly splicing the image features and the clinical features, automatically learning the weight distribution of the two modal features through an attention mechanism to obtain the fusion features; or, fusing the image features and the clinical features through at least one of a fully connected layer, a multilayer perceptron, and an attention mechanism to obtain the fusion features.

[0013] Preferably, the natural delivery time prediction result comprises whether natural delivery within one week and / or the number of days to natural delivery, and accordingly, the predicting the natural delivery time of the pregnant woman based on the fusion feature comprises: inputting the fusion feature into a full connection layer for nonlinear mapping, and then outputting a probability value of natural delivery within one week of the pregnant woman through a classification prediction output layer, so as to determine whether the pregnant woman delivers within one week by comparing the probability value with a preset threshold; and / or inputting the fusion feature into a full connection layer for nonlinear mapping, and then outputting the number of days to natural delivery of the pregnant woman through a regression prediction output layer, so as to determine the delivery date of the pregnant woman according to the gestational week and the number of days to natural delivery of the pregnant woman, or determine the delivery date of the pregnant woman according to the current date and the number of days to natural delivery of the pregnant woman.

[0014] Preferably, the method further comprises: generating a heat map based on a Grad-CAM method; adjusting the size of the heat map to be consistent with the third trimester ultrasound image input into the image feature extraction network; and using a pseudo-color heat map superimposition technology to superimpose the size-adjusted heat map on the third trimester ultrasound image, so as to intuitively present the region in the third trimester ultrasound image that plays a key role in the natural delivery time prediction result.

[0015] The present application provides a delivery time prediction system combining ultrasound images and clinical information, comprising: an image feature extraction unit for automatically extracting image features related to natural delivery time from a third trimester ultrasound image of a pregnant woman; a clinical information feature extraction unit for feature extraction processing of clinical information of the pregnant woman to obtain clinical features related to natural delivery time; a feature fusion module for fusion processing of the image features and the clinical features through a multi-modal fusion strategy to obtain fusion features with better delivery time prediction performance; and a delivery time prediction module for predicting the natural delivery time of the pregnant woman based on the fusion features to obtain a natural delivery time prediction result of the pregnant woman.

[0016] Preferably, the natural delivery time prediction result is whether natural delivery within one week and / or the number of days to natural delivery, the delivery time prediction module comprises: a deep learning unit, configured to input the fusion features into a full connection layer for nonlinear mapping, and then output a probability value of natural delivery of the pregnant woman within one week through a classification prediction output layer, and / or input the fusion features into a full connection layer for nonlinear mapping, and then output the number of days to natural delivery of the pregnant woman through a regression prediction output layer; and a result output unit, configured to determine whether the pregnant woman delivers within one week by comparing the probability value with a preset threshold, and / or determine the delivery date of the pregnant woman according to the gestational week and the number of days to natural delivery or according to the current date and the number of days to natural delivery.

[0017] Preferably, the system further comprises an interpretability module, configured to generate a heat map based on a Grad-CAM method, adjust the size of the heat map to be consistent with the third trimester antenatal ultrasound image input into the image feature extraction network, and superimpose the size-adjusted heat map on the third trimester antenatal ultrasound image by using a pseudo-color heat map superimposition technology, so as to intuitively present the region in the third trimester antenatal ultrasound image that plays a key role in the natural delivery time prediction result.

[0018] The present application combines automatically extracted ultrasound features with clinical features, can more accurately predict the occurrence time of natural delivery, is beneficial to the resource allocation of medical institutions, is beneficial to alleviate the anxiety of pregnant women, and in addition, does not need to manually extract information about head circumference, abdominal circumference and the like in the ultrasound image, but extracts high-dimensional image features, so that the prediction result is low in subjectivity, high in efficiency, and can reduce the burden of ultrasound doctors. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of a delivery time prediction method combining ultrasound images and clinical information provided by the present application;

[0020] Figure 2 is a framework diagram of a delivery time prediction system combining ultrasound images and clinical information provided by the present application;

[0021] Figure 3 is Figure 2 is a structural schematic diagram of the feature fusion module;

[0022] Figure 4 is Figure 3 is an output schematic diagram of the interpretability module shown in the figure;

[0023] Figure 5 is a scale diagram of the training set and the validation set provided by the application example of the present application;

[0024] Figure 6a ,Figure 6b , Figure 6c are ROC curve plots of independent verification sets, ROC curve plots of late pregnancy subgroups in independent verification sets, and ROC curve plots of preterm birth subgroups in independent verification sets, respectively;

[0025] Figure 7 are visual fetal brain region explainable heat maps using the Grad-CAM method, A, B, C, and D show four fetal brain ultrasound features for predicting the start time of natural childbirth, respectively;

[0026] Figure 8 are visual fetal internal organ region explainable heat maps using the Grad-CAM method, A, B, C, and D show four fetal internal organ regions, which are liver region, liver and intestine, intestine, and intestine, respectively. DETAILED DESCRIPTION

[0027] The embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the following described embodiments are only used to illustrate and explain the present application, and are not intended to limit the present application.

[0028] Referring to Figure 1 , a delivery time prediction method combining ultrasound images and clinical information includes:

[0029] Step S101: automatically extracting image features related to natural childbirth time from the pregnant woman's late pregnancy ultrasound images.

[0030] In the present application, an image feature extraction network is used to automatically extract high-dimensional image features from the pregnant woman's late pregnancy ultrasound images, and the extracted high-dimensional image features are determined as image features related to natural childbirth time; wherein the image feature extraction network uses at least one of a convolutional neural network, a residual network, a dense connection network, and a hybrid neural network combining convolution operation and self-attention mechanism.

[0031] It should be noted that the image features extracted by the image feature extraction network of the present application are not the head-hip length (CRL), biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), femur length (FL), etc. measured manually by an ultrasound doctor in the traditional sense, but a kind of high-dimensional image features (actually a group of numerical values) learned automatically from ultrasound images by the above deep learning network. These features comprehensively reflect multiple factors related to natural childbirth time, such as fetal development, amniotic fluid status, and placental position.

[0032] Step S102: feature processing of the clinical information of the pregnant woman to obtain clinical features related to natural childbirth time.

[0033] The clinical information of the pregnant woman includes numerical clinical information, such as gestational age. The present invention can directly use the standardized and normalized numerical clinical information as clinical features, or use a fully connected neural network or a self-attention mechanism to feature encode and embed the standardized and normalized numerical clinical information to obtain the clinical features of the numerical clinical information.

[0034] The clinical information of the pregnant woman also includes categorical clinical information, such as pregnancy and delivery history. The present invention can use an embedding layer to map the categorical clinical information into a dense vector, and determine the dense vector as the clinical feature of the categorical clinical information.

[0035] Step S103: The imaging features and the clinical features are fused through a multimodal fusion strategy to obtain fused features with greater predictive power for delivery time.

[0036] The present invention can directly splice the image features and the clinical features to obtain the fused features, or after directly splicing the image features and the clinical features, automatically learn the weight distribution of the two modal features through an attention mechanism to obtain the fused features, or fuse the image features and the clinical features through at least one of a fully connected layer, a multi-layer perceptron, and an attention mechanism to obtain the fused features.

[0037] Step S104: predicting the natural delivery time of the pregnant woman based on the fusion features to obtain a natural delivery time prediction result of the pregnant woman.

[0038] The natural delivery time prediction result is whether the natural delivery will occur within a week and / or the number of days to the natural delivery.

[0039] In one embodiment, the natural delivery time prediction result is whether the delivery will be within a week. Specifically, the fusion feature is input into the fully connected layer for nonlinear mapping, for example, from 2048 dimensions → 128 dimensions → 1 dimension, and then through the classification prediction output layer, which can use a Softmax function or a Sigmoid function to obtain the probability value of the pregnant woman giving birth within a week. After obtaining the probability value, it can be determined whether the pregnant woman will give birth within a week by comparing the probability value with a preset threshold. Taking the preset threshold of 0.5 as an example, if the probability value is greater than 0.5, it is determined that the pregnant woman will give birth within a week, otherwise she will not give birth within a week. In this embodiment, the classification loss can be a cross entropy loss.

[0040] In another embodiment, the natural delivery time prediction result is the number of days to natural delivery. Specifically, the fusion features are input into a fully connected layer for nonlinear mapping, for example, 2048 dimensions to 64 dimensions to 1 dimension, and then output by a regression prediction layer, which can be a single neuron or a linear activation function, to obtain the number of days to natural delivery of the pregnant woman, for example, delivery is expected in 5 days. After obtaining the number of days to natural delivery, the delivery date of the pregnant woman is determined according to the gestational age and the number of days to natural delivery of the pregnant woman, for example, 38+3 weeks, i.e., delivery at 38 weeks and 3 days of pregnancy, or the delivery date of the pregnant woman is determined according to the current date and the number of days to natural delivery of the pregnant woman, for example, the expected delivery date is July 31, 2025. In this embodiment, the regression loss can be MSE loss.

[0041] In yet another embodiment, the natural delivery time prediction result includes whether natural delivery is within one week and the number of days to natural delivery. Specifically, the fully connected layer is branched into two independent sub-networks, one is a classification branch fully connected layer, the fusion features are input into the classification branch fully connected layer for nonlinear mapping, and then output by a classification prediction output layer to output a probability value to determine whether the pregnant woman will deliver within one week, and the other is a regression branch fully connected layer, the fusion features are input into the regression branch fully connected layer for nonlinear mapping, and then output by a regression prediction output layer to output the number of days to natural delivery of the pregnant woman, so as to obtain the delivery date of the pregnant woman based on the number of days to natural delivery of the pregnant woman. In this way, the two independent branch fully connected layers will independently learn and optimize features according to the characteristics of their respective prediction tasks, avoiding conflicts between prediction tasks. In this embodiment, the total loss can be the weighted sum of the classification loss (such as cross-entropy loss) and the regression loss (such as MSE loss), which are optimized simultaneously through backpropagation.

[0042] Clinically, the expected date of delivery is generally estimated according to 40 weeks (280 days), however, the accuracy of this method is often limited by individual menstrual cycles and changes in fetal growth patterns, has individual differences, and traditional ultrasound analysis relies on manual feature extraction of crown-rump length (CRL), biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), femur length (FL), etc., which is low in efficiency and subjective. The present application uses machine learning network / deep learning network to automatically extract image features related to natural delivery time based on antenatal ultrasound images, and combines clinical features such as gestational age to predict the natural delivery time of pregnant women, which fully utilizes the image features in antenatal ultrasound, avoids the subjectivity of manual measurement by ultrasound physicians, and improves the accuracy of natural delivery time prediction.

[0043] Further, in order to enhance the clinical interpretability, the present application introduces an interpretability module (or Grad-CAM visualization module), which first generates a heat map based on the Grad-CAM method. Specifically, the last convolutional layer for convolutional feature extraction in the image feature extraction network is selected, and while obtaining the natural delivery time prediction result through forward propagation, the gradient of the last convolutional layer corresponding to the natural delivery time prediction result is obtained by performing back propagation. The average response value of each channel feature map is obtained by performing global average pooling on the spatial dimension of the gradient of the last convolutional layer corresponding to the natural delivery time prediction result. The average response value of each channel feature map is used as the weight of each channel feature map, and the heat map is obtained according to each channel feature map and the corresponding weight. Then, the size of the heat map is adjusted to be consistent with the third trimester antenatal ultrasound image input into the image feature extraction network, and a pseudo-color heat map superposition technology is used to superimpose the size-adjusted heat map on the third trimester antenatal ultrasound image, so as to intuitively present the region in the third trimester antenatal ultrasound image that plays a key role in the natural delivery time prediction result. The results show that the method used in the present application focuses on key parts such as the fetal head, fetal liver and intestinal canal, suggesting that the extracted features are related to the trend of changes in these structures, thereby assisting in the prediction of natural delivery time.

[0044] The use of the interpretability module can display the key areas for prediction, provide visual basis for the prediction result, and increase the trust of doctors and pregnant women.

[0045] Referring to Figure 2 A delivery time prediction system combining ultrasound images and clinical information includes a data acquisition module, a data preprocessing module, a feature extraction module including an image feature extraction unit and a clinical information feature unit, a feature fusion module, a delivery time prediction module including a deep learning unit and a result output unit, and shows the process of data acquisition, preprocessing, feature extraction, fusion and prediction. Among them,

[0046] 1. The data acquisition module is used for collecting ultrasound images and clinical information (or clinical data, clinical indicators) of pregnant women.

[0047] The module acquires the late trimester antenatal examination ultrasound image of a pregnant woman and clinical information, which includes gestational age, and can also include gestational age, age, pregnancy history, blood pressure, blood glucose, BMI, laboratory test data, lifestyle data, and genetic test data, etc. Among them, the laboratory test data includes blood routine (white blood cell count, platelet count), biochemical indicators (liver function, kidney function), sugar metabolism indicators (fasting blood glucose), etc. The lifestyle data includes dietary habits (high salt and high sugar), exercise frequency, sleep quality, smoking and drinking history, etc. The genetic test data includes chromosomal test results, single gene disease carrier screening results, etc.

[0048] 2、The data preprocessing module is used for standardization and normalization processing of data (i.e. ultrasound image and clinical information).

[0049] For ultrasound images, the module performs image format conversion, size scaling and cropping, pixel value standardization and normalization, data enhancement, etc.

[0050] For clinical information, the module performs missing value filling, outlier detection and processing, and standardization and normalization processing of numerical clinical information.

[0051] 3、Feature extraction module, using deep learning model (ResNet, etc.) to automatically extract high-dimensional features from ultrasound images, and to encode and embed the features of clinical information, see Figure 2 , the extraction of image features and clinical features uses two independent channels, respectively for feature representation of different modal data.

[0052] The image feature extraction unit of the feature extraction module is configured to automatically extract image features related to the natural delivery time from the third-trimester antenatal ultrasound image of the pregnant woman. Specifically, the image features are extracted from the multi-view ultrasound image by a deep learning model. The deep learning model can be implemented by the feature extraction part of the residual network ResNet50, which includes a plurality of residual blocks and convolutional layers. Similar network model structures exist, and without changing the overall structure of the system, the following types of networks can be used as alternatives: 1. Convolutional neural network (CNN) for extracting multi-level and high-dimensional features of the image, including edge, texture, shape, spatial relationship, and other high-level semantic information; 2. Residual network and dense connection network for extracting deep features and multi-scale features; 3. Transformer network (Transformer), including visual transformer (ViT) and SwinTransformer, for global context feature modeling and long-distance dependency capturing; 4. Hybrid neural network that combines convolution operation and self-attention mechanism, taking into account local details and global context information. In addition, multiple network models can be used in parallel or cascaded to enhance feature expression capability.

[0053] The clinical information feature extraction unit of the feature extraction module is configured to process the clinical information of the pregnant woman to obtain clinical features related to the natural delivery time. The clinical information of the pregnant woman includes numerical clinical information, and the clinical information feature extraction unit can directly use the standardized numerical clinical information as the clinical features, such as the number of gestational weeks. After standardization, the numerical information is directly used as a numerical input to splice with the image features. Alternatively, a fully connected neural network (MLP) or a self-attention mechanism can be used for feature encoding and embedding to obtain the clinical features of the numerical clinical information and the image features. The clinical information of the pregnant woman can also include categorical clinical information, such as pregnancy history, lifestyle, and medical history. The clinical information feature extraction unit uses an embedding layer to map the categorical features to a dense vector, such as a high-dimensional dense vector, to capture the semantic relationship between the features and achieve semantic expression.

[0054] 4. The feature fusion module is configured to fuse the image features and the clinical features by a multi-modal fusion strategy to obtain fusion features with better delivery time prediction capability.

[0055] In one embodiment, referring to Figure 3The feature fusion module fuses the image feature vector and the clinical feature vector by splicing, i.e., splicing the high-dimensional feature vector F_img output by the image feature extraction unit and the clinical feature vector F_clinical output by the clinical information feature extraction unit to obtain a fusion feature vector F_concat=[F_img;F_clinical]. This method is simple to implement and has high computational efficiency, and has achieved good results in actual experiments.

[0056] In another embodiment, to enhance the scalability of the system, one or more of the following fusion methods can also be used: 1. Fully connected layer (FC) fusion: image features and clinical features are spliced, weighted summed, or differentially combined through shared or independent fully connected layers, and then nonlinearly transformed and classified; 2. Multi-layer perception (MLP) fusion: different modal features are deeply fused and interactively modeled through multi-layer nonlinear mapping; 3. Attention mechanism fusion: including self-attention mechanism, cross-modal attention mechanism, etc., for adaptively adjusting the weights of each feature and realizing feature alignment and complementarity; 4. Feature splicing and mapping: image features and clinical features are spliced, weighted fused, and differentially mapped, and then high-dimensional space mapping and nonlinear transformation are performed through a fully connected layer or MLP.

[0057] 5. A delivery time prediction module for predicting the natural delivery time of the pregnant woman based on the fusion features, to obtain a natural delivery time prediction result of the pregnant woman. The natural delivery time prediction result includes whether to deliver naturally within a week and / or the number of days to natural delivery.

[0058] The delivery time prediction module includes a deep learning unit and a result output unit. The deep learning unit performs nonlinear mapping, classification / regression delivery time prediction on the fused features, and outputs prediction results such as probability values and / or numerical variables. The result output unit is used to post-process and interpret the output results of the deep learning unit, and convert them into prediction information understandable by users, i.e., generate a natural delivery time prediction result including a classification variable (such as whether to deliver within a week) and / or a continuous variable (such as a specific delivery period, i.e., the delivery date of the pregnant woman or the number of days to the prediction point, e.g., "expected to deliver at 38+3 weeks" or "expected to deliver in 6.2 days" etc.), for example: threshold judgment (such as >0.5 for "imminent delivery") on the probability value output by Sigmoid, and formatting (such as days to date) of the delivery time output by regression, the output results of the result output unit can be written into the system interface for doctors to view or used for report printing.

[0059] Obviously, the present application supports two output forms, one or both of which is selected according to different prediction tasks. When the prediction task is a classification prediction task, whether the purpose will be natural childbirth within a week, it belongs to a binary classification task. Specifically, the deep learning unit includes a fully connected layer (FC) and a classification prediction function, the fused features are input to the fully connected layer (FC) for nonlinear mapping, and finally the classification prediction output layer adopting a Softmax function or a Sigmoid function is used to output the natural childbirth time prediction result, i.e. the probability value of the pregnant woman delivering within a week. Correspondingly, the result output unit is used to determine whether the pregnant woman delivers within a week by comparing the probability value with a preset threshold. When the prediction task is a regression prediction task, the purpose is to determine the number of days from natural childbirth, and the output is a floating point number, which can be expressed as "expected to deliver in x days". Specifically, the deep learning unit includes a fully connected layer (FC) and a classification prediction function, the fused features are input to the fully connected layer for nonlinear mapping, and finally the regression prediction output layer is used to output the number of days from natural childbirth. Correspondingly, the result output unit determines the delivery date of the pregnant woman according to the gestational age and the number of days from natural childbirth of the pregnant woman or determines the delivery date of the pregnant woman according to the current date and the number of days from natural childbirth of the pregnant woman. When the prediction task is a classification prediction task and a regression prediction task, the two independent branches can predict the respective prediction tasks, avoiding the conflict of the prediction task key.

[0060] Further, referring to Figure 2 A delivery time prediction system combining ultrasound images and clinical information further comprises an explainability module which generates an explainability heat map based on a Grad-CAM (Gradient-weighted Class Activation Mapping) method for locating and identifying the key areas in the visualized ultrasound image that play a key role in the prediction result, and providing a visual basis for the predicted result. Functionally, the explainability module can include the following modules: a heat map calculation unit for obtaining gradient information of the last convolution layer and calculating the weight of each feature map; a heat map generation unit for calculating the heat map of the region activation according to the weight and the feature map, and adjusting it to the same size as the original image; a visualization unit for superimposing the generated heat map on the original ultrasound image in a pseudo-color manner, and providing dynamic visualization functions such as scrolling, zooming in, rotating and the like.

[0061] Specifically, to locate and identify the region (referred to as key region) in the visual ultrasound image which plays a key role in predicting the result, the explainability module first selects the last convolutional layer in the network, which usually retains strong spatial information, and performs back propagation to obtain the gradient corresponding to the prediction category while performing forward propagation to obtain the prediction result; then, the gradient corresponding to the category is globally averaged pooled in the spatial dimension to obtain the weight of each channel, which represents the importance of each channel to the prediction result, and then the weights are multiplied by the corresponding convolution feature map and added in the channel dimension to obtain a heat map; the size of the heat map is adjusted to be consistent with the third trimester ultrasound image input into the image feature extraction network, for example, the heat map is activated by ReLU (retaining the positive correlation part) and interpolated to the original ultrasound image size, and the size-adjusted heat map is superimposed on the third trimester ultrasound image using the pseudo-color heat map superimposition technology, to visually present the region in the third trimester ultrasound image which plays a key role in predicting the natural delivery time, and express the most critical image region for the prediction result. Obviously, the aforementioned "key region" is the convolution feature region that mainly contributes to the current prediction task in network decision-making. Taking the ResNet50 network structure as an example, the "last convolutional layer" refers to the layer4 module in ResNet50, i.e. the fourth stage residual block, and the output features of which are usually considered as the last layer of convolutional features that retain spatial information. The output dimension of this convolutional layer is usually [batch_size, 2048, H, W], where 2048 represents the number of channels, and HxW represents the spatial size. Each channel corresponds to a "feature map" representing the response of the channel to different spatial regions of the image. In Grad-CAM, the gradient of the current prediction category on the 2048 feature maps is globally averaged pooled to obtain the weight coefficient of each feature map, and then a weighted sum heat map is generated to finally reflect the region in the image which plays a key role in the current prediction task. Referring to Figure 4 , the left side is the original ultrasound image, and the right side is the heat map, and the red part in the heat map is the high contribution region.

[0062] The present application uses the Grad-CAM method to visualize the heat map of the features extracted by the image feature extraction unit, and the heat map can be located to the high contribution region, such as the fetal brain, fetal internal organs, etc., which reflects the correlation between the features extracted by the image feature extraction unit and the fetal development and amniotic fluid volume.

[0063] The system of the present application can effectively improve the accuracy of natural delivery time prediction and provide strong decision support for the clinical management of pregnant women.

[0064] The training method provided by the application example will be further explained below in combination with specific examples.

[0065] 1. Obtain training samples.

[0066] Single-center, multi-machine pregnancy ultrasound images were collected as sample images, and the ultrasound images covered multiple view images of fetal head, trunk, internal organs, amniotic fluid, etc. Each time of ultrasound image of each pregnant woman was saved in a folder. The natural delivery time of the pregnant woman was collected, and the data set was preprocessed. The task of the present application example was a classification prediction task, so a classification label was used, that is, whether delivery occurred within one week after ultrasound examination was used as a label. A sample image data set was established, and the name, age, and other information were hidden. The images with poor image quality were excluded, and the training set and the validation set were randomly divided according to a ratio of 7:3. The final training set included 26322 ultrasound images, and the independent validation set included 10736 ultrasound images, as shown in FIG. 1. The baseline information of the data is shown in Table 1. Figure 5

[0067] Table 1. Baseline table of original data

[0068]

[0069] 2. Delivery time prediction model construction.

[0070] In the application example, the image feature extraction unit: uses a ResNet50 model as the backbone structure, wherein the convolutional part (i.e., conv1 to layer4) of the ResNet50 network is used to extract multi-level and high-dimensional image features of the ultrasound image, which belongs to the feature extraction module. The clinical information feature unit: only the numerical clinical information “gestational age” is included, which is directly spliced into the feature fusion module after standardization. The feature fusion module: adopts an Early Fusion fusion strategy, first extracts and aggregates the features of the multi-view ultrasound image through the image feature extraction unit, and then splices the obtained image feature vector and the gestational age (Gestational Age) feature vector to form a unified fusion feature vector. The deep learning module: the spliced fusion features are sent to a fully connected layer (Linear) and output a binary classification result using a Sigmoid function, that is, whether to deliver naturally within one week, to complete the final prediction task. The result output module: the Sigmoid output probability is post-processed and can be converted into a Boolean label (whether to deliver within one week) for result display or interface calling. The explainability module: the Grad-CAM method is used to obtain the feature map and the gradient information of the prediction category from the layer4 layer of the ResNet50, and a visualized heat map is generated.

[0071] 3. Model training.

[0072] The ultrasound images and the clinical information (gestational age) were used for training. The trained initial deep learning model was used as a starting point, and the model was trained for 500 rounds. The optimal model was obtained according to the performance of the model. ​

[0073] 4. Model performance evaluation and validation

[0074] Model validation was performed using an independent validation set, and model performance was evaluated by ROC curve, accuracy, sensitivity, specificity, negative predictive value, positive predictive value, F1 score and kappa value.

[0075] The model performance is shown in Table 2, with an accuracy of 0.81 (95% CI: 0.77-0.84), a sensitivity of 0.87 (95% CI: 0.82-0.90), a specificity of 0.75 (95% CI: 0.70-0.80), a positive predictive value of 0.77 (95% CI: 0.72-0.81), a negative predictive value of 0.85 (95% CI: 0.81-0.89), a kappa value of 0.62 (95% CI: 0.56-0.68), and an F1 score of 0.82 (95% CI: 0.78-0.85).

[0076] Table 2. Model performance

[0077]

[0078] Validation was performed on an independent validation set, as well as on a late pregnancy subgroup and a preterm birth subgroup, and the results are shown in Table 3, with an AUC of 0.88 (95% CI: 0.85-0.91) for the late pregnancy group, an AUC of 0.86 (95% CI: 0.82-0.89) for the late pregnancy subgroup, and an AUC of 0.81 (95% CI: 0.68-0.91) for the preterm birth subgroup. Figure 6a Figure 6b Figure 6c Using the traditional 40-week (280-day) due date prediction method, the AUC in the independent validation set was 0.68 (95% CI: 0.65-0.71), the AUC in the late pregnancy subgroup was 0.61 (95% CI: 0.55-0.66), and the AUC in the preterm birth subgroup was 0.50 (95% CI: 0.50-0.50). Analysis of the results using the DeLong test showed that the prediction method in the present application was superior to the traditional 40-week (280-day) due date prediction method for validation of the independent validation set, as well as for the late pregnancy subgroup and the preterm birth subgroup (p<0.05).

[0079] It should be noted that due to the limited amount of data, early and mid-pregnancy data can be included in the negative class to learn features related to natural delivery. In the model training process, especially in the case of limited data, a small amount of early and mid-pregnancy data can be added for training. If the amount of data is sufficient, only late pregnancy data can be used for training. In the model application process, late pregnancy data is used for natural delivery time prediction.

[0080] ​​5. Explainability analysis.

[0081] Explainability analysis is performed on the prediction results using the CAM-Grad method, and the regions with prediction contribution (i.e., key regions) are visualized as shown in Figs. 8 and 9. Figure 7 and Figure 8 It is found that the regions with high contribution to the prediction results in the explainability heat map are concentrated in the brain region and the internal organ region of the fetus. This not only helps to explain why the model makes the prediction, but also indirectly verifies the rationality and reliability of the model results by comparing with the key anatomical structures judged by doctors, which helps to improve the trust of the prediction results by doctors and patients.

[0082] The explainability heat map not only provides a new perspective for ultrasound image analysis, but also provides potential insights for optimizing obstetric clinical practice. Moreover, in the case of good prediction accuracy, the visualization image of the explainability module can improve the rationality and reliability of the model, as evidence of the prediction results of the model.

[0083] The present application can also provide a computer storage medium having stored thereon a program of a delivery time prediction method combining ultrasound images and clinical information, which can implement the steps of the aforementioned delivery time prediction method combining ultrasound images and clinical information when executed by a processor.

[0084] The program can be written in any combination of one or more programming languages including, but not limited to, such as Java, C++, python, C language or similar programming languages. The program can be completely executed on the processor of the user computing device, partially executed on the processor of the user computing device, partially executed on the processor of a remote computing device, or completely executed on the processor of a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device using the Internet provided by an Internet service provider.

[0085] Compared with the traditional EDD prediction method, the present application can achieve the following technical effects:

[0086] 1. The present application can more accurately predict the occurrence time of natural delivery, which is beneficial for resource allocation of medical institutions and for alleviating the anxiety of pregnant women.

[0087] 2. The present application can automatically extract ultrasound features, without the need for ultrasound doctors to measure ultrasound parameters such as head and leg length (CRL), biparietal diameter (BPD), head circumference (HC), abdominal circumference (AC), and femur length (FL), which is low in subjectivity and high in efficiency, and can reduce the burden of ultrasound doctors.

[0088] 3. The prediction result of the present application has interpretability, and the predicted process can be visualized, which is more beneficial to clinical practice.

[0089] Although the present application is described in detail above, the present application is not limited thereto, and those skilled in the art can make various modifications according to the principles of the present application. Therefore, any modification made according to the principles of the present application should be understood to fall within the scope of the present application.

Claims

1. A method for predicting delivery time by combining ultrasound images and clinical information, characterized in that: The method comprises: Automatically extract image features related to the time of natural delivery from ultrasound images of pregnant women during late pregnancy; Characterizing the clinical information of the pregnant woman to obtain clinical characteristics related to the time of natural delivery; The imaging features and the clinical features are fused through a multimodal fusion strategy to obtain fusion features with greater predictive power for delivery time; The natural delivery time of the pregnant woman is predicted based on the fusion features to obtain a natural delivery time prediction result of the pregnant woman.

2. The method according to claim 1, characterized in that The automatic extraction of image features related to the time of natural delivery from the ultrasound images of the pregnant woman during the third trimester includes: Automatically extracting high-dimensional image features from the pregnant woman's third trimester ultrasound image using an image feature extraction network, and determining the extracted high-dimensional image features as image features associated with the time of natural delivery; The image feature extraction network adopts at least one of a convolutional neural network, a residual network, a densely connected network, and a hybrid neural network combining convolution operation and self-attention mechanism.

3. The method according to claim 1, characterized in that The clinical information of the pregnant woman includes numerical clinical information. Accordingly, the clinical information of the pregnant woman is characterized to obtain clinical characteristics related to the time of natural delivery, including: The standardized and normalized numerical clinical information is directly used as the clinical feature, or the standardized and normalized numerical clinical information is feature-encoded and embedded using a fully connected neural network or self-attention mechanism to obtain the clinical features of the numerical clinical information.

4. The method according to claim 3, characterized in that The clinical information of the pregnant woman also includes categorical clinical information. Accordingly, the clinical information of the pregnant woman is subjected to characterization processing to obtain clinical features related to the time of natural delivery, including: The categorical clinical information is mapped into a dense vector using an embedding layer, and the dense vector is determined as a clinical feature of the categorical clinical information.

5. The method according to claim 1, wherein The multimodal fusion strategy is used to fuse the image features and the clinical features to obtain fusion features with greater predictive power for delivery time, including: directly splicing the image features and the clinical features to obtain the fusion features; Alternatively, after directly concatenating the image features and the clinical features, the weight distribution of the two modality features is automatically learned through an attention mechanism to obtain the fusion feature; Alternatively, the image features and the clinical features are fused through at least one of a fully connected layer, a multi-layer perceptron, and an attention mechanism to obtain the fused features.

6. The method according to claim 1, characterized in that The natural delivery time prediction result includes whether the natural delivery will occur within a week and / or the number of days until the natural delivery. Accordingly, the natural delivery time of the pregnant woman is predicted based on the fusion feature, and the natural delivery time prediction result of the pregnant woman is obtained, including: Inputting the fused features into a fully connected layer for nonlinear mapping, and then passing through a classification prediction output layer to obtain a probability value of the pregnant woman giving birth within a week, so as to determine whether the pregnant woman will give birth within a week by comparing the probability value with a preset threshold; And / or, the fusion feature is input into the fully connected layer for nonlinear mapping, and then the regression prediction output layer is used to obtain the number of days until the pregnant woman's natural delivery, so as to determine the delivery date of the pregnant woman based on the gestational age and the number of days until the pregnant woman's natural delivery, or to determine the delivery date of the pregnant woman based on the current date and the number of days until the pregnant woman's natural delivery.

7. The method according to claim 2, characterized in that The method further comprises: Generate heatmap based on Grad-CAM method; Adjusting the size of the heat map to be consistent with the late pregnancy ultrasound image input to the image feature extraction network; The pseudo-color heat map overlay technology is used to overlay the resized heat map onto the late pregnancy ultrasound image to intuitively present the area in the late pregnancy ultrasound image that plays a key role in the prediction result of the natural delivery time.

8. A delivery time prediction system combining ultrasound images and clinical information, characterized in that: The system comprises: An image feature extraction unit, used to automatically extract image features related to the time of natural delivery from ultrasound images of pregnant women during late pregnancy; a clinical information characterization unit, configured to perform characterization processing on the clinical information of the pregnant woman to obtain clinical characteristics related to the time of natural delivery; A feature fusion module is used to fuse the image features and the clinical features through a multimodal fusion strategy to obtain a fusion feature with greater predictive power for delivery time; The delivery time prediction module is used to predict the natural delivery time of the pregnant woman based on the fusion feature to obtain a prediction result of the natural delivery time of the pregnant woman.

9. The system according to claim 8, characterized in that The natural delivery time prediction result is whether the natural delivery will occur within a week and / or the number of days until the natural delivery. The delivery time prediction module includes: a deep learning unit, configured to input the fused features into a fully connected layer for nonlinear mapping, and then obtain a probability value of the pregnant woman giving birth within one week through a classification prediction output layer, and / or input the fused features into a fully connected layer for nonlinear mapping, and then obtain the number of days until natural delivery of the pregnant woman through a regression prediction output layer; The result output unit is used to determine whether the pregnant woman will give birth within a week by comparing the probability value with a preset threshold, and / or determine the delivery date of the pregnant woman based on the gestational age and the number of days from natural delivery of the pregnant woman, or determine the delivery date of the pregnant woman based on the current date and the number of days from natural delivery of the pregnant woman.

10. The system according to claim 8, wherein: The system also includes an interpretability module for generating a heat map based on the Grad-CAM method, adjusting the size of the heat map to be consistent with the late pregnancy prenatal examination ultrasound image input into the image feature extraction network, and using pseudo-color heat map overlay technology to overlay the resized heat map on the late pregnancy prenatal examination ultrasound image to intuitively present the area in the late pregnancy prenatal examination ultrasound image that plays a key role in the prediction result of the natural delivery time.

Images