A dual-model fusion method for fetal age and weight prediction

Abstract

The present invention discloses a dual-model fused fetal age and weight prediction method in the technical field of fetal age and weight prediction, comprising the following steps: Step 1: reading fetal scan video data, and performing noise reduction preprocessing on the video data to obtain to-be-processed video data; Step 2: inputting the to-be-processed video data into a 3D regression network model and a 2D multi-task network model for processing respectively, so as to obtain first prediction data and second prediction data; Step 3: fusing the first prediction data and the second prediction data and performing weighted average calculation to obtain predicted values ​​of fetal age and fetal weight. In the present invention, the fetal age and weight estimation results of two different recognition paths are fused, and the possible large error deviation of a single model is reduced through this redundant method.

Description

Technical Field

[0001] The present invention relates to the technical field of fetal age and weight prediction, and in particular to a dual-model fusion fetal age and weight prediction method. Background Art

[0002] Fetal birth weight (FBW) is an important indicator of perinatal health prognosis. Accurate prediction of fetal birth weight, gestational age, pregnancy complications and maternal body parameters is crucial to determine the best mode of delivery (natural delivery or cesarean section). Ultrasound measurement of fetal organs is currently the most commonly used method to assess fetal growth status and pregnancy safety.

[0003] Accurately predicting age and weight using prenatal ultrasound is challenging because it requires images of specific fetal body parts taken in late pregnancy, which are difficult to capture due to poor image quality caused by the lack of amniotic fluid, and requires highly visible standard planes of the fetal body, which can only be identified by experienced sonographers. Unfortunately, weight predictions provided by experienced sonographers are often imprecise, with a mean absolute percentage error of up to 10%. Therefore, predictions that rely solely on standard methods often have significant errors. To this end, we propose a dual-model fusion method for fetal age and weight prediction. Summary of the invention

[0004] The purpose of the present invention is to provide a dual-model fusion method for predicting fetal age and weight, so as to solve the problem in the above-mentioned background technology that predictions relying solely on standard methods often result in significant errors.

[0005] To achieve the above object, the present invention provides the following technical solution: a dual-model fusion fetal age and weight prediction method, specifically comprising the following steps:

[0006] Step 1: Read fetal scan video data, and perform noise reduction preprocessing on the video data to obtain video data to be processed;

[0007] Step 2: inputting the to-be-processed video data into the 3D regression network model and the 2D multi-task network model for processing respectively, so as to obtain the first prediction data and the second prediction data;

[0008] Step 3: The first prediction data and the second prediction data are integrated and weighted averaged to obtain the predicted values ​​of fetal age and fetal weight;

[0009] The process of acquiring the first prediction data is specifically as follows: extracting image data of consecutive T frames in the video data to be processed and inputting them into a 3D regression network model for processing to obtain first preprocessed data; the first preprocessed data is filtered according to a first preset rule to obtain first prediction data;

[0010] The process of acquiring the second prediction data is specifically as follows: extracting each frame of image data in the video data to be processed and inputting it into a 2D multi-task network model for processing to obtain second preprocessed data; and the second preprocessed data is filtered according to a second preset rule to obtain second prediction data.

[0011] Preferably, after the video data is subjected to noise reduction preprocessing in step 1, the video data to be processed is obtained by using an ultrasonic image quality network to judge the image quality of each frame of the video, and the image frames with quality estimation values ​​lower than a threshold are eliminated to obtain the video data to be processed.

[0012] Preferably, the ultrasound image quality network is a regression network based on ResNet, the input of the ultrasound image quality network is 3×H×W in size, 3 represents the number of channels of the RGB image, H represents the height of the image, and W represents the width of the image, and the output of the ultrasound image quality network is an image quality score;

[0013] When the ultrasound image quality network judges the image quality of each frame of the video, the image height and width H and W are 244 video frames.

[0014] Preferably, the 3D regression network model includes a Transformer module arranged between a 3D global pooling layer and a fully connected layer, the input of the 3D regression network is of size T×3×H×W, T represents the number of consecutive frames of the video, 3 represents the number of channels of the RGB image, H represents the height of the image, and W represents the width of the image, and the output of the 3D regression network is the first prediction data;

[0015] When the 3D regression network model processes the extracted image data, the input segment length T is 16, and the video sequence of the image height and width H and W are 64.

[0016] Preferably, the first preset rule is specifically to compare the first preprocessed data with a threshold to obtain data with a score greater than the threshold, and filter out the data with the largest score among the data with a score greater than the threshold to obtain the first predicted data.

[0017] Preferably, the 2D multi-task network model includes a Transformer module arranged between the encoder 4 and the decoder 4, the input of the 2D multi-task network is 3×H×W, and the output of the 2D multi-task network is the second prediction data;

[0018] The 2D multi-task network output includes a key point output of size 10×H×W and a classification output of size 4. The first 0 to 3 channels of the 10 key point outputs are the key points a1, a2, b1, b2 of the fetus head, 4 to 7 are the key points c1, c2, d1, d2 of the abdomen, and 8 to 9 are the key points e of the femur. 1， e2, the classification output is four categories of classification output: background, head, abdomen and femur.

[0019] Preferably, the second preset rule is to compare the second preprocessed data with a threshold to obtain data with a classification prediction probability greater than the threshold, filter out the data with the largest classification prediction probability among the data with a classification prediction probability greater than the threshold, and use the filtered data to perform calculations to obtain second prediction data.

[0020] Preferably, the method of using the screened data for calculation to obtain the second predicted data specifically comprises using the screened data to calculate the major axis and the minor axis of the ellipse of the fetal head and the fetal abdomen, and then calculating the circumference of the ellipse by the major axis and the minor axis as the head circumference and the abdominal circumference, calculating the minor axis length of the fetal head ellipse as the biparietal diameter, calculating the distance between two key points of the fetal femur as the femur length, and then inputting the head circumference, abdominal circumference, biparietal diameter, and femur length into the MLP age regression network to obtain the fetal age, and inputting the head circumference, abdominal circumference, and femur length into the MLP weight regression network to obtain the fetal weight, so as to obtain the second predicted data;

[0021] The calculation formula involved in the calculation using the selected data includes: the straight line formed by the fetal head a1a2 is the major axis of the ellipse , the straight line formed by b1b2 is the minor axis of the ellipse ;

[0022] The straight line formed by c1c2 of the fetal abdomen is the major axis of the ellipse , the straight line formed by d1d2 is the minor axis of the ellipse ;

[0023] The straight line formed by the fetal femur e1e2 is the femur length ;

[0024] The calculation method for head circumference is 2πb+4 (ab), the calculation method for abdominal circumference is 2πd+4 (cd), the calculation method for biparietal diameter is b, and the calculation method for femur length is e.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: the ultrasound image quality network is used in the present invention to filter out image frames with lower quality, thereby avoiding interference of low-quality image frames on subsequent networks, and fusing the fetal age and weight estimation results of two different recognition paths. Through this redundant approach, the possible large error deviation of a single model is reduced, and at the same time, the accuracy of the model is significantly improved by combining information of time, global space and local space, thereby increasing the accuracy of prediction of fetal birth age and weight. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a workflow diagram for predicting fetal birth age and weight according to the present invention;

[0027] Figure 2 This is a schematic diagram of the image quality network structure of the present invention;

[0028] Figure 3 Schematic diagram of the 3D regression network structure of the present invention;

[0029] Figure 4 Schematic diagram of the 2D multi-task network structure of the present invention;

[0030] Figure 5 This is the flow chart of the age and weight MLP regression network of the present invention;

[0031] Figure 6 It is a schematic diagram of the standard surfaces of the three organs of the fetus head, abdomen and femur according to the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] See also Figure 1 , Figure 2 and Figure 6 , a dual-model fusion fetal age and weight prediction method, specifically comprising the following steps:

[0034] Step 1: Read the fetal scan video, perform noise reduction preprocessing on the read video, and obtain the video data to be processed;

[0035] The ultrasound image quality network is used to judge the quality of each image frame, and the image frames with quality estimation values ​​lower than the threshold are eliminated to obtain high-quality scanning videos.

[0036] Among them, (1) the image quality network is a regression network based on ResNet. The input of the network is 3×H×W, where 3 represents the number of channels of the RGB image, H represents the height of the image, and W represents the width of the image. The output of the network is the regression output img of the image quality score. — quality;

[0037] (2) The training data consists of ultrasound images and the true values ​​of image quality annotated by doctors. Image quality is a continuous value between 0 and 1. Low-quality images are usually blurred images caused by rapid movement, black screens due to poor contact between the probe and the body, ghost images caused by occlusion, and meaningless frames that do not contain any organs. The data is divided into training and test sets with a ratio of 4:1.

[0038] (3) Use the pytorch training framework to organize the training code, use NVIDIA GTX2080Ti for training, set the minimum batch size to 64, use the Adam optimizer with an initial learning rate of 0.0001, and reduce the learning rate by 0.1 times every time the entire training set is iterated for 80 rounds, for a total of 100 rounds. The training updates the model parameters by minimizing the mean square error (MSE) loss. Use data enhancement such as random rotation, brightness, contrast, horizontal flipping, image compression, and blurring to accelerate model training and improve generalization;

[0039] (4) During inference, the video frame with image height and width H and W of 244 is output as the image quality estimation value;

[0040] (5) For (4), when img — quality is smaller than img — quality — When the thr threshold is reached, the current frame is judged as a low-quality video frame, and the current frame is removed from the original video to obtain a high-quality scan video;

[0041] Use ultrasound image quality network to filter out low-quality image frames, as the lack of amniotic fluid in late pregnancy leads to poor image quality, to avoid interference of low-quality image frames on subsequent networks;

[0042] See also Figure 1 and Figure 3 , Step 2: Input the video data to be processed into the 3D regression network model and the 2D multi-task network model for processing, and obtain the fetal birth age and weight under the 3D regression network model and the fetal birth age and weight under the 2D multi-task network model;

[0043] Specifically:

[0044] Step 21: For the high-quality scan video obtained in step 1, extract the continuous The frame data is input into the 3D regression network to predict the fetal birth age (GA), weight (FBW) and video segment quality score, and the results with video segment quality scores greater than the threshold are recorded;

[0045] Step 22: When all frames of the scanned video are identified, the fetal birth age (GA) and weight (FBW) of the video segment with the largest video segment quality score are output as the final result;

[0046] Among them, (1) the Transformer module is added before the last fully connected layer of the 3D ResNet model to obtain a 3D regression network. The input of the network is T×3×H×W, where T represents the number of consecutive video frames and 3 represents the number of channels of the RGB image. Represents the height of the image, W represents the width of the image, and the output of the network is the regression output of age, weight, and video segment quality;

[0047] (2) The training data is T-frame video segments captured from fetal scan videos. Doctors annotate the true value of video quality for each video segment. Video quality is a continuous value between 0 and 1. The more organs such as the fetal head, abdomen, and femur are included in the captured video segment, the closer the video quality score is to 1, and vice versa. Only video segments that contain organs such as the fetal head, abdomen, and femur are annotated with the weight and age of the fetus. The true value of weight and age is the data collected when the fetus is born. The data is divided into training set and test set with a ratio of 4:1;

[0048] (3) Use the pytorch training framework to organize the training code, use NVIDIA GTX2080Ti for training, set the minimum batch size to 2, use the Adam optimizer with an initial learning rate of 0.0001, and reduce the learning rate by 0.1 times every time the entire training set is iterated for 150 rounds, for a total of 200 rounds of training. The training updates the model parameters by minimizing the mean square error (MSE) loss. Data enhancement such as random rotation, brightness, contrast, horizontal flipping, image compression and blurring are used to accelerate model training and improve generalization. The video clip length T is set to 16, and the image height and width H and W are set to 64;

[0049] (4) During inference, the input is a video sequence with a segment length T of 16 and image height and width H and W of 64, and the output is the estimated age and weight of the fetus;

[0050] (5) For the high-quality scan video obtained in Example 1, continuous The frame data is input into the 3D regression network to predict the fetal birth age (GA), weight (FBW) and video segment quality score, and the results with video segment quality scores greater than the threshold are recorded;

[0051] (6) When all frames of the scanned video are identified, the fetal birth age (GA) and weight (FBW) of the video segment with the largest video segment quality score recorded in (5) are output as the final result;

[0052] The role of the 3D regression network model: Use Transformer to extend 3D ResNet and propose an end-to-end 3D fetal age, weight and video segment quality regression neural network, which is used to simultaneously learn the temporal and spatial information based on fetal scan videos and model long sequence inputs. The robustness and accuracy are significantly better than the model based on single-frame images. In addition, through the video segment quality score indicator, the video segments input to the network are made to include video frames such as the fetal head, abdomen and femur as much as possible;

[0053] See also Figure 1 , Figure 4 and Figure 5 ,Step 23: For the high-quality scan video obtained in step 1, extract each frame and input it into the 2D multi-task network, and the network outputs the classification result of the current image and the coordinates of the key points of the organs;

[0054] Step 24: Record the results of the standard sections whose output classification category is one of the three categories of fetal head, fetal abdomen and fetal femur and whose classification prediction probability is greater than the threshold;

[0055] Step 25: After all frames of the scan video are identified, for the results of the three categories of fetal head, fetal abdomen and fetal femur, the frame with the largest prediction probability is selected as the best standard section;

[0056] Step 26: For the key points of the major axis and minor axis endpoints of the ellipse corresponding to the best frames of the three categories of fetal head, fetal abdomen and fetal femur, calculate the major axis and minor axis of the ellipse of the fetal head and fetal abdomen, and then calculate the circumference of the ellipse through the major axis and minor axis as the head circumference (HC) and abdominal circumference (AC), the minor axis length of the fetal head ellipse is used as the biparietal diameter (BPD), and the distance between the two key points of the fetal femur is calculated as the femur length (FL);

[0057] Step 27: When the best standard sections of the fetal head, abdomen, and femur are detected, HC, AC, BPD, and FL are input into the MLP age regression network to obtain the fetal age (GA), and HC, AC, and FL are input into the MLP weight regression network to obtain the fetal weight (FBW);

[0058] Among them, (1) the Transformer module is added between the encoder 4 and the decoder 4 of Unet to obtain a 2D multi-task network. The input of the network is 3×H×W, and the network output includes a key point output of size 10×H×W and a classification output of size 4. Among the 10 channel outputs of the key point, the first 0 to 3 channels are the key points of the fetal head a1, a2, b1, b2, 4 to 7 are the key points of the abdomen c1, c2, d1, d2, and 8 to 9 are the key points of the fetus e1e2. The classification output is 4 categories of classification outputs: background, head, abdomen and femur.

[0059] (2) The training data is a single-frame image captured from a fetal scan video. The video frames containing organs such as the fetal head, abdomen, and femur are annotated with corresponding categories and organ key points, and other video frames are used as background categories. The data is divided into a training set and a test set with a ratio of 4:1.

[0060] (3) Use the pytorch training framework to organize the training code, use NVIDIA GTX2080Ti for training, set the minimum batch size to 16, use the Adam optimizer with an initial learning rate of 0.0001, adopt the learning rate cosine descent strategy, and train for 100 rounds. The training updates the model parameters by minimizing the cross entropy loss (CE) and root mean square loss (MSE) losses. Use data enhancement such as random rotation, brightness, contrast, horizontal flip, and vertical flip to accelerate model training and improve generalization. Image height and width and Set to 320;

[0061] (4) Use the model trained in (3) for inference. Read the fetal scan video, extract each frame, and input it into the 2D segmentation and classification multi-task network. The network outputs the classification result and key points of the current image.

[0062] (5) For (4), the output classification category is one of the three categories of fetal head, fetal abdomen and fetal femur, and the classification prediction probability Greater than The results of the standard sections are recorded;

[0063] (6) For the key points in (4), the straight line formed by the fetal head a1a2 is the major diameter of the ellipse , the straight line formed by b1b2 is the short diameter of the ellipse , the straight line formed by c1c2 in the fetal abdomen is the long diameter of the ellipse , the straight line formed by d1d2 is the short diameter of the ellipse , the straight line formed by the femur e1e2 is the femur length ;

[0064] (7) After identifying all frames of the scan video, for the three categories of fetal head, fetal abdomen, and fetal femur, select the predicted probability in (5). The largest frame is taken as the best standard section;

[0065] (8) For the key points corresponding to the best frames of the three categories of fetal head, fetal abdomen and fetal femur screened out in (7), the circumference of the ellipse is obtained through (6), the head circumference (HC) is 2πb+4(ab) and the abdominal circumference (AC) is 2πd+4(cd), the minor axis length b of the fetal head ellipse is taken as the biparietal diameter (BPD), and the femur length (FL) is e;

[0066] (9) For the HC, AC, BPD and FL obtained in (8), input them into the MLP fetal age regression network to obtain fetal age (GA), and input HC, AC and FL into the MLP fetal weight regression network to obtain fetal weight (FBW);

[0067] The role of the 2D multi-task network model: Using Transformer to extend Unet, an end-to-end 2D fetal standard section recognition and organ key point multi-task network is proposed. The attention mechanism of Transformer is used to learn the local and global information of a single frame image, and then the fetal biological characteristics are measured through post-processing to obtain the measured values. Finally, the measured values ​​are input into the MLP regression network to obtain the final fetal age and weight estimation. Multi-task training can significantly reduce model misidentification and improve model robustness and accuracy. Different from other methods that use segmentation to obtain measurement values, the key point method is simpler and more direct, avoiding complex post-processing of segmentation, and significantly reducing the computational cost and annotation cost;

[0068] See also Figure 1 , Step 3: Fuse the fetal birth age and weight under the 3D regression network model in step 2 with the fetal birth age and weight under the 2D multi-task network model, and perform weighted average calculation on the fused data to obtain the estimated values ​​of fetal age and fetal weight;

[0069] By fusing the fetal age and weight estimation results of two different recognition paths, the possible large error deviation of a single model can be reduced through this redundant method. At the same time, the accuracy of the model can be significantly improved by combining the information of time, global space and local space.

[0070] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A dual-model fusion method for predicting fetal age and weight, characterized in that: The specific steps include: Step 1: Read fetal scan video data, and perform noise reduction preprocessing on the video data to obtain video data to be processed; Step 2: inputting the to-be-processed video data into the 3D regression network model and the 2D multi-task network model for processing respectively, so as to obtain the first prediction data and the second prediction data; Step 3: The first prediction data and the second prediction data are integrated and weighted averaged to obtain the predicted values ​​of fetal age and fetal weight; The first predicted data is the birth age and birth weight of the fetus. The specific process of obtaining the first predicted data is as follows: for the video data to be processed obtained in step 1, continuous T-frame data are extracted with overlapping sliding, input into the 3D regression network, and the birth age, birth weight and video segment quality score of the fetus are predicted. The results of the video segment quality score greater than the threshold are recorded. When all frames of the entire video to be processed are identified, the results of the recorded video segment quality scores are output, and the birth age and birth weight of the fetus of the video segment with the largest video segment quality score are output as the final result; The second predicted data are the birth age and birth weight of the fetus. The specific process of obtaining the second predicted data is as follows: for the video data to be processed obtained in step 1, each frame is extracted, input into the 2D multi-task network, the classification result of the current image and the coordinates of the key points of the organs are output, and the results of the standard sections whose output classification categories are one of the three categories of fetal head, fetal abdomen and fetal femur and whose classification prediction probability is greater than the threshold are recorded; when all frames of the entire video to be processed are identified, for the results of the three categories of fetal head, fetal abdomen and fetal femur recorded, the frame with the largest prediction probability is selected as the best standard section; for the key points of the major axis and minor axis endpoints of the ellipse corresponding to the best frames of the three categories of fetal head, fetal abdomen and fetal femur screened out, the head circumference, abdominal circumference, biparietal diameter and femur length of the fetus are calculated; when the best standard sections of the fetal head, abdomen and femur are all detected, the head circumference, abdominal circumference, biparietal diameter and femur length are input into the MLP age regression network to obtain the birth age of the fetus, and the head circumference, abdominal circumference and femur length are input into the MLP weight regression network to obtain the birth weight of the fetus.

2. The method for predicting fetal age and weight by dual-model fusion according to claim 1, characterized in that: After the video data is pre-processed for noise reduction in step 1, the video data to be processed is obtained. Specifically, the quality of each frame of the video is judged by an ultrasonic image quality network, and the image frames with quality estimation values ​​lower than a threshold are eliminated to obtain the video data to be processed.

3. The method for predicting fetal age and weight by dual-model fusion according to claim 2, characterized in that: The ultrasound image quality network is a regression network based on ResNet, and the input of the ultrasound image quality network is size, 3 represents the number of channels of the RGB image, H represents the height of the image, W represents the width of the image, and the output of the ultrasound image quality network is the image quality score; When the ultrasound image quality network judges the image quality of each frame of the video, the image height and width H and W are 244 video frames.

4. The method for predicting fetal age and weight by dual-model fusion according to claim 1, characterized in that: The 3D regression network model includes a Transformer module arranged between the 3D global pooling and the fully connected layer. The input of the 3D regression network model is Size, T represents the number of consecutive frames of the video, 3 represents the number of channels of the RGB image, H represents the height of the image, and W represents the width of the image; When the 3D regression network model processes the extracted image data, the input segment length T is 16, and the video sequence of the image height and width H and W are 64.

5. The method for predicting fetal age and weight by dual-model fusion according to claim 1, characterized in that: The 2D multi-task network model includes a Transformer module arranged between the encoder 4 and the decoder 4. The input of the 2D multi-task network model is ; Among them, the 2D multi-task network model output contains a size of Key point output and a classification output of size 4, the first 0 to 3 channels of the key point 10 channel output are fetal head key points a1, a2, b1, b2, 4 to 7 are abdominal key points c1, c2, d1, d2 and 8 to 9 are femur key points e 1， e2, the classification output is four categories of classification output: background, head, abdomen and femur, H represents the height of the image, and W represents the width of the image.

6. The method for predicting fetal age and weight by dual-model fusion according to claim 5, characterized in that: Calculate the major axis and minor axis of the ellipse of the fetal head and fetal abdomen, and then calculate the circumference of the ellipse through the major axis and minor axis as the head circumference and abdominal circumference, calculate the minor axis length of the fetal head ellipse as the biparietal diameter, and calculate the distance between the two key points of the fetal femur as the femur length; wherein the calculation formula includes that the straight line formed by the fetal head a1a2 is the major axis of the ellipse , the straight line formed by b1b2 is the minor axis of the ellipse ; The straight line formed by c1c2 of the fetal abdomen is the major axis of the ellipse , the straight line formed by d1d2 is the minor axis of the ellipse ; The straight line formed by the fetal femur e1e2 is the femur length ; Head circumference is calculated as , abdominal circumference calculation method is , the biparietal diameter is calculated as b, and the femur length is calculated as e.

Images