Real-time semantic segmentation method for maternal and fetal ultrasound images during delivery

Abstract

The invention discloses a real-time semantic segmentation method based on an intranatal maternal ultrasonic image, and the method comprises the following steps: inputting the obtained and extracted intranatal maternal ultrasonic image into a trained neural network RTNet, and segmenting an anatomical structure in the intranatal maternal ultrasonic image at a speed of 0.005 s per piece; according to the invention, the technical problems of large workload and long time consumption of an existing method based on manual screening and traditional medical image segmentation can be solved.

Description

Technical Field

[0001] The present invention relates to the fields of translational medicine and artificial intelligence technology, and in particular to a real-time semantic segmentation method for maternal and fetal ultrasound images during delivery. Background Art

[0002] Intrapartum maternal and fetal ultrasound examination is an important step for women to assess the progress of labor during delivery. It is an important basis for checking the position of the fetal head and the relative position of the fetal head to the maternal pelvis, thereby assessing the progress of labor, and can provide guidance for early intervention and treatment. During the clinical examination, professional clinicians with rich experience use ultrasound equipment to obtain two-dimensional ultrasound standard sections containing key anatomical structures, and then segment the anatomical structures according to the position of each section image, thereby measuring various evaluation indicators to assess the speed of labor progress and predict the risk of dystocia. Due to the low resolution and contrast of intrapartum maternal and fetal ultrasound images, unclear boundaries of anatomical structures, and the presence of artifacts, even experienced obstetricians cannot quickly and accurately segment the correct anatomical structures. Although image segmentation methods have made certain progress, the use of traditional image segmentation of intrapartum maternal and fetal ultrasound images still faces many problems, such as:

[0003] (1) The operation is difficult and requires extremely high operation skills from obstetricians, resulting in different segmentation results among different obstetricians;

[0004] (2) The accuracy of segmentation is low, resulting in large deviations in the measurement of delivery parameters after segmentation, and the reference value of delivery parameters cannot be guaranteed;

[0005] (3) Low clinical work efficiency may cause pregnant women to be examined for too long, causing adverse reactions.

[0006] Since there is still a lack of doctors with rich clinical experience and skilled operation skills, this situation is more serious in underdeveloped areas. On the other hand, the accuracy of maternal and fetal ultrasound image segmentation during delivery directly affects the calculation of labor progress parameters, which in turn affects obstetricians' delivery decisions.

[0007] Invention content information

[0008] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method for real-time semantic segmentation of maternal-fetal ultrasound images during delivery, so as to solve the technical problems that the existing methods based on traditional image segmentation and manual segmentation are labor-intensive, time-consuming and highly dependent on the operation of obstetricians, thereby affecting the accuracy of intrapartum ultrasound examination results.

[0009] To achieve the above object, the present invention provides the following technical solution: a real-time semantic segmentation method for maternal and fetal ultrasound images during delivery, the method comprising the following steps:

[0010] (1) Obtain multiple maternal and fetal ultrasound videos during delivery;

[0011] (2) extracting the maternal-fetal ultrasound video obtained in step (1) to obtain an extracted maternal-fetal ultrasound image during delivery;

[0012] (3) Input each maternal-fetal ultrasound image obtained in step (2) into the trained RTNet network to perform automatic segmentation of the anatomical structure and output the result;

[0013] Preferably, the RTNet network structure is as follows:

[0014] The first layer is the input layer, whose input size is 3*256*256 maternal and fetal ultrasound images during delivery.

[0015] The second layer is a distributed shift convolution layer, which receives the image from the input layer. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 16. This layer is filled in the SAME way and uses the Hswish function as the activation function. The final output size is a matrix of 16*256*256.

[0016] The third layer is the downsampling layer, which receives a matrix of size 16*256*256 from the output of the second layer. The pooling method of this layer is maximum pooling, the downsampling factor is 2*2, and the step size is 2.

[0017] The final output matrix size is 16*128*128.

[0018] The fourth layer is a distributed shift convolution layer, which receives a 16*128*128 matrix from the output of the third layer. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 32. This layer is filled in the SAME way and uses the Hswish function as the activation function. The final output is a matrix of size 32*128*128.

[0019] The fifth layer is the downsampling layer, which receives the matrix of size 32*128*128 from the output of the fourth layer. The pooling method of this layer is maximum pooling, the downsampling factor is 2*2, and the step size is 2. The final output size is a matrix of size 32*64*64.

[0020] The sixth layer is a distributed shift convolution layer, which receives a 32*64*64 matrix from the fifth layer. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 128. This layer is filled in the SAME way and uses the Hswish function as the activation function. The final output is a matrix of size 128*64*64.

[0021] The seventh layer is the downsampling layer, which receives the matrix of size 128*64*64 from the output of the sixth layer. The pooling method of this layer is maximum pooling, the downsampling factor is 2*2, and the step size is 2. The final output size is a matrix of size 128*32*32.

[0022] The eighth layer is the projection convolution layer, which receives the 128*32*32 matrix output from the seventh layer. First, the convolution kernel size is 3*3, the step size is 2, and the number of convolution kernels is 160. This layer uses the SAME filling method to reduce the dimension of the matrix and use layer normalization to obtain a 256*160 matrix.

[0023] The ninth layer is the shift perceptron layer, which receives the 256*160 matrix output from the eighth layer. First, the width axis of the matrix is ​​moved to induce the model to pay attention to the width information, and then the data is up-dimensioned. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 256*160 matrix. Similarly, the height axis of the matrix is ​​moved to induce the model to pay attention to the height information, and then the data is up-dimensioned. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 256*160 matrix. Finally, the data is up-dimensioned to obtain a 160*16*16 matrix.

[0024] The tenth layer is the projection convolution layer, which receives the 160*16*16 matrix output from the ninth layer. First, the convolution kernel size is 3*3, the step size is 2, and the number of convolution kernels is 256. This layer uses the SAME filling method to reduce the dimension of the matrix and use layer normalization to obtain a 64*256 matrix.

[0025] The eleventh layer is the shift perceptron layer, which receives the 64*256 matrix output from the tenth layer. First, the width axis of the matrix is ​​moved to induce the model to pay attention to the width information, and then the data is up-dimensioned. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 256. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 64*256 matrix. Similarly, the height axis of the matrix is ​​moved to induce the model to pay attention to the height information, and then the data is up-dimensioned. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 256. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 64*256 matrix. Finally, the data is up-dimensioned to obtain a 256*8*8 matrix.

[0026] The twelfth layer is the convolution layer, which receives the matrix of size 256*8*8 from the output of the eleventh layer. First, the convolution kernel size is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and then up-sampled by bilinear interpolation to obtain a matrix of size 160*16*16. Finally, it accepts the matrix of size 160*16*16 output by the ninth layer, and fuses the two inputs to output a matrix of 160*16*16.

[0027] The thirteenth layer is the shift perceptron layer, which receives the 160*16*16 matrix output from the twelfth layer. First, the matrix is ​​reduced in dimension, the width axis of the matrix is ​​moved, and the model is induced to pay attention to the width information. Then the data is increased in dimension. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 256*160 matrix. Similarly, the height axis of the matrix is ​​moved to induce the model to pay attention to the height information. Then the data is increased in dimension. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 256*160 matrix. Finally, the data is increased in dimension to obtain a 160*16*16 matrix.

[0028] The fourteenth layer is the convolution layer, which receives the 160*16*16 matrix output from the thirteenth layer. First, the convolution kernel size is 3*3, the step size is 1, and the number of convolution kernels is 128. This layer is filled in the SAME way, and then upsampled by bilinear interpolation to obtain a 128*32*32 matrix. Finally, it accepts the 128*32*32 matrix output from the ninth layer, and fuses the two inputs to output a 128*32*32 matrix.

[0029] The fifteenth layer is the shift perceptron layer, which receives the 128*32*32 matrix output from the thirteenth layer. First, the matrix is ​​reduced in dimension, the width axis of the matrix is ​​moved, and the model is induced to pay attention to the width information. Then the data is increased in dimension. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 128. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 1024*128 matrix. Similarly, the height axis of the matrix is ​​moved to induce the model to pay attention to the height information. Then the data is increased in dimension. The size of the convolution kernel is 3*3, the step size is 1, and the number of convolution kernels is 160. This layer is filled in the SAME way, and the matrix is ​​reduced in dimension. The GELU function is used as the activation function to obtain a 1024*128 matrix. Finally, the data is increased in dimension to obtain a 128*32*32 matrix.

[0030] The sixteenth layer is the convolution layer, which receives the 128*32*32 matrix output from the fifteenth layer. First, the convolution kernel size is 3*3, the step size is 1, and the number of convolution kernels is 32. This layer is filled in the SAME way, and then upsampled by bilinear interpolation to obtain a 32*64*64 matrix. Finally, it accepts the 32*64*64 matrix output from the fifth layer, and fuses the two inputs to output a 32*64*64 matrix.

[0031] The seventeenth layer is a convolution layer, which receives a 32*64*64 matrix from the output of the sixteenth layer. First, the convolution kernel size is 3*3, the step size is 1, and the number of convolution kernels is 16. This layer is filled in the SAME way, and then up-sampled by bilinear interpolation to obtain a 16*128*128 matrix. Finally, it receives a 16*128*128 matrix from the output of the third layer, and fuses the two inputs to output a 16*128*128 matrix. The eighteenth layer is a convolution layer, which receives a 16*128*128 matrix from the output of the seventeenth layer. First, the convolution kernel size is 3*3, the step size is 1, and the number of convolution kernels is 3. This layer is filled in the SAME way, and then up-sampled by bilinear interpolation to obtain a 3*256*256 matrix. Finally, it receives a 3*256*256 matrix from the output of the third layer, and fuses the two inputs to output a 3*256*256 matrix.

[0032] Preferably, the neural network is trained by the following steps:

[0033] (3-1) Obtain a data set of maternal and fetal ultrasound images during delivery, and divide the data set into a training set and a test set in a ratio of 8:2.

[0034] (3-2) sending the maternal and fetal ultrasound image dataset obtained in step (3-1) to an ultrasound physician for manual data labeling, so as to obtain a manual data labeling result as a labeled maternal and fetal ultrasound image dataset during delivery;

[0035] (3-3) Performing data enhancement on the training set in the labeled intrapartum maternal and fetal ultrasound image data set obtained in step (3-2) to obtain a data enhanced intrapartum maternal and fetal ultrasound image training set.

[0036] (3-4) The data-enhanced intrapartum maternal and fetal ultrasound image training set obtained in step (3-3) is standardized and normalized in succession to obtain a preprocessed intrapartum maternal and fetal ultrasound image training set.

[0037] (3-5) Input the preprocessed intrapartum maternal and fetal ultrasound images in the training set of intrapartum maternal and fetal ultrasound images obtained in step (3-4) into the neural network to obtain the corresponding reasoning output result, and input the reasoning output result and the intrapartum maternal and fetal ultrasound images annotated by the ultrasound physician in step (3-2) into the segmentation dice loss function of the neural network to obtain the loss value.

[0038] (3-6) Use the adaptive moment estimation optimizer and the loss value obtained in step (3-5) to optimize the loss function of the neural network to iteratively update the parameters in the neural network.

[0039] (3-7) Repeat the above steps (3-5) and (3-6) until the neural network converges to the optimal state, thereby obtaining a trained neural network.

[0040] Compared with the prior art, the present invention has the following advantages and effects:

[0041] The present invention proposes a real-time semantic segmentation method for maternal and fetal ultrasound images during delivery. The following randomly selects representative maternal and fetal ultrasound images during delivery as a test set to test the neural network. The test set includes 403 maternal and fetal ultrasound images. The accuracy and the inference speed of the neural network on Nivida RTX3090 are used as evaluation criteria to test the detection results of the neural network. The accuracy is as high as 90.94%, and the inference speed reaches 0.005s per image, which can meet the current clinical quality control requirements for real-time segmentation of maternal and fetal ultrasound images during delivery. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0043] Figure 1 It is a flow chart of a real-time semantic segmentation method of maternal and fetal ultrasound images during delivery disclosed by the present invention;

[0044] Figure 2 is an example of image data segmentation according to an embodiment of the present invention; DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Embodiment 1:

[0047] This embodiment discloses a method for automatically measuring the cervical dilation distance based on an ultrasound image during labor, which mainly includes the following steps:

[0048] S1, collect transperineal ultrasound images and label annotations of the fetal head and pubic symphysis;

[0049] The data source in step S1 is collected by the radiologist. The equipment used is Philip's SONOS5500B ultrasound for intrapartum ultrasound image examination. Under the guidance of the radiologist, four students performed the task of labeling the fetal head and pubic symphysis, and after completion, the doctor checked again to confirm that it was correct.

[0050] S2, further screening of ultrasound images;

[0051] Among them, select the clearer images of the fetal head and pubic symphysis.

[0052] S3, use RTNet to process and calculate the image obtained in S2 to obtain the cervical position and segmentation information of the image;

[0053] In step S3, the input is the intrapartum ultrasound image, and the output is the segmented region and category. Specifically, we use the RTNet model to process and calculate the input intrapartum ultrasound image. RTNet outputs the segmentation results of the pubic symphysis and fetal head.

[0054] Specifically, RTNet uses a symmetrical structure of encoding layer-decoding layer to capture ultrasound image features. After the input ultrasound image (size is H×W) passes through the encoding layer, feature maps of different sizes will be obtained. Different feature maps capture different information. Shallow feature maps capture shallow features, and deep feature maps capture deep features. Then, feature fusion is performed through upsampling. Specifically, it includes upsampling the encoding layer to obtain a feature map of the same size as the upper feature map, merging it, and then convolving it to the original number of channels. Iterate the cycle to finally obtain a feature map of the same size as the original image and output pixel-level segmentation information.

[0055] All experiments of the present invention are carried out under the PyTorch framework, and the graphics card used is NVIDIA GeForce RTX3090. The optimizer in the RTNet model uses the AdamW optimizer, with an initial learning rate of 0.001 and a weight decay of 0.00001. In the early stage of training, a larger learning rate can make the model converge faster, while in the later stage of training, a larger learning rate will cause the model to oscillate and converge. In order to stabilize the convergence of the model weight parameters, we use a learning rate decay strategy and a cosine annealing decay strategy. The training batch size is 4. Due to the small amount of data, the present invention also uses rotation, folding, cropping and other methods for data enhancement during the training process. In order to facilitate subsequent adjustments, the loss values ​​of the training are recorded throughout the training process, and the training is completed to test and select the optimal model. In addition, this model does not require the size of the input ultrasound image during production, and the input image will be automatically reshaped to a pixel size of 256×256 in the preprocessing of RTNet.

[0056] S4. Calculate the fetal head and pubic symphysis segmentation accuracy and inference time based on the segmentation information of S3.

[0057] In step S4, the segmentation accuracy is monitored according to the segmentation result in S3, such as Figure 2 Shown is a schematic diagram of the segmentation of the fetal head and pubic symphysis. The Dice coefficient and the average surface distance Asd are both important indicators for measuring the segmentation results of the model. Among them, the Dice coefficient is more sensitive to the internal filling of the segmentation, and the larger the Dice coefficient, the better, while the average surface distance Asd is more sensitive to the boundary of the segmentation, and the smaller the average surface distance Asd, the better. The commonly used segmentation model UNet has a segmentation dice coefficient of 91.57%, an average surface distance Asd of 2.97, and an inference time of about 0.05s per image. The segmentation dice coefficient of RTNet is 92.82%, the average surface distance Asd is 2.74, and the inference time is about 0.005s per image. It can be seen that the RTNet used in the present invention has better segmentation performance and faster inference speed.

[0058] In summary, this embodiment provides a real-time semantic segmentation method for maternal-fetal ultrasound images during delivery, which can achieve real-time and high-precision segmentation of the fetal head and pubic symphysis, and has strong reliability.

Claims

1. A real-time semantic segmentation method for maternal and fetal ultrasound images during delivery, characterized in that: The method for real-time automatic segmentation of fetal head and pubic symphysis in ultrasound images during delivery comprises the following steps: S1, collect transperineal ultrasound images and label annotations of the fetal head and pubic symphysis; S2, further screening the ultrasound images and selecting ultrasound images with clear anatomical structures; S3, using RTNet to process and calculate the image obtained in S2, and obtain the position and segmentation information of the pubic symphysis and fetal head in real time; S4. Calculate the fetal head and pubic symphysis segmentation accuracy and inference time based on the segmentation information of S3.

2. The real-time semantic segmentation method of maternal and fetal ultrasound images during delivery according to claim 1, characterized in that: The image data in step S1 comes from intrapartum ultrasound images with different relative positions of the pubic symphysis and the fetal head.

3. The real-time semantic segmentation method of maternal-fetal ultrasound images during delivery according to claim 1 or 2, characterized in that: The step S2 selects an intrapartum ultrasound image having a fetal head and pubic symphysis.

4. A real-time semantic segmentation method for maternal-fetal ultrasound images during delivery according to any one of claims 1 to 3, characterized in that: The network model structure of RTNet in step S3 includes an encoder and a decoder, and both the encoder and the decoder use only a single convolutional layer to reduce the computational complexity and memory usage of the neural network, so as to achieve real-time semantic segmentation of the fetal head and pubic symphysis.

5. The real-time semantic segmentation method of maternal and fetal ultrasound images during delivery according to claim 4, characterized in that: The convolution of the encoder of the RTNet network model structure adopts distributed shift convolution, and the convolution kernel size is 3*3, which reduces the computational complexity and memory usage of the neural network to achieve real-time semantic segmentation of the fetal head and pubic symphysis.

6. The real-time semantic segmentation method of maternal and fetal ultrasound images during delivery according to claim 4, characterized in that: The deep network structure of the RTNet network model structure adopts a shift perceptron to shift the width and height of the extracted image features respectively, and learn the relevant position features of the image to achieve high-precision segmentation of the fetal head and pubic symphysis.

7. A real-time semantic segmentation method for maternal and fetal ultrasound images during delivery according to any one of claims 4, 5, and 6, characterized in that: The RTNet network can be applied to real-time and high-precision semantic segmentation of the fetal head and pubic symphysis, and the segmentation reasoning time is greater than 24 transmission frames per second.

Images