Key frame detection method and device for endometrial image
Through the endometrial image keyframe detection method combined with DDR-Net and MSNet169 model, the problem of low keyframe recognition efficiency in uterine ultrasound video in the prior art is solved, and high accuracy and high efficiency endometrial detection is achieved. Enhanced the reliability of clinical applications.
Patent Information
- Application Number
- CN202411117638.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-24
AI Technical Summary
The prior art is difficult to accurately identify keyframes in uterine ultrasound videos in a short period of time, affecting the efficiency and accuracy of endometrial detection.
The endometrial image keyframe detection method combined with DDR-Net and MSNet169 model is adopted to extract and identify keyframe images containing the maximum thickness information through segmentation, classification and feature fusion modules, and enhance the robustness of the model using the focus cross-entropy loss function. .
It significantly improves the accuracy and efficiency of image segmentation and classification, reduces operation time, improves the reliability and accuracy of automated processing, can quickly and accurately identify keyframes, and enhances the support of clinical applications.
Smart Images
Figure CN119181038A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image analysis, and in particular to a key frame detection method and device for endometrial images. Background Art
[0002] Normal thickness of endometrium The normal thickness of endometrium is different in different periods of menstrual cycle. If the endometrium is thick, menstruation is easy and painful, and the fertilized egg is not easy to implant. Generally speaking, the normal thickness of endometrium is 0.2-1.0 cm. During menstruation, the functional layer of endometrium falls off and the basal layer is preserved. The proliferative period is also the 6th to 14th day of menstruation. The normal thickness of endometrium is about 1-3mm. The secretory period is from the 15th to 28th day of menstruation, and the thickness of endometrium is about 5-7mm. However, the thickness of endometrium of some women is obviously not in line with the normal thickness after examination. Then the thin hyperplasia of endometrium can lead to female infertility and some can also cause endometrial cancer. In recent years, uterine ultrasound examination is used to obtain endometrial ultrasound video. In the process of video analysis and processing, relevant and necessary information retrieval is a task that must be completed, because if the video is large, it is difficult to process the complete video in a short time without losing its semantic details. So how to accurately identify the key frames in ultrasound video is the basis for endometrial detection. Summary of the invention
[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a key frame detection method for endometrial images, which comprises the following steps:
[0004] Step 1: Acquire uterine ultrasound video data;
[0005] Step 2: extract and segment the uterine ultrasound video data to obtain a segmented endometrial image;
[0006] Step 3: Use the MSNet169 model to classify the segmented endometrial images, and determine all endometrial images suitable for measuring endometrial thickness based on the classification results;
[0007] Step 4: Find key frames in all endometrial images obtained in step 3;
[0008] Step 5: Output a key frame image containing maximum thickness information and the corresponding specific thickness value of the endometrium.
[0009] Furthermore, the specific steps of step 2 include:
[0010] Step 21: Segment the input video data into a number of single-frame images;
[0011] Step 22: Perform image enhancement on each single frame image;
[0012] Step 23: Use the DDRNets model to segment the enhanced image to obtain a segmented endometrial image and generate an image mask;
[0013] Step 24: Superimpose the generated mask and the original single-frame image to obtain a masked endometrial image.
[0014] Furthermore, the MSNet169 model includes two parallel branch networks, the first branch network is used to extract mask features to obtain a mask feature map, and the second branch network is used to extract image features. The two branch networks are connected through a feature fusion module, and the output of the feature fusion module is connected to a fully connected layer, which is used to map the fused features to the target category, thereby achieving classification.
[0015] Furthermore, the first branch network includes a plurality of convolutional layers, a spatial attention module, and an adaptive average pooling layer connected in sequence, and the spatial attention module is used to generate a spatial attention map;
[0016] The second branch network includes multiple convolutional layers using dense connections, the output features of each layer are cascaded with subsequent layers, and the channel features are connected between each layer through dense connections.
[0017] Furthermore, the feature fusion module fuses the mask features and the image features by performing element-wise multiplication operation on the mask feature map and the spatial attention map.
[0018] Furthermore, the specific steps of step 3 are:
[0019] Step 31: Input the masked endometrial image into the MSNet169 model;
[0020] Step 32: Use the pre-trained MSNet169 model to extract mask features and image features through two branch networks to obtain the final classification results;
[0021] Step 33: Determine whether the current segmented endometrial image is suitable for measuring endometrial thickness based on the classification result, and output the endometrial image with a correct segmentation result for measuring thickness.
[0022] Furthermore, the MSNet169 model uses the focal cross entropy loss function for training, and its calculation formula is:
[0023] FL(p t )=-∑(i=1to K)q i α i (1-p i ) γ log(p i )
[0024] in,
[0025] q i =(1-ε)p i +ε / K
[0026] In the formula, α i represents the sample weight, γ represents the adjustable focusing parameter, q i is the smoothed label value, p i is the original label, ε is the smoothing parameter, K is the number of categories; p t Represents the predicted probability value.
[0027] Furthermore, step 4 specifically includes:
[0028] Step 41: import all correctly classified endometrial images with masks, find the inscribed circle using the maximum inscribed circle search method, and calculate the endometrial thickness;
[0029] Step 42: The image frame corresponding to the time when the calculated endometrial thickness is the maximum is used as a key frame.
[0030] A key frame detection device for endometrial images is implemented based on the key frame detection method for endometrial images, and the device comprises:
[0031] An input module, used for collecting uterine ultrasound video data through an ultrasound machine;
[0032] The segmentation module is used to segment the input video data into several single-frame images, and perform image enhancement on each single-frame image. Finally, the enhanced image is segmented using the DDRNets model to generate a masked endometrial image.
[0033] A classification module, used for classifying the segmented endometrial images and determining all endometrial images suitable for measuring endometrial thickness;
[0034] A key frame detection module, for extracting key frames based on the correctly classified masked endometrial images;
[0035] The output module is used to output a key frame image containing maximum thickness information and the corresponding specific thickness value of the endometrium.
[0036] Furthermore, the classification module includes:
[0037] An MSNet169 model building unit, used to build an MSNet169 model including two branch networks, wherein the two branch networks are used to extract mask features and image features respectively, and the obtained mask features and image features are fused through a feature fusion module to realize classification;
[0038] An MSNet169 model classification unit, used to classify the input masked endometrial image using the MSNet169 model and output the classification result;
[0039] The segmentation result evaluation unit is used to determine whether the current segmented endometrial image is suitable for measuring the endometrial thickness according to the classification result, and output the endometrial image with the correct segmentation result.
[0040] Based on the DDR-Net and MSNet169 models, the present invention proposes a key frame detection method for endometrial images, which has the following technical effects:
[0041] First, the accuracy of image segmentation is improved:
[0042] The DDR-Net model uses a deep convolutional neural network to accurately segment the endometrial region in ultrasound images. Through a dual-path structure, one path generates a high-resolution feature map, and the other path extracts rich semantic information through multiple downsampling and performs information fusion through a multi-scale context module (DAPPM), which significantly improves the segmentation accuracy.
[0043] Second, the accuracy and efficiency of image classification are enhanced:
[0044] The MSNet169 model combines image features and mask features through a dual-channel input structure. The first branch network extracts mask features and generates a spatial attention map, and the second branch network extracts image features and fuses the two through a feature fusion module. The MSNet169 model has better accuracy in classification tasks than traditional convolutional neural network models. This structure effectively utilizes mask information and significantly improves classification performance.
[0045] Third, the efficiency of key frame detection is improved:
[0046] Through the segmentation and classification steps, the image with the largest endometrial thickness is extracted as the key frame, which can quickly and accurately identify the key frame. Compared with manual selection of key frames, the method of the present invention greatly reduces the operation time and improves efficiency.
[0047] Fourth, the robustness of automated processing has been improved:
[0048] The present invention adopts the focal cross entropy loss function for model training, solves the class imbalance problem, and enhances the robustness of the model. Experimental results show that the use of the MSNet169 model can better cope with complex endometrial images and improve the reliability of automatic processing.
[0049] In summary, the present invention significantly improves the accuracy and efficiency of endometrial image processing by improving the segmentation and classification model structure, providing more reliable technical support for clinical applications.
[0050] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 Schematic diagram of the DDRNets model structure framework.
[0052] Figure 2 This is a schematic diagram of the MSNet169 network structure.
[0053] Figure 3 Flowchart of the two-step automated method for measuring endometrial thickness.
[0054] Figure 4 A diagram showing the process of automatically identifying key frames. DETAILED DESCRIPTION
[0055] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art make various changes or modifications to the present invention, and these equivalent forms also fall within the scope limited by the claims attached to the application.
[0056] The present invention provides a key frame detection method for endometrial images, which mainly includes three parts: segmentation, classification, and key frame search.
[0057] 1. Endometrial segmentation
[0058] Endometrial segmentation is a key step in automatic key frame detection. The present invention adopts a deep dual-resolution network-DDRNets model for segmentation.
[0059] DDRNets uses deep convolutional neural networks to accurately segment the endometrial region in ultrasound images. Figure 1 As shown in the figure, DDRNets starts with the backbone network and then divides into two deep branches with different resolutions. One branch generates a relatively high-resolution feature map, and the other branch extracts rich semantic information through multiple downsampling operations. There are multiple bilateral connections between the two branches to achieve effective information fusion. In addition, DDRNets also contains a new module called DAPPM, which inputs low-resolution feature maps, extracts multi-scale contextual information, and merges them together in a cascaded manner.
[0060] First, we used the collected uterine ultrasound video data to construct a semantic segmentation dataset, which contains 349 uterine ultrasound videos, covering different patient characteristics and clinical scenarios. The age range of patients in the dataset ranges from 16 to 80 years old, ensuring the representation of a wide range of childbearing age groups.
[0061] Secondly, the DRRNets model is trained using the semantic segmentation dataset. Before the DRRNets model is trained on the semantic segmentation dataset, the DRRNets model has been trained on ImageNet.
[0062] In order to demonstrate the effect of the semantic segmentation model selected by the present invention, a series of evaluation parameters, such as dice coefficient, recall rate, precision and specificity, are used to evaluate the six commonly used models, and the results are shown in Table 1. As can be seen from the table, the DDRNets model has the highest dice coefficient. The model consists of a dual-resolution channel that can capture global and local context information, and can achieve accurate boundary positioning and robust segmentation performance.
[0063] Table 1 Comparison of results of six segmentation models
[0064] Model Dice coefficient Recall Accuracy Specificity DDRNets 0.928 0.88 0.892 0.999 Resnet50_unet 0.848 0.85 0.866 0.996 Resnet50_segnet 0.772 0.637 0.788 0.999 Unet 0.641 0.635 0.724 0.997 Unet_mini 0.455 0.373 0.759 0.999 Vgg_segnet 0.572 0.483 0.855 0.999
[0065] As can be seen from the above, the present invention has good segmentation accuracy by segmenting the enhanced image through the DDR-Nets model. The model generates a mask after image segmentation, and superimposes the generated mask with the input original image to obtain an image with mask.
[0066] 2. Classification
[0067] To ensure the accuracy and reliability of the endometrial segmentation results, the present invention proposes a dual-channel input model MSNet169 (MaskSupportNet169), and trains the binary classification model MSNet169 to evaluate the correctness of the segmented endometrial contour in representing the true endometrial area. The present invention uses a set of carefully annotated images to create a balanced training dataset specifically for this classification task.
[0068] like Figure 2 As shown in Figure 1, the input of the MSNet169 model is: image and mask. These two inputs are used to generate features, which are then fused together for subsequent classification tasks. The MSNet169 model consists of two branches, a feature fusion module connected to the outputs of the two branches, and a classifier.
[0069] (1) First branch: image features
[0070] Dense Connection: The original input image passes through multiple convolutional layers, using dense connections (Channel-wise concatenation), and the output features of each layer are cascaded with the subsequent layers. The channel features are connected between each layer through dense connections.
[0071] Average pooling (Avg Pool): After the dense connection layer, average pooling is performed on the feature map to reduce the dimension of the feature map.
[0072] (2) Second branch: mask features
[0073] Convolutional Layers (Conv2D, BatchNorm2D, ReLU): The input mask first passes through three convolutional layers, each of which includes a convolution operation (Conv2D), batch normalization (BatchNorm2D), and an activation function (ReLU) to extract features.
[0074] The number of output channels of the first layer is 64.
[0075] The number of output channels of the second layer is 128.
[0076] The number of output channels of the third layer is 256.
[0077] Spatial Attention Module: The extracted features pass through the spatial attention module to generate a spatial attention map.
[0078] The number of channels of the output feature map is 512.
[0079] (3) Feature Fusion
[0080] By performing element-wise multiplication operation on the mask feature map and the spatial attention map, the image features and mask features obtained by the first branch and the second branch are fused to obtain a fused feature map.
[0081] (4) Classifier
[0082] After obtaining the fused feature map, it is input into the fully connected layer, classified through the fully connected layer, and the final classification result is output.
[0083] Specifically, the MSNet169 model includes:
[0084] Dense connection: The present invention uses the pre-trained DenseNet169 network as the convolution layer to extract features from the image input. DenseNet169 is a deep convolutional neural network, which is characterized by each layer is directly connected to all subsequent layers. This dense connection method can effectively solve the problems of gradient disappearance and feature reuse.
[0085] Convolutional network is used to extract features from mask input. It contains multiple convolutional layers, each followed by a batch normalization layer and a ReLU activation function. This structure can effectively extract the features of the mask, and the nonlinearity of the model is increased by the ReLU activation function.
[0086] The adaptive average pooling layer is used to adjust the size of the mask feature to the same size as the DenseNet169 feature. This ensures the consistency of the spatial scale of the mask feature and the image feature, which facilitates subsequent feature fusion.
[0087] The spatial attention module is used to generate a spatial attention map, which is then applied to the mask feature to implement the spatial attention mechanism. This mechanism allows the model to pay more attention to important areas when processing mask features.
[0088] The fully connected layer is used to map the fused features to the target category. This layer is the output layer of the model, and its function is to map the high-dimensional feature vector to the low-dimensional category space. During the forward propagation process, the model first extracts features from the image input and the mask input, then fuses these two features together, and finally performs classification through the fully connected layer.
[0089] In the training process of the MSNet169 model, in order to solve the problem of imbalanced category data, we used the focal cross entropy loss function, but the present invention added a label smoothing operation to the loss to enhance the robustness of the model. The training was performed on a high-performance computing system, using GPU to accelerate convergence and achieve efficient parameter optimization.
[0090] The present invention achieves "downweighting" by adding a modulation factor in the cross entropy loss, so that difficult examples receive more attention.
[0091] The traditional focal cross entropy loss function is:
[0092] FL(p t )=-α t (1-p t ) γ log(p t ) (1)
[0093] Among them, α t represents the sample weight, p t represents the predicted probability value, and γ represents the adjustable focusing parameter;
[0094] It can be seen that the focal loss adds a factor (1-p t )γ If we set γ>0, we will reduce the relative loss of well-classified samples (pt>.5), thereby focusing more attention on misclassified samples that are difficult to classify.
[0095] The above p t The definition of p is: t ={p if y=1,1-p if y=0}, where y is the true label and p is the probability predicted by the model;
[0096] Label smoothing is to adjust the value of the true label so that it is no longer 0 or 1, but to assign some small non-zero values to other categories to alleviate the overfitting phenomenon. The label value after label smoothing is:
[0097] q i =(1-ε)p i +ε / K
[0098] Among them, q i is the smoothed label value, p i is the original label, ε is the smoothing parameter, and K is the number of categories;
[0099] The present invention adds a label smoothing operation to the traditional loss function to obtain a loss function after label smoothing, which is defined as:
[0100] FL(p t )=-∑(i=1to K)q i α i (1-p i ) γ log(p i )
[0101] The above loss function can enhance the robustness of the model.
[0102] In order to verify the effect of the MSNet169 model proposed in the present invention, the MSNet169 model was compared with the DenseNet169, DenseNet121, ResNet101 and ResNet50 models, and the effectiveness of the MSNet169 model in accurately classifying and segmenting the endometrial contour was evaluated. The experimental results are shown in Table 2.
[0103] As can be seen from Table 2, the MSNet169 model has the highest performance in endometrial segmentation and classification accuracy, with an accuracy of 0.9467, which verifies the superiority of the MSNet169 model in accurately capturing the complex structure of the endometrial region in ultrasound images.
[0104] In terms of accuracy, the MSNet169 model achieved the highest accuracy of 0.795, indicating its excellent ability to correctly identify positive cases.
[0105] In terms of specificity, the MSNet169 model has the highest specificity of 0.9672, indicating that it can accurately identify negative examples.
[0106] In summary, the experimental results demonstrate the superiority of the MSNet169 model in accurately capturing the complex structure of the endometrial region in ultrasound images.
[0107] Table 2 Binary classification results of five different models
[0108] Model Name Accuracy Recall Specificity Accuracy F-number ResNet50 0.8756 0.4649 0.9396 0.5455 0.502 ResNet101 0.9348 0.9315 0.9353 0.6918 0.794 DenseNet121 0.9434 0.8505 0.9579 0.7591 0.8022 DenseNet169 0.9444 0.9061 0.9504 0.7402 0.8148 MSNet169 0.9467 0.8149 0.9672 0.7950 0.8048
[0109] 3. Identify key frames
[0110] (1) Analysis of endometrial thickness
[0111] After the segmentation and classification steps, key frames will be identified from the correctly classified data. Since endometrial thickness is clinically significant in assessing uterine conditions, the present invention automatically measures endometrial thickness by identifying key frames. A two-step method for automatically measuring endometrial thickness has been disclosed in the existing published document “Automatic Measurement of Endometrial Thickness From Transvaginal Ultrasound lmages”. In this document, first, a semantic segmentation method based on deep learning is used to segment the endometrium from two-dimensional transvaginal ultrasound images. Secondly, a maximum inscribed circle search method is disclosed to calculate endometrial thickness. The detailed process is as follows. Figure 3 shown.
[0112] (2) Key frame analysis
[0113] The target region frame containing the endometrium is identified in the ultrasound video stream, that is, the masked endometrial image after correct segmentation, and the key frame with the largest endometrial thickness is extracted from the target region.
[0114] To verify the accuracy of keyframe identification, a team of professional sonographers manually selected keyframes for each video data case. The selection process took into account multiple factors, including visible endometrial borders, optimal image quality, and the presence of maximum endometrial thickness measurements. These manually selected keyframes served as a benchmark for evaluating the performance of the automatic keyframe detection method proposed in this paper. Figure 4 shown.
[0115] In order to evaluate the accuracy of the automatic method, an allowable error value for endometrial thickness was set. The endometrial thickness of the keyframes selected by the automatic method and the keyframes selected manually were compared to evaluate the correctness of keyframe identification. Keyframes within the allowable error range were considered accurate, indicating that the frame with the largest endometrial thickness was successfully detected.
[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A key frame detection method for endometrial images, characterized in that: The following steps are involved: Step 1: Acquire uterine ultrasound video data; Step 2: extract and segment the uterine ultrasound video data to obtain a segmented endometrial image; Step 3: Use the MSNet169 model to classify the segmented endometrial images, and determine all endometrial images suitable for measuring endometrial thickness based on the classification results; Step 4: Find key frames in all endometrial images obtained in step 3; Step 5: Output a key frame image containing maximum thickness information and the corresponding specific thickness value of the endometrium.
2. The key frame detection method of endometrial image according to claim 1, characterized in that: The specific steps of step 2 include: Step 21: Segment the input video data into a number of single-frame images; Step 22: Perform image enhancement on each single frame image; Step 23: Use the DDRNets model to segment the enhanced image to obtain a segmented endometrial image and generate an image mask; Step 24: Superimpose the generated mask and the original single-frame image to obtain a masked endometrial image.
3. The key frame detection method of endometrial image according to claim 1, characterized in that: The MSNet169 model includes two parallel branch networks. The first branch network is used to extract mask features to obtain a mask feature map, and the second branch network is used to extract image features. The two branch networks are connected through a feature fusion module. The output of the feature fusion module is connected to a fully connected layer. The fully connected layer is used to map the fused features to the target category, thereby achieving classification.
4. The key frame detection method of endometrial image according to claim 3, characterized in that: The first branch network includes a plurality of convolutional layers, a spatial attention module, and an adaptive average pooling layer connected in sequence, wherein the spatial attention module is used to generate a spatial attention map; The second branch network includes multiple convolutional layers using dense connections, the output features of each layer are cascaded with subsequent layers, and the channel features are connected between each layer through dense connections.
5. The key frame detection method of endometrial image according to claim 4, characterized in that: The feature fusion module fuses the mask features and the image features by performing element-wise multiplication operation on the mask feature map and the spatial attention map.
6. The key frame detection method of endometrial image according to claim 5, characterized in that: The specific steps of step 3 are: Step 31: Input the masked endometrial image into the MSNet169 model; Step 32: Use the pre-trained MSNet169 model to extract mask features and image features through two branch networks to obtain the final classification results; Step 33: Determine whether the current segmented endometrial image is suitable for measuring endometrial thickness based on the classification result, and output the endometrial image with a correct segmentation result for measuring thickness.
7. The key frame detection method of endometrial image according to claim 6, characterized in that: The MSNet169 model uses the focal cross entropy loss function for training, and its calculation formula is: FL(p t )=-∑(i=1to K)q i α i (1-p i ) γ log(p i ) in, q i =(1-ε)p i +e / K In the formula, α i represents the sample weight, γ represents the adjustable focusing parameter, q i is the smoothed label value, p i is the original label, ε is the smoothing parameter, K is the number of categories; p t Represents the predicted probability value.
8. The key frame detection method of endometrial image according to claim 6, characterized in that: Step 4 specifically includes: Step 41: import all correctly classified endometrial images with masks, find the inscribed circle using the maximum inscribed circle search method, and calculate the endometrial thickness; Step 42: The image frame corresponding to the time when the calculated endometrial thickness is the maximum is used as a key frame.
9. A key frame detection device for endometrial images, implemented based on the method according to any one of claims 1 to 8, characterized in that: The device includes: An input module, used for collecting uterine ultrasound video data through an ultrasound machine; The segmentation module is used to segment the input video data into several single-frame images, and perform image enhancement on each single-frame image. Finally, the enhanced image is segmented using the DDRNets model to generate a masked endometrial image. A classification module, used for classifying the segmented endometrial images and determining all endometrial images suitable for measuring endometrial thickness; A key frame detection module, for extracting key frames based on the correctly classified masked endometrial images; The output module is used to output a key frame image containing maximum thickness information and the corresponding specific thickness value of the endometrium.
10. The key frame detection device for endometrial images according to claim 9, characterized in that: The classification module comprises: An MSNet169 model building unit, used to build an MSNet169 model including two branch networks, wherein the two branch networks are used to extract mask features and image features respectively, and the obtained mask features and image features are fused through a feature fusion module to realize classification; An MSNet169 model classification unit, used to classify the input masked endometrial image using the MSNet169 model and output the classification result; The segmentation result evaluation unit is used to determine whether the current segmented endometrial image is suitable for measuring the endometrial thickness according to the classification result, and output the endometrial image with the correct segmentation result.