Uterine myoma multi-modal evaluation method and device

Abstract

The invention discloses a hysteromyoma multi-modal evaluation method and a related device, and the method comprises the steps: obtaining pelvic CT and MRI image data of a female patient, the image data comprising different sequence images of T2WI and CE-T1WI; pelvic CT and MRI image data are input into a multi-modal uterine fibroid fusion segmentation model, multi-dimensional quantitative features of the uterine fibroid are extracted and marked, and the multi-modal uterine fibroid fusion segmentation model comprises an input layer, a deformable convolutional layer, an RPN region layer, an RoI region selection layer, a multi-modal feature extraction network and an output layer; and inputting the multi-dimensional quantitative characteristics, the age of the patient and the hormone level into a hysteromyoma recurrence risk prediction model to obtain the postoperative recurrence risk probability of the hysteromyoma. The shape, texture, signal intensity distribution and diffusion coefficient quantitative characteristics are extracted through the multi-mode hysteromyoma fusion segmentation model, the microstructure and biological characteristics of the myoma are disclosed, and the hysteromyoma postoperative recurrence risk prediction precision is further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary diagnosis, and in particular to a multimodal evaluation method and device for uterine fibroids, and a computing device. Background Art

[0002] Uterine fibroids are benign tumors of the female reproductive system that are formed by the proliferation of uterine smooth muscle cells. Computed tomography (CT) and magnetic resonance imaging (MRI) can provide rich anatomical and pathological information, helping doctors to have a more comprehensive understanding of uterine fibroids.

[0003] Existing uterine fibroid assessment methods include level set-based methods and fuzzy C-means-based methods. Among them, the level set-based method constructs an energy function and uses image gradient information to guide the evolution of the curve to achieve target segmentation. However, when dealing with uterine fibroids with complex shapes, it is often difficult to obtain accurate segmentation results. In addition, the level set method is sensitive to initial conditions, and different initial conditions may lead to different segmentation results, thereby affecting the accuracy of the assessment. For example, the boundary of a sub-tumor may be ignored due to low contrast with surrounding tissues, or it may be mistakenly divided into adjacent tissues due to irregular boundaries. The fuzzy C-means-based method is a cluster analysis method, which divides pixels in the image into different categories by iteratively optimizing the objective function. However, when dealing with uterine fibroids with fuzzy boundaries, mis-segmentation is prone to occur. And it is also sensitive to changes in noise and image quality. When encountering uterine fibroids with fuzzy boundaries, it is difficult to accurately distinguish between fibroids and surrounding tissues, resulting in mis-segmentation. For example, pixels belonging to fibroids are mistakenly divided into surrounding tissues, and pixels belonging to surrounding tissues are mistakenly divided into fibroids.

[0004] To solve the above problems, the present invention proposes a multimodal evaluation method for uterine fibroids. The shape, texture, signal intensity distribution and diffusion coefficient quantitative characteristics are extracted through a multimodal uterine fibroid fusion segmentation model to reveal the microstructure and biological characteristics of the fibroids, and further improve the prediction accuracy of the postoperative recurrence risk of uterine fibroids. Summary of the invention

[0005] In view of the above problems, the present invention provides a multimodal evaluation method and device for uterine fibroids, and a computing device.

[0006] According to one aspect of the present invention, a multimodal evaluation method for uterine fibroids is provided, comprising:

[0007] Acquire pelvic CT and MRI image data of a female patient, wherein the image data includes different sequence images of T2WI and CE-T1WI;

[0008] Inputting the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model to extract and annotate multidimensional quantitative features of uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network, and an output layer;

[0009] The multidimensional quantitative features, the patient's age and hormone levels are input into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2D UNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN regional layer.

[0010] In an optional manner, the multidimensional quantitative features include the type, shape, size, number, location, texture, signal intensity distribution, diffusion coefficient and perfusion parameters of the uterine fibroids;

[0011] The types of uterine fibroids include smooth muscle cell hyperplasia, adenomyoma-like changes and malignant changes.

[0012] In an optional manner, the multimodal feature extraction network includes multiple residual networks RestNet-101, multiple CNN network layers, multiple attention gates, GCN graph convolution network layers, multiple gate signals, multiple connection layers, multiple 3D convolution blocks and multiple upsampling layers;

[0013] Each of the residual networks RestNet-101 is alternately connected to the corresponding CNN network layer in sequence, and the CNN network layer connected to the last residual network RestNet-101 is connected to the attention gate, and the CNN network layer is connected through 3×3 convolution and attention gate; wherein the residual network RestNet-101 abandons the last 3 layers of the output layer;

[0014] The CNN network layer is sequentially connected to the next CNN network layer through gate signals and 3×3 convolutions;

[0015] The GCN graph convolutional network layer is used to emphasize or suppress the output features and corresponding weights of each CNN network layer.

[0016] In an optional manner, the formula for adjusting the sampling position of the convolution kernel in the deformable convolution layer in the multimodal uterine fibroid fusion segmentation model is:

[0017]

[0018] Among them, y(p) is the value of position p in the output feature map; x(p+p k +Δp k ) is the input feature map after the offset Δpk The sample value after the offset Δp k Learned from the input feature map through the convolutional layer; w k is the weight of the kth convolution kernel; p k is the regular position offset of the convolution kernel; K is the total number of convolution kernels.

[0019] In an optional manner, the RPN region layer is used to generate a candidate region, and the coordinate offset prediction formula of the candidate frame of the candidate region is:

[0020]

[0021] Among them, (x, y, w, h) are the coordinates and size of the prediction box; (x a ,y a ,w a ,h a ) is the coordinates and size of the anchor box; the offset t x ,t y ,t w ,t h Used to train RPN to accurately predict candidate regions.

[0022] In an optional manner, the step of enhancing feature representation by the SENet hybrid attention layer includes:

[0023] The feature map of each channel is compressed into a real number through global average pooling to capture global information;

[0024] The weights of each channel are learned through two fully connected layers; the first fully connected layer is followed by ReLU activation and the second fully connected layer is followed by Sigmoid activation.

[0025] In an optional manner, the FPN regional layer enhances multi-scale feature representation through a top-down path and lateral connections;

[0026] Among them, the first layer feature map P in the feature pyramid in the FPN area layer l The construction formula is:

[0027] P l =Conv(P l+1 ↑+Resize(C l ))

[0028] Among them, P l+1 ↑ is the feature map obtained by upsampling the feature map of the l+1th layer; C l is the feature map corresponding to the original image size in the residual network; the Resize function is the resize function to match P l+1↑ Size; Conv is a convolution operation, which is used to fuse features and reduce the aliasing effect caused by upsampling.

[0029] In an optional manner, the method further includes:

[0030] Obtaining the patient's previous treatment feature information, including myomectomy, uterine artery embolization, treatment time, symptom improvement, and change in myoma size; inputting the previous treatment feature information into the 2DUNet variant network through one-hot encoding;

[0031] The contribution of each previous treatment feature to the uterine fibroid recurrence risk prediction model is evaluated by the SHAP value; wherein the SHAP value is obtained by calculating the average marginal contribution of each feature in the prediction of the uterine fibroid recurrence risk prediction model, and the prediction results affected by each feature are displayed in the form of a honey bee diagram.

[0032] According to another aspect of the present invention, a multimodal evaluation device for uterine fibroids is provided, comprising:

[0033] A CT and MRI image data acquisition module, used to acquire pelvic CT and MRI image data of female patients, wherein the image data includes different sequence images of T2WI and CE-T1WI;

[0034] A uterine fibroid fusion segmentation module, used to input the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model, extract and annotate multidimensional quantitative features of uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network and an output layer;

[0035] The uterine fibroid recurrence risk prediction module is used to input the multidimensional quantitative features, the patient's age and hormone levels into the uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein, the uterine fibroid recurrence risk prediction model is a 2DUNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN regional layer.

[0036] According to another aspect of the present invention, there is provided a computing device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus.

[0037] According to the scheme provided by the present invention, pelvic CT and MRI image data of female patients are obtained, wherein the image data include different sequence images of T2WI and CE-T1WI; the pelvic CT and MRI image data are input into a multimodal uterine fibroid fusion segmentation model, and the multidimensional quantitative features of uterine fibroids are extracted and annotated, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, a RoI region selection layer, a multimodal feature extraction network, and an output layer; the multidimensional quantitative features, the patient's age and hormone level are input into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2DUNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN region layer. The present invention extracts shape, texture, signal intensity distribution and diffusion coefficient quantitative features through a multimodal uterine fibroid fusion segmentation model, reveals the microstructure and biological characteristics of the fibroids, and further improves the accuracy of postoperative recurrence risk prediction of uterine fibroids.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0040] Figure 1 A schematic diagram showing a flow chart of a multimodal evaluation method for uterine fibroids according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a multimodal uterine fibroid fusion segmentation model according to an embodiment of the present invention is shown;

[0042] Figure 3 A schematic diagram of a multimodal feature extraction network MMFEN according to an embodiment of the present invention is shown;

[0043] Figure 4A A schematic diagram of a uterine fibroid in a T2WI sequence according to an embodiment of the present invention is shown;

[0044] Figure 4B A schematic diagram of standard apparent diffusion in an embodiment of the present invention is shown;

[0045] Figure 4CA schematic diagram of slow apparent diffusion according to an embodiment of the present invention is shown;

[0046] Figure 4D A schematic diagram of rapid apparent diffusion according to an embodiment of the present invention is shown;

[0047] Figure 4E A schematic diagram of perfusion fraction according to an embodiment of the present invention is shown;

[0048] Figure 5 A schematic diagram of the framework of a multimodal evaluation device for uterine fibroids according to an embodiment of the present invention is shown;

[0049] Figure 6 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0051] Figure 1 FIG. 4 is a flow chart of a multimodal evaluation method for uterine fibroids according to an embodiment of the present invention. Specifically, Figure 1 As shown, the following steps are included:

[0052] Step S101, obtaining pelvic CT and MRI image data of a female patient, wherein the image data includes different sequence images of T2WI and CE-T1WI.

[0053] In this embodiment, through the different sequence images of T2WI and CE-T1WI in the pelvic CT and MRI image data of female patients, the anatomical structure and pathological details of each organ in the pelvic cavity can be clearly displayed, thereby providing more abundant diagnostic information to more comprehensively evaluate the condition. Among them, T2WI uses cross-sectional, sagittal and coronal imaging, with a layer thickness of 3 to 5 mm and a spacing of 30%. CE-T1WI is enhanced by intravenous injection of contrast agent (such as 0.5 mol / L Gd-DTPA, 0.1 mmol / Kg, 0.5 to 1 ml / s speed) on the basis of T1WI. For example, the T2WI image shows a low signal area in the myometrium with clear boundaries, and uterine fibroids may be present. After intravenous injection of contrast agent, the CE-T1WI image shows no obvious enhancement in the uterine fibroid area, which forms a clear contrast with the surrounding normal myometrium, and uterine fibroids can be further diagnosed. At the same time, the size, morphology and adjacent relationship of uterine fibroids with surrounding tissues can be observed.

[0054] Step S102, input the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model, extract and annotate the multidimensional quantitative features of the uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network and an output layer.

[0055] In this embodiment, the multimodal uterine fibroid fusion segmentation model simultaneously processes image data of two different modalities, CT and MRI, and fully utilizes the advantages of different imaging technologies to capture more pathological information of uterine fibroids such as morphology, density, perfusion, etc., which helps to more comprehensively evaluate the condition. The RPN (Region Proposal Network) regional layer is combined with the RoI (Region of Interest) regional selection layer to further screen the region of interest containing uterine fibroids. The image data in the RoI area is feature extracted and annotated through the multimodal feature extraction network to obtain multidimensional quantitative features including shape, size, texture, density, etc.

[0056] Specifically, Figure 2 As shown in the figure, the input pelvic CT and MRI image data are preprocessed by denoising, normalization, and registration. The preprocessed CT and MRI image data are fused in the input layer to form multimodal image data. The fused multimodal image data is input into the deformable convolution layer, and the complex structure and geometric deformation in the image are captured by adaptive convolution kernel adjustment. The RPN layer (RPN layer identifies the area containing the target object in the image through the sliding window and anchor mechanism) is used to quickly generate candidate regions containing potential uterine fibroids in the image. According to the candidate regions generated by the RPN layer, the region of interest containing uterine fibroids is further screened out through the RoI region selection layer. The image data in the RoI region is extracted by the multimodal feature extraction network (hereinafter referred to as MMFEN network) including multidimensional quantitative features such as shape, size, texture, and density. The extracted features are annotated, and the final segmentation results and feature information are output, that is, the precise position and shape of the uterine fibroid in the image, as well as the related feature information, and displayed on the screen in the form of color enhancement.

[0057] In an optional manner, the multidimensional quantitative features include the type, shape, size, number, location, texture, signal intensity distribution, diffusion coefficient and perfusion parameters of the uterine fibroids;

[0058] The types of uterine fibroids include smooth muscle cell hyperplasia, adenomyoma-like changes and malignant changes.

[0059] like FIG. 4A to FIG. 4EAs shown in the figure, the multidimensional quantitative characteristics of fibroids include shape (oval), size (about 5 cm in diameter), number (single), location (intramural), texture (uniform), signal intensity distribution (high signal on T2WI), ADC value (1200×10^-6mm2 / s), and perfusion parameters (high Ktrans value). Based on the multidimensional quantitative characteristics of uterine fibroids, combined with clinical pathological information, uterine fibroids are classified and identified (such as smooth muscle cell hyperplasia, adenomyoma-like changes, and malignant changes).

[0060] In an optional manner, the multimodal feature extraction network includes multiple residual networks RestNet-101, multiple CNN network layers, multiple attention gates, GCN graph convolution network layers, multiple gate signals, multiple connection layers, multiple 3D convolution blocks and multiple upsampling layers;

[0061] Each of the residual networks RestNet-101 is alternately connected to the corresponding CNN network layer in sequence, and the CNN network layer connected to the last residual network RestNet-101 is connected to the attention gate, and the CNN network layer is connected through 3×3 convolution and attention gate; wherein the residual network RestNet-101 abandons the last 3 layers of the output layer;

[0062] The CNN network layer is sequentially connected to the next CNN network layer through gate signals and 3×3 convolutions;

[0063] The GCN graph convolutional network layer is used to emphasize or suppress the output features and corresponding weights of each CNN network layer.

[0064] In this embodiment, the residual network is alternately connected to the CNN network layer to further enhance the diversity of feature extraction and capture more levels of feature information. By screening the features through the attention gate, it is possible to focus on more important features and suppress irrelevant noise. The GCN graph convolutional network layer adjusts the weights of the output features of each CNN network layer according to the graph structure information to enhance the processing capability of graph structure data. The upsampling operation increases the resolution of the feature while maintaining the feature information.

[0065] Specifically, Figure 3As shown in the figure, multiple ResNet-101 residual networks are initialized and their structures are adjusted to discard the last three layers of the output layer to reduce the amount of computation and avoid overfitting. Multiple CNN network layers are constructed, each of which contains convolution operations, activation functions, and pooling operations. Attention gates and GCN graph convolution network layers are inserted. Each ResNet-101 residual network is alternately connected to a CNN network layer in sequence, and a CNN network layer is connected after the last residual network, and the features are further processed through attention gates and 3×3 convolutions. Gate signals are inserted between CNN network layers to control the transmission of feature flows. Convolution operations are performed using 3×3 convolution kernels to extract local features. Features from different CNN network layers are fused and weighted through the GCN graph convolution network layer, and processed multimodal features are output. Optionally, the green 3*3 convolution is used for upsampling, while the yellow 1*1 convolution is used to convert channels to obtain segmentation masks. After four downsamplings and four upsamplings, downsampling is achieved by alternating the connection between ResNet-101 and CNN network, and upsampling is achieved by 3*3 convolution. During one upsampling, the feature map is divided into two paths. One path passes through 3*3 convolution, the number of channels is doubled, and the image size is halved. The other path is input into the attention gate together with the result after the third downsampling, and then spliced ​​with the output of 3*3 convolution. The spliced ​​result is input into a CNN network layer, and then passes through the next upsampling part, and this step is repeated until the feature is restored to its original size. It is worth noting that during the third upsampling, the first feature map is combined with the features from different CNN network layers through the GCN graph convolution network layer to fuse and adjust the weights. Figure 1 It passes through the attention gate before being spliced ​​with the feature map after the second 3*3 convolution.

[0066] In an optional manner, the formula for adjusting the sampling position of the convolution kernel in the deformable convolution layer in the multimodal uterine fibroid fusion segmentation model is:

[0067]

[0068] Among them, y(p) is the value of position p in the output feature map; x(p+p k +Δp k ) is the input feature map after the offset Δp k The sample value after the offset Δp k Learned from the input feature map through the convolutional layer; w k is the weight of the kth convolution kernel; p k is the regular position offset of the convolution kernel; K is the total number of convolution kernels.

[0069] In this embodiment, the deformable convolution allows the sampling position of the convolution kernel to be dynamically adjusted according to the local information of the input feature map, so as to better adapt to the shape changes of the uterine fibroids and reduce the errors in the segmentation process, so as to more accurately capture the subtle features and complex structures (irregular shapes and boundaries) in the uterine fibroids image. In addition, the deformable convolution reduces the dependence on a large amount of labeled data, reduces the burden of labeling work, and improves the generalization ability of the model.

[0070] In an optional manner, the RPN region layer is used to generate a candidate region, and the coordinate offset prediction formula of the candidate frame of the candidate region is:

[0071]

[0072] Among them, (x, y, w, h) are the coordinates and size of the prediction box; (x a ,y a ,w a ,h a ) is the coordinates and size of the anchor box; the offset t x ,t y ,t w ,t h Used to train RPN to accurately predict candidate regions.

[0073] In this embodiment, RPN generates candidate regions directly on the feature map by full convolution, which reduces the amount of calculation of traditional methods (such as sliding windows or selective search methods) and improves the detection speed. The position and size of the candidate box are finely adjusted through the coordinate offset prediction formula to make it closer to the real target bounding box (candidate boxes of different scales and aspect ratios).

[0074] Specifically, multiple anchor boxes (anchors) are generated at each position of the feature map according to the preset scale and aspect ratio as the initial box of the candidate area. A sliding window is used to slide on the feature map, and the features generated at each position are processed by the convolution layer in the RPN. The processing includes two parallel branches: the classification branch and the regression branch. Among them, the classification branch is used to determine whether the target is contained in the anchor box (that is, to determine whether the anchor box is a positive sample or a negative sample). The regression branch uses the coordinate offset prediction formula to calculate the offset t from the anchor box to the true target bounding box x ,t y ,t w ,t h According to the result of the classification branch, the anchor boxes that do not contain the target are removed. The remaining anchor boxes are adjusted using the offset predicted by the regression branch to obtain the final candidate regions. Optionally, non-maximum suppression is performed on the candidate regions to eliminate redundant candidate boxes.

[0075] Step S103, inputting the multidimensional quantitative features, the patient's age and hormone levels into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2D UNet variant network, and the 2D UNet variant network introduces a SENet hybrid attention layer and an FPN regional layer.

[0076] In this embodiment, the SENet hybrid attention layer automatically learns the importance of each feature channel and recalibrates the features by explicitly modeling the interdependence between feature channels. In the prediction of the recurrence risk of uterine fibroids, more attention is paid to the features that are highly correlated with the recurrence risk. In addition, the SENet hybrid attention layer not only considers the attention between channels, but also optimizes the feature map in combination with spatial attention, which can more effectively capture the key information of the lesion area when extracting uterine fibroids and their related features.

[0077] In an optional manner, the step of enhancing feature representation by the SENet hybrid attention layer includes:

[0078] The feature map of each channel is compressed into a real number through global average pooling to capture global information;

[0079] The weights of each channel are learned through two fully connected layers; the first fully connected layer is followed by ReLU activation and the second fully connected layer is followed by Sigmoid activation.

[0080] Specifically, for a feature map X of shape (C, H, W), where C is the number of channels, H and W are the height and width respectively, global average pooling is performed on the feature map of each channel, that is, the feature map of each channel is compressed into a real number (scalar) to obtain a tensor of shape (C, 1, 1), in order to capture global information and reduce the spatial dimension of the feature map.

[0081] The first fully connected layer inputs the (C, 1, 1) tensor obtained in the previous step into the fully connected layer, which compresses the number of channels to a smaller value (for example, C / r, where r is the dimension reduction coefficient) to reduce the amount of calculation. After that, nonlinearity is added through the ReLU activation function.

[0082] The second fully connected layer inputs the ReLU activated features into another fully connected layer, which restores the number of channels to the original value C. After that, the output value is normalized to (0, 1) through the Sigmoid activation function to obtain the weight of each channel.

[0083] The learned weights are multiplied by the original feature map X channel by channel to obtain the weighted feature map, and the features are recalibrated according to the importance of each channel to enhance the useful feature representation.

[0084] In an optional manner, the FPN regional layer enhances multi-scale feature representation through a top-down path and lateral connections;

[0085] Among them, the first layer feature map P in the feature pyramid in the FPN area layer l The construction formula is:

[0086] P l =Conv(P l+1 ↑+Resize(C l ))

[0087] Among them, P l+1 ↑ is the feature map obtained by upsampling the feature map of the l+1th layer; C l is the feature map corresponding to the original image size in the residual network; the Resize function is the resize function to match P l+1 ↑ Size; Conv is a convolution operation, which is used to fuse features and reduce the aliasing effect caused by upsampling.

[0088] In this embodiment, FPN only performs one forward propagation on the input image in the feature extraction stage, and then reuses these features through top-down paths and lateral connections, which can significantly improve the model's ability to detect small targets.

[0089] Specifically, features are extracted layer by layer from the input image using ResNet, VGG, etc. as the backbone network. Each layer contains feature information of different scales and levels. Starting from the last feature map of the backbone network (e.g., C5), a 1x1 convolution is used to reduce the number of channels to generate a high-level feature map P5. P5 is upsampled (e.g., bilinear interpolation or deconvolution) to match its size with the feature map of the next layer (e.g., C4). A lateral connection is added between the upsampled P5 and C4, and the number of channels of C4 is adjusted by 1x1 convolution to ensure that it has the same number of channels as P5. The adjusted C4 is element-wise added to the upsampled P5 to obtain a new feature map P4. The above process is repeated until all required scales (e.g., P3, P2, etc.) are covered. Optionally, additional convolutional layers are attached to each fused feature map (P2 to P5) to further fuse features and eliminate the aliasing effect caused by upsampling. The additional convolutional layers use a 3x3 convolution kernel and include an activation function.

[0090] In an optional manner, the method further includes:

[0091] Obtaining the patient's previous treatment feature information, including myomectomy, uterine artery embolization, treatment time, symptom improvement, and change in myoma size; inputting the previous treatment feature information into the 2DUNet variant network through one-hot encoding;

[0092] The contribution of each previous treatment feature to the uterine fibroid recurrence risk prediction model is evaluated by the SHAP value; wherein the SHAP value is obtained by calculating the average marginal contribution of each feature in the prediction of the uterine fibroid recurrence risk prediction model, and the prediction results affected by each feature are displayed in the form of a honey bee diagram.

[0093] In this embodiment, the SHAP value provides an intuitive understanding of the importance of each feature by calculating the average marginal contribution of each feature in the model prediction, thereby enhancing the interpretability of the model. At the same time, the SHAP value analysis identifies which features have the greatest impact on the model prediction results, which helps with subsequent feature selection and model optimization. Displaying the SHAP value in the form of a bee diagram intuitively shows the impact of each feature on the prediction results.

[0094] For example, a dataset of 100 patients who underwent treatments such as myomectomy or uterine artery embolization, with information such as treatment time, symptom improvement, and changes in fibroid size, was used to train a 2DUNet variant network to predict the recurrence risk of uterine fibroids. The ADAM optimizer was used for training, with an initial learning rate of 1×10 -3 , the learning rate decay coefficient used is 3×10 -5 , bach_size is set to 16, 200 epochs are set for training, the cross entropy loss function is used to calculate the loss value, and then the model parameters are updated through back propagation, and the model parameters with the best effect in training are finally saved. After the training is completed, the SHAP value of each feature is calculated using the SHAP library, and it is found that "treatment time" and "fibroid size change" are the two most important features affecting the recurrence risk prediction results. The longer the "treatment time", the lower the recurrence risk; and the greater the "fibroid size change" (that is, the more obvious the reduction of fibroid volume after treatment), the lower the recurrence risk. To intuitively display these results, the honey bee chart is used to draw the distribution of the SHAP values ​​of the two features "treatment time" and "fibroid size change". It can be clearly seen from the figure that with the increase of "treatment time" and the increase of "fibroid size change", the number of data points predicted as low recurrence risk gradually increases, which helps clinicians to formulate more reasonable treatment plans.

[0095] According to the scheme provided by the present invention, pelvic CT and MRI image data of female patients are obtained, wherein the image data include different sequence images of T2WI and CE-T1WI; the pelvic CT and MRI image data are input into a multimodal uterine fibroid fusion segmentation model, and the multidimensional quantitative features of uterine fibroids are extracted and annotated, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, a RoI region selection layer, a multimodal feature extraction network, and an output layer; the multidimensional quantitative features, the patient's age and hormone level are input into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2DUNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN region layer. The present invention extracts shape, texture, signal intensity distribution and diffusion coefficient quantitative features through a multimodal uterine fibroid fusion segmentation model, reveals the microstructure and biological characteristics of the fibroids, and further improves the accuracy of postoperative recurrence risk prediction of uterine fibroids.

[0096] Figure 5 The schematic diagram of the framework of the multimodal evaluation device for uterine fibroids of an embodiment of the present invention is shown. The multimodal evaluation device for uterine fibroids comprises:

[0097] CT and MRI image data acquisition module 501, used to acquire pelvic CT and MRI image data of female patients, wherein the image data includes different sequence images of T2WI and CE-T1WI;

[0098] The uterine fibroid fusion segmentation module 602 is used to input the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model to extract and annotate multidimensional quantitative features of uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network, and an output layer;

[0099] The uterine fibroid recurrence risk prediction module 503 is used to input the multidimensional quantitative features, the patient's age and hormone levels into the uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2D UNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN regional layer.

[0100] Figure 6 The schematic diagram of the structure of the computing device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

[0101] like Figure 6As shown, the computing device may include: a processor (processor) 602 , a communication interface (Communications Interface) 602 , a memory (memory) 606 , and a communication bus 608 .

[0102] The processor 602, the communication interface 602, and the memory 606 communicate with each other via a communication bus 608. The communication interface 602 is used to communicate with other devices such as a client or other server network elements. The processor 602 is used to execute a program 610, which can specifically execute the relevant steps in the above-mentioned uterine fibroid multimodal assessment method embodiment.

[0103] Specifically, the program 610 may include program codes, which include computer operation instructions.

[0104] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computing device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0105] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0106] According to the scheme provided by the present invention, pelvic CT and MRI image data of female patients are obtained, wherein the image data include different sequence images of T2WI and CE-T1WI; the pelvic CT and MRI image data are input into a multimodal uterine fibroid fusion segmentation model, and the multidimensional quantitative features of uterine fibroids are extracted and annotated, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, a RoI region selection layer, a multimodal feature extraction network, and an output layer; the multidimensional quantitative features, the patient's age and hormone level are input into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2DUNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN region layer. The present invention extracts shape, texture, signal intensity distribution and diffusion coefficient quantitative features through a multimodal uterine fibroid fusion segmentation model, reveals the microstructure and biological characteristics of the fibroids, and further improves the accuracy of postoperative recurrence risk prediction of uterine fibroids.

[0107] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition they can be divided into multiple submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose. In addition, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments means being within the scope of the present invention and forming different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination. The present invention can be implemented by hardware including several different elements and by appropriately programmed computers. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The steps in the above embodiments should not be understood as limiting the execution order unless otherwise specified.

Claims

1. A multimodal evaluation method for uterine fibroids, characterized in that: include: Acquire pelvic CT and MRI image data of a female patient, wherein the image data includes different sequence images of T2WI and CE-T1WI; Inputting the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model to extract and annotate multidimensional quantitative features of uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network, and an output layer; The multidimensional quantitative features, the patient's age and hormone levels are input into a uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein the uterine fibroid recurrence risk prediction model is a 2D UNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN regional layer.

2. The multimodal assessment method for uterine fibroids according to claim 1, characterized in that: The multidimensional quantitative features include the type, shape, size, number, location, texture, signal intensity distribution, diffusion coefficient and perfusion parameters of uterine fibroids; The types of uterine fibroids include smooth muscle cell hyperplasia, adenomyoma-like changes and malignant changes.

3. The multimodal evaluation method for uterine fibroids according to claim 1 or 2, characterized in that: The multimodal feature extraction network includes multiple residual networks RestNet-101, multiple CNN network layers, multiple attention gates, GCN graph convolution network layers, multiple gate signals, multiple connection layers, multiple 3D convolution blocks and multiple upsampling layers; Each of the residual networks RestNet-101 is alternately connected to the corresponding CNN network layer in sequence, and the CNN network layer connected to the last residual network RestNet-101 is connected to the attention gate, and the CNN network layer is connected through 3×3 convolution and attention gate; wherein the residual network RestNet-101 abandons the last 3 layers of the output layer; The CNN network layer is sequentially connected to the next CNN network layer through gate signals and 3×3 convolutions; The GCN graph convolutional network layer is used to emphasize or suppress the output features and corresponding weights of each CNN network layer.

4. The multimodal assessment method for uterine fibroids according to claim 3, characterized in that: The formula for adjusting the sampling position of the convolution kernel in the deformable convolution layer in the multimodal uterine fibroid fusion segmentation model is: Among them, y(p) is the value of position p in the output feature map; x(p+p k +Δp k ) is the input feature map after the offset Δp k The sample value after the offset Δp k Learned from the input feature map through the convolutional layer; w k is the weight of the kth convolution kernel; p k is the regular position offset of the convolution kernel; K is the total number of convolution kernels.

5. The multimodal assessment method for uterine fibroids according to claim 1, characterized in that: The RPN region layer is used to generate a candidate region, and the coordinate offset prediction formula of the candidate frame of the candidate region is: Among them, (x, y, w, h) are the coordinates and size of the prediction box; (x a ,y a ,w a ,h a ) is the coordinates and size of the anchor box; the offset t x ,t y ,t w ,t h Used to train RPN to accurately predict candidate regions.

6. The multimodal evaluation method for uterine fibroids according to claim 1, characterized in that: The steps of SENet hybrid attention layer enhancing feature representation include: The feature map of each channel is compressed into a real number through global average pooling to capture global information; The weights of each channel are learned through two fully connected layers; the first fully connected layer is followed by ReLU activation and the second fully connected layer is followed by Sigmoid activation.

7. The multimodal evaluation method for uterine fibroids according to claim 1, characterized in that: The FPN regional layer enhances multi-scale feature representation through top-down paths and lateral connections; Among them, the first layer feature map P in the feature pyramid in the FPN area layer l The construction formula is: P l =Conv(P l+1 ↑+Resize(C l )) Among them, P l+1 ↑ is the feature map obtained by upsampling the feature map of the l+1th layer; C l is the feature map corresponding to the original image size in the residual network; the Resize function is the resize function to match P l+1 ↑ Size; Conv is a convolution operation, which is used to fuse features and reduce the aliasing effect caused by upsampling.

8. The multimodal evaluation method for uterine fibroids according to claim 1, characterized in that: The method further comprises: Obtaining the patient's previous treatment feature information, including myomectomy, uterine artery embolization, treatment time, symptom improvement, and change in myoma size; inputting the previous treatment feature information into the 2D UNet variant network through one-hot encoding; The contribution of each previous treatment feature to the uterine fibroid recurrence risk prediction model is evaluated by the SHAP value; wherein the SHAP value is obtained by calculating the average marginal contribution of each feature in the prediction of the uterine fibroid recurrence risk prediction model, and the prediction results affected by each feature are displayed in the form of a honey bee diagram.

9. A multimodal evaluation device for uterine fibroids, characterized in that: include: A CT and MRI image data acquisition module, used to acquire pelvic CT and MRI image data of female patients, wherein the image data includes different sequence images of T2WI and CE-T1WI; A uterine fibroid fusion segmentation module, used to input the pelvic CT and MRI image data into a multimodal uterine fibroid fusion segmentation model, extract and annotate multidimensional quantitative features of uterine fibroids, wherein the multimodal uterine fibroid fusion segmentation model includes an input layer, a deformable convolution layer, an RPN region layer, an RoI region selection layer, a multimodal feature extraction network and an output layer; The uterine fibroid recurrence risk prediction module is used to input the multidimensional quantitative features, the patient's age and hormone levels into the uterine fibroid recurrence risk prediction model to obtain the postoperative recurrence risk probability of uterine fibroids; wherein, the uterine fibroid recurrence risk prediction model is a 2DUNet variant network, and the 2D UNet variant network introduces a SENet mixed attention layer and an FPN regional layer.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned uterine fibroid multimodal assessment method.

Images