Ovarian Mass Segmentation Device Based on Grouping of Multi-Morphological Mass Image Features

Abstract

The present invention discloses an ovarian mass segmentation device based on multi-modal mass image feature grouping, belonging to the field of medical image segmentation, including: after performing multi-modal mass image feature grouping and labeling category tags on the solid part in the ovarian cancer ultrasound image cyst, performing ultrasound image preprocessing; constructing an ovarian mass category recognition model including a first ovarian mass classification module based on a fusion collaborative adaptive Transformer, a second ovarian mass classification module based on radiomics features, and a confidence voting module, wherein the three types of preprocessed ultrasound images pass through the first ovarian mass classification module to obtain a first classification result, and the radiomics features extracted from the three types of preprocessed images pass through the second ovarian mass classification module to obtain a second classification result, and the two classification results obtain a final classification result through the confidence voting module; constructing an exclusive lesion area segmentation model corresponding to each ovarian mass category and accurately segmenting the lesion area.

Description

Technical Field

[0001] The present invention belongs to the technical field of image segmentation, and particularly relates to an ovarian mass segmentation device based on grouping of multi-morphological mass image features. Background Art

[0002] Ovarian masses are common physiological changes or pathological changes in the female reproductive system, presenting in various forms such as cystic, cystic-solid or solid. Ultrasound is the preferred non-invasive imaging method for screening and classifying the nature of ovarian masses. However, due to the complex images, especially the features of the solid part containing rich tumor heterogeneity features, such as all-solid, papillary projections, blood flow in the solid part, and multi-chambered septa in the cyst as mentioned in the IOTA ADNEX model and the ORADS ultrasound system, their interpretation depends on doctors' experience and knowledge. Research shows that for ultrasound features other than size and volume measurement, the inter-observer agreement is only 0.45 - 0.58.

[0003] Artificial intelligence can increase the consistency among ultrasound examiners. However, most artificial intelligence medical imaging technologies are based on the recognition and delineation of regions of interest in images. Manual delineation is a time-consuming and highly professional process, and may introduce subjective judgments of observers, resulting in inter-observer inconsistencies. Automatic segmentation technology greatly reduces the time consumption, commonly based on features such as the shape, color, and grayscale of the image, such as edge detection segmentation, threshold-based segmentation methods, and watershed algorithms. In recent years, the developed deep learning technology has achieved pixel-level semantic segmentation in image segmentation. Liu et al. trained the FCNResNet101 model to segment ovarian masses, and its Dice similarity coefficient (DSC) value reached 89.1%, and used the automatically segmented ultrasound images to evaluate low malignant risk and medium-high malignant risk adnexal masses based on O-RADS. Wang et al. proposed an ovarian ultrasound image segmentation algorithm that fuses multi-scale features, introduced a pyramid pooling structure while retaining the residual connection structure of the U-net network, increased the receptive field, and improved the recognition rate of features of different sizes, making up for the lost semantic information in downsampling. It is worth noting that these technologies mainly focus on the edge detection and extraction of the whole ovarian mass, and do not further segment the complex solid structure inside the ovarian mass.

[0004] The difficulties in the automatic segmentation of the solid part of ovarian masses in ultrasound images mainly include: 1) The tumor heterogeneity in the solid region is high, and there may be pathological conditions such as bleeding, necrosis, or calcification, resulting in large differences in the internal acoustic characteristics of the solid part; 2) Compared with cystic masses, the acoustic impedance of the solid region is similar to that of surrounding tissues such as the intestine, and the contrast of the acoustic interface determined by the difference in acoustic impedance is insufficient, making it difficult to identify the edge. The segmentation accuracy of masses mainly composed of solid lesions is 7% lower than that of masses mainly composed of cystic lesions; 3) For multi-chambered cystic masses of the ovary, the septa can be thin bands (less than 3 mm), requiring a relatively high pixel-level segmentation accuracy of the algorithm; 4) The ovarian appearance is affected by the physiological cycle of women of childbearing age, and physiological masses may also have complex internal echoes, such as bleeding in the corpus luteum, etc., which are easily confused with the solid part of the mass. These factors make it challenging to accurately identify and segment ovarian masses, especially the solid part. In addition, the anatomical location of the ovary is relatively deep, and generally 3-5 MHz (trans-abdominal) or 5-9 MHz (trans-vaginal) ultrasound probes are used to meet the penetration of sound waves, and the image resolution is not as good as that of superficial parts such as the breast and thyroid (generally 7-18 MHz probes), further increasing the difficulty of automatic segmentation and analysis of ultrasound images. Summary of the Invention

[0005] In view of the above, aiming at the difficulties in image analysis such as high solid heterogeneity, complex structure, insufficient contrast, interference from surrounding tissues, and deep location of ovarian masses in ultrasound images, as well as the technical requirements of difficult automatic segmentation and high segmentation accuracy requirements, the purpose of the present invention is to provide an ovarian mass segmentation device based on the grouping of multi-morphological mass image features. On the basis of grouping the multi-morphological mass image features, by introducing a collaborative attention mechanism and a deep learning automatic segmentation algorithm, it is possible to accurately segment the complex solid structure inside the ovarian mass, provide a more definite region of interest for the analysis of artificial intelligence-related technologies, and improve the accuracy of automatic differential diagnosis of ultrasound images.

[0006] To achieve the above invention purpose, an ovarian mass segmentation device based on the grouping of multi-morphological mass image features provided by an embodiment includes:

[0007] A data acquisition and preprocessing unit, which is used to acquire ultrasound images of ovarian cancer, group the multi-morphological mass image features of the solid part inside the cyst, mark the ovarian mass category labels, and then perform preprocessing of the ultrasound images;

[0008] An ovarian mass category recognition unit, which is used to construct an ovarian mass category recognition model and perform ovarian mass category recognition. The ovarian mass category recognition model includes a first ovarian mass classification module based on a fused collaborative adaptive Transformer, a second ovarian mass classification module based on radiomics features, and a confidence voting module. The three types of preprocessed ultrasound images pass through the first ovarian mass classification module to obtain a first classification result. The radiomics features extracted from the three types of preprocessed images pass through the second ovarian mass classification module to obtain a second classification result. The two classification results pass through the confidence voting module to obtain the final classification result;

[0009] A lesion area segmentation unit, which is used to construct an exclusive lesion area segmentation model corresponding to each ovarian mass category and perform lesion area segmentation. Each exclusive lesion area segmentation model is used to extract features from the preprocessed ultrasound images and then segment to obtain the lesion area segmentation result.

[0010] Preferably, the cystic-solid part is grouped according to the multi-modal mass image features and the ovarian mass category label is marked, including:

[0011] Group the morphological mass image features according to the cystic-solid part of ovarian cancer, including: when it is cystic-solid tissue, mark the smooth cyst wall as the ovarian mass category label;

[0012] When the cystic-solid component protrudes into the cyst by less than 3 mm, mark the non-smooth cyst wall as the ovarian mass category label;

[0013] When the solid part of the cystic-solid component protruding into the cyst is greater than 3 mm, mark the presence of cystic-solid components as the ovarian mass category label.

[0014] Preferably, the first ovarian mass classification module based on the fused collaborative adaptive Transformer includes a double-branch image segmentation sub-module, a linear projection and position encoding sub-module, a multi-scale Transformer encoder, and a branch fusion sub-module,

[0015] Among them, the double-branch image segmentation sub-module uses a multi-scale strategy to parallelly segment the input ultrasound image into regular image blocks of large / small sizes to respectively capture rich semantic information and fine local details;

[0016] The linear projection and position encoding sub-module maps the image blocks of each branch to the vector space through a linear layer and uses learnable absolute position encoding to retain the spatial information to obtain image features. The multi-scale Transformer encoder is used to perform adaptive perception encoding and collaborative attention encoding on the image features of each branch in the vector space to obtain the encoded features of each branch;

[0017] The branch fusion sub-module is responsible for integrating the encoded features of the two branches and predicting the final first classification result.

[0018] Preferably, the multi-scale Transformer encoder performs adaptive perception encoding and co-attention encoding on the image features of each branch in the vector space to obtain the encoded features of each branch, including:

[0019] The image features of different scales of each branch first undergo adaptive perception encoding through an adaptive Transformer. The large-scale image semantic features and small-scale image detail features obtained are simultaneously in the co-attention sub-module. With the idea of using the class representation of one branch to fuse the features of the other branch to achieve the purpose of exchanging information, co-attention fusion of features and class representations is performed to obtain the encoded features of each branch.

[0020] Preferably, in the co-attention sub-module, with the idea of using the class representation of one branch to fuse the features of the other branch to achieve the purpose of exchanging information, co-attention fusion of features and class representations is performed and the encoded features of each branch are predicted, including:

[0021] When the large-scale branch features fuse the small-scale branch features, the large-scale image class representation is adjusted in dimension through projection mapping and then spliced with the small-scale image detail features of the corresponding branch of the small scale to obtain the first spliced feature. Based on the first spliced feature and the large-scale image class representation after projection mapping, the first co-attention post-feature is constructed through the co-attention mechanism. This first co-attention post-feature combines the large-scale image class representation and the large-scale image semantic features to obtain the encoded features of the large-scale branch;

[0022] When the small-scale branch features fuse the large-scale branch features, the small-scale image class representation is adjusted in dimension through projection mapping and then spliced with the large-scale image semantic features of the corresponding branch of the large scale to obtain the second spliced feature. Based on the second spliced feature and the small-scale image class representation after projection mapping, the second co-attention post-feature is constructed through the co-attention mechanism. This second co-attention post-feature combines the small-scale image class representation and the small-scale image detail features to obtain the encoded features of the small-scale branch.

[0023] Preferably, the second ovarian mass classification module based on radiomics features includes a radiomics feature extraction sub-module and a logistic regression classifier;

[0024] Among them, the radiomics feature extraction sub-module statistically analyzes radiomics features based on each type of preprocessed ultrasound image and its ovarian mass category label. Specifically, it extracts first-order statistical features, morphological features, and second-order and high-order texture features. These features are screened by the variance method, Mann-Whitney U test, and Spearman correlation coefficient to retain the features most relevant to ovarian mass classification as the final radiomics features;

[0025] The logistic regression classifier performs logistic regression prediction based on the radiomics features to obtain the second classification result.

[0026] Preferably, the confidence voting module obtains the final classification result by averaging the first classification result and the second classification result.

[0027] Preferably, the exclusive lesion area segmentation model corresponding to each ovarian mass category uses 2DUnet, and 2DUnet is configured according to the training data characteristics in the corresponding category dataset. Specifically, it includes:

[0028] First, extract the distribution attributes of the training samples in the dataset, including the median image size, voxel spacing distribution, intensity value distribution, number of classes, number of training samples, and image modality, where the image modality is ultrasound image;

[0029] Then, configure the regular parameters and fixed parameters of 2DUnet based on the distribution attributes. The regular parameters include image resampling, intensity normalization, architecture configuration, batch size, and patch size. The fixed parameters include optimizer, learning rate, loss function, data augmentation, training process, and inference process.

[0030] To achieve the above invention purpose, the embodiment also provides a computing device, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to perform ovarian mass segmentation using the above ovarian mass segmentation device, including the following steps:

[0031] Use the data acquisition and preprocessing unit to obtain ovarian cancer ultrasound images and perform ultrasound image preprocessing;

[0032] Use the ovarian mass category recognition model in the ovarian mass category recognition unit to recognize the ovarian mass category of the preprocessed ultrasound image to obtain the final classification result;

[0033] Select the exclusive lesion area segmentation model corresponding to the final classification result from the lesion area segmentation unit and perform segmentation based on the preprocessed ultrasound image to obtain the lesion area segmentation result.

[0034] To achieve the above-mentioned invention object, the embodiment also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the above-mentioned ovarian mass segmentation device is used for ovarian mass segmentation, including the following steps:

[0035] Use the data acquisition and preprocessing unit to acquire ovarian cancer ultrasound images and perform ultrasound image preprocessing;

[0036] Use the ovarian mass category recognition model in the ovarian mass category recognition unit to recognize the category of ovarian masses in the preprocessed ultrasound images to obtain the final classification result;

[0037] Select the exclusive lesion area segmentation model corresponding to the final classification result from the lesion area segmentation unit and perform segmentation based on the preprocessed ultrasound images to obtain the lesion area segmentation result.

[0038] Compared with the prior art, the beneficial effects of the present invention at least include:

[0039] Based on the data acquisition and preprocessing unit, the ovarian mass category recognition unit, and the lesion area segmentation unit, the combined action of these three units can realize the accurate classification of each type of ovarian mass by constructing an ovarian mass category recognition model based on the multi-modal mass image feature grouping in the cystic solid part, combined with the collaborative attention mechanism and the deep learning automatic segmentation algorithm. Then, the exclusive lesion area segmentation model constructed for each type of ovarian mass can more accurately achieve the precise segmentation of each type of ovarian mass. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 is a schematic structural diagram of an ovarian mass segmentation device based on multi-modal mass image feature grouping provided by the embodiment;

[0042] Figure 2 is a flowchart of ovarian mass category marking and preprocessing provided by the embodiment;

[0043] Figure 3 is a schematic structural and process diagram of an ovarian mass category recognition model provided by the embodiment;

[0044] Figure 4 is a schematic structural and process diagram of the first ovarian mass classification module provided by the embodiment;

[0045] Figure 5 It is a schematic diagram of the structure and process of the collaborative attention sub-module provided by the embodiment;

[0046] Figure 6 It is a configuration and training flow chart of the exclusive lesion area segmentation model provided by the embodiment;

[0047] Figure 7 It is a flowchart for ovarian mass segmentation provided by the embodiment. Detailed implementation manners

[0048] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementation manners described herein are only used to explain the present invention and do not limit the protection scope of the present invention.

[0049] The inventive concept of the present invention is as follows: In order to realize the automated segmentation technology for the solid part within an ovarian mass, meet the requirements of artificial intelligence-related technical algorithms to extract more effective mass information, and meet the clinical application needs, the present invention proposes an ovarian mass segmentation device based on multi-modal mass image feature grouping. The basic idea is as follows: Construct an integrated collaborative attention mechanism and deep learning automatic segmentation algorithm, perform automatic classification and joint segmentation of ovarian masses based on different groupings of multi-modal mass image features, and evaluate the model segmentation performance through the Dice coefficient and intersection over union. This automatic classification and joint segmentation highlights the analysis of the presence or absence of solid structures and the heterogeneity characteristics of the solid region in ovarian masses, combines the overall edge detection and segmentation technology of ovarian masses, and can optimize the application of artificial intelligence-driven clinical scenarios from both precise local and global perspectives, such as major issues in tumor diseases such as the differential diagnosis of tumor benign and malignant, pathological typing, heterogeneity assessment, clinical staging, chemotherapy efficacy, survival prognosis, etc., as well as the assessment of ovarian remaining parenchymal function and premature ovarian failure, etc. At the same time, it eliminates the cumbersome work of manual image delineation and reduces the difference in image interpretation due to empirical knowledge.

[0050] As Figure 1 shown, an ovarian mass segmentation device 10 based on multi-modal mass image feature grouping provided by the embodiment includes a data acquisition and preprocessing unit 11, an ovarian mass category recognition unit 12, and a lesion area segmentation unit 13. The combined action of these three units can realize the construction of an ovarian mass category recognition model by combining the collaborative attention mechanism and the deep learning automatic segmentation algorithm on the basis of performing multi-modal mass image feature grouping in the solid part within the cyst, so as to achieve the accurate classification of each type of ovarian mass. Then, the exclusive lesion area segmentation model constructed for each type of ovarian mass can more accurately achieve the precise segmentation of each type of ovarian mass.

[0051] In an embodiment, the data acquisition and preprocessing unit 11 is configured to acquire ovarian cancer ultrasound images, group the image features of multi-shaped masses in the solid-cystic part, label the ovarian mass category labels, and then perform preprocessing on the ultrasound images.

[0052] The obtained ovarian cancer ultrasound images are from multi-section scans of the pelvic region of patients using a transabdominal (3-5 MHz) or transvaginal (5-9 MHz) ultrasound probe, including the maximum longitudinal section and orthogonal transverse section of the ovary, the maximum longitudinal section and orthogonal transverse section of the mass, and showing the ovarian and surrounding tissue structures from multiple angles. During the prospective image acquisition scan, the gain, depth, and focus settings are adjusted to optimize the image quality, and at the same time, two-dimensional gray-scale ultrasound images, color Doppler flow imaging, and spectral Doppler data are recorded. All image data are saved in DICOM format to ensure that the image resolution and dynamic parameters meet the diagnostic requirements. During the acquisition process, two experienced ultrasound physicians independently operate and label the lesion areas, including key features such as solid-cystic components. The acquired image data are strictly paired with the pathological diagnosis results to construct an ovarian cancer ultrasound image database, which is used for subsequent model training.

[0053] In an embodiment, the ovarian mass category is classified according to the morphological features of the solid-cystic part in the ultrasound image, that is, the image features of multi-shaped masses in the solid-cystic part are grouped and the ovarian mass category labels are labeled. Specifically, when it is a solid-cystic tissue, the smooth inner wall of the cyst is marked as the ovarian mass category label; when the solid-cystic component protrudes into the cyst by less than 3 mm, the non-smooth inner wall of the cyst is marked as the ovarian mass category label; when the solid part of the solid-cystic component protruding into the cyst is greater than 3 mm, the presence of solid-cystic components in the cyst is marked as the ovarian mass category label.

[0054] In an embodiment, as Figure 2 shown, preprocessing is also performed on the labeled ultrasound images, including image standardization, unified resolution, and image quality optimization. The specific process is as follows: The two-dimensional gray-scale ultrasound image is standardized. First, the pixel values of the image are normalized, specifically to the range of [0, 1] to eliminate the gray-scale differences between devices. Then, the image resolution is uniformly adjusted by bilinear interpolation, specifically to 512×512 pixels, to ensure data spatial consistency. The histogram equalization method is used to enhance the image contrast and highlight the texture and edge features of the lesion area. To further improve the image quality, the Gaussian filtering algorithm is used to smooth the image, effectively suppressing noise while retaining key lesion information. For color Doppler images, first, the blood flow signal is extracted based on the HSV color space, and the red (blood flow towards the probe) and blue (blood flow away from the probe) regions are separated by hue threshold and converted into grayscale images. The subsequent preprocessing steps are the same as those for two-dimensional gray-scale ultrasound images to achieve high-quality preprocessing of multi-modal ultrasound images.

[0055] In the embodiment, the ovarian mass category recognition unit 12 is used to construct an ovarian mass category recognition model and perform ovarian mass category recognition. As Figure 3 shown, the ovarian mass category recognition model includes a first ovarian mass classification module based on the fused collaborative adaptive Transformer, a second ovarian mass classification module based on radiomics features, and a confidence voting module. The three types of preprocessed ultrasound images pass through the first ovarian mass classification module to obtain a first classification result. The radiomics features extracted from the three types of preprocessed images pass through the second ovarian mass classification module to obtain a second classification result. The two classification results pass through the confidence voting module to obtain the final classification result.

[0056] As Figure 4 shown, the first ovarian mass classification module based on the fused collaborative adaptive Transformer includes a dual-branch image segmentation sub-module, a linear projection and position encoding sub-module, a multi-scale Transformer encoder, and a branch fusion sub-module. Among them, the dual-branch image segmentation sub-module uses a multi-scale strategy to parallelly segment the input ultrasound image into regularly sized large / small image patches to capture rich semantic information and fine local details respectively; the linear projection and position encoding sub-module maps the image patches of each branch to the vector space through a linear layer and uses learnable absolute position encoding to retain the spatial information to obtain image features; the multi-scale Transformer encoder is used to perform adaptive perception encoding and collaborative attention encoding on the image features of each branch in the vector space to obtain the encoded features of each branch; the branch fusion sub-module is responsible for integrating the encoded features of the dual-branch and predicting to obtain the final first classification result.

[0057] As Figure 4 shown, the multi-scale Transformer encoder includes a dual-branch adaptive Transformer and a collaborative attention sub-module. Each branch is optimized based on the basic vision Transformer: an adaptive enhancement sub-module composed of a downsampling layer, a ReLU activation function, and an upsampling layer is inserted after the second normalization. Among them, the basic vision Transformer part is initialized with ImageNet pre-trained parameters and kept frozen, and only the adaptive branch parameters are optimized; the proportion of the original feature and the adapted feature is adjusted by the scale coefficient α. In a specific implementation case, the scale coefficient is 0.8.

[0058] The image features of different scales of each branch first pass through the adaptive Transformer for adaptive perception encoding, and the obtained large-scale image semantic features and small-scale image detail features are simultaneously in the collaborative attention sub-module, as Figure 5As shown, with the idea of using the category representation (category token) of one branch to fuse the features of another branch to achieve the purpose of information exchange, co-attention fusion of features and category representations is performed to obtain the encoded features of each branch.

[0059] Specifically, when the large-scale branch features are fused with the small-scale branch features, first, the large-scale image category representation is adjusted in dimension through projection mapping and then concatenated with the small-scale image detail features of the corresponding branch of the small scale to obtain the first concatenated feature :

[0060] ;

[0061] Among them, represents the concatenation operation of directly expanding the dimension without changing the original data; represents the large-scale branch; represents the small-scale branch: is the projection function, and its purpose is to convert the dimension of the large-scale image into the dimension of the small-scale image; represents the image category Token of the large-scale branch; represents the image detail features of the small-scale branch;

[0062] Secondly, based on the first concatenated feature and the large-scale image category representation after projection mapping, a first co-attention post-feature is constructed through a co-attention mechanism, including: As the only query, in and the co-attention mechanism is executed, that is, the first key vector k and the first value vector v in the co-attention mechanism are constructed based on the first concatenated feature, and the first query vector is constructed based on the large-scale image category representation after projection mapping q :

[0063] , , ;

[0064] Among them, , , respectively represent , and corresponding weights;

[0065] After calculating the first co-attention weight based on the first query vector q and the first key vector k, the first co-attention post-feature is calculated based on the first co-attention weight and the first value vector v :

[0066] , ;

[0067] Among them, represents the normalized exponential function; the superscript represents transpose; and represent the large image patch size and the number of attention heads respectively; represents co-attention;

[0068] Finally, the feature after the first co-attention combines the large-size image class representation and the large-size image semantic feature to obtain the encoded feature of the large-size branch, including: the feature after the first co-attention and the projected large-size image class representation are added and then back-projected. The result of the first back-projection is concatenated with the large-size image semantic feature of the corresponding branch of the large scale to obtain the encoded feature of the large-size branch :

[0069] ;

[0070] ;

[0071] Among them, represents the feature after addition, represents the multi-head co-attention mechanism; represents layer normalization, that is, the first concatenated feature is layer-normalized and then multi-head co-attention is performed to obtain the feature after the first co-attention, is the back-projection function, and its purpose is to convert the dimension of the small-size image to the dimension of the large-size image.

[0072] When fusing the small-size branch feature with the large-size branch feature, the same logic as that for fusing the large-size branch feature with the small-size branch feature above is adopted. First, the small-size image class representation is adjusted in dimension through projection mapping and then concatenated with the large-size image semantic feature of the corresponding branch of the large scale to obtain the second concatenated feature:

[0073] ;

[0074] Among them, is the projection function, and its purpose is to convert the dimension of the small-size image to the dimension of the large-size image; represents the image class Token of the small-size branch; represents the image meaning feature of the large-size branch;

[0075] Secondly, based on the second splicing feature and the class representation of the small-sized image after projection mapping, a second co-attention post-feature is constructed through a co-attention mechanism, including: As the only query, in and Execute the co-attention mechanism to construct the second key vector in the co-attention mechanism based on the second splicing feature and the second value vector , and construct the second query vector based on the class representation of the small-sized image after projection mapping : :

[0076] , , ;

[0077] Among them, , , respectively represent , and corresponding weights;

[0078] Based on the second query vector and the second key vector , after calculating the second co-attention weight, calculate the second co-attention post-feature based on the second co-attention weight and the second value vector : :

[0079] , ;

[0080] Among them, represents the small image patch size;

[0081] Finally, the second co-attention post-feature is combined with the class representation of the small-sized image and the detailed features of the small-sized image to obtain the encoded feature of the small-sized branch, including: this second co-attention post-feature and the class representation of the small-sized image after projection mapping are added and then subjected to inverse projection mapping, and the second inverse projection mapping result is spliced with the detailed features of the small-sized image of the corresponding branch of the small scale to obtain the encoded feature of the small-sized branch :

[0082] ;

[0083] ;

[0084] Among them, is the inverse projection function, and the purpose is to convert the dimension of the large-sized image into the dimension of the small-sized image.

[0085] The multi-scale Transformer encoder includes a dual-branch adaptive Transformer and a co-attention sub-module, which significantly reduces the training parameters of the model while maintaining the powerful representation ability of the pre-trained model; the co-attention mechanism is used to perform efficient interaction between input sequences of different scales, so as to better utilize multi-scale features.

[0086] As Figure 3 shown, the second ovarian mass classification module based on radiomics features includes a radiomics feature extraction sub-module and a logistic regression classifier;

[0087] Among them, the radiomics feature extraction sub-module statistically analyzes radiomics features based on each type of preprocessed ultrasound image and its ovarian mass category label, specifically extracting first-order statistical features, morphological features, and second-order and high-order texture features. These features are screened by variance method, Mann-Whitney U test, and Spearman correlation coefficient to retain the features most relevant to ovarian mass classification as the final radiomics features; the logistic regression classifier performs logistic regression prediction based on the radiomics features to obtain the second classification result.

[0088] In the embodiment, the confidence voting module integrates the two classification results from the outputs of the first ovarian mass classification module and the second ovarian mass classification module, sums the classification probability distributions predicted by the two modules by category and calculates the average value, so as to obtain the final classification confidence, that is, the final classification result. The calculation method is as follows:

[0089] ;

[0090] Among them, represents the probability of the irregular cyst inner wall calculated by the confidence voting module; represents the probability of the irregular cyst inner wall of the first ovarian mass classification module based on Transformer; represents the probability of the irregular cyst inner wall of the second ovarian mass classification module based on radiomics features; the calculation method of the regular cyst inner wall probability is the same. By comparing the calculated category probabilities, the category with the larger probability value is the final classification result of the ultrasound image.

[0091] In the embodiment, the lesion area segmentation unit 13 is used to construct an exclusive lesion area segmentation model corresponding to each ovarian mass category and perform lesion area segmentation, where each exclusive lesion area segmentation model is used to extract features from the preprocessed ultrasound image and then segment to obtain the lesion area segmentation result.

[0092] Specifically, the exclusive lesion area segmentation model corresponding to each ovarian mass category uses 2DUnet, and configures 2DUnet according to the training data characteristics in the corresponding category dataset, such as Figure 6 shown, specifically including:

[0093] First, extract the distribution attributes of the training samples in the dataset, including the median of the image size, the voxel spacing distribution, the intensity value distribution, the number of categories, the number of training samples, and the image modality. The intensity value distribution includes statistical information such as the mean, standard deviation, and median of the foreground region intensity values; the number of categories is the number of categories to be segmented; the image modality is the ultrasound image.

[0094] Then, configure the regular parameters and fixed parameters of 2DUnet based on the distribution attributes. The regular parameters include image resampling, intensity normalization, architecture configuration, batch size, and patch size. The image resampling strategy refers to resampling the original image to a fixed resolution that can be directly processed by the model, and its parameters depend on the voxel spacing distribution; intensity normalization is to perform z-score normalization independently on each ultrasound image; the architecture configuration, batch size, and patch size depend on the memory limit of the GPU, and nnU-Net will optimize this trade-off.

[0095] The fixed parameters include the optimizer, learning rate, loss function, data augmentation, training process, and inference process. In a specific implementation case, the optimizer is the stochastic gradient descent optimizer; the learning rate is set to 0.01; the loss function uses both Dice and cross-entropy loss functions; the data augmentation method is the same as the general data augmentation method in deep learning, including rotation, cropping, etc.; the training process data is 2256, and the inference process data is 625.

[0096] Based on the above configuration, train 2DUnet and perform five-fold cross-validation, and also use the Dice coefficient to evaluate the exclusive lesion area segmentation model.

[0097] Take the differential diagnosis of ovarian borderline tumors as an example:

[0098] The automatic segmentation accuracy of solid regions of interest is comparable to that of experienced ultrasound doctors: In the actual clinical work scenario, affected by factors such as experience level, visual limitations, visual fatigue, and overlapping of lesion structures in ultrasound images, ultrasound doctors cannot always accurately detect and describe image abnormalities, resulting in inter-observer inconsistency, which is more significant in ultrasound features related to the differentiation of solid lesions other than size and volume measurement. Based on this, gray-scale and color Doppler ultrasound images of ovarian benign tumors were retrospectively collected respectively. For the training set, experienced ultrasound doctors outlined the solid lesion regions, which were used as the true regions for subsequent calculation of the Dice index. For the test set, the solid regions were not outlined, and the device proposed in the present invention was used to train the segmentation model for ovarian tumor lesion regions, and the automatic segmentation test results of the solid lesion regions were output. The predicted regions were compared with the true regions to evaluate the automatic segmentation performance of the model. As shown in Table 1, the Dice value index reflecting the automatic segmentation performance of the model reached 0.892 and 0.890 in the validation set and the test set respectively, indicating that the ovarian mass segmentation device based on the multi-modal mass image feature grouping in the present invention can achieve a relatively superior automatic segmentation effect on ultrasound images of ovarian benign tumors. The manual outlining of the solid regions was performed by ultrasound doctors with more than 5 years of work experience. When it was difficult to clarify the manual outlining boundary, it was determined after discussion with ultrasound doctors with more than 10 years of experience.

[0099] Table 1

[0100]

[0101] The present invention realizes the automation of extracting blood flow texture information of solid lesions in color images and improves the differential diagnosis ability of borderline ovarian tumors: At present, the training of artificial intelligence-driven differential diagnosis models is mostly based on the overall edge detection and segmentation of ultrasound images, without considering the importance of accurate local segmentation of lesions in the parenchymal regions of ovarian masses for the diagnostic performance of the models. The International Ovarian Tumor Analysis (IOTA) expert consensus in ultrasound clearly points out the importance of blood flow signals for the evaluation of ultrasound diagnostic results. However, its evaluation of ultrasound blood flow characteristics is mainly based on the subjective evaluation of doctors, which is limited by the experience level of ultrasound doctors. Based on this, the present invention realizes the automatic extraction of IOTA blood flow signal indicators on color Doppler ultrasound images by introducing the new method for automatic segmentation of solid lesion regions proposed in the present invention and automatically segmenting the solid regions of color Doppler ultrasound images.

[0102] In summary, the present invention has multiple technological innovations and application innovations: 1) Technologically, this method first identifies the automatic classification of smooth cyst inner walls, non-smooth cyst inner walls, and cystic solid components in ovarian masses. Based on the classification of lesions in the solid region, the nnU-Net is used to train a deep learning-based lesion region segmentation model respectively to achieve stable segmentation accuracy and robustness. 2) In terms of application, by comparing with the segmentation results of professional ultrasound physicians (with more than 5 years of work experience), the accuracy of the automatic segmentation algorithm is verified, and the Dice value of automatic segmentation in grayscale ultrasound images can reach 0.890. Further, it is applied to the segmentation of the solid lesion region in color Doppler ultrasound images to realize the automation of extracting blood flow texture information of solid lesions in color images by radiomics, demonstrating that the segmentation of the region of interest in the solid part of the mass improves the differential diagnosis value in borderline ovarian tumors ( P = 0.005).

[0103] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the ovarian mass segmentation using the above-mentioned ovarian mass segmentation device, as Figure 7 shown, including the following steps:

[0104] S1, using the data acquisition and preprocessing unit to acquire ovarian cancer ultrasound images and perform ultrasound image preprocessing;

[0105] S2, using the ovarian mass category recognition model in the ovarian mass category recognition unit to recognize the ovarian mass category of the preprocessed ultrasound image to obtain the final classification result;

[0106] S3, selecting the exclusive lesion region segmentation model corresponding to the final classification result from the lesion region segmentation unit to perform segmentation on the preprocessed ultrasound image to obtain the lesion region segmentation result.

[0107] For the computing device provided by the embodiment, at the hardware level, in addition to including a processor and a memory, it also includes other hardware required for other services such as an internal bus, a network interface, and a memory. The memory is a non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the ovarian mass segmentation steps described in S1-S3 above. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or a logical device.

[0108] Based on the same inventive concept, the embodiment also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it realizes ovarian mass segmentation by using the above-mentioned ovarian mass segmentation device, as Figure 7 shown, including the following steps:

[0109] S1, obtaining an ovarian cancer ultrasound image by using the data acquisition and preprocessing unit and performing ultrasound image preprocessing;

[0110] S2, using the ovarian mass category recognition model in the ovarian mass category recognition unit to recognize the category of ovarian masses in the preprocessed ultrasound image to obtain the final classification result;

[0111] S3, selecting the exclusive lesion area segmentation model corresponding to the final classification result from the lesion area segmentation unit and performing segmentation based on the preprocessed ultrasound image to obtain the lesion area segmentation result.

[0112] In the embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be realized by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data.

[0113] The above-mentioned specific embodiments have elaborated on the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, supplements, equivalent replacements, etc. made within the principle scope of the present invention should be included in the protection scope of the present invention.

Claims

1. An ovarian mass segmentation device based on grouping of multi-modal mass image features, characterized in that Including: A data acquisition and preprocessing unit, which is used to acquire ovarian cancer ultrasound images, group the image features of multi-morphology masses in the cystic-solid part, mark the ovarian mass category labels, and then perform ultrasound image preprocessing; An ovarian mass category recognition unit, which is used to construct an ovarian mass category recognition model and perform ovarian mass category recognition. The ovarian mass category recognition model includes a first ovarian mass classification module based on a fused collaborative adaptive Transformer, a second ovarian mass classification module based on radiomics features, and a confidence voting module. The three types of preprocessed ultrasound images pass through the first ovarian mass classification module to obtain a first classification result, and the radiomics features extracted from the three types of preprocessed images pass through the second ovarian mass classification module to obtain a second classification result. The two classification results pass through the confidence voting module to obtain the final classification result; Among them, the first ovarian mass classification module based on the fused collaborative adaptive Transformer includes a dual-branch image segmentation sub-module, a linear projection and position encoding sub-module, a multi-scale Transformer encoder, and a branch fusion sub-module. The dual-branch image segmentation sub-module uses a multi-scale strategy to parallelly segment the input ultrasound image into regularly sized large / small image patches to capture rich semantic information and fine local details respectively; the linear projection and position encoding sub-module maps the image patches of each branch to a vector space through a linear layer and uses learnable absolute position encoding to retain spatial information to obtain image features; the multi-scale Transformer encoder is used to perform adaptive perception encoding and collaborative attention encoding on the image features of each branch in the vector space to obtain the encoded features of each branch. Specifically, the image features of different scales of each branch first pass through an adaptive Transformer for adaptive perception encoding, and the large-scale image semantic features and small-scale image detail features obtained are simultaneously in the collaborative attention sub-module. With the idea of using the category representation of one branch to fuse the features of the other branch to achieve the purpose of information exchange, the collaborative attention fusion of features and category representation is performed to obtain the encoded features of each branch; the branch fusion sub-module is responsible for integrating the encoded features of the dual-branch and predicting to obtain the final first classification result; A lesion area segmentation unit, which is used to construct an exclusive lesion area segmentation model corresponding to each ovarian mass category and perform lesion area segmentation. Among them, the exclusive lesion area segmentation model corresponding to the final classification result selected from the lesion area segmentation unit is used to extract features from the preprocessed ultrasound image and then segment to obtain the lesion area segmentation result.

2. The ovarian mass segmentation device based on the grouping of multi-modal mass image features according to claim 1, wherein Grouping the image features of multi-morphology masses in the cystic-solid part and marking the ovarian mass category labels, including: Grouping the image features of morphological masses according to the cystic-solid part of ovarian cancer, including: when it is cystic-solid tissue, marking the smooth inner wall of the cyst as the ovarian mass category label; When the cystic-solid component protrudes into the cyst by less than 3 mm, marking the uneven inner wall of the cyst as the ovarian mass category label; When the solid part of the inner solid component protruding into the cyst is greater than 3 mm, the cyst containing the solid component inside is marked as the ovarian mass category label.

3. The ovarian mass segmentation device based on grouping of multi-modal mass image features according to claim 1, characterized in that, In the co-attention sub-module, with the idea of using the category representation of one branch to fuse the features of the other branch to achieve the purpose of exchanging information, co-attention fusion of features and category representations is carried out, and the encoded features of each branch are predicted, including: When the large-size branch features are fused with the small-size branch features, the category representation of the large-size image is projected and mapped to adjust the dimension and then spliced with the small-size image detail features of the corresponding branch of the small scale to obtain the first spliced feature. Based on the first spliced feature and the category representation of the large-size image after projection mapping, the first co-attention post-feature is constructed through the co-attention mechanism. This first co-attention post-feature combines the category representation of the large-size image and the semantic features of the large-size image to obtain the encoded features of the large-size branch; When the small-size branch features are fused with the large-size branch features, the category representation of the small-size image is projected and mapped to adjust the dimension and then spliced with the semantic features of the large-size image of the corresponding branch of the large scale to obtain the second spliced feature. Based on the second spliced feature and the category representation of the small-size image after projection mapping, the second co-attention post-feature is constructed through the co-attention mechanism. This second co-attention post-feature combines the category representation of the small-size image and the detail features of the small-size image to obtain the encoded features of the small-size branch.

4. The ovarian mass segmentation device based on multi-modal mass image feature grouping according to claim 1, characterized in that, The second ovarian mass classification module based on radiomics features includes a radiomics feature extraction sub-module and a logistic regression classifier; Among them, the radiomics feature extraction sub-module statistically analyzes radiomics features based on each type of preprocessed ultrasound image and its ovarian mass category label, specifically extracting first-order statistical features, morphological features, and second-order and high-order texture features. These features are screened through the variance method, Mann-Whitney U test, and Spearman correlation coefficient to retain the features most relevant to ovarian mass classification as the final radiomics features; The logistic regression classifier performs logistic regression prediction based on the radiomics features to obtain the second classification result.

5. The ovarian mass segmentation device based on multi-modal mass image feature grouping according to claim 1, characterized in that, The confidence voting module uses the method of taking the average to average the first classification result and the second classification result to obtain the final classification result.

6. The ovarian mass segmentation device based on multi-modal mass image feature grouping according to claim 1, wherein, The exclusive lesion area segmentation model corresponding to each ovarian mass category uses 2DUnet and configures 2DUnet according to the training data characteristics in the corresponding category dataset, specifically including: First, extract the distribution attributes of the training samples in the dataset, including the median of the image size, voxel spacing distribution, intensity value distribution, number of categories, number of training samples, and image modality, where the image modality is an ultrasound image; Then, configure the regular parameters and fixed parameters of 2DUnet based on the distribution attributes. The regular parameters include image resampling, intensity normalization, architecture configuration, batch size, and patch size, and the fixed parameters include optimizer, learning rate, loss function, data augmentation, training process, and inference process.

7. A computing device includes a memory and one or more processors, and executable code is stored in the memory, characterized in that, When the one or more processors execute the executable code, they are used to perform ovarian mass segmentation using the ovarian mass segmentation device according to any one of claims 1-6, including the following steps: The data acquisition and preprocessing unit is used to acquire ovarian cancer ultrasound images and perform preprocessing on the ultrasound images; The ovarian mass category recognition model in the ovarian mass category recognition unit is used to recognize the category of ovarian masses in the preprocessed ultrasound images to obtain the final classification result; The exclusive lesion area segmentation model corresponding to the final classification result is selected from the lesion area segmentation unit to perform segmentation on the preprocessed ultrasound images to obtain the lesion area segmentation result.

8. A computer-readable storage medium, characterized in that, A program is stored thereon, and when the program is executed by a processor, it realizes ovarian mass segmentation by using the ovarian mass segmentation device according to any one of claims 1-6, including the following steps: The data acquisition and preprocessing unit is used to acquire ovarian cancer ultrasound images and perform preprocessing on the ultrasound images; The ovarian mass category recognition model in the ovarian mass category recognition unit is used to recognize the category of ovarian masses in the preprocessed ultrasound images to obtain the final classification result; The exclusive lesion area segmentation model corresponding to the final classification result is selected from the lesion area segmentation unit to perform segmentation on the preprocessed ultrasound images to obtain the lesion area segmentation result.

Images