Ovarian adnexal tumor risk grading system based on multimodal deep learning

Abstract

The present invention relates to the technical field of tumor risk assessment and discloses an ovarian adnexal tumor risk grading system based on multimodal deep learning. The system comprises: a multimodal data acquisition module for acquiring ultrasound imaging data, CA125 serum concentration data, and gold standard data of patients with ovarian adnexal tumors; an ultrasound image segmentation module for automatically locating and structurally segmenting ultrasound image lesion areas using adaptive bidirectional displacement; a multimodal feature extraction module for extracting O-RADS vectors and CEUS vectors based on lesion area masks from segmented images, and for performing numerical normalization processing on CA125 serum concentration data to obtain standardized feature vectors; a multimodal feature fusion module for performing feature fusion to obtain OCC-US vector representations; and a risk grading decision module for classifying the OCC-US vector representations and outputting OCC-US scores and benign and malignant risk levels. The present invention improves the diagnostic accuracy and reliability of ovarian adnexal tumors.

Description

Technical Field

[0001] The present invention relates to the technical field of tumor risk assessment, and in particular to an ovarian adnexal tumor risk grading system based on multimodal deep learning. Background Art

[0002] Early diagnosis of ovarian adnexal tumors is crucial for patient prognosis and treatment. However, current diagnostic methods, particularly those based on the combined use of O-RADS (Ovarian-Adnexal Reporting and Data System), CEUS (Contrast-Enhanced Ultrasound), and CA125 (a serum tumor marker), still face several challenges. These include false positives due to low specificity, misdiagnosis due to blurred lesion boundaries, and inconsistency between modalities. Traditional diagnostic methods often struggle to provide adequate differentiation, especially in situations of poor image quality, atypical lesion presentation, and high volatility in biochemical markers.

[0003] Therefore, there is an urgent need for a new intelligent diagnostic system that can effectively integrate multi-source heterogeneous medical data and improve the diagnostic accuracy and reliability of ovarian adnexal tumors through adaptive learning and highly robust segmentation strategies. This requires not only the precise identification of lesion areas, but also the ability to process and integrate different types of medical information to accurately stratify and predict tumor risk. Summary of the Invention

[0004] The main purpose of this invention is to address the limitations of existing ovarian adnexal tumor diagnosis methods and propose a risk grading system based on multimodal deep learning. By fusing multi-source heterogeneous medical data and using adaptive learning and highly robust segmentation strategies, the diagnostic accuracy and reliability of ovarian adnexal tumors are improved, and accurate grading and prediction of tumor risk are achieved.

[0005] To achieve the above objectives, the following technical solutions are adopted:

[0006] A multimodal deep learning-based risk grading system for ovarian adnexal tumors, including:

[0007] Multimodal data acquisition module, used to obtain ultrasound imaging data, CA125 serum concentration data and gold standard data of patients with ovarian adnexal tumors;

[0008] Ultrasound image segmentation module, which is used to automatically locate and structure the ultrasound image lesion area based on the pre-processed ultrasound image data using adaptive bidirectional displacement, and output the lesion area mask of the ovarian adnexal tumor in each image;

[0009] a multimodal feature extraction module for extracting an O-RADS feature vector of the B-ultrasound image and a dynamic feature vector of the CEUS image based on the lesion area mask of the ovarian adnexal tumor in each image, and performing numerical normalization processing on the CA125 serum concentration data to obtain a standardized feature vector;

[0010] a multimodal feature fusion module for performing feature fusion on the O-RADS feature vector of the B-ultrasound image, the dynamic feature vector of the CEUS image, and the normalized feature vector using a dual-branch fusion strategy that fuses self-attention and cross-attention mechanisms to obtain an OCC-US vector representation;

[0011] The risk grading decision module is used to input the fused OCC-US vector representation into the classifier, output the OCC-US score and benign and malignant risk level, and use reinforcement learning to dynamically optimize the scoring decision.

[0012] Furthermore, the ultrasound image segmentation module includes:

[0013] A self-supervised guidance unit (ABD-R) is used to generate weakly enhanced and strongly enhanced samples for unlabeled ultrasound images. Through patch-level confidence evaluation and semantic matching, low-confidence regions are replaced with semantically similar high-confidence regions to generate enhanced training image samples.

[0014] The difficult sample enhancement unit (ABD-I) is used to perform an inverse confidence shift strategy on the annotated ultrasound image, replacing the high-confidence region with the low-confidence region in another enhanced view to construct a difficult version of the training image sample;

[0015] The segmentation map generation unit is used to synchronously train two segmentation models with different structures using the enhanced version training image samples, the difficult version training image samples, the original annotated ultrasound images and their corresponding labels, use the two trained segmentation models to infer the newly input ultrasound images, and output the lesion area mask of the ovarian adnexal tumor in each image.

[0016] Furthermore, the self-supervisory guidance unit (ABD-R) performs the following operations:

[0017] Divide the unlabeled ultrasound image into multiple patches and calculate the average confidence and category distribution of each patch;

[0018] Filter the patch with the lowest confidence as the area to be replaced, and select the replacement source from the semantically similar high-confidence patches;

[0019] New samples are generated via bidirectional displacement: low-confidence patches of weakly enhanced samples are replaced with matching high-confidence patches of strongly enhanced samples, and vice versa.

[0020] Furthermore, the difficult sample enhancement unit (ABD-I) performs the following operations:

[0021] Locate the patch with the highest confidence in the annotated ultrasound image;

[0022] Replace it with the patch with the lowest confidence in another enhanced version of the same annotated sample, and adjust the label mask simultaneously;

[0023] Generate synthetic samples that force the model to learn complex lesion edges, i.e., difficult versions of training image samples.

[0024] Furthermore, the two segmentation models with different structures trained by the segmentation map generation unit are the UNet model and the SwinUNet model; the UNet model and the SwinUNet model respectively generate a prediction map A and a prediction map B for the new ultrasound image; the average prediction map C of the prediction map A and the prediction map B is calculated; the threshold segmentation is binarized according to the average prediction map C to generate the lesion area and the background area, and then after post-processing, a binary segmentation mask is generated.

[0025] Furthermore, the multimodal feature fusion module adopts a dual-branch fusion strategy that integrates self-attention and cross-attention mechanisms, including:

[0026] The self-attention (SA) branch is used to model the spatial and temporal correlation of unimodal internal features through the self-attention mechanism to enhance the ability to perceive lesion structures;

[0027] The cross-attention (CA) branch is used to focus on cross-modal low-correlation regions through the inverse attention mechanism and extract complementary features.

[0028] Furthermore, the operations performed by the self-attention (SA) branch include:

[0029] Decompose the unimodal feature tensor into query vector, key vector and value vector through a learnable linear transformation;

[0030] Calculate the similarity matrix between the query vector and the key vector, and generate the attention weight after scaling and normalization;

[0031] Use attention weights to perform weighted fusion on the value vectors to obtain enhanced unimodal features;

[0032] The high-order feature associations are further extracted through the multi-layer perceptron to strengthen the structural consistency within the single modality.

[0033] Furthermore, the operations performed by the crisscross attention (CA) branch include:

[0034] The first modality feature is used as the query vector, and the second modality feature is used as the key vector and value vector;

[0035] Calculate the cross-modal similarity matrix and use the inverse softmax operation to generate complementary attention weights;

[0036] Reweighting the second modality features by the complementary attention weights to focus on cross-modal low correlation regions;

[0037] Output cross-modal features that enhance difference information to resolve modal conflicts.

[0038] Furthermore, the risk classification decision module combines the multimodal fusion feature vector As input, the system trains a deep classification model to perform multi-level classification of tumor types and outputs a prediction of the risk level of benign or malignant tumors: benign, suspicious, or malignant.

[0039] When the benign or malignant risk level prediction result is benign, follow-up is recommended; when the benign or malignant risk level prediction result is suspicious, enhanced reexamination is recommended; when the benign or malignant risk level prediction result is malignant, surgical intervention is recommended.

[0040] Furthermore, the SARSA(λ) algorithm is used to dynamically optimize the scoring decision by using reinforcement learning, including:

[0041] The OCC-US vector is used as the state, the risk level prediction is used as the action, and the consistency of the pathological result is used as the reward signal;

[0042] Dynamically update the state-action value function through temporal difference error;

[0043] The qualification trace is used to trace the historical decision weights and dynamically adjust the parameters of the classifier in the risk grading decision module.

[0044] Compared with the prior art, the present invention achieves the following beneficial effects:

[0045] 1. This paper proposes an adaptive bidirectional displacement (ABD) mechanism, combined with a confidence-guided strategy, to achieve highly robust lesion region segmentation under semi-supervision. This mechanism, combining two modules: self-supervised guidance based on unlabeled ultrasound images (ABD-R) and hard-to-sample enhancement based on labeled ultrasound images (ABD-I), improves the segmentation accuracy of ovarian adnexal tumors, particularly in cases of poor image quality or blurred boundaries. By introducing annotation information for uncertain regions, this mechanism enables the model to effectively learn even on weakly labeled data, improving segmentation stability and accuracy.

[0046] 2. This paper proposes annotated ultrasound image-based hard-sample enhancement (ABD-I), which employs an "inverse confidence shift" strategy to force the model to learn regions that would otherwise be overlooked. Traditional segmentation methods tend to overlook tumor regions with blurred boundaries or weak features, particularly the complex, mixed masses found in ovarian adnexal tumors. Using the inverse confidence shift strategy, the model is forced to learn these "hard samples," thereby improving its ability to recognize complex tumors.

[0047] 3. This paper proposes a dual-branch fusion strategy that enhances both unimodal and cross-modal features based on multimodal data. The SA (Self-Attention) branch enhances the structural awareness of unimodal internal features, enabling the model to capture both local and global context within the same modality, improving lesion recognition accuracy. The CA (Cross-Attention) branch captures intermodal complementarity, specifically focusing on the differential interpretation of the same lesion by different modalities, such as the synergistic relationship between CEUS and O-RADS. This fusion effectively addresses conflicts between modalities and enhances the model's diagnostic capabilities.

[0048] 4. This paper proposes an inverse attention mechanism (re-softmax), which applies an inverse softmax strategy to cross-modal fusion, focusing on low-correlation regions, thereby enhancing the model's ability to recognize atypical lesions (such as multilocular cysts with unclear structures). This mechanism effectively suppresses redundant information between modalities and emphasizes complementary features that are overlooked by traditional methods.

[0049] 5. This paper proposes the SARSA(λ) algorithm, which simulates the "diagnosis-decision-feedback" process to enable the scoring system to adaptively correct for the risk of misdiagnosis. This algorithm, optimized on the traditional reinforcement learning framework, dynamically adjusts the scoring model's decision-making process, reducing the misdiagnosis rate and enhancing the long-term adaptability of the diagnostic system. By simulating the repeated feedback within the clinical pathway, the model can automatically correct errors in the early stages and improve long-term diagnostic accuracy.

[0050] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for a better understanding of the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0052] Figure 11 is a schematic diagram of a module of an ovarian adnexal tumor risk grading system based on multimodal deep learning according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the architecture of an ovarian adnexal tumor risk grading system based on multimodal deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0056] Figure 1 1 is a schematic diagram of a module of an ovarian adnexal tumor risk grading system based on multimodal deep learning according to an embodiment of the present invention; Figure 2 FIG. 1 is a schematic diagram of the architecture of an ovarian adnexal tumor risk grading system based on multimodal deep learning according to an embodiment of the present invention. Figure 1 and Figure 2 As shown, a multimodal deep learning-based ovarian adnexal tumor risk grading system 100 includes:

[0057] Multimodal data acquisition module 110: used to obtain ultrasound imaging data, CA125 serum concentration data and gold standard data of patients with ovarian adnexal tumors;

[0058] The multimodal data acquisition module 110 is used to implement a multimodal data acquisition and annotation system.

[0059] (1) Multimodal data acquisition

[0060] Data from patients with ovarian adnexal masses diagnosed pathologically or followed up for a long time were retrospectively collected (609 samples, randomly divided into training set, validation set, and independent test set in a ratio of 7:1.5:1.5), including:

[0061] Ultrasound imaging data: transvaginal / abdominal ultrasound B-mode images and CEUS angiography dynamic sequences (including arterial and portal venous phase images) were acquired using devices such as Samsung RS80A and GE LOGIQ E10 and stored in the PACS system.

[0062] Laboratory data: CA125 serum concentration measured 1-14 days before surgery (threshold ≥37.815 U / ml as an indicator of malignancy).

[0063] Gold standard data: surgical pathology results (such as serous cystadenocarcinoma, mature teratoma, etc.) and / or follow-up records.

[0064] (2) Multimodal data preprocessing

[0065] Image standardization: The ultrasound images were grayscale normalized and the resolution was unified (e.g., 512 × 512 pixels). The key frames of the peak enhancement phase (e.g., 20 s arterial phase, 42 s portal venous phase) of the CEUS sequence were extracted.

[0066] Feature annotation: Experienced ultrasound physicians will annotate the morphological features of the lesions (such as cyst wall thickness and papillary protrusions) and CEUS scores (≥4 points is defined as malignant vascular features) according to the ACR O-RADS 2022 version, and simultaneously annotate the CA125 value.

[0067] The ultrasound image segmentation module 120 uses adaptive bidirectional displacement to automatically locate and structure the ultrasound image lesion area based on the pre-processed ultrasound image data, and outputs a lesion area mask of the ovarian adnexal tumor in each image;

[0068] The ultrasound image segmentation module 120 is used to implement ultrasound image segmentation. By replacing model-uncontrollable regions (low-confidence patches) in an image with reliable regions (high-confidence, semantically similar patches) from another perturbed image, more consistent training samples are generated. ABD-R leverages unlabeled data to enhance generalization, while ABD-I uses inverse confidence to mine difficult examples in labeled data, creating a complementary approach.

[0069] In the multimodal deep learning-based ovarian adnexal tumor risk grading system, ultrasound image segmentation, as a prerequisite, aims to achieve automatic localization and structured segmentation of the lesion region, providing high-quality input for subsequent O-RADS feature extraction and CEUS dynamic analysis. Because both B-ultrasound and CEUS images suffer from grayscale blur, unclear boundaries, and strong lesion heterogeneity, conventional segmentation methods are difficult to handle stably. Therefore, this system introduces an adaptive bidirectional displacement (ABD) mechanism, combined with a confidence-guided strategy, to achieve highly robust lesion region segmentation under semi-supervision. ABD is divided into two major modules: self-supervised guidance based on unlabeled ultrasound images (ABD-R) and hard sample enhancement (ABD-I) based on labeled ultrasound images.

[0070] Furthermore, the ultrasound image segmentation module 120 includes:

[0071] The self-supervised guidance unit (ABD-R) 121 generates weakly enhanced and strongly enhanced samples for unlabeled ultrasound images. Through patch-level confidence evaluation and semantic matching, it replaces low-confidence areas with semantically similar high-confidence areas to generate enhanced training image samples. Furthermore, the self-supervised guidance unit (ABD-R) performs the following operations: dividing the unlabeled ultrasound image into multiple patches, calculating the average confidence and category distribution of each patch; selecting the patch with the lowest confidence as the area to be replaced, and selecting the replacement source from semantically similar high-confidence patches; generating new samples through bidirectional displacement: replacing the low-confidence patch of the weakly enhanced sample with the matching high-confidence patch of the strongly enhanced sample, and vice versa. More specifically:

[0072] (1) Self-supervised guidance based on unlabeled ultrasound images (ABD-R):

[0073] Considering that there are a large number of unlabeled ultrasound image samples in clinical ultrasound imaging data, in order to improve the generalization ability of the model, this module first generates a set of weak enhancement and strong enhancement pairs (such as color perturbation and blur processing) for each unlabeled ultrasound image, inputs them into two initially different sub-networks (UNet and Swin-UNet), and outputs the predicted probability distribution. .

[0074]

[0075] : Weakly enhanced samples (such as mild blurring and brightness perturbation processing) of unlabeled ovarian adnexa B-ultrasound images are used to preserve the original lesion structure. : Strong enhancement samples (such as color shift, morphological perturbation, etc.) of unannotated ovarian adnexa B-ultrasound images are used to enhance the robustness of the model to complex boundaries or heterogeneous masses. : Two semantic segmentation neural network structures with different initializations, UNet and SwinUNet, are used to predict the probability of the lesion area of ​​the input image. : Weakly enhanced image via the Internet The output logarithmic probability feature map (unnormalized) reflects the confidence level of each pixel belonging to different lesion categories. : Strongly enhanced image via the Internet The output logarithmic probability feature map reflects the segmentation consistency ability under image perturbations. :Will The predicted probability map obtained after normalization by the softmax activation function, the value of each pixel represents the probability of it belonging to a certain lesion category. :and , is the predicted probability map of the strongly enhanced image.

[0076] Then, the image is divided into multiple patches of equal size and the average confidence of each patch is calculated. and category distribution .

[0077]

[0078] :The original image The jth patch (size is Pixels), each patch is used to evaluate the prediction confidence of the local area. G is the full set of pixel-level classification categories: in the lesion segmentation task, it represents all tissue categories that need to be distinguished. For example, in ovarian tumor segmentation: G = {background, cystic tumor, solid tumor, mixed tumor}. Mathematically expressed: G = {0, 1, 2, 3}, where each number corresponds to a category, such as: g = 0 (background), representing normal ovarian tissue / peripheral organs; g = 1 (cystic tumor), representing a pure cystic tumor; g = 2 (solid tumor), representing a solid malignant tumor; g = 3 (mixed tumor), representing a mixed cystic and solid tumor; etc.). : represents the average of the logarithmic prediction scores for category g within the jth patch area, measuring the preference of the local area for this category; : Indicates the predicted probability that the m-th pixel in the j-th patch belongs to category g, that is, The value of the corresponding pixel position in . : The average confidence value of the jth patch, obtained by averaging the maximum category probability of each pixel, reflects the reliability of the model prediction in this area; higher values ​​indicate clearer classification, which is seen in lesion areas with clear boundaries.

[0079] Based on this, the patch with the lowest confidence score is selected As "confusing areas"; from the most semantically similar high confidence patches Select candidate regions.

[0080]

[0081] : In unlabeled weakly enhanced images The patch index with the lowest confidence corresponds to the area where the model is most uncertain, which often appears in areas with fuzzy boundaries or unclear lesion morphology. : The index of the patch with the highest confidence ranking i, where i=1 represents the patch with the highest confidence; such regions represent areas with clear ovarian tumor structures and stable model prediction results. :Depend on An index set consisting of the top n high-confidence patches in [1] is used to provide semantically reliable candidate regions in patch replacement.

[0082] KL divergence is used for semantic matching (for example, the KL divergence between the category probability distribution vectors of pixels in two patch areas is calculated), and a bidirectional displacement operation is performed, that is, a low-confidence patch is replaced with a semantically similar high-confidence patch to generate a new training sample.

[0083]

[0084] : KL divergence function, used to measure the two patches The smaller the value, the higher the semantic similarity between the two regions. :from Selected from The index of the high-confidence patch that is semantically closest to the low-confidence patch of the model; used to "replace" the latter and improve the performance of the model in abnormal perfusion or atypical edge areas. :from Selected from The index of the high-confidence patch that is semantically closest to the low-confidence patch of ; it is also used to assist in generating stable training samples.

[0085] Generate new samples: (Will The low confidence patch of ), otherwise generate .

[0086]

[0087] : The jth patch in the new synthesized sample. At patch index j equal to When the confidence level is the lowest, the patch will be The model is used to replace the semantically similar high-confidence lesion area in the image, which is used to "patch" the uncertain areas predicted by the model (such as masses with weak cyst walls and blurred boundaries). : The jth patch in the new synthesized sample. If the position is In the region where the model prediction is unreliable, use Replacing semantically similar high-confidence regions helps enhance the model's recognition stability of complex structures (such as mixed cystic and solid lesions) under image perturbations. :express The patch index with the lowest confidence corresponds to the area that is difficult for the model to identify, such as the edge of the lesion with uneven echo and unclear blood perfusion boundary. :exist Selected from, The high-confidence patch index with the closest semantics to the region indicates that the model is highly certain about the region and it is a typical lesion feature region.

[0088] This operation automatically fills in "difficult-to-identify ovarian lesion regions" (e.g., blurred perfusion margins or unclear papillary protrusions) with "semantically stable regions," thereby enhancing the model's ability to identify complex or heterogeneous lesions (e.g., borderline tumors and mixed cystic-solid lesions). This strategy effectively improves the model's segmentation robustness for cystic, solid, and mixed ovarian lesions without labeled supervision.

[0089] The difficult sample enhancement unit (ABD-I) 122 performs an inverse confidence shift strategy on the annotated ultrasound image, replacing high-confidence regions with low-confidence regions from another enhanced view to construct a difficult version of the training image sample. Furthermore, the difficult sample enhancement unit (ABD-I) performs the following operations: locating the patch with the highest location confidence in the annotated ultrasound image; replacing it with the patch with the lowest confidence from another enhanced view, while simultaneously adjusting the label mask; and generating synthetic samples that force the model to learn complex lesion edges, i.e., difficult version training image samples. More specifically:

[0090] (2) Hard-to-sample enhancement based on labeled ultrasound images (ABD-I)

[0091] Specifically: In order to improve the model's ability to learn complex structures such as lesion edges and subtle lesions (such as papillary protrusions and cyst wall enhancement areas) in existing annotated ultrasound images, this unit adopts the "inverse confidence shift strategy":

[0092] Identify the most stably predicted high-confidence patch in the current enhanced image; replace it with the least confident hard-to-predict region from another enhanced view (another enhanced version of the same annotated sample, such as a weakly enhanced view against a strongly enhanced view, or vice versa).

[0093]

[0094] : "Inverse replacement sample" in the labeled image. When the jth patch is When the model identifies the most certain area (i.e., the area with obvious lesions and clear boundaries), it is replaced by The most uncertain patches are used to create difficult training examples and enhance the model's ability to learn atypical tumor structures. :The direction is opposite to the above formula, use Replace with low confidence patches The high confidence regions in the image are used to challenge the model's ability to consistently identify small structures (such as papillary processes or cyst wall calcifications) under different enhancement conditions. : represent weakly enhanced samples and strongly enhanced samples from labeled images, respectively, which are used to construct contrastive learning conditions; their source images have clear pathological labels (such as "serous prostate carcinoma", "mature teratoma", etc.) and can be used for supervised learning. :Indicates The patch index that the model recognizes most confidently is the lesion center or the echo uniform area, which is "too helpful" for model training and needs to be replaced to avoid overfitting. :exist The most uncertain patch index may be a region with fuzzy boundaries, weak signals, or close to normal tissue areas, which is used to strengthen the model's willingness to learn "difficult areas".

[0095] The essence of the patch replacement strategy is to forcibly shift the model's attention from "easy-to-predict areas" to "difficult-to-predict areas," thereby enhancing its robustness in identifying areas with "blurred boundaries," "atypical perfusion," and "large tissue heterogeneity" in ovarian adnexal tumor images.

[0096] Synchronously adjust the label mask to ensure the consistency of the supervision signal.

[0097]

[0098] : The original labeled image comes from the annotated sample. The value of each pixel represents its category (such as normal adnexal tissue, benign mass, malignant lesion, etc.). It is generated after alignment with the pathological diagnosis result and serves as the supervision signal of the segmentation model. : The image is divided into Patch label area, representing the image patch The corresponding classification mask; each patch area retains the distribution information of its corresponding original lesion category (for example, the jth patch may cover the "solid tumor edge" or "blood flow enhancement area"). :For generating images When j is consistent with the highest confidence patch position in the original weakly enhanced image (i.e., it is replaced), the label area is replaced by The labels of the lowest confidence areas are used to "cover" the originally easy areas with labels of more difficult-to-identify areas, guiding the model to learn complex structures such as papillary protrusions and edge perfusion areas. : Pseudo labels generated by reverse synchronization. If the jth patch is The replaced area in (originally high confidence), then the label is changed to The labels of the regions where the model is most uncertain are simulated in the model to enhance the robustness of the model by simulating the “mismatch” of regions with dynamic perfusion changes or blurred boundaries. : In the strongly enhanced image In , the patch index with the lowest model prediction confidence often corresponds to the area with poor image quality, low contrast, or unclear lesion features. : In weakly enhanced images In , the index of the region where the model predicts the least confidence is often used to replace the target. : The location where the replacement occurs, that is, the location of the patch whose content is replaced in the new sample. These indices ensure that the replacement of the label is exactly consistent with the location of the change in the image patch content, preventing the supervision label from mismatching the input image.

[0099] This operation is equivalent to artificially creating "difficult samples" during the training phase, forcing the model to learn areas that were originally easily overlooked, and improving its recognition ability under complex lesions such as blurred edges, uneven perfusion, and mixed structures.

[0100] The segmentation map generation unit 123 is used for the segmentation map generation unit, which is used to synchronously train two segmentation models with different structures using the enhanced version training image samples, the difficult version training image samples, the original annotated ultrasound images and their corresponding labels, so as to utilize the complementarity of learning features of different network architectures, use the two trained segmentation models to infer the newly input ultrasound images, and output the lesion area mask of the ovarian adnexal tumor in each image.

[0101] (3) Segmentation model output and post-processing

[0102] After completing the two-stage ABD-R and ABD-I training, the system outputs a structured segmentation map (lesion region mask) of the ovarian adnexal tumor in each image. Its segmentation accuracy has been verified in preliminary experiments using the Dice coefficient and Intersection over Union (IoU) metrics. Specifically, the predicted segmentation map is compared with the gold standard of physician annotations. The metrics are calculated: the Dice coefficient, which measures the degree of overlap (0-1, higher is better); and the Intersection over Union (IoU), which measures segmentation accuracy (0-1, higher is better).

[0103] Furthermore, the two segmentation models trained by the segmentation map generation unit are the UNet model and the SwinUNet (Swin Transformer-based UNet) model, respectively. The inputs are: the enhanced image generated by ABD-R + the difficult image generated by ABD-I + the original annotated ultrasound image and its corresponding labels; the standard segmentation loss function (Dice Loss) is used; the two segmentation models generate prediction maps A (probability maps) and prediction maps B (probability maps) for the new ultrasound images respectively; the average prediction map C = (A + B) / 2 of the prediction maps A and B is calculated; the threshold segmentation binarization processing is performed according to the average prediction map C (the threshold is set according to the average probability map C, such as 0.5), and the lesion area and background area are generated, including: lesion area = area with average prediction value ≥ 0.5, background area = area with average prediction value < 0.5; then after post-processing, a binary segmentation mask is generated.

[0104] Among them, post-processing includes: removing small noise (areas with an area of ​​<10 pixels); filling small holes (holes with an area of ​​<20 pixels).

[0105] ABD-R provides a more diverse training image set, helping the model better handle fuzzy boundaries. ABD-I provides more challenging learning samples, enhancing the model's ability to recognize complex regions. Combining the strengths of both models yields more reliable segmentation results. Simple post-processing optimizes the visual quality of the segmentation map, ensuring it better meets physician expectations. The final output is a clear binary segmentation map of the lesion area, ready for subsequent feature extraction.

[0106] a multimodal feature extraction module 130 for extracting an O-RADS feature vector of the B-ultrasound image and a dynamic feature vector of the CEUS image based on the lesion area mask of the ovarian adnexal tumor in each image, and performing numerical normalization processing on the CA125 serum concentration data to obtain a standardized feature vector;

[0107] (1) Ultrasound image feature extraction

[0108] 1) O-RADS Features (Structural Features): A 3D ResNet network is used to automatically identify key O-RADS grading indicators such as lesion boundaries, echo type, and solid component ratio in B-mode ultrasound. Leveraging its three-dimensional convolutional capabilities, the 3D ResNet network not only extracts two-dimensional spatial features (such as lesion boundary shape and cyst wall thickness), but also integrates depth information from multiple image frames (such as slight angular changes caused by probe displacement) to achieve dynamic perception of the lesion's spatial structure.

[0109] For example: Input: original B-ultrasound image: 512×512 pixel grayscale image, segmentation mask: binary lesion area output by ABD module (1: lesion, 0: background);

[0110] Processing: The segmentation mask is superimposed on the original image to generate a focused image of the lesion area (with non-lesion areas set to zero); multi-frame sequence construction: adjacent frames (approximately 3-5 frames) generated by slight probe displacement are selected to form a spatial sequence; 3D ResNet processing: Input - spatial sequence (size: T × 512 × 512 × 1, where T is the number of frames) is used to extract joint spatiotemporal features through a 3D convolution kernel; feature mapping: The terminal fully connected layer outputs a structured vector: [boundary clarity score, solid component ratio, papillary protrusion landmark, septum thickness].

[0111] Output O-RADS feature vector: for example, [0.87, 0.35, 1, 2.1mm] (the quantitative result predicted / extracted by the model, with a dimension of 4), which respectively represents: boundary fuzziness (0-1), solid component ratio (0-1), presence of papillary protrusions (0 / 1), and maximum separation thickness (mm).

[0112] 2) CEUS Features (Hemodynamic Characteristics): A CNN+BiLSTM network is used to extract dynamic perfusion patterns (enhancement, filling rate, and clearance rate) from CEUS time-series images. The CNN extracts spatial features from each frame of the angiographic image, such as the distribution of the angiographic signal, perfusion brightness, and shape changes. The BiLSTM utilizes a bidirectional long short-term memory network to model the time series during the angiographic process, capturing dynamic patterns such as the onset of perfusion, peak enhancement, and clearance.

[0113] For example: Input: CEUS dynamic sequence: key frames (about 10-15 frames) of the arterial phase (20s) and portal venous phase (42s); segmentation mask: segmentation results of the same lesion at different times (time-varying ROI area);

[0114] Processing: Time-varying ROI extraction: A segmentation mask at the corresponding moment was applied to each frame of CEUS images to extract the dynamic perfusion lesion area; CNN spatial feature extraction: Input - single-frame CEUS lesion area (512×512), network - ResNet-18 convolutional layer, output - 256-dimensional feature vector for each frame; BiLSTM temporal modeling: Input - 256-dimensional feature vector of frame sequence (length = T), network - bidirectional LSTM unit, output - 128-dimensional temporal features; Dynamic parameter decoding: Fully connected layer mapping to clinical indicators: [time to peak (s), peak intensity (%), washout slope (ratio)].

[0115] Output CEUS dynamic feature vector: for example, [8.2s, 180%, 0.15] (model prediction / extraction quantitative results, dimension is 3), which respectively represent: contrast agent peak time, enhancement intensity compared to baseline, and washout rate (decreasing slope).

[0116] After CEUS feature extraction, the extracted feature vector [time to peak (s), peak intensity (%), washout slope (ratio)] is mapped to a CEUS malignancy risk score (grade 1-5) through a small classifier (such as a fully connected layer + softmax). Finally, this score can be used to calculate the OCC-US score (2 points for grade ≥4).

[0117] (2) CA125 numerical feature processing

[0118] CA125 is a commonly used serum tumor marker in the diagnosis of ovarian tumors, but its absolute value varies significantly between patients and is affected by factors such as the menstrual cycle and inflammation. Therefore, to enable it to more effectively participate in multimodal feature fusion, the system normalizes the CA125 value to obtain a standardized feature vector.

[0119] First, the raw CA125 concentration (range: 0-500 U / mL) is mapped to the [0, 1] interval through a normalization layer to prevent interference from scale differences in network learning. Next, a fully connected layer is introduced, using a nonlinear activation function to learn the underlying complex relationship between CA125 and tumor malignancy risk. For low concentrations accompanied by imaging abnormalities, the model may still classify them as malignant. For high concentrations but benign imaging structures, the model can be adjusted based on other modalities.

[0120] The CA125 feature (a serum tumor marker) is ultimately embedded as an independent vector and participates in the cross-attention calculation as a numerical feature in multimodal fusion. Its diagnostic weight is dynamically adjusted based on the complementarity of the modalities in the fusion strategy to avoid a single indicator dominating the judgment.

[0121] The multimodal feature fusion module 140 adopts a dual-branch fusion strategy that combines self-attention and cross-attention mechanisms to fuse the O-RADS feature vector of the B-ultrasound image, the dynamic feature vector and the normalized feature vector of the CEUS image, and obtain an OCC-US vector representation;

[0122] After completing B-mode ultrasound image segmentation and O-RADS structural feature extraction, learning the perfusion pattern of the CEUS dynamic image sequence, and normalizing the CA125 concentration, each modality outputs a structured feature representation. However, relying solely on a single modality (such as O-RADS images or CA125 values) can lead to one-sided diagnosis, especially when lesions have atypical appearances or blurred boundaries. Therefore, this system introduces a multimodal feature fusion method that combines self-attention and cross-attention mechanisms in the multimodal feature fusion module 140 to construct a unified risk score representation, namely the OCC-US score vector, which is used to achieve final benign and malignant differentiation and risk grading.

[0123] In order to effectively extract the complementary information between modalities and avoid redundant enhancement, the system adopts a "dual-branch fusion strategy": Self-attention (SA) branch: used to model the spatial and temporal correlation of single-modal internal features through the self-attention mechanism, and enhance the structural perception ability of single-modal internal features (i.e., lesion structure); Cross-attention (CA) branch: through the inverse attention mechanism, focus on cross-modal low-correlation areas, extract complementary features, capture the complementarity between modalities, and pay special attention to the different explanations of different modalities for the manifestation of the same lesion.

[0124] (1) Constructing the self-attention (SA) branch: Strengthening the unimodal internal representation

[0125] The feature tensor of each modality first models its local and global contextual relationship through the self-attention mechanism, such as the consistency between the CEUS perfusion peak and the enhancement timing, the correlation between the echo in O-RADS and the lesion edge morphology, etc. Furthermore, the operations performed by the self-attention (SA) branch include: decomposing the unimodal feature tensor into a query vector, a key vector, and a value vector through a learnable linear transformation; calculating the similarity matrix between the query vector and the key vector, and generating the attention weights after scaling and normalization; using the attention weights to perform weighted fusion on the value vector to obtain the enhanced unimodal feature; further extracting high-order feature associations through a multi-layer perceptron to strengthen the structural consistency within the unimodality. In the self-attention (SA) branch, each modality (O-RADS feature vector, CEUS feature vector, CA125 standardized feature vector) independently passes through the self-attention branch. Perform internal feature enhancement, that is, model the spatial and temporal correlation of the internal features of each modality through the self-attention mechanism.

[0126] The specific calculation method is as follows:

[0127]

[0128] : Input feature tensor, representing the mode c: Modality identifier, indicating the medical data modality currently being processed. For example, c = "O-RADS" indicates the structural features of B-ultrasound images, c = "CEUS" indicates the blood flow features of dynamic angiography, and c = "CA125" indicates the numerical features of serum tumor markers. : A learnable linear transformation matrix used to transform input features These are mapped into three vector representations: query, key, and value, thereby enabling the attention mechanism to model relevance. In ovarian tumor identification, this process helps determine whether, for example, a combination of "hypoechoic + solid component + micropapillary protrusions" constitutes a malignant sign. : Respectively represent the query (Query), key (Key), and value (Value) vector sets extracted from modality c. : Indicates what semantic information the current feature hopes to obtain (such as the features of the lesion center area); : Provides an "information index" of all locations in the image; : Contains the actual representation content of these positions (such as edge shape, reflection brightness, etc.). d: Feature dimension, representing The number of channels at each position (for example, each pixel is described by a 128-dimensional vector), which is used for normalization and scaling when calculating the attention score ( . : Represents the similarity scoring matrix between the query and the key. Larger values ​​indicate more semantically related locations. For example, in CEUS, the reinforcement starting point and the peak converging region have high similarity, and can enhance each other's feature expression. : Perform softmax normalization on the above scoring matrix row by row to obtain the "attention weight" of each position to other positions, that is, which areas the model pays attention to to update the current position features. : Normalization operation, generally LayerNorm, is used to stabilize training and standardize feature distribution to prevent gradient explosion or degradation. : Represents the enhanced features of modality c after processing by the SA module, with a stronger ability to understand the internal structure. For example, it can better capture the spatial synergistic features of "multi-locular swelling + irregular septation" in the image, or capture the internal consistency of "CEUS injection pattern + time curve". MLP : A multi-layer perceptron module, which often contains two fully connected layers and a ReLU activation function. Its function is to introduce nonlinear transformations and further dimensional reconstruction, so that the model can learn complex associations between features from higher-level semantics.

[0129] The SA module enhances the feature association within a single modality and captures local and global dependencies through the self-attention mechanism.

[0130] (2) Constructing the Cross Attention (CA) branch: extracting cross-modal complementary information (key to fusion)

[0131] In order to enhance the inter-modal linkage understanding of the same lesion in terms of "structural manifestation", "blood flow pattern" and "serum markers", the system introduces an inverse attention mechanism (re-softmax, i.e. inverse softmax or softmax(-X)), which focuses attention on low-correlation areas to discover complementary information. Furthermore, the operations performed by the cross-attention (CA) branch include: taking the first modality features as the query vector and the second modality features as the key vector and value vector; calculating the cross-modal similarity matrix, and using the inverse softmax operation (i.e. softmax(-X)) to generate complementary attention weights; re-weighting the second modality features by the complementary attention weights to focus on cross-modal low-correlation areas; outputting cross-modal features that enhance the difference information to resolve modal conflicts. In the cross-attention (CA) branch, between key modality pairs, it is performed: using the enhanced features of the CEUS modality as the query vector and the features of the O-RADS modality as the key vector and value vector to generate cross-modal enhanced features for the O-RADS modality. ; Using the enhanced features of the O-RADS modality as the query vector and the features of the CEUS modality as the key vector and value vector, generate cross-modal enhanced features for the CEUS modality The CA125 normalized feature vector does not participate in the cross-attention calculation, and only directly participates in the splicing of the final fusion vector after internal representation optimization through its self-attention branch.

[0132] Specifically:

[0133]

[0134] : Input features of the other modality. :With another modal feature Generate query vector, Still in current mode generate. : Cross-modal correlation matrix. Each position represents One area pair The semantic attention level of a region in . : Cross-modal enhancement features of the current modality c, composed of features from modal The query guides the re-weighted expression of the value region in the current modality. For example, if CEUS emphasizes abnormal perfusion in a certain blood supply area, the diagnostic weight of the corresponding regional feature in the O-RADS image can be increased. Complementary attention weight generation function : Inverse softmax function, in the traditional attention mechanism, the model will focus on the "highly relevant" area, while Reversely emphasize the "low correlation" area to extract the differences and complementary information between modalities. Shifting attention from high correlation areas to complementary areas, the expression is: The meanings of other parameters are the same as those of the SA module.

[0135] The CA branch uses re-softmax to focus on complementary cross-modal information and suppress redundant associations. For example, if the O-RADS image identifies a low-risk condition but CEUS shows abnormal early perfusion, the CA branch can amplify the diagnostic weight of the CEUS feature. Similarly, if the CA125 value is normal but the imaging structure shows multi-papillary projections, the CA branch can enhance the explanatory power of the image features.

[0136] (3) Fusion output: Construct a unified OCC-US (O-RADS, CEUS, CA125-based Ultrasound Scoring System) vector representation.

[0137] After processing by the SA and CA branches, the fused features of each modality are spliced ​​and sent to a unified fusion layer to form the final OCC-US vector representation, which integrates three types of data: O-RADS (structural features), CEUS (hemodynamic features), and CA125 (serum tumor marker). The vector structure is as follows:

[0138]

[0139] : The self-attention enhanced features of the O-RADS modality represent the spatial features (such as wall thickness and echo structure) extracted under single-modality internal structure recognition. : The cross-modality enhancement features of O-RADS modalities are guided by CEUS / CA125 to emphasize complementary information that it fails to pay attention to itself. : Self-attention enhancement characteristics of CEUS modality, such as internal temporal dynamic manifestations such as perfusion starting point, enhancement rate, and extinction pattern. : Cross-modality enhancement features of CEUS modality, using O-RADS image structure or CA125 abnormal values ​​to enhance the expression of its non-significant areas. : The feature vector of CA125 modality represents the embedded representation of serum tumor marker values, assisting the understanding of biochemical signals and the collaborative interpretation of image information. : The final spliced ​​multimodal fusion feature vector comprehensively considers the performance of ovarian adnexal tumors in three dimensions: "image structure (ORADS)", "blood flow dynamics (CEUS)", and "tumor marker (CA125)", and is used for subsequent benign and malignant discrimination and risk scoring.

[0140] Final output OCC-US vector The data will be sent to the risk grading decision module 150 to achieve classification and risk scoring of benign and malignant tumors.

[0141] The risk grading decision module 150 is used to input the fused OCC-US vector representation into the classifier, output the OCC-US score and the benign or malignant risk level, and dynamically optimize the scoring decision using reinforcement learning.

[0142] After completing multimodal feature fusion and generating a unified OCC-US vector representation, the system enters the critical risk scoring and classification phase. This step aims to integrate imaging, dynamic blood flow, and biochemical information into quantifiable and interpretable diagnostic indicators to assist in clinical judgment of benign and malignant ovarian adnexal tumors and guide surgical intervention decisions.

[0143] (1) Risk level output and identification strategy

[0144] The multimodal fusion feature vector As input, the system trains a deep classification model (multi-layer perceptron (MLP)) to perform multi-level classification of tumor types, and the output is the prediction result of the benign and malignant risk level (such as three-category probability: benign, suspicious, and malignant):

[0145] The probability of benign tumor is high (such as functional cysts and teratomas), and regular follow-up is recommended;

[0146] For suspicious areas in the middle (such as borderline tumors), enhanced imaging review is recommended;

[0147] If malignancy is highly suspected (such as serous cystadenocarcinoma, sarcoma), surgical treatment is recommended as soon as possible.

[0148] At the same time, according to the preset scoring rules: O-RADS ≥4 is assigned 2 points, CEUS score ≥4 is assigned 2 points, and CA125 ≥37.815 U / mL is assigned 1 point. The OCC-US score (0-5 points) of this case is calculated; the accuracy of the model prediction results is verified based on the OCC-US score.

[0149] During the model training process, the pathological diagnosis results are used as the standard, and the optimized loss function is a combination of cross entropy and FocalLoss to adapt to the imbalance problem of benign and malignant samples.

[0150] (3) Reinforcement learning optimization strategy

[0151] To further enhance the scoring system's adaptability in long-term clinical pathways, a reinforcement learning strategy was introduced to dynamically optimize the classifier. Using the SARSA(λ) algorithm as the core framework, the scoring system adaptively corrects for misdiagnosis risk by simulating the "diagnosis-decision-feedback" process.

[0152] Specifically, the reinforcement learning optimization adopts the SARSA(λ) algorithm. In the SARSA(λ) algorithm, the model must adjust the value of its "current scoring strategy" in each diagnosis-judgment-feedback cycle. Specifically, state (s): the multimodal fusion feature vector of the current patient , action (a): risk level action space {benign (0), suspicious (1), malignant (2)} predicted by the classification model (corresponding to scores 0-1, 2, ≥3 points), reward (R): if the predicted level is consistent with the final pathological result: +1; if the predicted level is inconsistent with the final pathological result: -1 (the reward value can also be adjusted according to clinical risk, such as a penalty of -2 for a malignant case being judged as benign); dynamically update the state-action value function through the time difference error; use the eligibility trace to trace the historical decision weights and adjust the parameter parameters of the classifier (MLP) in the risk grading decision module. This adjustment is completed by the following three parts:

[0153] 1) Temporal difference error (TD) error δ: measures the deviation between the prediction and the actual feedback. TD error is a signal to evaluate the quality of the current behavior, indicating the model's accuracy for a state-action pair. The gap between the value prediction and the actual feedback.

[0154]

[0155] s: The current system state, i.e., the OCC-US multimodal score vector, which is a fusion of multimodal features such as structural images (O-RADS), CEUS perfusion patterns, and CA125 biochemical markers, representing the diagnostic "context" of a particular ovarian adnexal mass. a: The action taken by the current classification model, i.e., the risk level prediction label (benign / suspicious / malignant). R: A positive reward (+1) is given if the result is consistent with the actual pathological result; otherwise, a penalty (-1) is given. : the expected cumulative discounted reward for taking action a in state s; : Next state and action The corresponding Q value is used to estimate the expected reward of the future path; in the process of "diagnosis-adjustment-rejudgment", this value simulates the long-term benefits that may be brought by the next step. δ: TD error, which measures the difference between the current prediction and the next state prediction : Future reward discount factor, weighing current and long-term benefits: the closer γ is to 1, the more emphasis is placed on long-term effects (such as the subsequent intervention costs caused by misdiagnosis); this system uses γ=0.5, which reflects the medium- and long-term impact of the diagnosis consequences.

[0156] If δ>0, it means that the model underestimates the value of the current behavior (for example, it should have been judged as malignant but was not); if δ<0, it means that the model overestimates the value of the current behavior (for example, it misclassifies benign as malignant).

[0157] 2) Qualification trace : Determine who should be "focused on updating".

[0158] The qualification trace plays a "memory role", recording which state-action pairs have recently participated in the decision-making and the "intensity" of participation; the currently executing Yes, its Set to 1; other non-current state-action pairs, by The proportion of participation gradually decays (the longer ago the participation, the smaller the impact); the existence of qualification traces not only updates the current behavior, but also retroactively adjusts related decisions in the past.

[0159] The qualification trace update formula is:

[0160]

[0161] : State-action pair The qualification trace value of is used to record its importance in recent learning, that is, "memory trace": if the current decision is ,but Otherwise, it gradually decays to ; This helps to give stronger adjustment signals to decisions that had a greater impact in the past when future returns arrive. : The attenuation coefficient of the qualification trace, which controls the weight range of historical influence: Only update the current state; All historical decisions are kept in memory; recommended in ovary identification , taking into account both current scores and past learning behaviors.

[0162] 3) Q-value update : Value correction is performed by superposition of TD error and qualification trace.

[0163] The Q-value (i.e., prediction of future reward) of each state-action pair is updated according to the following formula:

[0164]

[0165] : Learning rate, which controls the magnitude of each Q value update If it is set too large, the model may become unstable; if it is set too small, the update will be slow and the learning efficiency will be low.

[0166] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0167] It should also be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device comprising the elements.

[0168] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined in the embodiments of the present application may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the embodiments of the present application, but rather will conform to the widest scope consistent with the principles and novel features disclosed in the embodiments of the present application.

Claims

1. A risk grading system for ovarian adnexal tumors based on multimodal deep learning, characterized by: include: Multimodal data acquisition module, used to obtain ultrasound imaging data, CA125 serum concentration data and gold standard data of patients with ovarian adnexal tumors; Ultrasound image segmentation module, which is used to automatically locate and structure the ultrasound image lesion area based on the pre-processed ultrasound image data using adaptive bidirectional displacement, and output the lesion area mask of the ovarian adnexal tumor in each image; a multimodal feature extraction module for extracting an O-RADS feature vector of the B-ultrasound image and a dynamic feature vector of the CEUS image based on the lesion area mask of the ovarian adnexal tumor in each image, and performing numerical normalization processing on the CA125 serum concentration data to obtain a standardized feature vector; a multimodal feature fusion module for performing feature fusion on the O-RADS feature vector of the B-ultrasound image, the dynamic feature vector of the CEUS image, and the normalized feature vector using a dual-branch fusion strategy that fuses self-attention and cross-attention mechanisms to obtain an OCC-US vector representation; The risk grading decision module is used to input the fused OCC-US vector representation into the classifier, output the OCC-US score and benign or malignant risk level, and dynamically optimize the scoring decision using reinforcement learning; The ultrasound image segmentation module includes: A self-supervised guidance unit is used to generate weakly enhanced and strongly enhanced samples for unlabeled ultrasound images. Through patch-level confidence evaluation and semantic matching, low-confidence regions are replaced with semantically similar high-confidence regions to generate enhanced training image samples. A difficult sample enhancement unit is used to perform an inverse confidence shift strategy on the annotated ultrasound image, replacing the high-confidence area with the low-confidence area in another enhanced view to construct a difficult version of the training image sample; The segmentation map generation unit is used to synchronously train two segmentation models with different structures using the enhanced version training image samples, the difficult version training image samples, the original annotated ultrasound images and their corresponding labels, use the two trained segmentation models to infer the newly input ultrasound images, and output the lesion area mask of the ovarian adnexal tumor in each image.

2. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 1, characterized in that: The self-supervision guidance unit performs the following operations: Divide the unlabeled ultrasound image into multiple patches and calculate the average confidence and category distribution of each patch; Filter the patch with the lowest confidence as the area to be replaced, and select the replacement source from the semantically similar high-confidence patches; New samples are generated via bidirectional displacement: low-confidence patches of weakly enhanced samples are replaced with matching high-confidence patches of strongly enhanced samples, and vice versa.

3. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 1, characterized in that: The difficult sample strengthening unit performs the following operations: Locate the patch with the highest confidence in the annotated ultrasound image; Replace it with the patch with the lowest confidence in another enhanced version of the same annotated sample, and adjust the label mask simultaneously; Generate synthetic samples that force the model to learn complex lesion edges, i.e., difficult versions of training image samples.

4. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 3, characterized in that: The two segmentation models with different structures trained by the segmentation map generation unit are respectively a UNet model and a SwinUNet model; the UNet model and the SwinUNet model respectively generate a prediction map A and a prediction map B for a new ultrasound image; the average prediction map C of the prediction map A and the prediction map B is calculated; the threshold segmentation is binarized based on the average prediction map C to generate a lesion area and a background area, and then after post-processing, a binary segmentation mask is generated.

5. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 1, characterized in that: The multimodal feature fusion module adopts a dual-branch fusion strategy that integrates self-attention and cross-attention mechanisms, including: The self-attention branch is used to model the spatial and temporal correlation of single-modal internal features through the self-attention mechanism, thereby enhancing the ability to perceive lesion structures; The cross-attention branch is used to focus on cross-modal low-correlation regions through the inverse attention mechanism and extract complementary features.

6. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 5, characterized in that: The operations performed by the self-attention branch include: Decompose the unimodal feature tensor into query vector, key vector and value vector through a learnable linear transformation; Calculate the similarity matrix between the query vector and the key vector, and generate the attention weight after scaling and normalization; Use attention weights to perform weighted fusion on the value vectors to obtain enhanced unimodal features; The high-order feature associations are further extracted through the multi-layer perceptron to strengthen the structural consistency within the single modality.

7. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 5, characterized in that: The operations performed by the cross-attention branch include: The first modality feature is used as the query vector, and the second modality feature is used as the key vector and value vector; Calculate the cross-modal similarity matrix and use the inverse softmax operation to generate complementary attention weights; Reweighting the second modality features by the complementary attention weights to focus on cross-modal low correlation regions; Output cross-modal features that enhance difference information to resolve modal conflicts.

8. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 1, characterized in that: The risk grading decision module combines the multimodal fusion feature vector As input, the system trains a deep classification model to perform multi-level classification of tumor types and outputs a prediction of the risk level of benign or malignant tumors: benign, suspicious, or malignant. When the benign or malignant risk level prediction result is benign, follow-up is recommended; when the benign or malignant risk level prediction result is suspicious, enhanced reexamination is recommended; when the benign or malignant risk level prediction result is malignant, surgical intervention is recommended.

9. The ovarian adnexal tumor risk grading system based on multimodal deep learning according to claim 1, characterized in that: The SARSA algorithm is used to dynamically optimize the scoring decision by using reinforcement learning, including: The OCC-US vector is used as the state, the risk level prediction is used as the action, and the consistency of the pathological result is used as the reward signal; Dynamically update the state-action value function through temporal difference error; The qualification trace is used to trace the historical decision weights and dynamically adjust the parameters of the classifier in the risk grading decision module.

Images