Prostate cancer diagnosis method based on multimodal large model prompt learning mechanism

Abstract

The invention belongs to the field of characterization learning, and particularly relates to a prostate cancer diagnosis method based on a multimodal large model prompt learning mechanism. Comprising the following steps: step 1, data preprocessing; 2, key frames and similarity are calculated, and irrelevant editing is compressed; step 3, text and image alignment training; step 4, a test stage; according to the method, a similarity-based screening mechanism is provided, ultrasonic videos under coarse-grained labels are preliminarily screened under segmentation of a large model, and focus areas are focused on in time sequence; meanwhile, a pre-processing mechanism based on a large model is provided, and the influence of a large amount of irrelevant information existing in prostate ultrasonic image scanning on the model is compressed at a data end, so that the diagnosis effect of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of representation learning, specifically relating to a method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism. Background Technology

[0002] Prostate cancer is the second most common type of cancer among men worldwide and a leading cause of cancer-related deaths. Magnetic resonance imaging (MRI) is the most commonly used imaging tool for prostate cancer diagnosis. MRI has high specificity and can be used to predict the T-stage of prostate cancer, but its sensitivity is low, and its widespread clinical application is limited by factors such as cost. Ultrasound imaging, another common imaging diagnostic tool, is also used in prostate cancer detection. Compared to MRI, ultrasound imaging has advantages such as lower cost, wider applicability, and less harm to the human body, making it more suitable for low-cost and efficient prostate cancer detection. However, the multifocal and scattered nature of prostate cancer reduces the diagnostic value of a single image. Therefore, ultrasound video encompassing the entire prostate gland and surrounding tissues and organs is used to compensate for the shortcomings of single-image diagnosis. However, ultrasound video imaging suffers from low resolution, making accurate diagnosis difficult. Furthermore, the sensitivity and specificity of ultrasound fluctuate significantly, especially in cases of small lesions and indistinct borders, making diagnostic accuracy dependent on the physician's personal experience.

[0003] Therefore, how to effectively improve the resolution of ultrasound videos to provide clearer and higher-fidelity images, and improve doctors' diagnostic accuracy for small lesions and complex areas with blurred boundaries, has become one of the important research directions and key technological development areas for prostate cancer diagnosis based on ultrasound imaging.

[0004] Video classification is an important means to improve the diagnostic effectiveness of ultrasound video. In the diagnosis of prostate cancer, the main diagnostic results are divided into three categories: Noca (no cancer), T0 stage cancer, and T1 stage cancer.

[0005] Unlike traditional single static images, ultrasound video provides temporal features through continuous frame information, which is of great significance for capturing the dynamic changes of prostate cancer, lesion development, and its relationship with surrounding tissues. By analyzing multiple frames in ultrasound video, the structure, blood flow, and morphology of prostate tissue can be more comprehensively assessed, thereby improving the detection rate and diagnostic accuracy of the disease.

[0006] Among existing technologies, MedSAM2 is an open-source large-scale segmentation model. MedSAM2 is a large-scale medical image segmentation model based on the classic SegmentAnything Model. By introducing spatial and semantic cue encoders, it significantly improves the segmentation accuracy and generalization ability for complex medical images. This model possesses powerful pre-trained weights that can provide segmentation knowledge of lesions, thus it can serve as a large-scale model cue for prostate cancer tumor staging and classification.

[0007] The CLIP method (Contrastive Language-Image Pretraining) provides an alignment approach for medical information and medical image alignment. In video modalities, each frame needs to be extracted and aligned with the text. Specifically, for video features, ResNet or a Vision Transformer is used to extract modal features; for text features, a pre-trained text encoder is used.

[0008] After obtaining features from two modalities, the similarity between corresponding images and text is reduced through comparative learning, and finally the parameters are updated using InfoNCE loss.

[0009] However, prostate ultrasound images contain a large amount of irrelevant information, especially in lesion-free areas. Utilizing the entire T-frame of the video in an approximation scheme introduces a significant amount of irrelevant information, affecting the model's feature learning of prostate ultrasound images and thus impacting the model's accuracy in diagnosing prostate cancer.

[0010] In addition, existing models do not fully consider the fluctuations in diagnostic styles and data quality among multiple doctors' sampling, resulting in unstable validation results that cannot meet the high requirements for accuracy and robustness in medical-engineering integration applications. Summary of the Invention

[0011] To address the problems of existing technologies, this invention proposes a prostate cancer diagnosis method based on a multimodal large model prompting learning mechanism. It utilizes the segmentation results output by a large medical segmentation model as prior knowledge for filtering and prompting, thereby compressing irrelevant content and highlighting important information in the sample. Simultaneously, the CLIP framework is employed to process both visual and textual modalities, improving the diagnostic effectiveness of this method.

[0012] Technical solution A method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism includes the following steps: Step 1 Data Preprocessing Input a prostate cancer ultrasound video, perform sliding window sampling on the video to obtain multiple clips of equal length, input the video clips into a pre-trained MedSAM2 segmentation model to obtain segmented images containing regions of interest.

[0013] Step 2: Calculate keyframes and similarity, and compress irrelevant clips. By analyzing the segmentation results of the entire video, n clusters are selected using clustering. The center frame of each cluster is selected as the keyframe for that cluster. A unique keyframe is then selected based on the lesion size ranking across several frames. The similarity between each frame and the keyframes is then calculated. Clips with no segmented content or whose similarity across all frames is below a threshold are discarded. The category labels of the filtered video clips and the original ultrasound video are then determined. (Noca without cancer, T0 stage cancer, T1 stage cancer) constitute multiple pairs of clips and text.

[0014] Step 3: Text and Image Alignment Training For the multiple clip-text pairs obtained in step 2, frame tokens are first generated for the clips using 2D convolution and linear layers. These tokens are then processed by a pre-trained vision encoder and used as visual features. For the text, these tokens are processed by a text encoder and used as text features. Finally, similarity is calculated. Cosine distance is used to reduce the distance between the same pair of clips and text, while increasing the distance between different pairs of clips and text.

[0015] Step 4 Testing Phase The video uses a pre-trained visual encoder and a text encoder. During testing, the three labels for prostate cancer—Noca (no cancer), T1 stage cancer, and T2 stage cancer—are encoded to obtain text features. The video is then segmented into multiple clips. After passing through the visual encoder, the cosine similarity between the visual features of each clip and the text features of the three labels is calculated. The clip with the highest similarity is used as the final classification result. The final classification result for the video is obtained by combining the classification results of all clips.

[0016] Beneficial effects The advantages of this invention for different tasks in the same field are summarized as follows: (1) Screening coarse-grained label mechanism: For the coarse labels of the main data, this invention proposes a similarity-based screening mechanism. Under the segmentation of the large model, ultrasound videos under coarse-grained labels are initially screened, and the lesion area is the focus in time sequence.

[0017] (2) Irrelevant data compression mechanism: This invention proposes a preprocessing mechanism based on a large model, which compresses the influence of a large amount of irrelevant information in prostate ultrasound imaging on the model at the data end, thereby improving the model's diagnostic effect. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the processing framework of the method of the present invention; Figure 2 This is a framework for the visual feature extraction process in an embodiment of the present invention; Figure 3 This is a framework for extracting text features according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating precision and recall in an embodiment of the present invention. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0020] A method for prostate cancer diagnosis based on a multimodal large model cueing learning mechanism, such as Figure 1 As shown. It includes the following steps: Step 1: Data Preprocessing Input a prostate cancer ultrasound video V, including T frames, with a height of H and a width of W; the video is labeled with a prostate cancer classification category label: Noca, T0, or T1.

[0021] A sliding window sampling with a step size of 2 is adopted, and the sliding window length num_frames is set to 8. Specifically, the first, third, fifth, ... frames of the video (i.e. odd frames) are taken as the starting frames, and the sliding window length (8) frames are taken continuously to form a video clip.

[0022] The final sample yields (T-8) / 2 video clips, denoted as ,in This represents frames 1-8 of the i-th clip. The tag for each clip comes from the video's tags.

[0023] The video clip is input into the pre-trained MedSAM2 segmentation model to obtain segmented images containing regions of interest. ,in This represents the segmented image corresponding to the i-th frame.

[0024] Step 2: Calculate keyframes and similarity, and compress irrelevant frames. Ultrasound imaging data exhibits strong continuity; that is, effective content (including lesion content) is relatively concentrated, while ineffective content (excluding lesion content) is also formed as continuous fragments. Therefore, clustering methods are used to aggregate data based on the distance between frames and the segmented lesion content. Specifically, the local density of the i-th frame is defined. for Where d ij The density distance between frame i and frame j is defined as follows: Among them, IoU ij The overlap rate of the segmented content between frame i and frame j is given by the formula for calculating the overlap rate of frames A and B: T represents the total number of frames in the video; Idx ij The calculation formula is: , which represents the absolute value of the difference between the index values ​​of the i-th frame and the j-th frame.

[0025] Then calculate the relative density for each data point. Minimum density distance at higher points Based on the minimum density distance of each frame Select high-density centers: Calculate the density product for each point. And calculate the threshold of the density product. as follows: in This refers to calculating the average of a set of data. This refers to calculating the standard deviation of a set of data.

[0026] Greater than the threshold The point is used as the high-density center (total) indivual):{ }, statistical analysis is performed on the segmented images corresponding to high-density centers. The frame with the largest foreground area in the segmented image corresponding to the high-density center is selected as the unique keyframe F. key_frame .

[0027] Definition of the first Frames and keyframes F key_frame similarity ,as follows: Define similarity threshold for: Calculate the video clip in step 1 Corresponding similarity , and make the following judgment: If the similarity of the entire video clip is lower than the similarity threshold , then this video clip will be discarded and not used in subsequent steps.

[0028] When sampling ultrasonic data, there are a large number of irrelevant frames at the beginning and end, without lesion content. Through the above processing, these interference information can be compressed.

[0029] After screening, M clips are obtained , where M < N. The class labels of the video clips and the original ultrasonic video constitute video clip and label pairs (Pairs), which are provided to Step 3.

[0030] Step 3: Text and Image Alignment Training Perform the following processing on the video clip and label pairs (Pairs) sampled and screened from the video: First, extract visual features, such as Figure 2 shown. It includes the following steps: Each video clip is represented as , where , H and W represent the height and width of the image, each clip is 8 frames, and perform two-dimensional convolution on the video clip , and its formula is expressed as Perform non-overlapping sampling on the convolution result in the spatial dimension, and each sampling block is in size, obtaining ; then divide it into num_frames individual vectors according to the frame dimension, obtaining , where, N= ( ) , and the token size is .

[0031] Use a linear layer to convert the vector converted from each frame into a D -dimensional vector. Then add a randomly initialized classification token , and the finally output vector is: where e represents cosine position encoding, and then the vector group goes through L layers of Video Encoder. VideoEncoder is a feature extractor based on Transformers. In this embodimentL It is 12. Representing the Layer VideoEncoder, the first The layer output is represented as Take the output of the last layer The 0th classification token of the visual features of the frame image After passing through a linear layer As the final visual feature .in The dimension of the final visual features.

[0032] in This represents the final visual encoder layer, containing features of the entire video. v Represented as in The features of each frame in a segment representing 8 frames. This represents the average pooling operation, which divides 8... Average This is how the video representation was extracted. .

[0033] Then extract text features, such as Figure 3 As shown. It includes the following steps: The text encoder uses a pre-trained BERT model, which has a total of L Layers are defined in this embodiment. L The number of layers is 12, and each layer consists of an MHSA multi-head self-attention neural network and an FFN feedforward neural network. (i=1,...,12) is the output of the i-th layer text encoder, then we have Finally in i When =12, text features are obtained. In summary, input editing and text After passing through the text encoder and visual encoder, the text features and visual features obtained are as follows: in and Representing visual encoders and text encoders, the input is a clip (clip) and text (C), which extract visual features.v and text features c Define cosine loss During training, maximize the pairings v,c Yes And minimize other v,c Yes Among them, the paired ones v,c Visual features obtained from the same pair of video clips and tags (Pair) v and text features c Other v,c Representation of visual features v and text features c From different video clips and tag pairs.

[0034] Step 4: Testing Phase During the testing phase, given the input video V and the list of tag texts List, a series of filtered clips from video V are first obtained using steps 1 and 2. M represents the total number of clips. The text list List is a list of cancer predictions for three categories: {"No cancer", "T0 stage cancer", "T1 stage cancer"}.

[0035] Using the previously pre-trained visual encoder and text encoder, the clip is obtained. Visual representation Text representation of a text list (List) ,in Calculate the visual representation of each clip. Cosine similarity to the text representation of a list of texts. The maximum cosine similarity is the classification prediction result (Noca: no cancer, T0 stage cancer, or T1 stage cancer).

[0036] Obtain prediction results for M clips. Then, a judgment is made: the prediction results of M clips are counted. If one clip is predicted to be stage T1 cancer, then the video is predicted to be a stage T1 cancer video; if no clip is predicted to be stage T1 cancer, but there is a clip predicted to be stage T0 cancer, then the video is predicted to be a stage T0 cancer video; if all clips are predicted to be stage Noca (no cancer), then the video is predicted to be a stage Noca (no cancer) video.

[0037] Example All experiments in this invention were performed on two NVIDIA RTX 3090 GPUs. The specific values ​​of the hyperparameters for this method are shown in Table 1.

[0038] Table 1. Value settings for various hyperparameters in this invention Dataset: The experiment was conducted on the dataset from Shanghai Tenth People's Hospital for both training and validation. The data statistics are shown in Table 2 below. Table 2 presents the statistics of the training and validation sets in this embodiment: Table 2 Statistical information of the dataset of this invention Method application: This experiment employed a sliding window sampling with a stride of 2, setting the training batch size to 64. Throughout the process, the learning rate (LR) was maintained at 4e-4. Pre-training was performed for 500 epochs at a learning rate of 1e-3 to minimize reconstruction and sampling losses. Subsequently, fine-tuning was performed for 300 epochs at a learning rate of 4e-4 to minimize cross-entropy loss. Training was conducted on two Tesla V100 GPUs.

[0039] Effect Comparison The experimental results are shown in Table 3 below. Figure 4 As shown, the evaluation metrics are ACC (Accuracy) and Pre (Precision), which measure the model's accuracy and discriminative ability in the prediction task, respectively. Table 3 Experimental Results Among them, PVTv2 (Pyramid Visual Transformer v2) is an efficient visual backbone network that performs excellently on dense prediction tasks. VideoMAEv2 is a video pre-training method based on mask autoencoders, which learns powerful representations by reconstructing a large number of occluded video frames.

[0040] As can be seen from the table, compared with some existing excellent classification methods, the method of this invention improves the ACC accuracy by 1.3% and the Pre accuracy by 1.56% in this experimental data set.

[0041] The above description is merely a description of preferred embodiments of this application and is not intended to limit the scope of this application in any way. Any changes or modifications made by those skilled in the art based on the above-disclosed technical content should be considered as equivalent and valid embodiments and fall within the scope of protection of the technical solution of this application.

Claims

1. A method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism, characterized in that, Includes the following steps: Step 1: Data preprocessing; Input a prostate cancer ultrasound video, perform sliding window sampling on the video to obtain multiple clips of equal length, input the video clips into a pre-trained MedSAM2 segmentation model to obtain segmented images containing regions of interest; Step 2: Calculate keyframes and similarity, and compress irrelevant clips; By analyzing the segmentation results of the entire video, n clusters are selected using clustering. The center frame of each cluster is selected as the keyframe for that cluster. A unique keyframe is selected based on the lesion size ranking of several frames. The similarity between each frame and the keyframe is then calculated. Clips with no segmented content or whose similarity is below a threshold are discarded. The category labels of the filtered video clips and the original ultrasound video are then determined. This constitutes multiple pairs of clips and text; Step 3: Text and image alignment training; For the clip and text pairs obtained in step 2, frame tokens are first generated for the clips through two-dimensional convolution and linear layers; after being processed by a pre-trained visual encoder, they are used as visual features. After passing through a text encoder, the text features are used to calculate similarity. Cosine distance is used to reduce the distance between the same pair of clips and text, and to increase the distance between different pairs of clips and text. Step 4: Testing Phase; The video is divided into multiple clips. After passing through the visual encoder, the visual features of each clip are compared with the text features of the three labels for prostate cancer: Noca (no cancer), T1 stage cancer, and T2 stage cancer. The clip is then divided into multiple clips. After passing through the visual encoder, the cosine similarity between the visual features of each clip and the text features of the three labels is calculated. The clip with the highest similarity is used as the final classification result. The final classification result of the video is obtained by combining the classification results of all clips of the video.

2. The method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism according to claim 1, characterized in that, Step 1 is as follows: Input a prostate cancer ultrasound video V, including T frames of images, with a height of H and a width of W, respectively; the video is labeled with a prostate cancer classification category tag: Noca, T0, or T1. A sliding window sampling with a step size of 2 is used, and the length of the sliding window num_frames is set to 8. Specifically, the first, third, fifth, ... frames of the video are taken as the starting frames, and the length of the sliding window is taken continuously to form a video clip. The final sample yields (T-8) / 2 video clips, denoted as ,in Represents frames 1-8 of the i-th clip; the tag for each clip comes from the video's tags; The video clip is input into the pre-trained MedSAM2 segmentation model to obtain segmented images containing regions of interest. ,in This represents the segmented image corresponding to the i-th frame.

3. The method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism according to claim 1, characterized in that, Step 2 is as follows: Define the local density of the i-th frame for Where d ij The density distance between frame i and frame j is defined as follows: Among them, IoU ij The overlap rate of the segmented content between frame i and frame j is given by the formula for calculating the overlap rate of frames A and B: T represents the total number of frames in the video; Idx ij The calculation formula is: , representing the absolute value of the difference between the index values ​​of the i-th frame and the j-th frame; Then calculate the relative density for each data point. Minimum density distance at higher points Based on the minimum density distance of each frame Select high-density centers: Calculate the density product for each point. And calculate the threshold of the density product. as follows: in This refers to calculating the average of a set of data. This refers to calculating the standard deviation of a set of data; Greater than the threshold The point is taken as the high-density center: { }, statistical analysis is performed on the segmented images corresponding to high-density centers. The frame with the largest foreground area in the segmented image corresponding to the high-density center is selected as the unique keyframe F. key_frame ; Definition of the first Frames and keyframes F key_frame similarity ,as follows: Define similarity threshold for: Calculate the video clip in step 1 Corresponding similarity And make the following judgment: if the similarity of the entire video segment is lower than the similarity threshold If so, this video clip will be discarded and will not be used in subsequent steps; After screening, M clips are obtained , where M < N; the category labels of the video clips and the original ultrasound video form pairs of video clips and labels and are provided to step 3.

4. The method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism according to claim 1, characterized in that, Step 3 specifically involves: The following processing is performed on the video clips and tag pairs obtained from video sampling and filtering: First, extract visual features. Includes the following steps: Each video clip is represented as ,in H and W represent the height and width of the image, and each clip consists of 8 frames for a video clip. The formula for performing a two-dimensional convolution is as follows: The convolution results are sampled spatially without overlap, with each sampling block being... Size, get Then, it is divided into num_frames individual vectors according to the frame dimension, resulting in... ,in, N=( ) The token size is ; Use a linear layer The vector converted from each frame Transform into a D A dimensional vector; then add a randomly initialized classification token. The final output vector for: in e Represents cosine position encoding, followed by vector group After L Layer Video Encoder, settings L The value is 12. VideoEncoder is a feature extractor based on Transformers. Representing the Layer Video Encoder, th The layer output is represented as: Take the output of the last layer The 0th classification token of the visual features of the frame image After passing through a linear layer As the final visual feature ,in Dimensions of the final visual features: in Representing the final visual encoder layer, the features of the entire video. v Represented as in The features of each frame in a segment representing 8 frames. This represents the average pooling operation, which divides 8... Average Extracting the video representation ; Then, text features are extracted, including the following steps: The text encoder uses a pre-trained BERT model, which has a total of L Layer, setting L The number of layers is 12, and each layer consists of an MHSA multi-head self-attention neural network and an FFN feedforward neural network. (i=1,...,12) represents the output of the i-th layer text encoder, then: Finally in i When =12, text features are obtained. In summary, input editing and text After passing through the text encoder and visual encoder, the text features and visual features obtained are as follows: in and Representing visual encoders and text encoders, the input is a clip (clip) and text (C), which extract visual features. v and text features c ; Define cosine loss During training, maximize the pairings v,c Yes And minimize other v,c Yes ; among which the paired v,c Visual features obtained from the same pair of video clips and labels v and text features c Other v,c Representation of visual features v and text features c From different video clips and tag pairs.

5. The method for diagnosing prostate cancer based on a multimodal large model cueing learning mechanism according to claim 1, characterized in that, Step 4 is as follows: During the testing phase, given the input video V and the list of tag texts List, a series of filtered clips from video V are first obtained using steps 1 and 2. M represents the total number of clips; List is a list of three types of cancer predictions {"Noca", "T0 stage cancer", "T1 stage cancer"}; Using the previously pre-trained visual encoder and text encoder, the clip is obtained. Visual representation Text representation of a text list (List) ,in Represents the predicted staging of three types of cancer; Calculate the visual representation of each clip Cosine similarity to the text representation of a list of texts. The maximum cosine similarity is the classification result; Obtain prediction results for M clips. Then, a judgment is made: the prediction results of M clips are statistically analyzed. If one clip is predicted to be stage T1 cancer, then the video is predicted to be a stage T1 cancer video; if no clip is predicted to be stage T1 cancer, but a clip is predicted to be stage T0 cancer, then the video is predicted to be a stage T0 cancer video; if all clips are predicted to be stage Noca (no cancer), then the video is predicted to be a stage Noca (no cancer) video.

Images