A prostate cancer diagnosis method based on a multi-modal large model prompt learning mechanism

Abstract

The application belongs to the field of characterization learning, and particularly relates to a prostate cancer diagnosis method based on a multi-modal large model prompt learning mechanism. The method comprises the following steps: step 1, data preprocessing; step 2, calculation of key frames and similarity, compression of irrelevant clips; step 3, text and image alignment training; and step 4, a test stage. The application proposes a screening mechanism based on similarity, under the segmentation of a large model, preliminarily screens the ultrasound videos under a coarse-grained label, and focuses on the lesion area in time sequence. Meanwhile, a pre-processing mechanism based on a large model is proposed, irrelevant information existing in prostate ultrasound image scanning is compressed at the data end, so as to improve the diagnosis effect of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of feature learning, and particularly relates to a prostate cancer diagnosis method based on a multi-modal large model prompt learning mechanism. BACKGROUND

[0002] Prostate cancer is the second most common type of cancer among men worldwide and one of the leading causes of cancer-related deaths. In the diagnosis of prostate cancer, magnetic resonance imaging (MRI) is the most commonly used imaging tool. MRI has high specificity and can be used for T-stage prediction of prostate cancer, but its sensitivity is low and it is difficult to popularize to extensive clinical applications due to factors such as price. As another common imaging diagnostic tool, ultrasound imaging has also been applied in the detection of prostate cancer. Compared with MRI, ultrasound imaging has the advantages of low cost, wide applicability and small harm to the human body, and is more suitable for low-cost and efficient detection of prostate cancer. However, the multiple and scattered features of prostate cancer reduce the diagnostic value of a single image, so ultrasound videos that wrap the entire prostate gland and surrounding tissues and organs are used to make up for the shortcomings of single image diagnosis. However, ultrasound video imaging has the problem of low resolution, which makes it difficult to accurately diagnose. In addition, the sensitivity and specificity of ultrasound have large fluctuations, especially in the case of small lesions and blurred boundaries, the accuracy of diagnosis depends on the personal experience of doctors.

[0003] Therefore, how to effectively improve the resolution of ultrasound videos to provide clearer and more high-fidelity images and improve the diagnostic accuracy of doctors for complex areas such as small lesions and blurred boundaries has become one of the important research directions and technical development focuses in the current diagnosis of prostate cancer based on ultrasound imaging.

[0004] Video classification task is an important means to improve the diagnosis effect of ultrasound video. In the diagnosis of prostate cancer, the main diagnostic results are divided into three categories: Noca (no cancer), T0 stage cancer, and T1 stage cancer.

[0005] Unlike traditional single static images, ultrasound videos provide temporal features through consecutive frame information, which is of great significance for capturing the dynamic changes of prostate cancer, the development of lesions, and their relationship with surrounding tissues. By analyzing multiple frames in ultrasound videos, the structure of prostate tissue, blood flow, and tumor morphology can be more comprehensively evaluated, thereby improving the detection rate of diseases and the accuracy of diagnosis.

[0006] In the prior art, MedSAM2 is an open-source segmentation large model. MedSAM2 is a medical image segmentation large model based on the classic work SegmentAnything Model, which greatly improves the segmentation accuracy and generalization ability of complex medical images by introducing a spatial and semantic prompt encoder. This model has strong pre-training weights and can provide some lesion segmentation knowledge, so it can be used as a large model prompt for prostate cancer tumor staging classification methods.

[0007] The CLIP method (Contrastive Language-Image Pretraining, basic picture-text alignment) provides an alignment idea for medical information and medical image alignment. In the video modality, each frame and the text need to be extracted and aligned. Specifically, for video features, a residual network (ResNet) or a visual transformer (Vision Transformer) is used to extract modal features; in the extraction of text features, a pre-trained text encoder is used to extract modal features.

[0008] After obtaining the features of the two modalities, the similarity between the corresponding images and texts is narrowed through contrastive learning, and finally the parameters are updated through the InfoNCE loss.

[0009] However, there is a lot of irrelevant information in the data of prostate ultrasound images, especially in the areas without lesions. In the approximate scheme, the T frames of the entire video are all used, which introduces a lot of irrelevant information, affecting the model's learning of the features of the prostate ultrasound image, and thus affecting the accuracy of the model's final prostate cancer diagnosis.

[0010] In addition, the existing model does not fully consider the diagnosis style and data quality fluctuations between multiple doctors' sampling, resulting in unstable validation results and failing to meet the high requirements of medical and engineering combined applications for accuracy and robustness. SUMMARY

[0011] To solve the problems in the prior art, the present application provides a prostate cancer diagnosis method based on a multi-modal large model prompt learning mechanism, which uses the segmentation results output by the medical segmentation large model as prior knowledge for screening and prompting, thereby compressing irrelevant content and highlighting important content in the sample. At the same time, the CLIP framework is used to process the visual and text modalities, improving the diagnostic effect of the present application.

[0012] TECHNICAL SCHEME

[0013] A prostate cancer diagnosis method based on a multi-modal large model prompt learning mechanism, comprising the following steps:

[0014] Step 1: Data preprocessing

[0015] Input the prostate cancer ultrasound video, sample the video with a sliding window to obtain multiple equal-length clips, input the video clips into a pre-trained MedSAM2 segmentation model to obtain a segmented image containing a region of interest.

[0016] Step 2: Calculate key frames and similarity, and compress irrelevant clips

[0017] By analyzing the segmentation results of the entire video, n classes are selected by clustering, and the center frame of each class is selected as the key frame of the class. The unique key frame is selected according to the size of the lesion in several frames. Then, the similarity between each frame and the key frame is calculated. For clips with all non-segmented frames or all frames with a similarity below the threshold, discard. After screening, the video clips and the class labels of the original ultrasound video (Noca, T0 cancer, T1 cancer) constitute a pair of clips and text pairs.

[0018] Step 3: Text and image alignment training

[0019] For the multiple pairs of clips and text pairs obtained in step 2, first generate frame tokens by two-dimensional convolution and linear layer for the clips. After passing through the pre-trained vision encoder, the visual features are obtained; after passing through the text encoder, the text features are obtained. Finally, the similarity is calculated. Use cosine distance to reduce the distance between the same pair of clips and text, and increase the distance between different pairs of clips and text.

[0020] Step 4: Test phase

[0021] The vision encoder and text encoder obtained by pre-training. In the test, the three labels of prostate cancer text Noca, T1 cancer, and T2 cancer are all text encoded to obtain text features. The video is divided into multiple clips, and the visual features of each clip and the text features of the three label texts are calculated after passing through the vision encoder. The cosine similarity is calculated, and the largest similarity is taken as the final classification result of the clip. The final classification result of the video is obtained by classifying all clips in the video.

[0022] Advantages

[0023] The superiority of the present application for other tasks in the same field is as follows:

[0024] (1) Screening coarse-grained label mechanism: For coarse labels of main data, the present application proposes a screening mechanism based on similarity. Under the segmentation of large models, the ultrasound video under the coarse-grained label is preliminarily screened, and the lesion area is focused on in the time sequence.

[0025] (2) Irrelevant data compression mechanism: This invention proposes a preprocessing mechanism based on a large model, which compresses the influence of a large amount of irrelevant information in prostate ultrasound imaging on the model at the data end, thereby improving the model's diagnostic effect. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the processing framework of the method of the present invention;

[0027] Figure 2 This is a framework for the visual feature extraction process in an embodiment of the present invention;

[0028] Figure 3 This is a framework for extracting text features according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram illustrating precision and recall in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0031] A method for prostate cancer diagnosis based on a multimodal large model cueing learning mechanism, such as Figure 1 As shown. It includes the following steps:

[0032] Step 1: Data Preprocessing

[0033] Input a prostate cancer ultrasound video V, including T frames, with a height of H and a width of W; the video is labeled with a prostate cancer classification category label: Noca, T0, or T1.

[0034] A sliding window sampling with a step size of 2 is adopted, and the sliding window length num_frames is set to 8. Specifically, the first, third, fifth, ... frames of the video (i.e. odd frames) are taken as the starting frames, and the sliding window length (8) frames are taken continuously to form a video clip.

[0035] The final sample yields (T-8) / 2 video clips, denoted as ,in This represents frames 1-8 of the i-th clip. The tag for each clip comes from the video's tags.

[0036] The video clip is input into the pre-trained MedSAM2 segmentation model to obtain segmented images containing regions of interest. ,in This represents the segmented image corresponding to the i-th frame.

[0037] Step 2: Calculate key frame and similarity, compress irrelevant frames

[0038] The data obtained by ultrasound image scanning has strong continuity, that is, the effective content (including lesion content) is relatively concentrated, and the invalid content (not including lesion content) is also continuous in fragments. Therefore, the clustering method is used for aggregation according to the distance between frames and lesion segmentation content. Specifically, the local density of the i-th frame is defined as

[0039]

[0040] Where d ij is the density distance of the i-th frame to the j-th frame, defined as

[0041]

[0042] Where IoU ij is the overlap rate of the segmentation content between the i-th frame and the j-th frame, and the overlap rate calculation formula of frames A and B is: ; T is the frame number of the entire video; Idx ij The calculation formula is: , which represents the absolute value of the index value difference between the i-th frame and the j-th frame.

[0043] Then calculate the minimum density distance of each data point to the relative density higher point

[0044]

[0045] According to the minimum density distance of each frame Select high density center: calculate the density product of each point , and calculate the threshold of the density product As follows:

[0046]

[0047] Where refers to calculating the average value of a group of data, refers to calculating the standard deviation of a group of data.

[0048] Points greater than the threshold are selected as high density centers (a total of ): { }, and the frame corresponding to the segmentation image of the high density center with the largest foreground area is selected as the unique key frame F key_frame .

[0049] Define the i-th frame and the key frame F key_frame ​​similarity of the video clip As follows:

[0050]

[0051] Define a similarity threshold For:

[0052]

[0053] Compute step 1 video clip Corresponding similarity And make the following judgments: if the similarity of the entire video clip is lower than the similarity threshold Then this video clip will be discarded and not used in subsequent steps.

[0054] There are a large number of irrelevant frames at the beginning and end of the ultrasound data sampling, and there is no lesion content. Through the above processing, these interference information can be compressed.

[0055] After screening, M video clips are obtained Where M < N. Video clips and original ultrasound video category labels Form a video clip and label pair (Pair) and provide it to step 3.

[0056] Step 3: Text and image alignment training

[0057] The video clip and label pair (Pair) obtained by sampling and screening from the video are processed as follows:

[0058] First, extract visual features, such as Figure 2 As shown. The following steps are included:

[0059] Each video clip is represented as Where H and W represent the height and width of the image, and each clip is 8 frames. Two-dimensional convolution is performed on the video clip The formula is represented as

[0060]

[0061] The convolution result is sampled in space without overlap, and each sampling block is Size, obtaining ; Then, according to the frame dimension, it is divided into num_frames individual vectors, obtaining Where, N=( ) The token size is .

[0062] A linear layer The vector converted from each frame is converted into a D dimensional vector. Then a randomly initialized classification token is added . The final output vector is:

[0063]

[0064] where e represents the cosine positional encoding, and the vector group has undergone L layers of Video Encoder, which is a Transformer-based feature extractor. In this embodiment L is 12, represents the i-th layer of Video Encoder, and the i-th layer output is represented as

[0065]

[0066] The 0th classification token of the i-th frame image visual feature of the last layer output is taken through a linear layer as the final visual feature . Wherein is the dimension of the final visual feature.

[0067]

[0068] wherein represents the last layer of the visual encoder, and the feature of the entire video is represented as v

[0069]

[0070] wherein represents the feature of each frame of the 8-frame segment, represents the average pooling operation, and the average of the 8 is , so that the representation of the video is extracted .

[0071] Then the text feature is extracted, as shown in Figure 3 . The following steps are included:

[0072] The Text Encoder uses a pre-trained BERT model, which has a total of L layers, and in this embodiment LFor 12, each layer is composed of MHSA multi-head self-attention and FFN feedforward neural network, set (i = 1,..., 12) is the output of the i-th layer of the text encoder, then

[0073]

[0074] Finally, when i = 12, the text feature is obtained.

[0075] In summary, the input clip and the text , after the text encoder and the visual encoder, the text feature and the visual feature

[0076] are obtained as follows

[0077]

[0078] Where and represent the visual encoder and the text encoder, the input clip clip and the text C, the visual feature v and the text feature c are extracted, the cosine loss is defined, when training, the v,c of the paired is maximized, and the v,c of the other is minimized. Where the paired v,c pair means that the visual feature v and the text feature c are obtained from the same pair of video clip and label pair (Pair); the other v,c pair means that the visual feature v and the text feature c come from different video clip and label pair (Pair).

[0079] Step 4: Test phase

[0080] In the test phase, for the input video V and the list of label texts List, first, a series of filtered clips from the video V are obtained by the method of step 1 and step 2, and M is the total number of clips. The list of texts List is the list of three categories of cancer prediction {“Noca no cancer”, “T0 stage cancer”, “T1 stage cancer”}.

[0081] Through the previously pre-trained visual encoder and text encoder, the visual representation of the clip and the text representation of the list of texts List are obtained. wherein The cosine similarity between each clip visual representation and the text representation of the text list List The maximum cosine similarity is the classification prediction result (Noca, no cancer or T0 stage cancer or T1 stage cancer).

[0082] Obtain the prediction results of M clips Then make a decision: count the prediction results of M clips, if one clip is predicted as T1 stage cancer, then the video is predicted as a T1 stage cancer video; if no clip is predicted as T1 stage cancer, but a clip is predicted as T0 stage cancer, then the video is predicted as a T0 stage cancer video; if all clips are predicted as Noca, no cancer, then the video is predicted as a Noca, no cancer video.

[0083] Embodiment

[0084] All experiments of the present application are completed on two NVIDIA RTX 3090 GPUs. The specific values of the various hyperparameters of the present method are shown in Table 1.

[0085] Table 1: Value setting of various hyperparameters of the present application

[0086]

[0087] Dataset:

[0088] The experiment is trained and verified on the dataset of Shanghai Tenth People's Hospital, and the data statistics are as follows. Table 2 shows the statistics of the training set and the verification set of the present embodiment:

[0089] Table 2: Dataset statistics information of the present application

[0090]

[0091] Method application:

[0092] The present experiment uses a sliding window sampling step of 2, sets the training batch size (Batch Size) to 64, and sets the learning rate (Learning Rate, LR) to 4e-4 throughout the process. Pre-training for 500 rounds at a learning rate of 1e-3 to minimize reconstruction loss and sampling loss. Then fine-tune for 300 rounds at a learning rate of 4e-4 to minimize cross-entropy loss. Train on two Tesla V100 graphics cards.

[0093] Effect comparison

[0094] The experimental results are shown in Table 3 and Figure 4As shown, the evaluation indexes are ACC (Accuracy) accuracy and Pre (Precision) precision, which respectively measure the accuracy and discrimination ability of the model in the prediction task:

[0095] Table 3 Experimental effect table

[0096]

[0097] Among them, PVTv2 (Pyramid Vision Transformer v2) is an efficient visual backbone network, which performs excellently in dense prediction tasks. VideoMAEv2 is a video pre-training method based on mask autoencoder, which learns powerful representations by reconstructing a large number of masked video frames.

[0098] As can be seen from the table, compared with some existing excellent classification methods, the method of the application improves the ACC accuracy by 1.3% and the Pre precision by 1.56% in the experimental data set.

[0099] The above description is only a description of the preferred embodiments of the application, and is not any limitation on the scope of the application. Any modification or modification made by any person skilled in the art according to the above disclosed technical content shall be regarded as an equivalent effective embodiment, and shall belong to the protection scope of the technical scheme of the application.

Claims

1. A method for diagnosing prostate cancer based on a multi-modal large model prompt learning mechanism, characterized in that, Comprising the following steps: Step 1 data preprocessing; Input the prostate cancer ultrasound video, sample the video with a sliding window, obtain multiple equal-length clips, input the video clips into the pre-trained MedSAM2 segmentation model, and obtain segmented images containing the region of interest; Step 2 calculate key frames and similarity, and compress irrelevant clips; Through the analysis of the segmentation results of the entire video, n classes are selected by clustering, and the center frame of each class is selected as the key frame of the class. The unique key frame is selected according to the size sorting of several frames. Then the similarity between each frame and the key frame is calculated. For all frames without segmentation or all frames with similarity lower than the threshold, the cutting is discarded. The video cutting after screening and the category label of the original ultrasound video are selected constitute multiple pairs of cuttings and text pairs; Step 3 text and image alignment training; For the clip and text pair obtained in step 2, first generate frame tokens through two-dimensional convolution and linear layer for the clip; After the pre-trained visual encoder, it is used as a visual feature; After the text encoder, it is used as a text feature, and finally the similarity is calculated; Use cosine distance to reduce the distance between the same pair of clips and texts, and increase the distance between different pairs of clips and texts; Step 4 test stage; Through the pre-trained visual encoder and text encoder; When testing, encode the text features of the three labels of prostate cancer Noca, T1 cancer and T2 cancer, cut the video into multiple clips, and calculate the cosine similarity between the visual features of each clip and the text features of the three label texts after the visual encoder. The maximum similarity is used as the final classification result of the clip; The final classification result of the video is obtained by classifying all clips in the video.

2. The prostate cancer diagnosis method based on the multi-modal large model prompt learning mechanism according to claim 1, characterized in that, Step 1 is specifically: Input the prostate cancer ultrasound video V, including T frames of images, and the height and width of the images are H and W respectively; The video has prostate cancer classification category labels Noca, T0 or T1; Adopt sliding window sampling with a step of 2, and set the sliding window length num_frames to 8. Specifically, take the 1st, 3rd, 5th, … frames of the video as the starting frame, and continuously take a sliding window length frame to form a video clip; The final sampling results in (T-8) / 2 clips of videos, denoted as where represents the first 8 frames of the i-th clip; the label of each clip is derived from the label of the video. inputting the video clip into a pre-trained MedSAM2 segmentation model to obtain a segmented image containing a region of interest wherein represents the segmented image corresponding to the i-th frame. 3.The method of claim 1, wherein the method is characterized by, Step 2 is specifically: Defining the local density of the i-th frame For where d ij is the density distance between the i-th frame and the j-th frame, defined as where IoU ij is the overlap rate of the segmentation content between the i-th frame and the j-th frame, and the frame A and B overlap rate calculation formula is: ; T is the frame number of the entire video; Idx ij The calculation formula is: , which represents the absolute value of the index value difference between the i-th frame and the j-th frame. Recalculate the relative density for each data point to the Minimum density distance to higher points According to the minimum density distance of each frame Select high density centers: compute density product of each point And compute threshold of density product As follows: wherein means to calculate the mean of a set of data, means to calculate the standard deviation of a set of data; Greater than threshold The point as a high-density center: The segmentation image corresponding to the high-density center is counted, and the frame with the largest foreground area of the segmentation image corresponding to the high-density center is taken as the unique key frame F key_frame ; Definition of the first frame and key frame F key_frame similarity As follows: Defining a similarity threshold is: Compute step 1 video segment Corresponding similarity And make the following judgment: if the similarity of the entire video segment is lower than the similarity threshold Then this video segment will be discarded and not used in subsequent steps; After the screening, M video clips are obtained where M < N; the class labels of the video clips and the original ultrasound videos The video clips and the labels form video clip and label pairs, which are provided to step 3. 4.The method of claim 1, wherein the method is characterized by, Step 3 is specifically: The video clip and label pair obtained by sampling and screening from the video are processed as follows: First, extract the visual feature, Comprising the following steps: Each video clip is represented as where H and W represent the height and width of the image, each clip is 8 frames, and a two-dimensional convolution is performed on the video clip whose formula is represented as The convolution result is sampled in space without overlapping, each sampling block is of size ; and then divided into num_frames individual vectors according to the frame dimension, to obtain , wherein N=( ) , and the token size is ; Using a linear layer Transforming each frame into a vector Transforming into a D dimensional vector; then adding a randomly initialized classification token The final output vector is: wherein e representing the cosine position encoding, followed by the vector group have been L layer Video Encoder, set L is 12, the VideoEncoder is a Transformer-based feature extractor, representing the first layer Video Encoder, the first layer output is represented as: the last layer output 0th classification token for frame image visual features , through a linear layer as final visual features where is the dimension of the final visual features: wherein represents the last layer of the visual encoder, the features of the entire video v are represented as wherein representing the features of each frame of a segment of 8 frames, representing an average pooling operation, the 8 resulting in an average of , extracting the representation of the video ; Then extract the text feature, comprising the following steps: The text encoder uses a pre-trained BERT model, which has a total of L layers, set to 12, each of which is composed of an MHSA multi-head self-attention and an FFN feed-forward neural network, set L to 4, and an output dimension set to 768. where yi=1,...,12 is the output of the i-th layer of the text encoder, then we have: Finally at i =12, obtain text features In summary, input clips and text are passed through a text encoder and a visual encoder to obtain text features and visual features as follows wherein and represent a visual encoder and a text encoder, input a visual clip and text C, extract visual features v and text features c ; Define cosine loss During training, maximize the pairings v,c Yes And minimize other v,c Yes ; among which the paired v,c Visual features obtained from the same pair of video clips and labels v and text features c Other v,c Representation of visual features v and text features c From different video clips and tag pairs. 5.The method of claim 1, wherein the method is characterized by, Step 4 is specifically: In the testing phase, for the input video V and the list of label texts List, a series of clips from the video V after screening are obtained by the method of step 1 and step 2 , M is the total number of clips; the list of texts List is the list of three categories of cancer prediction {"Noca no cancer", "T0 stage cancer", "T1 stage cancer"}; The visual representation of the clip and the text representation of the text list List are obtained by the previously pre-trained visual encoder and text encoder wherein represents the 3 cancer prediction stages; computing a cosine similarity of each clip visual representation and the text representation of the text list List a maximum cosine similarity is the classification result; Obtaining the prediction results of the M clips After that, a decision is made: the prediction results of the M clips are counted, if one clip is predicted as T1 cancer, then the video is predicted as T1 cancer video; if no clip is predicted as T1 cancer, but there is a clip predicted as T0 cancer, then the video is predicted as T0 cancer video; if all clips are predicted as Noca non-cancer, then the video is predicted as Noca non-cancer video.

Images