Cancer auxiliary diagnosis and treatment method based on vision-language large model

Abstract

The invention discloses a cancer auxiliary diagnosis and treatment method based on a vision-language large model, and the method comprises the following steps: carrying out the decoupling of an original image-text pair information set, obtaining an independent image information set and an independent text information set, and respectively adding the original independent image information set and the original independent text information set into the corresponding independent image information set and the independent text information set; identifying and enhancing a key medical region of the image in the independent image information set by adopting a key information focusing mechanism to obtain a weighted image set; coupling the independent text information set, the weighted image set and the professional medical image-text information set under a cross-modal alignment mechanism to obtain an image-text pair information set; and mapping the image-text pair information set and other multi-modal information to the same semantic space for fusion, inputting the fused information into a target system model constructed based on a CLIP framework for training, and outputting the trained target system model. According to the invention, the accuracy of comprehensive judgment of the medical image and the text information can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a cancer auxiliary diagnosis and treatment method based on a large visual-language model. Background Art

[0002] The diagnosis and treatment of cancer has always received widespread attention. Cancer diagnosis and treatment is extremely cumbersome, requiring medical staff to process a large amount of medical images, text, and genome sequencing information. Large language models that can process multimodal information will undoubtedly greatly assist cancer treatment, improve diagnostic efficiency, and provide medical advice. In recent years, with the rapid development of large language models, large language models that can simultaneously process multimodal data and generate interactive texts have continued to increase. Many research teams are also paying attention to the application potential of multimodal large models in the medical field. Among them, the CLIP model under OpenAI has been pre-trained and can match text keywords and images with few or even zero samples, complete multimodal information processing, and generate interactive text. Because of its versatility, easy integration, and high efficiency, it provides research references for subsequent multimodal large models or directly participates in the construction of medical systems based on multimodal large models. This also demonstrates the excellent ability and unlimited potential of the CLIP model in multimodal medical information processing.

[0003] Although the CLIP model has strong multimodal information processing capabilities, it cannot be directly applied to medical problems. It is necessary to fine-tune the algorithm and select pre-trained information sets based on it. At present, with the CLIP model as the basic framework, many medical multimodal models are being continuously optimized, with stronger capabilities and higher information processing efficiency, but there are also certain defects. The MedCLIP model uses a different training method from CLIP. It decouples the original text-data pairs, incorporates more original independent images and independent texts for model training, and builds a semantic matching detection module to complete multimodal information alignment. This approach provides a data augmentation strategy, but the decoupling process causes the correspondence between text-image data pairs to be lost, and the model is also slightly powerless in the face of label error detection problems. At the same time, because there is less professional medical image-text annotation data, many research teams are forced to choose medical image-text data from the Internet and try to improve its professionalism. The MedKLIP model constructs a ternary extraction module to extract the entity, location, and existence of the disease from the original text, and uses the constructed medical knowledge base to enrich the above information for training. CLIPSyntel uses Chat-GPT to generate context related to medical queries, filter medical concepts, and generate visually aware summaries of medical questions. However, these models have failed to deeply extract key information from existing data and have certain defects in multimodal data alignment.

[0004] In summary, the above models provide a design solution for medical multimodal large models to a certain extent, but they still fail to solve the problems of low utilization of training sample data, false negatives in contrastive learning, and less utilization of multimodal information. In addition, the above models still have certain shortcomings in the problem of multimodal information alignment. Therefore, there is an urgent need for a new solution that can make full use of existing data and improve data utilization; at the same time, it has better multimodal information alignment effect and higher accuracy. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a cancer auxiliary diagnosis and treatment method based on a large visual-language model. In the method, the performance of the cancer auxiliary diagnosis and treatment system can be improved. On the premise of retaining the correspondence between the original image-text pair data, the amount of training data can be increased, and the model's attention to key cancerous sites can be improved; more modal information can be included to guide model training, and the accuracy of the target system model's comprehensive judgment of medical images and text information can be improved, thereby showing higher practicality in practical applications.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows:

[0007] A cancer auxiliary diagnosis and treatment method based on a visual-language large model comprises the following steps:

[0008] Acquire an original independent image information set, an original independent text information set, an original image-text pair information set, and a professional medical image-text information set, decouple the original image-text pair information set, obtain an independent image information set and an independent text information set, and add the original independent image information set and the original independent text information set to the corresponding independent image information set and independent text information set, respectively;

[0009] A key information focusing mechanism is used to identify and enhance key medical regions of images in the independent image information set to obtain a weighted image set;

[0010] The independent text information set, the weighted image set and the professional medical image-text information set are coupled under a cross-modal alignment mechanism to obtain the image-text pair information set;

[0011] The image-text pair information set and other multimodal information are mapped to the same semantic space for fusion and input into the target system model built based on the CLIP framework for training. The model parameters are optimized through the back-propagation algorithm to minimize the loss function until the preset requirements are met, and the trained target system model is output.

[0012] Input the data to be tested into the trained target system model to obtain cancer auxiliary diagnosis and treatment results.

[0013] Preferably, the method further comprises the following steps:

[0014] Performing data enhancement processing on the independent image information set, wherein the data enhancement processing includes space transformation and scale adjustment;

[0015] The independent text information set is cleaned, and the cleaning process includes removing stop words, eliminating punctuation interference, correcting spelling errors, and performing stemming or lemmatization.

[0016] Preferably, a key information focusing mechanism is used to identify and enhance key parts of the image in the independent image information set to obtain a weighted image information set, which specifically includes the following steps:

[0017] Automatically identify key medical regions in images using convolutional neural networks or deep learning image segmentation algorithms;

[0018] Calculate the weight ω of the key medical area i , and normalize the weight of the entire image. The weight calculation formula is as follows:

[0019]

[0020] In the formula, A ij is the similarity between the feature vector of the i-th key medical region and the feature vector of the j-th custom text label;

[0021] According to the calculated weights, the global feature representation F of the image is obtained image :

[0022]

[0023] Preferably, the image-text pair information set is mapped to the same semantic space for fusion with other multimodal information, specifically comprising the following steps:

[0024] Get the mapped joint feature vector Z, the formula is as follows:

[0025]

[0026] In the formula, F image is the global feature vector of the image; F text is the original feature vector of the text; F others is the original feature vector of other multimodal information; E image is the encoder of image features; E text is the encoder of text features; E others is the encoder of other multimodal information; concat is the connection function;

[0027] Calculate the similarity of the mapped multimodal feature vectors and normalize them using the softmax function to obtain the attention weights. Perform weighted summation on the fused multimodal features based on the calculated attention weights to generate aligned feature representations.

[0028] The fusion loss function can be expressed as:

[0029] L text_image_others =1-similarity(Z image ,Z text ,Z others ) (5)

[0030] Where Similarity(,,) represents the similarity measure between multidimensional feature vectors, such as cosine similarity. image ,Z text ,Z others They represent the components of the fused multimodal feature vector on image, text and other modal information respectively.

[0031] Preferably, the encoder of the image features adopts a CNN deep convolutional neural network; and the encoder of the text features adopts a Transformer convolutional neural network.

[0032] Preferably, the other multimodal information is genome sequencing data, physiological monitoring data and electrocardiogram numerical data.

[0033] Based on the above technical solution, the beneficial effects of the present invention are:

[0034] 1) The present invention increases the amount of model pre-training data: based on making full use of the coupling relationship between medical image-text data sets, the original independent text and independent image can be included in the training data set through data decoupling, thereby increasing the amount of data for sample training;

[0035] 2) The present invention expands the scope of application of the model and improves its accuracy: The model supports the inclusion of multimodal information such as genome sequencing of cancerous sites, medical expertise, medical images of cancerous sites and their text descriptions into the model training data set, which significantly improves the professionalism and scope of application of the model;

[0036] 3) The present invention improves the professionalism of the model: the model training embedded with professional medical knowledge improves the prediction accuracy of the model, reduces the "false negative" errors in comparative learning under the CLIP framework, and improves the professionalism of the model;

[0037] 4) The present invention reduces dependence on external data: the model is based on the CLIP large model framework, which gives full play to the excellent 0-sample learning ability of the CLIP model and can reduce the model's dependence on external data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The figure is a flowchart of a cancer auxiliary diagnosis and treatment method based on a visual-language large model in an embodiment. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] like Figure 1 As shown, this embodiment provides a cancer auxiliary diagnosis and treatment method based on a visual-language large model, in which a target system model is pre-built using a CLIP framework, including the following steps:

[0041] Acquire an original independent image information set, an original independent text information set, an original image-text pair information set, and a professional medical image-text information set, decouple the original image-text pair information set, obtain an independent image information set and an independent text information set, and add the original independent image information set and the original independent text information set to the corresponding independent image information set and independent text information set, respectively;

[0042] A key information focusing mechanism is used to identify and enhance key medical regions of images in the independent image information set to obtain a weighted image set;

[0043] The independent text information set, the weighted image set and the professional medical image-text information set are coupled under a cross-modal alignment mechanism to obtain the image-text pair information set;

[0044] The image-text pair information set and other multimodal information are mapped to the same semantic space for fusion and input into the target system model built based on the CLIP framework for training. The model parameters are optimized through the back-propagation algorithm to minimize the loss function until the preset requirements are met, and the trained target system model is output.

[0045] Input the data to be tested into the trained target system model to obtain cancer auxiliary diagnosis and treatment results.

[0046] In one embodiment, a cancer auxiliary diagnosis and treatment method based on a visual-language large model provides a method for obtaining an image-text pair information set, which uses data preprocessing and decoupling, and is designed to optimize the integration and analysis process of multimodal data, while not losing the corresponding relationship between image-text pair data, increasing the amount of training data, and improving the accuracy of the target system model's comprehensive judgment of medical images and text information. The implementation of this mechanism follows the following steps:

[0047] Original training data collection: In hospital databases and professional platforms, such as Dingxiang Doctor and other websites, we collect as much multimodal medical images containing various cancer pathology types, clinical text reports, and genome sequencing data corresponding to the cancerous sites of patients as possible to ensure the diversity and rigor of the constructed dataset in pathology. The professional medical image-text information set comes from the public dataset in the published papers, covering multimodal information such as X-ray, CT and MRI images, and is detailedly annotated and verified by professional radiologists.

[0048] Decoupling strategy: After incorporating the original image-text pair information into the target system model training data set, the original image-text pair information is decoupled, and the original independent image information set and the original independent text information set are incorporated into the decoupled data, reducing the dependence on paired data and expanding the scale of the available data set; at the same time, in-depth analysis of independent text information and independent image information is achieved.

[0049] Preprocessing of image and text data: A series of data enhancement techniques including spatial transformation and scale adjustment are used to enhance the model's robustness to image variations and improve its generalization ability. At the same time, a text cleaning process is performed, including but not limited to removing stop words, eliminating punctuation interference, correcting spelling errors, and performing stem extraction or word form restoration to improve the accuracy and efficiency of text processing.

[0050] Image-text matching matrix construction: CNN and Transformer neural networks are used to extract and encode medical entities from independent text and image data sets, combined with expert annotations and other professional medical knowledge, and the image-text matching matrix is ​​constructed using the similarity of high-dimensional vectors. The image-text matching process uses a key information focus mechanism to increase the decision weight of key parts of the image. The calculation formula for vector similarity is:

[0051] similarity(I i ,T i )(1)

[0052] Among them, I i ,T i Represents the high-dimensional image feature vector and text feature vector output by CNN and Transformer neural networks, and similarity(,) represents the similarity measure between feature vectors, such as cosine similarity.

[0053] Alignment and matching of decoupled data: Through the image-text matching matrix, the original independent image and text data are realigned and matched, serving the comparative learning process based on the CLIP framework, and incorporating the newly matched image-text data pairs into the model training dataset.

[0054] In one embodiment, a cancer auxiliary diagnosis and treatment method based on a visual-language large model provides a processing process for a key information focusing mechanism, which is a key technology for enhancing image understanding and highlighting key parts of the image, aiming to significantly improve the decision weight of key parts, such as cancerous lesions, adjust the calculation of the loss function, and improve the accuracy of medical image analysis. This mechanism is a component step of the above-mentioned image-text matching matrix construction. It is achieved by the following steps:

[0055] Key area identification: In the construction of an independent image-text matching matrix, a convolutional neural network or deep learning image segmentation algorithm is used to automatically identify key medical areas of the image, such as the location of cancerous or lesion sites in the image.

[0056] Regional feature enhancement and weight adjustment: Apply feature enhancement algorithms to the identified key medical regions to increase the prominence of these regions in the image, assign them higher weights, and normalize the weight of the entire image. The weight is calculated by using a convolutional neural network to extract image features and encode them into feature vectors, calculate the similarity between the feature vectors and custom text labels, and obtain a similarity matrix for each feature vector. Feature vectors and corresponding image regions that are more similar to custom text labels will receive higher weights. The weight calculation formula is:

[0057]

[0058] Where: i is the weight of the i-th key medical area. ij is the similarity between the feature vector of the i-th key medical region and the feature vector of the j-th custom text label. The similarity measure of the feature vectors is cosine similarity.

[0059] Weighted adjustment of loss function: A weight adjustment algorithm is introduced, which automatically adjusts the weight of the key area in the loss function according to its importance, and adjusts the model loss function accordingly. Misclassification of key medical areas will be subject to greater loss penalties, prompting the model to pay more attention to key medical areas. The loss function is a temperature-scaled cross entropy loss based on the CLIP model. According to the weights calculated in the previous step, the global feature representation F of the image is obtained. image :

[0060]

[0061] in, is the feature vector for each image region.

[0062] Contextual relevance enhancement: Combine textual context information to enhance the correlation between key medical areas and surrounding tissues, providing the model with a more comprehensive diagnostic basis.

[0063] In one embodiment, a cancer auxiliary diagnosis and treatment method based on a visual-language large model provides a processing process for mapping an image-text pair information set and other multimodal information to the same semantic space for fusion. This fusion mechanism can map image-text data pairs and other multimodal data such as genome sequencing information-text data to the same semantic space, and jointly perform model training, showing significant results. This is achieved by the following steps:

[0064] Extraction of other multimodal information and its corresponding text information: Use convolutional neural network tools adapted to various fields to extract key information from other modal data, such as using 1D-CNN convolutional neural network to extract mutation sites and expression change data from genome sequencing data; using 1D-CNN convolutional neural network to extract key features from patient physiological monitoring data, such as changes in heart rate, blood pressure, and blood sugar levels; using TCN time series convolutional neural network to extract abnormal waveforms from electrocardiogram numerical data, such as ST segment elevation or T wave inversion. At the same time, the convolutional neural network tools in the above fields encode these features with the corresponding medical text and convert them into standardized feature vector representations.

[0065] Cross-modal feature fusion: Through the cross-modal mapping function, image-text pair data and other multimodal data are mapped to a shared semantic space, and deep learning techniques such as autoencoders or variational autoencoders are used to achieve information fusion and knowledge transfer between different modalities. For example, the mapping function can learn the intrinsic representation of imaging and genomic data, allowing the model to capture cross-modal relationships and generate consistent feature representations in the shared space. The cross-modal mapping function can be expressed as:

[0066]

[0067] Where: F image is the global feature vector of the medical image. text is the original feature vector of medical text. others is the original feature vector of other multimodal information. image It is an encoder for medical image features, such as a CNN deep convolutional neural network. text Encoder for medical text features, such as Transformer convolutional neural network. others is the encoder of other multimodal information, including bioinformatics feature extraction methods and deep learning models in different fields, such as 1D-CNN convolutional neural network, TCN temporal convolutional neural network, etc. concat is the connection function, which is used to splice the output of the encoder of medical image features, medical text features and other multimodal information at the feature level. Z is the joint feature vector after mapping, which represents the fused multimodal feature vector in the shared semantic space.

[0068] Cross-modal alignment strategy: After multimodal information fusion, the semantic similarity of the fused multimodal feature vectors in the shared semantic space is calculated to align the multimodal information in the shared semantic space. Specifically, the similarity of the fused multimodal feature vectors is calculated, and the softmax function is used to normalize these similarities to obtain the attention weights. According to the calculated attention weights, the fused multimodal feature vectors are weighted and summed to generate the aligned feature representation.

[0069] Fusion loss function: Define a fusion loss function that not only considers the alignment between images and text, but also includes other modal data, such as genomic data, to ensure consistency and accuracy between multiple modalities. Fusion loss function L text_image_others It can be expressed as:

[0070] L text_image_others =1-similarity(Z image ,Z text ,Z others ) (5)

[0071] Where Similarity(,,) represents the similarity measure between multidimensional feature vectors, such as cosine similarity. image ,Z text ,Z others They represent the components of the fused multimodal feature vector on images, texts, and other modal information respectively. The number of dimensions is equal to the number of modalities of the multimodal information.

[0072] Iterative feedback loop: Implement an iterative feedback loop, where the output of the model will be used to further optimize the alignment process, optimizing the model parameters through the back-propagation algorithm to minimize the above loss function L text_image_others , fine-tuned based on clinical validation and expert feedback.

[0073] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0074] The above are only preferred embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A cancer-assisted diagnosis and treatment method based on a large visual-language model. It is characterized in that The steps include: Acquire an original independent image information set, an original independent text information set, an original image-text pair information set, and a professional medical image-text information set, decouple the original image-text pair information set, obtain an independent image information set and an independent text information set, and add the original independent image information set and the original independent text information set to the corresponding independent image information set and independent text information set, respectively; A key information focusing mechanism is used to identify and enhance key medical regions of images in the independent image information set to obtain a weighted image set; The independent text information set, the weighted image set and the professional medical image-text information set are coupled under a cross-modal alignment mechanism to obtain the image-text pair information set; The image-text pair information set and other multimodal information are mapped to the same semantic space for fusion and input into the target system model built based on the CLIP framework for training. The model parameters are optimized through the back-propagation algorithm to minimize the loss function until the preset requirements are met, and the trained target system model is output. Input the data to be tested into the trained target system model to obtain cancer auxiliary diagnosis and treatment results.

2. According to claim 1, a cancer auxiliary diagnosis and treatment method based on a visual-language large model, It is characterized in that The following steps are also included: Performing data enhancement processing on the independent image information set, wherein the data enhancement processing includes space transformation and scale adjustment; The independent text information set is cleaned, and the cleaning process includes removing stop words, eliminating punctuation interference, correcting spelling errors, and performing stemming or lemmatization.

3. According to claim 1, a cancer auxiliary diagnosis and treatment method based on a visual-language large model, It is characterized in that The key information focusing mechanism is used to identify and enhance the key parts of the image in the independent image information set to obtain the weighted image information set, which specifically includes the following steps: Automatically identify key medical regions in images using convolutional neural networks or deep learning image segmentation algorithms; Calculate the weight ω of the key medical area i , and normalize the weight of the entire image. The weight calculation formula is as follows: In the formula, A ij is the similarity between the feature vector of the i-th key medical region and the feature vector of the j-th custom text label; According to the calculated weights, the global feature representation F of the image is obtained image : in, is the feature vector for each image region.

4. According to claim 1, a cancer auxiliary diagnosis and treatment method based on a visual-language large model, It is characterized in that The image-text pair information set is mapped to the same semantic space with other multimodal information for fusion, which specifically includes the following steps: Get the mapped joint feature vector Z, the formula is as follows: In the formula, F image is the global feature vector of the image; F text is the original feature vector of the text; F others is the original feature vector of other multimodal information; E image is the encoder of image features; E text is the encoder of text features; E others is the encoder of other multimodal information; concat is the connection function; Calculate the similarity of the mapped multimodal feature vectors and normalize them using the softmax function to obtain the attention weights. Perform weighted summation on the fused multimodal features based on the calculated attention weights to generate aligned feature representations. The fusion loss function can be expressed as: L text_image_others =1-similarity(Z image ,Z text ,Z others ) (5) Where Similarity(,,) represents the similarity measure between multidimensional feature vectors, such as cosine similarity. image ,Z text ,Z others They represent the components of the fused multimodal feature vector on image, text and other modal information respectively.

5. According to claim 4, a cancer auxiliary diagnosis and treatment method based on a visual-language large model, It is characterized in that The encoder of the image features adopts a CNN deep convolutional neural network; the encoder of the text features adopts a Transformer convolutional neural network.

6. The cancer auxiliary diagnosis and treatment method based on the visual-language large model according to claim 4, It is characterized in that The other multimodal information includes genome sequencing data, physiological monitoring data and electrocardiogram numerical data.

Images