Breast image processing method, system and equipment based on artificial intelligence and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides a mammary gland image processing method, system and device based on artificial intelligence and a medium. The method comprises the steps of performing unsupervised learning pre-training on a multi-modal contrast learning network based on a mammary gland medical sample image and a corresponding diagnosis text sample report; based on the user sample question and the corresponding medical sample data, supervised learning training is carried out on the visual question and answer model VQA, the trained visual question and answer model VQA is obtained, and the visual question and answer model VQA comprises a pre-trained multi-modal contrast learning network, a classification network and a large language network model; the pre-trained multi-modal contrast learning network is connected with a large language network model through a classification network, the medical sample data comprises breast medical sample images and corresponding diagnosis text sample reports, and the medical data and user questions are input into a pre-trained visual question and answer model VQA to obtain a multi-modal contrast learning model; and enabling the visual question and answer model VQA to output a question answer corresponding to the user question.

Description

Technical Field

[0001] The present application relates to the fields of medical technology and artificial intelligence technology, and in particular to an artificial intelligence-based breast image processing method, system, device and storage medium. Background Art

[0002] Large-scale basic models for medical research are usually trained / fine-tuned using clinical data, which includes digital medical images (X-rays and CT scans, etc.), user-uploaded images, clinical metadata, and clinical reports. Among them, Visual Language Models (VLMs) for Visual Question Answering (VQA) tasks account for a large part. Image content VQA (generating questions based on entity and table information in medical reports) are two mainstream tasks of VQA tasks.

[0003] Traditional solutions include PubMedCLIP, BiomedCLIP, and HuaTuo for VQA tasks. PubMedCLIP and BiomedCLIP are a two-stream model based on CLIP, which can be fine-tuned for VQA. HuaTuo specifically uses traditional Chinese medicine knowledge to adjust LlaMA. ​​CLIP is used for visual feature training, LLaMa-7B is used for language understanding, and PMC-15 is fine-tuned for VQA. Traditional VQA only uses medical images unilaterally for single-modality or multi-modality training and use, resulting in limited output accuracy, and there is no effective VQA solution specifically for breast imaging. Summary of the invention

[0004] The embodiments of the present application provide a breast image processing method, system, device and storage medium based on artificial intelligence to implement an intelligent question-answering processing solution specifically for breast medical images.

[0005] Based on breast medical sample images and corresponding diagnostic text sample reports, unsupervised learning pre-training is performed on a multimodal contrastive learning network, wherein the multimodal contrastive learning network includes a single-modal contrastive learning branch and a multimodal contrastive learning branch; Based on user sample questions and their corresponding medical sample data, supervised learning training is performed on a visual question answering model VQA to obtain a trained visual question answering model VQA, wherein the visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network, and a large language network model, wherein a multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network, and the medical sample data includes breast medical sample images and corresponding diagnostic text sample reports; Obtaining medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports; The medical data and the user questions are input into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs answers corresponding to the user questions.

[0006] A breast image processing system based on artificial intelligence, characterized by comprising: The first training module is used to perform unsupervised learning pre-training on the multimodal contrastive learning network based on breast medical sample images and corresponding diagnostic text sample reports; The second training module performs supervised learning training on the visual question answering model VQA based on user sample questions and their corresponding medical sample data to obtain a trained visual question answering model VQA, wherein the visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network, and a large language network model, wherein the multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network, and the medical sample data includes breast medical sample images and corresponding diagnostic text sample reports; An acquisition module, used to acquire medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports; The output module is used to input the medical data and user questions into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs the answers corresponding to the user questions.

[0007] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the aforementioned artificial intelligence-based breast image processing methods when executing the computer program.

[0008] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the artificial intelligence-based breast image processing method as described in any of the above items.

[0009] In one of the solutions provided in the present application, a multimodal contrastive learning network, a classification network, and a large language network model are first obtained by pre-training the multimodal contrastive learning network for multimodal unsupervised learning based on breast medical sample images and corresponding diagnostic text sample reports to obtain network parameters in the multimodal contrastive learning network. The classification network and the large language network model are combined and supervised fine-tuning training is performed using user sample questions and their corresponding medical sample data. The single-modal contrastive learning branch and the multi-modal contrastive learning branch are used to pre-learn and train the single modality and the multi-modality respectively, and then fine-tuning is performed based on the constructed visual question answering model to obtain a visual question answering model VQA specifically for breast images, providing an effective visual question answering processing solution for breast images, and the single-modal contrastive learning branch and the multi-modal contrastive learning branch are used to pre-learn and train the single modality and the multi-modality respectively, so that the modal features can be effectively pre-learned, so that the model recognition accuracy of the visual question answering model using the multimodal contrastive learning network is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of a breast image processing method based on artificial intelligence in an embodiment of the present application; Figure 2 It is a schematic diagram of a training process of a multimodal contrastive learning network in one embodiment of the present application; Figure 3 1 is a schematic diagram of a network framework of a multimodal contrastive learning network in an embodiment of the present application; Figure 4 is a schematic diagram of a network framework for selecting an image encoder using density classification in one embodiment of the present application; Figure 5 It is a schematic diagram of a training process of a multimodal contrastive learning network and a classification network in one embodiment of the present application; Figure 6 is a flow chart of a breast image processing method based on artificial intelligence in one embodiment of the present application; Figure 7 is a structural schematic diagram of a video processing device based on artificial intelligence in one embodiment of the present application; Figure 8 It is a structural diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0012] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0013] The present application embodiment provides a breast image processing method based on artificial intelligence, which can be applied as follows: Figure 1 Specifically, the application environment includes: Figure 1 In the server shown, the client can be installed on but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The client can receive medical data uploaded by users, that is, corresponding user questions, and the server can receive the medical data, that is, corresponding user questions, and then input the medical data and user questions into the pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs the answer to the user question. The server can be a cluster server or a separate server, which is not specifically limited.

[0014] In the embodiment of the present application, a visual question answering model VQA, a corresponding training method and a method of use, that is, a breast image processing method, which can be specifically targeted at an intelligent processing solution for breast images, are provided. For ease of understanding, the visual question answering model VQA provided in the embodiment of the present application and its training process are first described.

[0015] In one embodiment, the visual question answering model VQA includes a multimodal contrastive learning network, a classification network and a large language network model, wherein the multimodal contrastive learning network is connected to the large language network model through the classification network, the multimodal contrastive learning network is obtained by multimodal unsupervised learning pre-training based on breast medical sample images and corresponding diagnostic text sample reports, the multimodal contrastive learning network includes a unimodal contrastive learning branch and a multimodal contrastive learning branch, the visual question answering model VQA is obtained by supervised fine-tuning training based on user sample questions and their corresponding medical sample data, the visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network and a large language network model, the multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network, and the medical sample data includes breast medical sample images and corresponding diagnostic text sample reports.

[0016] Multimodality refers to the sample reports based on breast medical images and the corresponding diagnostic text sample reports. The diagnostic text sample reports include text reports, that is, multimodality includes images and text. Sample images based on breast medical images include but are not limited to X-rays, CT scans or MRI images, etc., and there is no specific limitation. That is, for each type of breast medical sample image, the model framework can be constructed and trained in the above manner, thereby obtaining a VQA model for each type of breast medical image.

[0017] It can be seen that in this embodiment, a visual question answering model VQA framework is provided, including a multimodal contrastive learning network, a classification network and a large language network model. First, based on breast medical sample images and corresponding diagnostic text sample reports, the multimodal contrastive learning network is pre-trained for multimodal unsupervised learning to obtain network parameters in the multimodal contrastive learning network. After combining the classification network with the large language network model, supervised fine-tuning training is performed using user sample questions and their corresponding medical sample data. Thus, a visual question answering model VQA specifically for breast images is obtained.

[0018] In one embodiment, the image encoders of the unimodal contrastive learning branch and the multimodal contrastive learning branch share the same network coefficients, such as Figure 2 As shown, unsupervised learning pre-training is performed in the following way: S101, performing two random masking operations on the breast medical sample image to generate a first mask image and a second mask image of the breast medical sample image, and inputting the first mask image and the second mask image into two image encoders of the single-modality contrast learning branch respectively to perform single-modality contrast learning; S102, inputting the breast medical sample image and its corresponding diagnostic text sample report into the image encoder and text encoder of the multimodal contrastive learning branch respectively to perform multimodal contrastive learning until the unsupervised learning pre-training of the encoder is completed.

[0019] In this embodiment, in order to achieve better alignment of text and image modalities, a novel multimodal contrastive learning network is designed, which is called the projection model UMiCon in the embodiment of this application. Figure 3 As shown, the multimodal contrastive learning network includes a single-modal contrastive learning branch and a multimodal contrastive learning branch. Figure 3The upper dashed box in the figure represents a unimodal contrastive learning branch, and the lower dashed box represents a multimodal contrastive learning branch. The unimodal contrastive learning branch shares the same network coefficients with the image encoder of the multimodal contrastive learning branch. The shared network coefficients mainly include the image encoder in the learning branch. During contrastive learning training, unimodal and multimodal contrastive learning are used for training, bridging the gap between visual and textual modalities.

[0020] In order to improve the subsequent high-quality question and answer, first, the breast medical sample image is randomly masked twice to generate the first mask image and the second mask image of the breast medical sample image. Exemplarily, a mask of 8%d of the image area is randomly added twice to generate the first mask image and the second mask image respectively, that is, to generate a pair of mask input images, and the first mask image and the second mask image are respectively input into the two image encoders of the single-modal contrast learning branch to perform single-modal contrast learning. Among them, the two image encoders share weight coefficients (ShareWeights), and the mask image is subjected to pooling and flattening operations after the image encoder, and the visual features are mapped to the multi-layer perception layer (Multilayer Perceptron, MLP) therein, and a 1×2 vector is output, wherein the two corresponding mapping processes can also share weight coefficients (Share Weights). In addition, exemplary, the perception layer MLP can be a six-layer MLP.

[0021] In addition, the breast medical sample image and its corresponding diagnostic text sample report are respectively input into the image encoder and text encoder of the multimodal contrastive learning branch to perform multimodal contrastive learning until the unsupervised learning pre-training of the encoder is completed. The multimodal contrastive learning branch is executed by the Siamese contrastive learning module. The input breast medical sample image is also input into the image encoder of the branch with shared weights. Then, the encoded features are projected into two one-dimensional vectors (H×W×C) representing the possibility of its category, where C represents the number of channels, W represents the width, and H represents the height. The diagnostic text sample report corresponding to the breast medical sample image is input into the editor of this article, and after encoding, it forms a vector (T×D) and enters the projection layer to output the vector (T×C). T and D represent the rows and columns of the vector (T×D), respectively, and T and C represent the rows and columns of the projected vector (T×C), respectively. The corresponding vectors of the breast medical sample images and their diagnostic text sample reports are then globally pooled and mapped to output a 1×C vector. The contrast loss of the single-modality contrast learning branch and the multi-modality contrast learning branch aims to make breast medical sample images from the same category closer and separate breast medical sample images from different categories further in the projection space. Figure 3 The unsupervised learning phase of the multimodal contrastive learning network is shown in Figure .

[0022] It should be noted that Figure 3 In the example, the breast medical sample image is illustrated by taking a breast X-ray image and a diagnostic text sample report (Diagnostic report) as examples. The two are in a corresponding relationship (Corresponding). The content of the diagnostic text sample report (Diagnostic report) at least includes findings (abnormal information) and impression (overall diagnostic results).

[0023] It can be seen that in this embodiment, in order to better align the modalities of text and image, a novel multimodal contrastive learning network is designed in the embodiment of the present application, and unsupervised learning pre-training is performed using single-modal and multimodal contrastive learning, which bridges the gap between the visual and textual modalities of the samples, and is conducive to subsequent fine-tuning to obtain the final visual question answering model VQA.

[0024] In one embodiment, the breast medical sample images include breast medical sample images graded by the Breast Imaging Reporting and Data System (BI-RADS). It should be understood that the above BI-RADS grade is a graded evaluation standard that is currently widely used.

[0025] In an embodiment of the present application, in order to perform more ideal pre-training of the multimodal contrastive learning network UMiCon, a classification task between BI-RADS graded breast medical sample images can be used to pre-train the multimodal contrastive learning network UMiCon, thereby improving the pre-training effect of the multimodal contrastive learning network UMiCon and facilitating the subsequent achievement of more ideal training results.

[0026] As an exemplary explanation, the multimodal contrastive learning network UMiCon may be pre-trained using a classification task between BI-RADS 3 and 4 (the most difficult BI-RADS to classify), or other samples that are more difficult to classify, without specific limitation.

[0027] In one embodiment, the breast medical sample images are breast medical images of the same density classification category, and the image encoder in the multimodal contrast learning network is an image encoder corresponding to the density classification category.

[0028] Since breast medical sample images generally have different density categories, taking breast X-ray images as an example, they mainly include the following density categories: Almost entirely fat (A); There are scattered areas of dense fibroglandular breast tissue (B); There are many glands and connective tissue areas (C); Extremely dense (D).

[0029] Since there are differences between these density categories, it is feasible and necessary to select different image encoders for the multimodal contrastive learning network according to different density categories to adapt to and target training. Therefore, in this embodiment, the breast medical sample images are breast medical images of the same density classification category, and the image encoder in the multimodal contrastive learning network is the image encoder corresponding to the density classification category, so as to ensure that the multimodal contrastive learning network is pre-trained in a targeted manner.

[0030] As an example, Figure 4 As shown in Figure 1, the Sparse Mixture of Experts (Visual-MoE) architecture provides a scalable, efficient, and flexible approach to meet the requirements of encoder selection and build models while maintaining computational efficiency. Therefore, Visual-MoE is used to process mammograms as follows Figure 4 As shown, in the embodiment of the present application, different image encoders (Visual Encoder (s)) were tried and the three most sensitive to density were selected. Including: CLIP (Contrastive Language Image Pre-training) image encoder performs best for generating features for Class A images, ConvNeXt-Tiny image encoder performs best for generating features for Class C and Class D images, and Dinov2 image encoder is most suitable for Class B images. In the embodiment of the present application, a micro selector can be added to serve as a density classifier (Denslty Classlfler). Exemplarily, the density classifier includes classifiers such as ResNet-50. ResNet-50 is a type of deep residual network (Deep residual network, ResNet-). Taking this as an example, ResNet-50 first processes the input mammogram (Mammogram (s)) and then sends it to the selected image encoder. That is, according to the selection of the density classifier, the input images of different density classifications are matched to different encoders (Feed the image to different encoderbased on its density).

[0031] It should be noted that the above classification is only an example, and there may be other classification situations, which are not limited to the specific embodiments of the present application.

[0032] In one embodiment, the classification network includes a feature fusion layer and a classification head, wherein the feature fusion layer includes a Concat module and a Layernorm module connected in sequence.

[0033] In this embodiment, after the multimodal contrastive learning network UMiCon is obtained through pre-training, it can be transferred to the supervised learning training stage, such as Figure 5 As shown, the classification network includes a feature fusion layer and a classification head (i.e. Figure 5 The feature fusion layer includes a Concat module and a Layernorm module connected in sequence. The features of the breast medical sample image and the corresponding diagnostic text sample report are set to the same dimension (1×C). After direct connection and layer normalization, the cross-modal features are fused into a 2D vector (2×C) to achieve cross-modal contrastive learning (Across Various Patients'Cases), and then input into the classification head to output classification information for subsequent large language models. Of course, it can also be other classification networks, which are not specifically limited.

[0034] It can be seen that in the embodiment of the present application, the multimodal contrastive learning network obtained after pre-training is directly transferred to the supervised learning training stage in combination with the classification network, and it is a modal fusion pipeline without attention-based calculation, which greatly reduces the complexity of effective supervised learning training and training efficiency.

[0035] Finally, the multimodal contrastive learning network UMiCon is connected to the large language network model (LargeLanguage Model, LLM) through the classification network, and then the user sample questions and their corresponding medical sample data can be used for supervised fine-tuning training to obtain the required visual question answering model VQA. As an example, in the end, some large language model frameworks can be used as the initial large language model, connected to the previously trained multimodal block (multimodal contrastive learning network + classification network), and trained through supervised fine-tuning to generate the final answer.

[0036] It should be noted that the above embodiment provides a visual question answering model VQA and a training method. Based on the visual question answering model VQA, there can be many corresponding usage scenarios. In one embodiment, if Figure 6 As shown, a breast image processing method based on artificial intelligence is provided, comprising: S10, obtaining medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports of the breast medical images; Breast medical imaging includes but is not limited to X-rays, CT scans or MRI images, etc., without specific limitation.

[0037] S20, inputting the medical data and the user question into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs an answer corresponding to the user question; Among them, the visual question answering model VQA includes a multimodal contrastive learning network, a classification network and a large language network model. The multimodal contrastive learning network is connected to the large language network model through the classification network. The multimodal contrastive learning network is obtained by multimodal unsupervised learning pre-training based on breast medical sample images and corresponding diagnostic text sample reports. The visual question answering model VQA is obtained by supervised fine-tuning training based on user sample questions and their corresponding medical sample data. For specific training, please refer to the above description, which will not be repeated here.

[0038] It can be seen that in this embodiment, the embodiment of the present application provides a breast image processing method based on artificial intelligence, which can not only improve the accuracy and efficiency of diagnosis, but also play an important role in education, patient communication, telemedicine and scientific research. It has significant clinical value and social significance, and can help doctors quickly understand complex medical imaging data and provide accurate diagnostic suggestions.

[0039] For example, when a doctor uploads a breast X-ray and asks if there is any abnormality, professional advice can be provided through the embodiment of the present application. At the same time, misdiagnosis caused by human factors can be reduced. Considering that doctors need to process a large amount of imaging data every day, in addition, preliminary screening can be performed automatically and suspicious areas can be quickly marked, thereby saving doctors' time and allowing them to focus on dealing with more complex situations.

[0040] Moreover, through the embodiments of the present application, a consistent question-and-answer standard can be provided to reduce subjective differences between different doctors and improve the consistency and reliability of problem solving. In addition, through the embodiments of the present application, it can also be used for medical education and training to help medical students and junior doctors better understand and learn the interpretation methods of breast images. Students can understand the characteristics and diagnostic ideas of different lesions, and a rich case library can be provided for doctors and researchers to conduct in-depth research and analysis and improve clinical skills.

[0041] Finally, it can also help doctors explain imaging results to patients more clearly, enhance patients' understanding and trust. Better communication and explanation can reduce patients' anxiety and worries, and improve the quality of overall medical services and patient satisfaction.

[0042] The method of the embodiment of the present application can also be integrated into a telemedicine platform to help remote areas or medical institutions with limited resources to perform high-quality imaging diagnosis. Doctors can obtain professional diagnostic advice by remotely accessing the system. Through networking and digitization, efficient sharing of medical resources can be achieved, and the operating efficiency of the overall medical system can be improved. In addition, a large amount of imaging data can be collected and analyzed to support clinical research and epidemiological surveys, and help discover new disease characteristics and treatment plans. Through machine learning and big data technology, continuous optimization and improvement can be achieved to improve the accuracy and reliability of diagnosis.

[0043] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0044] In one embodiment, a breast image processing system based on artificial intelligence is provided, and the breast image processing system based on artificial intelligence corresponds one-to-one with the breast image processing method based on artificial intelligence in the above embodiment. Figure 8 As shown, the artificial intelligence-based breast image processing system includes an acquisition module 101 and an output module 102. Each functional module is described in detail as follows: An acquisition module 101 is used to acquire medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports; An output module 102, used for inputting the medical data and the user question into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs the answer to the user question; The visual question answering model VQA is obtained by supervised learning training of the visual question answering model VQA based on user sample questions and their corresponding medical sample data. The visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network, and a large language network model. The multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network. The medical sample data includes breast medical sample images and corresponding diagnostic text sample reports. The multimodal contrast learning network is pre-trained for unsupervised learning based on breast medical sample images and corresponding diagnostic text sample reports. The multimodal contrast learning network includes a single-modal contrast learning branch and a multimodal contrast learning branch.

[0045] In one embodiment, the image encoders of the unimodal contrastive learning branch and the multimodal contrastive learning branch share the same network coefficients, and unsupervised learning pre-training is performed in the following manner: Performing two random masking operations on the breast medical sample image to generate a first mask image and a second mask image of the breast medical sample image, and inputting the first mask image and the second mask image into two image encoders of the single-modality contrast learning branch respectively to perform single-modality contrast learning; The breast medical sample images and their corresponding diagnostic text sample reports are respectively input into the image encoder and the text encoder of the multimodal contrastive learning branch to perform multimodal contrastive learning until the unsupervised learning pre-training of the encoder is completed.

[0046] In this embodiment, in order to better align the modalities of text and image, a novel multimodal contrastive learning network is designed in the embodiment of the present application, and unsupervised learning pre-training is performed using single-modal and multimodal contrastive learning, which bridges the gap between the visual and textual modalities of the samples, and is conducive to subsequent fine-tuning to obtain the final visual question answering model VQA.

[0047] In one embodiment, the breast medical sample images include breast medical sample images graded by the Breast Imaging Reporting and Data System BI-RADS.

[0048] In this embodiment, in order to perform more ideal pre-training of the multimodal contrastive learning network UMiCon, a classification task between BI-RADS graded breast medical sample images can be used to pre-train the multimodal contrastive learning network UMiCon, thereby improving the pre-training effect of the multimodal contrastive learning network UMiCon and facilitating the subsequent achievement of more ideal training results.

[0049] In one embodiment: the classification network includes a feature fusion layer and a classification head, wherein the feature fusion layer includes a Concat module and a Layernorm module connected in sequence.

[0050] It can be seen that in the embodiment of the present application, the multimodal contrastive learning network obtained after pre-training is directly transferred to the supervised learning training stage in combination with the classification network, and it is a modal fusion pipeline without attention-based calculation, which greatly reduces the complexity of effective supervised learning training and training efficiency.

[0051] In one embodiment: the breast medical image samples are breast medical images of the same density classification category, and the image encoder in the multimodal contrast learning network is an image encoder corresponding to the density classification category.

[0052] In this embodiment, the breast medical sample images are breast medical images of the same density classification category, and the image encoder in the multimodal contrastive learning network is the image encoder corresponding to the density classification category, thereby ensuring that the multimodal contrastive learning network is pre-trained in a targeted manner.

[0053] In one embodiment, the medical data and the user question are input into a pre-trained visual question answering model VQA, and the answer to the user question output by the visual question answering model VQA includes: Inputting the breast medical image into a density classifier to identify the density classification category corresponding to the breast medical image; Select a pre-trained visual question answering model VQA corresponding to a density classification category corresponding to the breast medical image; The medical data and the user questions are input into the visual question answering model VQA corresponding to the density classification category to output the answers corresponding to the user questions.

[0054] It can be seen that in this embodiment, a breast image processing system based on artificial intelligence is provided. Developing a VQA system for breast images can not only improve the accuracy and efficiency of diagnosis, but also play an important role in education, patient communication, telemedicine and scientific research, and has significant clinical value and social significance. This system can help doctors quickly understand complex medical imaging data and provide accurate diagnostic suggestions.

[0055] For example, when a doctor uploads a breast X-ray and asks if there are any abnormalities, the system can provide professional opinions based on image analysis. At the same time, the VQA system can also detect tiny lesions and abnormal areas, thereby reducing misdiagnosis caused by human factors. Considering that doctors need to deal with a large amount of imaging data every day, the VQA system can automatically perform preliminary screening and quickly mark suspicious areas, thereby saving doctors' time and allowing them to focus on more complex situations.

[0056] The VQA system can provide consistent diagnostic standards, reduce subjective differences between different doctors, and improve the consistency and reliability of diagnosis. In addition, the VQA system can be used in medical education and training to help medical students and junior doctors better understand and learn how to interpret breast images. By interacting with the system, students can understand the characteristics and diagnostic ideas of different lesions. The system can provide a rich case library for doctors and researchers to conduct in-depth research and analysis to improve clinical skills.

[0057] Finally, the VQA system can help doctors explain imaging results to patients more clearly, by visually displaying the location and extent of lesions, enhancing patients' understanding and trust. Better communication and explanation can reduce patients' anxiety and concerns, and improve the overall quality of medical services and patient satisfaction.

[0058] This system can also be integrated into telemedicine platforms to help remote areas or medical institutions with limited resources to provide high-quality imaging diagnosis. Doctors can access the system remotely to obtain professional diagnostic advice. The VQA system can achieve efficient sharing of medical resources and improve the operating efficiency of the overall medical system. The VQA system can collect and analyze a large amount of imaging data, support clinical research and epidemiological surveys, and help discover new disease characteristics and treatment plans. Through machine learning and big data technology, the VQA system can be continuously optimized and improved to improve the accuracy and reliability of diagnosis.

[0059] For the specific definition of the artificial intelligence-based breast image processing system, please refer to the definition of the artificial intelligence-based breast image processing method above, which will not be repeated here. Each module in the above-mentioned artificial intelligence-based breast image processing system can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0060] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the functions of a breast image processing system based on artificial intelligence, or the steps of a breast image processing method based on artificial intelligence are realized.

[0061] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a breast image processing method based on artificial intelligence in the above embodiment is implemented, which will not be described in detail here. Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of a breast image processing system based on artificial intelligence are implemented, which will not be described in detail here to avoid repetition.

[0062] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, a breast image processing method based on artificial intelligence in the above embodiment is implemented, which is not described in detail here. Alternatively, when the processor executes the computer program, the functions of each module / unit in the embodiment of a breast image processing system based on artificial intelligence are implemented, which is not described in detail here to avoid repetition.

[0063] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A breast image processing method based on artificial intelligence, characterized in that: include: Based on breast medical sample images and corresponding diagnostic text sample reports, unsupervised learning pre-training is performed on a multimodal contrastive learning network, wherein the multimodal contrastive learning network includes a single-modal contrastive learning branch and a multimodal contrastive learning branch; Based on user sample questions and their corresponding medical sample data, supervised learning training is performed on a visual question answering model VQA to obtain a trained visual question answering model VQA, wherein the visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network, and a large language network model, wherein a multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network, and the medical sample data includes breast medical sample images and corresponding diagnostic text sample reports; Obtaining medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports; The medical data and the user questions are input into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs answers corresponding to the user questions.

2. The artificial intelligence-based breast image processing method according to claim 1, characterized in that: The image encoders of the unimodal contrastive learning branch and the multimodal contrastive learning branch share the same network coefficients and are pre-trained for unsupervised learning in the following manner: Performing two random masking operations on the breast medical sample image to generate a first mask image and a second mask image of the breast medical sample image, and inputting the first mask image and the second mask image into two image encoders of the single-modality contrast learning branch respectively to perform single-modality contrast learning; The breast medical sample images and their corresponding diagnostic text sample reports are respectively input into the image encoder and the text encoder of the multimodal contrastive learning branch to perform multimodal contrastive learning until the unsupervised learning pre-training of the encoder is completed.

3. The artificial intelligence-based breast image processing method according to claim 2, characterized in that: The breast medical image sample includes the breast medical image sample graded by the breast imaging report and data system BI-RADS.

4. The artificial intelligence-based breast image processing method according to claim 1, characterized in that: The classification network includes a feature fusion layer and a classification head, wherein the feature fusion layer includes a Concat module and a Layernorm module connected in sequence.

5. The artificial intelligence-based breast image processing method according to claim 1, characterized in that: The breast medical sample images are breast medical sample images of the same density classification category, and the image encoder in the multimodal contrast learning network is an image encoder corresponding to the density classification category.

6. The artificial intelligence-based breast image processing method according to any one of claims 1 to 5, characterized in that: The step of inputting the medical data and the user question into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs an answer to the user question, includes: Inputting the breast medical image into a density classifier to identify the density classification category corresponding to the breast medical image; Selecting a trained visual question answering model VQA corresponding to a density classification category corresponding to the breast medical image; The medical data and the user question are input into the visual question answering model VQA corresponding to the density classification category, so that the corresponding visual question answering model VQA outputs the answer corresponding to the user question.

7. A breast image processing system based on artificial intelligence, characterized in that: include: An acquisition module, used to acquire medical data and corresponding user questions, wherein the medical data includes breast medical images and corresponding diagnostic text reports; An output module, used to input the medical data and the user question into a pre-trained visual question answering model VQA, so that the visual question answering model VQA outputs the answer to the user question; The visual question answering model VQA is obtained by supervised learning training of the visual question answering model VQA based on user sample questions and their corresponding medical sample data. The visual question answering model VQA includes a pre-trained multimodal contrastive learning network, a classification network, and a large language network model. The multimodal contrastive learning branch of the pre-trained multimodal contrastive learning network is connected to the large language network model through the classification network. The medical sample data includes breast medical sample images and corresponding diagnostic text sample reports. The multimodal contrast learning network is pre-trained for unsupervised learning based on breast medical sample images and corresponding diagnostic text sample reports. The multimodal contrast learning network includes a single-modal contrast learning branch and a multimodal contrast learning branch.

8. The artificial intelligence-based breast image processing system according to claim 7, characterized in that: The image encoders of the unimodal contrastive learning branch and the multimodal contrastive learning branch share the same network coefficients and are pre-trained for unsupervised learning in the following manner: Performing two random masking operations on the breast medical sample image to generate a first mask image and a second mask image of the breast medical sample image, and inputting the first mask image and the second mask image into two image encoders of the single-modality contrast learning branch respectively to perform single-modality contrast learning; The breast medical sample images and their corresponding diagnostic text sample reports are respectively input into the image encoder and the text encoder of the multimodal contrastive learning branch to perform multimodal contrastive learning until the unsupervised learning pre-training of the encoder is completed.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the artificial intelligence-based breast image processing method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the artificial intelligence-based breast image processing method according to any one of claims 1 to 6 is implemented.

Images