Mammary gland optical image processing method and device, storage medium and electronic equipment
By processing pathological text and breast optical images into structured multimodal data, combined with questioning data and multimodal model analysis of breast dynamic optical images, the problem of slight differences in breast images is solved, and the diagnosis of breast cancer is improved accuracy.
Patent Information
- Application Number
- CN202411595914.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-11
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In early detection and diagnosis of breast cancer, it is difficult for doctors to fully capture subtle differences in breast images, affecting the accuracy of diagnosis results.
Process the pathological text and breast optical images into structured multimodal data, combine the question data with the pre-trained multimodal model of breast dynamic optical images for analysis, generate breast image analysis data, analyze breast images from multiple directions, and improve Data accuracy and accuracy.
Through the combination of structured multimodal data and questioning data, the accuracy of the description information and feature information of breast images is significantly improved, providing doctors with accurate and comprehensive auxiliary diagnostic data, and improving the accuracy of diagnostic results.
Smart Images

Figure CN119090895A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for processing mammary optical images, a storage medium and an electronic device. Background Art
[0002] Early detection and diagnosis of breast cancer is the key to breast cancer treatment. Doctors need to observe and analyze breast images when diagnosing, relying on their experience and interpretation of breast images. Breast images have a lot of characteristics and changes, and it is difficult for doctors to fully capture these subtle differences with their naked eyes, which can easily affect the doctor's diagnosis. Therefore, how to provide doctors with accurate and comprehensive description information of breast images and improve the accuracy of doctors' diagnostic results has become a problem that needs to be solved urgently. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a breast optical image processing method and device, a storage medium and an electronic device. The present application processes pathological text and breast optical images into structured multimodal data, and then a large multimodal model of breast dynamic optical images combines question data and structured multimodal data to obtain breast image analysis data. The structured multimodal data can be used to analyze breast images from multiple directions, and the analysis can be combined with the question data, thereby improving the precision and accuracy of the obtained data.
[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0005] A method for processing mammary gland optical images, comprising:
[0006] Acquiring pathological text and breast optical images, wherein the breast optical images include breast near-infrared images and breast dynamic near-infrared images;
[0007] Preprocessing the breast optical image to obtain a structured image;
[0008] Evaluating the text weight of the pathology text, and performing dimensionality reduction and text optimization processing on the high-dimensional word meaning space in the pathology text to obtain a structured pathology text;
[0009] Cascadingly fusing the structured image and the structured pathology text to obtain structured multimodal data;
[0010] Acquiring question data, wherein the question data includes image description questions for analyzing breast images;
[0011] The structured multimodal data and the question data are processed using a pre-trained breast dynamic optical imaging multimodal large model to output breast image analysis data, wherein the breast image analysis data includes a derived answer to each image description question in the question data.
[0012] In the above method, optionally, the preprocessing of the mammary optical image to obtain a structured image includes:
[0013] Performing denoising processing on the mammary optical image to obtain a denoised image;
[0014] Performing nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image;
[0015] The brightness and contrast of the denoised image are adjusted, and size scaling and spatial normalization operations are performed to obtain a standardized structured image.
[0016] The above method, optionally, includes a process of training a large multi-modal model of breast dynamic optical imaging, including:
[0017] Loading a two-dimensional model of a large multimodal model of breast dynamic optical imaging and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using two-dimensional near-infrared sample images of breasts, and the two-dimensional model is used to extract features from a two-dimensional image;
[0018] Using each three-dimensional breast dynamic near-infrared sample image to pre-train the three-dimensional model of the breast dynamic optical image multimodal large model until the three-dimensional model is pre-trained;
[0019] Acquire multimodal sample data, wherein the multimodal sample data includes a pathological sample text, a first sample image data, and a second sample image data;
[0020] The multimodal large model of breast dynamic optical imaging is fine-tuned using the multimodal sample data so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the multimodal large model of breast dynamic optical imaging meets preset convergence conditions, thereby completing the training of the multimodal large model of breast dynamic optical imaging.
[0021] The above method, optionally, uses a pre-trained breast dynamic optical imaging multimodal large model to process the structured multimodal data and the question data to output breast imaging analysis data, including:
[0022] Inputting the structured multimodal data into a large multimodal model of breast dynamic optical imaging, and inputting the question data into the large multimodal model of breast dynamic optical imaging;
[0023] The breast dynamic optical imaging multimodal large model extracts image feature data from the structured image of the structured multimodal data, and extracts text feature data from the structured pathology text of the structured multimodal data;
[0024] Generate first-stage analysis content based on the image feature data, the text feature data and the question data, wherein the first-stage analysis content includes screening content for each image description question in the question data;
[0025] The first-stage analysis content and the image feature data are further processed to obtain the second-stage analysis content, which includes the derived answer content of each image description question in the question data, and the second-stage analysis content is determined as breast image analysis data.
[0026] The above method may optionally further include:
[0027] Inputting preset description questions corresponding to the breast optical image into the fine-tuned breast dynamic optical image multimodal large model multiple times;
[0028] Obtaining the answer to the preset description question output by the breast dynamic optical imaging multimodal large model each time;
[0029] Obtaining the answer score of each answer to the question, and sorting the answers to the question based on the answer scores of the answers to the question, to obtain the score sorting results of the answers to the question;
[0030] The scoring ranking results are fed back to the breast dynamic optical imaging multimodal large model, and the breast dynamic optical imaging multimodal large model is adjusted through a back propagation algorithm.
[0031] A mammary gland optical image processing device, comprising:
[0032] A first acquisition unit is used to acquire pathological texts and breast optical images, wherein the breast optical images include breast near-infrared images and breast dynamic near-infrared images;
[0033] A preprocessing unit, used for preprocessing the mammary optical image to obtain a structured image;
[0034] An evaluation unit, used for evaluating the text weight of the pathological text, and performing dimensionality reduction processing and text optimization processing on the high-dimensional word meaning space in the pathological text to obtain a structured pathological text;
[0035] A fusion unit, used for cascading the structured image and the structured pathology text to obtain structured multimodal data;
[0036] A second acquisition unit is used to acquire question data, wherein the question data includes image description questions for analyzing breast images;
[0037] An analysis unit is used to process the structured multimodal data and the question data using a pre-trained breast dynamic optical imaging multimodal large model, and output breast image analysis data, wherein the breast image analysis data includes a derived answer to each image description question in the question data.
[0038] In the above device, optionally, the pre-processing unit comprises:
[0039] A denoising processing module, used for performing denoising on the mammary optical image to obtain a denoised image;
[0040] A calibration module, used for performing nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image;
[0041] The normalization module is used to adjust the brightness and contrast of the denoised image, and perform size scaling and spatial normalization operations to obtain a standardized structured image.
[0042] The above device may optionally further include: a training unit;
[0043] The training unit is used to load a two-dimensional model of a large model of breast dynamic optical imaging multimodal model and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using respective two-dimensional breast near-infrared sample images, and the two-dimensional model is used to extract features in a two-dimensional image; the three-dimensional model of the large model of breast dynamic optical imaging multimodal model is pre-trained using respective three-dimensional breast dynamic near-infrared sample images until the three-dimensional model completes pre-training; multimodal sample data is obtained, wherein the multimodal sample data includes a pathological sample text, a first sample image data, and a second sample image data; and the multimodal sample data is used to fine-tune the large model of breast dynamic optical imaging multimodal model so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the large model of breast dynamic optical imaging multimodal model meets a preset convergence condition, thereby completing the training of the large model of breast dynamic optical imaging multimodal model.
[0044] A storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned mammary optical image processing method.
[0045] An electronic device includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to execute the above-mentioned mammary optical image processing method by one or more processors.
[0046] Compared with the prior art, this application has the following advantages:
[0047] The present application provides a method and device for processing breast optical images, a storage medium, and an electronic device, including: obtaining pathological text and breast optical images; processing the pathological text and breast optical images to obtain structured multimodal data; obtaining question data, the question data containing image description questions for analyzing breast images; using a trained breast dynamic optical image multimodal large model to process the question data and structured multimodal data, and outputting breast image analysis data, the breast image analysis data containing the derived answer to each image description question in the question data. The present application processes the pathological text and breast optical images into structured multimodal data, and then the breast dynamic optical image multimodal large model combines the question data and the structured multimodal data to obtain breast image analysis data. The structured multimodal data can be used to analyze breast images from multiple directions, and the analysis is combined with the question data, thereby improving the precision and accuracy of the obtained data. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0049] Figure 1 A flow chart of a method for processing mammary optical images provided in an embodiment of the present application;
[0050] Figure 2 An example diagram of obtaining structured multimodal data provided in an embodiment of the present application;
[0051] Figure 3 An example diagram of the process of preprocessing mammary optical images to obtain structured images for this application;
[0052] Figure 4 A flowchart of using a large multimodal model of breast dynamic optical imaging to process structured multimodal data and question data and output breast imaging analysis data provided in an embodiment of the present application;
[0053] Figure 5A flowchart of the process of training a large multimodal model of breast dynamic optical imaging provided in an embodiment of the present application;
[0054] Figure 6 An example diagram of a scene for training a large multimodal model of breast dynamic optical imaging provided in an embodiment of the present application;
[0055] Figure 7 An example diagram of fine-tuning a large multi-modal model of breast dynamic optical imaging provided in an embodiment of the present application;
[0056] Figure 8 An example diagram of data processing of a large multimodal model of breast dynamic optical imaging provided in an embodiment of the present application;
[0057] Fig. 9 An example diagram of a reward model provided in an embodiment of the present application;
[0058] Fig.10 A scene example diagram of a method for processing mammary optical images provided for the implementation example of this application;
[0059] Fig.11 An example diagram of a scenario for obtaining a derived answer to an image description question in question data provided by an embodiment of the present application;
[0060] Fig.12 Another structural schematic diagram of a mammary optical image processing method provided in an embodiment of the present application;
[0061] Fig.13 A structural example diagram of a mammary optical image processing device provided in an embodiment of the present application;
[0062] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0065] To solve the above problems, the present application provides a breast optical image processing solution. Using the solution provided by the present application, the pathological text and the breast optical image are processed into structured multimodal data, the question data is obtained, and the structured multimodal data and the question data are processed using the fine-tuned breast dynamic optical image multimodal large model, and then the breast image analysis data is output, which contains the derived answer to each image description question in the question data. The present application processes the data into structured multimodal data, and then uses the breast dynamic optical image multimodal large model to process the structured multimodal data and the question data. It can analyze the breast optical image from multiple aspects, comprehensively obtain the descriptive information and feature information about the breast optical image, and use a multimodal method to obtain the breast image analysis data, which can improve the accuracy of the data. The breast image analysis data can be provided to the doctor later to assist the doctor in diagnosis and improve the accuracy of the doctor's diagnosis results.
[0066] The present application can be applied to a computer-aided diagnosis system of a medical system, which can be constructed using a variety of general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, and distributed computing environments including any of the above devices or devices.
[0067] Reference Figure 1 , is a flow chart of a mammary optical image processing method provided in an embodiment of the present application, and is specifically described as follows:
[0068] S101. Acquire pathology text and breast optical images, where the breast optical images include breast near-infrared images and breast dynamic near-infrared images.
[0069] Pathological text and breast optical images of a subject are obtained, and the subject may be a patient.
[0070] Breast optical images include breast near-infrared images and breast dynamic near-infrared images, wherein the breast near-infrared images can be represented as NIR images, which are 2D images; the breast dynamic near-infrared images can be represented as DNIR images, which are 3D images and time series images.
[0071] The pathology text includes, but is not limited to, information such as the subject's age, cup size, and ultrasound grade.
[0072] S102: Preprocess the breast optical image to obtain a structured image.
[0073] S103, evaluating the text weight of the pathology text, and performing dimensionality reduction and text optimization processing on the high-dimensional word meaning space in the pathology text to obtain a structured pathology text.
[0074] S104, cascadingly fusing the structured image and the structured pathological text to obtain structured multimodal data.
[0075] In the present application, obtaining structured multimodal data requires processing breast optical images to obtain structured images, processing pathological texts to obtain structured pathological texts, and then fusing the structured images and structured pathological texts to obtain structured multimodal data.
[0076] Reference Figure 2 , is an example diagram of obtaining structured multimodal data provided in an embodiment of the present application, and the relevant description is as follows:
[0077] The NIR images and DNIR images are denoised, calibrated and standardized, and the pathological texts are weighted and dimensionally reduced to obtain structured multimodal data.
[0078] Reference Figure 3 , which is a flowchart of preprocessing mammary optical images to obtain structured images in this application, and the relevant instructions are as follows:
[0079] S201, performing denoising processing on the breast optical image to obtain a denoised image.
[0080] Denoising the image can reduce the impact of noise on image quality and feature analysis. Preferably, a filter-based denoising method is used to denoise the breast optical image to obtain a denoised image. The filter-based denoising method effectively reduces Gaussian noise in the image and retains the details and edge information of the breast image.
[0081] S202: Perform nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image.
[0082] The mammary gland image in the denoised image is subjected to nonlinear grayscale transformation to calibrate the denoised image and obtain a calibrated image. The effect of calibration can eliminate systematic errors and ensure the accuracy and comparability between different images.
[0083] S203, adjusting the brightness and contrast of the denoised image, and performing size scaling and spatial normalization operations to obtain a standardized structured image.
[0084] Adjust the brightness and contrast of the denoised image, and perform size scaling and spatial normalization on the adjusted image to obtain a standardized structured image. The standardized image can eliminate the differences between different images and make them comparable and consistent.
[0085] The present application performs a series of processing on the breast optical images to improve the integrity and accuracy of the data and facilitate further analysis of the breast optical images.
[0086] When processing pathological texts, natural language processing techniques are used to evaluate the importance of medical terms in pathological texts and express text weights through the term frequency-inverse text frequency (TF-IDF) method. PCA is used to reduce the dimensionality of the high-dimensional word space and perform text optimization. The text optimization process includes converting synonyms in pathological texts and performing optimization operations such as ambiguity elimination to obtain structured pathological texts.
[0087] After obtaining the structured pathology text and structured images, the structured pathology text and structured images are cascaded and fused to fully explore the correlation information between multimodal data and improve the representation ability of comprehensive data.
[0088] S105: Obtain question data, where the question data includes image description questions used for analyzing breast images.
[0089] For example, image description problems may include: whether the image is clear, where the lesion is located in the image, what information is available to determine the lesion, etc. Image description problems may be viewed as medical problems or medical problems.
[0090] The question data can be regarded as a prompt project in the large multimodal model of breast dynamic optical imaging. The question data can be used to assist the large multimodal model of breast dynamic optical imaging to interpret breast images by deriving answers to image description questions.
[0091] S106. Use the pre-trained breast dynamic optical imaging multimodal large model to process the structured multimodal data and the question data, and output breast imaging analysis data, where the breast imaging analysis data includes the derived answers to each image description question in the question data.
[0092] Reference Figure 4 , which is a flowchart of using a large multimodal model of breast dynamic optical imaging to process structured multimodal data and question data and output breast imaging analysis data provided by an embodiment of the present application, and is specifically described as follows:
[0093] S301, inputting structured multimodal data into a large breast dynamic optical imaging multimodal model, and inputting question data into the large breast dynamic optical imaging multimodal model.
[0094] Preferably, the question data can be the prompt project constructed in this application, and the question data serves as a supplementary prompt part of the model input to further improve the reasoning ability of the model and the quality of the output data.
[0095] S302. The breast dynamic optical imaging multimodal large model extracts image feature data from the structured images of the structured multimodal data, and extracts text feature data from the structured pathology text of the structured multimodal data.
[0096] The extracted image feature data can also be called visual feature data.
[0097] Preferably, a visual feature extraction algorithm is used to extract corresponding first visual information from a 2D image in a structured image, and further, the first visual information is encoded data processed by a 2D image encoder; and corresponding second visual information is extracted from a 3D image in the structured image, and the second visual information is encoded information processed by a 3D image encoder, and visual feature data is obtained based on the first visual information and the second visual information.
[0098] S303: Generate first-stage analysis content based on the image feature data, text feature data and question data, where the first-stage analysis content includes screening content for each image description question in the question data.
[0099] S304. Further process the first-stage analysis content and the image feature data to obtain the second-stage analysis content, which includes the derived answer content for each image description question in the question data, and determine the second-stage analysis content as breast image analysis data.
[0100] It should be noted that the processing of the multimodal large model of breast dynamic optical imaging has two stages. The first stage is the basic principle generation stage, which will generate the first stage analysis content. In this stage, the model generates preliminary screening results based on the input data, and the final results can be regarded as preliminary results; the second stage is the answer reasoning stage, in this stage, the model further infers and optimizes the preliminary results, and finally obtains the second stage analysis content.
[0101] Preferably, the second stage analysis content can also be understood as the image analysis content of the breast optical image, and the second stage analysis content is used to assist doctors in understanding the patient's breast condition and making further judgments on the patient's breast.
[0102] In addition, the second-stage analysis content may include the relevant description of the patient's breast condition output by the multimodal large model of breast dynamic optical imaging, such as description information about breast lumps and breast tissues. The accuracy of the data of breast image analysis obtained by the scheme provided by the present application is high, thereby improving the accuracy of the relevant description of the breast condition, such as improving the accuracy of the description of breast lumps and breast tissues, providing doctors with high-accuracy auxiliary diagnosis data to assist doctors in diagnosing patients.
[0103] In the method provided in the embodiment of the present application, pathological text and breast optical images are obtained; the pathological text and breast optical images are processed to obtain structured multimodal data; question data are obtained, and the question data contains image description problems used to analyze breast images; the question data and structured multimodal data are processed using the trained breast dynamic optical image multimodal large model, and breast image analysis data is output, and the breast image analysis data contains the derived answers to each image description problem in the question data. The present application processes the pathological text and breast optical images into structured multimodal data, and then the breast dynamic optical image multimodal large model combines the question data and the structured multimodal data to obtain the breast image analysis data. The structured multimodal data can be used to analyze breast images from multiple directions, and the analysis is combined with the question data, thereby improving the precision and accuracy of the obtained data. Improve the accuracy of breast descriptions and provide doctors with accurate and comprehensive breast-related data.
[0104] Reference Figure 5 , which is a flow chart of the process of training a large multi-modal model of breast dynamic optical imaging provided in an embodiment of the present application, and is specifically described as follows:
[0105] S401, loading a two-dimensional model of a large multimodal model of breast dynamic optical imaging and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using two-dimensional breast near-infrared sample images, and the two-dimensional model is used to extract features from two-dimensional images.
[0106] S402, using each three-dimensional breast dynamic near-infrared sample image to pre-train the three-dimensional model of the breast dynamic optical image multimodal large model until the three-dimensional model is pre-trained.
[0107] S403: Acquire multimodal sample data, where the multimodal sample data includes pathological sample text, first sample image data, and second sample image data.
[0108] Preferably, the first sample image data includes a two-dimensional breast near-infrared sample image; and the second sample image data includes a three-dimensional breast dynamic near-infrared sample image.
[0109] S404. Use the multimodal sample data to train the large multimodal model of breast dynamic optical imaging so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the large multimodal model of breast dynamic optical imaging meets the preset convergence conditions, thereby completing the training of the large multimodal model of breast dynamic optical imaging.
[0110] The multimodal model of breast dynamic optical imaging is based on the PaliGemma model and adds a processing method for three-dimensional images.
[0111] Among them, the two-dimensional model is the basis of the PaliGemma large model, and the three-dimensional image processing method is implemented on the three-dimensional model.
[0112] Reference Figure 6 , is a scene example diagram for training a large multimodal model of dynamic optical imaging of breast provided in an embodiment of the present application, which includes a data processing example and a fine-tuning example of a large multimodal model of dynamic optical imaging of breast. The process of training a large multimodal model of dynamic optical imaging of breast includes training a three-dimensional module and then performing fine-tuning. It should also be noted that the figure also shows the general architecture of the large multimodal model of dynamic optical imaging of breast, such as the multi-head self-attention layer, the multilayer perceptron (MLP) layer and the LLM layer in the figure.
[0113] The specific process is as follows: DOBI multimodal large model loads two-dimensional weights (i.e., PaliGemma large model weights). The PaliGemma large model has been pre-trained on a large-scale two-dimensional image dataset and has strong feature extraction and pattern recognition capabilities, but lacks the ability to diagnose medical diseases and breast cancer. The two-dimensional image dataset contains multiple breast near-infrared sample images. After loading its weights, the two-dimensional model is frozen and the model is independently pre-trained using a collated open source general three-dimensional dataset, so that the model has preliminary feature extraction capabilities for three-dimensional images. The three-dimensional dataset contains multiple breast dynamic near-infrared sample images.
[0114] After completing the three-dimensional pre-training independently, the DOBI multimodal large model is adjusted using fine-tuning technology. Specifically, some two-dimensional models can be unfrozen during the fine-tuning process, allowing the unfrozen two-dimensional models to be adjusted together with the three-dimensional models to ensure that they can work together and adapt to the diagnosis of breast cancer. Subsequently, a series of hyperparameters are set through iterative learning based on the characteristics of DOBI multimodal data, including learning rate, training rounds, and batch size. At the same time, in order to better adapt to DOBI imaging data, it is also necessary to select appropriate loss functions and optimizers to ensure that the model can effectively adjust weights during training, thereby improving the accuracy of diagnosis and screening. The training of the DOBI multimodal large model is terminated until the preset convergence conditions are met. The convergence condition can reach a preset number of iterations or training times.
[0115] It should be noted that in order to improve the accuracy of the data obtained, the breast dynamic optical imaging multimodal large model is fine-tuned. The fine-tuning process is as follows: the model weight parameters in the trained breast dynamic optical imaging multimodal large model are converted from floating point numbers to integers, and the LoRA fine-tuning technology is used to fine-tune the breast dynamic optical imaging multimodal large model after the model weight parameters are converted.
[0116] For example, the process of fine-tuning the large multimodal model of breast dynamic optical imaging is as follows: applying QLoRA parameter efficient fine-tuning technology (Quantized Low-Rank Adaptation) to fine-tune the large multimodal model of breast dynamic optical imaging, while reducing computational costs and memory usage, while maintaining the performance of the large multimodal model of breast dynamic optical imaging.
[0117] Furthermore, in order to make the breast dynamic optical imaging multimodal large model analyze breast images more accurately, the general base large model in the breast dynamic optical imaging multimodal large model can be fine-tuned to obtain a fine-tuned breast dynamic optical imaging multimodal large model. Figure 7 , is an example diagram of fine-tuning the breast dynamic optical imaging multimodal large model provided in an embodiment of the present application. As shown in the figure, after the base large model in the breast dynamic optical imaging multimodal large model is iteratively learned for hyperparameters, the QLoRA parameter efficient fine-tuning technology is used to fine-tune the breast dynamic optical imaging multimodal large model, including 4-bit quantization, low-order decomposition, and double quantization, to obtain the fine-tuned breast dynamic optical imaging multimodal large model, which can also be called the DOBI multimodal large model.
[0118] It should be noted that Figure 6 The figure also contains the fine-tuning diagram content. The LoRA in the figure represents the LoRA fine-tuning technology.
[0119] Combination Figure 6 and Figure 7 The fine-tuning process is explained. First, by combining 4-bit quantization technology, the model weight parameters in the breast dynamic optical imaging multimodal large model are converted from the original 32-bit floating point numbers to 4-bit integers, thereby reducing memory space and computing resources. Then, the LoRA (Low-Rank Adaptation) technology is used to fine-tune the breast dynamic optical imaging multimodal large model, adjust some model weights in the breast dynamic optical imaging multimodal large model, and simulate the effect of full fine-tuning. For pre-trained weights , LoRA limits its update method and fine-tunes the incremental matrix of all parameters It is represented as two low-rank approximation matrices A and B with smaller parameters, such that: ;
[0120] in Both are weight matrices for LoRA low-rank adaptation, and r is much smaller than d.
[0121] At this time, the fine-tuning parameter amount is changed from the original The d*d of matrix B is converted to r*d of matrix B and A. At the beginning of training, matrix B is initialized by Gaussian function, and matrix A is initialized with all 0s, so that BA=0 at the beginning of training, and fine-tuning can start from the pre-trained weight W0. During training, the parameters of the pre-trained model are fixed, and only matrix A and matrix B are trained. After training, the B matrix is multiplied by the A matrix and then merged with the pre-trained model parameters as the fine-tuned model parameters. LoRA fine-tuning technology not only realizes efficient parameter adjustment to meet the needs of DOBI tasks, but also significantly reduces computational complexity, memory usage and resource requirements.
[0122] The structured multimodal data and the question data are input into the fine-tuned breast dynamic optical imaging multimodal large model, so that the breast dynamic optical imaging multimodal large model processes the structured multimodal data and the question data, thereby outputting breast imaging analysis data.
[0123] The fine-tuned model can better process DOBI multimodal data and provide accurate and reliable medical diagnosis and screening results. This approach not only improves the performance of the model, but also makes it more practical in resource-constrained environments and has broad application prospects.
[0124] Reference Figure 8, which is an example diagram of data processing of the multimodal large model of dynamic optical imaging of breast provided in the embodiment of the present application, and the relevant description is as follows: The multimodal large model of dynamic optical imaging of breast, based on the PaliGemma large model, also proposes a processing method for three-dimensional images. First, for the input DNIR image, the series of images are input into the 3D image encoder for encoding, and then, in order to solve the problem of high similarity of encoding of adjacent time series images, a self-learning encoding position mapping method is proposed. The NIR image is input into the 2D image encoder for encoding, and then linear mapping processing is performed; the 2D image data after linear mapping processing, the self-learning position mapping image data and the input text are serialized, and then processed using the Gemma model, and finally the breast image analysis data containing the question answer text and the related description text of the breast is output. It should be noted that when processing three-dimensional images, an attention mechanism is introduced to learn the correlation between different positions. By learning the attention weight, the DOBI multimodal large model can dynamically adjust the weight of the position encoding in each time step. In the process of training the large multimodal model of breast dynamic optical imaging, the weight parameters including position encoding are optimized by minimizing the loss function. Through continuous iteration, the DOBI large multimodal model can learn the correlation between different positions, thereby improving the performance and generalization ability of the model when processing sequence data. Through dynamic adjustment, the model can effectively capture the dependencies between different positions in the sequence and better understand the structure and characteristics of the sequence data. At the same time, in addition to accepting multimodal data input, the DOBI large multimodal model can still parse and encode the input single modality data in the absence of modality, and pass it to the model for application processing through its own mapping method.
[0125] Furthermore, in order to improve the performance of the large multimodal model of breast dynamic optical imaging, the present application can continuously optimize the model.
[0126] The optimization process includes: inputting preset description questions corresponding to breast optical imaging into the fine-tuned breast dynamic optical imaging multimodal large model multiple times; obtaining the question responses output by the breast dynamic optical imaging multimodal large model for each preset description question; obtaining the answer score for each question response, and sorting each question response based on the answer score of each question response to obtain the score sorting result of each question response; feeding back the score sorting result to the breast dynamic optical imaging multimodal large model, and adjusting the breast dynamic optical imaging multimodal large model through the back propagation algorithm.
[0127] By repeating the same question several times and passing it to the fine-tuned breast imaging multimodal large model, a set of different answers is generated. These answers are then evaluated by professional physicians. Physicians score and rank each answer based on its accuracy and clinical relevance. The scoring criteria can include multiple dimensions such as the correctness of the diagnosis, the completeness of the information, and the practicality of clinical application. Then, based on the scoring and ranking results given by the physician, this information is fed back to the DOBI multimodal large model and trained through the back-propagation algorithm. This process enables the DOBI multimodal large model to obtain the pros and cons of its own responses from the reward model, so as to make more reasonable and accurate results in future reasoning processes.
[0128] Furthermore, the optimization process of the multimodal large model of breast dynamic optical imaging can be constructed as a reward model, which can be used as a core component in the reinforcement learning process. Fig. 9 , which is an example diagram of the reward model provided in an embodiment of the present application.
[0129] In order to further optimize the DOBI multimodal model, the reward model uses the Proximal Policy Optimization (PPO) algorithm, which has strong stability and efficiency. In this stage, an objective function is set, which specifically includes indicators such as the accuracy and efficiency of the model in the film reading task. Through the stochastic gradient descent optimization strategy, the objective function is optimized and the model parameters are continuously adjusted to achieve higher performance.
[0130] The entire process goes through multiple rounds of feedback and iterative learning, allowing the model to continuously improve its intelligent reading results. Each round of feedback uses a reinforcement learning mechanism to enable the model to learn more accurate and reliable diagnostic patterns from the physician's scores, thereby gradually improving its output results. This method not only improves the diagnostic accuracy and reliability of the model, but also provides an efficient optimization path for the development of intelligent medical diagnosis systems.
[0131] In addition, by combining the feedback mechanism of reinforcement learning and advanced optimization algorithms, the present invention can be flexibly applied in different clinical scenarios and adapt to various complex medical question-answering tasks. Ultimately, after multiple iterations and optimizations, the model will have a higher intelligent reading capability, providing a powerful auxiliary tool for clinicians and significantly improving the efficiency and accuracy of breast imaging diagnosis.
[0132] Reference Fig.10 , which is a scene example diagram of a method for processing mammary optical images provided for the implementation example of this application, and the relevant descriptions are as follows:
[0133] First, NIR and DNIR images are input as image modality data, and denoising, calibration and DOBI standardization operations are performed on them to obtain structured DOBI image data. At the same time, the patient's pathological information, including his age, cup size, ultrasound grade and other pathological text information, is evaluated for its text weight using the term frequency-inverse text frequency (TF-IDF) method, and PCA is used to reduce the dimensionality of the high-dimensional word space and solve problems such as synonym conversion and ambiguity elimination to obtain structured pathological text.
[0134] The obtained structured multimodal data is input into the DOBI multimodal large model. At this time, the DOBI multimodal large model has been pre-trained in three dimensions using the three-dimensional general image dataset, and fine-tuned through QLoRA using the DOBI multimodal data.
[0135] The input text content is divided into pathological text and question data. The pathological text contains information such as the patient's age, cup size, and ultrasound grade, while the question data contains several questions that the DOBI multimodal model is expected to answer. For example, the input text content is "The above is DOBI multimodal imaging data. The patient is 35 years old, cup size is B, and BIRADS ultrasound grade is 4B. Is the patient's imaging clear? Where is the lesion located? Please output the analysis data of the benign and malignant tumors." The content is divided into pathological text and question data, and then combined with prompt data to assist in reasoning, using the CoT principle, the question data is further refined, for example: subdivided into "Is the imaging clear?", "Where is the lesion located?" and "Output the benign and malignant tumor analysis data" and other sub-questions (equivalent to the image description problem above).
[0136] Based on the above multiple sub-questions, the complex diagnosis and screening tasks are broken down into several easy-to-handle sub-tasks. The model will integrate pathological information and question information, and analyze from multiple angles such as imaging clarity and lesion location, thereby improving the accuracy and reliability of the overall reasoning. After obtaining the answers to multiple sub-questions, these answers are spliced to form an image analysis data that includes multiple aspects such as imaging clarity and lesion location. The image analysis data can also be regarded as an image analysis report. This splicing technology not only improves the coherence and completeness of the response, but also ensures that the details of each step can be fully utilized to form a more comprehensive diagnosis and screening results.
[0137] Reference Fig.11 , which is an example diagram of a scenario for obtaining a derived answer to an image description question in question data provided in an embodiment of the present application.
[0138] After the structured multimodal data and question data are input into the large multimodal model of breast dynamic optical imaging, the visual features are extracted and fused with the encoded text representation. Specifically, an advanced visual feature extraction algorithm is used to extract important visual information from 2D NIR image data and 3D DNIR image data, and then combine it with the encoded text representation. This fusion method can fully utilize the advantages of multimodal data and enhance the model's understanding and reasoning capabilities.
[0139] Next, the fused multimodal representation is fed back to the decoder for processing. In order to further improve the output effect of the model, the present invention combines the two modalities of pathological text data and DOBI image data into a two-stage framework. Among them, the first stage is the basic principle generation stage, in which the model generates preliminary screening results describing the problem based on the input data. The second stage is the answer reasoning stage, in which the model further infers and optimizes the preliminary screening results, and comprehensively considers the above-mentioned imaging clarity, lesion location, disease trend and other contents, and finally contains imaging analysis data of imaging clarity, lesion location, disease trend, breast description information and other contents.
[0140] Through this multimodal CoT method, the present invention effectively reduces the hallucination output of the DOBI large model, that is, reduces the possibility of the model generating inaccurate or unreasonable content, thereby significantly improving the output effect of the model. This method not only enhances the accuracy and reliability of the model in dealing with complex medical problems, but also provides new ideas and methods for the fusion and utilization of multimodal data.
[0141] Reference Fig.12 , is another structural schematic diagram of a breast optical image processing method provided in an embodiment of the present application.
[0142] The 2D image feature extraction module in the figure is the two-dimensional model mentioned above, the 3D image feature extraction module is the three-dimensional model mentioned above, the prompt project is the question data mentioned above, and the intelligent report is the breast image analysis data mentioned above.
[0143] This application first cleans and normalizes, structures and integrates various types of near-infrared (NIR) images, dynamic near-infrared (DNIR) images, and pathological text data; proposes a processing method for three-dimensional input images, and introduces self-learning position encoding mapping to enable the large model to accurately obtain DNIR three-dimensional image features containing time series; then, combined with the characteristics of multimodal input data, the pre-trained model is quantized to 4 bits, and the structure and parameters of the base model are fine-tuned through low-rank decomposition, so that the model can more accurately parse various types of information contained in DOBI images and adapt to downstream tasks of DOBI multimodal data input; finally, combined with prompt engineering technologies such as multimodal chain of thought (CoT) and contextual learning, complex problems are decomposed into multiple steps, and text and images are integrated into a framework to guide the large model to process key medical features and knowledge points in breast images and guide the model to reason, thereby realizing an intelligent reading method for automatically generating detailed breast cancer screening and diagnosis reports. At the same time, the intelligent reading results obtained by this method are evaluated by professional doctors, using reinforcement learning methods such as the proximal policy optimization (PPO) algorithm, to guide the large model to learn the optimal behavior strategy through reward signals, and to score, quantify and correct the model output. The model is also fed back to achieve iterative learning and model parameter updates, and to continuously improve the intelligent reading results.
[0144] Based on the PaliGemma model, this application introduces a self-learning position coding mapping method for time series three-dimensional image data. By learning the attention weight, the model can dynamically adjust the weight and position of each time series position coding, learn the correlation of different time series position coding, and adaptively learn to find the best coding method. DOBI multimodal data is classified, and targeted normalization and structured methods are proposed for different types of data to improve the integrity and accuracy of the data, fully understand the key details in the image or correctly associate it with relevant text information, thereby affecting the accuracy and reliability of diagnosis, thereby improving the accuracy and reliability of breast cancer detection and diagnosis. The image and text modality are merged into the same framework, and the problem reasoning and result generation are divided into two stages to assist the model in understanding and analyzing the image, fully explore the information in the DOBI image, and comprehensively capture the subtle differences in the dynamic optical image of the gland, guiding the model learning and reasoning process. The present invention introduces breast dynamic optical imaging, including single image NIR and time series image DNIR, which can be analyzed in the early stage of tumor growth based on tumor neovascularization to obtain relevant description data of breast tumors.
[0145] The present invention provides a mammary gland optical image processing device, which is used to support Figure 1The specific implementation of the method shown is that the device is set in a processor.
[0146] Reference Fig.13 , is a structural example diagram of a mammary optical image processing device provided in an embodiment of the present application, and is specifically described as follows:
[0147] A first acquisition unit 501 is used to acquire pathological texts and breast optical images, wherein the breast optical images include breast near-infrared images and breast dynamic near-infrared images;
[0148] A preprocessing unit 502, configured to preprocess the breast optical image to obtain a structured image;
[0149] An evaluation unit 503 is used to evaluate the text weight of the pathology text, and perform dimensionality reduction and text optimization processing on the high-dimensional word space in the pathology text to obtain a structured pathology text;
[0150] A fusion unit 504 is used to perform cascade fusion on the structured image and the structured pathology text to obtain structured multimodal data;
[0151] A second acquisition unit 505 is used to acquire question data, wherein the question data includes image description questions for analyzing breast images;
[0152] The analysis unit 506 is used to process the structured multimodal data and the question data using a pre-trained breast dynamic optical imaging multimodal large model, and output breast image analysis data, wherein the breast image analysis data includes a derived answer to each image description question in the question data.
[0153] In the device provided in the embodiment of the present application, pathological text and breast optical images are obtained; the pathological text and breast optical images are processed to obtain structured multimodal data; question data are obtained, and the question data contains image description questions for analyzing breast images; the question data and structured multimodal data are processed using a trained breast dynamic optical image large multimodal model, and breast image analysis data is output, and the breast image analysis data contains the derived answers to each image description question in the question data. The present application processes the pathological text and breast optical images into structured multimodal data, and then the breast dynamic optical image large multimodal model combines the question data and the structured multimodal data to obtain breast image analysis data. The structured multimodal data can be used to analyze breast images from multiple directions, and the analysis is combined with the question data, thereby improving the precision and accuracy of the obtained data.
[0154] In another embodiment provided by the present application, the preprocessing unit of the device includes:
[0155] A denoising processing module, used for performing denoising on the mammary optical image to obtain a denoised image;
[0156] A calibration module, used for performing nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image;
[0157] The normalization module is used to adjust the brightness and contrast of the denoised image, and perform size scaling and spatial normalization operations to obtain a standardized structured image.
[0158] In another embodiment provided by the present application, the device further includes: a training unit;
[0159] The training unit is used to perform the following operations:
[0160] Loading a two-dimensional model of a large multimodal model of breast dynamic optical imaging and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using two-dimensional near-infrared sample images of breasts, and the two-dimensional model is used to extract features from a two-dimensional image;
[0161] Using each three-dimensional breast dynamic near-infrared sample image to pre-train the three-dimensional model of the breast dynamic optical image multimodal large model until the three-dimensional model is pre-trained;
[0162] Acquire multimodal sample data, wherein the multimodal sample data includes a pathological sample text, a first sample image data, and a second sample image data;
[0163] The multimodal large model of breast dynamic optical imaging is fine-tuned using the multimodal sample data so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the multimodal large model of breast dynamic optical imaging meets preset convergence conditions, thereby completing the training of the multimodal large model of breast dynamic optical imaging.
[0164] In another embodiment provided by the present application, the analysis unit of the device includes:
[0165] An input module, used for inputting the structured multimodal data into a large multimodal model of breast dynamic optical imaging, and inputting the question data into the large multimodal model of breast dynamic optical imaging;
[0166] An extraction module, used for the breast dynamic optical imaging multimodal large model to extract image feature data from the structured image of the structured multimodal data, and to extract text feature data from the structured pathology text of the structured multimodal data;
[0167] A first generating module, configured to generate first-stage analysis content based on the image feature data, the text feature data and the question data, wherein the first-stage analysis content includes screening content for each image description question in the question data;
[0168] The second generation module is used to further process the first-stage analysis content and the image feature data to obtain the second-stage analysis content, which includes the derived answer content of each image description question in the question data, and determines the second-stage analysis content as breast image analysis data.
[0169] In another embodiment provided by the present application, the device further includes: an adjustment unit;
[0170] The adjustment unit is used to perform the following operations:
[0171] Inputting preset description questions corresponding to the breast optical image into the fine-tuned breast dynamic optical image multimodal large model multiple times;
[0172] Obtaining the answer to the preset description question output by the breast dynamic optical imaging multimodal large model each time;
[0173] Obtaining the answer score of each answer to the question, and sorting the answers to the question based on the answer scores of the answers to the question, to obtain the score sorting results of the answers to the question;
[0174] The scoring ranking results are fed back to the breast dynamic optical imaging multimodal large model, and the breast dynamic optical imaging multimodal large model is adjusted through a back propagation algorithm.
[0175] Although the present invention depicts operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.
[0176] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.
[0177] An embodiment of the present invention further provides a storage medium, wherein the storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned mammary optical image processing method.
[0178] The embodiment of the present invention further provides an electronic device, the structural diagram of which is shown in FIG. Fig.14As shown, it specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to execute the one or more instructions 602 to perform the above-mentioned breast optical image processing method.
[0179] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in relevant regions. The specific implementation process of each of the above embodiments and their derivative methods are all within the scope of protection of the present invention.
[0180] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0181] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0182] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for processing mammary gland optical images, characterized in that: include: Acquiring pathological text and breast optical images, wherein the breast optical images include breast near-infrared images and breast dynamic near-infrared images; Preprocessing the breast optical image to obtain a structured image; Evaluating the text weight of the pathology text, and performing dimensionality reduction and text optimization processing on the high-dimensional word meaning space in the pathology text to obtain a structured pathology text; Cascadingly fusing the structured image and the structured pathology text to obtain structured multimodal data; Acquiring question data, wherein the question data includes image description questions for analyzing breast images; The structured multimodal data and the question data are processed using a pre-trained breast dynamic optical imaging multimodal large model to output breast image analysis data, wherein the breast image analysis data includes a derived answer to each image description question in the question data.
2. The method according to claim 1, characterized in that The preprocessing of the mammary optical image to obtain a structured image includes: Performing denoising processing on the mammary optical image to obtain a denoised image; Performing nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image; The brightness and contrast of the denoised image are adjusted, and size scaling and spatial normalization operations are performed to obtain a standardized structured image.
3. The method according to claim 1, characterized in that The process of training a large multimodal model of breast dynamic optical imaging includes: Loading a two-dimensional model of a large multimodal model of breast dynamic optical imaging and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using two-dimensional near-infrared sample images of breasts, and the two-dimensional model is used to extract features from a two-dimensional image; Using each three-dimensional breast dynamic near-infrared sample image to pre-train the three-dimensional model of the breast dynamic optical image multimodal large model until the three-dimensional model is pre-trained; Acquire multimodal sample data, wherein the multimodal sample data includes a pathological sample text, a first sample image data, and a second sample image data; The multimodal large model of breast dynamic optical imaging is fine-tuned using the multimodal sample data so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the multimodal large model of breast dynamic optical imaging meets preset convergence conditions, thereby completing the training of the multimodal large model of breast dynamic optical imaging.
4. The method according to claim 1, characterized in that: The method uses the pre-trained breast dynamic optical imaging multimodal large model to process the structured multimodal data and the question data, and outputs breast imaging analysis data, including: Inputting the structured multimodal data into a large multimodal model of breast dynamic optical imaging, and inputting the question data into the large multimodal model of breast dynamic optical imaging; The breast dynamic optical imaging multimodal large model extracts image feature data from the structured image of the structured multimodal data, and extracts text feature data from the structured pathology text of the structured multimodal data; Generate first-stage analysis content based on the image feature data, the text feature data and the question data, wherein the first-stage analysis content includes screening content for each image description question in the question data; The first-stage analysis content and the image feature data are further processed to obtain the second-stage analysis content, which includes the derived answer content of each image description question in the question data, and the second-stage analysis content is determined as breast image analysis data.
5. The method according to claim 1, characterized in that Also includes: Inputting preset description questions corresponding to the breast optical image into the fine-tuned breast dynamic optical image multimodal large model multiple times; Obtaining the answer to the preset description question output by the breast dynamic optical imaging multimodal large model each time; Obtaining the answer score of each answer to the question, and sorting the answers to the question based on the answer scores of the answers to the question, to obtain the score sorting results of the answers to the question; The scoring ranking results are fed back to the breast dynamic optical imaging multimodal large model, and the breast dynamic optical imaging multimodal large model is adjusted through a back propagation algorithm.
6. A mammary gland optical image processing device, characterized in that: include: A first acquisition unit is used to acquire pathological texts and breast optical images, wherein the breast optical images include breast near-infrared images and breast dynamic near-infrared images; A preprocessing unit, used for preprocessing the mammary optical image to obtain a structured image; An evaluation unit, used for evaluating the text weight of the pathological text, and performing dimensionality reduction processing and text optimization processing on the high-dimensional word meaning space in the pathological text to obtain a structured pathological text; A fusion unit, used for cascading the structured image and the structured pathology text to obtain structured multimodal data; A second acquisition unit is used to acquire question data, wherein the question data includes image description questions for analyzing breast images; An analysis unit is used to process the structured multimodal data and the question data using a pre-trained breast dynamic optical imaging multimodal large model, and output breast image analysis data, wherein the breast image analysis data includes a derived answer to each image description question in the question data.
7. The device according to claim 6, characterized in that The pre-processing unit comprises: A denoising processing module, used for performing denoising on the mammary optical image to obtain a denoised image; A calibration module, used for performing nonlinear grayscale transformation on the breast image in the denoised image to calibrate the denoised image to obtain a calibrated image; The normalization module is used to adjust the brightness and contrast of the denoised image, and perform size scaling and spatial normalization operations to obtain a standardized structured image.
8. The device according to claim 6, characterized in that Also includes: Training unit; The training unit is used to load a two-dimensional model of a large model of breast dynamic optical imaging multimodal model and a model weight of the two-dimensional model, wherein the two-dimensional model is a model pre-trained using respective two-dimensional breast near-infrared sample images, and the two-dimensional model is used to extract features in a two-dimensional image; the three-dimensional model of the large model of breast dynamic optical imaging multimodal model is pre-trained using respective three-dimensional breast dynamic near-infrared sample images until the three-dimensional model completes pre-training; multimodal sample data is obtained, wherein the multimodal sample data includes a pathological sample text, a first sample image data, and a second sample image data; and the multimodal sample data is used to fine-tune the large model of breast dynamic optical imaging multimodal model so as to simultaneously adjust the model weights of the two-dimensional model and the three-dimensional model until the large model of breast dynamic optical imaging multimodal model meets a preset convergence condition, thereby completing the training of the large model of breast dynamic optical imaging multimodal model.
9. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the mammary optical image processing method according to any one of claims 1 to 5.
10. An electronic device, characterized in that: The system comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to execute the breast optical image processing method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Pathological classification method and system based on multi-modal deep learning
CN110298383A
Intelligent diagnosis and evaluation method based on mammary gland structured report
CN113159134A
Visual question and answer method for medical image diagnosis
CN114201592A
Intelligent method for rapid screening in early stage of mammary tissue sclerosis
CN117481672A
Breast cancer molecular typing prediction method under guidance of text information
CN117995386A
Cited By
Medical diagnosis system based on chain reasoning and multi-modal fusion and application thereof
CN119361138A
