Medical image intelligent analysis model and training method

Abstract

The invention relates to medical image intelligent analysis, in particular to a medical image intelligent analysis model, a training method and a visual feature encoder, which are used for performing medical image feature extraction on an input medical image and inputting the medical image features into a memory activation selector; the memory activation selector is used for extracting visual diagnosis characteristics according to the medical image characteristics, acquiring a memory index of corresponding activation by calculating the similarity between the visual diagnosis characteristics and disease diagnosis knowledge stored in a disease diagnosis memory bank, performing de-duplication on the memory index, acquiring a memory characteristic combination of corresponding activation according to the memory index after de-duplication, and outputting the memory characteristic combination of corresponding activation; the memory feature combination serves as an instruction prefix, and the instruction prefix and a user instruction are input into a large language model decoder together; according to the technical scheme provided by the invention, the defects that the medical image is difficult to accurately analyze and the medical image intelligent analysis model is inconvenient to conveniently and extensively learn in the prior art can be effectively overcome.

Description

Technical Field

[0001] The present invention relates to intelligent analysis of medical images, and in particular to a medical image intelligent analysis model and a training method. Background Art

[0002] There are two main types of existing training methods for large medical imaging models. One is a training method involving memory learning. This type of method implicitly learns from data and updates the memory bank through model training. The other is a training method that does not involve memory learning. Through model training, the distribution from input to output is directly modeled, that is, a traditional training method that does not involve a memory module.

[0003] At present, for large medical imaging models, a large portion of them adopt the traditional end-to-end training paradigm. During the data iteration process, the model is expected to implicitly store scene-related knowledge into learnable parameters to complete distribution modeling from input to output without giving explicit knowledge summary goals. Although this training method is relatively easy to implement, it has poor interpretability and cannot meet the requirements of medical scenarios for accurate AI output content.

[0004] In addition, some training methods have designed relevant memory banks, but still use implicit learning methods to complete memory updates through model training and learning the interaction between features. Although this type of method introduces the setting of memory banks, it fails to effectively give the memory banks the ability to be interpretable and manipulable. The set memory banks still contain intermediate features with mixed semantics, which are only auxiliary to complete the distribution modeling from input to output. There is no essential difference between them and the traditional end-to-end training paradigm. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the above-mentioned shortcomings of the prior art, the present invention provides a medical image intelligent analysis model and training method, which can effectively overcome the defects of the prior art that it is difficult to accurately analyze medical images and it is inconvenient to conduct convenient and extensible learning of the medical image intelligent analysis model.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A medical image intelligent analysis model, including a visual feature encoder, a memory activation selector, a large language model decoder, and a disease diagnosis memory bank;

[0010] A visual feature encoder extracts medical image features from the input medical image and inputs the medical image features into the memory activation selector;

[0011] A memory activation selector extracts visual diagnostic features based on medical image features, obtains the corresponding activated memory index by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the disease diagnosis memory bank, deduplicates the memory index, obtains the corresponding activated memory feature combination based on the deduplicated memory index, and uses the memory feature combination as an instruction prefix and inputs it into the large language model decoder together with the user instruction;

[0012] The large language model decoder outputs a medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions.

[0013] Preferably, the visual feature encoder extracts medical image features from the input medical image, comprising:

[0014] The visual feature encoder uses the following formula to extract medical image features from the input medical image:

[0015] x ve =E vis (x);

[0016] Among them, E vis represents the visual feature encoder, x is the input medical image, x ve is a medical imaging feature. represents the corresponding domain, p is the number of medical image features, and d is the feature dimension;

[0017] The input medical images are 2D medical images or 3D medical images. 2D medical images include X-ray medical images and ultrasonic medical images, and 3D medical images include CT and MR.

[0018] Preferably, the memory activation selector extracts visual diagnostic features according to medical image features, obtains corresponding activated memory indexes by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the disease diagnosis memory bank, deduplicates the memory indexes, and obtains corresponding activated memory feature combinations according to the deduplicated memory indexes, including:

[0019] S11, the memory activation selector uses the following formula to extract visual diagnostic features based on medical image features:

[0020] h=f MAS (x ve );

[0021] Among them, f MAS Represents a memory activation selector, x ve is the medical image feature, h is the visual diagnosis feature, represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension;

[0022] Visual diagnostic features h include disease type features h dis and disease location characteristics loc , represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension;

[0023] S12. The similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank is calculated using the following formula:

[0024]

[0025] Among them, M dis Disease type knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n1 is the number of disease types, d is the feature dimension, T represents transposition, s dis is the disease type feature h dis The disease type knowledge M stored in the disease diagnosis memory bank dis The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features;

[0026] M loc Disease location knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n2 is the number of disease locations, d is the feature dimension, s loc h is the disease location feature loc The disease location knowledge M stored in the disease diagnosis memory bank loc The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features;

[0027] S13, obtaining the corresponding activated memory index by taking argmax, including the disease type index and the corresponding disease direction index;

[0028] S14, deduplicate the memory index, and obtain the corresponding activated memory feature combination according to the deduplicated memory index:

[0029]

[0030] in, is the memory feature of the cth disease type, m dis is the disease type memory feature set obtained based on the memory index after deduplication. represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, and d is the feature dimension;

[0031] is the cth disease location memory feature, m loc is the disease location memory feature set obtained based on the memory index after deduplication, represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, d is the feature dimension, and m is the memory feature combination.

[0032] Preferably, the large language model decoder outputs a medical image analysis report by sequence decoding according to the input medical image, memory feature combination and user instruction, including:

[0033] The large language model decoder uses the following formula to output the medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions:

[0034] k text =D llm (m,x,prompt);

[0035] Among them, D llm represents the large language model decoder, m is the memory feature combination, x is the input medical image, prompt is the user instruction, and the user instruction prompt is "please output the corresponding medical image analysis report based on the input medical image", k text To provide medical imaging analysis reports;

[0036] The medical imaging analysis report includes the medical imaging diagnosis report and the medical imaging Q&A response.

[0037] Preferably, the visual feature encoder is composed of a stack of convolutional structures or transformer structures, the memory activation selector is composed of a stack of convolutional structures or transformer structures, and the large language model decoder is composed of a stack of transformer structures.

[0038] A training method for a medical image intelligent analysis model is proposed, which performs two-stage step-by-step training on the medical image intelligent analysis model based on a memory quantization step-by-step learning framework:

[0039] Phase 1 training: In the encoder learning phase, the disease diagnosis memory is used to drive the visual feature encoder to fit the disease diagnosis memory, and to drive the large language model decoder to generate modeling based on the disease diagnosis memory.

[0040] In the second stage of training, the memory adaptation learning stage, the disease diagnosis memory bank is fine-tuned in a dynamic process by interconnecting and fine-tuning various modules, and is linked to feature extraction of the visual feature encoder and generative modeling of the large language model decoder for optimization.

[0041] Preferably, in the encoder learning stage, the disease diagnosis memory library is used to drive the visual feature encoder to fit the disease diagnosis memory library, and the large language model decoder is driven to generate modeling based on the disease diagnosis memory library, including:

[0042] S21, for the visual feature encoder, according to the preset disease list and location word list, each element in the disease diagnosis memory bank is sequentially assigned a specific meaning, and at the beginning of training, the disease diagnosis memory bank is initialized using a random orthogonal vector;

[0043] S22. Based on the medical image-medical image analysis report text pair data, disease keywords and location keywords in the medical image analysis report text are extracted by text regularization or keyword extraction:

[0044] (d1,loc1;d2,loc2;...)=E keyword (text);

[0045] Among them, E keyword represents a keyword extractor, text is the text of the medical image analysis report, d1, d2... are the extracted disease keywords, loc1, loc2... are the extracted location keywords;

[0046] S23, construct the corresponding co-occurrence matrix ydis_loc according to the index of the disease keywords and the location keywords in the disease diagnosis memory bank, represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0047] S24, after the memory activation selector outputs the similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank, the response matrix of each visual diagnosis feature is obtained using the following formula:

[0048]

[0049] in, is the similarity matrix between the k-th disease type feature and the i-th disease type knowledge stored in the disease diagnosis memory bank, is the similarity matrix between the k-th disease location feature and the i-th disease location knowledge stored in the disease diagnosis memory bank, is the response matrix of the kth visual diagnostic feature, q is the number of visual diagnostic features, represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0050] S25, average the response matrix of each visual diagnostic feature to obtain the response matrix k of the whole image dis_loc , represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0051] S26. For the large language model decoder, according to the index of the disease keywords and location keywords in the disease diagnosis memory library, the corresponding medical image features and memory feature combinations are extracted and input into the large language model decoder together with the user instructions.

[0052] Preferably, the training supervision loss function of the visual feature encoder is as follows:

[0053]

[0054] Among them, y dis_loc is the co-occurrence matrix, k dis_loc is the response matrix of the whole image, α is the sample distribution weight modulation item, which is used to increase the weight of non-zero labels in the collinear matrix and alleviate the impact of label imbalance, α=max(n1,n2), n1 is the number of disease types, n2 is the number of disease directions, γ is the difficult sample weight modulation item, which is used to control and reduce the loss value of negative samples, γ=2, L vis_cog (y dis_loc ,k dis_loc ) is the training supervision loss function of the visual feature encoder;

[0055] The training supervision loss function of the large language model decoder is as follows:

[0056]

[0057] Among them, y text For reference to medical imaging analysis reports, k text Medical image analysis report output by the large language model decoder, y t is the output probability distribution at time t, k ＜t is the decoding history before time t, M is the decoding time, L llm (y text ,k text ) is the training supervision loss function of the large language model decoder.

[0058] Preferably, in the memory adaptation learning stage, the disease diagnosis memory bank is fine-tuned in a dynamic process by connecting and fine-tuning various modules, and is linked to feature extraction of the visual feature encoder and generative modeling of the large language model decoder for optimization, including:

[0059] The modules are fine-tuned by combining multi-task losses. The multi-task loss function is as follows:

[0060]

[0061] Among them, the first term L llm (y text ,k text ) is the training supervision loss function of the large language model decoder, and the second term is the alignment loss between the visual diagnostic features and the disease diagnostic knowledge stored in the disease diagnostic memory. To make the visual diagnosis features closer to the disease diagnosis memory, speed up the loss of the large language model decoder based on the disease diagnosis memory to generate modeling, the fourth term λ(L div (M dis )+L div (M loc )) is a regular term, which is used to ensure that there are differences between memory feature combinations, that is, the variance between memory feature combinations is large, which can represent different disease types and disease locations;

[0062] sg[·] is the gradient truncation function, which is used to cut off the gradient during back propagation. The operation of obtaining the corresponding activated memory feature combination according to the memory index after deduplication is not differentiable. During back propagation, the large language model decoder transfers the visual feature encoder to the visual feature encoder by gradient copy, which achieves end-to-end trainability.

[0063] L div (M dis ) is the disease type knowledge M stored in the disease diagnosis memory bank dis The sum of the cosine similarities of the two memory features in L div (M loc ) is the disease location knowledge M stored in the disease diagnosis memory bank loc The sum of the cosine similarities of the pairwise memory features.

[0064] Preferably, it also includes extended learning of medical image intelligent analysis models:

[0065] In the early stage, the disease diagnosis memory library of scene A is constructed. In the later stage, when facing the medical image intelligent analysis task of scene B, new disease type knowledge and disease location knowledge are spliced ​​in the disease diagnosis memory library of scene A, and corresponding memory feature combinations are assigned to them. Then, for the newly added memory feature combinations, the medical image intelligent analysis model is trained in two stages based on the memory quantization step-by-step learning framework;

[0066] During the repeated training of the model, the memory feature combination under scene A is frozen, and the image and text data of scene A is iterated in combination with the image and text data of scene B to complete the expansion of the diagnostic memory filling of the new scene. The model can maintain the ability to intelligently analyze medical images in old scenes while having the ability to intelligently analyze medical images in new scenes. Through continuous expansive learning, the integration of a general medical image intelligent analysis model is achieved.

[0067] (III) Beneficial effects

[0068] Compared with the prior art, the medical image intelligent analysis model and training method provided by the present invention designs a medical image intelligent analysis model composed of a visual feature encoder, a memory activation selector, a large language model decoder and a disease diagnosis memory bank, and a two-stage step-by-step training method for the medical image intelligent analysis model based on a memory quantization step-by-step learning framework. Combined with the extended learning of the medical image intelligent analysis model, it can enhance the disease recognition and disease location capabilities of the model, and comprehensively improve the capabilities of the medical image intelligent analysis model in scenarios such as medical image analysis report generation and medical image question-and-answer response. At the same time, it has interpretability and scalability, and can continuously learn and summarize new medical image knowledge by simply expanding the memory bank and quickly fine-tuning and adapting in the process of constantly changing task scenarios without adjusting the model structure.

[0069] At the same time, the memory quantization strategy designed by the present invention directly corresponds to the key disease type identification and disease location positioning in clinical diagnosis, which can not only improve the accuracy of medical image analysis, but also make the output results interpretable, with strong practicality and flexibility. Through explicit reinforcement modeling, the model has the ability to accurately understand and analyze disease types and disease locations, thereby realizing accurate intelligent auxiliary diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0071] Figure 1 It is a schematic diagram of the structure of the medical image intelligent analysis model of the present invention;

[0072] Figure 2 Schematic diagram of the process of training the intelligent analysis model of traditional Chinese medicine images according to the present invention. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] A medical image intelligent analysis model, such as Figure 1 As shown, it includes a visual feature encoder, a memory activation selector, a large language model decoder, and a disease diagnosis memory bank;

[0075] Visual-Encoder, which extracts medical image features from the input medical image and inputs the medical image features into the memory activation selector;

[0076] Memory Activation Selector: extracts visual diagnostic features based on medical image features, obtains the corresponding activated memory index by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the Disease Diagnosis Memory Bank, deduplicates the memory index, obtains the corresponding activated memory feature combination based on the deduplicated memory index, and uses the memory feature combination as the instruction prefix and inputs it into the large language model decoder together with the user instruction;

[0077] The Large Language Model Decoder (LLM-Decoder) outputs a medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions.

[0078] ① The visual feature encoder extracts medical image features from the input medical image, including:

[0079] The visual feature encoder uses the following formula to extract medical image features from the input medical image:

[0080] x ve =E vis (x);

[0081] Among them, E vis represents the visual feature encoder, x is the input medical image, x ve is a medical imaging feature. represents the corresponding domain, p is the number of medical image features, and d is the feature dimension;

[0082] The input medical images are 2D medical images or 3D medical images. 2D medical images include X-ray medical images and ultrasonic medical images, and 3D medical images include CT and MR.

[0083] ② The memory activation selector extracts visual diagnostic features based on medical image features, obtains the corresponding activated memory index by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the disease diagnosis memory bank, deduplicates the memory index, and obtains the corresponding activated memory feature combination based on the deduplicated memory index, including:

[0084] S11, the memory activation selector uses the following formula to extract visual diagnostic features based on medical image features:

[0085] h=f MAS (x ve );

[0086] Among them, f MAS Represents a memory activation selector, x ve is the medical image feature, h is the visual diagnosis feature, represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension;

[0087] Visual diagnostic features h include disease type features h dis and disease location characteristics loc , represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension;

[0088] S12. The similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank is calculated using the following formula:

[0089]

[0090] Among them, M dis Disease type knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n1 is the number of disease types (based on different parts of the human body, taking the lungs as an example, it corresponds to lung disease types such as pneumonia, lung nodules, and emphysema), d is the feature dimension, T represents transposition, and s dis is the disease type feature h dis The disease type knowledge M stored in the disease diagnosis memory bank dis The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features;

[0091] M loc Disease location knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n2 is the number of disease locations (based on different human body parts, taking the lungs as an example, it corresponds to the upper left lobe, middle left lobe, lower left lobe, upper right chest wall and other lung disease locations), d is the feature dimension, s loc h is the disease location feature loc The disease location knowledge M stored in the disease diagnosis memory bank loc The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features;

[0092] S13, obtaining the corresponding activated memory index by taking argmax, including the disease type index and the corresponding disease direction index;

[0093] S14, deduplicate the memory index, and obtain the corresponding activated memory feature combination according to the deduplicated memory index:

[0094]

[0095] in, is the memory feature of the cth disease type, m dis is the disease type memory feature set obtained based on the memory index after deduplication. represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, and d is the feature dimension;

[0096] is the cth disease location memory feature, m loc is the disease location memory feature set obtained based on the memory index after deduplication, represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, d is the feature dimension, and m is the memory feature combination.

[0097] ③ The large language model decoder outputs a medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions, including:

[0098] The large language model decoder uses the following formula to output the medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions:

[0099] k text =D llm (m,x,prompt);

[0100] Among them, D llm represents the large language model decoder, m is the memory feature combination, x is the input medical image, prompt is the user instruction, and the user instruction prompt is "please output the corresponding medical image analysis report based on the input medical image", k text To provide medical imaging analysis reports;

[0101] The medical imaging analysis report includes the medical imaging diagnosis report and the medical imaging Q&A response.

[0102] In the technical solution of the present application, the visual feature encoder is composed of a stack of convolutional structures or transformer structures, the memory activation selector is composed of a stack of convolutional structures or transformer structures, and the large language model decoder is composed of a stack of transformer structures.

[0103] A training method for a medical image intelligent analysis model, such as Figure 2 As shown in the figure, the medical image intelligent analysis model is trained in two stages based on the memory quantization step-by-step learning framework:

[0104] The first stage of training: Encoder learning stage (Encode-Decode Learning), using the disease diagnosis memory library to drive the visual feature encoder to fit the disease diagnosis memory library, and drive the large language model decoder to generate modeling based on the disease diagnosis memory library;

[0105] The second stage of training, the memory adaptation learning stage (MemoryAdaptation Learning), fine-tunes each module through interconnection, so that the disease diagnosis memory bank can be fine-tuned in a dynamic process, and is linked to the feature extraction of the visual feature encoder and the generative modeling of the large language model decoder for optimization.

[0106] ① In the encoder learning stage, the disease diagnosis memory library is used to drive the visual feature encoder to fit the disease diagnosis memory library, and to drive the large language model decoder to generate modeling based on the disease diagnosis memory library, including:

[0107] S21, for the visual feature encoder, according to the preset disease list and location word list, each element in the disease diagnosis memory bank is sequentially assigned a specific meaning, and at the beginning of training, the disease diagnosis memory bank is initialized using a random orthogonal vector;

[0108] S22. Based on the medical image-medical image analysis report text pair data, disease keywords and location keywords in the medical image analysis report text are extracted by text regularization or keyword extraction:

[0109] (d1,loc1;d2,loc2;...)=E keyword (text);

[0110] Among them, E keyword represents a keyword extractor, text is the text of the medical image analysis report, d1, d2... are the extracted disease keywords, loc1, loc2... are the extracted location keywords;

[0111] S23, construct the corresponding co-occurrence matrix ydis_loc according to the index of the disease keywords and the location keywords in the disease diagnosis memory bank, represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0112] S24, after the memory activation selector outputs the similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank, the response matrix of each visual diagnosis feature is obtained using the following formula:

[0113]

[0114] in, is the similarity matrix between the k-th disease type feature and the i-th disease type knowledge stored in the disease diagnosis memory bank, is the similarity matrix between the k-th disease location feature and the i-th disease location knowledge stored in the disease diagnosis memory bank, is the response matrix of the kth visual diagnostic feature, q is the number of visual diagnostic features, represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0115] S25, average the response matrix of each visual diagnostic feature to obtain the response matrix k of the whole image dis_loc , represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions;

[0116] S26. For the large language model decoder, according to the index of the disease keywords and location keywords in the disease diagnosis memory library, the corresponding medical image features and memory feature combinations are extracted and input into the large language model decoder together with the user instructions.

[0117] Specifically, the training supervision loss function of the visual feature encoder is as follows:

[0118]

[0119] Among them, y dis_loc is the co-occurrence matrix, k dis_loc is the response matrix of the whole image, α is the sample distribution weight modulation item, which is used to increase the weight of non-zero labels in the collinear matrix and alleviate the impact of label imbalance, α=max(n1,n2), n1 is the number of disease types, n2 is the number of disease directions, γ is the difficult sample weight modulation item, which is used to control and reduce the loss value of negative samples, γ=2, L vis_cog (y dis_loc ,k dis_loc ) is the training supervision loss function of the visual feature encoder;

[0120] The training supervision loss function of the large language model decoder is as follows:

[0121]

[0122] Among them, y text For reference to medical imaging analysis reports, k text Medical image analysis report output by the large language model decoder, y t is the output probability distribution at time t, k ＜t is the decoding history before time t, M is the decoding time, L llm (y text ,k text ) is the training supervision loss function of the large language model decoder.

[0123] ② In the memory adaptation learning stage, the disease diagnosis memory bank is fine-tuned in a dynamic process by connecting and fine-tuning various modules, and is optimized in conjunction with the feature extraction of the visual feature encoder and the generative modeling of the large language model decoder, including:

[0124] The modules are fine-tuned by combining multi-task losses. The multi-task loss function is as follows:

[0125]

[0126] Among them, the first term L llm (y text ,k text ) is the training supervision loss function of the large language model decoder, and the second term is the alignment loss between the visual diagnosis features and the disease diagnosis knowledge stored in the disease diagnosis memory. To make the visual diagnosis features closer to the disease diagnosis memory, speed up the loss of the large language model decoder based on the disease diagnosis memory to generate modeling, the fourth term λ(L div (M dis )+L div (M loc )) is a regular term, which is used to ensure that there are differences between memory feature combinations, that is, the variance between memory feature combinations is large, which can represent different disease types and disease locations;

[0127] sg[·] is the gradient truncation function, which is used to cut off the gradient during back propagation. The operation of obtaining the corresponding activated memory feature combination according to the memory index after deduplication is not differentiable. During back propagation, the large language model decoder transfers the visual feature encoder to the visual feature encoder by gradient copy, which achieves end-to-end trainability.

[0128] L div (M dis ) is the disease type knowledge M stored in the disease diagnosis memory bank dis The sum of the cosine similarities of the two memory features in Ldiv (M loc ) is the disease location knowledge M stored in the disease diagnosis memory bank loc The sum of the cosine similarities of the pairwise memory features.

[0129] ③ In the technical solution of this application, the training method for the medical image intelligent analysis model also includes extended learning of the medical image intelligent analysis model:

[0130] In the early stage, the disease diagnosis memory library of scene A is constructed. In the later stage, when facing the medical image intelligent analysis task of scene B, new disease type knowledge and disease location knowledge are spliced ​​in the disease diagnosis memory library of scene A, and corresponding memory feature combinations are assigned to them. Then, for the newly added memory feature combinations, the medical image intelligent analysis model is trained in two stages based on the memory quantization step-by-step learning framework;

[0131] During the repeated training of the model, the memory feature combination under scene A is frozen, and the image and text data of scene A is iterated in combination with the image and text data of scene B to complete the expansion of the diagnostic memory filling of the new scene. The model can maintain the ability to intelligently analyze medical images in old scenes while having the ability to intelligently analyze medical images in new scenes. Through continuous expansive learning, the integration of a general medical image intelligent analysis model is achieved.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A medical image intelligent analysis model, characterized by: Includes visual feature encoder, memory activation selector, large language model decoder and disease diagnosis memory bank; A visual feature encoder extracts medical image features from the input medical image and inputs the medical image features into the memory activation selector; A memory activation selector extracts visual diagnostic features based on medical image features, obtains the corresponding activated memory index by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the disease diagnosis memory bank, deduplicates the memory index, obtains the corresponding activated memory feature combination based on the deduplicated memory index, and uses the memory feature combination as an instruction prefix and inputs it into the large language model decoder together with the user instruction; The large language model decoder outputs a medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions.

2. The medical image intelligent analysis model according to claim 1, characterized in that: The visual feature encoder extracts medical image features from the input medical image, including: The visual feature encoder uses the following formula to extract medical image features from the input medical image: x ve =E vis (x); Among them, E vis represents the visual feature encoder, x is the input medical image, x ve is a medical imaging feature. represents the corresponding domain, p is the number of medical image features, and d is the feature dimension; The input medical images are 2D medical images or 3D medical images. 2D medical images include X-ray medical images and ultrasonic medical images, and 3D medical images include CT and MR.

3. The medical image intelligent analysis model according to claim 2, characterized in that: The memory activation selector extracts visual diagnostic features according to medical image features, obtains corresponding activated memory indexes by calculating the similarity between the visual diagnostic features and the disease diagnosis knowledge stored in the disease diagnosis memory bank, deduplicates the memory indexes, and obtains corresponding activated memory feature combinations according to the deduplicated memory indexes, including: S11, the memory activation selector uses the following formula to extract visual diagnostic features based on medical image features: h=f MAS (x ve ); Among them, f MAS Represents a memory activation selector, x ve is the medical image feature, h is the visual diagnosis feature, represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension; Visual diagnostic features h include disease type features h dis and disease location characteristics loc , represents the corresponding domain, q is the number of visual diagnostic features, and d is the feature dimension; S12. The similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank is calculated using the following formula: Among them, M dis The disease type knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n1 is the number of disease types, d is the feature dimension, T represents transposition, s dis is the disease type feature h dis The disease type knowledge M stored in the disease diagnosis memory bank dis The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features; M loc The disease location knowledge stored in the disease diagnosis memory bank, represents the corresponding domain, n2 is the number of disease locations, d is the feature dimension, s loc h is the disease location feature loc The disease location knowledge M stored in the disease diagnosis memory bank loc The similarity matrix between represents the corresponding domain, q is the number of visual diagnostic features; S13, obtaining the corresponding activated memory index by taking argmax, including the disease type index and the corresponding disease direction index; S14, deduplicate the memory index, and obtain the corresponding activated memory feature combination according to the deduplicated memory index: in, Memory characteristics for the cth disease type, m dis is the disease type memory feature set obtained based on the memory index after deduplication. represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, and d is the feature dimension; is the cth disease location memory feature, m loc is the disease location memory feature set obtained based on the memory index after deduplication, represents the corresponding domain, c is the total number of key diagnoses contained in the medical image, c≤q, d is the feature dimension, and m is the memory feature combination.

4. The medical image intelligent analysis model according to claim 3, characterized in that: The large language model decoder outputs a medical image analysis report by sequence decoding according to the input medical image, memory feature combination and user instruction, including: The large language model decoder uses the following formula to output the medical image analysis report through sequence decoding based on the input medical image, memory feature combination and user instructions: k text =D llm (m,x,prompt); Among them, D llm represents the large language model decoder, m is the memory feature combination, x is the input medical image, prompt is the user instruction, and the user instruction prompt is "Please output the corresponding medical image analysis report based on the input medical image", k text To provide medical imaging analysis reports; The medical imaging analysis report includes the medical imaging diagnosis report and the medical imaging Q&A response.

5. The medical image intelligent analysis model according to claim 4, characterized in that: The visual feature encoder is composed of a stack of convolutional structures or transformer structures, the memory activation selector is composed of a stack of convolutional structures or transformer structures, and the large language model decoder is composed of a stack of transformer structures.

6. A method for training a medical image intelligent analysis model, applied to the medical image intelligent analysis model of claim 1, characterized in that: Based on the memory quantization step-by-step learning framework, the medical image intelligent analysis model is trained in two stages: Phase 1 training: In the encoder learning phase, the disease diagnosis memory is used to drive the visual feature encoder to fit the disease diagnosis memory, and to drive the large language model decoder to generate modeling based on the disease diagnosis memory. In the second stage of training, the memory adaptation learning stage, the disease diagnosis memory bank is fine-tuned in a dynamic process by interconnecting and fine-tuning various modules, and is linked to feature extraction of the visual feature encoder and generative modeling of the large language model decoder for optimization.

7. The method for training a medical image intelligent analysis model according to claim 6, characterized in that: The encoder learning stage uses the disease diagnosis memory library to drive the visual feature encoder to fit the disease diagnosis memory library, and drives the large language model decoder to generate modeling based on the disease diagnosis memory library, including: S21, for the visual feature encoder, according to the preset disease list and location word list, each element in the disease diagnosis memory bank is sequentially assigned a specific meaning, and at the beginning of training, the disease diagnosis memory bank is initialized using a random orthogonal vector; S22. Based on the medical image-medical image analysis report text pair data, disease keywords and location keywords in the medical image analysis report text are extracted by text regularization or keyword extraction: (d1,loc1;d2,loc;...)=E keyword (text); Among them, E keyword represents a keyword extractor, text is the text of the medical image analysis report, d1, d2... are the extracted disease keywords, loc1, loc2... are the extracted location keywords; S23, construct the corresponding co-occurrence matrix y according to the index of disease keywords and location keywords in the disease diagnosis memory library dis_loc , represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions; S24, after the memory activation selector outputs the similarity between the visual diagnosis feature and the disease diagnosis knowledge stored in the disease diagnosis memory bank, the response matrix of each visual diagnosis feature is obtained using the following formula: in, is the similarity matrix between the k-th disease type feature and the i-th disease type knowledge stored in the disease diagnosis memory bank, is the similarity matrix between the k-th disease location feature and the i-th disease location knowledge stored in the disease diagnosis memory bank, is the response matrix of the kth visual diagnostic feature, q is the number of visual diagnostic features, represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions; S25, average the response matrices of each visual diagnostic feature to obtain the response matrix k of the entire image dis_loc , represents the corresponding domain, n1 is the number of disease types, and n2 is the number of disease directions; S26. For the large language model decoder, according to the index of the disease keywords and location keywords in the disease diagnosis memory library, the corresponding medical image features and memory feature combinations are extracted and input into the large language model decoder together with the user instructions.

8. The method for training a medical image intelligent analysis model according to claim 7, characterized in that: The training supervision loss function of the visual feature encoder is as follows: Among them, y dis_loc is the co-occurrence matrix, k dis_loc is the response matrix of the whole image, α is the sample distribution weight modulation item, which is used to increase the weight of non-zero labels in the collinear matrix and alleviate the impact of label imbalance, α=max(n1,n2), n1 is the number of disease types, n2 is the number of disease directions, γ is the difficult sample weight modulation item, which is used to control and reduce the loss value of negative samples, γ=2, L vis_cog (y dis_loc ,k dis_loc ) is the training supervision loss function of the visual feature encoder; The training supervision loss function of the large language model decoder is as follows: Among them, y text For reference to medical imaging analysis reports, k text Medical image analysis report output by the large language model decoder, y t is the output probability distribution at time t, k ＜t is the decoding history before time t, M is the decoding time, L llm (y text ,k text ) is the training supervision loss function of the large language model decoder.

9. The method for training a medical image intelligent analysis model according to claim 7, characterized in that: In the memory adaptation learning stage, the disease diagnosis memory bank is fine-tuned in a dynamic process by interconnecting and fine-tuning various modules, and is linked to feature extraction of the visual feature encoder and generative modeling of the large language model decoder for optimization, including: The modules are fine-tuned by combining multi-task losses. The multi-task loss function is as follows: Among them, the first term L llm (y text ,k text ) is the training supervision loss function of the large language model decoder, and the second term is the alignment loss between the visual diagnosis features and the disease diagnosis knowledge stored in the disease diagnosis memory. To make the visual diagnosis features closer to the disease diagnosis memory, speed up the loss of the large language model decoder based on the disease diagnosis memory to generate modeling, the fourth term λ(L div (M dis )+L div (M loc )) is a regular term, which is used to ensure that there are differences between memory feature combinations, that is, the variance between memory feature combinations is large, which can represent different disease types and disease locations; sg[·] is the gradient truncation function, which is used to cut off the gradient during back propagation. The operation of obtaining the corresponding activated memory feature combination according to the memory index after deduplication is not differentiable. During back propagation, the large language model decoder transfers the visual feature encoder to the visual feature encoder by gradient copy, which achieves end-to-end trainability. L div (M dis ) is the disease type knowledge M stored in the disease diagnosis memory bank dis The sum of the cosine similarities of the two memory features in L div (M loc ) is the disease location knowledge M stored in the disease diagnosis memory bank loc The sum of the cosine similarities of the pairwise memory features.

10. The method for training a medical image intelligent analysis model according to claim 9, characterized in that: It also includes extended learning of medical image intelligent analysis models: In the early stage, the disease diagnosis memory library of scene A is constructed. In the later stage, when facing the medical image intelligent analysis task of scene B, new disease type knowledge and disease location knowledge are spliced ​​in the disease diagnosis memory library of scene A, and corresponding memory feature combinations are assigned to them. Then, for the newly added memory feature combinations, the medical image intelligent analysis model is trained in two stages based on the memory quantization step-by-step learning framework; During the repeated training of the model, the memory feature combination under scene A is frozen, and the image and text data of scene A is iterated in combination with the image and text data of scene B to complete the expansion of the diagnostic memory filling of the new scene. The model can maintain the ability to intelligently analyze medical images in old scenes while having the ability to intelligently analyze medical images in new scenes. Through continuous expansive learning, the integration of a general medical image intelligent analysis model is achieved.

Images