A medical image classification method based on dynamic multi-granularity hierarchical prompt field

Abstract

The application belongs to the technical field of artificial intelligence and medical information processing, and particularly relates to a medical image classification method based on a dynamic multi-granularity hierarchical prompt field, which comprises the following steps: constructing three different prompt fields, taking the global prompt field, the medium prompt field and the fine-granularity prompt field as a first prompt vector, and taking the medium prompt field and the fine-granularity prompt field as a second prompt vector; performing double-layer cross-attention calculation in a field controller to obtain two text features based on the first prompt vector and the second prompt vector respectively; splicing the visual feature and the two text features respectively, then calculating the classification probability distribution of the two spliced vectors respectively, and finally obtaining the final image classification result by weighted fusion based on the entropy value. The application makes the prompt field gradually transition from the global / medium semantic bias to the fine-granularity semantic enhancement in the training process, effectively reduces the disturbance of early fine-granularity prompts on learning, improves the training stability and promotes the stable convergence of fine-granularity discrimination ability.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and medical information processing technology, and is applicable to various medical image classification tasks. Specifically, it relates to a medical image classification method based on a dynamic multi-granularity hierarchical cue field. Background Technology

[0002] In recent years, the emergence of CLIP models has marked a significant advancement in pre-trained visual-language models. CLIP-type models employ an approach that aligns both image and text modalities, demonstrating excellent few-shot and zero-shot generalization capabilities in current open dataset image recognition. In the fields of natural and medical image processing, current work has explored replacing manually written, fixed text cue templates with finely tuned text cue vectors, thereby significantly improving the performance of downstream tasks with minimal annotation and computational cost. Therefore, cue learning is gradually becoming an important direction in visual-language modeling.

[0003] Building upon CLIP, the medical field has also seen the emergence of large-scale vision-language pre-trained models specifically for medical scenarios, such as BiomedCLIP. This model is pre-trained across modalities using PubMed text and large-scale medical image data, demonstrating stronger domain-specific capabilities compared to the general-purpose CLIP in disease description, organ structure, and semantic understanding of medical images. Therefore, it has become a commonly used foundational model in downstream tasks such as medical image classification, segmentation, and retrieval. However, directly transferring cue learning to the medical image domain still presents several challenges.

[0004] First, the semantic information in medical images possesses multi-granularity features, including global structural features (such as organ location), mesoscopic lesion morphology (such as local abnormal areas), and fine-grained visual details (such as subtle signal changes and texture differences). Most existing cue learning methods employ static cue vectors of single granularity, which struggle to simultaneously represent the multi-level semantic structure of medical images, thus limiting the model's visual-linguistic alignment capabilities.

[0005] Secondly, medical imaging models exhibit significant stage-specific characteristics during training: in the early stages of training, models rely more on stable, coarse-grained semantic information, while in the mid-to-late stages, they require stronger fine-grained feature alignment capabilities. Existing cue learning methods generally employ fixed, static cues, lacking a gradual, transitional cue control mechanism similar to the "time-step scheduling" in diffusion models. The contribution of cue information cannot be dynamically adjusted with the training progress, making it difficult to adapt to the "coarse-to-fine" learning pattern of medical semantics.

[0006] Furthermore, due to the prevalence of imaging noise, low contrast, small lesions, and blurred boundaries in medical images, a large number of challenging samples exist. These challenging samples are prone to blurred boundaries, noise interference, and tiny lesion areas. These samples are often the most easily misclassified, and traditional cue-based learning mechanisms lack explicit attention to them, leading to performance bottlenecks.

[0007] Fourth, medical images exhibit high intra-group heterogeneity. The imaging manifestations of the same disease can vary significantly among different patients, and simple classification supervision is insufficient to guarantee the clustering of similar features in the feature space. Without constraints, the intra-class distribution is loose and the inter-class boundaries are blurred, further complicating the classification task.

[0008] Finally, in the inference phase, most existing cue learning methods employ fixed cues or simple static combinations. However, the uncertainty of different samples in medical images varies significantly. Under conditions of high noise, blurred lesions, and large variations in imaging conditions, the model's dependence on cues of different granularities is inconsistent. Fixed cue strategies cannot dynamically select more appropriate cue granularity combinations based on sample difficulty, resulting in insufficient robustness of the system across different case scenarios. Summary of the Invention

[0009] To effectively improve the accuracy, stability, and robustness of medical image classification, this invention proposes a medical image classification method based on a dynamic multi-granularity hierarchical cue field, which specifically includes the following steps:

[0010] The medical image to be classified is input into the visual encoder to obtain visual features, and the visual token obtained by the visual encoder is used as the input of the field controller.

[0011] In the field controller, three different cue fields are constructed through global cue, meso cue, and fine-grained cue. The global cue field, meso cue field, and fine-grained cue field are used as the first cue vector, and the meso cue field and fine-grained cue field are used as the second cue vector. Each row of each cue vector is a cue field.

[0012] Using the visual token as the key vector and value vector, and the first cue vector and the first cue vector cue field as the query vector, a two-layer cross-attention calculation is performed in the field controller to obtain two text features;

[0013] The visual features and the two text features are concatenated separately, and then the entropy values ​​of the classification probability distributions of the two concatenated vectors are calculated separately.

[0014] The entropy difference is scaled by a temperature coefficient and then input into the Sigmoid function. After linear mapping, it is applied to the set minimum-maximum adaptive weight space to obtain the adaptive cue fusion weight.

[0015] The two classification probabilities are weighted and fused using adaptive cueing fusion weights to obtain the final image classification result.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] (1) The present invention divides the explicit prompt into a global subfield, a meso-level subfield and a fine-grained subfield, and injects visual conditions into the prompt representation through a two-layer Transformer-type cross-modal controller. This structure enables the prompt to have both macro-level semantic carrying capacity and adaptive capture of local lesion details, thereby significantly enhancing the semantic expression capability of text-image cross-modal alignment and the image conditional prompt capability, which is beneficial to improving the recognition accuracy of multi-scale lesions in complex medical images.

[0018] (2) The present invention introduces a training step scheduling mechanism S(e), which enables the cue field to gradually and smoothly transition from emphasizing global / meso-level semantics to enhancing fine-grained semantics (from coarse to fine) during the training process. This effectively reduces the disturbance of early fine-grained cues to learning, improves training stability, and promotes the robust convergence of fine-grained discrimination ability.

[0019] (3) This invention adds intra-class consistency regularization terms under the current dynamic prompt conditions. This constrains the compact clustering of samples of the same category in the feature space, reduces intra-class dispersion and inter-class confusion, thereby improving the model's discriminative ability and robustness to heterogeneous clinical samples.

[0020] (4) The present invention applies a targeted refinement loss term to instances identified as difficult samples during training. This explicitly widens the score gap between the true class and the strongest negative class, enhances the model's sensitivity to samples with blurred boundaries, low contrast, or small lesions, and improves the detection rate and reliability of key clinical samples.

[0021] (5) In the inference process, this invention calculates the entropy difference between the two output entropy values ​​generated by the reference cues (partially steady-state / coarse-grained) and the dynamic cues (hybrid cues obtained from training), and maps it to adaptive fusion weights. The system dynamically combines different granularity prompts based on sample uncertainty. This mechanism enables the system to automatically select a more suitable combination of prompt granularity for each sample, thereby achieving an adaptive balance between robustness and fine-grained discrimination, and improving generalization ability and reliability under heterogeneous images and uncertain inputs. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the inference stage of a medical image classification method based on a dynamic multi-granularity hierarchical cue field according to the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention proposes a medical image classification method based on a dynamic multi-granularity hierarchical cue field, specifically including the following steps:

[0025] The medical image to be classified is input into the visual encoder to obtain visual features, and the visual token obtained by the visual encoder is used as the input of the field controller.

[0026] In the field controller, three different cue fields are constructed through global cue, meso cue, and fine-grained cue. The global cue field, meso cue field, and fine-grained cue field are used as the first cue vector, and the meso cue field and fine-grained cue field are used as the second cue vector. Each row of each cue vector is a cue field.

[0027] Using the visual token as the key vector and value vector, and the first cue vector and the first cue vector cue field as the query vector, a two-layer cross-attention calculation is performed in the field controller to obtain two text features;

[0028] The visual features and the two text features are concatenated separately, and then the entropy values ​​of the classification probability distributions of the two concatenated vectors are calculated separately.

[0029] The entropy difference is scaled by a temperature coefficient and then input into the Sigmoid function. After linear mapping, it is applied to the set minimum-maximum adaptive weight space to obtain the adaptive cue fusion weight.

[0030] The two classification probabilities are weighted and fused using adaptive cueing fusion weights to obtain the final image classification result.

[0031] In this embodiment, a medical vision-language basic model is first constructed, using a pre-trained visual encoder and a text encoder as the backbone network; a dynamic multi-granularity hierarchical cue field is then introduced on the input side of the text encoder. This prompt field is composed of global subfields. Middle-level subfield With fine-grained subfields The system is composed of three components: a global semantic unit, a meso-level conceptual unit, and a fine-grained morphological unit; a text sequence consisting of the cue vector group and the category name token is constructed for each category; and a cue field controller is set before the cue is sent to the text encoder. To achieve visual conditional injection, the controller is implemented as a two-layer Transformer with cross-modal attention. Its queries consist of learnable prompt seeds, and its keys / values ​​are provided by visual tokens output by the visual encoder. The controller injects visual information into the cue representation through cross-modal attention and outputs a cue embedding that is aligned with the context length of the text encoder. The cue embedding is concatenated with the category name token and then input into the text encoder to generate category text features.

[0032] This embodiment uses the pre-trained medical vision-language model BiomedCLIP as its backbone. The visual encoder receives the pre-processed medical images. Output normalized image feature vector:

[0033]

[0034] The text encoder receives a text sequence concatenated from a start marker, a cue context vector, a category name token, and an end marker, and outputs normalized text features:

[0035]

[0036] The dynamic multi-granularity hierarchical hint field is parameterized by three types of subfields:

[0037]

[0038] in, For global subfields, For the middle-level sub-field, This is a fine-grained subfield. (Note: This refers to the field controller.) The implementation is a two-layer Transformer (including cross-modal attention): Queries are learnable prompt seeds, and Keys / Values ​​are visual tokens output by the visual encoder. The controller injects visual conditions into each subfield representation and outputs a cue embedding aligned with the context length of the text encoder. Then concatenate it with the category name token to generate text features. .

[0039] Image-text similarity is calculated using a temperature-scaled inner product:

[0040]

[0041] in, Text features representing the c-th category Image features The degree of similarity, This is the temperature scaling factor; Representing medical images The probability of category c.

[0042] Prompt field controller It operates in a feedforward manner: its input Queries are a set of learnable cue seeds, and Keys / Values ​​are the visual encoder's input to the current image. Extracted visual tokens. The controller, through cross-modal attention computation, outputs a set of image-conditional contextual cue embeddings aligned with the visual features of the current image, denoted as... The prompt is embedded The length is aligned with the preset context length of the text encoder and is independent of the category.

[0043] For each category c, its category text sequence is constructed as follows:

[0044]

[0045] Prompt context The three subfields are controlled by a controller under visual conditions. Generate and weight the data according to progress in each training round (see Step 3). Categorical text features:

[0046]

[0047] When the number of training rounds is E, during the training process, in each round... Define the time step scaling factor:

[0048]

[0049] in, Provide a fine-grained hint for the initial lower bound (e.g., it can be set to 0.03). To increase the margin (for example, it can be set to 0.05, hence...) ).

[0050] Mixed weights , , satisfy + + =1, and throughout the entire training process, The proportion should decrease as the number of training rounds e increases, while and The weighting is adjusted accordingly to achieve a smooth transition from global / meso-level semantic dominance to fine-grained semantic dominance. The dynamic allocation of the mixed weights of the three subfields is driven by the time step scaling factor S(e), that is:

[0051]

[0052] in, and It is a preset, fixed proportionality coefficient, and satisfies + =1, for example, can be set to 1. =0.7, =0.3 indicates that global suggestions have the majority weight in the non-fine-grained parts. This mapping ensures the weight of fine-grained suggestions. The weights increase linearly from the initial value β to β+α, while the global and mesoscopic hints sum to ( + The cue values ​​decrease linearly from 1-β to 1-(β+α), achieving a progressive cue scheduling from coarse to fine. This results in a hybrid cue for training:

[0053]

[0054] In this embodiment, the overall training objective is to minimize the following loss:

[0055]

[0056] The classification crossover loss is expressed as:

[0057]

[0058] in, For classification crossover loss; The batch size; Indicates that in a given input image Under the given conditions, predict whether it belongs to the true category. The probability of.

[0059] In the current mixed prompt Next, use the "reference branch" (only) We obtain a set of logits as a reference and mine the top-K hard negative classes for each sample from it. Then, mixed prompts. True text embedding under the following categories With negative text embedding Construct a contrast set. Difficult negative class mining contrastive loss. Represented as:

[0060]

[0061] in, The temperature coefficient is a trainable parameter. Indicates calculation and The inner product, This represents the visual feature vector of the i-th sample. This represents the text feature vector corresponding to the true category of the i-th sample; Let represent the set of difficult negative classes corresponding to the i-th sample. This can be based on either the top-K values ​​of the reference logits or the statistics of difficult negative classes from the previous training round. Specifically, for each sample, based on the classification logits output by the model, the K classes with the highest predicted scores among all non-true classes are selected as the currently most easily confused negative classes. These classes constitute... ; This represents the text feature vector of the negative class.

[0062] In the current batch, for each actually occurring category Calculate the characteristic mean:

[0063]

[0064] in, For a batch of internal categories The set of sample indexes. The intra-class consistency regularization term is defined as:

[0065]

[0066] in, This represents the actual number of categories appearing in a batch. This is multiplied by the hyperparameter. By incorporating the total loss, this regularization guides similar samples to cluster more closely within the feature space, enhancing inter-class separability.

[0067] In the current mixed prompt Below, based on the model output, the set of difficult samples H is identified (judgment criteria: prediction error or true class prediction probability below a threshold). For each difficult sample Let the true class prediction score be... The strongest negative class prediction score is (The category with the highest model prediction score (logit) among all non-true categories), define the interval:

[0068]

[0069] The refinement loss for hard samples is:

[0070]

[0071] This item is multiplied by the weight When the total loss is included, and the score of the negative class is higher than that of the true class... Those who violate this rule will be penalized to explicitly boost the true class score for hard samples.

[0072] like Figure 1 During the inference phase, an adaptive fusion strategy using two outputs and entropy difference mapping is employed. The core idea is to automatically calculate fusion weights based on the uncertainty difference between the reference cues (partially stable, coarse-grained) and the dynamically trained cues (containing fine-grained information) on the current sample, thereby dynamically allocating the influence of coarse / fine-grained cues according to the sample difficulty. The specific implementation is as follows:

[0073] (1) Reference Tips (Partially Steady): Employs a robust, coarse-grained semantic hint configuration. Specifically, it constructs a fine-grained subfield with little or no weight, defined as follows:

[0074]

[0075] in, and For preset fixed weights (such as =0.7, =0.3, and (From step two) From the dynamic adjustment, P can be seen that g With P m The amount of updates is relatively large in the early stage, and it quickly converges and stabilizes in the early and mid-stages, so it serves as a reference for a relatively stable state.

[0076] (2) Dynamic prompts (Training Mix): Use the final mix cue at the end of training directly as the dynamic cue.

[0077] (3) Uncertainty weight calculation: For the input image x, respectively use and Calculate the classification logits to obtain two probability distributions. Calculate the entropy of these two distributions. and And calculate the entropy difference:

[0078]

[0079] The gap is scaled by a temperature coefficient, mapped using the Sigmoid function, and then linearly transformed to a predefined range. , obtain the fusion weight :

[0080]

[0081] (4) Final suggestion fusion and classification: using adaptive weights A linear mixture of the reference hints and dynamic hints is applied to obtain the sample-specific final hint:

[0082]

[0083] Based on this Text features are generated and their similarity to image features is calculated to obtain the final classification result. This mechanism makes the model tend to rely on more robust reference cues when facing samples with high uncertainty (large entropy differences); while when the model is relatively certain about its own dynamic predictions, it makes more use of dynamic cues containing fine-grained information.

[0084] As one specific implementation method, the various prompt fields in this embodiment are set according to Table 1:

[0085] Table 1 shows the field parameter settings.

[0086]

[0087] In this embodiment, the global cue subfield is used to capture category-independent overall image semantics, and its construction method includes:

[0088] 1. Construct category template text: For each category (In the CHMNIST example, C=16) Generate template text:

[0089]

[0090] 2. Obtain the category embedding using the BiomedCLIP text encoder:

[0091]

[0092] in, A text encoder for BiomedCLIP, used to map natural language phrases to a 512-dimensional embedding space; Let c be the vector representation of category c.

[0093] 3. Calculate the category mean vector:

[0094]

[0095] in, It is the center vector for global hint initialization.

[0096] 4. Initialize the global hint matrix:

[0097]

[0098] In the initialization of the global cue matrix, each row represents a D-dimensional global cue sub-vector. It is obtained by adding a small random perturbation to the center vector of the global cue initialization. In this embodiment, the initialized global cue matrix is ​​represented as follows: Normalize each of its elements, that is:

[0099]

[0100] Meso- and fine-grained cue subfields are used to capture more detailed medical image features, relying on large language models (LLMs, such as ChatGPT-5) to generate descriptions. The specific process includes:

[0101] 1. For each category c:

[0102]

[0103]

[0104] Where G represents the large language model generation function; For mesoscopic prompts, it could be set as: "Describe mesoscopic medical characteristics such as lesion shape, structural pattern, and regional distribution." For finer-grained prompts, it could be set to: "Describe fine-grained visual details such as texture irregularities, boundary sharpness, and subtle intensity variations." For category c, there is a set of meso-level descriptive texts; A fine-grained description of the text set for category c; These are the network parameters for a large language model.

[0105] 2. Embed and normalize the generated descriptive text set, that is:

[0106]

[0107]

[0108]

[0109] in, Represents the i-th mesoscopic text The embedding vector; Represents the i-th fine-grained text The embedding vector; This indicates the calculation of the L2 norm.

[0110] 3. Construct the hint matrix:

[0111] In this embodiment, let the dimension M of the mesoscopic vector be... m Given a value of 10, construct the initial mesoscopic hint matrix, represented as:

[0112]

[0113] In this embodiment, let the dimension M of the fine-grained vector be... f Given a value of 16, construct the initial fine-grained cue matrix, represented as follows:

[0114]

[0115] If there are enough texts, the texts are selected in order to construct the meso-level and fine-grained cue matrices. If there are not enough texts, the missing positions are padded. The i-th padded data in the meso-level cue matrix is ​​the sum of the mean of the elements in the current meso-level cue matrix and a random number. The random number follows a Gaussian distribution with a mean of 0 and a variance of 0.01. Similarly, the i-th padded data in the fine-grained cue matrix is ​​the sum of the mean of the elements in the current fine-grained cue matrix and a random number. The random number follows a Gaussian distribution with a mean of 0 and a variance of 0.01.

[0116] This embodiment also proposes a two-layer cross-attention calculation method in the field controller. The process of processing the first cue vector and the second cue vector cue field is exactly the same. This embodiment uses the first cue vector... Taking this as an example, the specific steps include:

[0117] 1. Training cue matrix:

[0118]

[0119] Each row corresponds to a hint vector (6 dimensions globally + 10 dimensions at the meso level + 16 dimensions at the fine-grained level), and each vector has 512 dimensions.

[0120] 2. Visual feature matrix:

[0121]

[0122] Each row is a patch embedding vector, with each vector having a dimension of 512, obtained through the BiomedCLIP visual encoder.

[0123] 3. In the first layer of cross-attention:

[0124] The global cue field, the meso-level cue field, and the fine-grained cue field are used as the first cue vector. The first cue vector is then linearly transformed to map into the first query matrix, i.e.:

[0125]

[0126] in, This is the first query matrix. The weight vector is used to map the first query matrix, where each row of the first query matrix is ​​a hint vector;

[0127] Each visual patch is based on a weight vector used to map the first key matrix and the first value matrix. , Mapping yields the first key matrix Second-value matrix ,Right now:

[0128]

[0129]

[0130] For each cue vector (32 rows) and each visual (In this example, there are 196 rows) Calculate the inner product, and then use... Scaling is performed to avoid Gradient vanishing or exploding, calculate the first cue-visual similarity matrix. ,Right now:

[0131]

[0132] Among them, the first hint - visual similarity matrix The element in the i-th row and j-th column represents the i-th cue vector for the j-th visual cue. Similarity;

[0133] Then, execute for each line. This ensures that each cue vector corresponds to each visual input. The sum of the attention weights is 1, and the first attention weight matrix is... Represented as:

[0134]

[0135] Wherein, the first attention weight matrix The element in the i-th row and j-th column represents the i-th cue vector for the j-th visual cue. The proportion of attention;

[0136] Next, the attention weight matrix AND-value matrix Multiplying them together yields the visual information representation enhanced by the first cue vector. , represented as:

[0137]

[0138] Among them, the visual information representation after the first cue vector enhancement The i-th row represents the representation of the i-th cue vector combined with the visual context;

[0139] Finally, the original cue matrix and the attention output are residually concatenated and then normalized at the layer, i.e.:

[0140]

[0141] Features obtained after residual concatenation .

[0142] 4. In the second layer of cross-attention:

[0143] The second layer of cross-attention is re-projected through the new query matrix. Key projection matrix Value projection matrix Projection yields the second query matrix Second bond matrix Second-value matrix ,Right now:

[0144]

[0145]

[0146]

[0147] This layer of cross-attention allows cue vectors to interact more deeply with visual features; similarly, the second cue-visual similarity is calculated. The calculation process also uses Scaling and normalization are performed, i.e.:

[0148]

[0149]

[0150] Next, the obtained first attention weight matrix is ​​used. For the second value matrix Weighted summation yields the visual information representation enhanced by the second cue vector. , represented as:

[0151]

[0152] Finally, the visual information enhanced by the second cue vector is represented. and Perform residual connections and normalization to obtain visual context information. ,Right now:

[0153]

[0154] This embodiment utilizes information including the original prompt information and visual context information. Construct a text sequence, that is:

[0155]

[0156] in, This serves as a marker for the start of the sequence; Indicates a splicing operation; Indicates category embedding; This is a marker indicating the end of the sequence.

[0157] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A medical image classification method based on a dynamic multi-granularity hierarchical cue field, characterized in that, Specifically, the following steps are included: The medical image to be classified is input into the visual encoder to obtain visual features, and the visual token obtained by the visual encoder is used as the input of the field controller. In the field controller, three different cue fields are constructed through global cue, meso cue, and fine-grained cue. The global cue field, meso cue field, and fine-grained cue field are used as the first cue vector, and the meso cue field and fine-grained cue field are used as the second cue vector. Each row of each cue vector is a cue field. Using visual tokens as key vectors and value vectors, and the first and second cue vectors as cue fields as query vectors, a two-layer cross-attention calculation is performed in the field controller to obtain two text features. The visual features and the two text features are concatenated separately, and then the entropy values ​​of the classification probability distributions of the two concatenated vectors are calculated separately. The entropy difference is scaled by a temperature coefficient and then input into the Sigmoid function. After linear mapping, it is applied to the set minimum-maximum adaptive weight space to obtain the adaptive cue fusion weight. The classification probability distributions corresponding to the two obtained text features are weighted and fused using adaptive cue fusion weights to obtain the final image classification result.

2. The medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 1, characterized in that, The construction of the global cue field includes: After concatenating the global-level cue words with the category name of each category, the embedding vector of the corresponding category at the global level is obtained through a text encoder. Calculate the average value of all classification embedding vectors at the global granularity, and use it as the initial center vector at the global granularity; If the length of the input visual token is Mg, then a column vector of length Mg is initialized based on the center vector, and each element of the column vector is the sum of the center vector and a random perturbation; Normalize each element and output the initialization hint vector at the global granularity.

3. The medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 2, characterized in that, The random disturbances follow a normal distribution with a mean of 0 and a variance of 0.

01.

4. The medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 1, characterized in that, The construction of the meso-level cue field and the fine-grained cue field includes: Obtain the prompt text at the current granularity, generate an embedding vector based on the text encoder, and normalize the embedding vector; Based on the current granularity's cue vector length N, extract the first N elements from the embedding vector as the initial cue vector for the current granularity.

5. A medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 4, characterized in that, If the number of elements in the embedding vector is less than N, the missing part is supplemented by the sum of the mean of the embedding vector and a random number.

6. The medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 1, characterized in that, The process of data processing based on two-level cascaded cross-attention includes: The cue vector is mapped to the first query vector, and the visual token is mapped to the value vector and the key vector. The first cue-visual similarity matrix is ​​calculated based on the cross-attention mechanism. After normalizing each row of the first cue-visual similarity matrix using the softmax function, the value vector is weighted using this matrix. The weighted value vector is then concatenated with the input cue vector using residuals and normalized again to obtain the intermediate text features. The intermediate text features are mapped to the second query vector, and the visual token is mapped to the value vector and key vector. The second cue-visual similarity matrix is ​​calculated based on the cross-attention mechanism. After normalizing each row of the second cue-visual similarity matrix using the softmax function, the value vector is weighted using this matrix. The weighted vector is then residually concatenated with the intermediate text features and normalized again to obtain the text features.

7. The medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 1, characterized in that, During training, if the total number of training epochs is E, defining the time step scaling factor during the e-th training epoch specifically includes the following steps: Calculate the weights of each cue field using the time step scaling factor: The first hint vector is represented as: in, This represents the time step scaling factor for the e-th training iteration; To indicate the initial lower bound at the fine-grained granularity level; To increase the margin; The weights of the global cue field during the e-th training iteration; The weights of the mesoscopic cue field during the e-th training iteration; Provide the weights of the field with fine-grained hints during the e-th training iteration; This is the preset global prompt ratio. The preset mesoscopic prompt ratio coefficient, and ; This is the first cue vector for the e-th training iteration; For global prompt field; This is a mid-level indicator field; Provides fine-grained cue fields.

8. A medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 1 or 7, characterized in that, During training, the network parameters are updated with the goal of minimizing the total loss function, which is expressed as: in, This is the total loss function; For classification crossover loss; For difficult negative classes, comparative loss is extracted. For the difficult negative class, extract the weights of the contrast loss; For intra-class consistency regularization, The weight of the intra-class consistency regularization; For the refinement loss of difficult samples, The weights for refining the loss for difficult samples.

9. A medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 8, characterized in that, Difficult Negative Class Mining Comparative Loss Represented as: Where B is the batch size; Temperature coefficient; Indicates calculation and The inner product, This represents the visual feature vector of the i-th sample. This represents the text feature vector corresponding to the true category of the i-th sample; Let i represent the set of difficult negative classes corresponding to the i-th sample. This represents the text feature vector of the negative class.

10. A medical image classification method based on a dynamic multi-granularity hierarchical cue field according to claim 8, characterized in that, Difficult sample refinement loss Represented as: in, To identify the number of samples in the difficult sample set H; This represents the prediction score of the i-th sample in the set of difficult samples H. The interval between the predicted score and the logit label.

Images