A multi-label medical image diagnosis method and device

Abstract

The application discloses a multi-label medical image diagnosis method and device, and belongs to the technical field of computer vision. The method comprises the following steps: performing multi-label feature extraction and reasoning on a medical image by using a trained medical image diagnosis model to obtain the prediction probability of each disease label and pathological evidence features; the model comprises the following units: a multi-scale bidirectional cross-attention fusion unit, which is used for performing cross-scale aggregation on the fused features at each scale to obtain multi-scale joint features; a medical knowledge graph embedding unit, which is used for constructing a label correlation graph and generating label embedding based on the semantic embedding of the fused disease labels and the statistical rules of the co-occurrence of each disease; a Transformer decoding unit, which is used for taking the label embedding as an initial query vector, simulating the clinical correlation between diseases through multi-head self-attention, and retrieving the corresponding pathological evidence features from the multi-scale joint features through cross-attention. The accuracy of the auxiliary diagnosis system in clinical chest radiograph screening is improved.

Description

Technical Field

[0001] This application belongs to the field of computer vision technology, and specifically relates to a multi-label medical image diagnosis method and device. Background Technology

[0002] In clinical medical image analysis, the automated diagnosis of chest X-ray images (CXR) often faces the challenge of multi-label classification, meaning that multiple lesions often coexist in a single chest X-ray. Traditional deep learning methods mainly rely on convolutional neural networks (CNNs) to extract image features. Although CNNs have advantages in capturing local textures of lesions, their limited receptive fields make it difficult to effectively model the long-range spatial dependencies between various anatomical structures in the human body.

[0003] While the visual transformer (ViT) introduced in recent years can establish a global context through a self-attention mechanism, it often suffers from insufficient recognition sensitivity when dealing with fine-grained lesions (such as small nodules or fibrosis) with low contrast and extremely weak features due to the lack of local feature sensing bias. This makes it difficult to achieve the optimal balance between global anatomical structure and local abnormalities.

[0004] On the other hand, complex clinical logic exists between disease categories in multi-label chest X-ray recognition; for example, cardiomegaly is often accompanied by pulmonary edema. Most current automated diagnostic systems treat different diseases as independent classification tasks, ignoring co-occurrence patterns and mutual exclusion logic among symptoms. While some studies attempt to assist in judgment by statistically analyzing label co-occurrence frequencies, such methods rely excessively on the statistical distribution of the training set and lack explicit incorporation of medical semantic knowledge. This results in weak generalization and logical reasoning abilities for automated diagnostic systems when faced with long-tailed distributions of rare diseases or cross-hospital datasets, failing to fully simulate the process by which radiologists combine medical common sense with visual evidence for comprehensive judgment.

[0005] In summary, existing medical imaging diagnostic methods suffer from several problems, including difficulty in balancing global and local features, neglect of clinical correlations with diseases, lack of guidance from medical knowledge, and weak cross-scenario generalization and logical reasoning abilities. Summary of the Invention

[0006] To address the aforementioned technical challenges, this application provides a multi-label medical imaging diagnostic method and device that achieves deep alignment and complementarity between local texture details and global anatomical background information in images. Furthermore, in complex scenarios with multiple co-occurring lesions, it can explicitly utilize medical semantic knowledge and disease association patterns for logical constraints, thereby improving the detection sensitivity of low-contrast, fine-grained lesions (such as nodules and fibrosis), enhancing the consistency of multi-label predictions, and ultimately significantly improving the accuracy and robustness of computer-aided diagnostic systems in large-scale clinical chest X-ray screening.

[0007] This application provides a multi-label medical imaging diagnostic method, including: Input the medical images to be diagnosed into the trained medical image diagnosis model; By using a trained medical image diagnostic model, multi-label feature extraction and inference are performed to obtain the predicted probability of each disease label and the corresponding pathological evidence features. The trained medical image diagnosis model includes: a heterogeneous feature extraction unit, a multi-scale bidirectional cross-attention fusion unit, a medical knowledge graph embedding unit, a Transformer decoding unit, and a classification output unit. The heterogeneous feature extraction unit is used to extract multi-scale hierarchical features and global contextual features in parallel from the input medical images; The multi-scale bidirectional cross-attention fusion unit is used to scale-align the extracted hierarchical features at each scale with the global context features, obtain the fused features at each scale through bidirectional cross-attention, and perform cross-scale aggregation on the fused features at each scale to obtain multi-scale joint features. The medical knowledge graph embedding unit is used to construct a label association graph based on the semantic embedding of fused disease labels and the statistical regularity of the co-occurrence of various diseases in the training set. The dependency relationship between labels is propagated through a graph convolutional network to generate label embeddings containing medical logic. The Transformer decoding unit is used to embed the labels as the initial query vector, simulate the clinical association between diseases through multi-head self-attention, and then retrieve the corresponding pathological evidence features from the multi-scale joint features through cross-attention and output them to the feedforward neural network. The classification output unit is used to output the predicted probability of each disease label and the corresponding pathological evidence features based on the feedforward neural network.

[0008] Furthermore, the heterogeneous feature extraction unit includes parallel convolutional feature extraction units and transformer feature extraction units, with the convolutional feature extraction unit adopting the DenseNet network architecture; The convolutional feature extraction unit is used to process the input medical images, extracting intermediate convolutional features from shallow networks that preserve high-resolution spatial local details, and deep semantic features from deep networks that capture abstract semantics. The transformer feature extraction unit is used to segment medical images into image sequence blocks and extract global contextual features containing global anatomical topology through a self-attention mechanism.

[0009] Furthermore, the multi-scale bidirectional cross-attention fusion unit includes: The first projection layer is used to project the global context features into the first query features through a linear transformation, and to project the intermediate convolutional features into the first key features and the first value features through a linear transformation, respectively. The first cross-attention layer is used to calculate the similarity between the first query feature and the first key feature and obtain the first attention weight by softmax normalization. The first value feature is weighted by the first attention weight and the first fusion feature is output. The second projection layer is used to project the global context features into the second query features through a linear transformation, and to project the deep semantic features into the second key features and the second value features through a linear transformation, respectively. The second cross-attention layer is used to calculate the similarity between the second query feature and the second key feature and obtain the second attention weight by softmax normalization. The second value feature is weighted by the second attention weight and the second fusion feature is output. The aggregation layer is used to add the first fused feature and the second fused feature element by element, and then add the sum to the scale-aligned global context feature element by element again to output the multi-scale joint feature.

[0010] Furthermore, the medical knowledge graph embedding unit is specifically used to calculate the co-occurrence matrix based on the common occurrence frequency of each disease label in the training set, and to construct an undirected adjacency matrix and initial node features by combining the text semantic vectors of each disease label name; the undirected adjacency matrix and initial node features are input into a propagation structure containing a multi-layer graph convolutional network, and feature message passing is performed through the normalized undirected adjacency matrix to calculate and output a label embedding that integrates disease co-occurrence logic and mutual exclusion rules.

[0011] Furthermore, the Transformer decoding unit is specifically used to directly map the embedded labels to the initial query vector of the decoder layer. Within the decoder layer, a self-attention operation is performed on the initial query vector to enable information interaction between query vectors of different diseases, thereby modeling the co-occurrence or mutual exclusion associations between disease labels and obtaining an updated query vector. Using the updated query vector as the query and the multi-scale joint features as the key and value, a cross-attention retrieval operation is performed to enable each disease label to retrieve the corresponding key pathological region features in the multi-scale joint features, and output the pathological evidence features to the feedforward neural network.

[0012] Furthermore, the medical image diagnostic model is trained and updated using a multi-task composite loss function, which is:

[0013] in, It is a compound loss. It is an asymmetric loss, used to balance positive and negative samples; It is a graph Laplacian regularization loss used to ensure the consistency of predictions for related labels; It is a pairwise ranking loss, used to improve the model's ability to distinguish difficult-to-diagnose diseases; It is a binary cross-entropy loss function, used to constrain the basic deviation between the predicted probabilities of each category and the true labels; It is an asymmetric loss weight. These are graph Laplacian regularized weights. It is a pairwise sorting loss weight. These are the weights of the binary cross-entropy loss, and the sum of the four weights is 1.

[0014] Furthermore, the medical images to be diagnosed include any one of chest X-ray images, chest CT images, or chest ultrasound images.

[0015] This application also provides an electronic device, including: Memory, which stores execution instructions; and The processor executes the instructions stored in memory, causing the processor to perform the methods described above.

[0016] This application also provides a readable storage medium storing executable instructions, which are executed by a processor to implement the above-described method.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0018] Compared with the prior art, this application has the following advantages: 1. By employing a multi-scale bidirectional cross-attention fusion (DCAF) unit, the limitations of traditional CNNs (limited receptive field) and pure Transformers (lack of local inductive bias) are overcome. The model utilizes the global anatomical background to guide the accurate extraction of local lesions, while fine-grained texture feedback corrects global judgments, significantly improving the model's sensitivity in detecting low-contrast, small lesions (such as lung nodules and fibrosis).

[0019] 2. An explicit logical reasoning mechanism based on medical semantics and co-occurrence patterns is proposed. By constructing a label association graph and using a Graph Convolutional Network (GCN) to generate label embeddings, the traditional method overcomes the limitation of treating diseases as independent classification tasks. Clinical medical knowledge and label statistical patterns are directly injected into the decoder query vector, enabling the model to simulate the clinical thinking of doctors, effectively capturing co-occurrence (such as cardiomegaly accompanied by pulmonary edema) or mutual exclusion logic between diseases, and enhancing the consistency and logical rigor of diagnostic results in complex multi-label scenarios.

[0020] 3. A precise evidence retrieval model based on label-aware queries was constructed. A graph-guided Transformer decoder architecture was adopted, using label-aware query vectors for interactive retrieval within the fused multi-scale joint features. This achieved a "semantic evidence-finding" diagnostic model, automatically locating and focusing on key pathological regions corresponding to each disease in the multi-scale feature space through a cross-attention mechanism. This not only improved the accuracy of multi-label recognition but also enhanced the interpretability of the model in the clinical diagnostic process.

[0021] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart of a multi-label medical image diagnostic method according to an embodiment of this application is shown; Figure 2 A network architecture diagram of a medical image diagnosis model according to an embodiment of this application is shown; Figure 3 A structural diagram of a DCAF unit according to an embodiment of this application is shown; Figure 4 A schematic diagram of a cross-attention structure according to an embodiment of this application is shown. Detailed Implementation

[0024] This application provides a multi-label medical image diagnosis method, which inputs the medical image to be diagnosed into a trained medical image diagnosis model; By using a trained medical image diagnostic model, multi-label feature extraction and inference are performed to obtain the predicted probability of each disease label and the corresponding pathological evidence features. The trained medical image diagnosis model includes: a heterogeneous feature extraction unit, a multi-scale bidirectional cross-attention fusion unit, a medical knowledge graph embedding unit, a Transformer decoding unit, and a classification output unit. The heterogeneous feature extraction unit is used to extract multi-scale hierarchical features and global contextual features in parallel from the input medical images; The multi-scale bidirectional cross-attention fusion unit is used to scale-align the extracted hierarchical features at each scale with the global context features, obtain the fused features at each scale through bidirectional cross-attention, and perform cross-scale aggregation on the fused features at each scale to obtain multi-scale joint features. The medical knowledge graph embedding unit is used to construct a label association graph based on the semantic embedding of fused disease labels and the statistical regularity of the co-occurrence of various diseases in the training set. The dependency relationship between labels is propagated through a graph convolutional network to generate label embeddings containing medical logic. The Transformer decoding unit is used to embed the labels as the initial query vector, simulate the clinical association between diseases through multi-head self-attention, and then retrieve the corresponding pathological evidence features from the multi-scale joint features through cross-attention and output them to the feedforward neural network. The classification output unit is used to output the predicted probability of each disease label and the corresponding pathological evidence features based on the feedforward neural network.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Figure 1 The multi-label medical imaging diagnostic method of the present invention is illustrated in the figure. Figure 2 A network architecture diagram using chest X-rays as medical images, according to an embodiment of this application, is given below. The following is in conjunction with… Figure 1 and Figure 2 The multi-label medical image diagnosis method in this embodiment of the invention is described, including the following steps: S100: The image preprocessing unit normalizes the input raw CXR; In this stage, the image preprocessing unit completes the processing of the raw chest X-ray (CXR) image. Adjust the size (e.g.) or The image undergoes normalization and other standardization processes to conform to the input distribution requirements of deep learning models. Here, H is the image height, W is the image width, and 3 is the number of channels.

[0027] S200: Extracts multi-scale hierarchical features and global contextual features in parallel from standardized medical images; This step S200 specifically includes S201 and S202: S201: The convolutional feature extraction unit adopts the DenseNet network architecture to extract intermediate convolutional features that preserve high-resolution spatial local details from shallow networks, as well as deep semantic features that capture abstract semantics from deep networks. The convolutional feature extraction unit mentioned in step S201 corresponds to Figure 2 The DenseNet branch consists of four dense blocks stacked in series. The convolutional feature extraction unit processes the input standardized medical image, extracting the intermediate convolutional features from the output of the third dense block and the deep semantic features from the output of the fourth dense block. The location of the third dense block mentioned above is the position of the shallow network, and the output is an intermediate convolutional feature that preserves high-resolution local spatial details. The location of the fourth dense block mentioned above represents the position of the deep network, and its output is the deep semantic feature of abstract semantics. .

[0028] S202: The transformer feature extraction unit segments the medical image into image sequence blocks and extracts global contextual features containing global anatomical topology through a self-attention mechanism.

[0029] The converter feature extraction unit mentioned in step S202 corresponds to Figure 2 The ViT branch consists of image segmentation, linear mapping, and a ViT encoder × 12 (12 identical Transformer encoder blocks stacked together). The image is segmented into sequence blocks, and linear mapping and encoding are performed sequentially to extract global contextual features. Where D is the number of feature channels.

[0030] S300: Multi-scale bidirectional cross-attention fusion (DCAF) for the intermediate convolutional features Deep semantic features and the global context features Scale alignment is performed, and the fused features at each scale are obtained through bidirectional cross-attention. The fused features at each scale are then aggregated across scales to obtain multi-scale joint features. Step S300 aims to achieve mutual calibration between local lesions and global anatomical structures through the DCAF unit, specifically including the following steps: S301: Scale Alignment and Projection: The Intermediate Convolutional Features Deep semantic features and the global context features Projecting to a uniform embedding dimension using a linear projection layer; projecting from independent projection layers... Two scale-dependent representations are derived from this. and And adjust their spatial resolution to be the same as and Consistent; S302: Calculate bidirectional cross-attention: bidirectional cross-attention between CNN→ViT and ViT→CNN.

[0031] Figure 3 A structural diagram of a DCAF unit according to an embodiment of this application is shown, and the specific description is as follows: The first projection layer is used to project the global context features. The intermediate convolutional features are then projected onto the first query feature through a linear transformation. After linear transformation, they are projected into the first key feature and the first value feature, respectively. The first cross-attention layer is used to calculate the similarity between the first query feature and the first key feature and obtain the first attention weight by softmax normalization. The first value feature is weighted by the first attention weight and the first fusion feature is output. The second projection layer is used to project the global context features into second query features through a linear transformation, and to project the deep semantic features... After linear transformation, the projections are respectively obtained as second bond features and second value features; The second cross-attention layer is used to calculate the similarity between the second query feature and the second key feature and obtain the second attention weight by softmax normalization. The second value feature is weighted by the second attention weight and the second fusion feature is output. The aggregation layer is used to add the first fused feature and the second fused feature element by element, and then add the sum to the scale-aligned global context feature element by element again to output the multi-scale joint feature.

[0032] The first key feature mentioned above is the intermediate convolution feature. Learnable bond linear projection matrix The first-valued feature obtained by linear transformation is Through a learnable numerical linear projection matrix The first query feature, obtained through linear transformation, is a global context feature. Learnable query linear projection matrix The second key feature is obtained through linear transformation. go through The second-valued feature obtained by linear transformation is go through The second query feature is obtained through linear transformation. go through Obtained by linear transformation; Figure 4The diagram shown is a cross-attention illustration according to an embodiment of this application, and its working principle is as follows: The correlation between the query feature matrix and the key feature matrix is ​​calculated, normalized using Softmax, and then weighted and aggregated using the value feature matrix.

[0033] At each scale Above, we calculate the bidirectional attention between convolutional features and transformer features; the cross-attention formula is defined as follows:

[0034] in, To query the feature matrix, The source feature matrix, For learnable query linear projection matrix, The learnable bond linear projection matrix. Scaling factor It is a learnable numerical linear projection matrix. To query the feature matrix, The key feature matrix, The characteristic matrix is ​​denoted by .

[0035] Based on this, the characteristics of mutual guidance are calculated:

[0036]

[0037] in, These are intermediate convolutional features guided by ViT. These are the deep semantic features guided by DenseNet.

[0038] S303: Perform cross-scale feature aggregation.

[0039] Low-resolution features Perform upsampling and compare with Aggregate to generate the final multi-scale joint features. This process can be expressed using a formula as follows:

[0040] in, Indicates aggregation, This indicates upsampling.

[0041] S400: Medical knowledge graph embedding unit construction graph guided by label semantic prior.

[0042] Medical knowledge graph embedded unit in Figure 2 Prior knowledge of the corresponding labels.

[0043] Step S400 utilizes medical knowledge and statistical laws to provide initial "knowledge guidance" for the decoder; specifically, it includes the following processes: 1. Construct an undirected adjacency matrix Calculate the co-occurrence matrix based on the common frequency of each symptom in the training set, and construct an undirected adjacency matrix by combining the semantic embedding of the label names (such as Word2Vec or BERT vectors). and initial node features .

[0044] 2. Generate the label query Q using graph convolutional network (GCN) message passing. A two-layer graph convolutional network is used to propagate dependencies between labels and compute the final label embeddings. The propagation logic at each layer is as follows:

[0045] in, For the number of floors, It is a non-linear activation function. For the first The learnable weight matrix of the layer, For the first Layer propagation labels; This is the normalized adjacency matrix.

[0046] S500: The Transformer decoding unit performs label-aware transformer decoding and evidence retrieval.

[0047] Specifically, the Transformer decoding unit uses the label query quantity Q as the initial query vector, simulates the clinical association between diseases through multi-head self-attention, and then retrieves the corresponding pathological evidence features from the multi-scale joint features stored in the feature memory F through cross-attention. In this step S500, the decoder matches the label prior with the visual features to achieve accurate lesion identification. Specifically, it includes the following processes: 1. Query vector initialization The tag embedding obtained in step 3 is used as the initial query vector for the decoder. :

[0048] in, This is the transpose of the embedding vector of the Lth disease label.

[0049] 2. Label Self-Attention Modeling In the decoder layer, firstly... The data is mapped to query, key, and value vectors through linear transformations, respectively. A multi-head attention mechanism is then used to capture the clinical correlations between different conditions (e.g., the co-occurrence relationship between cardiac hypertrophy and pulmonary edema), achieving mutual enhancement of labeled features.

[0050] It is a multi-head attention mechanism function. Is it input to the number The query vector of the layer; 3. Labeled Image Cross-Attention Retrieval Using the updated query vector From multi-scale joint features (Depend on Search for the corresponding pathological evidence in the flattened (obtained) field:

[0051] S600: The classification output unit uses an FFN-based feedforward neural network to output the predicted probability of each disease label and the corresponding pathological evidence features.

[0052] Specifically, the Sigmoid activation function can be used to map the output feature vector to 14 or 15 independent probability values ​​for each category. Each probability value is between 0 and 1, representing the confidence that the sample belongs to the corresponding category. The categories are not mutually exclusive.

[0053] It should be noted that during the model training phase, after S500, model prediction and multi-objective optimization training are performed, the prediction probabilities are calculated, and the parameters are updated according to the composite loss function; specifically, this includes: Step 1: Logistic Regression Prediction The vector output by the decoder Mapped to logits using a shared linear classifier. And use the Sigmoid function to calculate the probability of each label. ; Step 2: Calculation of Composite Loss Define the composite loss function The class imbalance and label noise problems are addressed by weighting multiple loss terms:

[0054] in, It is an asymmetric loss, used to balance positive and negative samples; It is a graph Laplacian regularization loss used to ensure the consistency of predictions for related labels; It is a pairwise ranking loss, used to improve the model's ability to distinguish difficult-to-diagnose diseases; It is a binary cross-entropy loss function, used to constrain the basic deviation between the predicted probabilities of each category and the true labels; It is an asymmetric loss weight. These are graph Laplacian regularized weights. It is a pairwise sorting loss weight. These are the weights of the binary cross-entropy loss, and the sum of the four weights is 1.

[0055] Step 5.3: Parameter Iterative Update Calculate using backpropagation algorithm For model parameters The gradient is calculated, and an optimizer (such as AdamW) is used to iterate continuously during the training cycle until the model converges.

[0056] When the medical image is a chest X-ray image, the intermediate convolutional features correspond to the texture features of pulmonary nodules or fibrosis, the global context features correspond to the anatomical features of the heart contour or mediastinal structure, and the clinical associations between the disease labels include the co-occurrence logic of cardiac hypertrophy and pulmonary edema.

[0057] Based on the same inventive concept as the above disclosure, this disclosure also provides an electronic device. The electronic device of this disclosure includes at least one processor and at least one memory electrically connected to the processor. The memory is electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.

[0058] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. The indirect connection method can be applied to the embodiments of this disclosure as long as it achieves the purpose of this disclosure.

[0059] Based on the same inventive concept, this disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described method.

[0060] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0061] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-label medical imaging diagnostic method, characterized in that, include: Input the medical images to be diagnosed into the trained medical image diagnosis model; The trained medical image diagnostic model is used to extract and infer multi-label features to obtain the predicted probability of each disease label and the corresponding pathological evidence features. The trained medical image diagnosis model includes: a heterogeneous feature extraction unit, a multi-scale bidirectional cross-attention fusion unit, a medical knowledge graph embedding unit, a Transformer decoding unit, and a classification output unit; The heterogeneous feature extraction unit is used to extract multi-scale hierarchical features and global context features in parallel from the input medical images; The multi-scale bidirectional cross-attention fusion unit is used to scale-align the extracted hierarchical features at each scale with the global context features, obtain the fused features at each scale through bidirectional cross-attention, and perform cross-scale aggregation on the fused features at each scale to obtain multi-scale joint features. The medical knowledge graph embedding unit is used to construct a label association graph based on the semantic embedding of fused disease labels and the statistical regularity of the co-occurrence of various diseases in the training set. The dependency relationship between labels is propagated through a graph convolutional network to generate label embeddings containing medical logic. The Transformer decoding unit is used to embed the label as an initial query vector, simulate the clinical association between diseases through multi-head self-attention, and then retrieve the corresponding pathological evidence features from the multi-scale joint features through cross-attention and output them to the feedforward neural network. The classification output unit is used to output the predicted probability of each disease label and the corresponding pathological evidence features based on the feedforward neural network.

2. The method according to claim 1, characterized in that, The heterogeneous feature extraction unit includes a parallel convolutional feature extraction unit and a transformer feature extraction unit, wherein the convolutional feature extraction unit adopts the DenseNet network architecture; The convolutional feature extraction unit is used to process the input medical image, extract intermediate convolutional features from the shallow network that preserve high-resolution spatial local details, and deep semantic features from the deep network that capture abstract semantics. The transformer feature extraction unit is used to segment medical images into image sequence blocks and extract global contextual features containing global anatomical topology through a self-attention mechanism.

3. The method according to claim 2, characterized in that, The multi-scale bidirectional cross-attention fusion unit includes: The first projection layer is used to project the global context features into a first query feature through a linear transformation, and to project the intermediate convolutional features into a first key feature and a first value feature through a linear transformation, respectively. The first cross-attention layer is used to calculate the similarity between the first query feature and the first key feature and obtain the first attention weight by softmax normalization. The first value feature is weighted by the first attention weight and the first fusion feature is output. The second projection layer is used to project the global context features into the second query features through a linear transformation, and to project the deep semantic features into the second key features and the second value features through a linear transformation, respectively. The second cross-attention layer is used to calculate the similarity between the second query feature and the second key feature and obtain the second attention weight by softmax normalization. The second value feature is weighted by the second attention weight and the second fusion feature is output. The aggregation layer is used to add the first fused feature and the second fused feature element by element, and then add the sum to the scale-aligned global context feature element by element again to output the multi-scale joint feature.

4. The method according to claim 1, characterized in that, The medical knowledge graph embedding unit is specifically used to calculate the co-occurrence matrix based on the common occurrence frequency of each disease label in the training set, and to construct an undirected adjacency matrix and initial node features by combining the text semantic vectors of each disease label name. The undirected adjacency matrix and initial node features are input into a propagation structure containing a multi-layer graph convolutional network. Feature message passing is performed through the normalized undirected adjacency matrix, and a label embedding that integrates disease co-occurrence logic and mutual exclusion rules is calculated and output.

5. The method according to claim 1, characterized in that, The Transformer decoding unit is specifically used to embed the label and directly map it to the initial query vector of the decoder layer. Within the decoder layer, a self-attention operation is performed on the initial query vector to enable information interaction between query vectors of different diseases, thereby modeling the co-occurrence or mutual exclusion association between disease labels and obtaining an updated query vector. Using the updated query vector as the query and the multi-scale joint features as the key and value, a cross-attention retrieval operation is performed to retrieve the corresponding key pathological region features for each disease label in the multi-scale joint features, and output the pathological evidence features to the feedforward neural network.

6. The method according to any one of claims 1 to 5, characterized in that, The medical image diagnosis model is trained and updated using a multi-task composite loss function, which is: in, It is a compound loss. It is an asymmetric loss, used to balance positive and negative samples; It is a graph Laplacian regularization loss used to ensure the consistency of predictions for related labels; It is a pairwise ranking loss, used to improve the model's ability to distinguish difficult-to-diagnose diseases; It is a binary cross-entropy loss function, used to constrain the basic deviation between the predicted probabilities of each category and the true labels; It is an asymmetric loss weight. These are graph Laplacian regularized weights. It is a pairwise sorting loss weight. These are the weights of the binary cross-entropy loss, and the sum of the four weights is 1.

7. The method according to claim 1, characterized in that, The medical images to be diagnosed include any one of chest X-ray images, chest CT images, or chest ultrasound images.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images