Double-flow collaborative transfer learning method for automatic staging of chest X-ray image
By employing a dual-stream collaborative transfer learning approach, utilizing a frozen ViT-B/16 backbone network and trainable ViT side networks, combined with cross-attention cueing and feature fusion modules, the problems of high computational complexity and overfitting in automatic staging of chest X-ray images are solved, thereby improving the model's generalization ability and device adaptability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TAIYUAN UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2026-02-06
- Publication Date
- 2026-05-19
AI Technical Summary
Existing deep learning methods have high computational complexity and large memory consumption in automatic staging of chest X-ray images. They are also prone to overfitting in small medical sample scenarios, making them difficult to deploy on conventional medical equipment. Furthermore, the differences in imaging equipment among different medical institutions affect the model's generalization ability.
A two-stream collaborative transfer learning approach is adopted, using a frozen ViT-B/16 as the backbone network and introducing a trainable ViT as a side network. Feature interaction and decoupling are achieved through a cross-attention cue fusion module and a feature mixing module, and classification is performed in conjunction with the Morlet-KAN classifier.
It effectively reduces computational complexity and memory consumption, avoids the risk of overfitting, and improves the model's generalization ability under different devices and imaging conditions.
Smart Images
Figure CN122067013A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a two-stream collaborative transfer learning method for automatic staging of chest X-ray images. Background Technology
[0002] Clinical staging of pneumoconiosis heavily relies on the accurate identification of subtle lesions (such as small nodules, reticular shadows, and abnormal lung texture) in high-resolution chest X-ray (HR-CXR) images. However, in practical applications, existing deep learning methods still face multiple challenges in this task: on the one hand, the original HR-CXR images have extremely high resolution, and directly inputting them into mainstream vision models will lead to a sharp increase in computational complexity and memory consumption; on the other hand, although large-scale pre-trained vision models (such as VisionTransformer, ViT) have strong feature representation capabilities, full-parameter fine-tuning in small medical sample scenarios can easily lead to overfitting, while also incurring significant computational and storage overhead, making them difficult to deploy on conventional medical equipment. In addition, differences in imaging equipment, acquisition protocols, and post-processing procedures among different medical institutions can significantly affect the grayscale distribution, contrast, and noise characteristics of chest X-rays, thereby limiting the generalization ability of the model. Summary of the Invention
[0003] The purpose of this invention is to provide a dual-stream collaborative transfer learning method for automatic staging of chest X-ray images, aiming to solve or improve at least one of the above-mentioned technical problems.
[0004] To achieve the above objectives, the present invention provides the following solution: A two-stream collaborative transfer learning method for automatic staging of chest X-ray images includes: Labeled chest X-ray images were acquired, preprocessed, and then divided into training and testing sets. A two-stream collaborative model is constructed; a frozen ViT-B / 16 is used as the backbone network; a trainable ViT is used as the side network; a cross-attention cue fusion module is introduced, which interacts with the patch tokens of the backbone network through learnable cue vectors, and the cue is projected onto the side network and concatenated with the category tokens of the intermediate layers; a feature mixing module is introduced, which decouples the structure and statistics of the category tokens output by the side network and performs asymmetric recombination on the final output of the side network during the training phase; a Morlet-KAN classifier is introduced to classify the features; After training and testing the two-stream collaborative model using the training and test sets, the final two-stream collaborative model is obtained; the feature fusion module is turned off in both the testing phase and the final two-stream collaborative model. The final dual-stream collaborative model was used to analyze the chest X-ray images to obtain the image recognition results.
[0005] Further preprocessing includes: The chest X-ray image was segmented to obtain a high-resolution image by the pneumoconiosis region; then, the high-resolution image was downsampled using bilinear interpolation to obtain a low-resolution image. The high-resolution image and the low-resolution image are divided into non-overlapping image blocks, and then a high-resolution token sequence and a low-resolution token sequence are generated by linear projection.
[0006] Furthermore, the frozen ViT-B / 16, obtained by pre-training a VisionTransformer Base on ImageNet-21K, has all parameters frozen and only performs forward inference.
[0007] Furthermore, ViT can be trained, and gradient backpropagation and parameter updates can be performed normally during the training process.
[0008] Furthermore, the execution flow of the cross-attention cue fusion module includes: Initialize a set of learnable cue vectors ; The learnable cue vectors are layer-normalized and used as queries. All patch tokens output by the backbone network are used as keys and values for multi-head cross-attention computation, expressed as: In the formula, For the extracted cue features; This is a multi-head cross-attention mechanism; A learnable cue vector; For layer normalization; The patch tokens output by the backbone network; The prompt features After projection, the class tokens are temporarily concatenated with the intermediate layers of the side network to inject semantic knowledge. After semantic guidance is completed, the prompt features are removed from the sequence.
[0009] Furthermore, the execution flow of the feature blending module includes: Any two feature sequences output by the inner-batch network are decomposed into structural components and statistical moments using strength normalization, as expressed by: In the formula, and These are feature sequence A and feature sequence B, respectively. and These are the mean and standard deviation of the characteristic sequence A, respectively; This refers to the structural information of the feature sequence A after normalization. and These are the mean and standard deviation of the characteristic sequence B, respectively. This refers to the structural information of the feature sequence B after normalization. The structural information of feature sequence A Recombining the mean and standard deviation of feature sequence B, a mixed feature is generated, expressed as: In the formula, It is a mixed feature.
[0010] Furthermore, the Morlet-KAN classifier employs a single-hidden-layer Kolmogorov–Arnold Network structure, and the activation function uses a learnable Morlet wavelet basis function, expressed as: In the formula, These are Morlet wavelet basis functions; These are the normalized input features; Use a Gaussian window; The center frequency; It is an oscillating core.
[0011] Furthermore, the execution flow of the Morlet-KAN classifier is as follows: The Morlet-KAN classifier maps input features to a 3D latent space, and the expression for the i-th latent unit is: In the formula, is the activation value of the i-th hidden unit; The dimension of the input feature; The weights are the edge weights of the first layer. These are Morlet wavelet basis functions; For scaling parameters; These are translation parameters; The dimension of the implicit space; The classification result of the chest X-ray image is obtained by mapping the latent representation through a linear layer. The expression of the k-th output component is: In the formula, This is the output value for the k-th category; The dimension of the implicit representation; Let be the activation value of the i-th hidden unit; The second-layer linear weights represent the contribution strength of the i-th hidden unit to the prediction of the k-th class. Output category index; This represents the number of output categories.
[0012] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a dual-stream collaborative transfer learning method for automatic staging of chest X-ray images. The method avoids the sequence explosion problem caused by directly inputting the original high-resolution image into a large Transformer through a complete freezing strategy of the backbone network and the collaborative design of the side networks. At the same time, it bypasses the severe memory pressure and overfitting risk caused by full fine-tuning of the ViT model with hundreds of millions of parameters in medical small sample scenarios. Attached Figure Description
[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0014] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the dual-flow cooperative model in this embodiment. Detailed Implementation
[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0016] The purpose of this invention is to provide a dual-stream collaborative transfer learning method for automatic staging of chest X-ray images, aiming to solve or improve at least one of the above-mentioned technical problems.
[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0018] like Figure 1 As shown, this invention provides a two-stream collaborative transfer learning method for automatic staging of chest X-ray images, comprising: Step 1: Obtain labeled chest X-ray images, perform preprocessing, and divide them into training and test sets; Preprocessing includes: segmenting the pneumoconiosis portion of the 2048×2048 chest X-ray image and scaling it down to 512×512 to obtain a high-resolution image; and downsampling the high-resolution image to 128×128 using bilinear interpolation to obtain a low-resolution image. The high-resolution image and the low-resolution image are divided into non-overlapping 16×16 image blocks, and then a high-resolution token sequence and a low-resolution token sequence are generated by linear projection.
[0019] like Figure 2 As shown, in step 2, a two-stream collaborative model is constructed; a frozen ViT-B / 16 is used as the backbone network; a trainable ViT is used as the side network; a cross-attention cue fusion module is introduced, which interacts with the patch tokens of the backbone network through the learnable cue vector, and the cue is projected onto the side network and concatenated with the category tokens of the intermediate layer; a feature mixing module is introduced, which decouples the structure-statistical moments and performs asymmetric recombination on the category tokens output by the side network during the training phase; and a Morlet-KAN classifier is introduced to classify the features. The frozen ViT-B / 16, including a 12-layer Transformer encoder, has an output dimension of 768. A Vision Transformer Base (ViT-B / 16) pre-trained on ImageNet-21K is used, with all parameters frozen and only forward inference performed to output high-resolution semantic features. .
[0020] The trainable ViT includes a 6-layer Transformer encoder with an embedding dimension of 384; during training, gradient backpropagation and parameter updates are performed normally, outputting a low-resolution feature sequence. .
[0021] The execution flow of the cross-attention cue fusion module includes: Initialize a set of 16 learnable cue vectors ; The learnable cue vectors are layer-normalized (LayerNorm) and used as a query. All patch tokens output by the backbone network are used as keys and values for multi-head cross-attention computation, expressed as: In the formula, For the extracted cue features; This is a multi-head cross-attention mechanism; A learnable cue vector; For layer normalization; The patch tokens output by the backbone network; The prompt features After projection to 384 dimensions, it is temporarily concatenated with the category tokens of the intermediate layers of the side network to inject semantic knowledge. After semantic guidance is completed, the suggested features are then... Remove from the sequence to keep the length of the side network output sequence constant and avoid increasing memory overhead.
[0022] The execution flow of the feature blending module includes: Any two low-resolution feature sequences output by the intra-batch network are decomposed into structural components and statistical moments using strength normalization, as expressed by: In the formula, and These are feature sequence A and feature sequence B, respectively. and These are the mean and standard deviation of the characteristic sequence A, respectively; This refers to the structural information of the feature sequence A after normalization. and These are the mean and standard deviation of the characteristic sequence B, respectively. This refers to the structural information of the feature sequence B after normalization. The structural information of feature sequence A Recombining the mean and standard deviation of feature sequence B, a mixed feature is generated, expressed as: In the formula, It is a mixed feature.
[0023] The aforementioned feature fusion module achieves "structure-distribution" decoupling in the feature space, effectively mitigating image style differences caused by different hospitals and equipment, and improving the model's robustness in distinguishing out-of-distribution samples.
[0024] The Morlet-KAN classifier uses a single-hidden-layer Kolmogorov–Arnold Network (KAN) structure, and the activation function is a learnable Morlet wavelet basis function, expressed as: In the formula, Morlet wavelet basis functions are used as nonlinear activation functions for each edge (i.e., each input-hidden unit connection) in the KAN network. Compared with the fixed activation function (such as ReLU) of traditional MLP, the activation function of KAN is a learnable function that can be parameterized in spline or wavelet form, and has a stronger function approximation ability. These are the normalized input features; Gaussian windows provide localization constraints to ensure that wavelets have tight support in the time domain (or spatial domain) and avoid infinite oscillations. The center frequency controls the speed of the cosine oscillation and is a key shape parameter of the Morlet wavelet; As the oscillation kernel, it endows the function with the ability to model periodic structures, and is particularly suitable for capturing quasi-periodic pathological textures such as reticular shadows, nodular arrangements, and fibrous strands commonly found in medical images.
[0025] The execution flow of the Morlet-KAN classifier is as follows: The Morlet-KAN classifier takes the input features Mapped to a 64-dimensional latent space, the expression for the i-th latent unit is: In the formula, Let be the activation value of the i-th hidden unit; The dimension of the input feature; represents the edge weights of the first layer, corresponding to the connection strength from input dimension j to hidden unit i; These are Morlet wavelet basis functions; The scaling parameter controls the frequency sensitivity of the Morlet wavelet on the input axis; The translation parameter controls the center position of the Morlet wavelet on the input axis, determining the wavelet's effect on the input features. Which range of values is most sensitive? The dimension of the implicit space; The 64-dimensional latent representation is mapped to a 4-dimensional classification using a linear layer. The expression for the k-th output component is: In the formula, The output value for the kth category is represented by the four dimensions, which correspond to the four diagnostic results of pneumoconiosis: no pneumoconiosis, stage I, stage II, and stage III. Let be the activation value of the i-th hidden unit; The second-layer linear weights represent the contribution strength of the i-th hidden unit to the prediction of the k-th class. This is the output category index.
[0026] Step 3: After training and testing the two-stream collaborative model using the training and test sets, the final two-stream collaborative model is obtained; the feature fusion module is turned off in both the testing phase and the final two-stream collaborative model. During the training phase, the optimizer uses AdamW with an initial learning rate of... Weight decay The batch size is 8, and the training takes 40 rounds.
[0027] During training, only the following learnable parameters are updated: All weights of the side network; the 16 initialized cue vectors; the query, construct, value, and output projection matrices in the cross-attention cue fusion module; the affine transformation parameters in the feature moment mixing module (such as scaling and offset terms in layer normalization, if enabled); all parameters in the Morlet-KAN classifier, including the scaling parameters of the Morlet wavelet basis. Translation parameters Center frequency and the two-layer connection weights and All parameters of the backbone network remain frozen and do not participate in gradient updates.
[0028] Step 4: Analyze the chest X-ray image using the final dual-stream collaborative model to obtain the image recognition results, including: Step 41: After segmenting the pneumoconiosis portion of the chest X-ray image, the image is scaled to obtain a high-resolution image; and the high-resolution image is downsampled using bilinear interpolation to obtain a low-resolution image. Step 42: Divide the high-resolution image and the low-resolution image into non-overlapping image blocks, and generate a high-resolution token sequence and a low-resolution token sequence through linear projection. Step 43: Input the high-resolution token sequence into the backbone network; input the low-resolution token sequence into the side network; Step 44, the cross-attention cue fusion module uses the learnable cue vector after layer normalization as the query and all patch tokens of the backbone network as the sum value to perform multi-head cross-attention calculation to obtain cue features; Step 45: After projecting the cue features, temporarily concatenate them with the category tokens of the intermediate layer of the side network to inject semantic knowledge. After completing the semantic guidance, remove the cue features from the sequence. Step 46: The Morlet-KAN classifier classifies the category tokens of the final output of the side network to obtain the analysis results of the chest X-ray image.
[0029] The technical solution of this invention, with its complete freezing strategy of the backbone network and the collaborative design of the side networks in the dual-stream architecture, avoids the sequence explosion problem caused by directly inputting the original high-resolution image into a large Transformer. At the same time, it bypasses the severe memory pressure and overfitting risk caused by fully fine-tuning the ViT model with hundreds of millions of parameters in medical small sample scenarios.
[0030] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0031] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A two-stream collaborative transfer learning method for automatic staging of chest X-ray images, characterized in that, include: Labeled chest X-ray images were acquired, preprocessed, and then divided into training and testing sets. Construct a dual-flow collaborative model; A frozen ViT-B / 16 is used as the backbone network; a trainable ViT is used as the side network; a cross-attention cue fusion module is introduced, which interacts with the patch tokens of the backbone network through learnable cue vectors, and the cue is projected onto the side network and concatenated with the category tokens of the intermediate layer. A feature fusion module is introduced to decouple the structure and statistics of the class tokens output by the side network and perform asymmetric recombination during the training phase; the Morlet-KAN classifier is introduced to classify the features. After training and testing the two-stream collaborative model using the training and test sets, the final two-stream collaborative model is obtained; the feature fusion module is turned off in both the testing phase and the final two-stream collaborative model. The final dual-stream collaborative model was used to analyze the chest X-ray images to obtain the image recognition results.
2. The dual-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The preprocessing includes: The chest X-ray image was segmented to obtain a high-resolution image by the pneumoconiosis region; then, the high-resolution image was downsampled using bilinear interpolation to obtain a low-resolution image. The high-resolution image and the low-resolution image are divided into non-overlapping image blocks, and then a high-resolution token sequence and a low-resolution token sequence are generated by linear projection.
3. The dual-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The frozen ViT-B / 16 is obtained by pre-training a Vision Transformer Base on ImageNet-21K, with all parameters frozen and only forward inference performed.
4. The dual-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The trainable ViT performs gradient backpropagation and parameter updates normally during training.
5. The dual-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The execution flow of the cross-attention cue fusion module includes: Initialize a set of learnable cue vectors ; The learnable cue vectors are layer-normalized and used as queries. All patch tokens output by the backbone network are used as keys and values for multi-head cross-attention computation, expressed as: In the formula, For the extracted cue features; For multi-head cross-attention mechanisms; A learnable cue vector; For layer normalization; The patch tokens output by the backbone network; The prompt features After projection, the class tokens are temporarily concatenated with the intermediate layers of the side network to inject semantic knowledge. After semantic guidance is completed, the prompt features are removed from the sequence.
6. The two-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The execution flow of the feature mixing module includes: Any two feature sequences output by the inner-batch network are decomposed into structural components and statistical moments using strength normalization, as expressed by: In the formula, and These are feature sequence A and feature sequence B, respectively. and These are the mean and standard deviation of the characteristic sequence A, respectively; This refers to the structural information of the feature sequence A after normalization. and These are the mean and standard deviation of the characteristic sequence B, respectively; This refers to the structural information of the feature sequence B after normalization. The structural information of feature sequence A Recombining the mean and standard deviation of feature sequence B, a mixed feature is generated, expressed as: In the formula, It is a mixed feature.
7. The dual-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The Morlet-KAN classifier employs a single-hidden-layer Kolmogorov–Arnold Network structure, and its activation function uses a learnable Morlet wavelet basis function, expressed as follows: In the formula, These are Morlet wavelet basis functions; These are the normalized input features; Use a Gaussian window; The center frequency; It is an oscillating core.
8. The two-stream collaborative transfer learning method for automatic staging of chest X-ray images according to claim 1, characterized in that, The execution flow of the Morlet-KAN classifier is as follows: The Morlet-KAN classifier maps input features to a 3D latent space, and the expression for the i-th latent unit is: In the formula, Let be the activation value of the i-th hidden unit; The dimension of the input feature; The weights are the edge weights of the first layer. These are Morlet wavelet basis functions; For scaling parameters; These are translation parameters; The dimension of the implicit space; The classification result of the chest X-ray image is obtained by mapping the latent representation through a linear layer. The expression of the k-th output component is: In the formula, This is the output value for the k-th category; The dimension of the implicit representation; Let be the activation value of the i-th hidden unit; The second-layer linear weights represent the contribution strength of the i-th hidden unit to the prediction of the k-th class. Output category index; This represents the number of output categories.