Medical image cross-domain auxiliary diagnosis method and system based on structural consistency enhancement

Abstract

The invention discloses a medical image cross-domain auxiliary diagnosis method and system based on structural consistency enhancement, and the method comprises the steps: carrying out the coding of the gray intensity relation between each pixel in an input image and a pixel in a neighborhood through the proposed neighborhood consistent binary transformation, constructing a multi-channel binary tensor, and explicitly retaining the relative relation of an anatomical structure; and meanwhile, style differences caused by imaging equipment are eliminated. And then decoding the tensor by using an intermediate domain style keeping auto-encoder, and reconstructing the tensor into a unified neutral style image, thereby forming image representation with both structure keeping and style normalization. The representation can serve as a standard image to be input into any existing disease classification or segmentation model, and the generalization performance of cross-center and cross-equipment can be remarkably improved without modifying a model structure or accessing target domain data. According to the method, the training stage only depends on a non-overlapping data set as an intermediate domain, the test stage is adaptive to any source image, and the method has good robustness and clinical practical value under various image devices.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image processing, image enhancement and artificial intelligence assisted diagnosis, and relates to a medical image cross-domain assisted diagnosis method and system, in particular to a method and system for enhancing the structural consistency of gray medical images (such as fetal ultrasound, chest X-ray, etc.) and improving the cross-domain generalization performance of an assisted diagnosis model. BACKGROUND

[0002] With the wide application of deep learning in medical image analysis, automated lesion detection, classification and segmentation systems are gradually landing in clinical assisted diagnosis practice. However, due to the significant style differences (such as brightness, contrast, texture changes, etc.) of medical images between different devices and different hospitals, the performance of the model is significantly reduced when applied in another center after training in one data center, which is called the "domain generalization problem". Although existing methods such as style transfer-based adversarial networks or domain adaptation algorithms can alleviate this problem to some extent, they often rely on target domain data for training, which is difficult to adapt to the "target domain data unobtainable" restriction in real scenarios. In addition, these methods often change the key structural information in the image, which is not conducive to the preservation of anatomical consistency in medical images. Therefore, how to effectively enhance the structural consistency of the image and improve the cross-domain generalization ability of the diagnosis model without accessing the target domain data has become a research hotspot in the field of medical artificial intelligence. SUMMARY

[0003] To solve the above problems, the application provides a method and system for enhancing the structural consistency of gray medical images (such as fetal ultrasound, chest X-ray, etc.) and improving the cross-domain generalization performance of an assisted diagnosis model.

[0004] The technical scheme adopted by the method of the application is: a medical image cross-domain assisted diagnosis method based on structural consistency enhancement, comprising the following steps: Step 1: for a gray medical image , perform neighborhood consistent binary transformation NCBT to generate a multi-channel binary tensor ; wherein each channel represents the gray level relationship between the current pixel and the neighborhood pixels; R represents the real number domain, H and W represent the height and width of the image respectively, and N represents the side length of the neighborhood window; Step 2: input the B to an intermediate domain style preserving autoencoder to generate a unified style image I R ; Step 3: input the I R to a medical image classification or segmentation network to perform disease assisted prediction or tissue recognition.

[0005] As preferred, in step 1, in the N×N neighborhood centered on each pixel, its gray value is compared with that of other pixels one by one to see if it is larger; the comparison result is coded as 1 or 0 each time, and then spliced into a binary vector to represent the local intensity relationship of the pixel; after splicing the coding tensors of all pixels, a multi-channel binary tensor B with a size of is formed; wherein the neighborhood size N is an odd number.

[0006] As preferred, the intermediate domain style preserving autoencoder in step 2 is composed of an encoder, a structure-aware residual bottleneck layer and a decoder. The encoder is provided with a plurality of combined layers composed of convolutional layers, ReLU activation layers and max-pooling layers, which extract semantic features of the image layer by layer while retaining spatial information with gradually decreasing resolution. The structure-aware residual bottleneck layer is composed of a plurality of stacked residual structure blocks, each of which contains two consecutive convolutional layers and a local attention unit inserted after the first layer. The decoder adopts a layer-by-layer upsampling manner, first processes the output of the structure-aware residual bottleneck layer through upsampling, then fuses it with the down-sampling output of the encoder, and then processes it through an AdaIN module, a convolutional layer and a ReLU activation layer, and performs upsampling again; the encoder down-sampling process and the decoder upsampling process correspond one by one to form a symmetric structure; the output after the last upsampling is processed through a convolutional layer and then output to the local structure loss module, which calculates the pixel reconstruction error and gradient direction consistency error of the output image I R and the original image I at multiple scales to enhance the network's reconstruction ability of texture details and edge information.

[0007] As preferred, the intermediate domain style preserving autoencoder IDSP-AE in step 2 is a trained encoder; during the training process, only the neutral dataset is trained, and the dataset has no overlap with the source / target diagnostic data.

[0008] As preferred, the intermediate domain style preserving autoencoder in step 2 is a trained encoder; during the training process, the reconstruction image I R and the original image I are trained through mean square error loss to constrain the style consistency while retaining the structure information. The specific loss function is:

[0009] wherein, is the i-th sample of the reconstruction image generated by the model, is the i-th sample of the original input image, is the i-th sample of the original input image, is the i-th sample of the original input image, is the number of samples in batch,

[0010] As preferred, the intermediate domain style preserving autoencoder in step 2 is a trained encoder; in the training process, in order to prevent overfitting, a random occlusion augmentation operation is added for data augmentation before performing the neighborhood consistent binary transformation on the gray image I in the training process; ; wherein, (·) represents a random occlusion function, which inserts an occlusion area at a random position of the image and sets the pixels in the area to zero: represents an occlusion area of random position and size in the image, and x, y represent the horizontal and vertical coordinates of the current pixel in the image.

[0011] As preferred, the intermediate domain style preserving autoencoder in step 2 is a trained encoder; in the training process, in order to prevent overfitting, a brightness perturbation augmentation operation is added for data augmentation after random occlusion on the gray image I in the training process before performing the neighborhood consistent binary transformation; ; wherein, T bright (·) represents a brightness perturbation function, which multiplies a brightness factor a: a ∈ [a, b] to the whole pixels of the image, usually [0.7, 1.3], and x, y represent the horizontal and vertical coordinates of the current pixel in the image.

[0012] As preferred, the intermediate domain style preserving autoencoder in step 2 is a trained encoder; in the training process, AUC, ACC and F1 Score are used as evaluation criteria; ; wherein, P is the number of positive samples, N is the number of negative samples, is the rank of the sample after sorting according to the predicted score; ; wherein, TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative; ; wherein, .

[0013] The technical scheme of the system of the application is: a medical image cross-domain auxiliary diagnosis system based on structure consistency enhancement, comprising: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the medical image cross-domain auxiliary diagnosis segmentation method based on structure consistency enhancement.

[0014] The technical scheme adopted by the product of the present application is: a medical image cross-domain auxiliary diagnosis product based on structure consistency enhancement, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the medical image cross-domain auxiliary diagnosis segmentation method based on structure consistency enhancement.

[0015] Compared with the prior art, the beneficial effects of the present application include: BRIEF DESCRIPTION OF DRAWINGS

[0016] The technical scheme of the present application is further illustrated below using examples and specific embodiments. In addition, some drawings are also used in the process of explaining the technical scheme. For those skilled in the art, other drawings and the intent of the present application can also be obtained from these drawings without creative labor.

[0017] Figure 1 The method flowchart of the embodiments of the present application is as follows: Figure 2 The pixel intensity comparison diagram of the neighborhood consistent binary transform (NCBT) in the 3x3 neighborhood window in the embodiments of the present application is as follows: Figure 3 The intermediate domain style preservation autoencoder IDSP-AE network structure diagram in the embodiments of the present application is as follows. DETAILED DESCRIPTION

[0018] In order to facilitate those skilled in the art to understand and implement the present application, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present application, and are not used to limit the present application.

[0019] See Figure 1 The medical image cross-domain auxiliary diagnosis method based on structure consistency enhancement provided by the embodiments has the following characteristics: Step 1: For a gray-scale medical image , execute neighborhood consistent binary transform NCBT to generate a multi-channel binary tensor ; wherein each channel represents the gray-scale size relationship of the current pixel and the neighborhood pixel; R represents the real number domain, H and W represent the image height and width respectively, N represents the side length of the neighborhood window, which is usually an odd number (such as N=3, 5, 15), which represents the construction of the N*N neighborhood with the current pixel as the center); In one embodiment, seeFigure 2 With each pixel as the center, compare its gray value with other pixels in its N x N neighborhood (preferably 15 x 15) one by one. Each comparison result is encoded as 1 or 0, and then spliced into a binary vector to represent the local intensity relationship of the pixel. After splicing the encoding tensors of all pixels, a multi-channel binary tensor B of size is formed. Figure 2 The following is a schematic diagram of pixel comparison when the neighborhood is 3 x 3, and a binary code of 8 channels is generated.

[0020] Step 2: input the B into the intermediate domain style preserving autoencoder IDSP-AE to generate a unified style image I R ; In one embodiment, please see Figure 3 , the intermediate domain style preserving autoencoder (Intermediate-Domain Style-Preserving Autoencoder, IDSP-AE) is composed of an encoder, a structure-aware residual bottleneck layer, and a decoder. The encoder part adopts a typical multi-layer convolution extraction module, which extracts the semantic features of the image layer by layer through the combination of convolution, ReLU activation and maximum pooling, while retaining the spatial information with gradually decreasing resolution. The binary tensor B formed after the NCBT conversion of the original image is input as the input, which first enters the encoder and performs two-dimensional convolution operation at each layer to extract local context information.

[0021] The structure-aware residual bottleneck module set between the encoder and the decoder is the core of the structural improvement of the present application. The module is composed of multiple stacked residual structure blocks, each residual block contains two consecutive 3 x 3 convolution layers, and a local attention unit is inserted after the first layer. The residual connection structure allows the original features to maintain the deep learning ability of the intermediate features while not being weakened too much; while the local attention unit enhances the sensitivity of the network to the edges and regional boundaries of the anatomical structure through the spatial-channel joint weighting mechanism, thereby improving the ability to restore structural details in image reconstruction. The bottleneck structure enables the network to explicitly encode the local intensity relationship from NCBT in the feature transformation space, thereby preserving the anatomical topological consistency in the original image.

[0022] The decoder part adopts a combination of layer-by-layer upsampling and convolution. First, the structural perception residual bottleneck layer output is processed by upsampling, then fused with the encoder downsampling output, and then processed by an adaptive instance normalization (AdaIN) module, a convolution layer and a ReLU activation layer, and then upsampling is performed again. The encoder downsampling process and the decoder upsampling process correspond to each other, forming a symmetric structure. The AdaIN module consists of two parts: feature standardization and style adjustment. First, the input feature map is processed by intra-channel mean and variance normalization, and then a learnable global scale and offset factor is introduced for style transformation, so that the decoding feature can actively adapt to the unified style distribution of the intermediate domain. This structure effectively avoids the feature offset problem caused by intra-batch style differences, helps the model to learn style-neutral representations, and thus improves the consistency reconstruction capability of cross-domain images. The output after the last upsampling processing is processed by a convolution layer and then output to a local structure loss module. The pixel reconstruction error and gradient direction consistency error of the output image I R and the original image I are calculated at multiple scales to enhance the network's reconstruction capability of texture details and edge information.

[0023] Overall, the intermediate domain style preserving autoencoder provided in this embodiment has the symmetry and local feature transmission capability of the U-Net structure, and also enhances the capture of local structure and the consistency control of global style through the structural perception bottleneck and AdaIN normalization module, which can effectively complete the mapping of multi-channel binary tensor to style-unified and structure-preserved images.

[0024] Step 3: input the I R into a medical image classification or segmentation network to assist in disease prediction or tissue identification.

[0025] In one embodiment, I R The image can be directly input into clip for disease text matching classification, or into a network such as nnUNet for cardiac structure segmentation. Since I R is unified in style, it no longer depends on the model to adapt to the target domain style, and fine-tuning is not required.

[0026] In one embodiment, the intermediate domain style preserving autoencoder IDSP-AE is a trained encoder; during training, only the neutral dataset is trained, and the dataset has no overlap with the source / target diagnostic data.

[0027] In one embodiment, during training, the training target is to minimize the pixel-level mean square error (MSE) between the original image I and the reconstructed image I R while constraining the style consistency and preserving the structure information. The specific loss function is:

[0028] wherein, is the i-th sample of the reconstructed image generated by the model, is the i-th sample of the original input image, is the number of samples in the batch,

[0029] In an embodiment, a random occlusion augmentation operation is added before the neighborhood consistent binary transformation is performed on the grayscale image I during the training process for data augmentation. ; wherein, (·) represents a random occlusion function, which inserts an occlusion region at a random position in the image and sets the pixels in the region to zero: represents an occlusion region of random position and size in the image, and x, y represent the horizontal and vertical coordinates of the current pixel in the image.

[0030] In an embodiment, a brightness perturbation augmentation operation is added before the neighborhood consistent binary transformation is performed on the grayscale image I after the random occlusion is added during the training process for data augmentation. ; wherein, T bright (·) represents a brightness perturbation function, which multiplies the pixels of the image by a brightness factor a: a ∈ [a, b], usually taken as [0.7, 1.3], x, y represent the horizontal and vertical coordinates of the current pixel in the image.

[0031] In an embodiment, AUC, ACC and F1 Score are used as evaluation criteria during the training process. ; wherein, P is the number of positive samples, N is the number of negative samples, is the rank of the sample after sorting by the predicted score; ; wherein, TP is the true positive, TN is the true negative, FP is the false positive, and FN is the false negative. ; wherein, .

[0032] In an embodiment, Adam is selected as the training optimizer during the training process, the initial learning rate is 1e-4, and the total number of iterations is about 30 rounds.​​

[0033] The embodiment also provides a medical image cross-domain auxiliary diagnosis system based on structure consistency enhancement, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the medical image cross-domain auxiliary diagnosis method based on structure consistency enhancement.

[0034] The embodiment also provides a medical image cross-domain auxiliary diagnosis product based on structure consistency enhancement, comprising computer program instructions, which, when running on a computer, cause the computer to execute the medical image cross-domain auxiliary diagnosis method based on structure consistency enhancement.

[0035] The following further describes the present application through experiments.

[0036] The training set contains 6000 fetal ultrasound images, which are used to train the intermediate domain style preserving autoencoder after being aligned and subjected to NCBT transformation. The network parameters are optimized using Adam, and the batchsize is 128, and the training is performed for 30 epochs.

[0037] Then, a data set containing 10000 pairs of fetal ultrasound image text pairs is used as the source domain of the downstream diagnosis task to train the Clip model, and then zero-shot testing is performed on four target domain diagnosis data sets containing hundreds of fetal ultrasound images. The original images of the source domain and the test target domain are used for training and testing, respectively, and the reconstructed source domain and the reconstructed target domain images obtained by the intermediate domain style preserving autoencoder after NCBT transformation are used for training and testing, respectively. The experimental results show that the proposed NCBT and intermediate domain style preserving autoencoder can improve the AUC, ACC and F1 of the neural network by 1.5%-7.4%, 1.3%-7.2% and 1.8%-6.6%, respectively, on the four test sets.

[0038] The experimental results show that the method of the present application can significantly improve the diagnosis effect of the neural network on medical images, and has good generalization performance.

[0039] It should be understood that the above-described embodiments are part of the embodiments of the present application, rather than all the embodiments. In addition, the technical features in each embodiment or single embodiment provided by the present application can be combined with each other to form a feasible technical solution, and such combination is not restricted by the order of steps and / or structure composition mode, but must be based on the implementation by ordinary skilled in the art. When the combination of technical solutions appears contradictory or unfeasible, it should be considered that such combination of technical solutions does not exist, and is not within the protection scope required by the present application.

[0040] It should be understood that the above description is merely a detailed example of the preferred embodiment and is not to be taken in a limiting sense, but is made merely to better describe the present application. Any equivalent variations are possible and are to be considered within the scope of the present application. The scope of the present application is to be interpreted in the broadest sense and equivalent alterations and modifications are within the scope of the present application. The scope of the present application is to be interpreted in the broadest sense and equivalent alterations and modifications are within the scope of the present application. The scope of the present application is to be interpreted in the broadest sense and equivalent alterations and modifications are within the scope of the present application. The scope of the present application is to be interpreted in the broadest sense and equivalent alterations and modifications are within the scope of the present application. The scope of the present application is to be interpreted in the broadest sense and equivalent alterations and modifications are within the scope of the present application. The scope of the present application is to be interpreted in the

Claims

1. A cross-domain assisted diagnosis method for medical images based on structural consistency enhancement, characterized by: The following steps are involved: Step 1: Targeting grayscale medical images , perform neighborhood consistent binary transform NCBT to generate a multi-channel binary tensor ; Each channel represents the grayscale relationship between the current pixel and the neighboring pixels; R represents the real number domain, H and W represent the image height and width respectively, and N represents the side length of the neighborhood window; Step 2: Input the B into the intermediate domain style preserving autoencoder to generate a unified style image I R ; Step 3: Place the R Input medical image classification or segmentation network for disease-assisted prediction or tissue identification.

2. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to claim 1, characterized in that: In step 1, with each pixel as the center, in its N×N neighborhood, compare its grayscale value with other pixels one by one to see if it is larger; each comparison result is encoded as 1 or 0, and sequentially spliced ​​into a binary vector to represent the local intensity relationship of the pixel; after splicing the encoded tensors of all pixels, a size of A multi-channel binary tensor B of , where the neighborhood size N is an odd number.

3. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to claim 1, characterized in that: The intermediate domain style-preserving autoencoder described in step 2 consists of an encoder, a structure-aware residual bottleneck layer, and a decoder; The encoder is provided with a combination of several convolutional layers, ReLU activation layers and maximum pooling layers, which extracts the semantic features of the image layer by layer while retaining the spatial information with gradually decreasing resolution; The structure-aware residual bottleneck layer is composed of several stacked residual structure blocks, each of which contains two consecutive convolutional layers and a local attention unit inserted after the first layer; The decoder adopts a layer-by-layer upsampling method. First, the output of the structure-aware residual bottleneck layer is upsampled, and then fused with the encoder downsampled output. Then, after processing through the AdaIN module, convolution layer and ReLU activation layer, upsampling is performed again. The encoder downsampling process corresponds to the decoder upsampling process one by one to form a symmetrical structure. The output after the last upsampling process is processed through the convolution layer and then output to the local structure loss module, and the output image I is R The pixel reconstruction error and gradient direction consistency error are calculated at multiple scales compared with the original image I to enhance the network's ability to reconstruct texture details and edge information.

4. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to any one of claims 1 to 3, characterized in that: The intermediate domain style preserving autoencoder IDSP-AE described in step 2 is a trained encoder; During training, training is performed only on a neutral dataset that has no overlap with the source / target diagnostic data.

5. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to any one of claims 1 to 3, characterized in that: The intermediate domain style preserving autoencoder described in step 2 is a trained encoder; During the training process, the reconstructed image I R The training is performed using the mean square error loss between the original image I, which constrains the style consistency while preserving the structural information; The specific loss function is: in, is the first reconstructed image generated by the model samples, is the first samples, is the number of samples in the batch, .

6. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to any one of claims 1 to 3, characterized in that: The intermediate domain style preserving autoencoder described in step 2 is a trained encoder; During the training process, to prevent overfitting, a random occlusion enhancement operation is added to perform data enhancement before performing neighborhood consistent binary transformation on the grayscale image I; ； in, (·) represents the random occlusion function, which inserts an occlusion region at a random position in the image and sets the pixels in the region to zero: Represents an occlusion area of ​​randomly selected position and size in the image, and x and y represent the horizontal and vertical coordinates of the current pixel in the image.

7. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to any one of claims 1 to 3, characterized in that: The intermediate domain style preserving autoencoder described in step 2 is a trained encoder; During the training process, to prevent overfitting, a brightness perturbation enhancement operation is added to the grayscale image I after random occlusion and before performing the neighborhood consistent binary transformation to perform data enhancement; ; Among them, T bright (·) represents the brightness perturbation function, which multiplies the entire pixel of the image by a brightness factor α: α∈ [a, b], usually taken as [0.7, 1.3], x, y represent the horizontal and vertical coordinates of the current pixel in the image.

8. The cross-domain assisted diagnosis method for medical images based on structural consistency enhancement according to any one of claims 1 to 3, characterized in that: The intermediate domain style preserving autoencoder described in step 2 is a trained encoder; During the training process, AUC, ACC and F1 Score were used as evaluation criteria; ； Among them, P is the number of positive samples, N is the number of negative samples, is the rank of the samples sorted by prediction score; ； Among them, TP is a true positive example, TN is a true negative example, FP is a false positive example, and FN is a false negative example; ； in, .

9. A cross-domain medical image assisted diagnosis system based on structural consistency enhancement, characterized by: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the medical image cross-domain auxiliary diagnosis method based on structural consistency enhancement as described in any one of claims 1 to 8.

10. A cross-domain medical image assisted diagnosis product based on structural consistency enhancement, comprising computer program instructions, characterized in that: When the computer program instructions are executed on a computer, the computer is enabled to execute the medical image cross-domain auxiliary diagnosis method based on structural consistency enhancement according to any one of claims 1 to 8.

Images