A skin pathology image diagnosis method combining heterogeneous networks and feature refinement

Abstract

The present invention proposes a skin pathology image diagnosis method combining heterogeneous networks and feature refinement, the method comprising: inputting images in a skin pathology image data set into a multi-scale feature extraction module to obtain second-stage features, third-stage features and fourth-stage features; inputting the second-stage features, third-stage features and fourth-stage features into a feature pyramid module to obtain the first-layer output features of the feature pyramid module and the second-layer output features of the feature pyramid module; inputting the four-stage features, the first-layer output features of the feature pyramid and the second-layer output features of the feature pyramid into an adaptive feature fusion module to obtain the final features; inputting the final features into a Softmax classifier, outputting classification results, and completing classification prediction. The present invention utilizes global and local features in images, mines spatial context information at different scales, improves the classification accuracy of skin pathology images, and better assists clinical diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a skin pathology image diagnosis method combining heterogeneous networks with feature refinement. Background Art

[0002] Skin cancer is one of the most common cancers worldwide, and its early diagnosis is crucial for the control and treatment of the disease. With the rapid development of computer-aided diagnosis (CAD) technology, CAD systems that incorporate mainstream artificial intelligence technology are gradually being used in the clinical diagnosis of cancer. More and more CAD systems based on artificial intelligence technology have achieved automatic diagnosis of skin pathology images. This technology can not only improve the diagnostic efficiency of dermatologists and reduce the impact of subjective differences on diagnostic results, but also provide dermatologists with more objective and comprehensive diagnostic results to better support their clinical work.

[0003] At present, the methods of skin pathology image classification use traditional machine learning or neural networks for disease prediction, which have achieved good classification performance and promoted the development of CAD-based intelligent diagnosis. However, the existing work has the following drawbacks: on the one hand, traditional machine learning methods based on manual features consume a lot of manpower and material resources, feature extraction is insufficient, the model accuracy is low and the generalization ability is insufficient; on the other hand, a single deep learning network also has limitations in extracting image features, and it cannot fully mine and utilize the local lesion information and global context information in the image; furthermore, there are a large number of skin pathology images of different scales in clinical diagnosis scenarios, and image features of a single scale cannot fully describe the spatial context information in skin pathology images; in addition, skin pathology images have serious data imbalance problems, which affects the final diagnostic accuracy. Summary of the invention

[0004] In view of the above situation, the main purpose of the present invention is to propose a skin pathology image diagnosis method combining heterogeneous networks and feature refinement to solve the above technical problems.

[0005] The present invention proposes a skin pathology image diagnosis method combining heterogeneous networks and feature refinement, the method comprising the following steps:

[0006] Step 1: Input the images in the skin pathology image dataset into the multi-scale feature extraction module to obtain and output the second stage features, the third stage features and the fourth stage features;

[0007] The multi-scale feature extraction module is divided into four stages, the first stage consists of a local aggregation module, and the second, third and fourth stages all consist of a global-local fusion module;

[0008] Step 2: Input the second stage features, the third stage features and the fourth stage features into the feature pyramid module, obtain and output the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module;

[0009] Step 3, input the fourth stage features, the first layer output features of the feature pyramid and the second layer output features of the feature pyramid into the adaptive feature fusion module, perform adaptive average pooling layer processing respectively and then splice all the processing results to obtain the spliced ​​features;

[0010] The concatenated features are processed point-by-point by convolution to obtain and output the final features;

[0011] Step 4: Input the final features into the Softmax classifier, output the classification results, and complete the classification prediction.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. The present invention designs a multi-scale feature extraction module, which fully utilizes the advantages of the rule network in local feature extraction and the Transformer in global context information extraction, and considers the local and global image pathology information at the same time. Through channel attention and residual connection, it promotes the deep fusion of local and global pathology information between the rule network and the Transformer, and enhances the discriminability of features;

[0014] 2. The present invention designs a feature pyramid module to simultaneously process skin pathology image features of different scales, capture information of features of different scales, and obtain a comprehensive and accurate image representation;

[0015] 3. The present invention designs an adaptive feature fusion module to adaptively fuse output feature maps from different levels, making full use of the rich complementary semantic information between features of different scales, including shape, color, and texture, to better characterize the pathological information in the image;

[0016] 4. The present invention adopts a polynomial loss function to dynamically adjust the weights of each category, so that the model pays more attention to categories with less data during training, thereby alleviating the problem of data imbalance.

[0017] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description or learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a skin pathology image diagnosis method combining heterogeneous networks and feature refinement proposed by the present invention;

[0019] Figure 2 A schematic diagram of the structure of a prediction model of a skin pathology image diagnosis method combining heterogeneous networks and feature refinement proposed by the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a multi-scale feature extraction module of a skin pathology image diagnosis method that combines heterogeneous networks and feature refinement proposed in the present invention. DETAILED DESCRIPTION

[0021] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0022] These and other aspects of the embodiments of the present invention will be apparent with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to represent some ways of implementing the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.

[0023] See also Figure 1 , Figure 2 and Figure 3 The embodiment of the present invention proposes a skin pathology image diagnosis method combining heterogeneous network and feature refinement, the method comprising the following steps:

[0024] Step 1: Input the images in the skin pathology image dataset into the multi-scale feature extraction module to obtain and output the second stage features, the third stage features and the fourth stage features;

[0025] The multi-scale feature extraction module is divided into four stages, the first stage consists of a local aggregation module, and the second, third and fourth stages all consist of a global-local fusion module;

[0026] In step 1, the multi-scale feature extraction module is divided into four stages. The first stage consists of a local aggregation module, and the second, third and fourth stages are all composed of a global-local fusion module. The specific composition of the local aggregation module is as follows:

[0027] The local aggregation module consists of a regular network block, which is composed of a convolutional layer with a convolution kernel size of 1, a normalization layer, a ReLu function, a group convolutional layer with a convolution kernel size of 3, a normalization layer, a ReLu function, and a convolutional layer with a convolution kernel size of 1.

[0028] The specific composition of the global-local aggregation module is as follows:

[0029] The global-local aggregation module consists of R regular network blocks connected in series as local aggregation, followed by a feature reconstruction layer, a Transformer block, and a feature reconstruction layer in sequence;

[0030] Furthermore, in the global-local aggregation module, the first feature reconstruction layer is responsible for expanding the features to fit the input size of the Transformer, and the second feature reconstruction layer is responsible for compressing the output features of the Transformer to the original dimension;

[0031] In step 1, the skin pathology image is input into the multi-scale feature extraction module, and the specific steps of obtaining and outputting the second stage features, the third stage features and the fourth stage features are as follows:

[0032] The images in the skin pathology image dataset are input into the multi-scale feature extraction module, and the first-stage features are obtained through the first-stage local aggregation module;

[0033] Input the first stage features into the global-local fusion module of the second stage to obtain and output the second stage features;

[0034] Input the second stage features into the global-local fusion module of the third stage to obtain and output the third stage features;

[0035] Input the third stage features into the global-local fusion module of the fourth stage to obtain and output the fourth stage features;

[0036] Furthermore, before inputting the images in the skin pathology image dataset into the multi-scale feature extraction module, a data enhancement operation is performed on the images, and the operation includes the following steps:

[0037] The images in the skin pathology image dataset are subjected to center cropping, horizontal flipping, color jittering, and normalization operations to improve the quality of image data, reduce noise and interference, and enhance the features and information of the images.

[0038] Step 2: Input the second stage features, the third stage features and the fourth stage features into the feature pyramid module, obtain and output the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module;

[0039] In step 2, the second stage features, the third stage features, and the fourth stage features are input into the feature pyramid module, and the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module are obtained and output, wherein the cross attention calculation formula in the feature pyramid module is:

[0040] ;

[0041] in, It means that after cross attention calculation, represents the query vector obtained by mapping the matrix to the sequence, represents the key vector obtained by mapping the matrix to the sequence, represents the value vector obtained by mapping the matrix to the sequence, It means that it has been processed by the Softmax function. represents transpose, represents the dimension of the key vector;

[0042] Furthermore, the low-dimensional features and high-dimensional features are input simultaneously and cross-attention calculation is performed. The query vector generated by the low-dimensional features is inner-producted with the key vector generated by the high-dimensional features, and the attention weight is obtained through the Softmax function. The attention weight is inner-producted with the value vector generated by the high-dimensional features to obtain the final attention result;

[0043] In step 2, the second stage features, the third stage features, and the fourth stage features are input into the feature pyramid module, and the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module are obtained and output. The specific steps are as follows:

[0044] The fourth stage features are upsampled to obtain the fourth stage upsampled features. The corresponding relationship is:

[0045] ;

[0046] in, represents the upsampling operation, represents the fourth stage upsampling feature, Indicates the fourth stage characteristics;

[0047] The third-stage features and the fourth-stage upsampled features are cross-attentionally calculated to obtain the fused features that combine the third-stage features and the fourth-stage upsampled features. The corresponding process has the following formula:

[0048] ;

[0049] in, It represents the fusion feature of the third stage features and the fourth stage upsampling features. Indicates the third stage characteristics;

[0050] The fourth stage features and the fused features that fuse the third stage features and the fourth stage up-sampled features are residually connected to obtain and output the first layer output features of the feature pyramid module. The relationship between the corresponding process is:

[0051] ;

[0052] in, Represents the first-layer output features of the feature pyramid module, represents residual connection;

[0053] The first layer output features of the feature pyramid module are upsampled to obtain the first layer upsampled features of the feature pyramid module. The corresponding relationship is:

[0054] ;

[0055] in, Represents the first layer of upsampled features of the feature pyramid module;

[0056] The second-stage features and the first-layer upsampled features of the feature pyramid module are cross-attentionally calculated to obtain the fused features that combine the second-stage features and the first-layer upsampled features of the feature pyramid module. The relationship between the corresponding processes is:

[0057] ;

[0058] in, Represents the fusion features of the second stage features and the first layer upsampled features of the feature pyramid module;

[0059] The first-layer output features of the feature pyramid module are residually connected with the fused features of the second-stage features and the first-layer upsampled features of the feature pyramid module to obtain and output the second-layer output features of the feature pyramid module. The relationship between the corresponding process is:

[0060] ;

[0061] in, Represents the second-level output features of the feature pyramid module.

[0062] Step 3: Input the four-stage features, the first-layer output features of the feature pyramid, and the second-layer output features of the feature pyramid into the adaptive feature fusion module, perform the adaptive average pooling layer processing respectively, and then splice all the processing results to obtain the spliced ​​features;

[0063] The concatenated features are processed point-by-point by convolution to obtain and output the final features;

[0064] In step 3, the fourth stage features, the first layer output features of the feature pyramid and the second layer output features of the feature pyramid are input into the adaptive feature fusion module, and the adaptive average pooling layer is used to process them respectively, and then all the processing results are spliced ​​to obtain the spliced ​​features. The relationship between the corresponding process is:

[0065] ;

[0066] in, represents the features after splicing, Represents a splicing operation, represents the adaptive average pooling operation;

[0067] The concatenated features are processed point-by-point by convolution to obtain and output the final features. The relationship between the corresponding processes is:

[0068] ;

[0069] in, represents the final feature, Indicates that it has been processed by the ReLu function. represents the normalization operation, Indicates the convolution operation;

[0070] Step 4: Input the final features into the Softmax classifier, output the classification results, and complete the classification prediction;

[0071] In step 4, the classification loss expression used in the process of executing step 4 is:

[0072] ;

[0073] in, represents multiple loss functions, Representation sample Belong to The probability of the class, Indicates the number of categories for classification, represents the polynomial coefficients, Represents the perturbation term, with a value of 1.

[0074] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0076] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A skin pathology image diagnosis method combining heterogeneous networks and feature refinement, characterized in that: The method is implemented based on a classification prediction model, which is composed of a multi-scale feature extraction module, a feature pyramid module, an adaptive feature fusion module and a Softmax classifier. The method includes the following steps: Step 1: Input the images in the skin pathology image dataset into the multi-scale feature extraction module to obtain and output the second stage features, the third stage features and the fourth stage features; The multi-scale feature extraction module is divided into four stages, the first stage consists of a local aggregation module, and the second, third and fourth stages all consist of a global-local fusion module; Step 2: Input the second stage features, the third stage features and the fourth stage features into the feature pyramid module, obtain and output the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module; Step 3, input the fourth stage features, the first layer output features of the feature pyramid and the second layer output features of the feature pyramid into the adaptive feature fusion module, perform adaptive average pooling layer processing respectively and then splice all the processing results to obtain the spliced ​​features; The concatenated features are processed point-by-point by convolution to obtain and output the final features; Step 4: Input the final features into the Softmax classifier, output the classification results, and complete the classification prediction; In the step 2, the second stage features, the third stage features and the fourth stage features are input into the feature pyramid module, and the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module are obtained and output, wherein the cross attention calculation formula in the feature pyramid module is: ； in, It means that after cross attention calculation, represents the query vector obtained by mapping the matrix to the sequence, represents the key vector obtained by mapping the matrix to the sequence, represents the value vector obtained by mapping the matrix to the sequence, It means that it has been processed by the Softmax function. represents transpose, represents the dimension of the key vector; Among them, the second stage features, the third stage features and the fourth stage features are input into the feature pyramid module, and the first layer output features of the feature pyramid module and the second layer output features of the feature pyramid module are obtained and output. The specific steps are as follows: The fourth stage features are upsampled to obtain the fourth stage upsampled features. The relationship between the corresponding process is: ； in, represents the upsampling operation, represents the fourth stage upsampling feature, Indicates the characteristics of the fourth stage; The third-stage features and the fourth-stage upsampled features are cross-attentionally calculated to obtain the fused features that combine the third-stage features and the fourth-stage upsampled features. The corresponding process has the following formula: ； in, It represents the fusion feature of the third stage features and the fourth stage upsampling features. Indicates the third stage characteristics; The fourth stage features and the fused features that fuse the third stage features and the fourth stage up-sampled features are residually connected to obtain and output the first layer output features of the feature pyramid module. The relationship between the corresponding process is: ； in, Represents the first-layer output features of the feature pyramid module, represents residual connection; The first layer output features of the feature pyramid module are upsampled to obtain the first layer upsampled features of the feature pyramid module. The corresponding relationship is: ； in, Represents the first layer of upsampled features of the feature pyramid module; The second-stage features and the first-layer upsampled features of the feature pyramid module are cross-attentionally calculated to obtain the fused features that combine the second-stage features and the first-layer upsampled features of the feature pyramid module. The corresponding relationship is: ； in, Represents the fusion features of the second stage features and the first layer upsampled features of the feature pyramid module; The first-layer output features of the feature pyramid module are residually connected with the fused features of the second-stage features and the first-layer upsampled features of the feature pyramid module to obtain and output the second-layer output features of the feature pyramid module. The relationship between the corresponding process is: ； in, Represents the second-level output features of the feature pyramid module.

2. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 1, characterized in that: In step 1, the multi-scale feature extraction module is divided into four stages. The first stage is composed of a local aggregation module, and the second, third and fourth stages are all composed of a global-local fusion module. The specific composition of the local aggregation module is as follows: The local aggregation module consists of a regular network block, which is composed of a convolutional layer with a convolution kernel size of 1, a normalization layer, a ReLu function, a group convolutional layer with a convolution kernel size of 3, a normalization layer, a ReLu function, and a convolutional layer with a convolution kernel size of 1.

3. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 2 is characterized in that: In step 1, the multi-scale feature extraction module is divided into four stages. The first stage consists of a local aggregation module, and the second, third and fourth stages are all composed of a global-local fusion module. The specific composition of the global-local aggregation module is as follows: The global-local aggregation module consists of R regular network blocks connected in series as local aggregation, followed by a feature reconstruction layer, a Transformer block, and a feature reconstruction layer.

4. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 3 is characterized in that: In step 1, the skin pathology image is input into the multi-scale feature extraction module, and the specific steps of obtaining and outputting the second stage features, the third stage features and the fourth stage features are as follows: The images in the skin pathology image dataset are input into the multi-scale feature extraction module, and the first-stage features are obtained through the first-stage local aggregation module; Input the first stage features into the global-local fusion module of the second stage to obtain and output the second stage features; Input the second stage features into the global-local fusion module of the third stage to obtain and output the third stage features; The third stage features are input into the global-local fusion module of the fourth stage to obtain and output the fourth stage features.

5. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 1, characterized in that: In step 3, the fourth stage features, the first layer output features of the feature pyramid and the second layer output features of the feature pyramid are input into the adaptive feature fusion module, and the adaptive average pooling layer is processed respectively, and then all the processing results are spliced ​​to obtain the spliced ​​features. The relationship between the corresponding process is: ； in, represents the features after splicing, Represents a splicing operation, Represents an adaptive average pooling operation.

6. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 5, characterized in that: In step 3, the concatenated features are processed by point-by-point convolution to obtain and output the final features. The relationship between the corresponding processes is: ； in, represents the final feature, Indicates that it has been processed by the ReLu function. represents the normalization operation, Indicates the convolution operation.

7. The skin pathology image diagnosis method combining heterogeneous network and feature refinement according to claim 6, characterized in that: The classification loss expression used in step 4 is: ； in, represents multiple loss functions, Representation sample Belong to The probability of the class, Indicates the number of categories for classification, represents the polynomial coefficients, represents the perturbation term.

Images