A method and system for training a neural network model for medical image classification

Abstract

A training method and system of a neural network for medical image classification, which adopts an improved visual transformer as a neural network model for medical image classification, has an aggregation function, aggregates local image region encoding features obtained based on medical images to form global image region encoding features, and aggregates the global image region encoding features with the local image region encoding features; the visual transformer is embedded into a contrast learning network as an encoding layer for pre-training, and a robust contrast self-supervised loss function (RS-InfoNCE) is used as a contrast loss function; after fine-tuning of the pre-trained visual transformer, a trained neural network model is obtained for classification of medical images. In this way, the embodiment of the present application pre-trains the neural network model for medical image classification using contrast learning technology, thereby improving the quality of the neural network model in classifying medical images.

Description

Technical Field

[0001] This application relates to image processing technology, and in particular to a training method and system for a neural network for medical image classification. Background Technology

[0002] In recent years, supervised learning-based neural network models have made significant contributions to the field of artificial intelligence. Supervised learning is a machine learning task that uses labeled data to train a neural network model to infer a function. However, training supervised learning-based neural network models requires a large amount of manually labeled training data. Due to the limited availability of manually labeled data and insufficient capacity to process training data, supervised learning-based neural network models only gradually became a practical technology for commercial application in recent years. Supervised learning-based neural network models are only suitable for situations where there is sufficient high-quality training data that can capture all possible scenarios. However, in the field of medical imaging, obtaining high-quality, accurate labeled training data is difficult, costly, and limited in quantity, making it impossible to train a highly accurate neural network model for classifying medical images.

[0003] There is a growing interest in using massive amounts of unlabeled training data to train neural network models that can accurately classify medical images for subsequent applications. Here, medical images include histopathological images such as cell images. Therefore, self-supervised learning techniques have been proposed, where neural network models learn directly from unlabeled data without the need for labeled training data.

[0004] Self-supervised learning techniques include various methods, one of which is contrastive learning. Contrastive learning effectively learns a pre-trained neural network model from unlabeled data. This pre-trained model is then fine-tuned using training data with limited labels to obtain a final neural network model, which is then applied to image classification. However, because medical images are a special type of image, directly applying contrastive learning to medical image classification can result in low classification quality. Therefore, how to effectively apply contrastive learning to medical image classification and improve classification quality has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this application provide a training method for a neural network for medical image classification. This method uses contrastive learning technology to pre-train a neural network model for medical image classification, thereby improving the quality of the neural network model in classifying medical images.

[0006] This application also provides a training system for a neural network for medical image classification. The system uses contrastive learning technology to pre-train a neural network model for medical image classification, thereby improving the quality of the neural network model in classifying medical images.

[0007] The embodiments of this application are implemented as follows:

[0008] A method for training a neural network for medical image classification, the method comprising:

[0009] a. A visual converter with feature aggregation function is preset, and the visual converter is embedded as an encoding layer into the contrastive learning network;

[0010] b. Provide multiple medical images for pre-training. For each medical image, input the medical image into the contrastive learning network. The contrastive learning network uses different image feature extraction methods to extract features of the first image region and features of the second image region.

[0011] c. Input the first image region features and the second image region features into the visual converter for multi-layer encoding. Before encoding in the final encoding layer, use the feature aggregation function to aggregate the local image region encoding features obtained from multi-layer encoding to form global image region encoding features. After encoding in the final encoding layer, aggregate the global image region encoding features and the local image region encoding features obtained in the final encoding layer to form image region embedding features, thereby obtaining the first image region embedding feature and the second image region embedding feature.

[0012] d. After processing the first image region embedding features and the second image region embedding features respectively using a multilayer perceptron (MLP) in a contrastive learning network, calculate the difference between the two using a preset contrastive loss function. If the difference between the two is greater than a preset threshold, adjust the parameters in the visual converter and return to the pre-training process of steps b to c until the difference is no greater than the preset threshold.

[0013] e. After training the pre-trained visual converter with labeled medical images, a trained visual converter is obtained and applied to the classification of medical images.

[0014] In the above method, the step of inputting the first image region features and the second image region features into the visual converter for multi-layer encoding includes:

[0015] The visual converter includes at least one cascaded Transformer-based coding layer. In each coding layer from the first coding layer to the coding layer before the last coding layer, attention weights corresponding to each coding layer are used to encode the features of the first image region and the features of the second image region, respectively.

[0016] For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

[0017] In the above method, when the difference between the two is calculated using a preset contrast loss function, the contrast loss function is the robust contrast self-supervised loss function RS-InfoNCE.

[0018] In the above method, the robustness contrastive self-supervised loss function includes:

[0019]

[0020] in,

[0021] In the formula z i and z j These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP, respectively. k This represents the first and second image region embedding features, sim(z), after other different medical images used for pre-training have been encoded by the visual converter and processed by MLP. i ,z j ) represents the similarity between the first image region embedding feature and the second image region embedding feature, |z i | This embeds features into the first image region after feature regularization, where t represents the hyperparameter, and is initially set to 0.07.

[0022] In the above method, after processing with a multi-layer MLP in a contrastive learning network, and before calculating the difference between the two using a preset contrastive loss function, the following steps are also included:

[0023] The predictor in the contrastive information network is used to make predictions to obtain the first image region embedding features and the second image region embedding features.

[0024] A training system for a neural network for medical image classification includes: a pre-training unit and a training unit, wherein,

[0025] The pre-training unit is used to embed a visual converter with feature aggregation function as an encoding layer into a contrastive learning network. Multiple medical images are provided for pre-training. For each medical image, the image is input into the contrastive learning network, which extracts first and second image region features using different image feature extraction methods. The first and second image region features are then input into the visual converter for multi-layer encoding. Before the final encoding layer, the feature aggregation function aggregates the local image region encoding features obtained from the multi-layer encoding to form global image region encoding features. After the final encoding layer, the global image region encoding features and the local image region encoding features obtained from the final encoding layer are aggregated to form image region embedding features, resulting in first and second image region embedding features. The first and second image region embedding features are processed using the MLP in the contrastive learning network, and a preset contrast loss function is used to calculate their difference. If the difference is greater than a preset threshold, the parameters in the visual converter are adjusted, and the pre-training process is repeated until the difference is no greater than the preset threshold.

[0026] The training unit is used to train the pre-trained visual converter using medical images with labeled information to obtain a trained visual converter, which is then applied to the classification of medical images.

[0027] In the above system, the pre-training unit is further configured to encode the first image region features and the second image region features respectively in each coding layer from the first coding layer to the coding layer before the last coding layer, using the attention weights corresponding to each coding layer.

[0028] For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

[0029] In the above system, when the difference between the two is calculated using a preset contrast loss function, the contrast loss function is the robust contrast self-supervised loss function RS-InfoNCE.

[0030] 9. The system as described in claim 8, wherein the robustness contrastive self-supervised loss function comprises:

[0031]

[0032] in,

[0033] In the formula z i and z j These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP, respectively. k This represents the first and second image region embedding features, sim(z), after other different medical images used for pre-training have been encoded by the visual converter and processed by MLP. i ,z j ) represents the similarity between the first image region embedding feature and the second image region embedding feature, |z i | This embeds features into the first image region after feature regularization, where t represents the hyperparameter, and is initially set to 0.07.

[0034] An electronic device, comprising:

[0035] processor;

[0036] A memory storing a program configured to implement, when executed by the processor, a training method for a neural network for medical image classification as described in any of the preceding claims.

[0037] As seen above, this embodiment employs an improved visual converter as the neural network model for medical image classification. This visual converter has an aggregation function, used to aggregate local image region encoding features obtained from encoding medical images into global image region encoding features, and to aggregate the global image region encoding features with the local image region encoding features to better capture the features of medical images. This visual converter is embedded as an encoding layer into a contrastive learning network for pre-training, and a robust contrastive self-supervised loss function (RS-InfoNCE) is used as the contrastive loss function to ensure robustness to spatial smoothing of noisy medical images and low-confidence negative samples generated from medical images. After fine-tuning the pre-trained visual converter, a trained neural network model is obtained for medical image classification. Thus, this embodiment uses contrastive learning technology to pre-train the neural network model for medical image classification, thereby improving the quality of the neural network model's classification of medical images. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a training method for a neural network for medical image classification, provided as an embodiment of this application;

[0039] Figure 2A schematic diagram of the structure of a training system for a neural network for medical image classification provided in an embodiment of this application;

[0040] Figure 3 A schematic diagram of the contrastive learning network framework provided in the embodiments of this application;

[0041] Figure 4 A schematic diagram of a noisy medical image and a low-confidence negative sample pair provided for an embodiment of this application;

[0042] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] In the background section, contrastive learning techniques have been applied to natural image classification and proven effective. However, directly transferring contrastive learning to the classification of medical images presents challenges. Histopathological images, such as cell images, differ from natural images. During pre-training, the extraction of image region features lacks fine-grained features and high-quality positive sample pairs, leading to inaccuracies in the subsequent classification of medical images by the pre-trained neural network model. Specifically, the pre-training process using contrastive learning for medical image classification suffers from two problems: 1) Incorrectly selecting regions in medical images that do not provide meaningful features as pre-training samples; 2) Extracting low-confidence negative sample pairs from medical images and inputting them into the neural network model for pre-training. Therefore, the quality of the neural network model trained using contrastive learning is low.

[0046] To address the aforementioned issues, this application employs an improved visual converter as the neural network model for medical image classification. This visual converter features an aggregation function, aggregating local image region encoding features obtained from encoding medical images to form global image region encoding features. Furthermore, it aggregates the global and local image region encoding features to better capture the characteristics of medical images. This visual converter is embedded as an encoding layer into a contrastive learning network for pre-training, using RS-InfoNCE as the contrastive loss function to ensure robustness to spatial smoothing of noisy medical images and low-confidence negative samples generated from medical images. After fine-tuning the pre-trained visual converter, a trained neural network model is obtained for medical image classification. Thus, this application employs contrastive learning technology to pre-train the neural network model for medical image classification, thereby improving the quality of the neural network model's medical image classification.

[0047] Figure 1 A flowchart of a training method for a neural network for medical image classification, provided in an embodiment of this application, includes the following steps:

[0048] Step 101: Preset a visual converter with feature aggregation function, and embed the visual converter as an encoding layer into the contrastive learning network;

[0049] Step 102: Provide multiple medical images for pre-training. For each medical image, input the medical image into the contrastive learning network. The contrastive learning network uses different image feature extraction methods to extract features of the first image region and features of the second image region.

[0050] Step 103: Input the first image region features and the second image region features into the visual converter for multi-layer encoding. Before encoding in the final encoding layer, use the feature aggregation function to aggregate the local image region encoding features obtained from multi-layer encoding to form global image region encoding features. After encoding in the final encoding layer, aggregate the global image region encoding features and the local image region encoding features obtained in the final encoding layer to form image region embedding features, thereby obtaining the first image region embedding feature and the second image region embedding feature.

[0051] Step 104: After processing the first image region embedding features and the second image region embedding features using a multilayer perceptron (MLP) in a contrastive learning network, calculate the difference between the two using a preset contrastive loss function. If the difference between the two is greater than a preset threshold, adjust the parameters in the visual converter and return to the pre-training process of steps 102 to 103 until the difference is no greater than the preset threshold.

[0052] Step 105: After training the pre-trained visual converter with labeled medical images, a trained visual converter is obtained and applied to the classification of medical images.

[0053] In the above method, the medical images include histopathological images such as cell images.

[0054] In the above method, the medical images used for pre-training are collected and organized, and are called a medical image dataset. Within the medical image dataset, 26,000 (k) unlabeled medical images are collected and organized for pre-training of self-supervised learning, while 20,000 labeled medical images are collected and organized for training in step 104, which is the transfer learning process for the classification task.

[0055] In the above method, the preset threshold is set according to actual needs, and there is no restriction here.

[0056] In the above method, the MLP includes an activation function for performing nonlinear feature space mapping on the first image region embedding features and the second image region embedding features.

[0057] In the above method, the step of inputting the first image region features and the second image region features into the visual converter for multi-layer encoding includes:

[0058] The visual converter includes at least one cascaded Transformer-based coding layer. In each coding layer from the first coding layer to the coding layer before the last coding layer, attention weights corresponding to each coding layer are used to encode the features of the first image region and the features of the second image region, respectively.

[0059] For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

[0060] In other words, this application provides an improved visual converter, which is a fine-grained visual converter, also known as a fine-grained transformer network. This visual converter is embedded in a contrastive learning network as the contrastive learning framework of this application. A feature aggregation function is set in the visual converter, which can more effectively capture medical image features by aggregating global image region coding features and local image region coding features.

[0061] In the above method, when the difference between the two is calculated using a preset contrast loss function, the contrast loss function is RS-InfoNCE.

[0062] Specifically, the robustness contrastive self-supervised loss function includes:

[0063]

[0064] in,

[0065] In the formula z i and z j These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP, respectively. k This represents the first and second image region embedding features, sim(z), after other different medical images used for pre-training have been encoded by the visual converter and processed by MLP. i ,z j ) represents the similarity between the first image region embedding feature and the second image region embedding feature, |z i | This embeds features into the first image region after feature regularization, where t represents the hyperparameter, and is initially set to 0.07.

[0066] In other words, during the pre-training of a visual converter using a contrastive learning network, noisy medical images and low-confidence negative samples extracted from them can degrade the pre-training process, leading to suboptimal representations when the loss information is used extensively during pre-training and is noisy. Furthermore, increasing the pre-training batch size does not always improve performance; if the batch size exceeds a certain threshold, it can even degrade the classification performance of the final pre-trained neural network model. Therefore, this application proposes a specific contrastive loss function—RS-InfoNCE—that is robust to spatial smoothing of noisy medical images and low-confidence negative samples generated from them.

[0067] In the above method, after processing with MLP in a contrastive learning network and before calculating the difference between the two using a preset contrastive loss function, the following steps are also included:

[0068] The predictor in the contrastive information network is used to make predictions to obtain the first image region embedding features and the second image region embedding features.

[0069] In other words, the contrastive learning network includes a feature extraction layer, an encoding layer, an MLP layer, a predictor layer, and a contrastive loss function calculation layer. The MLP is used to perform nonlinear feature space transformation on the obtained image region feature space, and the predictor layer is used to map the transformed image region feature space. These are all mature technologies.

[0070] In this embodiment, after pre-training the visual converter using contrastive learning technology, experimental verification is conducted to determine the pre-training effect. By setting an improved visual converter and a contrastive loss function in the contrastive learning technology, the proposed method demonstrates that the application of contrastive learning technology in medical image classification is also effective. A neural network model for medical image classification can be pre-trained on a large number of unlabeled medical images, and then the neural network model can be fine-tuned.

[0071] Figure 2 A schematic diagram of a training system for a neural network for medical image classification, provided in an embodiment of this application, includes: a pre-training unit and a training unit, wherein...

[0072] The pre-training unit is used to embed a visual converter with feature aggregation function as an encoding layer into a contrastive learning network. Multiple medical images are provided for pre-training. For each medical image, the image is input into the contrastive learning network, which extracts first and second image region features using different image feature extraction methods. The first and second image region features are then input into the visual converter for multi-layer encoding. Before the final encoding layer, the feature aggregation function aggregates the local image region encoding features obtained from the multi-layer encoding to form global image region encoding features. After the final encoding layer, the global image region encoding features and the local image region encoding features obtained from the final encoding layer are aggregated to form image region embedding features, resulting in first and second image region embedding features. The first and second image region embedding features are then processed using a multi-layer MLP in the contrastive learning network, and a preset contrast loss function is used to calculate their difference. If the difference is greater than a preset threshold, the parameters in the visual converter are adjusted, and the pre-training process is repeated until the difference is no greater than the preset threshold.

[0073] The training unit is used to train the pre-trained visual converter using medical images with labeled information to obtain a trained visual converter, which is then applied to the classification of medical images.

[0074] In the above system, the pre-training unit is further configured to encode the first image region features and the second image region features respectively in each coding layer from the first coding layer to the coding layer before the last coding layer, using the attention weights corresponding to each coding layer.

[0075] For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

[0076] In the above system, when the difference between the two is calculated using a preset contrast loss function, the contrast loss function is RS-InfoNCE.

[0077] In the above system, the training unit is actually the model fine-tuning unit, which is used to fine-tune the pre-trained visual converter.

[0078] The system may also include an experimental verification unit, which is used to perform experimental verification after the visual converter is pre-trained using contrastive learning techniques, in order to determine the pre-training effect.

[0079] As can be seen, the embodiments of this application propose an improved fine-grained visual converter, which, after pre-training and training, is applied to the classification of medical images without any other auxiliary tasks. When pre-training the visual converter using a contrastive learning network, RS-InfoNCE is employed as the contrastive loss function, and Sutdent's t-distribution vector is used as a smoothing strategy to mitigate the problem of low-confidence negative samples. The embodiments of this application propose a simple and general self-supervised learning framework, and experiments on the public dataset NCT-CRC-HE-100k achieve state-of-the-art performance.

[0080] Thus, the embodiments of this application can obtain high-quality positive sample pairs, high-confidence negative sample pairs, and fine-grained image region features from the contrastive learning network, solving the problem of directly applying contrastive learning to pre-train medical images. The embodiments of this application demonstrate that contrastive learning networks are also effective in the classification of medical images, allowing for the pre-training of a visual encoder on a large number of unlabeled medical images, followed by fine-tuning before application.

[0081] The following is a specific example to illustrate the embodiments of this application in detail.

[0082] Combination Figure 3 and Figure 4 This example will be used to illustrate the point, in which, Figure 3 This is a schematic diagram of the contrastive learning network framework provided in an embodiment of this application. Figure 4 This is a schematic diagram of a noisy medical image and a low-confidence negative sample pair provided in an embodiment of this application.

[0083] like Figure 3 As shown in the right half, this example is based on a fine-grained visual transducer, also known as a fine-grained transformer network, and sets up a contrastive learning network. The inspiration is to embed the improved visual transducer as the encoding layer in the contrastive learning network. This contrastive learning network also includes: a feature extraction layer, an encoding layer (represented as encoder in the figure), an MLP layer (represented as projector in the figure), and a loss function calculation layer (RS-InfoNCE). The contrastive learning network can further include a predictor layer (represented as predictor in the figure), where the MLP is used to perform a non-linear transformation on the obtained image region feature space, and the predictor layer is used to map the transformed image region feature space.

[0084] Here, the loss function computation layer uses RS-InfoNCE as the contrastive loss function, which employs the student's t-distribution vector as a smoothing strategy to mitigate the problem of low-confidence negative sample pairs. It is robust to smoothing noisy views and low-confidence negative sample pairs. In this example, the contrastive learning network was experimented on the public dataset NCT-CRC-HE-100K, achieving state-of-the-art performance.

[0085] use Figure 3 When the contrastive learning network shown pre-trains the visual converter, it uses a medical image dataset. In the medical image dataset, 26k unlabeled medical images are collected and organized for pre-training of self-supervised learning. 20k labeled medical images are collected and organized for training in step 104, which is the transfer learning process for the classification task.

[0086] In this example, a visual transformer, also known as a transformer network, is set up, which can be defined as TransFG++. It has a feature aggregation function to aggregate local image region encoded features obtained from encoding medical images into global image region encoded features. Furthermore, it aggregates the global image region encoded features with the local image region encoded features to better capture the features of the medical image. The visual transformer in this example aims to locate lesions in medical images and capture fine-grained features in histopathological images. Finally, it aggregates the features output from the encoding layer of the contrast learning network to obtain global features and features with significant classification regions. Here, the visual transformer uses multiple Transformer layers in series and includes a local selection module and a global feature aggregation module to obtain more effective image features.

[0087] Specifically, the visual transformer includes a local selection module that applies matrix multiplication to the attention weights corresponding to each Transformer layer in the visual transformer to integrate the attention weights of all previous layers. To fully utilize attention and discriminative information, a maximum selection model (represented in the figure as Largest Selection) is applied to the last Transformer layer. That is, the attention weights corresponding to the Transformer layers are integrated into the attention group weights of the last Transformer layer to guide the encoding process of the visual transformer, accurately selecting discriminative image regions and calculating the relationships between them.

[0088] The visual transformer includes a feature aggregation module. This example requires not only salient region features of the medical image but also global features. The final encoded features corresponding to the regions in the medical image are insufficient to provide global features. Therefore, this embodiment includes a global feature aggregation module using global average pooling (referred to as Global Features Aggregate in the figure). Before the last Transformer layer, all encoded local image region features are input for global average pooling to obtain global image region encoded features. Then, the local image region encoded features output from the last Transformer layer are aggregated with the global image region encoded features (i.e., concatenated) and output to the MLP layer of the contrastive learning network for further processing. Thus, in this example, features from a specific region selected by the last Transformer layer and global image region encoded features are used as the encoded classification features.

[0089] In this example, the contrastive learning network also uses a contrastive loss function. For example... Figure 4 The noisy medical images (upper part of the figure) and low-confidence negative samples (lower part of the figure) shown degrade the entire pre-training process of the contrastive learning network. Contrastive learning networks extensively use loss information, which produces suboptimal representations in the presence of noise. Increasing the pre-training batch size does not always bring improvement; if the pre-training batch size exceeds a threshold, performance degrades. Therefore, in this example, RS-InfoNCE is proposed as the contrastive loss function for the contrastive learning network, which is robust to noisy medical images and the smoothness of the negative sample space. The specific formula for calculating the contrastive loss function is as follows:

[0090] Specifically, the robustness contrastive self-supervised loss function includes:

[0091]

[0092] in,

[0093] In the formula z i and z j These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP, respectively. k This represents the first and second image region embedding features, sim(z), after other different medical images used for pre-training have been encoded by the visual converter and processed by MLP. i ,z j ) represents the similarity between the first image region embedding feature and the second image region embedding feature, |zi | This embeds features into the first image region after feature regularization, where t represents the hyperparameter, and is initially set to 0.07.

[0094] In this example, the pre-trained visual converter is fine-tuned using medical images with labeled information.

[0095] In this example, the process of pre-training the visual converter using the contrastive learning network described above can be experimentally verified. Experimental results show that, compared to directly using models trained on ImageNet or random initialization, this method provides a better way to initialize models for medical image classification tasks. It effectively improves the transfer learning performance of medical image classification tasks, utilizes a large number of unlabeled medical images, and reduces the need for a large number of labeled medical images in deep learning tasks.

[0096] Figure 5 This is a schematic diagram of an electronic device provided for another embodiment of this application. (See diagram below.) Figure 5 As shown, another embodiment of this application also provides an electronic device that may include a processor 401, wherein the processor 401 is used to execute the steps of the above-described training method for a neural network for medical image classification. From Figure 5 It can also be seen that the electronic device provided in the above embodiments further includes a non-transitory computer-readable storage medium 502, on which a computer program is stored. The computer program is executed by the processor 501 to perform the steps of the above-described training method for a neural network for medical image classification.

[0097] Specifically, the non-transitory computer-readable storage medium 502 can be a general-purpose storage medium, such as a mobile disk, hard disk, FLASH, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or portable compact disk read-only memory (CD-ROM), etc. When the computer program on the non-transitory computer-readable storage medium 502 is run by the processor 501, it can cause the processor 501 to execute the various steps of the above-described training method for a neural network for medical image classification.

[0098] In practical applications, the non-transitory computer-readable storage medium 502 may be included in the device / apparatus / system described in the above embodiments, or it may exist independently without being assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, can perform the various steps of the above-described method for training a neural network for medical image classification.

[0099] Another embodiment of this application provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described training method for a neural network for medical image classification.

[0100] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connected together may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0101] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0102] This document uses specific embodiments to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this application, and are not intended to limit this application. For those skilled in the art, changes can be made to the specific implementation methods and application scope based on the ideas, spirit and principles of this application. Any modifications, equivalent substitutions, improvements, etc., made should be included within the scope of protection of this application.

Claims

1. A method for training a neural network for medical image classification, characterized in that, The method includes: a. A visual converter with feature aggregation function is preset, and the visual converter is embedded as an encoding layer into the contrastive learning network; b. Provide multiple medical images for pre-training. For each medical image, input the medical image into the contrastive learning network. The contrastive learning network uses different image feature extraction methods to extract features of the first image region and features of the second image region. c. Input the first image region features and the second image region features into the visual converter for multi-layer encoding. Before encoding in the final encoding layer, use the feature aggregation function to aggregate the local image region encoding features obtained from multi-layer encoding to form global image region encoding features. After encoding in the final encoding layer, aggregate the global image region encoding features and the local image region encoding features obtained in the final encoding layer to form image region embedding features, thereby obtaining the first image region embedding feature and the second image region embedding feature. d. After processing the first image region embedding features and the second image region embedding features respectively using a multilayer perceptron (MLP) in a contrastive learning network, calculate the difference between the two using a preset contrastive loss function. If the difference between the two is greater than a preset threshold, adjust the parameters in the visual converter and return to the pre-training process of steps b to c until the difference is no greater than the preset threshold. e. After training the pre-trained visual converter with labeled medical images, a trained visual converter is obtained and applied to the classification of medical images.

2. The method as described in claim 1, characterized in that, The step of inputting the first image region features and the second image region features into the visual converter for multi-layer encoding includes: The visual converter includes at least one cascaded Transformer-based coding layer. In each coding layer from the first coding layer to the coding layer before the last coding layer, attention weights corresponding to each coding layer are used to encode the features of the first image region and the features of the second image region, respectively. For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

3. The method as described in claim 1, characterized in that, When the difference between the two is calculated using a preset contrast loss function, the contrast loss function is the robust contrast self-supervised loss function RS-InfoNCE.

4. The method as described in claim 3, characterized in that, The robust contrastive self-supervised loss function includes: in, = ; in the formula and These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP. The first and second image region embedding features represent the results of encoding and MLP processing of other different medical images used in pre-training by the visual converter. The similarity between the embedding features of the first image region and the embedding features of the second image region. Features are embedded into the first image region after feature regularization. This represents the hyperparameters, the initial settings. It is 0.

07.

5. The method as described in claim 1, characterized in that, After processing the data using a multi-layer MLP in a contrastive learning network, and before calculating the difference between the two values ​​using a pre-defined contrastive loss function, the following steps are also included: The predictor in the contrastive information network is used to make predictions to obtain the first image region embedding features and the second image region embedding features.

6. A training system for a neural network for medical image classification, characterized in that, include: Pre-training units and training units, wherein, The pre-training unit is used to embed a visual converter with feature aggregation function as an encoding layer into a contrastive learning network. Multiple medical images are provided for pre-training. For each medical image, the image is input into the contrastive learning network, which extracts first and second image region features using different image feature extraction methods. The first and second image region features are then input into the visual converter for multi-layer encoding. Before the final encoding layer, the feature aggregation function aggregates the local image region encoding features obtained from the multi-layer encoding to form global image region encoding features. After the final encoding layer, the global image region encoding features and the local image region encoding features obtained from the final encoding layer are aggregated to form image region embedding features, resulting in first and second image region embedding features. The first and second image region embedding features are processed using the MLP in the contrastive learning network, and a preset contrast loss function is used to calculate their difference. If the difference is greater than a preset threshold, the parameters in the visual converter are adjusted, and the pre-training process is repeated until the difference is no greater than the preset threshold. The training unit is used to train the pre-trained visual converter using medical images with labeled information to obtain a trained visual converter, which is then applied to the classification of medical images.

7. The system as described in claim 6, characterized in that, The pre-training unit is further configured to encode the first image region features and the second image region features respectively in each coding layer from the first coding layer to the coding layer before the last coding layer, using the attention weights corresponding to each coding layer. For the last encoding layer, the attention weights of each encoding layer from the first encoding layer to the last encoding layer are multiplied together to obtain the attention weights of the last encoding layer. The attention weights of the last encoding layer are then used to encode the features of the first image region and the features of the second image region, respectively.

8. The system as described in claim 6, characterized in that, When the difference between the two is calculated using a preset contrast loss function, the contrast loss function is the robust contrast self-supervised loss function RS-InfoNCE.

9. The system as described in claim 8, characterized in that, The robust contrastive self-supervised loss function includes: in, = ; in the formula and These represent the first and second image region embedding features of the same medical image used for pre-training, after being encoded by the visual converter and processed by the MLP. The first and second image region embedding features represent the results of encoding and MLP processing of other different medical images used in pre-training by the visual converter. The similarity between the embedding features of the first image region and the embedding features of the second image region. Features are embedded into the first image region after feature regularization. This represents the hyperparameters, the initial settings. It is 0.

07.

10. An electronic device, characterized in that, include: processor; A memory storing a program configured to implement, when executed by the processor, a method for training a neural network for medical image classification as described in any one of claims 1 to 5.

Images