Pre-training method and device of medical image coding network, equipment and storage medium

Abstract

Embodiments of the present application disclose a pre-training method and device of a medical image encoding network, an apparatus and a storage medium, which include: obtaining a medical image divided into a plurality of image blocks; selecting a to-be-masked image block from the plurality of image blocks; masking the to-be-masked image block using a mask image block to obtain a masked image; obtaining a query feature and an average feature using an encoding network and a momentum average network respectively, the encoding network and the momentum average network having the same structure, one of the query feature and the average feature being a global feature of the masked image and the other being a local feature of the mask image block; calculating a contrast loss based on the query feature and the average feature and updating model parameters of the encoding network according to the contrast loss; updating model parameters of the momentum average network according to the updated model parameters of the encoding network; and continuing training until the encoding network meets a pre-training stop condition. The above method can solve the technical problem of lack of a pre-training model for medical images in related technologies.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a pre-training method, apparatus, device, and storage medium for a medical image coding network. Background Technology

[0002] Deep learning (DL) refers to learning the inherent patterns and hierarchical representations of sample data to enable machines to possess analytical and learning capabilities similar to humans. With the development of deep learning technology, deep learning models are widely used in medical image processing. For example, deep learning models can be used to segment gliomas in brain images obtained through magnetic resonance imaging (MR) scans, thus aiding in glioma detection. Similarly, deep learning models can be used to detect lung nodules in lung images obtained through computed tomography (CT) scans, thus aiding in lung nodule detection. Furthermore, deep learning models can be used to detect chest abnormalities in chest images obtained through direct digital radiography (DR) systems, thus aiding in chest abnormality detection.

[0003] However, compared to deep learning models' processing of natural images (such as facial images), deep learning models lack pre-trained models on large datasets for medical image processing tasks. Pre-trained models can be considered as deep learning models pre-trained on large datasets, independent of specific tasks. During application, they can be fine-tuned directly on these pre-trained models to adapt them quickly to specific tasks. However, currently, there is a lack of pre-trained models specifically for medical images. Therefore, when processing medical images, deep learning models need to be trained from randomly initialized parameters. This requires a large amount of labeled medical image data for training, sophisticated model tuning strategies, and suffers from difficulties in model training, limited accuracy, and insufficient generalization ability. Summary of the Invention

[0004] This application provides a pre-training method, apparatus, device, and storage medium for medical image coding networks to address the technical problem of the lack of pre-trained models for medical images in related technologies.

[0005] In a first aspect, one embodiment of this application provides a pre-training method for a medical image coding network, comprising:

[0006] Acquire three-dimensional medical images, which are divided into multiple image blocks of the same size;

[0007] Select the image block to be masked from among the plurality of image blocks;

[0008] The image block to be masked in the medical image is masked using a masked image block to obtain the masked image;

[0009] The query feature and the average feature are obtained by using an encoding network and a momentum averaging network, respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0010] The contrast loss is calculated based on the query features and the average features, and the model parameters of the encoding network are updated according to the contrast loss.

[0011] The model parameters of the momentum averaging network are updated based on the updated model parameters of the encoding network.

[0012] Continue acquiring 3D medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition.

[0013] Secondly, one embodiment of this application also provides a pre-training device for a medical image coding network, comprising:

[0014] An acquisition module is used to acquire three-dimensional medical images, which are divided into multiple image blocks of the same size;

[0015] The selection module is used to select the image block to be masked from among the plurality of image blocks;

[0016] The masking module is used to mask the image block to be masked in the medical image using a mask image block, so as to obtain the masked image;

[0017] The feature determination module is used to obtain query features and average features using an encoding network and a momentum averaging network, respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0018] The first update module is used to calculate the contrast loss based on the query features and the average features, and update the model parameters of the encoding network according to the contrast loss;

[0019] The second update module is used to update the model parameters of the momentum averaging network based on the updated model parameters of the encoding network.

[0020] The retraining module is used to continue acquiring three-dimensional medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition.

[0021] Thirdly, one embodiment of this application also provides a pre-training device for a medical image coding network, comprising:

[0022] One or more processors;

[0023] Memory, used to store one or more programs;

[0024] When the one or more programs are executed by the one or more processors, the one or more processors implement the pre-training method for the medical image coding network as described in the first aspect.

[0025] Fourthly, one embodiment of this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the pre-training method for the medical image coding network as described in the first aspect.

[0026] In one embodiment of this application, a three-dimensional medical image is acquired, which is divided into multiple image blocks of the same size. Then, an image block to be masked is selected from each image block, and a mask image block is used to mask the image block to be masked, resulting in a masked image. The query features of the masked image are obtained by an encoding network, and the average features of the masked image are obtained by a momentum averaging network. Contrastive learning is performed based on the query features and the average features to calculate the contrast loss. The model parameters of the encoding network are updated according to the contrast loss, and the model parameters of the momentum averaging network are updated according to the updated model parameters of the encoding network to complete one pre-training. This process continues until the encoding network meets the pre-training stopping condition. This technical means solves the technical problem of the lack of pre-trained models for medical images in related technologies. When performing comparative learning, the global features of the masked image and the local features of the masked image patches in the masked image are utilized. Based on the idea that the matching degree of global and local features of the same masked image is high and the matching degree of global and local features of different masked images is low, a sample pair is constructed to perform a comparative learning mechanism for matching global and local features. This can solve the problem that the high similarity of medical images in the same location and the same modality is not conducive to comparative learning. Moreover, there is no need to augment the medical images, which reduces the dependence on data augmentation. Attached Figure Description

[0027] Figure 1A flowchart illustrating a pre-training method for a medical image coding network, provided as an embodiment of this application;

[0028] Figure 2 A medical image provided as an embodiment of this application;

[0029] Figure 3 A flowchart illustrating a pre-training method for a medical image coding network, provided as an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of a multilayer sensing network provided in one embodiment of this application;

[0031] Figure 5 A flowchart illustrating a pre-training method for an image coding network provided in one embodiment of this application;

[0032] Figure 6 A schematic diagram of a reconstruction module provided in one embodiment of this application;

[0033] Figure 7 A schematic diagram of the momentum averaging model processing flow using MoCo provided in one embodiment of this application;

[0034] Figure 8 A flowchart illustrating the construction of a masked image according to one embodiment of this application;

[0035] Figure 9 A flowchart of self-supervised pre-training provided for one embodiment of this application;

[0036] Figure 10 A schematic diagram of training loss provided for one embodiment of this application;

[0037] Figure 11 A schematic diagram of verification loss provided for one embodiment of this application;

[0038] Figure 12 A schematic diagram of the structure of a pre-training device for a medical image coding network provided in one embodiment of this application;

[0039] Figure 13 This is a schematic diagram of the structure of a pre-training device for a medical image coding network provided in one embodiment of this application. Detailed Implementation

[0040] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0041] For natural image processing tasks, self-supervised pre-training of deep learning models can be performed using methods such as Momentum Contrastive Learning (MOCO) or Visual Representation Contrastive Learning (SIMCLR). Self-supervised pre-training refers to learning directly from unlabeled data without the need for annotation. The idea behind this training method is to allow comparative learning between images from different individuals. After augmentation, the high-level semantic features extracted by the encoding network for the same image show high similarity, while the similarity between different images is low. This training method has achieved relatively good or even better results in supervised pre-training on millions of data points. However, for medical images, the similarity between different individuals in the same part and modality is very high. For example, chest images from different individuals obtained using CT scans have very similar grayscale spaces and structural content, resulting in minimal data augmentation and poor pre-training performance. Therefore, it is difficult to directly perform self-supervised pre-training on data-augmented medical images using the contrastive learning approach.

[0042] Based on this, this application provides a pre-training method, apparatus, device, and storage medium for a medical image coding network to achieve self-supervised pre-training of a deep learning model for medical images, obtain the corresponding pre-trained model, and avoid the problem of poor image pre-training effect due to high similarity between medical images of different individuals.

[0043] One embodiment of this application provides a pre-training method for a medical image coding network, which can be executed by a pre-training device for the medical image coding network. This pre-training device can be implemented through software and / or hardware. The pre-training device can consist of two or more physical entities, or it can consist of a single physical entity; this embodiment does not limit the specific implementation. In one embodiment, the pre-training device for the medical image coding network can be an electronic device such as a desktop computer, laptop computer, interactive smart panel, server, CT equipment, MR equipment, or DR equipment.

[0044] For example, Figure 1 A flowchart illustrating a pre-training method for a medical image coding network, as provided in one embodiment of this application. (Reference) Figure 1 The pre-training methods for this medical image coding network include:

[0045] Step 110: Obtain a three-dimensional medical image, which is divided into multiple image blocks of the same size.

[0046] Medical images are used in the pre-training process and are three-dimensional images; therefore, they can also be referred to as 3D medical images. The method of acquiring medical images is not currently limited; for example, they can be acquired through CT, MR, or DR methods, or by searching the internet or databases for publicly available medical images. The corresponding human body part for the medical images is not currently limited; for example, the medical images can be chest images, lung images, or brain images. In one embodiment, a medical image dataset is pre-constructed, and the medical images in this dataset are used in the pre-training process. Subsequently, during each pre-training iteration, medical images are directly retrieved from this dataset. It is understood that the same medical image can be reused during the pre-training process.

[0047] For example, a medical image is pre-divided into multiple image blocks, each of which can be considered a sub-3D image of the medical image, and the image blocks together constitute the medical image. In one embodiment, all image blocks are of the same size. In this case, the size of each image block can be set according to the size of the medical image and actual needs, and it is necessary to ensure that the size of the medical image is divisible by the size of the image blocks. For example, Figure 2 A medical image provided as an embodiment of this application, with reference to Figure 2 The dimensions of the medical image are D×H×W, and the size of the image patch is p. d ×p h ×p w It is divided into multiple image blocks of equal size. It should be noted that... Figure 2 The image shown is a single plane patch from a medical image. Assume D = 32, W = H = 256, and set p... d =4, p h =p w =32. After segmenting the medical image based on the size of the image patch, an 8×8×8 image patch can be obtained. It is understood that the unit of measurement for the aforementioned size is not currently limited and can be in pixels or in distance (such as cm, mm).

[0048] In one embodiment, the medical images in the medical image dataset are pre-divided and stored as image patch combinations. In this case, acquiring the medical image can be done by directly acquiring the three-dimensional medical image obtained by combining image patches. In another embodiment, the medical images are not pre-divided. After each acquisition of a medical image, it is cut according to the size of the image patch to obtain the image patches that make up the medical image.

[0049] Step 120: Select the image block to be masked from among multiple image blocks.

[0050] An image patch to be masked refers to an image patch that needs to be masked. The number of image patches to be masked is typically one. In one embodiment, one image patch from the various image patches of a medical image is selected as the image patch to be masked. When the deep learning model subsequently learns and predicts information from the medical image, it no longer learns to predict the information expressed by the image patch to be masked.

[0051] For example, the entropy of an image patch represents the amount of information it conveys. Here, image entropy can be understood as an estimate of the "busyness" of the image; the higher the entropy, the more information the image patch conveys. The entropy of an image patch can be one-dimensional, two-dimensional, or three-dimensional, etc., and is not limited here. After calculating the entropy of each image patch, the image patch with the most information is selected as the image patch to be masked, so that the subsequent learning and prediction of each image patch by the deep learning model is meaningful. In one embodiment, the selection of an image patch can be: pre-setting an entropy threshold, and then randomly selecting an image patch from those with entropy greater than the threshold as the image patch to be masked; or it can be: selecting a certain number or proportion of image patches in descending order of entropy, and then randomly selecting one image patch from the selected image patches as the image patch to be masked. It is understood that by randomly selecting, the image patch to be masked can be different each time the medical image is reused.

[0052] Step 130: Use masked image blocks to mask the image blocks to be masked in the medical image to obtain the masked image.

[0053] A mask image patch is a pre-generated image patch used to mask out the image patch to be masked in a medical image. In other words, it replaces the image patch to be masked, preventing subsequent deep learning models from learning and predicting the information represented by the image patch to be masked. The size of the mask image patch is the same as the size of each image patch. The generation rules for the mask image patch can be set according to the actual situation. For example, an image patch with each pixel value of 0 can be generated as the mask image patch. Alternatively, an image patch can be randomly selected from all image patches in the medical image (excluding the image patch to be masked) as the mask image patch. To ensure the diversity of the mask image patch, image patches with fixed pixel values, other image patches in the medical image excluding the image patch to be masked, and the image patch to be masked can all be used as mask image patches. Then, a mask image patch is selected from the three types of mask image patches for use.

[0054] In one embodiment, after masking the image block to be masked using a mask image block, the medical image changes; the image block to be masked is no longer displayed, but the mask image block is displayed instead. Currently, the medical image that displays the mask image block is denoted as the masked image.

[0055] Step 140: Obtain the query feature and the average feature using the encoding network and the momentum averaging network respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0056] The encoding network is a pre-built neural network model that processes input medical images to encode relevant features. In one embodiment, the encoding network includes at least a deep learning model that needs to be pre-trained. This deep learning model is a feature encoder, primarily used to extract features from the medical images. Optionally, in addition to the deep learning model, the encoding network may also include other structures required for pre-training. For example, the encoding network may also include a projector composed of multilayer perceptrons, which is used to project and map the features extracted by the deep learning model. This decouples the projected features from the specific task of the deep learning model (such as a classification task), ensuring that the projected features are only related to the medical images and not to the current specific task of the deep learning model, thereby guaranteeing the effectiveness of pre-training.

[0057] For example, after the masked image is input into the encoding network, the encoding network outputs global features and local features. The global features represent the features of the masked image, obtained by learning the information expressed by each image patch of the masked image (including the mask image patch). The local features represent the features of the mask image patch, obtained by learning the information expressed by the mask image patch and the surrounding image patches. It can be understood that when extracting features of the masked image through the deep learning model in the encoding network, each image patch of the masked image (including the mask image patch) has corresponding features. Then, global features can be obtained based on the features of each image patch, and local features can be obtained through the features corresponding to the mask image patch.

[0058] It is understandable that when the encoding network obtains local features, it references the information expressed by the surrounding image patches of the masked image patch. Therefore, the matching degree between the local feature and the global features of the same masked image is high, while the matching degree between the local feature and the global features of other masked images is low. Based on this, in one embodiment, global features and local features are constructed as sample pairs, and the global and local features of the same masked image are used as positive sample pairs (positive sample pairs have a high matching degree), while the global and local features of different masked images are used as negative sample pairs (negative sample pairs have a low matching degree). This allows the encoding network to be pre-trained through comparative learning using positive and negative sample pairs. Comparative learning is a commonly used self-supervised learning method, the core idea of ​​which is to bring positive samples closer together and widen the distance between positive and negative samples. In one embodiment, the momentum averaging model of MoCo is used to implement comparative learning. MoCo is an application of unsupervised machine learning in the field of image recognition (representation), and it is a pre-trained method that can be used for transfer learning. When using MoCo, a momentum averaging network also needs to be constructed, which has the same structure as the encoding network. After the masked image is input into the momentum averaging network, it can also output global and local features. To distinguish them, the global and local features output by the encoding network are denoted as the first global feature and the first local feature, respectively, while the global and local features output by the momentum learning network are denoted as the second global feature and the second local feature, respectively. The difference between the momentum averaging network and the encoding network is that the medical image encoding network uses conventional gradient backpropagation to update the model parameters, while the momentum averaging network updates the momentum based on the model parameters updated by the encoding network.

[0059] Based on the idea of ​​contrastive learning, a sample pair is formed by combining the first local feature output by the encoding network with the second global feature output by the momentum learning network, or vice versa. Since the subsequent processing methods are the same for both combination methods, the example of forming a sample pair with the first local feature and the second global feature will be described below.

[0060] In the implementation of MOCO, the momentum learning model uses a dynamic dictionary, which can be considered a dynamic queue, the length of which can be set according to actual conditions. After obtaining each second global feature, the momentum learning model adds it to the dynamic dictionary and removes the first recorded second global feature. After obtaining the first local feature, the encoding network combines the first local feature with each second global feature in the dynamic dictionary to form a sample pair. The common features in each sample pair are the features output by the encoding network (here referring to the first local feature), and the different features in each sample pair are the features output by the momentum learning model (here referring to the second global feature). At this point, the common features in each sample pair are denoted as the query feature, and the different features are denoted as the average feature, and the corresponding loss function is calculated for each sample pair composed of the query feature and the average feature. Understandably, at the beginning of pre-training, a certain number of masked images can be generated, and one of them can be selected and input into the encoding network to obtain query features. Each masked image is then sequentially input into the momentum learning network to obtain average features, which are written into the dynamic dictionary. Afterward, positive and negative sample pairs are formed based on the query features and the average features. During the next training iteration, the same masked image can be input into both the encoding and momentum learning networks. In this case, the momentum learning network writes the output average features into the dynamic dictionary, removes the first average feature written into the dynamic dictionary, and forms new sample pairs again.

[0061] After obtaining the average features and query features, and forming template pairs, the loss function can be calculated. That is, proceed to step 150.

[0062] Step 150: Calculate the contrastive loss based on the query features and average features, and update the model parameters of the encoding network according to the contrastive loss.

[0063] After obtaining the query features and average features, and forming sample pairs, contrastive learning is performed, and the model parameters of the encoding network are updated based on the results of the contrastive learning. The results of the contrastive learning are reflected by the contrastive loss. The contrastive loss is a specific value calculated using the contrastive loss function, a common loss function that reflects the clustering effect of the same category (i.e., positive sample pairs) and the separation effect of different categories (i.e., negative sample pairs). Substituting the positive and negative sample pairs into the contrastive loss function yields a loss value; the lower the contrastive loss, the better the contrastive learning result.

[0064] Subsequently, the model parameters of the encoding network are adjusted based on the contrastive loss. These parameters include the weights and biases of each convolutional kernel used in the encoding network. It can be understood that during pre-training, minimizing the contrastive loss is the objective. By continuously adjusting the model parameters of the encoding network using the obtained contrastive loss each time, the contrastive learning performance of the encoding network improves. As the contrastive learning performance of the encoding network improves, the deep learning model within the encoding network extracts features more accurately from the masked image. Since the masked image patch covers image patches with abundant information, even if the masked image lacks information-rich image patches, the deep learning model can still extract relatively accurate features, further improving the performance of the deep learning model.

[0065] Step 160: Update the model parameters of the momentum-averaged network based on the updated model parameters of the encoding network.

[0066] For example, the momentum of the momentum averaging network is updated by using the model parameters of the encoding network. In one embodiment, the formula for momentum update is as follows:

[0067] θ k ←mθ k +(1-m)θ q

[0068] Where, θ k θ represents the model parameters of the momentum-averaged network. q This represents the model parameters of the encoding network, and m is the updated momentum (i.e., momentum coefficient), which can optionally be between 0 and 1. For example, substituting the updated model parameters of the encoding network into the above formula yields the new model parameters of the momentum-averaged network, thus updating the momentum-averaged network model parameters. It can be understood that the model parameters of the momentum-averaged network also include parameters such as the weights and biases of each convolutional kernel used in the momentum-averaged network.

[0069] Step 170: Continue acquiring 3D medical images and return to the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition.

[0070] For example, after updating the model parameters of both the encoding network and the momentum averaging network, the pre-training can be considered complete. Then, it is determined whether the current encoding network meets the pre-training stopping condition. If it does, it means the deep learning model in the encoding network has reached the expected performance (e.g., it has accurately extracted features from medical images), and further pre-training is unnecessary; therefore, the pre-training of the encoding network ends. If the pre-training stopping condition is not met, it means the deep learning model in the encoding network has not reached the expected performance, and further pre-training is required. When continuing pre-training, a new medical image is selected. Since medical images can be reused, the new medical image can be the same as or different from the previously used image. Then, steps 120 to 160 are repeated to update the model parameters of the encoding network and the momentum averaging network again. Afterward, it is again determined whether the pre-training stopping condition is met, until the encoding network meets the pre-training stopping condition.

[0071] The pre-training stopping condition can be set according to actual needs. For example, the pre-training stopping condition can be a pre-set threshold for the number of iterations. After each update of the model parameters of the encoding network and the momentum averaging network, the number of pre-training iterations is incremented by 1. Then, the currently recorded number of pre-training iterations is compared with the threshold. If the threshold is reached, the pre-training stopping condition is confirmed to be met; otherwise, it is confirmed that the pre-training stopping condition has not been met. Another example is that the pre-training stopping condition is the convergence of the loss function, i.e., the contrastive loss for consecutive iterations is within a preset range. It can be understood that after the loss function converges, the performance of the encoding network is relatively stable. Loss function convergence means that the contrastive loss within consecutive training iterations is within a preset range.

[0072] In one embodiment, after pre-training, a deep learning model in the encoding network is obtained. This deep learning model serves as a pre-trained model suitable for medical images, capable of accurately extracting features from medical images. This pre-trained model can be used as the initial deep learning model for downstream tasks. By using a small number of labeled samples from the downstream task to train and fine-tune the pre-trained model, it can be adapted to the downstream task with good accuracy. For example, if the downstream task is to detect lung nodules, a small number of lung medical images with lung nodule markings are obtained as labeled samples. The pre-trained model is then trained using these labeled samples, and its parameters are adjusted during training. It is understood that because the number of labeled samples is small, only a small amount of training is needed to adapt the pre-trained model to the task of lung nodule detection.

[0073] The above-described method involves acquiring a three-dimensional medical image, dividing it into multiple image blocks of the same size, selecting an image block to be masked within each block, and using a mask image block to mask the image block, resulting in a masked image. A coding network obtains the query features of the masked image, and a momentum averaging network obtains the average features of the masked image. Contrastive learning is then performed based on the query features and the average features to calculate the contrast loss. The model parameters of the coding network are updated according to the contrast loss, and the model parameters of the momentum averaging network are updated according to the updated model parameters of the coding network to complete one pre-training cycle. This process continues until the coding network meets the pre-training stopping condition. This technique addresses the technical problem of the lack of pre-trained models specifically for medical images in related technologies. When performing comparative learning, the global features of the masked image and the local features of the masked image patches in the masked image are utilized. Based on the idea that the matching degree of global and local features of the same masked image is high and the matching degree of global and local features of different masked images is low, a sample pair is constructed to perform a comparative learning mechanism for matching global and local features. This can solve the problem that the high similarity of medical images in the same location and the same modality is not conducive to comparative learning. Moreover, there is no need to augment the medical images, which reduces the dependence on data augmentation.

[0074] Figure 3 This is a flowchart illustrating a pre-training method for a medical image coding network according to one embodiment of this application. This embodiment is a further refinement based on the above-described embodiment. (See reference...) Figure 3 The pre-training method specifically includes:

[0075] Step 210: Acquire a three-dimensional medical image, which is divided into multiple image blocks of the same size.

[0076] Step 220: Calculate the one-dimensional entropy of each image patch.

[0077] In one embodiment, one-dimensional entropy is used to represent the amount of information expressed by each image patch. Here, one-dimensional entropy is the amount of information contained in the clustering features of gray-level distribution within an image patch. Each image patch has a corresponding one-dimensional entropy.

[0078] Step 230: Filter the image blocks based on one-dimensional entropy.

[0079] For example, based on the one-dimensional entropy of each image patch, image patches with more information are selected for retention, while image patches with less information are filtered out. In one embodiment, step 230 includes steps 231-232:

[0080] Step 231: Select one-dimensional entropy greater than a preset threshold from all one-dimensional entropy, or, based on the sorting results of the size of each one-dimensional entropy, select a preset number of one-dimensional entropy starting from the largest one-dimensional entropy.

[0081] In one embodiment, the preset threshold is a pre-defined one-dimensional entropy threshold, which can be set according to actual conditions. If the one-dimensional entropy is greater than the preset threshold, it indicates that the corresponding image block contains a large amount of information, and the image block is retained. If the one-dimensional entropy is not greater than the preset threshold, it indicates that the corresponding image block contains less information, and the image block is filtered out. Accordingly, the one-dimensional entropy of each image block in the three-dimensional medical image is compared sequentially with the preset threshold, and then the one-dimensional entropy higher than the preset threshold is selected.

[0082] In one embodiment, the preset number is obtained by combining the total number of image blocks with a preset ratio. The preset ratio is a pre-defined proportion, the specific value of which can be set according to actual conditions. The larger the preset ratio, the more image blocks are selected. For example, the number of image blocks to be selected, i.e., the preset number, is determined based on the preset ratio and the total number of image blocks in the medical image. Then, starting from the largest one-dimensional entropy, one-dimensional entropies are selected according to the preset number. In one embodiment, when selecting one-dimensional entropies, they can be sorted from largest to smallest, and the preset number of one-dimensional entropies is selected based on the sorting result from largest to smallest.

[0083] Step 232: Retain the image patch corresponding to the selected one-dimensional entropy.

[0084] The image block corresponding to the one-dimensional entropy selected in step 231 is retained, and the remaining image blocks are filtered out.

[0085] Step 240: Select one image block from the remaining image blocks as the image block to be masked.

[0086] The retained image blocks represent a large amount of information, and each image block can be used as the image block to be masked. In one embodiment, an image block is randomly selected from the retained image blocks as the image block to be masked.

[0087] Step 250: Select the currently used mask image block from the fixed value image block, the image block to be masked, and other image blocks of the medical image.

[0088] In one embodiment, a fixed-value image block, an image block to be masked, and other image blocks of the medical image are used as mask image blocks. A fixed-value image block refers to an image block where each pixel has a fixed value. The fixed value can be set according to actual conditions; for example, when the fixed value is 0, the fixed-value image block refers to an image block where all pixels are 0. The image block to be masked can also be used as a mask image block. In this case, after masking the image block to be masked with a mask image block, the corresponding medical image remains unchanged. Other image blocks are other image blocks in the medical image besides the image block to be masked.

[0089] For the three types of masked image patches mentioned above, corresponding selection probabilities are assigned. Each selection probability can be set according to actual conditions to ensure the diversity and rationality of the masked image patches. Specifically, fixed-value image patches correspond to a first selection probability, other image patches correspond to a second selection probability, and the image patch to be masked corresponds to a third selection probability. The first selection probability is greater than the second selection probability, and the second selection probability is greater than the third selection probability. The first, second, and third selection probabilities can be set according to actual conditions. For example, a first selection probability of 80%, a second selection probability of 15%, and a third selection probability of 5% means that the probability of a masked image patch being a fixed-value image patch is 80%, the probability of it being another image patch is 15%, and the probability of it being the image patch to be masked is 5%. When medical images are reused, the randomness of the mask patch to be masked and the diversity of the masked image patches ensure the diversity of the masked images corresponding to the medical images, thus ensuring the diversity of the training samples.

[0090] Optionally, all image patches of all medical images used in the pre-training process can be used as mask image patches. In this case, each image patch of all medical images can be considered as another image patch and corresponds to the second selection probability.

[0091] Step 260: Use masked image blocks to mask the image blocks to be masked in the medical image to obtain the masked image.

[0092] Step 270: Obtain the query feature and the average feature using the encoding network and the momentum averaging network respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0093] For example, the query features of the masked image are obtained using an encoding network, and the average features of the masked image are obtained using a momentum averaging network.

[0094] In one embodiment, the encoding network consists of a feature encoder, a first pooling layer, a projector, a second pooling layer, and a predictor. In this case, obtaining the query features using the encoding network may include steps 271-275:

[0095] Step 271: The feature encoder extracts features from the masked image to obtain a three-dimensional feature map.

[0096] The feature encoder is a deep learning model in the encoding network that needs to be pre-trained to extract features from medical images. The feature encoder can be a Convolutional Neural Network (CNN) model, a Transformer model, or a combination of CNN and Transformer models. In one embodiment, the feature encoder uses a 3D convolutional neural network to process three-dimensional medical images. For example, the feature encoder parameters are shown in Table 1.

[0097]

[0098] Table 1

[0099] Referring to Table 1, Layer name represents the name of the network layer. Conv1 represents the first convolutional layer in the feature encoder, Conv2_x represents the second convolutional layer, Conv3_x represents the third convolutional layer, Conv4_x represents the fourth convolutional layer, and Conv5_x represents the fifth convolutional layer. The 'x' in the second to fifth convolutional layers indicates the number of convolutional layers stacked within that layer; the 'x' can be different for different convolutional layers. Output_size represents the size of the feature map output by each layer, which reflects the downsampling ratio of the convolutional layer. For example, if the size of the 3D image input to a convolutional layer is D×H×W, the size of the output of that convolutional layer would be... At this point, the downsampling ratios of the convolutional layer in the three directions are s, respectively. d s h s w As shown in Table 1, the input feature map size of Conv2 is 32×128×128, and the output feature map size is 32×64×64, indicating that Conv2 uses an asymmetric downsampling ratio. The 33-layer represents the parameters used in the network layer. In Conv1, the convolutional kernel size is 1×5×5, the number of output channels is 16, and the stride is 1×2×2. Conv2 is a 3D convolutional layer. This represents the kernel size and number of output channels in a 3D convolutional layer. The kernel size is 1×3×3, and the number of output channels is 32. This indicates that Conv2 contains three stacked convolutional layers, meaning x is 3. Based on the kernel size and output feature map size of Conv2, it can be determined that Conv2 is an asymmetric 3D convolution. Conv3 is a 3D convolutional layer. This represents the kernel size and number of output channels in 3D convolution, where the kernel size is 3×3×3 and the number of output channels is 64. This indicates that Conv3 contains four stacked convolutional layers, i.e., x is 4. The parameters for Conv4 and Conv5 are similar to those for Conv3, and will not be elaborated upon here. Based on the above parameters, it can be seen that the number of convolutional layers in the feature encoder is the same as that in ResNet34 (a common residual network). The difference lies in the fact that the feature encoder uses 3D convolutional layers, and the front-end part of the feature encoder (Conv2 in Table 1) uses asymmetric convolutional layers and an asymmetric downsampling ratio. In this case, the feature encoder can be considered to use an asymmetric 3D convolutional network. It should be understood that Table 1 is only an exemplary description of a feature encoder. In practical applications, the feature encoder can also use other structures and parameters, which are not limited here.

[0100] It is understandable that when pre-training the encoding network, the specific process involves self-supervised pre-training of the feature encoder.

[0101] For example, after the image to be masked is input into the feature encoder, the feature encoder extracts the features of the masked image and outputs a three-dimensional feature map. In one embodiment, the feature map output by the feature encoder is denoted as a three-dimensional feature map. It can be understood that the features of each image block (including the masked image block) in the masked image are reflected in the three-dimensional feature map.

[0102] In one embodiment, the downsampling ratio of the feature encoder is less than or equal to the size of the image patch. Since medical images are three-dimensional images, containing dimensions in three directions, the downsampling ratio in each direction is smaller than the size of the image patch in the corresponding direction. For example, the size of the masked image is D×H×W, and the size of the three-dimensional feature map is... The downsampling ratios of this 3D feature map in the three directions are s d s h s w The size of each image patch in the masked image is p. d ×p h ×p w The number of image patches is when Greater than or equal to At that time, i.e., s d Less than or equal to p d s h Less than or equal to p h s w Less than or equal to p wIn this case, the features of each image patch in the masked image are easily distinguishable in the 3D feature map; that is, the feature vector represented by a feature point in the 3D feature map is the feature encoding vector corresponding to one image patch. It is understandable that when the downsampling ratio of the 3D feature map in a corresponding direction is greater than the size of the image patch in that direction, the feature vector represented by one feature point in the 3D feature map may be the feature encoding vector of at least two image patches. In this case, it is difficult to distinguish the features corresponding to each image patch. To achieve better correspondence, in one embodiment, the size of the image patch is an integer multiple of the downsampling ratio of the feature encoder; that is, the size of the image patch in each direction is an integer multiple of the downsampling ratio in that direction. In this case, the feature vector of each feature point in the 3D feature map is the feature encoding vector corresponding to one image patch, facilitating the subsequent extraction of the first local features of the masked image patch. For example, the size of the masked image is 32×256×256, the size of the image patch is 4×32×32, the number of image patches contained in the masked image (including the mask image patch) is 8×8×8, the downsampling ratio in the three directions of the 3D feature map is 4×32×32, and the size of the 3D feature map is 8×8×8. In this case, each feature point in the 3D feature map corresponds to an image patch, representing the feature encoding vector of that image patch.

[0103] After the feature encoder outputs the 3D feature map, step 272 is executed.

[0104] Step 272: The first pooling layer obtains the mask features of the masked image block and the image features of other image blocks in the masked image from the three-dimensional feature map. Each image block corresponds to one image feature.

[0105] Pooling layers can achieve functions such as downsampling and dimensionality reduction, and removal of redundant information. In one embodiment, by processing the 3D feature map through a pooling layer, the feature encoding vectors of each image patch (including the masked image patch) in the masked image can be obtained. That is, the 3D features describing the image patch in the 3D feature map are transformed into one-dimensional feature vectors describing the image patch. Here, the currently used pooling layer is denoted as the first pooling layer, the feature encoding vector of the masked image patch is denoted as the mask feature, and the feature encoding vectors of other non-masked image patches are denoted as the image feature. It can be understood that both the mask feature and the image feature are one-dimensional feature vectors.

[0106] In one embodiment, when the size of the image patch is an integer multiple of the downsampling ratio of the feature encoder, Region of Interest (ROI) pooling is used to obtain mask features and image features; that is, the first pooling layer uses ROI pooling. Specifically, this step includes: using ROI pooling to obtain the mask features of the mask feature patch and the image features corresponding to other image patches in the masked 3D image from the 3D feature map. Here, ROI pooling is a neural network layer used for object detection tasks. Currently, the object detection task of ROI pooling is to use the mask features and image features as targets to detect the 3D feature map to obtain the mask features and image features, i.e., to obtain the feature encoding vector of each image patch. Since the size of the image patch is an integer multiple of the downsampling ratio of the feature encoder, ROI pooling can accurately find the 3D features corresponding to each image patch in the 3D feature map, and then obtain the corresponding one-dimensional feature vector based on the 3D features.

[0107] Step 273: The projector maps and projects the mask features and each image feature to obtain each isolated feature block.

[0108] In one embodiment, the feature encoder needs to extract 3D feature maps according to the specific task during use. For example, when the feature encoder is used for classification, it extracts 3D feature maps based on the classification category; similarly, when the feature encoder is used for segmentation, it extracts 3D feature maps based on the segmentation target. To complete the pre-training of the feature encoder, a task needs to be assumed for the feature encoder during self-supervised training. However, the pre-trained feature encoder needs to be applicable to different tasks. In this case, the 3D feature maps obtained based on the assumed task need to be isolated from the assumed task, i.e., the influence of the assumed task on mask features and image features during self-supervised training needs to be isolated.

[0109] To isolate the influence of the hypothetical task on the mask features and image features, a projector is designed after the first pooling layer. This projector is used to map and project the mask features and image features, that is, to perform a mapping transformation on the mask feature blocks and image feature blocks. In one embodiment, the features obtained after mapping and projection are denoted as isolated features. The isolated features are one-dimensional features. At this time, each image feature and mask feature has a corresponding isolated feature. The isolated features can be considered as features mapped to a new space. The features in the new space are independent of the hypothetical task; that is, the isolated features are features obtained after being isolated from the hypothetical task.

[0110] In one embodiment, the projector employs a multilayer perceptron network. A multilayer perceptron (MLP), also referred to as a multilayer perceptron, is a feedforward artificial neural network model that maps multiple input datasets to a single output dataset. Currently, the multilayer perceptron network used by the projector is referred to as the first multilayer perceptron network. The number of hidden layers and model parameters in the first multilayer perceptron network can be set according to actual conditions and are not currently limited.

[0111] Step 274: Use the second pooling layer to fuse all isolated features to obtain the first global feature of the masked image.

[0112] For example, a pooling layer is connected after the projector to fuse the isolated features. Currently, this pooling layer is referred to as the second pooling layer. In one embodiment, the second pooling layer is a Global Average Pooling (GAP) layer. That is, the isolated features are fused using global average pooling. The fused features contain the features of all image patches in the masked image; therefore, the fused features can be understood as the first global feature of the masked image, which is a one-dimensional feature.

[0113] In practical applications, the second pooling layer can be replaced by a convolutional layer to fuse the isolated features, or the second pooling layer can be omitted and a weighted average method can be used to fuse the isolated features. The parameters used in the weighted average process (such as weights) are learnable parameters, which can be adjusted according to the pre-training process.

[0114] Step 275: The predictor predicts the isolated feature blocks corresponding to the masked image blocks to obtain the first local features corresponding to the masked image blocks.

[0115] In one embodiment, a predictor is connected after the projector to obtain the first local feature corresponding to the masked image block. This predictor predicts the isolated feature block corresponding to the masked image block to obtain the first local feature of the masked image block; that is, it projects and maps the isolated features related to the masked image block to obtain the first local feature representing the masked image block. The first local feature is a one-dimensional feature.

[0116] In one embodiment, the predictor is also a multilayer perceptron network, and the multilayer perceptron network used by the predictor is referred to as the second multilayer perceptron network. That is, the predictor uses a second multilayer perceptron network, and the number of hidden layers and model parameters in the second multilayer perceptron network can be set according to actual conditions and are not currently limited. In one embodiment, both the projector and the predictor's multilayer perceptron networks have one hidden layer. Figure 4 This is a schematic diagram of a multilayer sensing network provided in one embodiment of this application. (Reference) Figure 4 It shows the specific structure of the hidden layers, where the Linear layer is a linear layer, the Norm layer is a normalization layer, and the Activation layer is an activation function layer.

[0117] It is understandable that when the encoding network obtains the first local features and the first global features, the momentum averaging network also processes the masked image to obtain the second local features and the second global features. The momentum averaging network has the same structure as the encoding network, and the way the momentum averaging network obtains the second global features and the second local features is the same as the way the encoding network obtains the first global features and the first local features, which will not be elaborated here.

[0118] After obtaining the first local feature, the first global feature, the second global feature, and the second local feature, the first local feature and the second global feature are used as the query feature and the average feature, respectively. Alternatively, the first global feature and the second local feature are used as the query feature and the average feature, respectively. Then, step 280 is executed.

[0119] Step 280: Add the currently obtained average features to the dynamic dictionary, which also records the average features of other masked images.

[0120] The dynamic dictionary can also be considered a dynamic queue, used to record the average features obtained by the momentum averaging network. The length of the dynamic dictionary can be set according to the actual situation. Each time an average feature is added to the dynamic dictionary, the first average feature added to the dynamic dictionary is removed. Optionally, during the initial training, the number of masked images processed by the momentum averaging network is equal to the length of the dynamic dictionary, so that during subsequent pre-training, the dynamic dictionary will remove the first recorded average feature whenever a new average feature is added.

[0121] Step 290: Take the average features and query features belonging to the same masked image as positive sample pairs, and take the average features and query features belonging to different masked images as negative sample pairs.

[0122] In one embodiment, each average feature in the dynamic dictionary can form a sample pair with a query feature obtained from the encoding network. Average features obtained from the same masked image and query features with a high degree of matching are considered positive sample pairs. Average features obtained from different masked images and query features with a low degree of matching are considered negative sample pairs.

[0123] Step 2100: Calculate the contrast loss using positive and negative sample pairs.

[0124] For example, after obtaining positive and negative sample pairs, contrastive learning is performed, i.e., calculating the specific loss value of the contrastive loss function. The contrastive loss function is mainly used in dimensionality reduction; that is, samples that were originally similar remain similar in the feature space after dimensionality reduction (feature extraction), while samples that were originally dissimilar remain dissimilar in the feature space. Similarly, this loss function can also well express the degree of matching between pairs of samples (i.e., positive and negative sample pairs). The contrastive loss function is often used in momentum averaging models utilizing MoCo. Substituting positive and negative sample pairs into the contrastive loss function yields the contrastive loss.

[0125] Step 2110: Update the model parameters of the encoding network according to the contrastive loss function.

[0126] For example, after obtaining the contrast loss, the model parameters of the medical image coding network can be updated based on the contrast loss. Specifically, updating the model parameters through gradient calculation allows for a smaller contrast loss, which better represents the matching degree of paired samples. Gradient calculation is the most commonly used optimization algorithm during neural network training; its specific implementation process will not be elaborated upon here.

[0127] Step 2120: Update the model parameters of the momentum-averaged network based on the updated model parameters of the encoding network.

[0128] Step 2130: Continue acquiring three-dimensional medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition.

[0129] As described above, when selecting image patches to be masked, filtering each image patch of the 3D medical image using one-dimensional entropy avoids low-information image patches interfering with the pre-training of the encoding network, preventing meaningless comparative learning and ensuring pre-training accuracy. By setting different types of mask image patches and their selection probabilities, the diversity of mask image patches is ensured, increasing the diversity of input samples (i.e., the masked images). A reasonable downsampling ratio is set for the feature encoder based on the image patch size, ensuring that each feature point in the 3D feature map corresponds to only one image patch, facilitating subsequent feature extraction from each image patch. The projector and predictor avoid the influence of the assumed task on features during pre-training, decoupling the pre-trained feature encoder from the specific task. Through a matching and comparative learning mechanism of global and local features, high matching degrees between global and local features of the same masked image and low matching degrees between global and local features of different masked images are used to construct positive and negative sample pairs for comparative learning. This avoids the problem of image pre-training effectiveness due to high similarity between medical images of different individuals, ensuring the effectiveness of pre-training.

[0130] Figure 5 This is a flowchart illustrating a pre-training method for an image coding network according to one embodiment of this application. This embodiment is a further refinement based on the above-described embodiment. (See reference...) Figure 5 The pre-training method specifically includes:

[0131] Step 310: Obtain a three-dimensional medical image, which is divided into multiple image blocks of the same size.

[0132] Step 320: Calculate the one-dimensional entropy of each image patch.

[0133] Step 330: Filter the image blocks based on one-dimensional entropy;

[0134] Step 340: Select one image block from the remaining image blocks as the image block to be masked.

[0135] Step 350: Select the currently used mask image block from the fixed value image block, the image block to be masked, and other image blocks of the medical image.

[0136] Step 360: Use masked image blocks to mask the image blocks to be masked in the medical image to obtain the masked image.

[0137] Step 370: Obtain the query feature and the average feature using the encoding network and the momentum averaging network respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0138] Step 380: Add the currently obtained average features to the dynamic dictionary, which also records the average features of other masked images.

[0139] Step 390: Take the average features and query features belonging to the same masked image as positive sample pairs, and take the average features and query features belonging to different masked images as negative sample pairs.

[0140] Step 3100: Calculate the contrast loss using positive and negative sample pairs.

[0141] The implementation process of steps 310-3100 is the same as that of steps 210-2100, and will not be repeated here.

[0142] Step 3110: Reconstruct low-resolution image patches based on the first local features or the second local features.

[0143] The more accurate the first or second local features, the higher the similarity between the image patch reconstructed based on the first or second local features and the image patch to be masked. That is, during the encoding network processing, even if the image patch to be masked is masked, the encoding network can still combine it with the surrounding image patches to obtain relatively accurate features of the image patch to be masked. In one embodiment, a low-resolution 3D image is reconstructed based on the first or second local features. This 3D image can be considered as a low-resolution reconstructed image of the image patch to be masked. Currently, the reconstructed 3D image is denoted as the low-resolution image patch.

[0144] In one embodiment, the example is described using a first local feature. Optionally, the encoding network also includes a reconstruction module connected after the predictor, which is used to reconstruct low-resolution image patches based on the first local features output by the predictor. Figure 6 This is a schematic diagram of a reconstruction module provided in one embodiment of this application. (Reference) Figure 6 The Reshape layer in this reconstruction module is used to spatially rearrange the features described by the first local features to obtain the corresponding three-dimensional features. The first local features are understood to be one-dimensional feature vectors. To reconstruct a three-dimensional image, these one-dimensional feature vectors need to be transformed into three-dimensional features, represented as feature maps. The Upsample layer is used to upsample the rearranged three-dimensional feature maps. The Conv3D layer is a three-dimensional convolutional layer, whose specific parameters can be set according to actual conditions. Conv3D is used to convolve the input content. The BN layer is a batch normalization layer used for normalization processing. The ReLU layer is an activation parameter layer using ReLU as the activation function. After inputting the first local features into the reconstruction module, low-resolution image patches can be obtained.

[0145] The resolution of low-resolution image patches can be set according to the actual situation and is not currently limited.

[0146] Step 3120: Downsample the image block to be masked to obtain a downsampled image block.

[0147] For example, the image patch to be masked is downsampled to reduce its resolution. The downsampled image patch is referred to as the downsampled image patch. It is understood that the downsampled image patch has the same resolution as the low-resolution image patch. The specific implementation method of downsampling is not currently limited.

[0148] Step 3130: Calculate the reconstruction loss based on the low-resolution image patch and the downsampled image patch.

[0149] The more accurate the first or second local features used in the reconstruction, the more similar the low-resolution image patch and the downsampled image patch will be. In one embodiment, a loss function describing the similarity of the reconstructed image is constructed based on the low-resolution image patch and the downsampled image patch. Currently, this constructed loss function is denoted as the reconstruction loss function. The type of reconstruction loss function can be set according to actual conditions. In one embodiment, the reconstruction loss function is the Smooth-L1 loss function, which reflects the difference between the predicted value (i.e., the low-resolution image patch) and the true value (i.e., the downsampled image patch). The more similar the low-resolution image patch and the downsampled image patch are, the smaller the Smooth-L1 loss function will be. It can be understood that by substituting the low-resolution image patch and the downsampled image patch into the reconstruction loss function, the reconstruction loss (i.e., the specific value calculated by the reconstruction loss function) can be obtained, and the reconstruction loss is calculated using the Smooth-L1 loss function.

[0150] It is understood that steps 380-3100 and steps 3110-3130 can be executed simultaneously or sequentially, and no restriction is currently imposed.

[0151] Step 3140: Update the model parameters of the encoding network based on the contrast loss and reconstruction loss.

[0152] In one embodiment, the contrast loss and reconstruction loss are combined as the total loss of the encoding network, and the model parameters of the encoding network are updated based on this total loss. In this case, the loss function of the encoding network can be expressed as:

[0153] Loss = L reconstruct +α*L contrast

[0154] Where Loss represents the loss function of the encoding network, L reconstruct L represents the reconstruction loss function. contrast Let represent the contrastive loss function, and α represent the balance coefficient. The value of α can be set according to the actual situation. Substituting the reconstruction loss and contrastive loss into the above formula, the total loss of the encoding network can be obtained.

[0155] Step 3150: Update the model parameters of the momentum-averaged network based on the updated model parameters of the encoding network.

[0156] Step 3160: Continue acquiring three-dimensional medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition.

[0157] As mentioned above, during the pre-training process, a low-resolution reconstruction of the image patch to be covered based on local features was designed. This allows the features learned by the encoding network to represent the main structural information of the image patch, ignoring details or noise information, thus improving the indicative nature of the features. Furthermore, compared to low-resolution reconstruction of the covered image, low-resolution reconstruction of the image patch to be covered requires less computation and is unaffected by damage to the covered image. Even if the covered image is damaged, low-resolution reconstruction of the image patch to be covered is still possible. Combining contrastive learning and low-resolution reconstruction as a pre-training step combines the strong robustness of contrastive learning with the high indicative nature of low-resolution reconstruction, giving the feature encoder strong feature extraction and representation capabilities. In downstream tasks, only a small amount of labeled data is needed for fine-tuning to achieve good accuracy.

[0158] The following is an exemplary description of the pre-training method for the medical image coding network provided in this application embodiment. The loss function of the medical image coding network includes a contrast loss function and a reconstruction loss function, and the balance coefficient is set to 0.5. The medical image is a medical image captured by MR. During pre-training, the momentum averaging model of MoCo is used, and the query feature is a first local feature, and the average feature is a second global feature.

[0159] Figure 7 This is a schematic diagram illustrating the momentum-average model processing flow using MoCo, provided as an embodiment of this application. (Reference) Figure 7 The masked image x( Figure 7 The image (denoted as masked image x) is input into the encoding network and the momentum averaging network, respectively, to obtain the query feature q and the average feature q, respectively. average characteristics Add the feature to the dynamic dictionary queue and remove the first recorded average feature from the dynamic dictionary queue. Query feature q and each average feature in the dynamic dictionary queue. Each can form a sample pair Next, the contrastive loss is obtained based on the dynamic dictionary queue and the query feature q, and the model parameters of the encoding network are updated based on the contrastive loss. Then, the model parameters of the momentum-averaged network are updated based on the momentum of the encoding network.

[0160] During a pre-training process, Figure 8 This is a flowchart illustrating the construction of a masked image according to one embodiment of this application. (See also...) Figure 8 Acquire medical images with dimensions D×H×W, and then segment them. There are n image blocks, each image block having a size of p. d ×p h×p w Next, an image patch to be masked is selected according to an image patch selection strategy. This strategy involves calculating the one-dimensional entropy of each image patch, filtering the patches based on this entropy, retaining those with higher entropy, and randomly selecting one patch from the retained patches as the masked image patch. Then, a mask image patch is generated and used to replace the image patch to be masked, resulting in the masked image. The masked image and the image patch to be masked form a sample pair used in the pre-training process to calculate the contrast loss and reconstruction loss.

[0161] Figure 9 A flowchart illustrating a self-supervised pre-training process provided in one embodiment of this application. (Reference) Figure 9 After obtaining the masked image, it is input into the feature encoder f of the medical image coding network to obtain a 3D feature map. Then, ROI pooling is used to obtain the image features of each image patch (excluding the image patch to be masked) in the masked 3D image and the mask features corresponding to the mask image patch. Next, a projector is used to map and project the image features and mask features to obtain isolated features. Then, GAP is used to fuse the isolated features to obtain the first global feature k. Furthermore, a predictor is used to map the isolated feature blocks corresponding to the mask image patch to obtain the first local feature q. Similarly, a momentum averaging network is used to obtain the second local feature. Second global features Next, the second global feature A dynamic dictionary queue is added, and a contrastive loss is calculated based on the first local feature q and the dynamic dictionary queue. A low-resolution reconstructed image is then reconstructed based on the first local feature q. The image patch to be masked is downsampled to obtain a downsampled image patch. Then, a reconstruction loss is obtained based on the low-resolution image patch and the downsampled image patch. Next, the model parameters of the encoding network are updated based on the reconstruction loss and the contrastive loss. Finally, the model parameters of the momentum-averaged network are updated based on the momentum of the encoding network, thus completing one pre-training cycle. A new pre-training cycle then begins until the pre-training stopping condition is met.

[0162] Figure 10 This is a schematic diagram of training loss provided for one embodiment of this application. (Reference) Figure 10 This is a diagram illustrating the loss values ​​obtained based on the training set during pre-training. The training set is the dataset used to train the encoding network during pre-training, which contains multiple 3D medical images in modalities such as t1 and t2. Figure 10In the diagram, line 1 represents the contrast loss, line 2 represents the reconstruction loss, and line 3 represents the total loss (based on the contrast loss and reconstruction loss) when the balance coefficient is 0.5. The horizontal axis represents the number of iterations (i.e., the number of pre-training iterations), and the vertical axis represents the loss value. Figure 11 This is a schematic diagram of verification loss provided for one embodiment of this application. (Reference) Figure 11 This diagram illustrates the loss values ​​obtained based on the validation set during pre-training. The validation set is used to fine-tune the model parameters in the encoding network during pre-training. The validation set and training set can be constructed based on the same dataset. For example, given a dataset, 80% of the data can be selected as the training set, and the remaining 20% ​​as the validation set. Figure 11 In the diagram, line 1 represents the contrast loss, line 2 represents the reconstruction loss, and line 3 represents the total loss (based on the contrast loss and reconstruction loss) when the balance coefficient is 0.5. Figure 10 and Figure 11 It can be seen that the loss function converges faster as the number of pre-training iterations increases.

[0163] In one embodiment, after pre-training, the feature encoder is used as the backbone network and applied to the nasopharyngeal carcinoma classification and segmentation models, respectively. Referring to Table 2, it includes performance data for the nasopharyngeal carcinoma classification and segmentation models using the pre-trained feature encoder and performance data for the nasopharyngeal carcinoma classification and segmentation models using a randomly initialized (i.e., unpre-trained) feature encoder.

[0164]

[0165] Table 2

[0166] As shown in Table 2, compared with the feature encoder using random initialization, the performance of the nasopharyngeal carcinoma classification model on the test set is improved by 3.8 percentage points and the performance of the nasopharyngeal carcinoma segmentation model on the test set is improved by 2.4 percentage points after using the pre-trained feature encoder.

[0167] Figure 12 This is a schematic diagram of the structure of a pre-training device for a medical image coding network provided in one embodiment of this application, with reference to... Figure 12 The pre-training device for the medical image coding network includes an acquisition module 401, a selection module 402, a masking module 403, a feature determination module 404, a first update module 405, a second update module 406, and a repetition training module 407.

[0168] The system comprises the following modules: an acquisition module 401, used to acquire three-dimensional medical images, which are divided into multiple image blocks of the same size; a selection module 402, used to select an image block to be masked from the multiple image blocks; a masking module 403, used to mask the image block to be masked in the medical image using a mask image block to obtain a masked image; a feature determination module 404, used to obtain query features and average features using an encoding network and a momentum averaging network, respectively. When the query feature is the first local feature of the mask image block, the average feature is the second global feature of the masked image; when the query feature is the first global feature of the masked image, the average feature is the second local feature of the mask image block. The encoding network and the momentum averaging network have the same structure; a first update module 405, used to calculate the contrast loss based on the query feature and the average feature, and update the model parameters of the encoding network according to the contrast loss; a second update module 406, used to update the model parameters of the momentum averaging network according to the updated model parameters of the encoding network; and a retraining module 407, used to continue acquiring three-dimensional medical images and return to the operation of selecting the image block to be masked until the encoding network meets the pre-training stopping condition.

[0169] In one embodiment of this application, the first update module 405 includes: an addition unit, used to add the currently obtained average features to a dynamic dictionary, the dynamic dictionary also recording the average features of other masked images; a sample pair construction unit, used to use the average features and query features belonging to the same masked image as positive sample pairs, and the average features and query features belonging to different masked images as negative sample pairs; a contrastive learning unit, used to calculate the contrastive loss using the positive and negative sample pairs; and a parameter update unit, used to update the model parameters of the encoding network according to the contrastive loss.

[0170] In one embodiment of this application, the method further includes: a reconstruction module, configured to reconstruct a low-resolution image patch based on a first local feature or a second local feature; a downsampling module, configured to downsample the image patch to be masked to obtain a downsampled image patch; and a reconstruction loss calculation module, configured to calculate a reconstruction loss based on the low-resolution image patch and the downsampled image patch. Correspondingly, the first update module 405 updates the model parameters of the encoding network based on the contrast loss specifically by updating the model parameters of the encoding network based on the contrast loss and the reconstruction loss.

[0171] In one embodiment of this application, the reconstruction loss is calculated using the Smooth-L1 loss function.

[0172] In one embodiment of this application, the encoding network consists of a feature encoder, a first pooling layer, a projector, a second pooling layer, and a predictor. The feature determination module 404 includes: an extraction unit, used to extract features from the masked image by the feature encoder to obtain a three-dimensional feature map; a first pooling unit, used to obtain the mask features of the masked image blocks and the image features of other image blocks in the masked image from the three-dimensional feature map by the first pooling layer, with each image block corresponding to one image feature; a projection unit, used to map and project the mask features and each image feature by the projector to obtain each isolation feature; and a second pooling unit... The first global feature of the masked image is obtained by fusing all isolation features using the second pooling layer; the prediction unit is used to predict the isolation features corresponding to the masked image blocks by the predictor to obtain the first local features corresponding to the masked image blocks; the average feature determination unit is used to obtain the average feature using the momentum averaging network. When the query feature obtained by the encoding network is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure.

[0173] In one embodiment of this application, the feature encoder employs an asymmetric 3D convolutional network, the projector employs a first multilayer perceptron network, and the predictor employs a second multilayer perceptron network.

[0174] In one embodiment of this application, the downsampling ratio of the feature encoder is less than or equal to the size of the image block.

[0175] In one embodiment of this application, the size of the image patch is an integer multiple of the downsampling ratio of the feature encoder, and the first pooling layer uses ROI pooling.

[0176] In one embodiment of this application, the selection module 402 includes: an entropy calculation unit for calculating the one-dimensional entropy of each image block; a filtering unit for filtering the image blocks according to the one-dimensional entropy; and an image block selection unit for selecting one image block from the retained image blocks as the image block to be masked.

[0177] In one embodiment of this application, the filtering unit includes: an entropy selection subunit, used to select one-dimensional entropies greater than a preset threshold from all one-dimensional entropies, or to select a preset number of one-dimensional entropies starting from the largest one-dimensional entropy based on the size of each one-dimensional entropy; and a retention subunit, used to retain the image block corresponding to the selected one-dimensional entropy.

[0178] In one embodiment of this application, it further includes: a mask selection module, used to select the currently used mask image block from fixed value image blocks, the image block to be masked, and other image blocks of the medical image before masking the image block to be masked in the medical image using a mask image block.

[0179] In one embodiment of this application, a fixed-value image block corresponds to a first selected probability, other image blocks correspond to a second selected probability, and the image block to be masked corresponds to a third selected probability. The first selected probability is greater than the second selected probability, and the second selected probability is greater than the third selected probability.

[0180] The pre-training device for the medical image coding network provided above can be used to execute the pre-training method for the medical image coding network provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0181] It is worth noting that in the embodiments of the pre-training device for the medical image coding network described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0182] Figure 13 This is a schematic diagram of the structure of a pre-training device for a medical image coding network provided in one embodiment of this application. Figure 13 As shown, the pre-training device for the medical image coding network includes a processor 50, a memory 51, an input device 52, and an output device 53; the number of processors 50 in the pre-training device for the medical image coding network can be one or more. Figure 13 Taking a processor 50 as an example, in the pre-training device for the medical image coding network, the processor 50, memory 51, input device 52, and output device 53 can be connected via a bus or other means. Figure 13 Taking the example of a connection between China and Israel via a bus.

[0183] The memory 51, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the pre-training method of the medical image coding network in one embodiment of this application (e.g., the acquisition module 401, selection module 402, masking module 403, feature determination module 404, first update module 405, second update module 406, and repetitive training module 407 in the pre-training device for the medical image coding network). The processor 50 executes various functional applications and data processing of the pre-training device for the medical image coding network by running the software programs, instructions, and modules stored in the memory 51, thereby implementing the aforementioned pre-training method for the medical image coding network.

[0184] The memory 51 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the pre-training device for the medical image coding network. Furthermore, the memory 51 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 51 may further include memory remotely located relative to the processor 50, which can be connected to the pre-training device of the medical image coding network via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0185] Input device 52 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the pre-trained equipment for the medical image coding network. It may also include devices required for acquiring medical images. Output device 53 may include display devices such as a display screen.

[0186] The aforementioned pre-training device for medical image coding networks includes a pre-training apparatus for medical image coding networks, which can be used to perform pre-training methods for any medical image coding network and has corresponding functions and beneficial effects.

[0187] Furthermore, embodiments of this application also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the pre-training method of the medical image coding network provided in any embodiment of this application, and have corresponding functions and beneficial effects.

[0188] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0189] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0191] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0193] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A pre-training method for a medical image coding network, characterized in that, include: Acquire three-dimensional medical images, which are divided into multiple image blocks of the same size; Select the image block to be masked from among the plurality of image blocks; The image block to be masked in the medical image is masked using a masked image block to obtain the masked image; The query feature and the average feature are obtained by using an encoding network and a momentum averaging network, respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure. The contrast loss is calculated based on the query features and the average features, and the model parameters of the encoding network are updated according to the contrast loss. The model parameters of the momentum averaging network are updated based on the updated model parameters of the encoding network. Continue acquiring 3D medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition; The calculation of the contrast loss based on the query features and the average features includes: The obtained average features are added to a dynamic dictionary, which also records the average features of other masked images. The average features and query features belonging to the same masked image are used as positive sample pairs, and the average features and query features belonging to different masked images are used as negative sample pairs. The contrast loss is calculated using the positive sample pairs and the negative sample pairs.

2. The pre-training method according to claim 1, characterized in that, Also includes: Based on the first local feature or the second local feature, a low-resolution image patch is reconstructed; The image block to be masked is downsampled to obtain a downsampled image block; Calculate the reconstruction loss based on the low-resolution image patch and the downsampled image patch; The step of updating the model parameters of the encoding network based on the contrastive loss includes: The model parameters of the encoding network are updated based on the contrast loss and the reconstruction loss.

3. The pre-training method according to claim 2, characterized in that, The reconstruction loss is calculated using the Smooth-L1 loss function.

4. The pre-training method according to claim 1, characterized in that, The encoding network consists of a feature encoder, a first pooling layer, a projector, a second pooling layer, and a predictor. The query features obtained using the coding network include: The feature encoder extracts features from the masked image to obtain a three-dimensional feature map; The first pooling layer obtains the mask features of the masked image block and the image features of other image blocks in the masked image from the three-dimensional feature map, with each image block corresponding to one image feature; The projector maps and projects the mask features and each of the image features to obtain the isolation features. The first global feature of the masked image is obtained by fusing all the isolation features using the second pooling layer; The predictor predicts the isolation features corresponding to the masked image block to obtain the first local features corresponding to the masked image block.

5. The pre-training method according to claim 4, characterized in that, The feature encoder employs an asymmetric 3D convolutional network, the projector employs a first multilayer perceptron network, and the predictor employs a second multilayer perceptron network.

6. The pre-training method according to claim 4, characterized in that, The downsampling ratio of the feature encoder is less than or equal to the size of the image block.

7. The pre-training method according to claim 6, characterized in that, The size of the image patch is an integer multiple of the downsampling ratio of the feature encoder, and the first pooling layer uses ROI pooling.

8. The pre-training method according to claim 1, characterized in that, Selecting the image block to be masked from among the plurality of image blocks includes: Calculate the one-dimensional entropy of each of the image blocks; The image blocks are filtered based on the one-dimensional entropy; From the retained image blocks, select one image block as the image block to be masked.

9. The pre-training method according to claim 8, characterized in that, The filtering of the image patch based on the one-dimensional entropy includes: Select one-dimensional entropy greater than a preset threshold from all the one-dimensional entropy, or select a preset number of one-dimensional entropy starting from the largest one-dimensional entropy based on the size of each one-dimensional entropy. Retain the image patch corresponding to the selected one-dimensional entropy.

10. The pre-training method according to claim 1, characterized in that, Before masking the image block to be masked in the medical image using masked image blocks, the method further includes: Select the currently used mask image block from the fixed value image block, the image block to be masked, and other image blocks of the medical image.

11. The pre-training method according to claim 10, characterized in that, The fixed-value image block corresponds to a first selected probability, the other image blocks correspond to a second selected probability, and the image block to be masked corresponds to a third selected probability. The first selected probability is greater than the second selected probability, and the second selected probability is greater than the third selected probability.

12. A pre-training device for a medical image coding network, characterized in that, include: An acquisition module is used to acquire three-dimensional medical images, which are divided into multiple image blocks of the same size; The selection module is used to select the image block to be masked from among the plurality of image blocks; The masking module is used to mask the image block to be masked in the medical image using a mask image block, so as to obtain the masked image; The feature determination module is used to obtain query features and average features using an encoding network and a momentum averaging network, respectively. When the query feature is the first local feature of the masked image block, the average feature is the second global feature of the masked image. When the query feature is the first global feature of the masked image, the average feature is the second local feature of the masked image block. The encoding network and the momentum averaging network have the same structure. The first update module is used to calculate the contrast loss based on the query features and the average features, and update the model parameters of the encoding network according to the contrast loss; The second update module is used to update the model parameters of the momentum averaging network based on the updated model parameters of the encoding network. The retraining module is used to continue acquiring three-dimensional medical images and return to perform the operation of selecting image patches to be masked until the encoding network meets the pre-training stopping condition; The first update module includes: an addition unit, used to add the currently obtained average features to a dynamic dictionary, which also records the average features of other masked images; a sample pair construction unit, used to take the average features and query features belonging to the same masked image as positive sample pairs and the average features and query features belonging to different masked images as negative sample pairs; and a contrastive learning unit, used to calculate the contrastive loss using the positive sample pairs and the negative sample pairs.

13. A pre-training device for a medical image coding network, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the pre-training method for the medical image coding network as described in any one of claims 1-11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the pre-training method for the medical image coding network as described in any one of claims 1-11.

Images