ViT model-based chromosome image processing general model and application method thereof

Abstract

The invention provides a ViT model-based chromosome image processing general model and an application method thereof, the ViT model-based chromosome image processing general model and the application method thereof, the used general model comprises an embedded layer, a multi-layer Transform coding layer and a multi-layer sensing machine head, the multi-layer Transform coding layer is an encoder of a pre-trained MAE model; wherein the pre-trained encoder can automatically extract useful features from a large amount of chromosome data which is randomly shielded, so that the dependence on large-scale labeled data is reduced, and the accuracy of an image processing model and the training efficiency of the model are effectively improved; meanwhile, when the encoder network is trained, input data is partially shielded and then recovered by a decoder, so that important information in the chromosome sequence can be efficiently captured, the robustness and generalization ability of the encoder network can be enhanced, and when the long and complex chromosome sequence is processed, the method has the advantages that the robustness and generalization ability of the encoder network are improved, and the robustness and generalization ability of the encoder network are improved. And the problem of data over-fitting can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of medical image processing, in particular to a universal model for chromosome image processing based on a ViT model and an application method thereof. Background Art

[0002] Chromosomes are the carriers of genes. Chromosome abnormalities will inevitably cause imbalanced gene expression, thereby causing a series of clinical symptoms. Chromosome structural abnormalities: deletion, duplication, inversion, insertion, translocation. Doctors can determine whether the chromosome is abnormal by analyzing the structure of the chromosome in the image. At present, the use of deep learning algorithms is widely used in the fields of recognition and detection. It only requires collecting enough samples and selecting a suitable deep learning network. After training, a better recognition model or detection model can be obtained.

[0003] Existing deep learning methods can automatically extract image features by learning from a large amount of labeled data. However, in order to achieve high-precision chromosome recognition and segmentation, they rely on a large amount of chromosome data, process a large amount of data and have low accuracy. Summary of the invention

[0004] To solve the above problems, the present invention proposes a universal chromosome image processing model based on the ViT model and an application method thereof.

[0005] The main contents of the present invention include:

[0006] A general model for chromosome image processing based on the ViT model includes an embedding layer, a multi-layer Transformer encoding layer and a multi-layer perceptron head; wherein the multi-layer Transformer encoding layer is an encoder of a pre-trained MAE model; and the working process of the encoder is as follows:

[0007] Perform image segmentation on the chromosome image to be processed to obtain the corresponding background area and target area;

[0008] Perform differential coding processing on the background area and the target area to obtain a corresponding merged mask image; the differential coding processing refers to performing random mask processing with different masking rates on the background image and the target image respectively;

[0009] The corresponding merged mask image is input into the pre-trained encoder to obtain the corresponding feature representation.

[0010] Preferably, the pre-training method of the MAE model comprises the following steps:

[0011] Dataset preparation: Prepare a dataset containing chromosome images; N is the total number of chromosome-like images;

[0012] Construct the MAE model: construct the encoder network E and the decoder network D, and set the coding weights of the chromosome image;

[0013] Coding processing: Differentiate the coding of the background and target parts of the chromosome image, and input the obtained image into the pre-trained encoder to obtain the corresponding feature representation;

[0014] Decoding process: input the feature representation into the decoder to obtain a reconstructed image;

[0015] Loss function setting: Set the loss function of the MAE model to in: x i is the original image of the chromosome image in the dataset; is the reconstructed image; y i For x i The corresponding category label;

[0016] Model update: Use optimization algorithms to minimize the loss function and update model parameters;

[0017] Save the pre-trained encoder to replace the multi-layer transformer encoding layer in the ViT model.

[0018] Preferably, it further includes a model training step, wherein the model training step includes iteratively training the MAE model until the MAE model converges or reaches a preset number of training times.

[0019] Preferably, setting the coding weight of the chromosome image includes selecting a band region in the chromosome image, and setting the coding weight of the band region to be greater than the coding weights of other regions.

[0020] Preferably, differential coding is performed on the background area and the target area to obtain a corresponding merged mask image, including:

[0021] Using a first mask rate to randomly mask a background area of ​​the chromosome image to be processed to obtain a first mask image;

[0022] Using a second mask rate to randomly mask a target region of the chromosome image to be processed to obtain a second mask image;

[0023] After merging the background area in the first mask image with the target area in the second mask image, a merged mask image is obtained;

[0024] The first masking rate is smaller than the second masking rate.

[0025] The present invention also proposes an application method of a universal model for chromosome image processing based on the ViT model, including:

[0026] Build image processing models according to task requirements;

[0027] The above-mentioned general model for chromosome image processing based on the ViT model is used as the feature extractor of the image processing model.

[0028] Preferably, the method further comprises the step of training the image processing model, the step comprising fine-tuning the image processing model using a labeled task dataset;

[0029] The image processing model after fine-tuning training is tested using a test set of the corresponding task. If the corresponding performance requirements are met, the image processing model is deployed.

[0030] The beneficial effects of the present invention are as follows: the present invention proposes a general model for chromosome image processing based on the ViT model and an application method thereof, wherein the general model used includes an embedding layer, a multi-layer Transformer encoding layer and a multi-layer perceptron head, wherein the multi-layer Transformer encoding layer is an encoder of a pre-trained MAE model; the present invention utilizes a pre-trained encoder network to perform feature extraction tasks in an image processing model, and can automatically extract useful features from a large amount of randomly masked chromosome data, thereby reducing dependence on large-scale annotated data and effectively improving the accuracy of the image processing model and the training efficiency of the model; at the same time, when the encoder network is trained, by partially masking the input data and then restoring it using a decoder, not only can the important information in the chromosome sequence be efficiently captured, but also the robustness and generalization ability of the encoder network can be enhanced, and when faced with long and complex chromosome sequences, the problem of data overfitting can also be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the training process of the encoder network of the present invention. DETAILED DESCRIPTION

[0032] The technical solution protected by the present invention will be described in detail below with reference to the accompanying drawings.

[0033] The present invention proposes a general model for chromosome image processing based on the ViT model. The existing ViT model is a visual recognition model based on the Transformer architecture, which is used for computer vision tasks such as image classification. The ViT model mainly consists of three parts: Embedding layer, Transformer Encoder (multi-layer Transformer encoding layer) and MLPHead (multi-layer perceptron head). Among them, the Embedding layer is responsible for converting image data into a form suitable for Transformer processing; and the Transformer Encoder extracts features through a multi-head attention mechanism;

[0034] The MLP Head is used for the final classification task.

[0035] In order to be applicable to the processing of chromosome images, the present invention proposes a general model based on the ViT model, which has good robustness and generalization ability in feature extraction of chromosome images, and does not rely on large-scale labeled data, has higher feature recognition accuracy, and higher model training efficiency. Specifically, the present invention improves the existing ViT model and replaces the Transformer Encoder in the existing ViT model with the encoder of the pre-trained MAE model; the working process of the encoder of the pre-trained MAE model is as follows:

[0036] First, the chromosome image x to be processed i Perform image segmentation to obtain the corresponding background area and target area; specifically, the existing image segmentation algorithm, such as the U-Net model, can be used to obtain the Mask mask area of ​​the chromosome image in the image; or the target detection algorithm, such as the Faster R-CNN model, can be used to obtain the bounding box target detection area of ​​the chromosome in the image, so as to convert the chromosome image to be processed into i Divided into background area B i and target area F i ;

[0037] Then, the background area and the target area are differentially coded to obtain a corresponding merged mask image; wherein the differential coding refers to performing random masking with different masking rates on the background area and the target area respectively; specifically, a random mask is generated using a first masking rate to process the chromosome image x i The background area is randomly masked to obtain the first mask image; then the second mask rate is used to generate a random mask for the chromosome image x to be processed. iThe target area is randomly masked to obtain a second mask image; then the background area in the first mask image and the target area in the second mask image are merged to obtain a complete image, which is recorded as a merged mask image; the background area and the target area of ​​the merged mask image have different mask rates, that is, differentiated coding processing is achieved for the background area and the target area.

[0038] Furthermore, the first mask rate is less than the second mask rate. In one embodiment, the first mask rate is selected to be 50%, that is, 50% of the background area of ​​the chromosome image to be processed is randomly selected for coding; and the second mask rate is selected to be 75%, that is, 75% of the target area of ​​the chromosome image to be processed is randomly selected for coding; then the coded first mask image and the second mask image are merged, the background area of ​​the first mask image has a mask rate of 50%, and the target area of ​​the second mask image has a mask rate of 75%, and the two parts are merged into a complete image to obtain a merged mask image.

[0039] The above two parts are randomly masked with different masking rates respectively, and then the encoder performs feature extraction to obtain the corresponding feature representation, which is the important information in the chromosome sequence. Through this method, the influence of the target area on the potential feature learning is enhanced, while the influence of the background area is relatively weakened. In this process, the background area plays a greater role in maintaining the global coordinate and position information. In addition, the use of random masking can reduce the dependence on large-scale labeled data.

[0040] The encoder in the pre-trained MAE model is different from the encoder of the existing MAE model. The encoder of the MAE model of the present invention performs differentiated masking on different parts of the chromosome image to be processed, and then restores the original data through the encoder, so that it can efficiently capture the important information in the chromosome sequence, so that the encoder can automatically extract useful features from a large amount of randomly masked chromosome data, reducing the dependence on large-scale labeled data; at the same time, the masking strategy not only reduces the amount of data to be processed, but also enables the MAE model to learn the global structure instead of memorizing local features, which can reduce the problem of data overfitting; and through this pre-training method, the robustness and generalization ability of the model can be enhanced, so that it can also perform better when processing new data.

[0041] In this embodiment, the step of training the MAE model includes:

[0042] Dataset preparation: Prepare a dataset containing chromosome images; N is the total number of chromosome-like images; x i is the original image of the chromosome image in the data set; i For xi The corresponding category label;

[0043] Constructing the MAE model: Constructing the encoder network E and the decoder network D, setting the coding weight of the chromosome image; in this embodiment, setting the stripe region in the target region to have a larger coding weight, that is, setting the coding weight of the stripe region to be greater than the coding weight of other regions. This strategy can force the model to pay more attention to these important regions, aiming to strengthen the feature learning of the position and stripe information of the chromosome in different forms, thereby extracting more meaningful features in the learning process.

[0044] Coding processing: The background part and the target part of the chromosome image are differentially coded, and the obtained background mask image and target mask image are merged and input into the pre-trained encoder to obtain the corresponding feature representation; this differentiated coding processing method can more effectively utilize limited data resources and learn more representative features in the pre-training stage.

[0045] Decoding process: input the feature representation into the decoder to obtain a reconstructed image;

[0046] Loss function setting: Set the loss function of the MAE model to in: is the reconstructed image;

[0047] Model update: Use optimization algorithms to minimize the loss function and update model parameters;

[0048] Then, the MAE line is iteratively trained through the model training step until the MAE model converges or reaches a preset number of training times, and then the trained encoder is saved and used as the multi-layer Transformer encoding layer of the ViT model, thereby obtaining a general model suitable for chromosome image processing.

[0049] At the same time, the present invention also proposes an application method of a universal model for chromosome image processing based on the ViT model, that is, using the universal model as a feature extractor of the network to which it is applied. The decoder is only used in the pre-training stage of the encoder. After the pre-training is completed, the encoder is used for the recognition task, and the decoder is discarded. This design reduces the computational burden and makes the training more efficient.

[0050] Specifically, the application method comprises the following steps:

[0051] An image processing model is constructed, wherein the image processing model includes a task network, that is, firstly, according to the image processing task, such as chromosome classification, chromosome target detection and the like, a corresponding task network model is selected, wherein the task network can be an existing deep learning network, such as a resent network model, a YOLO network model and the like; then, the feature extractor in the task network is replaced by the general model, thereby completing the construction of the image processing model; subsequently, the image model is fine-tuned using a labeled task data set; and the performance of the fine-tuned image processing model is evaluated on a test set. If the corresponding requirements are met, the corresponding image processing model can be deployed in an actual application environment.

[0052] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A general model for chromosome image processing based on the ViT model, characterized in that: It includes an embedding layer, a multi-layer Transformer encoding layer and a multi-layer perceptron head; wherein the multi-layer Transformer encoding layer is an encoder of a pre-trained MAE model; the working process of the encoder is as follows: Perform image segmentation on the chromosome image to be processed to obtain the corresponding background area and target area; Perform differential coding processing on the background area and the target area to obtain a corresponding merged mask image; the differential coding processing refers to performing random mask processing with different masking rates on the background image and the target image respectively; The corresponding merged mask image is input into the pre-trained encoder to obtain the corresponding feature representation.

2. The universal model for chromosome image processing based on the ViT model according to claim 1, characterized in that: The pre-training method of the MAE model includes the following steps: Dataset preparation: Prepare a dataset containing chromosome images; N is the total number of chromosome-like images; Construct the MAE model: construct the encoder network E and the decoder network D, and set the coding weights of the chromosome image; Coding processing: Differentiate the coding of the background and target parts of the chromosome image, and input the obtained image into the pre-trained encoder to obtain the corresponding feature representation; Decoding process: input the feature representation into the decoder to obtain a reconstructed image; Loss function setting: Set the loss function of the MAE model to in: x i is the original image of the chromosome image in the dataset; is the reconstructed image; y i For x i The corresponding category label; Model update: Use optimization algorithms to minimize the loss function and update model parameters; Save the pre-trained encoder to replace the multi-layer transformer encoding layer in the ViT model.

3. The universal model for chromosome image processing based on the ViT model according to claim 2, characterized in that: The method further includes a model training step, wherein the model training step includes iteratively training the MAE model until the MAE model converges or reaches a preset number of training times.

4. The universal model for chromosome image processing based on the ViT model according to claim 2, characterized in that: The coding weight of the chromosome image is set, including selecting a band region in the chromosome image, and setting the coding weight of the band region to be greater than the coding weights of other regions.

5. The universal model for chromosome image processing based on the ViT model according to claim 1, characterized in that: Perform differential coding on the background area and the target area to obtain the corresponding merged mask image, including: Using a first mask rate to randomly mask a background area of ​​the chromosome image to be processed to obtain a first mask image; Using a second mask rate to randomly mask a target region of the chromosome image to be processed to obtain a second mask image; After merging the background area in the first mask image with the target area in the second mask image, a merged mask image is obtained; The first masking rate is smaller than the second masking rate.

6. An application method of a universal model for chromosome image processing based on the ViT model, characterized in that: Envelope: Build image processing models according to task requirements; The chromosome image processing general model based on the ViT model as described in any one of claims 1 to 5 is used as the feature extractor of the image processing model.

7. The application method of the universal model for chromosome image processing based on the ViT model according to claim 6 is characterized in that: The step of training the image processing model is also included, and the step includes fine-tuning the image processing model using a labeled task dataset; The image processing model after fine-tuning training is tested using a test set of the corresponding task. If the corresponding performance requirements are met, the image processing model is deployed.

Images