Pathological section image analysis method and device based on image coding large model

Abstract

The invention provides a pathological section image analysis method and device based on an image coding large model, and belongs to the technical field of medical images. The method comprises the following steps: segmenting an original pathological section image to obtain a pathological section sub-image set; sequentially inputting the pathological section sub-images in the set into a target region segmentation large model to obtain a corresponding local target region distribution diagram, further obtaining an overall target region distribution diagram, and generating a positive local target region distribution diagram set; based on the positive local target area distribution map set, generating an abnormal focus distribution map by using an abnormal focus detection network; and finally realizing statistical analysis of the original pathological section image based on the target area distribution diagram and the abnormal focus distribution diagram. According to the method, various pathological section images can be analyzed, abnormal focus areas in the images are automatically identified and classified, key clinical indexes such as the lesion rate are obtained through automatic statistical analysis, and the diagnosis accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the field of medical imaging technology, and in particular relates to a pathological slice image analysis method and device based on an image coding large model. Background Art

[0002] With the rapid development of medical imaging technology, pathological sections have become an important tool for diagnosing various diseases. However, traditional pathological diagnosis methods mainly rely on pathologists to manually observe and analyze sections to identify and classify lesions. This method is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in low diagnostic efficiency and insufficient accuracy. Especially when faced with complex and diverse pathological sections, the limitations of traditional methods become more and more obvious.

[0003] At present, there are many studies on pathological slice image analysis methods at home and abroad. For example, the pathology team of Harvard Medical School trained a self-supervised learning model called UNI designed specifically for pathological slice image analysis based on the MetaAI team's DinoV2 visual model. The UNI model is trained in a self-supervised manner. Based on the visual transformer (ViT-L / 16 architecture) of DinoV2, it does not rely on labeled data and can automatically extract efficient features from pathological slices. The model can perform well in different types of pathological feature extraction tasks, such as tissue classification and tumor segmentation, and has strong generalization capabilities. Although pathological slice image analysis methods represented by UNI have certain capabilities in feature extraction, they have not yet achieved direct quantitative representation of pathological features, which makes the practical application value of such methods low, and it is difficult for doctors to efficiently obtain clinically valuable information from complex and trivial image features. Summary of the invention

[0004] The purpose of the present invention is to overcome the problem of lack of quantitative characterization of the degree of lesions in the existing pathological slice image analysis method. In order to overcome the shortcomings of the existing technology, a pathological slice image analysis method and device based on a large image coding model are proposed. The present invention uses deep learning technology to analyze various pathological slice images, automatically identify and classify abnormal lesion areas in the image, and automatically perform statistical analysis to obtain key clinical indicators such as lesion rate, thereby solving the problems of low efficiency and poor real-time performance caused by the existing manual observation of slices for diagnosis, improving the accuracy and efficiency of diagnosis, and promoting the progress and development of digital pathology.

[0005] The first embodiment of the present invention provides a pathological slice image analysis method based on an image coding large model, comprising:

[0006] The original pathological slice image is segmented according to a preset size to obtain a set of pathological slice sub-images;

[0007] The pathological slice sub-images in the pathological slice sub-image set are sequentially input into a preset target region segmentation model to obtain a corresponding local target region distribution map; wherein the target region segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model;

[0008] By processing the local target area distribution map, a target area distribution map corresponding to the original pathological slice image is obtained and a positive local target area distribution map set is generated;

[0009] Based on the positive local target area distribution map set, using a preset abnormal lesion detection network, generating an abnormal lesion distribution map corresponding to the original pathological slice image;

[0010] Based on the target area distribution map and the abnormal lesion distribution map, statistical analysis of the original pathological slice image is performed.

[0011] In a specific implementation of the present invention, obtaining a set of pathological slice sub-images includes:

[0012] 1) Obtaining a single pathological section image containing a complete lesion as an original pathological section image;

[0013] 2) Segmenting the original pathological slice image obtained in step 1) to obtain a set of pathological slice sub-images;

[0014] The size of the pathological slice sub-image is H×W, where H represents the length of the pathological slice sub-image and W represents the width of the pathological slice sub-image.

[0015] The split expression is as follows:

[0016]

[0017] Among them, height and width represent the length and width of the original pathological slice sub-image respectively; n is the number of sub-images to be segmented in the length direction, and m is the number of sub-images to be segmented in the width direction; for the part that reaches the image boundary and cannot be completely segmented, it is padded to H×W pixel size by mirror filling for segmentation, and the final number of pathological slice sub-images is n×m.

[0018] In a specific implementation of the present invention, the encoder of the target area segmentation model uses the DinoV2 image information coding model, which processes the input pathological slice sub-image to generate a multi-scale feature map; the decoder of the target area segmentation model uses a main decoder UPerNet and an auxiliary decoder FCNNet in combination, UPerNet and FCNNet work in parallel on the multi-scale feature map generated by the encoder, and then the results of UPerNet and FCNNet processing are weightedly fused at the output end of the decoder, and finally a local target area distribution map of H×W pixel size corresponding to the input pathological slice sub-image is output.

[0019] In a specific implementation of the present invention, the step of obtaining a target region distribution map corresponding to the original pathological slice image and generating a set of positive local target region distribution maps includes:

[0020] Based on the local target area distribution map of each pathological slice sub-image in the pathological slice sub-image set output by the target area segmentation large model, the sub-images corresponding to the local target area distribution map are spliced ​​according to the positions of the sub-images in the original pathological slice image to obtain the target area distribution map corresponding to the original pathological slice image;

[0021] All local target area distribution maps are traversed, and local target area distribution maps with target area distribution are positively marked, and then the positively marked local target area distribution maps and their corresponding pathological slice sub-images are numbered in sequence according to the order when they are input into the target area segmentation large model, and finally a set of positive local target area distribution maps is generated.

[0022] In a specific implementation of the present invention, before sequentially inputting the pathological slice sub-images in the pathological slice sub-image set into a preset target region segmentation large model, the method further includes:

[0023] Training the target region segmentation model;

[0024] The training of the target area segmentation model comprises:

[0025] 1)) Construct a training set of a large model for target area segmentation;

[0026] Wherein, the pathological type of the image in the training set of the target region segmentation large model is consistent with the pathological type of the original pathological slice image;

[0027] Annotating the target region of the image of the training set of the target region segmentation model as a training label of the image;

[0028] Then, each image in the training set is divided into a set of sub-images of size H×W, which are used as input for training the large model for target region segmentation;

[0029] 2) Construct a large model for target area segmentation;

[0030] 3) Using the training set obtained in step 1), the target region segmentation large model constructed in step 2) is trained to obtain a trained target region segmentation large model.

[0031] In a specific implementation of the present invention, the abnormal lesion detection network adopts Attention-UNet, and the Attention-UNet is composed of an encoder and a decoder, wherein the encoder of Attention-UNet is used to gradually reduce the spatial dimension of the image while increasing the number of feature channels to capture more abstract features; the decoder of Attention-UNet is used to gradually restore the spatial dimension of the image and reduce the number of feature channels to achieve precise positioning; in the Attention-UNet, each jump connection stage of the encoder and the decoder is provided with an attention module, which controls the features from the encoder and the features of the corresponding positions of the decoder by weighting.

[0032] In a specific implementation of the present invention, the method of generating an abnormal lesion distribution map corresponding to the original pathological slice image based on the positive local target area distribution map set and using a preset abnormal lesion detection network includes:

[0033] 1) inputting the pathological slice sub-image corresponding to each distribution map in the positive local target area distribution map set into a preset abnormal lesion detection network in sequence, and the network outputs the corresponding local abnormal lesion distribution map;

[0034] 2) The local abnormal lesion distribution map obtained in step 1) is spliced ​​according to the position of the corresponding pathological slice sub-image in the original pathological slice image, wherein the vacant parts appearing during splicing are filled with a full distribution map, and a complete abnormal lesion distribution map is obtained after the splicing is completed.

[0035] In a specific implementation of the present invention, before using a preset abnormal lesion detection network to generate an abnormal lesion distribution map corresponding to the original pathological slice image, the method further includes:

[0036] Training the abnormal lesion detection network;

[0037] The training of the abnormal lesion detection network comprises:

[0038] 1) Constructing a training set for abnormal lesion detection network;

[0039] Wherein, the pathological type of the image in the training set of the abnormal lesion detection network is consistent with the pathological type of the original pathological section image;

[0040] Annotating the image of the training set of the abnormal lesion detection network with the lesion area as the training label of the image;

[0041] Then, each image in the training set is divided into a set of sub-images of size H×W, which are used as input for abnormal lesion detection network training;

[0042] 2) Construct an abnormal lesion detection network;

[0043] 3) Using the training set of the abnormal lesion detection network model obtained in step 1), the abnormal lesion detection network constructed in step 2) is trained to obtain a trained abnormal lesion detection network.

[0044] In a specific implementation of the present invention, the statistical analysis of the original pathological slice image includes:

[0045] According to the abnormal lesion distribution map, the number of abnormal lesions and the total area of ​​lesions were obtained by using the connected domain counting algorithm and area statistics;

[0046] Among them, the number of closed connected domains surrounded by lesion labels in the abnormal lesion distribution map is the number of abnormal lesions, which is recorded as n1;

[0047] The sum of the number of pixels in the closed connected domain surrounded by the lesion labels in the statistical abnormal lesion distribution map is the total lesion area, denoted as S1, and the expression is as follows:

[0048]

[0049] Among them, S i is the number of pixels occupied by the i-th lesion area;

[0050] The number of target areas and the proportion of abnormal lesions are obtained according to the target area distribution map;

[0051] Among them, the number of closed connected domains surrounded by target area labels in the target area distribution map is the number of target areas, which is recorded as n2;

[0052] The sum of the number of pixels in the closed connected domain surrounded by the target area labels in the target area distribution map is the total area of ​​the target area, denoted as S2, and the expression is as follows:

[0053] Among them, S i is the number of pixels occupied by the i-th target area;

[0054] The abnormal lesion ratio is the ratio of the number of abnormal lesions to the number of all target areas, and the expression is as follows:

[0055]

[0056] Among them, r num It was recorded as the percentage of the number of abnormal lesions;

[0057] The abnormal lesion area ratio is the ratio of the total lesion area to the total target area, and the expression is as follows:

[0058]

[0059] Among them, r area It was recorded as the percentage of abnormal lesion area.

[0060] The second embodiment of the present invention provides a pathological slice image analysis device based on an image coding large model, comprising:

[0061] An image segmentation module is used to segment the original pathological slice image according to a preset size to obtain a set of pathological slice sub-images;

[0062] A local target area distribution map generation module is used to sequentially input the pathological slice sub-images in the pathological slice sub-image set into a preset target area segmentation model to obtain a corresponding local target area distribution map; wherein the target area segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model;

[0063] A target region distribution map generation module, configured to obtain a target region distribution map corresponding to the original pathological slice image and generate a positive local target region distribution map set by processing the local target region distribution map;

[0064] An abnormal lesion distribution map generating module is used to generate an abnormal lesion distribution map corresponding to the original pathological slice image based on the positive local target area distribution map set and using a preset abnormal lesion detection network;

[0065] A statistical analysis module is used to implement statistical analysis of the original pathological slice image based on the target area distribution map and the abnormal lesion distribution map.

[0066] A third aspect of the present invention provides an electronic device, including:

[0067] at least one processor; and a memory communicatively coupled to the at least one processor;

[0068] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on the image coding large model.

[0069] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on the image coding large model.

[0070] The characteristics and beneficial effects of the present invention are:

[0071] The present invention uses deep learning technology to analyze various pathological section images, automatically identify and classify abnormal lesion areas in the images; specifically, it can identify a variety of lesion characteristics, such as changes in tissue structure, abnormal cell morphology, etc., and provide quantitative indicators such as the classification result image and the proportion of the number of lesion areas.

[0072] The main advantages of the present invention are its efficiency and accuracy. Through the automated analysis of the deep learning model, the workload of pathologists can be greatly reduced and the diagnostic efficiency can be improved. At the same time, since the influence of human subjective factors is eliminated, the present invention can provide more objective and accurate diagnostic results. In addition, based on the technical means of big data and image coding, the present invention can effectively integrate and analyze pathological slice data from different hospitals, break data silos, and realize the sharing and optimization of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The present invention is an overall flow chart of a pathological slice image analysis method based on an image coding large model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The present invention proposes a pathological slice image analysis method and device based on an image coding large model, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0075] The first embodiment of the present invention provides a pathological slice image analysis method based on an image coding large model, comprising:

[0076] The original pathological slice image is segmented according to a preset size to obtain a set of pathological slice sub-images;

[0077] The pathological slice sub-images in the pathological slice sub-image set are sequentially input into a preset target region segmentation model to obtain a corresponding local target region distribution map; wherein the target region segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model;

[0078] By processing the local target area distribution map, a target area distribution map corresponding to the original pathological slice image is obtained and a positive local target area distribution map set is generated;

[0079] Based on the positive local target area distribution map set, using a preset abnormal lesion detection network, generating an abnormal lesion distribution map corresponding to the original pathological slice image;

[0080] Based on the target area distribution map and the abnormal lesion distribution map, a statistical analysis of the original pathological slice image is performed.

[0081] In a specific implementation of the present invention, the pathological slice image analysis method based on the image coding large model has the following overall process: Figure 1 As shown, the following steps are included:

[0082] 1) Segment the acquired pathological slice image to obtain a set of pathological slice sub-images; the specific steps are as follows:

[0083] 1-1) Obtain a single pathological section image containing a complete and intact lesion.

[0084] In this embodiment, the pathological slice image can be loaded from a slice scanner, a storage device or a microscope. The image needs to include the complete field of view of the entire slide, and the file format can be .jpg and .png; the storage device can be a mobile hard disk, a U disk or a computer internal storage that stores the pathological slice image.

[0085] In this embodiment, the pathological section image can also be obtained by a microscope, that is, the stained pathological section is placed on the microscope stage, the microscope is focused, a camera is used to obtain a pathological section scan image of the pathological section under the microscope field of view, the pathological section is moved, and finally all the pathological section scan images are spliced ​​together to form a complete pathological section image.

[0086] 1-2) Segmenting the pathological slice image obtained in step 1) to obtain a set of pathological slice sub-images;

[0087] The size of each sub-image in the slice sub-image set is H×W, that is, the length of the sub-image is H, the width is W, and the unit is pixel (px). For different pathological slice images, H and W are not fixed.

[0088] In a specific embodiment of the present invention, the expression for image segmentation of the original pathological slice image is as follows, where H=512 and W=512:

[0089]

[0090] Wherein, height and width represent the length and width of the pathological slice sub-image, respectively, in pixels (px); in a specific implementation of the present invention, height = 512, width = 512. n is the number of sub-images to be segmented in the height direction, and m is the number of sub-images to be segmented in the width direction. In this embodiment, the pathological slice image is segmented from the upper left to the lower right in a step size of 512 pixels. For the part that cannot be completely segmented due to the image boundary, it is padded to 512×512 pixels in size by mirror filling for segmentation. The number of pathological slice sub-images of 512×512 pixels obtained in the end should be n×m, and all pathological slice sub-images constitute a pathological slice sub-image set.

[0091] 2) Inputting the pathological slice sub-images obtained after the segmentation in step 1) into the preset target region segmentation large model F-model in sequence to obtain the corresponding local target region distribution map; by processing the local target region distribution map, obtaining the target region distribution map corresponding to the original pathological slice image and generating a positive local target region distribution map set; the specific steps are as follows:

[0092] 2-1) The pathological slice sub-image segmented in step 1) is input into a preset target region segmentation large model F-model, and the large model outputs a local target region distribution map corresponding to the input sub-image.

[0093] Among them, the target area segmentation large model F-model consists of two parts: an encoder and a decoder. In this embodiment, the encoder uses the DinoV2 image information encoding large model, which extracts features from the input image and outputs multi-scale multi-layer feature maps. In this embodiment, for each input pathological slice sub-image of 512×512 pixels, the encoder will output 64x64 (8 times downsampling), 32x32 (16 times downsampling), and 16x16 (32 times downsampling) feature maps of a total of 3 scales; the decoder uses the main decoder UPerNet and the auxiliary decoder FCNNet to combine, that is, UPerNet and FCNNet work in parallel on the multi-scale feature maps generated by the encoder, UPerNet and FCNNet process these feature maps independently, and then weightedly fuse the results processed by UPerNet and FCNNet at the output end of the decoder. This method ensures that the strengths of the two decoders are simultaneously utilized, where UPerNet is responsible for processing global structural information, while FCNNet is responsible for processing detail information.

[0094] In this embodiment, the multi-scale multi-layer feature map output by the encoder is input into the decoder, and the main decoder and the auxiliary decoder complete the recognition of the target area, and finally the recognition results obtained by the main decoder and the auxiliary decoder are weighted and fused. Among them, the respective weights (such as 0.5 and 0.5) are initially set, and the value of the weight can be adjusted manually: the weights are gradually adjusted through experiments to observe their impact on the segmentation accuracy; an adaptive learning mechanism can also be introduced to achieve automatic weight adjustment. The specific method is: set two initial weights w1 and w2, corresponding to the main decoder (UPerNet) and the auxiliary decoder (FCNNet), respectively. These weights can be initialized to the same value (0.5), and the constraint w1+w2=1. During the training process of the model, all parameters in the model will be optimized by the gradient descent method, including the network weights of the decoder.

[0095] This embodiment is processed by the target region segmentation large model F-model, and finally obtains a local target region distribution map of 512×512 pixels corresponding to the input pathological slice sub-image;

[0096] 2-2) All local target area distribution maps obtained in step 2-1) are spliced ​​according to the positions of their corresponding sub-images in the original pathological slice image, and finally a complete target area distribution map is obtained.

[0097] 2-3) Traverse all local target area distribution maps, positively mark the local target area distribution maps with target area distribution, and negatively mark the local target area distribution maps without target area distribution, and then number the positively marked local target area distribution maps and the negatively marked local target area distribution maps in sequence according to the input order of their corresponding pathological slice sub-images in step 2-1), and finally form a positive local target area distribution map set and a negative local target area distribution map set, respectively.

[0098] Furthermore, before this step, the method of this embodiment further includes training the target region segmentation model, specifically including:

[0099] a) Construct a training set of a large model for target region segmentation;

[0100] In this embodiment, the image sources of the training set of the target area segmentation large model are diverse, including public data sets, pathology slice scanners, microscopes and other equipment. The pathology type of the image in the training set needs to be consistent with the pathology type of the pathology slice image in step 1), and needs to include the complete field of view of the entire slide. This embodiment does not require the brand or type of the acquisition equipment to be consistent, and the size of the complete pathology slice image does not require consistency, but the magnification of the acquisition equipment and the pixel density (DPI) of the image need to be consistent.

[0101] Each complete pathology slide image must be annotated by a clinical pathologist to outline the target area as the training label of the image. The more training set images the target area segmentation model has, the better the final training model effect will be. Generally, more than 50 images are needed as much as possible.

[0102] Furthermore, the training set images need to be segmented. In this embodiment, each image in the training set is segmented into a set of sub-images of size H×W (eg, 512×512 pixels) as input for training the large target region segmentation model.

[0103] b) Build a large model for target area segmentation.

[0104] In this embodiment, the target region segmentation model is composed of an encoder and a decoder. The encoder uses a DinoV2 model to extract multi-scale features, and the decoder uses a combination of UPerNet and FCNNet to process feature maps of different scales and output them after combination.

[0105] c) Using the training set of the target region segmentation large model to train the target region segmentation large model.

[0106] Optimizer: The Adam optimizer is usually used, and the learning rate is usually 1e-4, but it can be adjusted according to the size of the training set and the complexity of the model.

[0107] Loss function: Cross-Entropy Loss is used for segmentation tasks.

[0108] Batch size: Usually set to 16 or 32, the specific size depends on the hardware memory and training speed.

[0109] Learning Rate Scheduler: Use CosineAnnealing to gradually reduce the learning rate to avoid overfitting during training.

[0110] Regularization method: In order to prevent overfitting, L2 regularization is added during training.

[0111] Training end conditions:

[0112] In this embodiment, when the loss on the preset validation set no longer decreases, or the loss stabilizes after multiple epochs, the training is stopped. According to the actual situation, the maximum number of epochs can be set, such as 100 epochs, and indicators such as Dice coefficient and IoU (Intersection over Union) can also be set to measure the segmentation effect of the model. When these indicators reach a certain threshold on the validation set, such as 90%, the training is stopped.

[0113] Specifically, the training process is as follows:

[0114] In each epoch, the model generates predictions through forward propagation, calculates errors through the loss function, and updates network weights using back propagation. The validation set is used to evaluate the generalization performance of the model and prevent overfitting. During the training process, the loss and segmentation accuracy on the validation set are monitored. When the early stopping condition is met or the maximum number of iterations is reached, the training process ends and the optimal model is saved as the final trained target region segmentation model.

[0115] 3) Based on the positive local target area distribution map set obtained in step 2), a preset abnormal lesion detection network is used to generate an abnormal lesion distribution map corresponding to the original pathological slice image; the specific steps are as follows

[0116] 3-1) The pathological slice sub-image corresponding to each distribution map in the positive local target area distribution map set is sequentially input into the preset abnormal lesion detection network Attention-UNet, which outputs the corresponding local abnormal lesion distribution map.

[0117] The Attention-UNet is an advanced deep learning architecture. It is based on the classic U-Net model and enhances the model's ability to recognize key features in the image by introducing an attention mechanism, that is, an attention module is added to the jump connection part between the encoder and decoder of the original U-Net. Attention-UNet consists of an encoder and a decoder, wherein the encoder of Attention-UNet is used to gradually reduce the spatial dimension of the image and increase the number of feature channels to capture more abstract features; the decoder of Attention-UNet is used to gradually restore the spatial dimension of the image and reduce the number of feature channels to achieve accurate positioning. In Attention-UNet, each jump connection stage of the encoder and decoder is provided with an attention module, which passes more important feature information to the corresponding decoding layer by weighted control of the features from the encoder and the features of the corresponding positions of the decoder, and suppresses those features that are irrelevant or secondary to the segmentation task. In this embodiment, the network structure that combines the context perception ability of U-Net and the focusing ability of the attention mechanism can locate abnormal lesions with high precision. A specific embodiment of the present invention inputs a 512×512 pixel pathological slice sub-image into the network, and finally obtains a 512×512 pixel local abnormal lesion distribution map output by Attention-UNet processing.

[0118] In this embodiment, through this step, all pathological slice sub-images corresponding to the local target area distribution map marked as positive obtain their corresponding local abnormal lesion distribution maps.

[0119] 3-2) The local abnormal lesion distribution map obtained in step 3-1) is spliced ​​according to the position of the corresponding pathological slice sub-image in the original pathological slice image to obtain a complete abnormal lesion distribution map.

[0120] Since this embodiment only processes the positively marked sub-image, some gaps will appear during the stitching. For the missing parts, it is considered that there is no abnormal lesion distribution, and they are directly filled with a full-empty distribution map (if a lesion exists, it is marked with 1, and if a lesion does not exist, it is marked with 0, then all the missing parts are marked with 0). After the stitching is completed, a complete abnormal lesion distribution map is finally obtained.

[0121] Furthermore, before this step, the method of this embodiment further includes training the abnormal lesion detection network, specifically including:

[0122] a) constructing a training set for abnormal lesion detection network;

[0123] In this embodiment, the image sources of the training set of the abnormal lesion detection network are diverse, including public data sets, pathology slice scanners, microscopes and other equipment. The pathology type of the images in the training set needs to be consistent with the pathology type of the pathology slice image in step 1), and needs to include the complete field of view of the entire slide. This embodiment does not require the brand or type of the acquisition equipment to be consistent, and the size of the complete pathology slice image does not require consistency, but the magnification of the acquisition equipment and the pixel density (DPI) of the image need to be consistent.

[0124] Each complete pathological section image must be annotated by a clinical pathologist to outline the lesion area as the training label of the image. The more images in the training set of the abnormal lesion detection network, the better the final training model effect will be. Generally, more than 50 images are needed as much as possible.

[0125] Furthermore, the training set images need to be segmented. In this embodiment, each image in the training set is segmented into a set of sub-images of size H×W (eg, 512×512 pixels) as input for abnormal lesion detection network training.

[0126] b) Construct an abnormal lesion detection network.

[0127] In this embodiment, the abnormal lesion detection network uses Attention-UNet, in which the model uses an attention mechanism between the encoder and the decoder to enhance the model's attention to key areas.

[0128] c) Using the training set of the abnormal lesion detection network model to train the abnormal lesion detection network.

[0129] The training parameters in this embodiment are set as follows:

[0130] Optimizer: The Adam optimizer is usually used, and the learning rate is usually 1e-4, but it can be adjusted according to the size of the training set and the complexity of the model.

[0131] Loss function: Cross-Entropy Loss is used for segmentation tasks.

[0132] Batch size: Usually set to 16 or 32, the specific size depends on the hardware memory and training speed.

[0133] Learning rate scheduler: Use the cosine annealing algorithm to gradually reduce the learning rate to avoid overfitting during training.

[0134] Regularization method: In order to prevent overfitting, L2 regularization is added during training.

[0135] Training end conditions:

[0136] In this embodiment, when the loss on the preset validation set no longer decreases, or the loss stabilizes after multiple epochs, the training is stopped. According to the actual situation, the maximum number of epochs can be set, such as 100 epochs, and indicators such as Dice coefficient and IoU (Intersection over Union) can also be set to measure the segmentation effect of the model. When these indicators reach a certain threshold on the validation set, such as 90%, the training is stopped.

[0137] Specifically, the training process is as follows:

[0138] In each epoch, the model generates prediction results through forward propagation, calculates the error through the loss function, and updates the network weights using back propagation. The validation set is used to evaluate the generalization performance of the model and prevent overfitting. During the training process, the loss and segmentation accuracy on the validation set are monitored. When the early stopping condition is met or the maximum number of iterations is reached, the training process ends and the optimal model is saved as the final trained abnormal lesion detection network model.

[0139] 4) Based on the target area distribution map of step 2) and the abnormal lesion distribution map of step 3), statistics are collected on the distribution of abnormal lesions in the pathological slice image, and lesion count and lesion rate analysis is performed; specifically including:

[0140] According to the abnormal lesion distribution map, the number of abnormal lesions and the total area of ​​lesions can be obtained by using the connected domain counting algorithm and area statistics. These two indicators reflect the distribution area and severity of the lesions in the pathological slice image, which are important bases for doctors to make diagnoses and are of great significance.

[0141] The number of abnormal lesions can be obtained in the following way: counting the number of closed connected domains surrounded by lesion labels in the abnormal lesion distribution map, which is defined as the number of abnormal lesions, denoted as n1.

[0142] The total lesion area can be obtained by counting the sum of the number of pixels in the closed connected domain surrounded by the lesion labels in the abnormal lesion distribution map. The number of pixels is defined as the total lesion area, recorded as S1, and the calculation method is as follows:

[0143]

[0144] Among them, S i is the number of pixels occupied by the i-th lesion area.

[0145] According to the target area distribution map, the number of target areas and the proportion of abnormal lesions can be obtained. Among them, the target area is the suspected lesion area, that is, the area where the lesion may occur; the proportion of abnormal lesions, including the proportion of number and area, can provide doctors with the overall lesion situation in the pathological section map on a larger scale, which is a very important basis for diagnosis.

[0146] The number of target areas can be obtained in the following way: count the number of closed connected domains surrounded by target area labels in the target area distribution map, and define this number as the number of target areas, denoted as n2.

[0147] The total area of ​​the target region can be obtained by counting the sum of the number of pixels in the closed connected domain surrounded by the target region labels in the target region distribution map. The number of pixels is defined as the total area of ​​the target region, denoted as S2. The calculation method is as follows:

[0148]

[0149] Among them, S i is the number of pixels occupied by the i-th target area.

[0150] The abnormal lesion ratio can be obtained in the following way: The abnormal lesion ratio is the ratio of the number of abnormal lesions to the number of all target areas, and the calculation method is as follows:

[0151]

[0152] Among them, r num It is recorded as the percentage of abnormal lesions.

[0153] The abnormal lesion area ratio can be obtained in the following way: The abnormal lesion area ratio is the ratio of the total lesion area to the total target area, and the calculation method is as follows:

[0154]

[0155] Among them, r area It was recorded as the percentage of abnormal lesion area.

[0156] Furthermore, the method described in this embodiment also includes:

[0157] The target area distribution map, abnormal lesion distribution map and generated quantitative indicators, including the total lesion area, the proportion of abnormal lesions and the proportion of abnormal lesion area, are visualized and output as images and texts to assist physicians in diagnosis.

[0158] To implement the above embodiment, a second aspect of the present invention provides a pathological slice image analysis device based on an image coding large model, comprising:

[0159] An image segmentation module is used to segment the original pathological slice image according to a preset size to obtain a set of pathological slice sub-images;

[0160] A local target area distribution map generation module is used to sequentially input the pathological slice sub-images in the pathological slice sub-image set into a preset target area segmentation model to obtain a corresponding local target area distribution map; wherein the target area segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model;

[0161] A target region distribution map generation module, configured to obtain a target region distribution map corresponding to the original pathological slice image and generate a positive local target region distribution map set by processing the local target region distribution map;

[0162] An abnormal lesion distribution map generating module is used to generate an abnormal lesion distribution map corresponding to the original pathological slice image based on the positive local target area distribution map set and using a preset abnormal lesion detection network;

[0163] A statistical analysis module is used to implement statistical analysis of the original pathological slice image based on the target area distribution map and the abnormal lesion distribution map.

[0164] It should be noted that the above-mentioned explanation of the embodiment of a pathological slice image analysis method based on an image coding large model is also applicable to a pathological slice image analysis device based on an image coding large model of the present embodiment, and will not be repeated here. According to a pathological slice image analysis device based on an image coding large model proposed in an embodiment of the present invention, the original pathological slice image is segmented according to a preset size to obtain a pathological slice sub-image set; the pathological slice sub-images in the pathological slice sub-image set are sequentially input into a preset target region segmentation large model to obtain a corresponding local target region distribution map; wherein the target region segmentation large model is composed of an encoder and a decoder, and the encoder adopts an image coding large model; by processing the local target region distribution map, a target region distribution map corresponding to the original pathological slice image is obtained and a positive local target region distribution map set is generated; based on the positive local target region distribution map set, a preset abnormal lesion detection network is used to generate an abnormal lesion distribution map corresponding to the original pathological slice image; based on the target region distribution map and the abnormal lesion distribution map, a statistical analysis of the original pathological slice image is realized. This makes it possible to analyze various pathological slice images, automatically identify and classify abnormal lesion areas in the images, and automatically perform statistical analysis to obtain key clinical indicators such as lesion rate, thereby solving the problems of low efficiency and poor real-time performance caused by the existing manual observation of slices for diagnosis, improving the accuracy and efficiency of diagnosis, and promoting the advancement and development of digital pathology.

[0165] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0166] at least one processor; and a memory communicatively coupled to the at least one processor;

[0167] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on the image coding large model.

[0168] To implement the above-mentioned embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on the image coding large model.

[0169] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0170] The computer readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the pathological slice image analysis method based on the image coding large model of the above embodiment.

[0171] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0172] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0173] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0174] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0175] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0176] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0177] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0179] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pathological slice image analysis method based on an image coding large model, characterized in that: include: The original pathological slice image is segmented according to a preset size to obtain a set of pathological slice sub-images; The pathological slice sub-images in the pathological slice sub-image set are sequentially input into a preset target region segmentation model to obtain a corresponding local target region distribution map; wherein the target region segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model; By processing the local target area distribution map, a target area distribution map corresponding to the original pathological slice image is obtained and a positive local target area distribution map set is generated; Based on the positive local target area distribution map set, using a preset abnormal lesion detection network, generating an abnormal lesion distribution map corresponding to the original pathological slice image; Based on the target area distribution map and the abnormal lesion distribution map, statistical analysis of the original pathological slice image is performed.

2. The method according to claim 1, characterized in that The step of obtaining a set of pathological slice sub-images comprises: 1) Obtaining a single pathological section image containing a complete lesion as an original pathological section image; 2) Segmenting the original pathological slice image obtained in step 1) to obtain a set of pathological slice sub-images; The size of the pathological slice sub-image is H×W, where H represents the length of the pathological slice sub-image and W represents the width of the pathological slice sub-image. The split expression is as follows: Among them, height and width represent the length and width of the original pathological slice sub-image respectively; n is the number of sub-images to be segmented in the length direction, and m is the number of sub-images to be segmented in the width direction; for the part that reaches the image boundary and cannot be completely segmented, it is padded to H×W pixel size by mirror filling for segmentation, and the final number of pathological slice sub-images is n×m.

3. The method according to claim 1, characterized in that The encoder of the target area segmentation model uses the DinoV2 image information coding model, which processes the input pathological slice sub-image to generate a multi-scale feature map; the decoder of the target area segmentation model uses a combination of the main decoder UPerNet and the auxiliary decoder FCNNet. UPerNet and FCNNet work in parallel on the multi-scale feature map generated by the encoder, and then the results of UPerNet and FCNNet processing are weightedly fused at the output end of the decoder, and finally a local target area distribution map of H×W pixel size corresponding to the input pathological slice sub-image is output.

4. The method according to claim 3, characterized in that The step of obtaining the target area distribution map corresponding to the original pathological slice image and generating a set of positive local target area distribution maps includes: Based on the local target area distribution map of each pathological slice sub-image in the pathological slice sub-image set output by the target area segmentation large model, the sub-images corresponding to the local target area distribution map are spliced ​​according to the positions of the sub-images in the original pathological slice image to obtain the target area distribution map corresponding to the original pathological slice image; All local target area distribution maps are traversed, and local target area distribution maps with target area distribution are positively marked, and then the positively marked local target area distribution maps and their corresponding pathological slice sub-images are numbered in sequence according to the order when they are input into the target area segmentation large model, and finally a set of positive local target area distribution maps is generated.

5. The method according to claim 3, characterized in that: Before sequentially inputting the pathological slice sub-images in the pathological slice sub-image set into a preset target region segmentation large model, the method further includes: Training the target region segmentation model; The training of the target area segmentation model comprises: 1)) Construct a training set of a large model for target area segmentation; Wherein, the pathological type of the image in the training set of the target region segmentation large model is consistent with the pathological type of the original pathological slice image; Annotating the target region of the image of the training set of the target region segmentation model as a training label of the image; Then, each image in the training set is divided into a set of sub-images of size H×W, which are used as input for training the large model for target region segmentation; 2) Construct a large model for target area segmentation; 3) Using the training set obtained in step 1), the target region segmentation large model constructed in step 2) is trained to obtain a trained target region segmentation large model.

6. The method according to claim 4, characterized in that The abnormal lesion detection network adopts Attention-UNet, which is composed of an encoder and a decoder, wherein the encoder of Attention-UNet is used to gradually reduce the spatial dimension of the image and increase the number of feature channels to capture more abstract features; the decoder of Attention-UNet is used to gradually restore the spatial dimension of the image and reduce the number of feature channels to achieve accurate positioning; in the Attention-UNet, each jump connection stage of the encoder and the decoder is provided with an attention module, which controls the features from the encoder and the features of the corresponding positions of the decoder by weighting.

7. The method according to claim 6, characterized in that The method of generating an abnormal lesion distribution map corresponding to the original pathological slice image based on the positive local target area distribution map set and using a preset abnormal lesion detection network comprises: 1) inputting the pathological slice sub-image corresponding to each distribution map in the positive local target area distribution map set into a preset abnormal lesion detection network in sequence, and the network outputs the corresponding local abnormal lesion distribution map; 2) The local abnormal lesion distribution map obtained in step 1) is spliced ​​according to the position of the corresponding pathological slice sub-image in the original pathological slice image, wherein the vacant parts appearing during splicing are filled with a full-empty distribution map, and a complete abnormal lesion distribution map is obtained after the splicing is completed.

8. The method according to claim 6, characterized in that Before using the preset abnormal lesion detection network to generate the abnormal lesion distribution map corresponding to the original pathological slice image, the method further includes: Training the abnormal lesion detection network; The training of the abnormal lesion detection network comprises: 1) Constructing a training set for abnormal lesion detection network; Wherein, the pathological type of the image in the training set of the abnormal lesion detection network is consistent with the pathological type of the original pathological section image; Annotating the image of the training set of the abnormal lesion detection network with the lesion area as the training label of the image; Then, each image in the training set is divided into a set of sub-images of size H×W, which are used as input for abnormal lesion detection network training; 2) Construct an abnormal lesion detection network; 3) Using the training set of the abnormal lesion detection network model obtained in step 1), the abnormal lesion detection network constructed in step 2) is trained to obtain a trained abnormal lesion detection network.

9. The method according to claim 7, characterized in that: The statistical analysis of the original pathological slice image includes: According to the abnormal lesion distribution map, the number of abnormal lesions and the total area of ​​lesions were obtained by using the connected domain counting algorithm and area statistics; Among them, the number of closed connected domains surrounded by lesion labels in the abnormal lesion distribution map is the number of abnormal lesions, which is recorded as n1; The sum of the number of pixels in the closed connected domain surrounded by the lesion labels in the statistical abnormal lesion distribution map is the total lesion area, denoted as S1, and the expression is as follows: Among them, S i is the number of pixels occupied by the i-th lesion area; The number of target areas and the proportion of abnormal lesions are obtained according to the target area distribution map; Among them, the number of closed connected domains surrounded by target area labels in the target area distribution map is the number of target areas, which is recorded as n2; The sum of the number of pixels in the closed connected domain surrounded by the target area labels in the target area distribution map is the total area of ​​the target area, denoted as S2, and the expression is as follows: Among them, S i is the number of pixels occupied by the i-th target area; The abnormal lesion ratio is the ratio of the number of abnormal lesions to the number of all target areas, and the expression is as follows: Among them, r num It was recorded as the percentage of the number of abnormal lesions; The abnormal lesion area ratio is the ratio of the total lesion area to the total target area, and the expression is as follows: Among them, r area It was recorded as the percentage of abnormal lesion area.

10. A pathological slice image analysis device based on an image coding large model, characterized in that: include: An image segmentation module is used to segment the original pathological slice image according to a preset size to obtain a set of pathological slice sub-images; A local target area distribution map generation module is used to sequentially input the pathological slice sub-images in the pathological slice sub-image set into a preset target area segmentation model to obtain a corresponding local target area distribution map; wherein the target area segmentation model is composed of an encoder and a decoder, and the encoder adopts an image coding model; A target region distribution map generation module, configured to obtain a target region distribution map corresponding to the original pathological slice image and generate a positive local target region distribution map set by processing the local target region distribution map; An abnormal lesion distribution map generating module is used to generate an abnormal lesion distribution map corresponding to the original pathological slice image based on the positive local target area distribution map set and using a preset abnormal lesion detection network; A statistical analysis module is used to implement statistical analysis of the original pathological slice image based on the target area distribution map and the abnormal lesion distribution map.

Images