Pathological section image analysis method and device based on online learning

Abstract

The invention provides a pathological section image analysis method and device based on online learning, and belongs to the technical field of computer vision, deep learning and medical image analysis. The method comprises the steps that a pathological section analysis model is trained offline, the input of the pathological section analysis model is a pathological section image, and the output of the pathological section analysis model is a pathological section focus segmentation result of the image; performing online learning on the current pathological section analysis model by using a newly obtained pathological section image to obtain an updated pathological section analysis model; and inputting a to-be-analyzed pathological section image into the updated pathological section analysis model, and finally obtaining a pathological section focus segmentation result of the to-be-analyzed pathological section image. According to the method, the new pathological section images are continuously transmitted into the pathological section analysis model subjected to offline training in real time for fine adjustment of the model, so that the model dynamically learns from the new images and optimizes model parameters, and the analysis accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, deep learning and medical image analysis, and in particular to a pathological slice image analysis method and device based on online learning. Background Art

[0002] Traditional pathological analysis mainly relies on visual judgment by experienced pathologists through a microscope. Although this method is intuitive, it has significant limitations. The judgments of different pathologists may be subjective, resulting in inconsistent analysis results. In addition, this manual method is inefficient, and each slice needs to be carefully examined. When faced with a large number of pathological samples, it is often difficult for pathologists to complete all analyses in a short period of time. Long-term microscopic observation can also cause pathologists to experience eye fatigue and lack of concentration, thereby increasing the possibility of misdiagnosis.

[0003] With the development of computer vision and deep learning technologies, automated pathology image analysis has been widely promoted. These technologies have significantly improved the efficiency and accuracy of pathology diagnosis through methods such as image preprocessing and enhancement, feature extraction and representation, and image segmentation and classification. However, most existing automated pathology analysis systems use an offline training mode. After the model is trained on a fixed data set, it is no longer updated and cannot adapt to new sample data in clinical practice, which limits the generalization ability and adaptability of the model. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the existing technology and propose a pathological slice image analysis method and device based on online learning. The present invention continuously transfers new pathological slice images to the offline trained pathological slice analysis model in real time to fine-tune the model, so that the model can dynamically learn from the new images and optimize the model parameters, thereby improving the accuracy and efficiency of the analysis.

[0005] The first embodiment of the present invention proposes a pathological section image analysis method based on online learning, comprising:

[0006] Offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image;

[0007] Using the newly acquired pathological section image to perform online learning on the current pathological section analysis model to obtain an updated pathological section analysis model;

[0008] The pathological slice image to be analyzed is input into the updated pathological slice analysis model, and finally a pathological slice lesion segmentation result of the pathological slice image to be analyzed is obtained.

[0009] In a specific embodiment of the present invention, the offline training pathological section analysis model includes:

[0010] Construct pathological slide image training set and validation set;

[0011] Constructing a pathological section analysis model, wherein the pathological section analysis model is composed of a pre-trained model and an additionally added task head;

[0012] Freeze the pre-trained model, train the task head using the training set and test it using the validation set to obtain the task head that has been preliminarily trained;

[0013] The pre-trained model is unfrozen, the pathological section analysis model is fine-tuned using the training set and tested using the validation set to obtain the pathological section analysis model that has been finally trained offline.

[0014] In a specific embodiment of the present invention, the method of performing online learning on the current pathological section analysis model using the newly acquired pathological section image to obtain the updated pathological section analysis model includes:

[0015] Get a batch of new pathological slice images D new , D new The sample consisting of the pathological slice image and its true label in is denoted as where x i is the i-th pathological slice image, y i is the true label corresponding to the i-th pathological slice image, and n is the number of samples; new The pathological slice image in is input into the current pathological slice analysis model, and the output result of the model is the predicted label of the image. After the predicted label is corrected, the real label of the image is obtained;

[0016] D new The pathological slice images in the image are input into the current pathological slice analysis model again for forward propagation to obtain the predicted labels corresponding to each pathological slice image. The cross entropy loss function is used to calculate the error between the predicted label and the true label, where the loss function calculation expression is as follows:

[0017]

[0018] Among them, c is the lesion category index, y i,c is the true label corresponding to the c-th lesion, is the predicted label result corresponding to the c-th type of lesion;

[0019] Back propagation and model weight update are performed based on the calculation results of the loss function. The small batch stochastic gradient descent method is used to update the model weight. The weight update calculation expression is as follows:

[0020]

[0021] Among them, θ t is the model parameter before weight update, θ t+1 is the model parameter after one weight update, η is the learning rate;

[0022] When the online learning reaches the preset training parameters, the pathological section analysis model is obtained after fine-tuning the single batch data.

[0023] In a specific embodiment of the present invention, the pathological section analysis model uses a pre-trained DinoV2 model as the main body, and adds a MaskFormer task head to the output layer of the DinoV2 model.

[0024] In a specific embodiment of the present invention, the step of constructing a training set and a validation set of pathological slice images includes:

[0025] 1) Acquire pathological slice images to form a pathological slice image dataset;

[0026] The pathological slice image dataset includes pathological slice images corresponding to a plurality of pathological types, wherein each pathological slice image has a consistent size;

[0027] 2) annotating each pathological slice image of the pathological slice image data set of step 1); wherein the annotation content of each pathological slice image includes a polygonal boundary of the lesion area and a corresponding lesion type label;

[0028] 3) Preprocessing the images of the pathological slice image dataset annotated in step 2) to normalize the pixel value of each image to the range of [0, 1];

[0029] 4) The pathological section data set preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

[0030] In a specific embodiment of the present invention, it also includes:

[0031] The performance of the pathological section analysis model that has been trained offline is evaluated using a preset test set, and the pathological section analysis model is adjusted according to the evaluation result to obtain an optimized pathological section analysis model; wherein the pathological section images in the test set are different from the pathological section images in the training set and the validation set.

[0032] In a specific embodiment of the present invention, it also includes:

[0033] Using the dynamic model fusion mechanism, the pathological section analysis model before online learning adjustment is weightedly fused with the adjusted pathological section analysis model;

[0034] Among them, the model before single batch data adjustment using online learning is taken as the initial model, denoted as M0, and the parameters of the model are denoted as θ0; the model after single batch data adjustment using online learning is taken as the fine-tuned model, denoted as M1, and the parameters of the model are denoted as θ1.

[0035] The models before and after adjustment are integrated, and the expression is as follows:

[0036] θ fusion =αθ0+(1-α)θ1

[0037] Among them, θ fusion is the parameter of the fusion model, α is the fusion coefficient, and its value range is [0,1];

[0038] The fusion The pathological section analysis model used as a parameter is used as a new current pathological section analysis model.

[0039] The second aspect of the present invention provides a pathological slice image analysis device based on online learning, comprising:

[0040] An offline training module, used for offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image;

[0041] An online learning module, used to perform online learning on the current pathological section analysis model using the newly acquired pathological section image to obtain an updated pathological section analysis model;

[0042] The analysis module is used to input the pathological slice image to be analyzed into the updated pathological slice analysis model, and finally obtain the pathological slice lesion segmentation result of the pathological slice image to be analyzed.

[0043] A third aspect of the present invention provides an electronic device, including:

[0044] at least one processor; and a memory communicatively coupled to the at least one processor;

[0045] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on online learning.

[0046] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on online learning.

[0047] The characteristics and beneficial effects of the present invention are:

[0048] The present invention combines online learning technology with intelligent analysis technology of pathological sections, and uses new pathological section images obtained in actual use to perform online training on the current pathological section analysis model, thereby realizing real-time updating and dynamic optimization of the model, thereby enabling the model to have stronger generalization performance and significantly improving the accuracy and efficiency of pathological image analysis.

[0049] The present invention can not only retain the analysis capabilities of the existing model while learning new pathological slice analysis knowledge, but also accelerate the analysis process and improve the work efficiency of pathological slice image analysis. In general, the present invention has significant advantages and application prospects in improving the accuracy of pathological slice image analysis, enhancing the adaptability and stability of the model, and accelerating the clinical analysis process. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The figure is an overall flow chart of a pathological slice image analysis method based on online learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention proposes a pathological section image analysis method and device based on online learning, which is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0052] The first embodiment of the present invention proposes a pathological section image analysis method based on online learning, comprising:

[0053] Offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image;

[0054] Using the newly acquired pathological section image to perform online learning on the current pathological section analysis model to obtain an updated pathological section analysis model;

[0055] The pathological slice image to be analyzed is input into the updated pathological slice analysis model, and finally a pathological slice lesion segmentation result of the pathological slice image to be analyzed is obtained.

[0056] In a specific implementation of the present invention, the pathological section image analysis method based on online learning has the following overall process: Figure 1 As shown, the following steps are included:

[0057] 1) Construct pathological slice image training set and validation set; the specific steps are as follows:

[0058] 1-1) Acquire pathological slice images to form a pathological slice image dataset.

[0059] In this embodiment, the pathological slice image dataset contains pathological slice images corresponding to multiple pathological types, including but not limited to cancer, inflammation, benign lesions, etc., and the number of pathological slice images of each type should be greater than 100. The size of all pathological slice images in the dataset is consistent, and in a specific embodiment of the present invention, they are all 512×512 pixels to ensure consistency and standardization when inputting the model.

[0060] In a specific embodiment of the present invention, a total of 20,000 pathology slide images were collected from multiple pathology laboratories and medical institutions to form a pathology slide image dataset. In order to ensure the diversity and representativeness of the dataset, these images cover different pathological features, patient backgrounds, and sample sources.

[0061] Furthermore, in the process of collecting slice images in this embodiment, special attention is paid to the following points:

[0062] Diversity of pathological types to ensure that the dataset contains a wide range of pathological types, including cancer (such as breast cancer, lung cancer, gastric cancer, etc.), where the segmentation content is mainly the cancerous area and its boundaries; inflammation (such as hepatitis, gastritis, etc.), where the segmentation content is mainly the inflammatory reaction area and its boundaries; benign lesions (such as benign tumors, cysts, etc.), where the segmentation content is mainly the lesion area and its boundaries.

[0063] Diversity of sample sources,This embodiment collects samples from different geographical regions and medical institutions,,covering different patient groups and pathological characteristics to ensure,the broad representativeness of the dataset.

[0064] In addition, this embodiment selects high-quality pathological slice images to ensure the clarity and details of the images so that the model can learn more useful information.

[0065] 1-2) Annotate each pathological slice image in the pathological slice image dataset obtained in step 1-1).

[0066] In this embodiment, the annotation content of each pathological slice image includes the polygonal boundary of the lesion area and the corresponding lesion type label, and these annotation data will be used for subsequent model training and verification.

[0067] In a specific embodiment of the present invention, the labeling process includes: preliminary inspection, pathology experts first perform preliminary inspection on each pathology slice image to determine the existence and approximate location of the lesion area; detailed labeling, using professional labeling tools to perform detailed polygonal labeling on each lesion area and accurately outline the lesion boundary; type labeling, assigning a corresponding lesion type label to each labeled lesion area, such as cancer, inflammation or benign lesions; review, all labeling results are reviewed multiple times to ensure the accuracy and consistency of the labeling and avoid mislabeling or missing labels.

[0068] 1-3) Preprocessing the images of the pathological slice image dataset annotated in step 1-2).

[0069] In an embodiment of the present invention, the preprocessing includes: normalizing the image pixel values ​​to a range of [0, 1] to improve the training effect and convergence speed of the model.

[0070] 1-4) Divide the data set preprocessed in step 1-3) into a training set and a validation set.

[0071] In a specific implementation of the present invention, 15,000 images are randomly selected from a pathology slice image data set consisting of 20,000 images to form a training set, and the remaining 5,000 images form a validation set. Generally speaking, the ratio of the training set to the validation set can be 3:1.

[0072] Optionally, this embodiment can also perform data enhancement on the pathological slice image data set in the training set, such as random cropping, rotation, flipping, etc., to increase the diversity of the training data and improve the robustness of the model. Through these preprocessing, it is ensured that the pathological slice image data has a consistent format and quality when input into the model, thereby improving the training effect and performance of the model.

[0073] 2) Construct a pathological section analysis model.

[0074] In this embodiment, the existing pre-trained DinoV2 model is used as the main body of the pathological section analysis model, and a MaskFormer task head is added to the output layer of the DinoV2 model, and the pathological section analysis model is constructed.

[0075] It should be noted that DinoV2 is an advanced self-supervised visual representation learning model that can extract rich image features by training on large-scale image datasets. DinoV2 mainly uses Vision Transformers (ViT), also known as visual attention methods, as the architectural basis, and can efficiently process and understand the global content of images through a mechanism called "self-attention". In the process of self-supervised learning, DinoV2 uses a variety of data enhancement techniques, such as cropping, color perturbation, and Gaussian blur, to generate multiple views of the image and let the model learn the relationship between these views, so as to obtain a more in-depth and robust image representation.

[0076] In this embodiment, in the pathological slice analysis model, the input of DinoV2 is a preprocessed 512×512 pixel pathological slice image, and the output is a high-dimensional image feature vector, usually a feature map of 16×16×768. This feature map is generated by DinoV2 through its deep visual feature extraction mechanism and contains rich information of the input image.

[0077] MaskFormer is an advanced image segmentation model specially designed to handle complex image segmentation tasks. In this embodiment, the input of MaskFormer is the high-dimensional image feature vector output by DinoV2, and the output is the segmentation mask of each pixel in the pathological section image. Specifically, the MaskFormer task head consists of three convolutional layers, each with a convolution kernel size of 3×3 and equipped with 8 self-attention heads. Each self-attention head can capture different feature dimensions of the image and enhance the segmentation ability of the model. The classification probability of each pixel is generated by the last softmax in the MaskFormer task head to ensure that the output segmentation mask has accurate classification results to complete the lesion identification task of pathological sections.

[0078] 3) Using the training set obtained in step 1) to train the pathological section analysis model constructed in step 2); the specific steps are as follows:

[0079] 3-1) Pre-trained model in frozen pathology section analysis model, training task head.

[0080] In this embodiment, the pre-trained parameters of DinoV2 in the pathological section analysis model are frozen to ensure that these parameters remain unchanged during the training process at this stage. Among them, the parameters of the pre-trained model are frozen in order to maintain the feature extraction capabilities learned by the pre-trained model on a large-scale data set to avoid the destruction of these features during the initial training process; and only training the parameters of the task head allows the task head to focus on learning the specific features of the pathological section images. In this way, the model can utilize the rich features learned in the pre-training stage to enhance the segmentation ability of the task head. The advantage of this method is that it can make full use of the powerful feature extraction capabilities of the pre-trained model, while focusing on the optimization of the task head to improve the segmentation performance of the overall model.

[0081] Then, the MaskFormer task head is trained using the pathological slice image training set obtained in step 1). During the training process, the parameters of the task head are optimized by the back propagation algorithm using the same or similar loss function and optimization algorithm as the pre-trained model. Appropriate training parameters, including learning rate, training cycle, batch size, etc., are set to ensure that the task head converges stably during the training process.

[0082] In a specific embodiment of the present invention, the specific steps of training the MaskFormer task head are as follows:

[0083] First, the parameters of the MaskFormer task head are randomly initialized, and the parameters required for training are set, including the learning rate, training cycle, and batch size. In a specific embodiment of the present invention, the learning rate is set to 0.001, the training cycle is set to 50 epochs, and the batch size is set to 32. The cross entropy loss function and the Adam optimizer are selected, and the parameters of the MaskFormer task head are optimized by the back propagation algorithm. The cross entropy loss function is suitable for classification tasks and can effectively measure the difference between the predicted results and the actual labels, while the Adam optimizer ensures stability and rapid convergence during training through adaptive learning rate adjustment.

[0084] Then, the MaskFormer task head is trained using the pathological slice image training set. During the training process, the parameters of the task head are continuously adjusted so that it can accurately segment the lesion area. Through multiple rounds of iterative training, the task head gradually learns how to identify and segment the lesion area on the high-dimensional feature map, thereby improving the segmentation accuracy.

[0085] Finally, monitor the loss value and validation set performance during the training process in real time to ensure the stability and effectiveness of the training. When the loss value on the validation set no longer decreases significantly, you can consider ending the training. Monitoring the training process can help detect overfitting in a timely manner and optimize model performance by adjusting training parameters. For example, the original initial learning rate is 0.001. If the training process is too slow, it can be adjusted to 0.002. If the training process fails to converge, it can be adjusted to 0.0005.

[0086] Through the above steps, the MaskFormer task head can fully learn the specific features of pathological slice images based on the frozen DinoV2 pre-trained model, laying a solid foundation for the subsequent fine-tuning of the full model. This process not only improves the segmentation ability of the model, but also ensures the accuracy and stability of the model when processing complex pathological slice images.

[0087] When the MaskFormer task head is trained, proceed to step 3-2).

[0088] 3-2) Unfreeze the pre-trained model, and then fine-tune the pathology section analysis model; specifically include:

[0089] Unfreeze the pre-trained parameters of the DinoV2 model so that the parameters of all layers of the DinoV2 model can be updated in the subsequent training process.

[0090] Set the parameters required for fine-tuning training, including learning rate, training cycle, and batch size. The learning rate in the fine-tuning phase should be set low to avoid large perturbations to the learned features. A commonly used learning rate is 0.0001, the training cycle can be set to 20 epochs, and the batch size can be kept at 32. Select the cross entropy loss function and the Adam optimizer to optimize the parameters of the entire model through the back-propagation algorithm.

[0091] It should be noted that the setting of the learning rate is crucial during the fine-tuning training stage. The learning rate is usually reduced in stages. The initial learning rate is high so that the model parameters can be adjusted quickly. As the training progresses, the learning rate is gradually reduced to refine the adjustment of the large model parameters. This method can ensure that the pathology section analysis model converges stably during the fine-tuning process and avoid training oscillation or overfitting.

[0092] Through the above steps, the pre-trained model and the task head can be jointly optimized in the pathological slice image segmentation task to improve the performance of the overall model. The fine-tuned pathological slice analysis model not only has a powerful feature extraction capability, but also has a high-precision segmentation capability for pathological slice images, thereby meeting the needs of practical applications. The method described in this embodiment can significantly improve the overall performance and adaptability of the large model by making full use of the existing knowledge of the pre-trained model and combining the specific optimization of the task head, providing reliable technical support for actual pathological slice image analysis.

[0093] After fine-tuning is completed, the final offline trained pathological section analysis model for pathological sections is obtained.

[0094] 3-3) Further, optionally, this embodiment can also further optimize the model by evaluating the trained pathological section analysis model for pathological sections; the specific steps are as follows:

[0095] 3-3-1) Evaluate model performance;

[0096] After completing the fine-tuning training of the large model, the next step is to conduct a comprehensive evaluation of the large model to ensure that its performance in the pathology image segmentation task meets expectations. The evaluation process includes using a validation set and a test set to measure the segmentation accuracy and robustness of the model. Key evaluation indicators include intersection over union (IoU), pixel accuracy (PA), and Dice coefficient, which can fully reflect the performance of the model in the segmentation task. Specifically, they include:

[0097] First, prepare an independent test set to ensure that the images in the test set have not been used for the training and verification of the pathology slice analysis model to ensure the objectivity of the evaluation results. Then, input the pathology slice images in the test set into the trained pathology slice analysis model to generate the segmentation results of each pathology slice image in the test set. Based on the segmentation results, calculate the key evaluation indicators of the pathology slice analysis model on the test set, including intersection over union (IoU), pixel accuracy (PA), and Dice coefficient.

[0098] The intersection over union (IoU) is used to measure the overlap between the predicted segmentation and the actual segmentation, the pixel accuracy (PA) is used to measure the proportion of correctly classified pixels in the predicted results, and the Dice coefficient is used to evaluate the consistency between the predicted segmentation and the actual segmentation. By combining these evaluation indicators, we can fully understand the segmentation performance of the model.

[0099] 3-3-2) Optimize and adjust the model;

[0100] According to the evaluation results of step 3-3-1), the model is further optimized and adjusted to improve its performance in the pathological slice image segmentation task. The optimization process includes adjusting model parameters, improving training strategies, and redesigning data enhancement methods. The goal is to gradually improve the segmentation accuracy and robustness of the model through continuous iteration. Specifically, it includes:

[0101] First, analyze the evaluation results to identify the strengths and weaknesses of the model. For example, if the model performs poorly on certain types of lesions, consider increasing the amount of training data for these types of lesions or performing specialized parameter tuning for these types of lesions.

[0102] Then, adjust the model's hyperparameters, including learning rate, batch size, regularization coefficient, etc., and conduct multiple experiments to find the best parameter combination. You can use methods such as grid search or random search to systematically explore the effects of different parameter combinations. In addition, you can consider improving the training strategy, such as using more advanced optimization algorithms, introducing new loss functions, or increasing the diversity of training data.

[0103] Next, improve the data augmentation method to increase the diversity of training data and the robustness of the model. For example, more types of image transformation techniques can be introduced, such as color jitter, blur, noise, etc., so that the model can still maintain high performance when facing different image changes.

[0104] Finally, further fine-tuning training is performed to gradually improve the segmentation performance of the model by combining new data and optimization strategies. After each training, the performance of the model is re-evaluated and the next round of optimization is performed based on the evaluation results.

[0105] For example, this embodiment optimizes the pathological section analysis model finally obtained in step 3-2). First, an independent test set of pathological section data is input into the model for lesion identification and segmentation. The result of lesion segmentation is compared with the actual result of actual annotation, and the classification accuracy at the pixel level is calculated. Assuming that the average classification accuracy of different categories finally calculated is 80%, if the model is expected to better complete the recognition task and achieve a higher accuracy, for example, more than 90%, the model needs to be adjusted. First, find out which type of lesion recognition accuracy of the large model is low, and increase the weight of the corresponding category in the loss function so that the large model can better learn the characteristics of this type of lesion. For example, if the classification rate of cell edema is low, the weight corresponding to the original edema can be doubled in the loss function. In addition, the number of different types of lesions can be counted. If the number of a certain type of lesions is too small, the large model will not be able to learn the characteristics of this type of lesions, resulting in poor classification results. For example, if there are too few samples of cell proliferation lesions, it is necessary to enhance the data of this type of samples, such as distorting, rotating, adding noise and other operations to increase the number of samples of this type, and then train the large model. After the training is completed, retest the recognition accuracy and continue to look for parts that can be optimized.

[0106] In this embodiment, the pathological slice analysis model after fine-tuning and optimization has better pathological slice feature extraction and encoding effects, and can be applied to practical tasks such as pathological image analysis and diagnosis.

[0107] 4) Use the current pathology section analysis model to process the newly acquired pathology section images.

[0108] In this embodiment, in the actual usage scenario, a new pathological slice image is obtained and adjusted to 512×512 pixels. In principle, the number of pathological slices can be greater than 1, but the greater the number, the more obvious the effect of subsequent online learning; the resized pathological slice image is input into the current pathological slice analysis model (the initial current pathological slice analysis model is the pathological slice analysis model trained in step 3), and the model outputs a 512×512 pixel pathological slice lesion segmentation result as the predicted label corresponding to the input image.

[0109] 5) The pathological section lesion segmentation result obtained in step 4) is corrected by a professional pathological section analyst, and the corrected result is used as the true label of the corresponding pathological section image.

[0110] 6) Based on the predicted labels obtained in step 4) and the true labels obtained in step 5), the pathology section analysis model is adjusted using online learning.

[0111] Online learning is a technology that learns and optimizes prediction models in sequentially arriving data stream instances. It dynamically learns and adjusts the model in turn for the samples in the data stream. This dynamic learning method makes online learning more scalable and has higher memory utility, while retaining a certain degree of migration capability for the model. For pathological sections, the styles of pathological sections obtained by different section staining reagents and section scanning instruments can vary greatly, which can bring great difficulties to the lesion identification task of the pathological section analysis model. Therefore, it is necessary to adjust and optimize the model through sample data in actual use, so as to make the lesion identification and section analysis results more accurate.

[0112] In this embodiment, for a batch of new pathological slice images D new , using the small batch stochastic gradient descent method to achieve online learning, D new The sample consisting of the pathological slice image and its true label in is denoted as where x i is the i-th pathological slice image, y i is the true label corresponding to the i-th pathological section image, and n is the number of samples.

[0113] D new The pathological slice images in the image are input into the current pathological slice analysis model again for forward propagation to obtain the predicted labels corresponding to each pathological slice image. The cross entropy loss function is used to calculate the error between the predicted label and the true label, where the loss function calculation expression is as follows:

[0114]

[0115] Among them, c is the lesion category index, y i,c is the true label corresponding to the c-th lesion, is the predicted label result corresponding to the c-th type of lesion. The left side of the equation is the Loss value calculated by the loss function. This value needs to be reduced through back propagation and weight update to optimize the model.

[0116] After that, back propagation and model weight update are performed. This process uses the small batch stochastic gradient descent method to update the model weight. The weight update calculation expression is as follows:

[0117]

[0118] Among them, θ t is the model parameter before weight update, θ t+1 is the model parameter after one weight update, η is the learning rate, and the learning rate here should be a small value, such as 0.0001.

[0119] Set the training cycle to repeat the training 10 times, set the batch size to 1, and use this batch of data D new The current pathological section analysis model is trained to continuously optimize and update the model parameters, and eventually a pathological section analysis model fine-tuned with a single batch of data can be obtained, thus completing a round of model online learning process.

[0120] Furthermore, the method described in this embodiment also includes:

[0121] 7) Using the dynamic model fusion mechanism, the pathological section analysis model before online learning adjustment is weightedly fused with the adjusted pathological section analysis model, so as to retain the analysis ability of the existing model while learning new pathological section analysis knowledge, ensure the smooth transition of model update and avoid functional fluctuations.

[0122] Specifically, let the model before single batch data adjustment using online learning be the initial model, denoted by M0, and the parameters of the model be denoted by θ0; let the model after single batch data adjustment using online learning be the fine-tuned model, denoted by M1, and the parameters of the model be denoted by θ1.

[0123] Then, a weighted fusion strategy is used to fuse the models before and after adjustment. The specific fusion expression is as follows:

[0124] θ fusion =αθ0+(1-α)θ1

[0125] Among them, θ fusion is the parameter of the fused model, α is the fusion coefficient, and its value range is [0,1]. The value can be adjusted according to the model performance. fusion The model used as a parameter is recorded as the fused model M2. Therefore, this method can take into account the learning of new knowledge and the retention of existing knowledge in the online learning process of the pathological section analysis model, and achieve smooth updating of the model and stable improvement of performance.

[0126] 8) The fused pathological section analysis model obtained in step 7) is saved and used as a new current pathological section analysis model, and then the process returns to step 4).

[0127] During use of the pathological section analysis model in this embodiment, for newly acquired pathological section samples, steps 4)-8) are continuously repeated, and the current model is fine-tuned and trained with new sample data to obtain a fine-tuned model. The initial model and the fine-tuned model are fused through a model fusion strategy to form a fused model. The fused model is used to analyze and predict new sample data, and it is used as the initial model for the next round of online learning for iteration.

[0128] In this way, this embodiment enables the pathology section analysis model to be continuously iteratively optimized, thereby achieving better performance and better stability.

[0129] To implement the above embodiment, a second aspect of the present invention provides a pathological slice image analysis device based on online learning, comprising:

[0130] An offline training module, used for offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image;

[0131] An online learning module, used to perform online learning on the current pathological section analysis model using the newly acquired pathological section image to obtain an updated pathological section analysis model;

[0132] The analysis module is used to input the pathological slice image to be analyzed into the updated pathological slice analysis model, and finally obtain the pathological slice lesion segmentation result of the pathological slice image to be analyzed.

[0133] It should be noted that the above explanation of the embodiment of a pathological slice image analysis method based on online learning is also applicable to a pathological slice image analysis device based on online learning in this embodiment, and will not be repeated here. According to an embodiment of the present invention, a pathological slice image analysis device based on online learning is proposed, through offline training of a pathological slice analysis model, the input of the pathological slice analysis model is a pathological slice image, and the output is the pathological slice lesion segmentation result of the image; the current pathological slice analysis model is online learned using the newly acquired pathological slice image to obtain the updated pathological slice analysis model; the pathological slice image to be analyzed is input into the updated pathological slice analysis model, and finally the pathological slice lesion segmentation result of the pathological slice image to be analyzed is obtained. In this way, it is possible to fine-tune the model by continuously transferring new pathological slice images to the offline trained pathological slice analysis model in real time, so that the model can dynamically learn from the new image and optimize the model parameters, thereby improving the accuracy and efficiency of the analysis.

[0134] To implement the above embodiment, a third aspect of the present invention provides an electronic device, including:

[0135] at least one processor; and a memory communicatively coupled to the at least one processor;

[0136] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned pathological section image analysis method based on online learning.

[0137] To implement the above-mentioned embodiment, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned pathological section image analysis method based on online learning.

[0138] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0139] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes a pathological slice image analysis method based on online learning in the above embodiment.

[0140] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0141] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0142] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0143] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0144] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0145] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0146] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0148] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A pathological section image analysis method based on online learning, characterized in that: include: Offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image; Using the newly acquired pathological section image to perform online learning on the current pathological section analysis model to obtain an updated pathological section analysis model; The pathological slice image to be analyzed is input into the updated pathological slice analysis model, and finally a pathological slice lesion segmentation result of the pathological slice image to be analyzed is obtained.

2. The method according to claim 1, characterized in that The offline training pathological section analysis model comprises: Construct pathological slide image training set and validation set; Constructing a pathological section analysis model, wherein the pathological section analysis model is composed of a pre-trained model and an additionally added task head; Freeze the pre-trained model, train the task head using the training set and test it using the validation set to obtain the task head that has been preliminarily trained; The pre-trained model is unfrozen, the pathological section analysis model is fine-tuned using the training set and tested using the validation set to obtain the pathological section analysis model that has been finally trained offline.

3. The method according to claim 1, characterized in that The method of using the newly acquired pathological section image to perform online learning on the current pathological section analysis model to obtain the updated pathological section analysis model includes: Get a batch of new pathological slice images D new , D new The sample consisting of the pathological slice image and its true label in is denoted as where x i is the i-th pathological slice image, y i is the true label corresponding to the i-th pathological slice image, and n is the number of samples; new The pathological slice image in is input into the current pathological slice analysis model, and the output result of the model is the predicted label of the image. After the predicted label is corrected, the real label of the image is obtained; D new The pathological slice images in the image are input into the current pathological slice analysis model again for forward propagation to obtain the predicted label corresponding to each pathological slice image. The cross entropy loss function is used to calculate the error between the predicted label and the true label, where the loss function calculation expression is as follows: Among them, c is the lesion category index, y i,c is the true label corresponding to the c-th lesion, is the predicted label result corresponding to the c-th type of lesion; Back propagation and model weight update are performed based on the calculation results of the loss function. The small batch stochastic gradient descent method is used to update the model weight. The weight update calculation expression is as follows: Among them, θ t is the model parameter before weight update, θ t+1 is the model parameter after one weight update, η is the learning rate; When the online learning reaches the preset training parameters, the pathological section analysis model is obtained after fine-tuning the single batch data.

4. The method according to claim 2, characterized in that: The pathological section analysis model uses the pre-trained DinoV2 model as the main body, and adds a MaskFormer task head to the output layer of the DinoV2 model.

5. The method according to claim 2, characterized in that: The step of constructing a training set and a validation set of pathological slice images includes: 1) Acquire pathological slice images to form a pathological slice image dataset; The pathological slice image dataset includes pathological slice images corresponding to a plurality of pathological types, wherein each pathological slice image has a consistent size; 2) annotating each pathological slice image of the pathological slice image data set of step 1); wherein the annotation content of each pathological slice image includes a polygonal boundary of the lesion area and a corresponding lesion type label; 3) Preprocessing the images of the pathological slice image dataset annotated in step 2) to normalize the pixel value of each image to the range of [0, 1]; 4) The pathological section data set preprocessed in step 3) is randomly divided into a training set and a validation set according to a preset ratio.

6. The method according to claim 2, characterized in that Also includes: The performance of the pathological section analysis model that has been trained offline is evaluated using a preset test set, and the pathological section analysis model is adjusted according to the evaluation result to obtain an optimized pathological section analysis model; wherein the pathological section images in the test set are different from the pathological section images in the training set and the validation set.

7. The method according to claim 3, characterized in that Also includes: Using the dynamic model fusion mechanism, the pathological section analysis model before online learning adjustment is weightedly fused with the adjusted pathological section analysis model; Among them, the model before single batch data adjustment using online learning is taken as the initial model, denoted as M0, and the parameters of the model are denoted as θ0; the model after single batch data adjustment using online learning is taken as the fine-tuned model, denoted as M1, and the parameters of the model are denoted as θ1. The models before and after adjustment are integrated, and the expression is as follows: i fusion =αθ0+(1-α)θ1 Among them, θ fusion is the parameter of the fusion model, α is the fusion coefficient, and its value range is [0,1]; The fusion The pathological section analysis model used as a parameter is used as a new current pathological section analysis model.

8. A pathological slice image analysis device based on online learning, characterized in that: include: An offline training module, used for offline training of a pathological section analysis model, wherein the input of the pathological section analysis model is a pathological section image, and the output is a pathological section lesion segmentation result of the image; An online learning module, used to perform online learning on the current pathological section analysis model using the newly acquired pathological section image to obtain an updated pathological section analysis model; The analysis module is used to input the pathological slice image to be analyzed into the updated pathological slice analysis model, and finally obtain the pathological slice lesion segmentation result of the pathological slice image to be analyzed.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images