Sample size expansion method and apparatus, device, and storage medium

Abstract

Embodiments of the present application disclose a sample quantity expansion method and device, equipment and a storage medium, the method comprising: obtaining a digital pathology image comprising a first image region labeled with a positive sample label, a second image region labeled with a negative sample label, and a third image region without a label; performing image segmentation on the third image region to obtain a plurality of sub-images of the third image region; performing similarity clustering processing on image features using the first image region, the second image region, and each sub-image to determine a similarity pair for each sub-image; and determining a target sample label corresponding to each sub-image according to the similarity pair for each sub-image and a preset similarity coordinate system. The method realizes expansion of sample quantity from small data labeling to large data labeling of the digital pathology image, and compared with manual labeling, the method shortens labeling time and improves labeling efficiency.

Description

Technical Field

[0001] This invention relates to the field of pathological image processing technology, and in particular to a method, apparatus, device and storage medium for expanding sample size. Background Technology

[0002] Artificial intelligence (AI) technology has been widely applied in medical image research and application, achieving remarkable progress. Medical images are one of the important bases for clinicians to diagnose and treat diseases, and the application of AI technology can improve the accuracy and efficiency of doctors' diagnoses, while also potentially discovering new disease characteristics and treatment options.

[0003] Generating deep learning models for medical images requires training on a large amount of labeled data from doctors. The quantity and quality of the training dataset are crucial factors limiting the accuracy of deep learning algorithms. The process of generating deep learning models for medical images involves using computers to gradually learn features from the data. Therefore, in medical image recognition tasks, to enable computers to perform classification, detection, and segmentation tasks as accurately as possible, model training on a large dataset is necessary.

[0004] However, clear, complete, and high-quality medical image data is not easy to obtain. It requires professional knowledge and experience for annotation and review, making it relatively difficult to acquire a sufficient number of datasets. The quantity and quality of the training dataset have a significant impact on the accuracy of deep learning algorithm models. Insufficient sample data can lead to underfitting, preventing the model from capturing the details and patterns in the data; while excessive training data can cause the model to become overly complex and lose its generalization ability. Furthermore, the quality of the training dataset also affects the accuracy of the deep learning algorithm model. During data annotation, human factors (such as the professional level of the annotators and the formulation of annotation rules) can influence the annotation results; at the same time, inconsistencies in data samples and noise can also reduce data quality. To improve the accuracy of deep learning models, a significant investment of time, manpower, and resources is needed to collect, organize, annotate, and screen usable samples to obtain sufficient and high-quality training data. In addition, the acquisition of training data should maximize data diversity to avoid overfitting. Only in this way can the accuracy and generalization ability of deep learning models be ensured in medical image recognition tasks, providing practical assistance for clinical applications.

[0005] Currently, experienced doctors often lack the time for repetitive image annotation work. Younger doctors, who do have time, often have lower accuracy in image annotation compared to senior doctors. The amount of annotated data for pathological images remains limited by the individual doctor's energy and ability; obtaining a large number of annotated samples often consumes the annotating doctor's time, resulting in low efficiency. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for expanding the sample size, which can solve the problem that in the prior art, a large number of labeled samples often consumes the time of labeling doctors and is inefficient.

[0007] To achieve the above objectives, a first aspect of the present invention provides a method for expanding the sample size, the method comprising:

[0008] Acquire digital pathological images, which include a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels;

[0009] The third image region is segmented to obtain several sub-images of the third image region;

[0010] Image feature similarity clustering is performed using the first image region, the second image region, and each of the sub-images to determine the similarity pairs of each sub-image. The similarity pairs are used to reflect the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region.

[0011] Based on the similarity pairs of each sub-image and a preset similarity coordinate system, the target sample label corresponding to the sub-image is determined. The similarity coordinate system is used to indicate the sample label corresponding to the similarity pairs. The target sample label includes at least a positive sample label or a negative sample label.

[0012] In one feasible implementation, the coordinate axes of the similarity coordinate system are composed of a first similarity and a second similarity, wherein the first similarity and the second similarity are inversely proportional. Then, determining the target sample label corresponding to the sub-image based on the similarity pairs of each sub-image and the preset similarity coordinate system includes:

[0013] Using the first similarity and the second similarity of each of the sub-images, at least two similarity thresholds are determined, the similarity thresholds including at least a minimum similarity threshold and a maximum similarity threshold;

[0014] Based on the minimum similarity threshold and the maximum similarity threshold, the similarity coordinate system is divided horizontally and vertically to determine the various distribution regions after division. The distribution regions include at least the upper left distribution region and the lower right distribution region. The upper left distribution region is used to reflect the distribution region of the first similarity pair of the first image region in the similarity coordinate system, and the lower right distribution region is used to reflect the distribution region of the second similarity pair of the second image region in the similarity coordinate system.

[0015] The first similarity and second similarity of each of the sub-images are input into a preset similarity coordinate system to determine the target distribution area of ​​each of the sub-images in the similarity coordinate system;

[0016] When the target distribution area is the upper left corner distribution area, the target sample label of the sub-image is determined to be a positive sample label;

[0017] When the target distribution area is the lower right corner distribution area, the target sample label of the sub-image is determined to be a negative sample label.

[0018] In one feasible implementation, the image features include at least brightness values. Then, the step of performing similarity clustering processing on the image features using the first image region, the second image region, and each of the sub-images to determine the similarity pairs for each sub-image includes:

[0019] Determine the first brightness value of the first image region and the second brightness value of the second image region;

[0020] Calculate the target brightness value for each of the sub-images;

[0021] A first similarity is determined by comparing the target brightness value with the first brightness value; a second similarity is determined by comparing the target brightness value with the second brightness value.

[0022] In one feasible implementation, the positive sample label includes a tumor label, the negative sample label includes a non-tumor label, and the image features include at least cell nuclear density. Then, the step of performing similarity clustering of image features using the first image region, the second image region, and each of the sub-images to determine the similarity pairs for each sub-image includes:

[0023] Determine the first cell nucleus density of the first image region and the second cell nucleus density of the second image region;

[0024] Calculate the target cell nucleus density of each of the sub-images;

[0025] A first similarity is determined by comparing the target cell nuclear density with the first cell nuclear density; a second similarity is determined by comparing the target cell nuclear density with the second cell nuclear density.

[0026] In one feasible implementation, the image features include at least the morphological features of the cell nucleus, which reflect the shape and area of ​​the cell nucleus. Then, the step of performing similarity clustering processing on the image features using the first image region, the second image region, and each of the sub-images to determine the similarity pairs of each sub-image includes:

[0027] Determine the first morphological features of the first image region and the second morphological features of the second image region;

[0028] Calculate the target morphological features of each of the sub-images;

[0029] A first similarity is determined by comparing the target morphological features with the first morphological features; a second similarity is determined by comparing the target morphological features with the second morphological features.

[0030] In one feasible implementation, the image features include at least the number of cell nucleus division images. Then, the step of clustering the first image region, the second image region, and each of the sub-images to determine the similarity pairs of each sub-image includes:

[0031] Determine the number of first split images in the first image region and the number of second split images in the second image region;

[0032] Calculate the number of target split images in each of the sub-images;

[0033] A first similarity is determined by comparing the number of target split images with the number of first split images; a second similarity is determined by comparing the number of target split images with the number of second split images.

[0034] In one feasible implementation, segmenting the third image region to obtain several sub-images of the third image region includes:

[0035] The target pixel size is determined using the first image region and the second image region;

[0036] The third image region is divided into equal segments according to the target pixel size to obtain several sub-images with the same pixel size as the target pixel size.

[0037] To achieve the above objectives, a second aspect of the present invention provides a sample size expansion device, the device comprising:

[0038] Image acquisition module: used to acquire digital pathological images, the digital pathological images including a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels;

[0039] Image segmentation module: used to segment the third image region to obtain several sub-images of the third image region;

[0040] Similarity clustering module: used to perform similarity clustering processing of image features using the first image region, the second image region and each of the sub-images, to determine the similarity pair of each sub-image, the similarity pair being used to reflect the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region;

[0041] Label determination module: used to determine the target sample label corresponding to the sub-image based on the similarity pairs of each sub-image and a preset similarity coordinate system. The similarity coordinate system is used to indicate the sample label corresponding to the similarity pairs. The target sample label includes at least a positive sample label or a negative sample label.

[0042] To achieve the above objectives, a third aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0043] To achieve the above objectives, a fourth aspect of the present invention provides a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps shown in the first aspect and any feasible implementation.

[0044] The embodiments of the present invention have the following beneficial effects:

[0045] This invention provides a method for expanding the sample size. The method includes: acquiring a digital pathological image, the digital pathological image including a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels; segmenting the third image region to obtain several sub-images of the third image region; performing image feature similarity clustering processing using the first image region, the second image region, and each sub-image to determine the similarity pair of each sub-image, the similarity pair reflecting the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region; determining the target sample label corresponding to each sub-image based on the similarity pair of each sub-image and a preset similarity coordinate system, the similarity coordinate system indicating the sample label corresponding to the similarity pair, the target sample label including a positive sample label or a negative sample label.

[0046] Using the above method, unlabeled third-image regions in digital pathology images can be segmented into multiple sub-images. Then, similarity clustering of image features is performed using labeled image regions and sub-images. Combining first and second similarity scores, the target sample label corresponding to the sub-image is determined within a pre-defined similarity coordinate system that indicates the sample label corresponding to the similarity score pair. This achieves a significant increase in sample size for digital pathology image annotation, expanding from small-scale to large-scale annotation, while shortening annotation time and improving annotation efficiency compared to manual annotation. Furthermore, since the images are from the same digital pathology image, they share similar imaging environment characteristics such as belonging to the same patient, using the same staining method, and exhibiting the same type of pathological features, reducing interference in similarity judgment and improving judgment efficiency and accuracy. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] in:

[0049] Figure 1 This is a flowchart of a method for expanding the sample size in an embodiment of the present invention;

[0050] Figure 2(a) is a schematic diagram of a digital pathological image in an embodiment of the present invention;

[0051] Figure 2(b) is another schematic diagram of a digital pathological image in an embodiment of the present invention;

[0052] Figure 3 This is another flowchart of a method for expanding the sample size in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of the segmentation of a third image region in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of a similarity coordinate system in an embodiment of the present invention;

[0055] Figure 6 This is a structural block diagram of a sample size expansion device according to an embodiment of the present invention;

[0056] Figure 7 This is a structural block diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figure 1 , Figure 1 This is a flowchart of a method for expanding the sample size according to an embodiment of the present invention, such as... Figure 1 The method shown can be applied to both terminals and servers. The terminal can be a desktop terminal or a mobile terminal; a mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server can be a standalone server or a server cluster consisting of multiple servers. This embodiment uses a terminal application as an example. Figure 1 The method shown includes the following steps:

[0059] 101. Acquire digital pathological images, wherein the digital pathological images include a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels;

[0060] It should be noted that the sample size expansion method shown in this application is used for expanding the sample size of digital pathological images. Specifically, the sample size refers to the number of image samples, which are digital pathological images with known pathological information. These image samples are specifically divided into positive samples and negative samples. These image samples can serve as training data for deep learning models, learning data for artificial intelligence, and analysis data for big data analysis. Furthermore, to distinguish different types of image samples, different identity labels need to be assigned to them. These identity labels are called sample labels, which are further divided into positive sample labels and negative sample labels based on the distinction between positive and negative samples. For example, a positive sample label is a tumor label, and a negative sample label is a non-tumor label.

[0061] Furthermore, in order to expand the sample size of digital pathological images, it is necessary to obtain digital pathological images, that is, to obtain digital pathological images through step 101. The number of digital pathological images is one, that is, the sample size expansion method of this application is to obtain several image samples through one digital pathological image. The digital pathological image includes a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels.

[0062] Please see Figures 2(a) to 2(b)Figure 2(a) is a schematic diagram of a digital pathological image in an embodiment of the present invention, and Figure 2(b) is another schematic diagram of a digital pathological image in an embodiment of the present invention. For example, the scanned original digital pathology image shown in Figure 2(a) is displayed on the screen of a computer or other terminal for experienced annotators to perform preliminary annotations, marking tumor tissue components and non-tumor tissue components. The preliminary annotation is incomplete, meaning there will be image areas that should be marked but are not. Therefore, the annotated original digital pathology image is shown in Figure 2(b). The digital pathology image shown in Figure 2(b) is the digital pathology image to be obtained in step 101 above. Figure 2(b) includes three first image regions: first image region 201, first image region 202, and first image region 203. Figure 2 also includes three second image regions: second image region 204, second image region 205, and second image region 206. In Figure 2(b), the solid thick rectangle is used to mark tumor tissue components, while the dashed thick rectangle is used to mark non-tumor tissue components. The image regions of the digital pathology image in Figure 2(b) other than the first and second image regions mentioned above all belong to the third image regions. In this figure, the first image region 201 and its positive sample label can be considered as one image sample, the first image region 202 and its positive sample label can be considered as another image sample, and so on. In Figure 2(b), the number of image samples is 6. It is understood that Figure 2(b) uses a small number of annotations as an example, showing 6 rectangular box annotations. Depending on the actual needs, there can be more than 6 annotations or fewer than 3 annotations. This is just an example and no specific limitation is made.

[0063] 102. Perform image segmentation on the third image region to obtain several sub-images of the third image region;

[0064] 103. Perform image feature similarity clustering processing using the first image region, the second image region, and each of the sub-images to determine the similarity pair of each sub-image. The similarity pair is used to reflect the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region.

[0065] Furthermore, having obtained digital pathology images with a small number of annotations, we obtain image samples consisting of the first image region and its corresponding positive sample label, and image samples consisting of the second image region and its corresponding negative sample label, thus obtaining a small number of image samples. This small number of image samples can then be used to expand the sample size. To achieve this expansion, the unlabeled third image region in a full panoramic digital pathology image can be segmented to obtain several sub-images of the third image region, for example, by segmenting the third image region into two or more images. Then, using the small number of image samples, sample labels are assigned to the sub-images, thereby obtaining more image samples and expanding the sample size. Specifically, in step 102, image feature similarity clustering is performed using the first image region, the second image region, and the sub-images. That is, the similarity between the image features of the sub-images and the first image region is used to determine whether they belong to the same class of images with similar image features, and the similarity between the image features of the sub-images and the second image region is used to determine whether they belong to the same class of images with similar image features. For example, image features include, but are not limited to, image brightness values, cell nuclear density, cell nuclear morphology features, number of cell nuclear division images, etc. Similarity determination can be achieved by calculating the similarity between the image features of the first image region and the second image region and the sub-images, respectively. Similarity clustering of image features is performed on the first image region, the second image region, and each of the sub-images to determine the similarity pairs for each sub-image. Each similarity pair includes at least a first similarity and a second similarity. The first similarity reflects the degree of similarity between the sub-image and the image features of the first image region, and the second similarity reflects the degree of similarity between the sub-image and the image features of the second image region. The similarity value can range from [0, 1]. The more similar the image features, the closer the similarity is to 1; the less similar the image features, the closer the similarity is to 0.

[0066] 104. Based on the similarity pairs of each sub-image and a preset similarity coordinate system, determine the target sample label corresponding to the sub-image. The similarity coordinate system is used to indicate the sample label corresponding to the similarity pairs. The target sample label includes a positive sample label or a negative sample label.

[0067] It should be noted that after obtaining the similarity pairs of sub-images, the target sample label of the sub-image can be determined as a positive or negative sample label by the first similarity between the sub-image and the first image region and the second similarity between the sub-image and the second image region. Specifically, the target sample label corresponding to the sub-image is determined according to the similarity pairs of each sub-image and the preset similarity coordinate system. The similarity coordinate system is used to indicate the sample label corresponding to the similarity pairs, and the target sample label includes a positive or negative sample label. In this similarity coordinate system, the axes are composed of similarity values. For example, the X-axis can be the range of values ​​for the first similarity, and the Y-axis can be the range of values ​​for the second similarity, or the Y-axis can be the range of values ​​for the first similarity, and the X-axis can be the range of values ​​for the second similarity. Furthermore, the coordinate pairs in the similarity coordinate system include both the first and second similarities. Different coordinate positions in the similarity coordinate system correspond to different sample labels. By inputting the similarity pairs into the similarity coordinate system, we can obtain which coordinate pair corresponds to the similarity of the sub-image, and thus obtain the target sample label of the sub-image, thereby expanding the sample size.

[0068] This invention provides a method for expanding the sample size. It segments unlabeled third-image regions in digital pathology images into multiple sub-images, and uses labeled image regions and sub-images to perform image feature similarity clustering. Combining first and second similarity scores, the target sample label corresponding to each sub-image is determined within a preset similarity coordinate system that indicates the sample label corresponding to the similarity score pair. This expands the sample size of digital pathology images from small-scale to large-scale annotation, shortening annotation time and improving annotation efficiency compared to manual annotation. Furthermore, since the images are from the same digital pathology image, they share similar imaging environment characteristics such as belonging to the same patient, using the same staining method, and exhibiting the same type of pathological features, reducing interference in similarity determination and improving judgment efficiency and accuracy.

[0069] Please see Figure 3 , Figure 3 This is another flowchart of a method for expanding the sample size according to an embodiment of the present invention, as shown below. Figure 3 The method shown includes the following steps:

[0070] 301. Acquire a digital pathological image, wherein the digital pathological image includes a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels;

[0071] 302. Perform image segmentation on the third image region to obtain several sub-images of the third image region;

[0072] It should be noted that steps 301 to 302 are related to... Figure 1 Steps 101 to 102 shown are similar and will not be repeated here to avoid repetition. Please refer to [link to relevant documentation] for details. Figure 1 The contents of steps 101 to 102 shown.

[0073] In one feasible implementation, to further improve the accuracy of similarity judgment, the third image region can be segmented according to the pixel size of the first image region and the second image region. Therefore, step 302 above may include steps A1 to A2:

[0074] A1. Determine the target pixel size using the first image region and the second image region;

[0075] A2. The third image region is divided into equal segments according to the target pixel size to obtain several sub-images with the same pixel size as the target pixel size.

[0076] It should be noted that segmentation of other unlabeled third image regions on the digital pathology image specifically involves using the first and second image regions to determine the target pixel size. This target pixel size reflects the pixel size of the first and second image regions. For example, the target pixel size can be... Figure 4 The solid-lined bounding box corresponds to the first image region, and the dashed-lined bounding box corresponds to the second image region. The dimensions of the dashed and solid-lined bounding boxes are the same, therefore their corresponding pixel dimensions are also the same. Thus, the final target pixel size can be either the first or second image region. Then, based on the target pixel size, the image is divided into equal-sized sub-images at equal intervals from the top left to the bottom right. The segmentation results can be found in [reference needed]. Figure 4 , Figure 4 This is a schematic diagram of the segmentation of a third image region in an embodiment of the present invention. The image blocks can also be numbered, with the vertical image blocks numbered sequentially from 1 to n. The horizontal image blocks are numbered sequentially from 1 to m. Each image block is named T(1,1), T(2,1), T(3,1)...T(m,n) according to its coordinate position in the panoramic digital pathology image.

[0077] 303. Perform image feature similarity clustering processing using the first image region, the second image region, and each of the sub-images to determine the similarity pair of each sub-image. The similarity pair is used to reflect the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region.

[0078] It should be noted that step 303 and Figure 1 The content of step 103 is similar, and will not be repeated here to avoid repetition. Please refer to [link to relevant documentation] for details. Figure 1 The content of step 103.

[0079] In one feasible implementation, the image features include at least brightness value, cell nuclear density, cell nuclear morphology features, and the number of cell nuclear mitotic figures. Furthermore, similarity can be determined using one or more of these features. Therefore, step 303 may include steps U1 to U3 below, using brightness values ​​to determine similarity and obtain similarity pairs:

[0080] U1. Determine the first brightness value of the first image region and the second brightness value of the second image region;

[0081] U2. Calculate the target brightness value of each of the sub-images;

[0082] U3. Based on the target brightness value and the first brightness value, a similarity judgment is made to determine the first similarity; based on the target brightness value and the second brightness value, a similarity judgment is made to determine the second similarity.

[0083] It should be noted that the positive sample label can be a tumor label, in which case the first image region can be a tumor region. The overall color of a tumor region is more bluish, which is reflected in the RGB values ​​of the color channels. The RGB values ​​of a tumor region might be (R12; G 56; B 240). For the same area, the B color channel has a higher parameter weight, while the R color channel has a lower parameter weight. Conversely, the negative sample label can be a non-tumor label, in which case the second image region can be a non-tumor region. The overall color of a non-tumor region is more reddish and white, and the RGB values ​​of a non-tumor region might be (R 210; G 26; B 24). For the same area, the B color channel has a lower parameter weight, while the R color channel has a higher parameter weight.

[0084] Then, a first brightness value for the first image region and a second brightness value for the second image region can be determined. The first brightness value includes the parameters of the B color channel and the R color channel of the first image region, and the second brightness value includes the parameters of the B color channel and the R color channel of the second image region. Further, a target brightness value for the image region of a single sub-image is calculated, which includes the parameters of the B color channel and the R color channel of the sub-image. Then, the similarity of the brightness values ​​is calculated respectively, resulting in similarity pairs.

[0085] To further improve the accuracy of similarity, the similarity between two images can be determined by the difference between the R and B color channels. For example, the target difference between the R and B parameters of the target brightness value can be calculated, as can the first difference between the R and B parameters of the first brightness value and the second difference between the R and B parameters of the second brightness value. The similarity between the target difference and the first and second differences is then calculated, yielding a first similarity P and a second similarity Q. It should be noted that, as shown in the example above, in non-tumor regions with the same area, the B color channel parameter weight is lower and the R color channel parameter weight is higher. In this case, the difference during RB may be positive. Conversely, in tumor regions with the same area, the B color channel parameter weight is higher and the R color channel parameter weight is lower. In this case, the difference during RB may be negative. Therefore, the difference can be used as a basis for similarity judgment to calculate the similarity.

[0086] In one feasible implementation, step 303 may further include the following steps F1 to F3, using cell nuclear density to determine similarity:

[0087] F1. Determine the first cell nucleus density of the first image region and the second cell nucleus density of the second image region;

[0088] F2. Calculate the target cell nucleus density of each of the sub-images;

[0089] F3. Based on the target cell nuclear density and the first cell nuclear density, a similarity judgment is made to determine the first similarity; based on the target cell nuclear density and the second cell nuclear density, a similarity judgment is made to determine the second similarity.

[0090] It should be noted that the cell density values ​​are higher in tumor regions, and the density of blue circular cell nuclei is higher within a single region. Conversely, the cell density values ​​are lower in non-tumor regions, and the density of blue circular cell nuclei is even lower within a single region. Therefore, the cell nucleus density values ​​of the first image region, the second image region, and the sub-images can be calculated, and similarity pairs can be obtained by calculating the cell nucleus density values. Specifically, the cell nucleus boundaries in the image can be extracted using a preset morphological algorithm to calculate the density.

[0091] In one feasible implementation, the image features include at least the morphological features of the cell nucleus, which reflect the shape and area of ​​the cell nucleus. Then, step 303 may also include the following steps O1 to O3, using the morphological features of the cell nucleus to perform similarity judgment:

[0092] O1. Determine the first morphological features of the first image region and the second morphological features of the second image region;

[0093] O2. Calculate the target morphological features of each of the sub-images;

[0094] O3. Based on the target morphological features and the first morphological features, a similarity judgment is made to determine the first similarity; based on the target morphological features and the second morphological features, a similarity judgment is made to determine the second similarity.

[0095] It should be noted that the cell nuclei in tumor regions are larger and more irregular in shape (potato-shaped, sweet potato-shaped); while the cell nuclei in non-tumor regions are smaller and more regular in shape (round, elongated spindle-shaped). Therefore, for unknown labeled regions, the morphological features of individual regions can be calculated to obtain the morphological features of the cell nuclei, such as area and shape, of the sub-image. Then, the first morphological features of the first image region and the second morphological features of the second image region are used to calculate the similarity with the target morphological features of the sub-image, resulting in the first and second similarity scores of the sub-image.

[0096] In one feasible implementation, if the image features include at least the number of cell nuclei splitting images, then step 303 may further include steps J01 to J03:

[0097] J01. Determine the number of first split images in the first image region and the number of second split images in the second image region;

[0098] J02. Calculate the number of target split images in each of the sub-images;

[0099] J03. Based on the number of target split images and the number of first split images, a similarity judgment is made to determine the first similarity; based on the number of target split images and the number of second split images, a similarity judgment is made to determine the second similarity.

[0100] It should be noted that tumor regions show a higher number of mitotic figures, while non-tumor regions show fewer. For unknown regions (yellow boxes), the number of mitotic figures within a single region is calculated. Therefore, the number of first mitotic figures in the first image region and the number of second mitotic figures in the second image region can be determined. The target number of mitotic figures in each of the sub-images is then calculated, i.e., the number of mitotic figures in these three image regions is calculated. A similarity judgment is made based on the target number of mitotic figures and the number of first mitotic figures to determine the first similarity; a similarity judgment is also made based on the target number of mitotic figures and the number of second mitotic figures to determine the second similarity, thus obtaining the similarity data.

[0101] 304. Using the first similarity and the second similarity of each of the sub-images, determine at least two similarity thresholds, wherein the similarity thresholds include at least a minimum similarity threshold and a maximum similarity threshold;

[0102] 305. Based on the minimum similarity threshold and the maximum similarity threshold, the similarity coordinate system is divided horizontally and vertically to determine the various distribution regions after division. The distribution regions include at least the upper left distribution region and the lower right distribution region. The upper left distribution region is used to reflect the distribution region of the first similarity pair of the first image region in the similarity coordinate system, and the lower right distribution region is used to reflect the distribution region of the second similarity pair of the second image region in the similarity coordinate system.

[0103] It should be noted that the coordinate axes of the similarity coordinate system are composed of similarity values. The Y-axis can be composed of the range of values ​​for the first similarity, and the X-axis can be composed of the range of values ​​for the second similarity. Alternatively, the X-axis can be composed of the range of values ​​for the first similarity, and the Y-axis can be composed of the range of values ​​for the second similarity. The first similarity and the second similarity are inversely proportional.

[0104] After obtaining the first similarity and second similarity of each of the sub-images, at least two similarity thresholds can be determined using these thresholds. These similarity thresholds include at least a minimum similarity threshold and a maximum similarity threshold, which are used to divide the similarity coordinate system. Further, based on the minimum and maximum similarity thresholds, the similarity coordinate system is divided horizontally and vertically to determine the resulting distribution regions.

[0105] For example, you can refer to Figure 5 , Figure 5 This is a schematic diagram of a similarity coordinate system in an embodiment of the present invention, as shown below. Figure 5 The Y-axis of the similarity coordinate system shown is composed of the value range [0,1] of the first similarity, and the X-axis is composed of the value range [0,1] of the second similarity. In the figure, the minimum similarity threshold is 0.2 and the maximum similarity threshold is 0.8. The vertical dividing line includes X=0.2 and X=0.8, and the horizontal dividing line includes Y=0.2 and Y=0.8. Thus, there are 9 distribution regions, including the upper left distribution region where the first similarity is greater than 0.8 and the second similarity is less than 0.2, the lower right distribution region where the first similarity is less than 0.8 and the second similarity is greater than 0.2, and other regions. It should be noted that if an image is more similar to a tumor image, it is less similar to a non-tumor image. In this case, the first similarity score will be much higher than the second similarity score. Therefore, the distribution area in the upper left corner is used to reflect the distribution area of ​​the first similarity score pairs of the first image region in the similarity coordinate system. Similarly, if an image is less similar to a tumor image, it is more similar to a non-tumor image. In this case, the first similarity score will be much lower than the second similarity score. Therefore, the distribution area in the lower right corner is used to reflect the distribution area of ​​the second similarity score pairs of the second image region in the similarity coordinate system.

[0106] 306. Input the first similarity and second similarity of each of the sub-images into a preset similarity coordinate system to determine the target distribution area of ​​each of the sub-images in the similarity coordinate system;

[0107] 307. When the target distribution area is the upper left corner distribution area, the target sample label of the sub-image is determined to be a positive sample label;

[0108] 308. When the target distribution area is the lower right corner distribution area, the target sample label of the sub-image is determined to be a negative sample label.

[0109] Furthermore, the target distribution area of ​​each sub-image in the similarity coordinate system can be determined by the similarity pairs of the sub-images. If the first similarity and second similarity of each sub-image are input into the preset similarity coordinate system, and its target distribution area falls in the upper left distribution area 501, it means that it is more similar to the first image area. Therefore, its sample label is determined to be a positive sample label, that is, a tumor label. Similarly, if its target distribution area falls in the lower right distribution area 502, it means that it is more similar to the second image area. Therefore, its sample label is determined to be a negative sample label, that is, a non-tumor label.

[0110] For example, image patches, or sub-images, use labeled tumor and non-tumor image patches from the same image as the initial training set. Unlabeled image patches are then subjected to similarity clustering analysis using a deep learning model. Unlabeled image patches yield two probability values: the similarity P-value with labeled tumor image patches and the similarity Q-value with labeled non-tumor image patches. In Figure 2(b), the P-value and Q-value of the T(1,1) image patch are both probability values ​​between 0 and 1. A vertical coordinate axis is established between the P-values ​​and Q-values. Labeled tumor image patches have P-values ​​close to 1 and Q-values ​​close to 0, located in the upper left corner of the coordinate axis. Labeled non-tumor image patches have P-values ​​close to 0 and Q-values ​​close to 1, located in the lower right corner of the coordinate axis. The P-values ​​and Q-values ​​of the remaining unlabeled image patches are distributed at different coordinate positions. Some of these are close to the upper left corner, indicating a higher probability that these image patches are tumor image patches. A portion of these images are located near the bottom right corner, indicating that the probability of these images being tumor patches is relatively low. A small portion is randomly distributed in other locations, suggesting that these images cannot be accurately classified.

[0111] To further improve the accuracy of the expanded results, senior physicians randomly sampled unlabeled image patches in the upper left corner to verify the accuracy of the cluster analysis results. After confirmation, the unlabeled image patches in the upper left corner were labeled as tumors, forming new labeled data. Thus, x labeled data points were generated from a small number of 3 labeled image patches in a short time. Similarly, the unlabeled image patches in the lower right corner were labeled as non-tumor, forming new labeled data. y labeled data points were generated from a small number of 3 labeled image patches in a short time. This application uses thresholds of 0.8 and 0.2 as examples. In most cases, tumor and non-tumor areas in the pathological images of each patient show significant differences in color and morphology. Based on the typically labeled tumor areas and the typically labeled non-tumor areas (manually labeled), clustering phenomena can be observed in the upper left and lower right corners. In each pathological image, when the proportion of tumor components is high, there are more image patches in the upper left corner; conversely, there are more image patches in the lower right corner. In rare cases, tumor and non-tumor areas may show little difference in color but exhibit morphological differences. The data distribution may be concentrated randomly in the middle. The reliability of cluster analysis can be improved by expanding the amount of labeled image patches provided by doctors.

[0112] Alternatively, for sub-images where the target distribution area falls outside the distribution areas of 501 and 503, such as... Figure 5 If a sub-image falls into 503, it means that the sample label of the sub-image has not yet been determined. Therefore, the image region corresponding to the sub-image whose sample label has been determined can be updated to the first image region or the second image region according to the label type. Then, return to step 301 to use the expanded sample and the manually labeled sample to determine the label of the sub-image that falls outside the distribution area of ​​501 and 503, so as to further expand the sample size.

[0113] Furthermore, during two-level cluster analysis, senior doctors can determine the upper and lower thresholds (minimum threshold and maximum threshold). Figure 5 Let's take 0.8 and 0.2 as examples. The upper and lower thresholds can be redefined based on the actual clustering results, such as 0.9 and 0.2, 0.95 and 0.1, etc. A smaller area in the upper left corner results in higher accuracy, but may generate less new labeled data. Doctors should confirm that for each pathology image, the amount of data should be increased as much as possible while ensuring high accuracy (e.g., greater than 95%). For newly generated labeled data, new labeled data can be generated after doctor confirmation. Generate x labeled data from the original 3 labeled data. This can be used as the parent data for a new clustering analysis, performing another two-layer clustering analysis. Repeat steps 303 to 308.

[0114] In one feasible implementation, for unknown regions, the RGB values ​​of individual regions are calculated, especially the changes in the R and B parameters. If the R parameter is >128 and the B parameter is <128, the region is more likely to be classified as a non-tumor region. If the B parameter is >128 and the R parameter is <128, it is classified as a tumor region. The threshold parameters can be adjusted based on the actual single panoramic pathological image. For example, if the R parameter is >180 and the B parameter is <60, the region is more likely to be classified as a non-tumor region. If the B parameter is >180 and the R parameter is <60, the region is classified as a tumor region and assigned a corresponding label. The latter has higher image similarity but may have less image data. Many image patches not falling into the above categories may not be classified into the "sub-label dataset," and the label cannot be determined. The method of this application is then used to determine the label. The threshold setting and the number of newly classified "sub-label datasets" are closely related. The stricter the threshold, the fewer new sub-label data are generated. Sub-label data includes sub-images and sample labels. Controlling this threshold requires confirmation by humans (experts) in actual image annotation work and subsequent verification. Each panoramic pathology image may differ significantly from others in terms of staining quality, pathological type, morphological characteristics, and other aspects. Therefore, this application aims to develop a method more suitable for experts to complete a large number of labeled images in a short time with high quality.

[0115] This invention provides a method for expanding the sample size by creating a small dataset of N small local pathological image blocks from a single panoramic pathological image. These blocks share characteristics such as belonging to the same patient, using the same staining method, and belonging to the same disease category. The small local pathological image blocks exhibit similar image classifications within a few categories. This facilitates the expansion from small-scale annotation to large-scale annotation on a single panoramic pathological image. The invention offers the following advantages: 1) Experienced physicians can perform medical image annotation in a short time. Using the method described in this application, a large number of image blocks can be automatically generated by computer in a short time, significantly improving the efficiency and quality of annotation. 2) Annotation work is transformed from a large amount of repetitive and monotonous work into a simple and fast process, avoiding the physiological effects of fatigue and boredom on the annotation physician. Simultaneously, annotation quality can be improved. 3) Traditional manual annotation methods present a trade-off between efficiency and quality; improving efficiency often leads to a decrease in quality. The method described in this application significantly improves annotation efficiency while maintaining the same annotation quality. 4) Compared to manual annotation, annotation of the same quality and quantity can significantly reduce both time and economic costs. 5) When errors occur due to human error or systemic data labeling, the labeled dataset can be corrected and reconstructed more quickly, improving the efficiency of annotation and correction.

[0116] Please see Figure 6, Figure 6 This is a structural block diagram of a sample size expansion device according to an embodiment of the present invention, such as... Figure 6 The apparatus shown includes:

[0117] Image acquisition module 601: used to acquire digital pathological images, the digital pathological images including a first image region labeled with positive sample labels, a second image region labeled with negative sample labels, and a third image region without labels;

[0118] Image segmentation module 602: used to segment the third image region to obtain several sub-images of the third image region;

[0119] Similarity clustering module 603: used to perform similarity clustering processing of image features using the first image region, the second image region and each of the sub-images, to determine the similarity pair of each sub-image, the similarity pair being used to reflect the first similarity between the sub-image and the first image region, and the second similarity between the sub-image and the second image region;

[0120] Label determination module 604: used to determine the target sample label corresponding to the sub-image based on the similarity pairs of each sub-image and a preset similarity coordinate system, wherein the similarity coordinate system is used to indicate the sample label corresponding to the similarity pairs, and the target sample label includes a positive sample label or a negative sample label.

[0121] It should be noted that, Figure 6 The contents of each module in the device shown are... Figure 1 The steps in the method shown are similar, and will not be repeated here to avoid repetition. Please refer to [reference needed]. Figure 1 The content of each step in the method shown.

[0122] This invention provides a device for expanding the sample size. It can segment unlabeled third image regions in digital pathology images into multiple sub-images, and perform image feature similarity clustering using labeled image regions and sub-images. Combining first and second similarity scores, the device determines the target sample label corresponding to each sub-image within a preset similarity coordinate system that indicates the sample label corresponding to the similarity score pair. This expands the sample size of digital pathology images from small-scale to large-scale annotation, shortening annotation time and improving annotation efficiency compared to manual annotation. Furthermore, since the images are from the same digital pathology image, they share similar imaging environment characteristics such as belonging to the same patient, using the same staining method, and exhibiting the same type of pathological features, reducing interference in similarity determination and improving judgment efficiency and accuracy.

[0123] Figure 7An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 7 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. The internal memory may also store a computer program, which, when executed by the processor, causes the processor to perform the aforementioned methods. Those skilled in the art will understand that… Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0124] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform actions such as... Figure 1 or Figure 3 The steps are shown.

[0125] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the following actions: Figure 1 or Figure 3 The steps are shown.

[0126] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method of augmenting a sample size, comprising: The method comprises: acquiring a digital pathology image, the digital pathology image comprising a first image region labeled with a positive sample label, a second image region labeled with a negative sample label, and a third image region without a label; performing image segmentation on the third image region to obtain a plurality of sub-images of the third image region; performing similarity clustering processing on image features of the first image region, the second image region, and each of the sub-images to determine a similarity pair for each of the sub-images, the similarity pair reflecting a first similarity between the sub-image and the first image region, and a second similarity between the sub-image and the second image region; determining a target sample label corresponding to each of the sub-images according to the similarity pair of each of the sub-images and a preset similarity coordinate system, the similarity coordinate system being used to indicate a sample label corresponding to the similarity pair, and the target sample label comprising at least a positive sample label or a negative sample label; wherein the coordinate axes of the similarity coordinate system are composed of the first similarity and the second similarity, the first similarity and the second similarity are inversely proportional, and the determination of the target sample label corresponding to each of the sub-images according to the similarity pair of each of the sub-images and the preset similarity coordinate system comprises: determining at least two similarity thresholds using the first similarity and the second similarity of each of the sub-images, the similarity thresholds comprising at least a minimum similarity threshold and a maximum similarity threshold; dividing the similarity coordinate system horizontally and vertically according to the minimum similarity threshold and the maximum similarity threshold to determine each of the divided distribution regions, the distribution regions comprising at least a top-left corner distribution region and a bottom-right corner distribution region, the top-left corner distribution region being used to reflect the distribution of the first similarity pair of the first image region in the similarity coordinate system, and the bottom-right corner distribution region being used to reflect the distribution of the second similarity pair of the second image region in the similarity coordinate system; inputting the first similarity and the second similarity of each of the sub-images into the preset similarity coordinate system to determine a target distribution region of each of the sub-images in the similarity coordinate system; when the target distribution region is the top-left corner distribution region, determining that the target sample label of the sub-image is the positive sample label; when the target distribution region is the bottom-right corner distribution region, determining that the target sample label of the sub-image is the negative sample label.

2. The method of claim 1, wherein, The image features comprise at least luminance values, and the determination of the similarity pair for each of the sub-images by performing similarity clustering processing on image features of the first image region, the second image region, and each of the sub-images comprises: determining a first luminance value of the first image region and a second luminance value of the second image region; calculating a target luminance value of each of the sub-images; determining the first similarity according to the target luminance value and the first luminance value, and determining the second similarity according to the target luminance value and the second luminance value.

3. The method of claim 1, wherein, The positive sample label comprises a tumor label, the negative sample label comprises a non-tumor label, and the image feature at least comprises a cell nucleus density, and the similarity clustering processing of the image features is performed on the first image area, the second image area and each sub-image to determine a similarity number pair of each sub-image, comprising: determining a first cell nucleus density of the first image area and a second cell nucleus density of the second image area; calculating a target cell nucleus density of each sub-image; determining a first similarity according to the target cell nucleus density and the first cell nucleus density, and determining a second similarity according to the target cell nucleus density and the second cell nucleus density.

4. The method of claim 1, wherein, The image feature at least comprises a morphological feature of a cell nucleus, and the morphological feature is used to reflect the morphology and area of the cell nucleus, and the similarity clustering processing of the image features is performed on the first image area, the second image area and each sub-image to determine a similarity number pair of each sub-image, comprising: determining a first morphological feature of the first image area and a second morphological feature of the second image area; calculating a target morphological feature of each sub-image; determining a first similarity according to the target morphological feature and the first morphological feature, and determining a second similarity according to the target morphological feature and the second morphological feature.

5. The method of claim 1, wherein, The image feature at least comprises a number of mitotic figures of a cell nucleus, and the clustering processing is performed on the first image area, the second image area and each sub-image to determine a similarity number pair of each sub-image, comprising: determining a first number of mitotic figures of the first image area and a second number of mitotic figures of the second image area; calculating a target number of mitotic figures of each sub-image; determining a first similarity according to the target number of mitotic figures and the first number of mitotic figures, and determining a second similarity according to the target number of mitotic figures and the second number of mitotic figures.

6. The method of claim 1, wherein, The image segmentation of the third image area is performed to obtain a plurality of sub-images of the third image area, comprising: determining a target pixel size by using the first image area and the second image area; performing an equi-ratio segmentation processing on the third image area according to the target pixel size to obtain a plurality of sub-images with the same pixel size as the target pixel size.

7. A sample volume augmentation device, comprising: The device comprises: an image acquisition module: configured to acquire a digital pathology image, wherein the digital pathology image comprises a first image area labeled with a positive sample label, a second image area labeled with a negative sample label and a third image area without a label; an image segmentation module: configured to perform image segmentation on the third image area to obtain a plurality of sub-images of the third image area; The similarity clustering module is configured to perform similarity clustering processing on image features by using the first image region, the second image region, and each of the sub-images, to determine a similarity pair for each of the sub-images, the similarity pair being used to reflect a first similarity between the sub-image and the first image region and a second similarity between the sub-image and the second image region. The label determining module is configured to determine a target sample label corresponding to each of the sub-images according to the similarity pair of each of the sub-images and a preset similarity coordinate system, the similarity coordinate system being used to indicate a sample label corresponding to the similarity pair, and the target sample label including at least a positive sample label or a negative sample label. The coordinate axes of the similarity coordinate system are composed of the first similarity and the second similarity, the first similarity and the second similarity are inversely proportional to each other, and the label determining module is specifically configured to determine at least two similarity thresholds, including a minimum similarity threshold and a maximum similarity threshold, by using the first similarity and the second similarity of each of the sub-images, to perform horizontal and vertical division on the similarity coordinate system according to the minimum similarity threshold and the maximum similarity threshold, to determine each of the divided distribution regions, including at least a top-left corner distribution region and a bottom-right corner distribution region, the top-left corner distribution region being used to reflect a distribution region of the first similarity pair of the first image region in the similarity coordinate system, and the bottom-right corner distribution region being used to reflect a distribution region of the second similarity pair of the second image region in the similarity coordinate system, to input the first similarity and the second similarity of each of the sub-images into the preset similarity coordinate system, to determine a target distribution region of each of the sub-images in the similarity coordinate system, to determine the target sample label of each of the sub-images as a positive sample label when the target distribution region is the top-left corner distribution region, and to determine the target sample label of each of the sub-images as a negative sample label when the target distribution region is the bottom-right corner distribution region.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to enable the processor to perform the steps of the method in any one of claims 1 to 6. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores the computer program, and the computer program is executed by the processor to enable the processor to perform the steps of the method in any one of claims 1 to 6.

Images