Pathological section image analysis system, method and computer readable medium

Abstract

The embodiments of the present disclosure relate to the field of image processing technology, and specifically to a pathological slice image analysis system, method, and computer-readable medium. The implementation method of the system includes: a pathological slice scanning device, a pathological image analysis server, and a pathological information storage server, wherein: the pathological slice scanning device is used to obtain a digital pathological whole slide image; the pathological image analysis server is used to block the digital pathological whole slide image to obtain an initial sub-slice image sequence; perform dyeing standardization processing on each initial sub-slice image to obtain a sub-slice image sequence; perform classification processing on each sub-slice image to obtain a qualified sub-slice image sequence; input the qualified sub-slice image into an image feature extraction model to obtain a sub-slice image feature vector sequence; generate a pathological slice classification confidence result; determine the pathological slice classification information, and send it to the pathological information storage server. This implementation method can improve the classification accuracy of pathological slices.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of image processing technology, and in particular to a pathology slice image analysis system, method, and computer-readable medium. Background Art

[0002] The size of digital pathology images often reaches billions of pixels. Doctors often need the assistance of AI (Artificial Intelligence) technology to improve efficiency and quality when reading pathology images.

[0003] Due to the high resolution of pathological images, the amount of annotations and costs involved are often very huge. Methods based on supervised learning are often difficult to reproduce in medicine. Therefore, existing AI technologies are usually active learning, weakly supervised learning, and unsupervised learning. Among them, the purpose of active learning methods is to improve the quality and efficiency of annotations through human-in-the-loop methods, and to train a satisfactory model with fewer annotations by screening the most useful annotations. However, the effect of active learning depends in part on the performance of the initial model. If the initial model performance is poor, low-value samples may be mistakenly selected for annotation. Weakly supervised methods aim to use image-level information to guide the model to make better judgments. The requirements for annotations are lower. It is not necessary to accurately frame the specific diseased cells in the image, but to determine their categories at the level of the entire image. Due to the reduced annotation requirements, weakly supervised learning may cause differences in the quality of the labeled data, which may further have a negative impact on the training of the model. Unsupervised learning methods can completely rely on unlabeled data for training by designing some tasks, such as mask reconstruction and contrastive learning. They can discover hidden patterns and structures in the data and help understand the inherent laws of image data. Since unsupervised learning does not have a clear training objective (such as classification, regression, etc.), its training process may lack clear directionality and be difficult to converge.

[0004] In addition, although the pathology slice image analysis system can distinguish different types of lesions by deploying AI technology based on the above three forms of learning, adjusting the weight of category loss, iteratively optimizing the annotation quality, etc., the classification characteristics of diseased cells are easy to confuse, and there are few lesion samples. The image representations of diseased cells are diverse and cannot be exhausted in the annotation, resulting in uneven annotation quality of medical images. As a result, most of the current pathology slice image analysis systems have a high misjudgment rate of pathology slice categories, which hinders the further implementation of AI technology to assist doctors in interpretation. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a pathology slice image analysis system, method, and computer-readable medium to solve the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a pathology slice image analysis system, the system comprising: a pathology slice scanning device, a pathology image analysis server and a pathology information storage server, wherein: the pathology slice scanning device is used to scan the pathology slice to obtain a digital pathology whole slide image, and send the digital pathology whole slide image to the pathology image analysis server; the pathology image analysis server is used to perform block division processing on the received digital pathology whole slide image to obtain an initial sub-slice image sequence; the pathology image analysis server is used to perform staining standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference staining information to obtain a sub-slice image sequence; the pathology image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence, wherein a qualified sub-slice image in the qualified sub-slice image sequence is a sub-slice image whose quality assessment score is greater than a preset score threshold; The analysis server is used to input each qualified sub-slice image in the above-mentioned qualified sub-slice image sequence into a pre-trained image feature extraction model in sequence to obtain a sub-slice image feature vector sequence, wherein the above-mentioned image feature extraction model is obtained by training an initial image feature extraction model according to the sample qualified sub-slice image after foreground target annotation, and the above-mentioned initial image feature extraction model is a visual neural network; the above-mentioned pathological image analysis server is used to generate a pathological slice classification confidence result corresponding to the above-mentioned digital pathological whole slide image based on the above-mentioned sub-slice image feature vector sequence and a preset attention mechanism network; the above-mentioned pathological image analysis server is used to determine the above-mentioned digital pathological whole slide image and the above-mentioned pathological slice classification confidence result as pathological slice classification information, and send the above-mentioned pathological slice classification information to the above-mentioned pathological information storage server for storage; the above-mentioned pathological information storage server is used to send the above-mentioned pathological slice classification information to the above-mentioned user terminal in response to receiving a viewing request information corresponding to the above-mentioned pathological slice classification information sent by the user terminal.

[0008] In a second aspect, some embodiments of the present disclosure provide a pathology slice image analysis method, the method comprising: performing block division processing on a pre-scanned digital pathology whole slide image to obtain an initial sub-slice image sequence; performing staining standardization processing on each initial sub-slice image in the above-mentioned initial sub-slice image sequence based on pre-generated reference staining information to obtain a sub-slice image sequence; performing classification processing on each sub-slice image in the above-mentioned sub-slice image sequence to obtain a qualified sub-slice image sequence, wherein a qualified sub-slice image in the above-mentioned qualified sub-slice image sequence is a sub-slice image whose corresponding quality assessment score is greater than a preset score threshold; Each qualified sub-slice image is sequentially input into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence, wherein the image feature extraction model is obtained by training an initial image feature extraction model based on the sample qualified sub-slice image after foreground target annotation, and the initial image feature extraction model is a visual neural network; based on the sub-slice image feature vector sequence and a preset attention mechanism network, a pathology slice classification confidence result corresponding to the digital pathology whole slide image is generated; the digital pathology whole slide image and the pathology slice classification confidence result are determined as pathology slice classification information, and the pathology slice classification information is stored for user acquisition.

[0009] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in the second aspect above.

[0010] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the pathological slice image analysis system of some embodiments of the present disclosure, the accuracy of pathological slice classification can be improved. Specifically, the reason for the high category misjudgment rate is that the classification features of diseased cells are easily confused, and there are few diseased samples, and the image representations of diseased cells are diverse and cannot be exhausted in the annotation, which makes the annotation quality of medical images uneven, thus causing most of the current pathological slice image analysis systems to have a high pathological slice category misjudgment rate. Based on this, some embodiments of the present disclosure provide a pathological slice image analysis system, wherein: the pathological slice scanning device is used to scan the pathological slice to obtain a digital pathological whole slide image, and send the digital pathological whole slide image to the pathological image analysis server; the pathological image analysis server is used to perform block division processing on the received digital pathological whole slide image to obtain an initial sub-slice image sequence; the pathological image analysis server is used to perform staining standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference staining information to obtain a sub-slice image sequence; the pathological image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence, wherein the qualified sub-slice images in the qualified sub-slice image sequence are sub-slice images whose corresponding quality assessment scores are greater than a preset score threshold; the pathological image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence. Each qualified sub-slice image in the column is sequentially input into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence, wherein the above-mentioned image feature extraction model is obtained by training an initial image feature extraction model based on the sample qualified sub-slice image after foreground target annotation, and the above-mentioned initial image feature extraction model is a visual neural network; the above-mentioned pathological image analysis server is used to generate a pathological slice classification confidence result corresponding to the above-mentioned digital pathological whole slide image based on the above-mentioned sub-slice image feature vector sequence and a preset attention mechanism network; the above-mentioned pathological image analysis server is used to determine the above-mentioned digital pathological whole slide image and the above-mentioned pathological slice classification confidence result as pathological slice classification information, and send the above-mentioned pathological slice classification information to the above-mentioned pathological information storage server for storage; the above-mentioned pathological information storage server is used to send the above-mentioned pathological slice classification information to the above-mentioned user terminal in response to receiving a viewing request information corresponding to the above-mentioned pathological slice classification information sent by the user terminal.Therefore, the pathological image analysis server in the pathological slice image analysis system of some embodiments of the present disclosure can obtain image features with less background interference and more obvious feature differences between abnormal cells and normal cells by performing quality screening on each sub-slice image corresponding to the pathological slice, and on this basis, by extracting features from each sub-slice image after the foreground target is annotated, so as to determine the classification information of the pathological slice, thereby reducing the misjudgment rate of the pathological slice category. Thus, the accuracy of pathological slice classification can be improved. Furthermore, it can be used to assist users in interpreting pathological slices and improve the accuracy of users' interpretation of pathological slices. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0012] Figure 1 is a schematic structural diagram of an embodiment of a pathological section image analysis system according to the present disclosure;

[0013] Figure 2 It is a schematic diagram of the process of foreground and background segmentation and annotation during the training of the image feature extraction model disclosed in the present invention;

[0014] Figure 3 is a flow chart of image feature extraction model training according to the present disclosure;

[0015] Figure 4 is a flow chart of an embodiment of a pathological section image analysis method according to the present disclosure. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0018] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0019] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0020] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0021] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0022] First, see Figure 1 , Figure 1 The schematic diagram of the structure of an embodiment of a pathological slice image analysis system according to the present disclosure is shown. The pathological slice image analysis system 100 comprises: a pathological slice scanning device 101 , a pathological image analysis server 102 and a pathological information storage server 103 .

[0023] The pathology slice scanning device 101 is used to scan the pathology slice to obtain a digital pathology whole slide image, and send the digital pathology whole slide image to a pathology image analysis server.

[0024] In some embodiments, the pathological slice scanning device 101 can be used to scan the pathological slice to obtain a digital pathological whole slide image, and send the digital pathological whole slide image to the pathological image analysis server 102. The pathological slice scanning device 101 can be a high-resolution scanner for scanning pathological slices. For example, the pathological slice scanning device 101 can be but not limited to one of the following: an optical microscopic slice scanner, an electron microscopic slice scanner. The digital pathological whole slide image can be a digital image of the pathological slice in RGB format. The pathological slice corresponds to a unique number. The pathological image analysis server 102 can be a server deployed with an image feature extraction model for analyzing pathological images.

[0025] The pathology image analysis server 102 is used to perform block division processing on the received digital pathology full-slide image to obtain an initial sub-slice image sequence.

[0026] In some embodiments, the pathological image analysis server 102 may be used to perform block division processing on the received digital pathological whole slide image to obtain an initial sub-slice image sequence. The initial sub-slice images in the initial sub-slice image sequence may be image blocks of completely identical size and non-overlapping obtained by uniformly dividing the digital pathological whole slide image. The pathological image analysis server may perform block division processing on the digital pathological whole slide image by a sliding segmentation method with a preset sliding window and a preset sliding step length to obtain an initial sub-slice image sequence. The preset sliding window may be a preset window of a fixed size for dividing image blocks. For example, the preset sliding window may be a sliding window of 640 pixels * 640 pixels. The preset sliding step length may be the distance that the preset sliding window moves each time. It should be noted that the preset sliding step length is usually the same as the size of the preset sliding window, thereby avoiding missing information on the digital pathological whole slide image and preventing overlap between the sub-slice images. In addition, the length and width of the preset sliding window may be modified according to the training resources and the length and width statistics of the diseased cells.

[0027] Continuing, before the subsequent pathology image analysis server classifies and identifies each initial sub-slice image after the above block division, it is also necessary to perform quality screening to reduce data noise and improve the accuracy of classification prediction. In addition, the present disclosure also takes into account that the different staining levels of pathology whole slide images will affect the results of quality screening, and the conventional slice image staining scheme has the technical problem of "selecting the template image for benchmarking through artificial vision, which is prone to inconsistent staining results due to the instability of artificial vision. Even if a generative neural network can be used to generate an image with a uniform staining style, for high-resolution images such as whole slide images, the generated staining image is prone to noise in details, and the boundary connection information between sub-slices may also be lost, resulting in a decrease in the quality of the sub-slice image." Therefore, in order to unify the staining levels of each initial sub-slice image, the present disclosure decides to adopt the following scheme:

[0028] The pathological image analysis server 102 is configured to perform a staining standardization process on each initial sub-slice image in the initial sub-slice image sequence based on the pre-generated reference staining information to obtain a sub-slice image sequence.

[0029] In some embodiments, the pathological image analysis server 102 may be used to perform dyeing standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on the pre-generated reference dyeing information to obtain a sub-slice image sequence. The sub-slice images in the sub-slice image sequence may be initial sub-slice images in the LAB (color-opponent space) color space after the image channel values ​​are standardized. The reference dyeing information may be information of parameters pre-generated for dyeing standardization of images. The reference dyeing information may include a first dyeing parameter, a second dyeing parameter, a third dyeing parameter, a fourth dyeing parameter, a fifth dyeing parameter, and a sixth dyeing parameter. The first dyeing parameter and the fourth dyeing parameter correspond to the first image channel. The first image channel may be a brightness channel in the LAB color space. The second dyeing parameter and the fifth dyeing parameter correspond to the second image channel. The second image channel may be a color channel of red and green in the LAB color space. The third dyeing parameter and the sixth dyeing parameter correspond to the third image channel. The third image channel may be a color channel of yellow and blue in the LAB color space. The first dyeing parameter, the second dyeing parameter, and the third dyeing parameter respectively represent the average values ​​of each image on the corresponding image channel. The fourth dyeing parameter, the fifth dyeing parameter, and the sixth dyeing parameter respectively represent the standard deviation of each image in the corresponding image channel.

[0030] As an example, the pathological image analysis server may perform the following steps for each initial sub-slice image in the initial sub-slice image sequence to generate a sub-slice image in the sub-slice image sequence:

[0031] The first step is to perform color space conversion processing on the initial sub-slice image to obtain a sub-slice image to be dyed. The sub-slice image to be dyed may be an initial sub-slice image in a LAB color space whose dyeing degree is to be standardized. The initial sub-slice image may be converted into a LAB color space by a method of converting an RGB color space into a LAB color space to obtain a sub-slice image to be dyed.

[0032] The second step is to determine the first image channel average value, the second image channel average value, the third image channel average value, the first image channel standard deviation, the second image channel standard deviation and the third image channel standard deviation corresponding to the sub-section image to be stained. The first image channel average value may be the average value of each component of each pixel of the image on the first image channel. The second image channel average value may be the average value of each component of each pixel of the image on the second image channel. The third image channel average value may be the average value of each component of each pixel of the image on the third image channel. The first image channel standard deviation may be the standard deviation of each pixel of the image on the first image channel. The second image channel standard deviation may be the standard deviation of each pixel of the image on the second image channel. The third image channel standard deviation may be the standard deviation of each pixel of the image on the third image channel.

[0033] In the third step, the sub-slice image to be stained may be subjected to a staining standardization process to generate a sub-slice image according to the pre-generated reference staining information, the first image channel average value, the second image channel average value, the third image channel average value, the first image channel standard deviation, the second image channel standard deviation and the third image channel standard deviation by using the following first formula group: .

[0034] in, Represents the first image channel value of a single pixel in the sub-slice image after staining normalization. Represents the second image channel value of a single pixel in the sub-slice image obtained after staining normalization. Represents the third image channel value of a single pixel in the sub-slice image after staining normalization. Represents the first image channel value of a single pixel in the initial sub-slice image before stain normalization. Represents the second image channel value of a single pixel in the original sub-slice image before stain normalization. Represents the third image channel value of a single pixel in the original sub-slice image before stain normalization. Represents the first staining parameter. Represents the second staining parameter. Represents the third staining parameter. Represents the fourth staining parameter. Represents the fifth staining parameter. Represents the sixth staining parameter. Represents the average value of the first image channel corresponding to the initial sub-slice image. Represents the average value of the second image channel corresponding to the initial sub-slice image. Indicates the average value of the third image channel corresponding to the initial sub-slice image. Indicates the standard deviation of the first image channel corresponding to the initial sub-slice image. Indicates the standard deviation of the second image channel corresponding to the initial sub-slice image. Indicates the standard deviation of the third image channel corresponding to the initial sub-slice image.

[0035] Optionally, before the pathological image analysis server performs dyeing standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on the pre-generated reference dyeing information to obtain the sub-slice image sequence, the pathological image analysis server may further perform the following steps to generate the reference dyeing information:

[0036] The first step is to obtain a reference digital pathology whole slide image set. The reference digital pathology whole slide image in the reference digital pathology whole slide image set can be a digital image in RGB format corresponding to the pathology section used to generate the parameters required for the dyeing standardization process. The reference digital pathology whole slide image set can be obtained from a pathology image database.

[0037] The second step is to perform block division processing on each reference digital pathology whole slide image in the reference digital pathology whole slide image set to obtain a reference slice image set. In which, the block division processing method can be referred to to perform block division processing on each reference digital pathology whole slide image in the reference digital pathology whole slide image set to generate each reference slice image to obtain a reference slice image set.

[0038] The third step is to perform color space conversion on each reference slice image in the reference slice image set to obtain a converted reference slice image set. The method of converting the RGB color space into the LAB color space can be used to perform color space conversion on each reference slice image in the reference slice image set to obtain a converted reference slice image set.

[0039] The fourth step is to perform clustering processing on each converted reference slice image in the above-mentioned converted reference slice image set to obtain each converted reference slice image cluster. Wherein, each converted reference slice image cluster in the above-mentioned converted reference slice image clusters can be a collection of each converted reference slice image with similar staining degree. The converted reference slice images in the above-mentioned converted reference slice image set can be clustered by a preset clustering method to obtain each converted reference slice image cluster. For example, the above-mentioned clustering method can be but is not limited to one of the following: K-means clustering method, density-based clustering method.

[0040] Step 5: Select a converted reference slice image cluster that meets a preset quantity condition from the above converted reference slice image clusters as a target slice image cluster. The preset quantity condition may be that the number of images corresponding to the converted reference slice image cluster is the maximum value among the converted reference slice image clusters.

[0041] Step 6: Generate reference staining information based on the target slice image cluster. Specifically, the following steps can be performed:

[0042] In the first sub-step, the first dyeing parameter, the second dyeing parameter and the third dyeing parameter corresponding to the target slice image cluster are determined by the following second formula group: .

[0043] in, The number of images representing the above target slice image cluster. Indicates the sequence number of the image. Indicates the pixel number. Indicates The first image channel value of pixels. Indicates The second image channel value of pixels. Indicates The third image channel value of pixels. represents the first The total number of pixels in the image.

[0044] In the second sub-step, the fourth dyeing parameter, the fifth dyeing parameter and the sixth dyeing parameter corresponding to the target slice image cluster are determined by the following third formula group: .

[0045] In a third sub-step, the first dyeing parameter, the second dyeing parameter, the third dyeing parameter, the fourth dyeing parameter, the fifth dyeing parameter and the sixth dyeing parameter are determined as reference dyeing information.

[0046] In some optional implementations of some embodiments, the pathological image analysis server may also randomly select an image from the target slice image cluster, and determine the respective average values ​​and standard deviations corresponding to the respective image channels of the selected image as reference staining information. Each image channel of the selected image corresponds to a mean value and a standard deviation. It should be noted that the above-mentioned method of randomly selecting an image from the target slice image cluster for generating reference staining information can simplify the calculation amount of the reference staining information of the pathological image analysis server, save computing resources, and shorten the computing time.

[0047] The above-mentioned sub-slice image sequence generation and its related contents, as an inventive point of an embodiment of the present disclosure, solve the technical problem of "inconsistent staining results and reduced quality of sub-slice images" existing in conventional slice image staining schemes. The reason why the conventional slice image staining scheme has inconsistent staining results and reduced quality of sub-slice images is that the template image used for calibration is selected by artificial vision, and the instability of artificial vision easily leads to inconsistent staining results. Even if a generative neural network can be used to generate an image with a uniform staining style, for a high resolution such as a full slide image, the generated staining image is prone to noise in details, and the boundary connection information between sub-slices may also be lost. If this solution solves the above-mentioned problems, it can achieve the effect of improving the consistency of staining degree and image quality of sub-slice images. Therefore, in order to achieve this effect, this solution first obtains a more representative slice image cluster by clustering. Then, according to the representative slice image cluster, the target parameters for staining standardization are determined. Finally, each initial sub-slice image is stained according to the target parameters. In this way, it can be ensured that the staining style of each initial sub-slice image is uniform and no noise is increased. Thus, the noise caused by staining differences can be reduced and the quality of sub-slice images can be improved. Furthermore, the accuracy of subsequent classification predictions can be improved.

[0048] The pathological image analysis server 102 is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence.

[0049] In some embodiments, the pathological image analysis server 102 may be used to classify each sub-slice image in the sub-slice image sequence in various ways to obtain a qualified sub-slice image sequence. The qualified sub-slice images in the qualified sub-slice image sequence may be sub-slice images whose corresponding quality assessment scores are greater than a preset score threshold. The preset score threshold may be a preset lower limit of the quality assessment score of the image. The quality assessment score may characterize the clarity of the image and the proportion of impurities in the image.

[0050] In some optional implementations of some embodiments, the pathological image analysis server may classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence through the following steps:

[0051] In the first step, each sub-slice image in the above sub-slice image sequence is sequentially input into a pre-trained slice image quality classification model to obtain a sub-slice image classification information sequence. The sub-slice image classification information in the above sub-slice image classification information sequence may correspond to the sub-slice images of the same order in the above sub-slice image sequence. The sub-slice image classification information in the above sub-slice image classification information sequence may include a quality classification probability and a quality category identifier. The above quality category identifier may be a unique identifier of an image quality category. The above image quality category may be one of the following: normal quality and abnormal quality. Abnormal quality may represent an image whose quality assessment score is less than or equal to the above preset score threshold. For example, an image with abnormal quality may be an image with blurred imaging, impurities or pollution. Normal quality may represent an image whose quality assessment score is greater than the above preset score threshold. For example, an image with normal quality may be an image without impurities or pollution. The above quality classification probability may represent the predicted probability of the sub-slice image being of the corresponding quality type. The above quality classification probability may be linearly mapped to a quality assessment score. The above slice image quality classification model may be a classification model that takes a sub-slice image as input and takes sub-slice image classification information as output. For example, the slice image quality classification model can be obtained by training based on the YOLO model.

[0052] As an example, when the quality category identifier corresponding to the sub-slice image indicates that the quality is normal and the quality classification probability is 0.85, through linear mapping, the quality assessment score corresponding to the sub-slice image can be 85 points. When the quality category identifier corresponding to the sub-slice image indicates that the quality is abnormal and the quality classification probability is 0.85, through linear mapping, the quality assessment score corresponding to the sub-slice image can be 35 points.

[0053] In practice, the pathological image analysis server can screen out sub-slice images with normal quality through the slice image quality classification model, and effectively reduce interfering slice data with blurred imaging, impurities or pollution.

[0054] Optionally, the pathological slice image analysis system may further include a model training server. The model training server may train various models (such as target detection models and segmentation models) for processing images. The model training server may send the trained image feature extraction model to the pathological image analysis server for deployment.

[0055] Optionally, the slice image quality classification model may be pre-trained based on the first sample image data set and the second sample image data set. The first sample image data set may include each first sample image data of normal quality. The second sample image data set may include each second sample image data of abnormal quality. Each first sample image data and each second sample image data may be a sub-slice image obtained by evenly dividing the digital pathology whole slide image and performing a dyeing standardization process.

[0056] Optionally, the above-mentioned slice image quality classification model can also be pre-trained based on the third sample slice image data set and the fourth sample slice image data set. Among them, the above-mentioned third sample slice image data set can include each third sample slice image data with normal quality. The above-mentioned fourth sample slice image data set can include each fourth sample slice image data with abnormal quality. Each third sample slice image data and each fourth sample slice image data can be a thumbnail of a different digital pathology whole slide image. In practice, the above-mentioned pathology image analysis server 102 trains the slice image quality classification model based on the thumbnail of the entire digital pathology whole slide image, and there is no need to divide the digital pathology whole slide image, thereby accelerating the speed of data quality screening and improving the efficiency of data quality screening.

[0057] In the second step, sub-slice image classification information satisfying a preset quality category condition is selected from the above sub-slice image classification information sequence in turn as target sub-slice image classification information to obtain a target sub-slice image classification information sequence. The above preset quality category condition may be that the quality category identifier included in the sub-slice image classification information indicates that the quality is normal.

[0058] In the third step, in response to determining that the target sub-slice image classification information sequence meets the preset image quantity condition, for each target sub-slice image classification information in the target sub-slice image classification information sequence, the following steps are performed:

[0059] The first sub-step is to perform image segmentation processing on the sub-slice image corresponding to the target sub-slice image classification information to obtain a sub-slice foreground image. The preset image quantity condition may be: the difference between the first number corresponding to the sub-slice image sequence and the second number corresponding to the target sub-slice image classification information sequence, and the ratio of the difference to the first number is less than or equal to a preset quality abnormality threshold. The first number may be the number of sub-slice images in the sub-slice image sequence. The second number may be the number of each sub-slice image corresponding to the target sub-slice image classification information sequence. The preset quality abnormality threshold may be an upper limit of the ratio between the number of acceptable sub-slice images with abnormal quality and the first number. The sub-slice image corresponding to the target sub-slice image classification information may be segmented by an image segmentation method based on the maximum inter-class variance method to obtain a sub-slice foreground image.

[0060] In the second sub-step, the ratio between the number of pixels corresponding to the sub-slice foreground image and the number of pixels of the sub-slice image corresponding to the target sub-slice image classification information is determined as the foreground proportion.

[0061] In the third sub-step, in response to determining that the foreground proportion is greater than a preset ratio threshold, the sub-slice image corresponding to the target sub-slice image classification information is determined as a qualified sub-slice image. The preset ratio threshold may be a preset lower limit of the proportion of all pixels of the foreground image to the entire sub-slice image. For example, the preset ratio threshold may be 50%.

[0062] Optionally, the pathology image analysis server 102 may also, in response to determining that the target sub-slice image classification information sequence does not meet the preset image quantity condition, annotate the digital pathology whole slide image based on a preset low-quality label text. The low-quality label text may be a text used to identify abnormal image quality. For example, the low-quality label text may be "abnormal". The low-quality label text may be set to the image quality attribute value corresponding to the digital pathology whole slide image. It should be noted that when the digital pathology whole slide image is marked as low quality, image analysis will no longer be performed on the image.

[0063] The pathological image analysis server 102 is used to sequentially input each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence.

[0064] In some embodiments, the pathological image analysis server 102 may sequentially input each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence. The image feature extraction model may be obtained by training an initial image feature extraction model based on a sample qualified sub-slice image annotated with a foreground target. The foreground target may be each cell in a digital pathology whole slide image. The initial image feature extraction model may be a visual neural network. For example, the initial image feature extraction model may be a ViT (Vision Transformer) model or a Resnet residual network model. The sub-slice image feature vectors in the sub-slice image feature vector sequence may correspond one-to-one to the qualified sub-slice images in the qualified sub-slice image sequence. The sub-slice image feature vectors in the sub-slice image feature vector sequence may characterize the features of the corresponding qualified sub-slice images.

[0065] Optionally, the model training server may train the initial image feature extraction model to obtain an image feature extraction model through the following steps:

[0066] The first step is to obtain a sample digital pathology whole slide image set and an initial image feature extraction model. The sample digital pathology whole slide images in the sample digital pathology whole slide image set can be digital pathology whole slide images that have been stained and standardized for model training.

[0067] In the second step, sample digital pathology whole slide images are selected from the sample digital pathology section image and digital pathology whole slide image set, and the following training steps are performed:

[0068] The first sub-step is to perform block division processing on the sample digital pathology whole slide image to obtain a sample sub-slice image sequence. The sample sub-slice image sequence may be a sub-slice image sequence corresponding to the sample digital pathology whole slide image. The model training server may perform block division processing on the sample digital pathology whole slide image according to the block division processing method to obtain a sample sub-slice image sequence, which will not be described in detail here.

[0069] The second sub-step is to classify each sample sub-slice image in the sample sub-slice image sequence to obtain a sample qualified sub-slice image sequence. The sample qualified sub-slice images in the sample qualified sub-slice image sequence may be qualified sub-slice images corresponding to the sample sub-slice images. The model training server may classify each sample sub-slice image in the sample sub-slice image sequence according to the classification processing step to obtain a sample qualified sub-slice image sequence, which will not be described in detail here.

[0070] The third sub-step is to determine the image foreground annotation information corresponding to each sample qualified sub-slice image in the above-mentioned sample qualified sub-slice image sequence based on a preset image segmentation model, and obtain an image foreground annotation information sequence. The image foreground annotation information in the above-mentioned image foreground annotation information sequence may be the information of the segmentation mask obtained after the foreground target is annotated for the sample qualified sub-slice image. The segmentation mask may be used to identify the segmented object or area in the image. The above-mentioned image segmentation model may be a foreground and background segmentation model that takes the sample qualified sub-slice image as input and takes the foreground target annotated image and the image foreground annotation information as output. The above-mentioned foreground target annotated image may be a sample qualified sub-slice image after the cells in the foreground image are annotated. For example, the above-mentioned image segmentation model may be a SAM (Segment Anything Model) model or a fine-tuned MedSAM (Segment Anything in Medical Images).

[0071] As an example, when the above-mentioned image segmentation model is a SAM model, the SAM model may include an image encoder, a prompt encoder and a mask decoder. Among them, the above-mentioned image encoder can be used to extract features of the input image. The above-mentioned prompt encoder can be used to receive and encode prompt information input by the user. The prompt information can be obtained by the user by clicking or framing the cells in the sub-slice image. The above-mentioned mask decoder can be used to convert the outputs of the image encoder and the prompt encoder into a segmentation mask. The above-mentioned model training server can determine the corresponding image foreground annotation information for each sample qualified sub-slice image in the above-mentioned sample qualified sub-slice image sequence based on the above-mentioned SAM model through the following sub-steps:

[0072] Sub-step 1: input the above-mentioned sample qualified sub-slice image into the image encoder included in the above-mentioned SAM model to obtain image embedding information.

[0073] Sub-step 2: using the prompt encoder included in the SAM model, receiving the annotation prompt information corresponding to the qualified sub-slice image of the sample, and generating prompt embedding information based on the annotation prompt information.

[0074] Sub-step three: input the above image embedding information and the above prompt embedding information into the mask decoder included in the above SAM model to obtain the foreground target annotated image and image foreground annotated information.

[0075] It should also be noted that when the above-mentioned image segmentation model is a fine-tuned MedSAM model, the workload of manual labeling can be further saved, and only the image mask information generated by the image segmentation model is used to assist the subsequent initial image feature extraction model in training for foreground target optimization.

[0076] As an example, Figure 2 A schematic diagram of the process of foreground and background segmentation and annotation during training of the image feature extraction model disclosed in the present invention is shown. Figure 2 It includes the image before annotation, the image during annotation, and the foreground target annotated image. Among them, the image before annotation is the image before the foreground target is annotated. The image during annotation is generated by the image segmentation model (such as the SAM model) for the user to annotate the foreground target by clicking, framing, etc. The image segmentation model can further refine the image during annotation according to the user's annotation prompt information to obtain the foreground target annotated image.

[0077] In the fourth sub-step, the sample qualified sub-slice images in the above sample qualified sub-slice image sequence are sequentially input into the initial image feature extraction model to obtain a sample sub-slice image feature vector sequence. Among them, each sample qualified sub-slice image can correspond to a sample sub-slice image feature vector in the sample sub-slice image feature vector sequence. The sample sub-slice image feature vector in the above sample sub-slice image feature vector sequence can characterize the features of the corresponding sample qualified sub-slice image. The sample sub-slice image feature vector in the above sample sub-slice image feature vector sequence can be represented by (1, C, d_model). Among them, one dimension in the C dimension (channels channel dimension) represents the class token (classification tag), and the remaining (C-1) dimensions represent the features of each sample sub-image block obtained by the initial image feature extraction model for dividing the input image. The above classification tag can characterize the category information of the input image. d_model is the feature dimension. The sample sub-image block can be obtained by the initial image feature extraction model by block division of the sample qualified sub-slice image. It should be noted that the segmentation mask at the corresponding position in the sample qualified sub-slice image can be found according to the position of the sub-image block.

[0078] As an example, when the single sample qualified sub-slice image input to the initial image feature extraction model is (3, 224, 224), 3 is the color channel, and 224 is the width and height of the image. The above single sample qualified sub-slice image can be divided into (16*16) sub-image blocks by the initial image feature extraction model, and then after the sub-image block embedding layer tiling operation, it can be obtained (1,768,14,14). Among them, 1 is batch_size, so it is 1 for a single image. 768 is the feature dimension, which is generally obtained according to the formula (224*224*3) / (14*14). 14 can represent the width and height of the sub-image block, which is obtained according to the formula (224 / 16). After tiling, the final feature embedding (1,197,768) can be obtained, that is, the sample sub-slice image feature vector. Among them, 197 represents the channel dimension plus a position embedding dimension. The channel dimension can be obtained according to the formula (14*14).

[0079] The fifth sub-step is to determine the foreground optimization loss value corresponding to each sample qualified sub-slice image in the sample qualified sub-slice image sequence based on the preset foreground loss function, the image foreground annotation information sequence and the sample sub-slice image feature vector sequence. The foreground optimization loss value can characterize the degree of difference between the foreground feature distribution and the background feature distribution and the degree of similarity between the feature distribution within the background region. For each sample qualified sub-slice image in the sample qualified sub-slice image sequence, the foreground optimization loss value corresponding to the sample qualified sub-slice image can be determined according to the image foreground annotation information corresponding to the sample qualified sub-slice image by the following foreground loss function: .

[0080] in, Represents the foreground loss function to be optimized. Represents the sum of each feature distance corresponding to each first target feature vector group. The first target feature vector group may be composed of two feature vectors corresponding to two sub-image blocks whose segmentation masks at corresponding positions in the sample qualified sub-slice image are both background. Each first target feature vector group corresponds to one feature distance. represents the sum of the feature distances corresponding to each second target feature vector group. The second target feature vector group can be composed of two feature vectors corresponding to two sub-image blocks corresponding to the background and foreground of the sample qualified sub-slice image. Each second target feature vector group corresponds to a feature distance. To minimize , which can reduce , increase , and now the distribution of background features is more similar, and the distribution of foreground and background features is more different. Represents the cosine similarity function. Represents the height of the input image to the image feature extraction model. Represents the width of the input image to the image feature extraction model. Indicates the sub-image block is the horizontal axis, is the position coordinate of the ordinate. Indicates the sub-image block is the horizontal axis, is the position coordinate of the ordinate. Represents the features of a sub-image block. Indicates the position on the sub-image block The image features at . Indicates the position on the sub-image block The image features at . , The coefficient that adjusts the weight.

[0081] It should be noted that when the input sub-slice image is divided into smaller sub-image blocks after the initial image feature extraction model, according to the foreground target labeling result output by the image segmentation model, in these sub-image blocks, if the ratio of the foreground area of ​​the sub-image block to the total image area is greater than a preset threshold, such as 50%, then when calculating the above foreground loss function, the features of the sub-image block will be used to calculate ; Otherwise, the features of the sub-image block will be used to calculate and Therefore, through the foreground loss function, the feature distance between the image foreground and background can be increased and the feature distance within the background can be reduced, making the background feature distribution more similar and the difference between the foreground and background feature distribution more obvious. This will enhance the model's feature representation of the foreground target and help improve the performance of subsequent classification prediction.

[0082] In the sixth sub-step, the sum of the obtained foreground optimization loss values ​​is determined as the total loss value.

[0083] In the seventh sub-step, in response to determining that the total loss value is less than a preset loss threshold, the trained initial image feature extraction model is determined as the image feature extraction model. The preset loss threshold may be an upper limit of a preset loss value. For example, the preset loss threshold may be 0.05.

[0084] Optionally, the model training server may also adjust the network parameters of the initial image feature extraction model in response to determining that the total loss value is greater than or equal to the preset loss threshold, and reselect sample digital pathology whole slide images from each sample digital pathology whole slide image information that has never been used, and use the adjusted initial image feature extraction model as the initial image feature extraction model to perform the training step again. The network parameters of the initial image feature extraction model may be adjusted according to the preset loss threshold by using a stochastic gradient descent method.

[0085] Optionally, the above-mentioned model training server can also adopt a batch training method to train the initial image feature extraction model. Specifically, each sample digital pathology whole slide image in the sample digital pathology whole slide image set is first divided into blocks to generate each sample sub-slice image, and a sample sub-slice image set is obtained. Then, according to the preset batch size, the sample sub-slice image set is divided into each training batch data set. Among them, the number of sample sub-slice images in each training batch is the same. Finally, according to each training batch data set, the initial image feature extraction model is trained. It should be noted that during the training process, the above-mentioned model training server usually uses a small batch gradient descent method or an Adam optimizer to adjust the network parameters of the initial image feature extraction model.

[0086] In practice, unsupervised training with foreground optimization loss, such as image mask reconstruction, can obtain a more robust image feature extraction model that focuses on foreground targets, which facilitates the development of high-performance fine-tuning for downstream tasks. In addition, the foreground optimization loss can also be used in conjunction with weak supervision methods, such as multi-instance learning, to perform cluster analysis and other operations on the final extracted visual image features to predict the judgment results of the overall pathological section.

[0087] Optionally, the model training server may also determine the trained initial image feature extraction model as the image feature extraction model in response to determining that the initial image feature extraction model has been iterated for a preset number of rounds. The preset number of rounds may be a preset number of iterations. For example, the preset number of rounds may be 100. In each iteration cycle of the model training, the foreground optimization loss is calculated, and according to the foreground optimization loss, the network parameters in the initial image feature extraction model are adjusted by gradient descent and back propagation.

[0088] The training steps and related contents of the above-mentioned image feature extraction model are an inventive point of the embodiments of the present disclosure. By using a large number of AI-assisted simple annotated foreground and background mask information pathological slice images and combining foreground optimization loss to train the image feature extraction model, the distance between the foreground and background area features can be widened, and the background area features can be kept in a similar distribution. Therefore, in the subsequent classification and prediction process of pathological slice features, more attention can be paid to differentiated foreground features, while reducing the influence of similarly distributed background features, thereby achieving the purpose of focusing on foreground target information and ignoring background interference. Furthermore, the accuracy of judging whether a cell is pathological can be improved.

[0089] As an example, Figure 3 The figure shows a flow chart of the image feature extraction model training disclosed in the present invention. Figure 3 It includes the input image, the sub-image blocks after model division, the output classification marks and sub-image block feature sequences, and the schematic image of the foreground optimization strategy. The above input image can be a qualified sub-slice image of the sample. The foreground optimization strategy is to optimize the foreground target by increasing the feature distance between the foreground and the background and reducing the feature distance inside the background. The yellow dots in the schematic image of the foreground optimization strategy represent the and The red dots represent the sub-image blocks used to calculate It should be noted that the yellow dots and red dots in the schematic image of the foreground optimization strategy correspond to the yellow dots and red dots representing the features of the sub-image block, respectively.

[0090] In addition, Figure 3 In the above, first, the input image is segmented by the image feature extraction model to obtain sub-image blocks. For the convenience of explanation, the input image is divided into 4*4 sub-image blocks of the same size, and each sub-image block has a corresponding position coordinate. The position coordinates of the sub-image blocks can be (1, 1), (1, 2), ..., (4, 2) (4, 3) (4, 4) in sequence. Then, the image feature extraction model can perform feature extraction processing on each sub-image block, and finally obtain classification labels and sub-image block feature sequences. Finally, the foreground target is optimized through the foreground optimization strategy.

[0091] The pathology image analysis server 102 is used to generate a pathology slice classification confidence result corresponding to the digital pathology full slide image based on the sub-slice image feature vector sequence and a preset attention mechanism network.

[0092] In some embodiments, the pathological image analysis server 102 can generate the pathological slice classification confidence result corresponding to the digital pathological whole slide image in various ways based on the sub-slice image feature vector sequence and the preset attention mechanism network. Among them, the attention mechanism network can generate different attention scores when the sub-slice image is predicted to be different pathological slice categories according to the characteristics of the sub-slice image. The pathological slice category can be the category of the pathological slice. The attention score can be used to measure the importance of different features of the sub-slice image. The pathological slice classification confidence result can be information on different confidences corresponding to different categories when the pathological slice is predicted to be.

[0093] As an example, when the pathological section category is binary, the pathological section category can be one of the following: positive, negative. Positive can indicate the presence of diseased cells in the corresponding pathological section. Negative can indicate the absence of diseased cells in the corresponding pathological section. When the pathological section category is multi-classified, taking cervical sections as an example, the pathological section category can be but is not limited to one of the following: ASC-US (Atypical Squamous Cells of Undetermined Significance) category, LSIL (Low-grade Squamous Intraepithelial Lesion) category, HSIL (High-grade Squamous Intraepithelial Lesion) category, no intraepithelial lesion category. It should be noted that the pathological section category is usually classified by the classification model according to the diseased cells that occupy a larger proportion in the pathological section.

[0094] Continuing, in the process of solving the above technical problems, the present disclosure considers that conventional classification models usually only output the category with the highest confidence for the input pathological image features, or output a classification result for each sub-slice image of the pathological image, and then aggregate the classification results to obtain the final classification result of the pathological image, and the conventional classification model also has the following technical problems: if only the pathological category with the highest confidence is output, it is difficult to provide a relatively comprehensive feedback for the pathological image, and if the final classification result of the pathological image is determined based on the local classification result of each sub-slice image, the classification result of the pathological image will be less accurate due to ignoring the connection between the sub-slice images. Therefore, the present disclosure decides to adopt the following solution.

[0095] In some optional implementations of some embodiments, the pathological image analysis server 102 may generate a pathological slice classification confidence result corresponding to the digital pathological full slide image based on the sub-slice image feature vector sequence and a preset attention mechanism network through the following steps:

[0096] In the first step, the sub-slice image feature vectors in the above sub-slice image feature vector sequence are sequentially input into a preset attention mechanism network to obtain a sequence of classified attention information groups. Each classified attention information group in the above classified attention information group sequence can correspond one-to-one to a sub-slice image feature vector in the above sub-slice image feature vector sequence. Each classified attention information can include a category identifier and an attention score. The above category identifier can be a unique identifier of a pathological slice category.

[0097] As an example, when the pathological slice category is binary and the category identifier included in the classified attention information indicates that the pathological slice category corresponding to the pathological slice is positive, the process of the attention mechanism network determining the attention score included in the above-mentioned classified attention information can be expressed by the following formula: .

[0098] in, Indicates the number of each qualified sub-slice image in the above qualified sub-slice image sequence. , Indicates the sequence number of the above qualified sub-slice image sequence. Indicates positive. Indicates The attention score when the pathological sub-slice corresponding to the qualified sub-slice image is judged as positive. represents the output of the positive branch in the above attention mechanism network, The positive branch features of qualified sub-slice images. represents the output of the positive branch in the above attention mechanism network, The positive branch features of qualified sub-slice images. represents the natural exponential function.

[0099] It should be noted that each feature branch in the attention mechanism network can correspond to a pathological slice category one by one. The attention scores of qualified sub-slice images under different pathological slice categories are calculated in the same way as the attention scores under the lesion cell category in the above example.

[0100] In the second step, for each pathological slice category identifier in the preset pathological slice category identifier sequence, the following steps are performed:

[0101] In the first sub-step, the classified attention information whose category identifier is the same as the pathological slice category identifier is selected from each of the classified attention information included in the classified attention information group sequence as the same category attention information to obtain the same category attention information group. The pathological slice category identifier in the pathological slice category identifier sequence can be a preset pathological slice category identifier.

[0102] The second sub-step is, for each attention information of the same category in the attention information group of the same category, based on the attention score included in the attention information of the same category, weighting the sub-slice image feature vector corresponding to the attention information of the same category to obtain a weighted sub-slice image feature vector. The weighted sub-slice image feature vector may be the result of multiplying the sub-slice image feature vector by the attention score.

[0103] The third sub-step is to aggregate the generated weighted sub-slice image feature vectors to obtain an aggregated sub-slice image feature vector. The generated weighted sub-slice image feature vectors can be aggregated by a pooling operation to obtain an aggregated sub-slice image feature vector. For example, the above pooling operation can be global average pooling or global maximum pooling.

[0104] The fourth sub-step is to determine the classification confidence score corresponding to the pathological slice category identifier based on the above-mentioned aggregated sub-slice image feature vector. The above-mentioned classification confidence score can represent the probability that the sub-slice image is classified as the corresponding pathological slice category. The classification confidence score corresponding to the pathological slice category identifier can be determined by a classification prediction head based on the above-mentioned aggregated sub-slice image feature vector. The above-mentioned classification prediction head can include a fully connected layer and a softmax activation function. When the pathological slice category is a binary classification, the softmax activation function can also be replaced by a sigmoid activation function.

[0105] The third step is to generate the classification confidence result of the pathology section corresponding to the digital pathology whole slide image based on the obtained classification confidence scores. Specifically, the following steps can be performed:

[0106] In the first sub-step, for each of the obtained classification confidence scores, the classification confidence score and the corresponding pathological section category identifier are determined as classification confidence information.

[0107] In the second sub-step, each classification confidence information is determined as a classification confidence result of the pathology section corresponding to the digital pathology full slide image.

[0108] The above-mentioned pathological slice classification confidence result generation step and its related content, as an inventive point of the present disclosure, can solve the technical problems existing in the above-mentioned conventional classification model. In order to solve the technical problems existing in the above-mentioned conventional classification model, first, the above-mentioned pathological image analysis server performs pooling processing on the results of the above-mentioned image feature extraction model through the above-mentioned attention mechanism network to aggregate the features of each sub-slice. Thus, dynamic feature aggregation can be performed on each sub-slice image feature by weighting, and the interference of irrelevant features can be further avoided. Then, the confidence of different pathological slice categories of the entire pathological slice can be predicted according to the fused features through the classification network. And because the above-mentioned attention mechanism network can independently calculate the attention scores under each classification for the target classification task, and perform more refined feature aggregation, thereby, the expression ability of the input aggregated features of each classification prediction branch can be improved, thereby, the accuracy of pathological slice classification can be improved. Furthermore, it can be used to better assist users in interpreting pathological results and improve the accuracy of interpretation.

[0109] The pathological image analysis server 102 is used to determine the digital pathological whole slide image and the pathological slice classification confidence result as pathological slice classification information, and send the pathological slice classification information to the pathological information storage server 103 for storage.

[0110] In some embodiments, the pathological image analysis server 102 may determine the digital pathological whole slide image and the pathological section classification confidence result as pathological section classification information, and send the pathological section classification information to the pathological information storage server for storage. The pathological information storage server may be a server deployed with a database for storing pathological information. It should be noted that the pathological section classification information corresponding to the pathological section may be queried from the pathological information storage server according to the number of the pathological section.

[0111] The pathology information storage server 103 is configured to send the pathology section classification information to the user terminal in response to receiving the viewing request information corresponding to the pathology section classification information sent by the user terminal.

[0112] In some embodiments, the pathological information storage server 103 may send the pathological section classification information to the user terminal in response to receiving the viewing request information corresponding to the pathological section classification information sent by the user terminal. The user terminal may be a terminal with a display screen. The viewing request information may include the pathological section number. The pathological information storage server 103 may query the corresponding pathological section classification information according to the pathological section number included in the viewing request information and send the found pathological section classification information to the user terminal.

[0113] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the pathological slice image analysis system of some embodiments of the present disclosure, the accuracy of pathological slice classification can be improved. Specifically, the reason for the high category misjudgment rate is that the classification features of diseased cells are easily confused, and there are few diseased samples, and the image representations of diseased cells are diverse and cannot be exhausted in the annotation, which makes the annotation quality of medical images uneven, thus causing most of the current pathological slice image analysis systems to have a high pathological slice category misjudgment rate. Based on this, some embodiments of the present disclosure provide a pathological slice image analysis system, wherein: the pathological slice scanning device is used to scan the pathological slice to obtain a digital pathological whole slide image, and send the digital pathological whole slide image to the pathological image analysis server; the pathological image analysis server is used to perform block division processing on the received digital pathological whole slide image to obtain an initial sub-slice image sequence; the pathological image analysis server is used to perform staining standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference staining information to obtain a sub-slice image sequence; the pathological image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence, wherein the qualified sub-slice images in the qualified sub-slice image sequence are sub-slice images whose corresponding quality assessment scores are greater than a preset score threshold; the pathological image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence. Each qualified sub-slice image in the column is sequentially input into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence, wherein the above-mentioned image feature extraction model is obtained by training an initial image feature extraction model based on the sample qualified sub-slice image after foreground target annotation, and the above-mentioned initial image feature extraction model is a visual neural network; the above-mentioned pathological image analysis server is used to generate a pathological slice classification confidence result corresponding to the above-mentioned digital pathological whole slide image based on the above-mentioned sub-slice image feature vector sequence and a preset attention mechanism network; the above-mentioned pathological image analysis server is used to determine the above-mentioned digital pathological whole slide image and the above-mentioned pathological slice classification confidence result as pathological slice classification information, and send the above-mentioned pathological slice classification information to the above-mentioned pathological information storage server for storage; the above-mentioned pathological information storage server is used to send the above-mentioned pathological slice classification information to the above-mentioned user terminal in response to receiving a viewing request information corresponding to the above-mentioned pathological slice classification information sent by the user terminal.Therefore, the pathological image analysis server in the pathological slice image analysis system of some embodiments of the present disclosure can obtain image features with less background interference and more obvious feature differences between abnormal cells and normal cells by quality screening of each sub-slice image corresponding to the pathological slice, and on this basis, by extracting features of each sub-slice image after the foreground target is annotated, so as to determine the classification information of the pathological slice, thereby reducing the misjudgment rate of the pathological slice category. Thus, the accuracy of pathological slice classification can be improved. Furthermore, it can be used to assist users in interpreting pathological slices and improve the accuracy of users' interpretation of pathological slices.

[0114] Figure 4 A flowchart of an embodiment of a medical image analysis method according to the present disclosure is shown. The medical image analysis method 400 comprises the following steps:

[0115] Step 401 : performing block division processing on the digital pathology whole slide image obtained by pre-scanning to obtain an initial sub-slice image sequence.

[0116] In some embodiments, the execution subject of the medical image analysis method (eg, a computing device, a pathology slice image analysis system) may perform block division processing on the pre-scanned digital pathology whole slide image to obtain an initial sub-slice image sequence.

[0117] Step 402 : Based on the pre-generated reference staining information, a staining standardization process is performed on each initial sub-slice image in the initial sub-slice image sequence to obtain a sub-slice image sequence.

[0118] In some embodiments, the execution subject may perform dyeing normalization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference dyeing information to obtain the sub-slice image sequence.

[0119] Step 403: Classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence.

[0120] In some embodiments, the execution entity may classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence.

[0121] Step 404: input each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model in sequence to obtain a sub-slice image feature vector sequence.

[0122] In some embodiments, the execution subject may sequentially input each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence. The image feature extraction model is obtained by training an initial image feature extraction model based on sample sub-slice images with foreground objects labeled, and the initial image feature extraction model is a visual neural network.

[0123] Step 405 , based on the sub-slice image feature vector sequence and the preset attention mechanism network, a pathology slice classification confidence result corresponding to the digital pathology full slide image is generated.

[0124] In some embodiments, the execution subject may generate a pathology slice classification confidence result corresponding to the digital pathology whole slide image based on the sub-slice image feature vector sequence and a preset attention mechanism network.

[0125] Step 406: determine the digital pathology whole slide image and the pathology section classification confidence result as pathology section classification information, and store the pathology section classification information for the user to obtain.

[0126] In some embodiments, the execution subject may determine the digital pathology whole slide image and the pathology slice classification confidence result as pathology slice classification information, and store the pathology slice classification information for users to obtain. The users may be, but are not limited to, the following persons: medical image researchers, doctors.

[0127] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the pathological slice image analysis method of some embodiments of the present disclosure, the accuracy of pathological slice classification can be improved. Specifically, the reason for the high category misjudgment rate is that the classification characteristics of the diseased cells are easily confused, and the diseased samples are few, and the image representations of the diseased cells are diverse and cannot be exhausted in the annotation, which makes the annotation quality of the medical image uneven, thereby causing the current mainstream AI-assisted analysis methods to generally have a high category misjudgment rate. Based on this, the medical image analysis based on foreground target optimization of some embodiments of the present disclosure, first, the digital pathology full slide image obtained by pre-scanning is subjected to block division processing to obtain an initial sub-slice image sequence. Thus, the initial sub-slice images of different regions can be obtained, so as to filter out the low-quality interfering sub-slices later. Secondly, based on the pre-generated reference staining information, each initial sub-slice image in the above-mentioned initial sub-slice image sequence is subjected to staining standardization processing to obtain a sub-slice image sequence. Thus, each sub-slice image with the same staining degree can be obtained, reducing the noise caused by the staining difference. Then, each sub-slice image in the above-mentioned sub-slice image sequence is subjected to classification processing to obtain a qualified sub-slice image sequence. Thus, each sub-slice image whose corresponding quality assessment score is greater than a preset score threshold can be obtained. Then, each qualified sub-slice image in the above qualified sub-slice image sequence is sequentially input into a pre-trained image feature extraction model to obtain a sub-slice image feature vector sequence. Among them, the above image feature extraction model is obtained by training the initial image feature extraction model according to the sample sub-slice image after the foreground target is annotated, and the above initial image feature extraction model is a visual neural network. Thus, the interference of the background image can be reduced, and the image features of the prominent foreground target features corresponding to each sub-slice image can be obtained. Afterwards, based on the above sub-slice image feature vector sequence and the preset attention mechanism network, the pathological slice classification confidence result corresponding to the above digital pathological full slide image is generated. Thus, the classification prediction result of the pathological slice can be obtained according to the extracted image features of the prominent foreground target. Finally, the above digital pathological full slide image and the above pathological slice classification confidence result are determined as pathological slice classification information, and the above pathological slice classification information is stored for users to obtain. Therefore, the pathological section image analysis method of some embodiments of the present disclosure can reduce the misjudgment rate of pathological section categories, improve the accuracy of pathological section classification, and can also assist users in interpreting pathological sections, thereby improving the accuracy of users' interpretation of pathological sections.

[0128] It should be noted that the computer-readable medium mentioned above in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0129] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0130] The computer-readable medium may be included in the pathological slice image analysis system; or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the pathological slice image analysis system: performs block division processing on the digital pathological whole slide image obtained by pre-scanning to obtain an initial sub-slice image sequence; performs staining standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference staining information to obtain a sub-slice image sequence; performs classification processing on each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence; inputs each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model in sequence to obtain a sub-slice image feature vector sequence, wherein the image feature extraction model is obtained by training the initial image feature extraction model based on the sample sub-slice image after foreground target annotation, and the initial image feature extraction model is a visual neural network; generates a pathological slice classification confidence result corresponding to the digital pathological whole slide image based on the sub-slice image feature vector sequence and a preset attention mechanism network; determines the digital pathological whole slide image and the pathological slice classification confidence result as pathological slice classification information, and stores the pathological slice classification information for user acquisition.

[0131] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A pathological section image analysis system, comprising: Pathological slice scanning device, pathological image analysis server and pathological information storage server, wherein: The pathology slice scanning device is used to scan the pathology slice to obtain a digital pathology whole slide image, and send the digital pathology whole slide image to the pathology image analysis server; The pathology image analysis server is used to perform block division processing on the received digital pathology full-slide image to obtain an initial sub-slice image sequence; The pathological image analysis server is used to perform dyeing standardization processing on each initial sub-slice image in the initial sub-slice image sequence based on pre-generated reference dyeing information to obtain a sub-slice image sequence; The pathological image analysis server is used to classify each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence, wherein the qualified sub-slice images in the qualified sub-slice image sequence are sub-slice images whose corresponding quality assessment scores are greater than a preset score threshold; The pathological image analysis server is used to input each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model in sequence to obtain a sub-slice image feature vector sequence, wherein the image feature extraction model is obtained by training an initial image feature extraction model based on the sample qualified sub-slice images after foreground object annotation, and the initial image feature extraction model is a visual neural network; The pathological image analysis server is used to generate a pathological slice classification confidence result corresponding to the digital pathological full slide image based on the sub-slice image feature vector sequence and a preset attention mechanism network; The pathological image analysis server is used to determine the digital pathological whole slide image and the pathological section classification confidence result as pathological section classification information, and send the pathological section classification information to the pathological information storage server for storage; The pathology information storage server is used to send the pathology section classification information to the user terminal in response to receiving viewing request information corresponding to the pathology section classification information sent by the user terminal.

2. The pathological section image analysis system according to claim 1, wherein: The pathological image analysis server is configured to: Inputting each sub-slice image in the sub-slice image sequence into a pre-trained slice image quality classification model in sequence to obtain a sub-slice image classification information sequence; Selecting sub-slice image classification information that meets a preset quality category condition from the sub-slice image classification information sequence in turn as target sub-slice image classification information, and obtaining a target sub-slice image classification information sequence; In response to determining that the target sub-slice image classification information sequence meets a preset image quantity condition, for each target sub-slice image classification information in the target sub-slice image classification information sequence, the following steps are performed: Performing image segmentation processing on the sub-slice image corresponding to the classification information of the target sub-slice image to obtain a sub-slice foreground image; Determine the ratio between the number of pixels corresponding to the sub-slice foreground image and the number of pixels of the sub-slice image corresponding to the target sub-slice image classification information as the foreground proportion; In response to determining that the foreground proportion is greater than a preset ratio threshold, the sub-slice image corresponding to the target sub-slice image classification information is determined as a qualified sub-slice image.

3. The pathological section image analysis system according to claim 1, wherein: The pathological section image analysis system further includes: a model training server, wherein the model training server is configured to: Obtaining a sample digital pathology full slide image set and an initial image feature extraction model; Select sample digital pathology whole slide images from the sample digital pathology whole slide image set and perform the following training steps: Performing block division processing on the sample digital pathology full slide image to obtain a sample sub-slice image sequence; Classifying and processing each sample sub-slice image in the sample sub-slice image sequence to obtain a sample qualified sub-slice image sequence; Based on a preset image segmentation model, determining the image foreground annotation information corresponding to each sample qualified sub-slice image in the sample qualified sub-slice image sequence to obtain an image foreground annotation information sequence; Inputting the sample qualified sub-slice images in the sample qualified sub-slice image sequence into the initial image feature extraction model in sequence to obtain a sample sub-slice image feature vector sequence, wherein each sample qualified sub-slice image corresponds to a sample sub-slice image feature vector in the sample sub-slice image feature vector sequence; Determine a foreground optimization loss value corresponding to each sample qualified sub-slice image in the sample qualified sub-slice image sequence based on a preset foreground loss function, the image foreground annotation information sequence and the sample sub-slice image feature vector sequence; The sum of the obtained optimization loss values ​​of each prospect is determined as the total loss value; In response to determining that the total loss value is less than a preset loss threshold, the trained initial image feature extraction model is determined as the image feature extraction model.

4. The pathological section image analysis system according to claim 3, wherein: The model training server is further configured to: In response to determining that the total loss value is greater than or equal to the preset loss threshold, the network parameters of the initial image feature extraction model are adjusted, and sample digital pathology whole slide images are reselected from each sample digital pathology whole slide image information that has never been used, and the adjusted initial image feature extraction model is used as the initial image feature extraction model to perform the training step again.

5. The pathological section image analysis system according to claim 2, wherein: The slice image quality classification model is pre-trained based on a first sample image data set and a second sample image data set, wherein the first sample image data set includes first sample image data of normal quality, and the second sample image data set includes second sample image data of abnormal quality.

6. The pathological section image analysis system according to claim 2, wherein: The pathological image analysis server is further configured to: In response to determining that the target sub-slice image classification information sequence does not satisfy the preset image quantity condition, the digital pathology whole slide image is annotated based on preset low-quality label text.

7. A pathological section image analysis method, applied to the pathological section image analysis system according to any one of claims 1 to 6, the method comprising: Performing block division processing on the digital pathology full-slide image obtained by pre-scanning to obtain an initial sub-slice image sequence; Based on the pre-generated reference staining information, performing staining standardization processing on each initial sub-slice image in the initial sub-slice image sequence to obtain a sub-slice image sequence; Classifying each sub-slice image in the sub-slice image sequence to obtain a qualified sub-slice image sequence; Inputting each qualified sub-slice image in the qualified sub-slice image sequence into a pre-trained image feature extraction model in sequence to obtain a sub-slice image feature vector sequence, wherein the image feature extraction model is obtained by training an initial image feature extraction model based on the sample qualified sub-slice images after foreground object annotation, and the initial image feature extraction model is a visual neural network; Based on the sub-slice image feature vector sequence and a preset attention mechanism network, generating a pathology slice classification confidence result corresponding to the digital pathology full slide image; The digital pathology whole slide image and the pathology section classification confidence result are determined as pathology section classification information, and the pathology section classification information is stored for users to obtain.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to claim 7 is implemented.

Images