Digital pathological image enhancement method and device, equipment and storage medium

Abstract

The invention discloses a digital pathological image enhancement method, device and equipment and a storage medium, and the method comprises the steps: extracting a first image feature of a representative visual field image for training and a second image feature of a first clear pathological image for training through a trained first image processing model; training a second image processing model based on the loss between the first image feature and the second image feature to obtain a trained second image processing model, processing the pre-enhanced slide specimen to obtain at least one new representative view image, and inputting each new representative view image into the trained second image processing model to obtain at least one representative view image; and outputting the new first clear pathological images, and splicing all the new first clear pathological images to obtain a second clear pathological image. According to the method, the representative view in the slide specimen can be better subjected to pathological image enhancement, and details better conforming to biological morphological characteristics are obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of digital pathology image processing, and in particular to a digital pathology image enhancement method, device, equipment and storage medium. Background Art

[0002] The core clinical basis for pathological diagnosis is microscopic imaging of patient tissue or cell specimens. The current mainstream method for acquiring these images is through digital pathology scanners, which scan and digitally archive slides. To ensure efficient scanning of large numbers of specimens, the equipment must strictly control the scanning and imaging time for each slide. Scanning tissue regions at the microscopic scale often requires multiple, high-magnification objective lens movements (e.g., capturing dozens to thousands of fields of view). High-throughput digital pathology scanners, which require speed, are generally unable to perform fine mechanical focusing on each field of view captured by the objective lens (e.g., capturing multiple images up and down the Z axis and calculating the optimal focal plane). A common compromise is the focus map survey, which involves evenly selecting several pre-focus points (e.g., 5-8) within the specimen tissue for fine focusing. The focal plane positions of these points are then used to perform surface fitting on the entire specimen surface, thereby estimating the focal planes of the remaining fields of view. With the focal plane information known in advance, the device can quickly control the objective lens to move between fields of view, eliminating the need for additional focusing steps. Since the thickness distribution of tissue specimens is irregular and non-uniform, the fitted focal plane may not be the best shooting position for the current field of view, or the tissue within the same field of view may be uneven and have no optimal focal plane; in addition, the motion control of the scanner equipment at the micron level (such as the XY axis direction of the slide stage and the Z axis direction of the objective lens) also has inherent errors and repeatability accuracy limitations, which will also affect the clarity of the captured image.

[0003] These issues directly impact the clinical readability of digital slide image files. For example, blurring or out-of-focus areas of key tissues or lesions can hinder accurate judgment and lead to misdiagnosis. To mitigate these serious issues, technical measures are needed to ensure consistent and high-quality digital slide images produced by scanners. The rapid development of artificial intelligence (AI) technology in recent years has provided a new perspective to address this need. Through the design of appropriate image algorithm models, low-quality input images can be post-processed and enhanced to produce outputs with improved clarity and quality (e.g., denoising, deblurring, super-resolution, or image style conversion). However, common image restoration or enhancement models are often trained and optimized based on images of everyday scenes and may be based on generative algorithms. Direct application on pathology images can result in distortion of details or alteration of biological morphological features, a significant concern in diagnostic applications. The industry still lacks an image restoration method that truly leverages pathology image features to enhance digital pathology images consistent with the professional perception of medical professionals. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the purpose of the present invention is to provide a digital pathology image enhancement method, device, equipment and storage medium, which can better enhance the representative field of view in pathology images, obtain details that are more consistent with biological morphological characteristics, and achieve a balance between image sharpness, granularity and visual authenticity.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to a first aspect of the present application, a digital pathology image enhancement method is provided, comprising: Establishing a first image processing model, inputting a first training data set into the first image processing model for training to extract image features, and obtaining a trained first image processing model, wherein the first training data set includes a plurality of specimen image blocks for training; Establishing a second image processing model, inputting a second training data set into the second image processing model, where the second training data set includes a plurality of representative field of view images for training and corresponding first clear pathological images for training, extracting first image features of the representative field of view images for training based on the trained first image processing model, and extracting second image features of the corresponding first clear pathological images for training, calculating a loss between the first image features and the second image features using a loss function, and training the second image processing model based on the loss to obtain a trained second image processing model; The pre-enhanced slide specimen is processed to obtain at least one new representative field of view image, each new representative field of view image is input into the trained second image processing model, a new first clear pathology image is output, and all the new first clear pathology images are spliced ​​together to obtain a second clear pathology image.

[0007] In some embodiments of the present application, based on the aforementioned solution, the basic architecture of the first image processing model is the DINOv2 visual model, and the Vision Transformer backbone network in the DINOv2 visual model is replaced with the Mamba architecture.

[0008] In some embodiments of the present application, based on the above solution, the method for obtaining the specimen image block is: A glass slide specimen is obtained, scanned, and digitally archived to obtain a pathological image. The pathological image is a pyramid image. For the bottom layer of the pyramid image with the original objective lens magnification, a sliding window patch of a specified size is used to traverse the local image without overlap to obtain the specimen image block.

[0009] In some embodiments of the present application, based on the above solution, the method for acquiring the representative field of view image is: Select a representative field of view for the slide specimen, perform precise focusing on the representative field of view, and obtain the optimal focal plane Z; Taking the optimal focal plane Z as the initial reference, a number of underfocus or overfocus images are captured upward and downward along the optimal focal plane Z, and the underfocus or overfocus images are aggregated into a representative field of view image.

[0010] In some embodiments of the present application, based on the aforementioned solution, a method for acquiring a first clear pathological image for training is: Image fusion is performed on multiple images in the representative field of view images used for training in a progressive manner, and the fused image obtained by each image fusion is fused again with the remaining images in sequence until all images in the representative field of view images used for training have been image fused, thereby obtaining the first clear pathological image used for training.

[0011] In some embodiments of the present application, based on the aforementioned solution, the image fusion method includes: Reading two images in the representative visual field image for training or the fused image and one remaining image in the representative visual field image for training in a matrix array format, and performing normalization processing to obtain a first image and a second image; Perform large window mean filtering on the first image and the second image to obtain a first base layer and a second base layer; Obtain a first detail layer and a second detail layer based on the first base layer and the second base layer respectively; Performing standard Laplace filtering and Gaussian filtering on the first detail layer and the second detail layer respectively, to obtain a first structure intensity map and a second structure intensity map; Obtain a first initial binary weight map and a second initial binary weight map by comparing the maximum values ​​of corresponding elements from the first structural strength map to the second structural strength map and from the second structural strength map to the first structural strength map; Using the first image and the second image as guide images for the first initial binary weight map and the second initial binary weight map, respectively, using guided filtering, base layer fusion and detail fusion are performed on the first initial binary weight map and the second initial binary weight map to obtain a first base layer weight and a first detail weight corresponding to the first initial binary weight map, and a second base layer weight and a second detail weight corresponding to the second initial binary weight map; The first base layer and the second base layer are weighted based on the first base layer weight and the second base layer weight to obtain a final base layer, the first detail and the second detail are weighted based on the first detail weight and the second detail weight to obtain a final detail layer, and the final base layer and the final detail layer are fused.

[0012] In some embodiments of the present application, based on the aforementioned solution, a second image processing model is established, and a second training data set is input into the second image processing model. The second training data set includes a number of representative field of view images for training and corresponding first clear pathology images for training. Based on the trained first image processing model, first image features of the slice images for training are extracted, and second image features of the corresponding first clear pathology images for training are extracted. The loss of the first image features and the second image features is calculated using a loss function. The second image processing model is trained based on the loss to obtain a trained second image processing model. The method further includes: Set the trained first image processing model to evaluation mode; using several layers in the registration block in the second image processing model as feature extraction layers, the feature extraction layers being used to extract first image features of a representative field of view image for training and second image features of a first clear pathological image for training; Forward propagation is performed sequentially on the representative visual field image used for training and the first clear pathological image used for training; The content loss and style loss are calculated for each feature extraction layer respectively, and the perceptual loss is obtained by summing the content loss and style loss.

[0013] According to a second aspect of the present application, a digital pathology image enhancement device is provided, comprising: a first image processing model establishment module, configured to establish a first image processing model, input a first training data set into the first image processing model for training to extract image features, and obtain a trained first image processing model, wherein the first training data set includes a plurality of specimen image blocks for training; a second image processing model establishment module, configured to establish a second image processing model, input a second training data set into the second image processing model, the second training data set including a plurality of representative field of view images for training and corresponding first clear pathological images for training, extract first image features of the representative field of view images for training based on the trained first image processing model, and extract second image features of the corresponding first clear pathological images for training, calculate a loss between the first image features and the second image features using a loss function, and train the second image processing model based on the loss to obtain a trained second image processing model; The clear image acquisition module is used to process the pre-enhanced slide specimen to obtain at least one new representative field of view image, input each new representative field of view image into the trained second image processing model, output a new first clear pathology image, and splice all the new first clear pathology images to obtain a second clear pathology image.

[0014] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by a processor, the above method is implemented.

[0015] According to a fourth aspect of the present application, an electronic device is provided, including: one or more processors; The memory is used to store executable instructions of the processor. When the executable instructions are executed by one or more processors, the one or more processors implement the above method.

[0016] The beneficial effects of this application are as follows: The digital pathology image enhancement method, apparatus, device and storage medium provided in the present application extract the first image features and the second image features corresponding to the representative field of view image and the first clear pathology image respectively based on the trained first image processing model, perform similarity-based optimization, train the second image processing model through the loss function, and process the new representative field of view image in the pre-enhanced glass slide specimen by the trained second image processing model. It can better learn the high-level semantic features and texture structure features of the representative field of view image, generate output that is more in line with the human visual system, splice all new first clear pathology images, and obtain a high-quality second clear pathology image output, which is suitable for image restoration of pathology image features, can obtain the optimal field of view, and has high clarity and short image processing time.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a digital pathology image enhancement method according to this embodiment; FIG2( a ) is a schematic diagram of a new slice image of the first embodiment; FIG2( b ) is a schematic diagram of a new first clear pathological image of the first specific embodiment; FIG2( c ) is a schematic diagram of a new slice image of the second specific embodiment; FIG2( d ) is a schematic diagram of a new first clear pathological image according to the second embodiment; FIG2( e ) is a schematic diagram of a new slice image of the second specific embodiment; FIG2( f ) is a schematic diagram of a new first clear pathological image according to the second embodiment; Figure 3 This is a schematic diagram of a digital pathology image enhancement device according to this embodiment; Figure 4 FIG. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Reference will now be made in detail to specific embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Although the present invention will be described in conjunction with specific embodiments, it will be understood that this is not intended to limit the invention to those embodiments. On the contrary, it is intended to cover variations, modifications, and equivalents within the spirit and scope of the present invention as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of the two.

[0020] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Note: The following example is only a specific example and is not intended to limit the embodiments of the present invention to the following specific steps, values, conditions, data, sequence, etc. Those skilled in the art can apply the concepts of the present invention to construct more embodiments not described in this specification by reading this specification.

[0022] Figure 1 A flow chart of a digital pathology image enhancement method according to this embodiment is shown.

[0023] This embodiment provides a digital pathology image enhancement method, including: Step 1: Establish a first image processing model, input a first training data set into the first image processing model for training to extract image features, and obtain a trained first image processing model, wherein the first training data set includes a number of specimen image blocks for training.

[0024] In some implementations of this embodiment, the method for obtaining the specimen image block is: A glass slide specimen is obtained, scanned, and digitally archived to obtain a pathological image. The pathological image is a pyramid image with multiple layers of different magnifications. For the bottom layer of the pyramid image with the original objective lens magnification, a sliding window patch of a specified size is used to traverse the image without overlap to obtain the specimen image block.

[0025] In some implementations of this example, slide specimens with common staining types are collected. Specifically, these include HE, IHC, Papanicolaou (Pap) stains, and other special stains (such as Masson's, Oil Red, AB-PAS, Sirius Red, and Wright's stains). Given that clinical practice primarily focuses on HE staining, as many slide specimens as possible (e.g., more than 300, encompassing a variety of organs, tissues, and disease types) should be collected; immunohistochemistry (IHC) and Papanicolaou (Pap) stains should each comprise at least 100 slides, and special stains should comprise at least 100 slides in total. Slide specimen collection should encompass as many centers as possible to ensure a diverse range of staining and slide preparation techniques, thereby improving the generalization performance of model training.

[0026] In some implementations of this embodiment, due to differences in optical imaging device configuration parameters between digital pathology scanners of different brands and models, the scanned image files may exhibit different visual attributes (such as brightness, contrast, saturation, and channel color differences). To enable the first image processing model to more purposefully learn imaging characteristics, the collected slide specimens are uniformly scanned and digitally archived on the target device.

[0027] In this specific embodiment, the bottom layer is magnified at 20X, and the image is intercepted using a sliding window patch of a specified size without overlapping to obtain a specimen image block. In this specific embodiment, the specimen image block is 448*448 pixels. Only patches with a certain proportion of tissue area are retained, and the rest of the images containing too much background are discarded. In this specific embodiment, the proportion is set to 30%, and judgment is made through binarization.

[0028] The constructed patch dataset of this embodiment may reach the millions level.

[0029] In some implementations of this embodiment, the first training dataset does not need to be labeled, and self-supervised image model training is performed. There are many options for image self-supervised models, such as DINOv2, BYOL, and MAE.

[0030] In some implementations of this embodiment, the first image processing model selects a structure that combines DINOv2 and Mamba (linear state space model). Specifically, the first image processing model's underlying architecture is the DINOv2 visual model. This embodiment replaces the Vision Transformer (ViT) backbone network in the DINOv2 visual model with the Mamba architecture, while maintaining the remaining architecture. This approach not only preserves DINOv2's strong transfer and semantic structure extraction capabilities, but also leverages the Mamba architecture's computational efficiency. Its implicit state space propagation mechanism is more stable on long sequences than the Transformer's self-attention mechanism, and has lower computational cost (compared to ViT's quadratic complexity (O(N²)), Mamba's is linear complexity (O(N)). This also enables the first training dataset to select patches with a larger field of view while maintaining the same computing power. Furthermore, DINOv2 provides a structured, buffered, and adaptive supervision mechanism, making it ideal for stable training of Mamba, a novel architecture that is highly sensitive to prior training. This optimization approach combines the strengths of both architectures and addresses their respective bottlenecks, achieving both excellent performance and computational efficiency in high-resolution pathology image processing, making it suitable for deployment on digital pathology scanners with limited computing power.

[0031] Step 2: Establish a second image processing model and input the second training data set into the second image processing model. The second training data set includes several representative field of view images for training and corresponding first clear pathological images for training. Based on the trained first image processing model, extract the first image features of the representative field of view images for training, and extract the second image features of the corresponding first clear pathological images for training. Calculate the loss of the first image features and the second image features through the loss function, train the second image processing model based on the loss, and obtain a trained second image processing model.

[0032] In some implementations of this embodiment, the second image processing model is an end-to-end enhancement model. Currently, there is a lack of cost-effective solutions that maintain overall scanning speed. For example, certain fast-focusing technologies (such as liquid lenses or voice coil motors) require significant additional hardware costs and still require focusing each field of view. As a result, overall scanning speeds lag far behind the fast linear scanning of traditional focus map surveys. Furthermore, employing multi-layer fusion scanning to achieve optimal image quality significantly reduces scanning speed.

[0033] This embodiment uses the second image processing model to focus on the representative field of view image, which can achieve deblurring and enhancement processing of the pathological image, greatly improving the processing speed without requiring significant additional hardware costs.

[0034] In some implementations of this embodiment, the method for obtaining the representative visual field image is: Select a representative field of view for the slide specimen, perform precise focusing on the representative field of view, and obtain the optimal focal plane Z; Taking the optimal focal plane Z as the initial reference, several underfocus or overfocus images are taken upward and downward along the optimal focal plane Z, and the underfocus or overfocus images are combined into a representative field of view image to cover the thickness variation of the slide specimen.

[0035] In some implementations of this embodiment, the representative field of view is one or more. In this specific embodiment, 3-6 are selected according to the size of the tissue specimen, and the camera capture resolution is 2048*2448.

[0036] In some implementations of this embodiment, the method for performing a precise focusing operation on the representative field of view is: calculating a Brenner Gradient based on multiple images of the representative field of view.

[0037] In this specific embodiment, 8 images were acquired in each direction, both upward and downward from the optimal focal plane, with a step size of 0.5 μm. This resulted in a total of 8+1+8=17 images for each representative field of view. The ±4 μm range covers the thickness variations of most slide specimens.

[0038] In some implementations of this embodiment, considering that the optimal focal plane Z may correspond to the global clearest image rather than the clearest image in all details, image fusion is performed on all underfocus and / or overfocus images of each representative field of view, and the first clear pathological image obtained is used as the benchmark of the representative field of view.

[0039] In some implementations of this embodiment, the method for obtaining the first clear pathological image for training is: Image fusion is performed on multiple images in the representative field of view images used for training in a progressive manner, and the fused image obtained by each image fusion is fused again with the remaining images in sequence until all images in the representative field of view images used for training have been image fused, thereby obtaining the first clear pathological image used for training.

[0040] Specifically, the above operation is repeated for the sequence images of the same field of view (the first image is fused with the second image, and the result is fused with the third image, and so on). Until the final high-quality fused image (ground truth) is obtained Image fusion methods include: Read two images from the representative field of view image used for training or the fused image and one remaining image from the representative field of view image used for training in matrix array format. Here, the size is 2048*2448*3, and perform normalization (dividing by 255) to obtain the first image and the second image. Performing a large window mean filter (uniform filter) on the first image and the second image to obtain a first base layer base1 and a second base layer base2; Obtain a first detail layer detail1 and a second detail layer detail2 based on the first base layer base1 and the second base layer base2 respectively; Performing standard Laplacian filtering and Gaussian filtering on the first detail layer detail1 and the second detail layer detail2 respectively, to obtain a first structure intensity map and a second structure intensity map; Obtain a first initial binary weight map and a second initial binary weight map by comparing the maximum values ​​of corresponding elements from the first structural strength map to the second structural strength map and from the second structural strength map to the first structural strength map; Using the first image and the second image as guide images for the first initial binary weight map and the second initial binary weight map, respectively, using a guided filter, base layer fusion and detail fusion are performed on the first initial binary weight map and the second initial binary weight map, thereby obtaining a first base layer weight wb1 and a first detail weight wb2 corresponding to the first initial binary weight map, and a second base layer weight wd1 and a second detail weight wd2 corresponding to the second initial binary weight map; The first base layer and the second base layer are weighted based on the first base layer weight and the second base layer weight to obtain a final base layer, the first detail and the second detail are weighted based on the first detail weight and the second detail weight to obtain a final detail layer, and the final base layer and the final detail layer are fused.

[0041] Specifically, large windows and strong regularization are used for base-level fusion, while small windows and weak regularization are used for detail fusion.

[0042] In some implementations of this embodiment, before weighting the first base layer and the second base layer based on the first base layer weight and the second base layer weight, it is necessary to normalize the first base layer weight and the second base layer weight, and the calculation formula is:

[0043]

[0044] In some implementations of this embodiment, the first detail and the second detail are weighted based on the first detail weight and the second detail weight to obtain a final detail layer. The calculation formula is:

[0045]

[0046] In some implementations of this embodiment, the final base layer and the final detail layer are fused, and the calculation formula is:

[0047] In some implementations of this embodiment, all underfocus and / or overfocus images captured in each representative field of view serve as input to a second image processing module, and the first clear pathological image serves as the ground truth of the second image processing module; the second image processing module reconstructs the learned input into a pattern regularity of the ground truth.

[0048] In some implementations of this embodiment, to avoid artifacts from generative models that do not conform to biologically realistic texture features, the second image processing model utilizes an end-to-end image restoration model. This embodiment uses NAFNet (Nonlinear Activation Free Network), a lightweight model based on the classic UNET architecture. This model's key features include its elimination of nonlinear activation computations and its streamlined and efficient residual block design, which simplifies the inference process and improves computational speed. It is well-suited for inference on large input images and deployment on edge devices (it can support fast inference on large inputs even on non-high-end GPUs).

[0049] Unlike the standard NAFNet training process, this embodiment innovatively designs a loss function specifically for pathological images. This loss function combines the design principles of perceptual loss with the first image processing model trained in step S2. This loss function better captures the unique hierarchical features of pathological images and adjusts the output to produce high-quality, clear images that are more suitable for the user's visual perception. (The original perceptual loss was implemented based on a pre-trained model for everyday images and is not suitable for direct transfer to the medical field.)

[0050] In some implementations of this embodiment, a second image processing model is established, a second training data set is input into the second image processing model, the second training data set includes a number of representative field of view images for training and corresponding first clear pathological images for training, first image features of the representative field of view images for training are extracted based on the trained first image processing model, and second image features of the corresponding first clear pathological images for training are extracted, a loss function is used to calculate the loss of the first image features and the second image features, and the second image processing model is trained based on the loss to obtain a trained second image processing model. The method further includes: Set the trained first image processing model to evaluation mode; using several layers in the registration block in the second image processing model as feature extraction layers, the feature extraction layers being used to extract first image features of a representative field of view image for training and second image features of a first clear pathological image for training; Forward propagation is performed sequentially on the representative visual field image used for training and the first clear pathological image used for training; The content loss and style loss are calculated for each feature extraction layer respectively, and the perceptual loss is obtained by summing the content loss and style loss.

[0051] In some implementations of this embodiment, the representative field of view image used for training and the first clear pathological image used for training are pre-processed before input to keep consistent with the input settings of the first image processing model training. The preprocessing method is normalization and standardization, etc., which is not limited in this embodiment.

[0052] In some implementations of this embodiment, the first image features and the second image features include texture details and edge features, structure and partial semantics, global semantics and scene consistency, etc. This embodiment does not limit this and generates output that is more consistent with the human visual system.

[0053] Specifically, the 3rd, 6th, and 11th layers in the registered blocks are selected as feature extraction layers, and the first image features and the second image features are dynamically extracted from these layers, where the 3rd layer corresponds to texture details and edge features, the 6th layer corresponds to structure and partial semantics, and the 9th layer corresponds to global semantics and scene consistency.

[0054] In some implementations of this embodiment, content loss (ContentLoss) and style loss (Style Loss) are calculated for each feature extraction layer respectively: the content loss uses L1 loss (absolute difference) to calculate the content difference of the extracted feature pairs, and the style loss obtains the correlation structure (representing the style) between the features through the Gram matrix and then calculates the L1 difference; each type of loss is defined as the sum of the content loss and style loss of each corresponding feature extraction layer.

[0055] In some implementations of this embodiment, since the representative field of view image has a high resolution (2048*2448), the second training dataset is input into the second image processing model for training, further comprising: During each training round, a fixed-size crop is randomly taken from the representative field of view image and the first clear pathology image used for training to maximize data enrichment for the model (pictures are cropped at the same coordinates to obtain paired images). During training, structural similarity (SSMI) and peak signal-to-noise ratio (PSNR) are used as performance metrics. Once the trained weights of the second image processing model are stabilized, they are saved. Because the representative field of view image will be fully processed in practice, the weights of the second image processing model must be adjusted to the input size of the representative field of view image (2048*2448). To facilitate computational acceleration on Nvidia GPUs, the adjusted weights of the second image processing model are converted to TensorRT format.

[0056] Step 3: Process the pre-enhanced slide specimen to obtain at least one new representative field of view image, input each new representative field of view image into the trained second image processing model, output a new first clear pathology image, and splice all new first clear pathology images to obtain a second clear pathology image, which is a pyramid image.

[0057] In some implementations of this embodiment, during the scanner's imaging process, the objective camera captures the field of view image and saves the data to memory. The software backend then uses a separate thread to invoke a trained second image processing model, a deblurring model, for processing. The image stitching process then proceeds according to the existing workflow. In this approach, scanning and deblurring are performed asynchronously in separate threads (the deblurring calculation inherently consumes time, such as 0.5 to 1 second). The final output of the second clear pathology image will experience a delay compared to traditional scanning methods, but this does not affect the normal mechanical scanning process of the slide specimen.

[0058] During the scanner's scanning process, the objective camera captures the field of view image and saves the data locally, maintaining the same workflow. If the output image clarity and quality do not meet the required requirements, the user can manually deblur the sample. The software background then activates a separate thread to invoke the model in step S4 to process the entire locally saved field of view image data, reassembling it to generate a new pyramid map. This approach also does not affect the inherent sample scanning process time.

[0059] It should be emphasized that the process ideas described in the present invention are not limited to image deblurring, but can also be applied to the enhancement or restoration of other pathological images, such as super-resolution, denoising and dehazing, etc., only need to construct a second training data set for different purposes with corresponding means.

[0060] In a specific implementation of this embodiment, as shown in FIG. 2 ( a ) to FIG. 2 ( f ), a digital pathology image enhancement method provided by this embodiment is used to identify three new slice images and input three new first clear pathology images.

[0061] Based on the same inventive concept, Figure 3 As shown, the embodiment of the present application further provides a digital pathology image enhancement device, the device comprising: A first image processing model establishment module 201 is used to establish a first image processing model, input a first training data set into the first image processing model for training to extract image features, and obtain a trained first image processing model, wherein the first training data set includes a plurality of specimen image blocks for training; A second image processing model establishing module 202 is configured to establish a second image processing model, input a second training data set into the second image processing model, the second training data set including a plurality of representative field of view images for training and corresponding first clear pathological images for training, extract first image features of the representative field of view images for training based on the trained first image processing model, and extract second image features of the corresponding first clear pathological images for training, calculate the loss of the first image features and the second image features using a loss function, and train the second image processing model based on the loss to obtain a trained second image processing model; The clear image acquisition module 203 is used to process the pre-enhanced slide specimen to obtain at least one new representative field of view image, input each new representative field of view image into the trained second image processing model, output a new first clear pathology image, and splice all the new first clear pathology images to obtain a second clear pathology image.

[0062] Specifically, this embodiment corresponds one-to-one to the above method embodiment, and the functions of each module have been described in detail in the corresponding method embodiment, so they will not be repeated here.

[0063] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. The computer program includes executable instructions. When the executable instructions are executed by the processor 301, the above method is implemented.

[0064] The present invention may implement all or part of the above-described method processes by instructing related hardware through a computer program. The computer program may be stored in a computer-readable storage medium. When executed by processor 301, the computer program may implement the steps of each of the above-described method embodiments. The computer program includes computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include any entity or device capable of carrying computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunications signals, and software distribution media. It should be noted that the content of computer-readable media may be appropriately expanded or reduced based on the requirements of legislation and patent practice within a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, computer-readable media do not include electric carrier signals or telecommunications signals.

[0065] Based on the same inventive concept, the present application also provides an electronic device, such as Figure 4 As shown, including: One or more processors 301; The memory 302 is used to store executable instructions of the processor 301. When the executable instructions are executed by one or more processors 301, the one or more processors 301 implement the above method.

[0066] The electronic device is implemented as a general-purpose computing device. Components of the electronic device may include, but are not limited to, the at least one processor 301, the at least one memory 302, and a bus 303 connecting different device components (including the memory 302 and the processor 301).

[0067] The processor 301 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. Processor 301 is the control center of the computer device, connecting various components of the entire computer device using various interfaces and circuits.

[0068] Memory 302 can be used to store computer programs and / or modules. Processor 301 implements various functions of the computer device by running or executing the computer programs and / or modules stored in memory 302 and accessing data stored in memory 302. Memory 302 may primarily include a program storage area and a data storage area. The program storage area may store applications required to operate the device and at least one function (such as sound playback and image playback); the data storage area may store data generated based on the use of the mobile phone (such as audio data and video data). Memory 302 may also include high-speed random access memory (RAM) and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0069] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, servers, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0070] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (apparatus), servers, and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor 301 of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor 301 of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0071] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0073] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A digital pathology image enhancement method, characterized in that: include: Establishing a first image processing model, inputting a first training data set into the first image processing model for training to extract image features, and obtaining a trained first image processing model, wherein the first training data set includes a plurality of specimen image blocks for training; Establishing a second image processing model, inputting a second training data set into the second image processing model, wherein the second training data set includes a plurality of representative field of view images for training and corresponding first clear pathological images for training, extracting first image features of the representative field of view images for training and second image features of the corresponding first clear pathological images for training based on the trained first image processing model, calculating a loss between the first image features and the second image features using a loss function, and training the second image processing model based on the loss to obtain a trained second image processing model; The pre-enhanced slide specimen is processed to obtain at least one new representative field of view image, each new representative field of view image is input into the trained second image processing model, a new first clear pathology image is output, and all the new first clear pathology images are spliced ​​together to obtain a second clear pathology image.

2. The method according to claim 1, wherein: The infrastructure of the first image processing model is the DINOv2 visual model, and the VisionTransformer backbone network in the DINOv2 visual model is replaced with the Mamba architecture.

3. The method according to claim 1, characterized in that The method for obtaining the specimen image block is: A glass slide specimen is obtained, scanned, and digitally archived to obtain a pathological image, wherein the pathological image is a pyramid diagram. For the bottom layer of the pyramid diagram at the original objective lens magnification, a sliding window patch of a specified size is used to traverse the local image without overlap to obtain a specimen image block.

4. The method according to claim 1, wherein The method for acquiring the representative visual field image is: Selecting a representative field of view for the slide specimen, performing a precise focusing operation on the representative field of view to obtain the optimal focal plane Z; Taking the optimal focal plane Z as an initial reference, a number of underfocus or overfocus images are captured upward and downward along the optimal focal plane Z, and the plurality of underfocus or overfocus images are aggregated into a representative field of view image.

5. The method according to claim 1, wherein The method for obtaining the first clear pathological image for training is: Image fusion is performed on multiple images in the representative field of view image used for training in a progressive manner, and the fused image obtained by each image fusion is fused again with the remaining images in sequence until all images in the representative field of view image used for training have been image fused, thereby obtaining the first clear pathological image used for training.

6. The method according to claim 5, characterized in that The image fusion method includes: Reading two images or a fused image and a remaining image in the representative visual field image for training in a matrix array format, and performing normalization processing to obtain a first image and a second image; Perform large window mean filtering on the first image and the second image to obtain a first base layer and a second base layer; Obtain a first detail layer and a second detail layer based on the first base layer and the second base layer respectively; Performing standard Laplace filtering and Gaussian filtering on the first detail layer and the second detail layer respectively, to obtain a first structure intensity map and a second structure intensity map; Obtain a first initial binary weight map and a second initial binary weight map by comparing the maximum values ​​of corresponding elements from the first structural strength map to the second structural strength map and from the second structural strength map to the first structural strength map; Using the first image and the second image as guide images for the first initial binary weight map and the second initial binary weight map, respectively, using guided filtering, base layer fusion and detail fusion are performed on the first initial binary weight map and the second initial binary weight map to obtain a first base layer weight and a first detail weight corresponding to the first initial binary weight map, and a second base layer weight and a second detail weight corresponding to the second initial binary weight map; The first base layer and the second base layer are weighted based on the first base layer weight and the second base layer weight to obtain a final base layer, the first detail and the second detail are weighted based on the first detail weight and the second detail weight to obtain a final detail layer, and the final base layer and the final detail layer are fused.

7. The method according to claim 1, characterized in that The method further includes establishing a second image processing model, inputting a second training data set into the second image processing model, wherein the second training data set includes a plurality of representative field of view images for training and corresponding first clear pathological images for training; extracting first image features of the slice images for training and second image features of the corresponding first clear pathological images for training based on the trained first image processing model; calculating the loss of the first image features and the second image features using a loss function; and training the second image processing model according to the loss to obtain a trained second image processing model. Set the trained first image processing model to evaluation mode; using several layers in the registration block in the second image processing model as feature extraction layers, wherein the feature extraction layers are used to extract first image features of the representative field of view image used for training and second image features of the first clear pathological image used for training; Forward propagation is performed sequentially on the representative visual field image used for training and the first clear pathological image used for training; The content loss and style loss are calculated for each feature extraction layer respectively, and the perceptual loss is obtained by summing the content loss and style loss.

8. A digital pathology image enhancement device, characterized in that: The device comprises: a first image processing model establishment module, configured to establish a first image processing model, input a first training data set into the first image processing model for training to extract image features, and obtain a trained first image processing model, wherein the first training data set includes a plurality of specimen image blocks for training; a second image processing model establishment module, configured to establish a second image processing model, input a second training data set into the second image processing model, wherein the second training data set includes a plurality of representative field of view images for training and corresponding first clear pathological images for training; based on the trained first image processing model, extract first image features of the representative field of view images for training, and extract second image features of the corresponding first clear pathological images for training; calculate the loss of the first image features and the second image features using a loss function; and train the second image processing model based on the loss to obtain a trained second image processing model; The clear image acquisition module is used to process the pre-enhanced slide specimen to obtain at least one new representative field of view image, input each new representative field of view image into the trained second image processing model, output a new first clear pathology image, and splice all the new first clear pathology images to obtain a second clear pathology image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program includes executable instructions, and when the executable instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A memory for storing executable instructions of the processor, wherein when the executable instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images