Method, device and equipment for pathological section image splicing and storage medium

Abstract

The application discloses a pathological section image splicing method, device and equipment and a storage medium, and belongs to the technical field of image processing. The method comprises the following steps: displaying a plurality of thumbnail section images; the thumbnail section image is a thumbnail of a pathological section image; in response to a splicing operation on the thumbnail section image, moving the thumbnail section image for splicing; in response to a splicing completion instruction, recording position information of each thumbnail section image in the plurality of thumbnail section images; and splicing pathological section images corresponding to each thumbnail section image based on the recorded position information to obtain a pathological section splicing image. The pathological section image operation efficiency can be improved by adopting the application.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for stitching together pathological slide images. Background Technology

[0002] In the field of telemedicine, remote experts can analyze tissue lesions using pathological slide images. However, a single pathological slide image can only represent a small area of ​​tissue, which may not be sufficient for accurate analysis of tissue lesions. In such cases, it is necessary to acquire multiple pathological slide images of the tissue to be analyzed and stitch these images together to obtain a stitched image of the pathological slide that can present a larger area of ​​tissue.

[0003] Currently, software capable of stitching pathological slides all directly stitch together the original images of the slides. Specifically, the software loads and displays the original images of each pathological slide, and the pathologist moves these original images to stitch them together.

[0004] The original images of pathological slides are in the gigabyte (GB) range. Loading and displaying the original images of multiple pathological slides at the same time requires a lot of memory resources, which may cause computer lag and affect the efficiency of pathologists in image stitching. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for stitching pathological slide images, which can solve the problem of low efficiency in pathological slide image operation in related technologies. The technical solution is as follows:

[0006] In a first aspect, a method for stitching together pathological slide images is provided, the method comprising:

[0007] Displaying multiple thumbnail slice images; the thumbnail slice images are thumbnails of pathological slice images;

[0008] In response to the stitching operation of the thumbnail slice image, the thumbnail slice image is moved and stitched together;

[0009] In response to the stitching completion command, the position information of each thumbnail slice image in the plurality of thumbnail slice images is recorded;

[0010] Based on the recorded location information, the pathological slide images corresponding to each thumbnail slide image are stitched together to obtain a stitched pathological slide image.

[0011] In one possible implementation, the step of stitching together the pathological slide images corresponding to each thumbnail slide image based on the recorded location information to obtain a stitched pathological slide image includes:

[0012] For each pathological slide image, based on the initial position information of the pathological slide image, the recorded position information of the corresponding thumbnail slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image, the affine transformation parameters corresponding to the pathological slide image are determined, and the affine transformation is performed on the pathological slide image according to the affine transformation parameters.

[0013] The pathological slide images after each affine transformation are stitched together to obtain a stitched pathological slide image.

[0014] In one possible implementation, stitching together the affine-transformed pathological slide images to obtain a stitched pathological slide image includes:

[0015] When multiple pathological slide images after affine transformation have overlapping areas, if the overlapping area of ​​the topmost pathological slide image belongs to the foreground, then the alpha channel value of the overlapping area of ​​the lower pathological slide image is adjusted to 0.

[0016] If the overlapping area of ​​the topmost pathological slide image belongs to the background, then the α channel value of the overlapping area of ​​the topmost pathological slide image is adjusted to 0.

[0017] In one possible implementation, before displaying the multiple thumbnail slice images, the method further includes:

[0018] Multiple pathological slide images are downsampled according to a preset downsampling factor to obtain thumbnail slide images corresponding to the pathological slide images.

[0019] In one possible implementation, displaying multiple thumbnail slice images includes:

[0020] Receive a triggered request to acquire pathological slide images;

[0021] Based on the storage path in the pathological slide image acquisition request, read the thumbnail slide images corresponding to the pathological slide images in the same folder under the storage path;

[0022] The retrieved thumbnail slice images are displayed on the interface.

[0023] In one possible implementation, after obtaining the thumbnail slice images corresponding to the pathological slice images, the method further includes:

[0024] For each thumbnail slice image, determine the foreground image in the thumbnail slice image;

[0025] The α channel value of the images other than the foreground image in the thumbnail slice image is adjusted to 0.

[0026] In one possible implementation, determining the foreground image in the thumbnail slice image includes:

[0027] Calculate the standard deviation of the RGB color channels for each pixel in the thumbnail image;

[0028] The image composed of pixels whose standard deviation is greater than the standard deviation threshold is used as the foreground image in the thumbnail slice image.

[0029] In one possible implementation, before displaying the multiple thumbnail slice images, the method further includes:

[0030] For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image;

[0031] In the thumbnail image, connected regions with an area smaller than a preset threshold are deleted.

[0032] Secondly, an apparatus for displaying pathological slide images is provided, the apparatus comprising:

[0033] The display module is used to display multiple thumbnail slice images; the thumbnail slice images are thumbnails of pathological slice images;

[0034] A manual stitching module is used to move the thumbnail slice image for stitching in response to a stitching operation on the thumbnail slice image;

[0035] A recording module is used to record the position information of each thumbnail slice image in the plurality of thumbnail slice images in response to the stitching completion command;

[0036] The automatic stitching module is used to stitch together the pathological slide images corresponding to each thumbnail slide image based on the recorded location information to obtain a stitched pathological slide image.

[0037] In one possible implementation, the automatic stitching module is used for:

[0038] For each pathological slide image, based on the initial position information of the pathological slide image, the recorded position information of the corresponding thumbnail slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image, the affine transformation parameters corresponding to the pathological slide image are determined, and the affine transformation is performed on the pathological slide image according to the affine transformation parameters.

[0039] The pathological slide images after each affine transformation are stitched together to obtain a stitched pathological slide image.

[0040] In one possible implementation, the automatic stitching module is used for:

[0041] When multiple pathological slide images after affine transformation have overlapping areas, if the overlapping area of ​​the topmost pathological slide image belongs to the foreground, then the alpha channel value of the overlapping area of ​​the lower pathological slide image is adjusted to 0.

[0042] If the overlapping area of ​​the topmost pathological slide image belongs to the background, then the α channel value of the overlapping area of ​​the topmost pathological slide image is adjusted to 0.

[0043] In one possible implementation, the device further includes a downsampling module for:

[0044] Multiple pathological slide images are downsampled according to a preset downsampling factor to obtain thumbnail slide images corresponding to the pathological slide images.

[0045] In one possible implementation, the display module is used for:

[0046] Displays multiple thumbnail slice images, including:

[0047] Receive a triggered request to acquire pathological slide images;

[0048] Based on the storage path in the pathological slide image acquisition request, read the thumbnail slide images corresponding to the pathological slide images in the same folder under the storage path;

[0049] The retrieved thumbnail slice images are displayed on the interface.

[0050] In one possible implementation, the apparatus further includes a foreground segmentation module for:

[0051] For each thumbnail slice image, determine the foreground image in the thumbnail slice image;

[0052] The α channel value of the images other than the foreground image in the thumbnail slice image is adjusted to 0.

[0053] In one possible implementation, the foreground segmentation module is used for:

[0054] Calculate the standard deviation of the RGB color channels for each pixel in the thumbnail image;

[0055] The image composed of pixels whose standard deviation is greater than the standard deviation threshold is used as the foreground image in the thumbnail slice image.

[0056] In one possible implementation, the device further includes a speckle noise removal module for:

[0057] For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image;

[0058] In the thumbnail image, connected regions with an area smaller than a preset threshold are deleted.

[0059] Thirdly, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method for stitching pathological slide images as described in the first aspect above.

[0060] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method for stitching pathological slide images as described in the first aspect above.

[0061] Fifthly, a computer program product is provided, the computer program product storing at least one instruction, the at least one instruction being loaded and executed by the processor to implement the method for stitching pathological slide images as described in the first aspect above.

[0062] The beneficial effects of the technical solutions provided in this application include at least the following:

[0063] In this embodiment, instead of loading and displaying the original pathological slide image, a thumbnail image of the pathological slide is displayed. Since the file size of the thumbnail image is smaller than the pathological slide image, loading and displaying the thumbnail image does not require excessive memory and will not cause computer lag. Furthermore, the user can move the displayed thumbnail images for stitching, resulting in a smoother process. After stitching the thumbnail images, the position information of each thumbnail image is recorded. Then, based on the position information of the thumbnail images, the pathological slide image is stitched together. The entire process does not require loading and displaying the pathological slide image, effectively avoiding the computer lag problem caused by excessive memory consumption due to loading and displaying the pathological slide image, thus allowing the user to complete the stitching of pathological slide images more smoothly and efficiently. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of a method for stitching pathological slide images provided in an embodiment of this application;

[0066] Figure 2 This is a flowchart of a method for stitching pathological slide images provided in an embodiment of this application;

[0067] Figure 3 This is a schematic diagram of the interface of an application provided in an embodiment of this application;

[0068] Figure 4 This is a schematic diagram of the interface of an application provided in an embodiment of this application;

[0069] Figure 5 This is a schematic diagram of a stitched pathological slide image provided in an embodiment of this application;

[0070] Figure 6 This is a schematic diagram of a thumbnail slice stitched image provided in an embodiment of this application;

[0071] Figure 7 This is a schematic diagram of an affine transformation of a pathological slide image provided in an embodiment of this application;

[0072] Figure 8 This is a schematic diagram of a device structure for creating a pathological slide image, provided in an embodiment of this application.

[0073] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0075] This application provides a method for stitching together pathological slide images, which can be implemented using a computer device. The computer device can be a laptop, desktop computer, tablet computer, mobile phone, etc. The following describes the method in conjunction with... Figure 1 The method for stitching pathological slide images provided in the embodiments of this application will be briefly described. For example... Figure 1 As shown, for example, if five pathological slide images (pathological slide image 1, pathological slide image 2, pathological slide image 3, pathological slide image 4, and pathological slide image 5) need to be stitched together, the user first downsamples each pathological slide image using the target application to obtain corresponding thumbnail slide images (thumbnail slide image 1, thumbnail slide image 3, thumbnail slide image 4, and thumbnail slide image 5). Then, the user manually moves each thumbnail slide image using the target application to stitch them together, obtaining a stitched thumbnail image, and records the position information of the thumbnail images. Finally, the target application stitches the pathological slide images together based on the position information of each thumbnail slide image to obtain the stitched pathological slide image.

[0076] The following is combined Figure 2 The processing flow of the pathological slide image stitching method shown in the embodiments of this application will be described. See [link to relevant documentation]. Figure 2 The method may include the following processing steps:

[0077] Step 201: Obtain the thumbnail slice images corresponding to the multiple pathological slice images.

[0078] Among them, the thumbnail slice image is a thumbnail of the pathological slice image.

[0079] In practice, users scan pathological slides to obtain pathological slide images. Each pathological slide image can then be named and saved in a folder. When it is necessary to stitch these pathological slide images together, the user first opens the target application for image stitching installed on their computer.

[0080] See Figure 3 This illustrates the main interface of a target application, which is the interface displayed when a user opens the target application. The main interface includes an image display area and a function area. The image display area shows thumbnail slices of images, which the user can interact with. The function area of ​​the main interface may display the following options: File, Styles, Stitching & Annotation, About, Layout Settings, Window Management, View, Scroll Bar, etc.

[0081] After the user selects the "File" option in the main interface's function area, sub-function options are displayed below the "File" option. For example... Figure 3 As shown, the file options include sub-options for "Open" and "Exit". The "Open" option also displays the shortcut text "Ctrl+O".

[0082] When the user selects the "Open" option or presses the "Ctrl+O" shortcut, the target application displays a file selection interface. The user then selects the target folder containing the pathology slide images, triggering a pathology slide image retrieval request. This request includes the storage path of the pathology slide images. The computer device then reads the pathology slide images from the target folder based on the storage path specified in the retrieval request.

[0083] The pathological slide images can be in TIFF (Tag Image File Format), and TIFF images can be read at different resolutions depending on the requirements. When reading pathological slide images, a thumbnail image of the pathological slide image at a preset downsampling factor can be retrieved. The preset downsampling factor can be pre-set by the technician as needed; for example, it can be set to 64x, meaning the resolution of the retrieved thumbnail image is 1 / 64th of the pathological slide image.

[0084] In one possible implementation, the downsampling factor can also be specified by the user. The target application can display a downsampling factor setting option, which can be located in the main interface's function area or in the file selection interface. The user can then select the downsampling factor setting option to set the downsampling factor. After the user selects the downsampling factor setting option, the target application can detect the selection and display a downsampling factor input box. The user can then enter the desired target downsampling factor in the downsampling input box. Consequently, when reading the thumbnail image, the target application can read the thumbnail image with the target downsampling factor.

[0085] Furthermore, to prevent the downsampling factor from being too large or too small, an upper limit and a lower limit for the downsampling factor can be set. If the downsampling factor entered by the user is detected to be less than the lower limit, the user will be prompted that the current input downsampling factor is too small. For example, the message "The input downsampling factor is too small. Please adjust and re-enter" can be displayed. If the downsampling factor entered by the user is detected to be greater than the lower limit, the user will be prompted that the current input downsampling factor is too large. For example, the message "The input downsampling factor is too large. Please adjust and re-enter" can be displayed.

[0086] In another possible scenario, where the downsampling factor is specified by the user, multiple candidate downsampling factors can be provided for the user to choose from. Accordingly, after the user selects the downsampling factor setting option, the target application can detect this selection and display various candidate downsampling factors. The user can then select the desired downsampling factor from these options, and the target application can detect this selection and use the chosen downsampling factor as the downsampling factor when reading the thumbnail slice image.

[0087] In addition, if the user does not set the downsampling factor through the downsampling factor setting option, the default preset downsampling factor will be used when reading the thumbnail slice image of the pathological slide image.

[0088] In one possible implementation, to improve efficiency, the target application can read thumbnail images of each pathological slide in parallel.

[0089] In one possible implementation, during the scanning of pathological sections, noise inevitably appears in the images. This noise could be man-made spots or uneven gray backgrounds caused by imaging. If the background is opaque, it may result in one thumbnail image's background obscuring the tissue area of ​​another thumbnail image during manual stitching. To avoid this, the alpha (α) channel value of the background can be adjusted to make it transparent. Accordingly, the following processing can be performed before displaying the read thumbnail image:

[0090] For each thumbnail slice image, determine the foreground image in the thumbnail image, and adjust the α channel value of the images other than the foreground image in the thumbnail slice image to a preset value.

[0091] In cases where the pathological slide image is obtained by scanning the pathological slide, the foreground image of the thumbnail slide image is the tissue image.

[0092] In practice, the foreground image in the thumbnail slice image can be determined by performing image standard deviation thresholding on the thumbnail slice image. The formula is as follows:

[0093] M0 = std(I) > threshold

[0094] The formula means: for any pixel in the thumbnail slice image, calculate the standard deviation of the RGB three color channel values ​​of that pixel, and then select pixels in the thumbnail slice image whose standard deviation is greater than a preset standard deviation threshold. The image composed of these selected pixels is the foreground image of the thumbnail slice image. In the above formula, M0 represents the foreground image, std() represents calculating the standard deviation, I represents the thumbnail slice image, and threshold represents the preset standard deviation threshold.

[0095] Therefore, the image other than the foreground image in the thumbnail slice is the background image. Then, the α channel value of each pixel in the background image is adjusted to a preset value to make the background transparent or nearly transparent. For example, the preset value can be 0.

[0096] In one possible implementation, when the pathological slide image is an image obtained by scanning a pathological slide, the holes in the pathological tissue in the thumbnail slide image can be filled before displaying the thumbnail slide image.

[0097] In practice, a fill hole algorithm can be used to fill the holes in the tissue in each thumbnail slice image.

[0098] In one possible implementation, speckle noise in the background areas of each thumbnail image can be removed before displaying the acquired thumbnail slice images.

[0099] In implementation, the "remove small hole" algorithm can be used to remove speckle noise from the background areas of each thumbnail image. The algorithm's processing can be as follows:

[0100] For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image. If the area of ​​a connected region is less than a preset threshold, then delete the connected region from the thumbnail slice image.

[0101] Step 202: Display multiple thumbnail slice images.

[0102] In practice, after reading the thumbnail images of each pathological slide, the target application displays each thumbnail image in the image display area of ​​the main interface.

[0103] When displaying thumbnail slice images, the thumbnail slice images can be displayed separately. That is, when displaying each thumbnail image, there is no overlap between the thumbnail slice images.

[0104] In one possible implementation, the user can name each pathology slide image saved in a folder. Correspondingly, when the target application reads a pathology slide image, it can simultaneously read the filename of that image. Furthermore, when displaying a thumbnail of the pathology slide image, the corresponding filename can be shown within the thumbnail image for the user to identify and distinguish. To prevent the displayed filename from obscuring useful images within the thumbnail image, the filename can be displayed in any of the four corners of the thumbnail image.

[0105] For example, a folder may contain six pathological slide images. Users can name these images 1, 2, 3, 4, 5, and 6, or a, b, c, d, e, and f. When the target application reads the pathological slide images in this folder, it also reads the filenames of each image. Then, when displaying thumbnail images of the pathological slides, it displays the corresponding filename in the upper right corner of the thumbnail image.

[0106] Step 203: In response to the stitching operation of the thumbnail slice images, move the thumbnail slice images to stitch them together; in response to the stitching completion command, record the position information of each thumbnail slice image in the multiple thumbnail slice images.

[0107] In implementation, users can perform operations such as translation, rotation, and mirroring on the thumbnail slice images to manually stitch them together. In the interface displaying the thumbnail slice images, when a user needs to manipulate a particular thumbnail slice image, they can first select that image. The target application then detects the selection and selects one thumbnail slice image from the displayed pool. The user can then move the selected thumbnail image. Correspondingly, when the target application receives this movement operation, it determines the movement parameters and moves the selected thumbnail slice image based on these parameters. The movement parameters can include the movement direction and distance, as well as the movement type, which can be translation, rotation, mirroring, etc.

[0108] For example, if the user is a pathologist, the pathologist can quickly determine the actual location of different pathological sections within the whole tissue based on the differences in the incisions of the sections in the thumbnail images. The pathologist can then perform operations such as moving, rotating, and mirroring the thumbnail images to achieve stitching. See also Figure 4 The image shows thumbnail images of five pathological sections, and a stitched image of thumbnail sections obtained from these five thumbnail images.

[0109] Users can manipulate thumbnail slice images in various ways. For example, users can drag thumbnail slice images with the mouse to perform operations such as translation, rotation, and mirroring. Correspondingly, whenever a user manipulates a thumbnail slice image with the mouse, the target application can detect the operation command corresponding to that thumbnail slice image and then perform the corresponding operation on the thumbnail slice image in response to the operation command.

[0110] In addition, users can also perform operations such as translation, rotation, and mirroring of the displayed thumbnail slice images through touch screen, shortcut keys, motion sensing devices, and development tools (such as MATLAB).

[0111] After stitching the thumbnail slices, users can select the stitching and annotation options in the target application's ribbon, and then, as shown... Figure 5 As shown, the target application displays sub-options under the stitching and annotation options. These sub-options may include manual stitching completion, start annotation, and statistical indicators. The user can then select the manual stitching completion option, and the target application can then detect the stitching completion command.

[0112] Upon detecting a stitching completion command, the target application, in response, captures the currently displayed stitched thumbnail image and records the current position information of each thumbnail image (also known as the rotating bounding box of the thumbnail image). This capture operation can also be a screenshot operation. The position information of each thumbnail image can be the pixel coordinates of the four corner points of each thumbnail image in the pixel coordinate system of the stitched thumbnail image. The origin of this pixel coordinate system can be any corner point of the stitched thumbnail image (such as the top left corner), and the coordinate axes are the two sides adjacent to that corner point.

[0113] In addition to recording the position information of each thumbnail slice image in the thumbnail slice mosaic, it can also record the size of the pathological slice image and the size of the corresponding thumbnail slice image.

[0114] In one possible implementation, the aforementioned location information and dimensions can be saved in JSON (JavaScript Object Notation) format.

[0115] In one possible implementation, the above-mentioned thumbnail slice mosaic can be in PNG (Portable Network Graphics) format.

[0116] Step 204: Based on the recorded position information of each thumbnail slice image, stitch together multiple pathological slice images in the background to obtain a stitched pathological slice image.

[0117] See Figure 6 This diagram illustrates the positional information of the thumbnail slices after manual stitching. The dashed boxes represent the positions of the thumbnail slices after manual stitching, and the four corner points of the dashed boxes formed by identical dashed lines represent the position of a single thumbnail slice in the pixel coordinate system of the stitched image. Because, in practice, overlap may occur between thumbnail slices when manually stitching them together, therefore... Figure 6 The dashed boxes in the text overlap.

[0118] In practice, after the stitching of the thumbnail slide images is completed, the target application stitches the pathological slide images in the background, so that the user can continue to use the target application to stitch other thumbnail slide images.

[0119] The stitching of pathological slide images by the target application can include two parts: affine transformation and image fusion. These two parts will be explained separately below.

[0120] Affine transformation:

[0121] For each pathological slide image, the affine transformation parameters corresponding to the pathological slide image are determined based on the initial position information of the pathological slide image, the recorded position information of the pathological slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image. The affine transformation is then performed on the pathological slide image based on the affine transformation parameters.

[0122] The following explains how to solve for the affine transformation parameters of any pathological slide image.

[0123] The parameters of an affine transformation include rotation, scaling, and translation. Let the rotation parameters be a1 and a2, the scaling parameters be b1 and b2, and the translation parameters be c1 and c2. These six parameters are unknowns and need to be solved.

[0124] Assuming the top-left corner of the pathological slide image is the origin (0, 0), and the two sides intersecting at the top-left corner are the coordinate axes, then the pixel coordinates of each corner point in the initial position information of the pathological slide image are: (x1, y1) = (0, 0), (x2, y2) = (0, w), (x3, y3) = (h, 0), and (x4, y4) = (h, w), where h and w are the length and width of the thumbnail slide image, which are known quantities.

[0125] In addition, the scaling factor k between the pathological slide image and the thumbnail slide image is a known quantity, and the positional information of the thumbnail slide image is recorded and is also a known quantity.

[0126] Therefore, the affine transformation parameters can be formed into the following affine transformation matrix A:

[0127]

[0128] Correspondingly, the initial position information and the position information after affine transformation of a pathological slide image can be related as follows:

[0129] |k*x1′ k*y1′ 1|=|x1 y1 1|·A (2)

[0130] |k*x2′ k*y2′ 1|=|x2 y2 1|·A (3)

[0131] |k*x3′ k*y3′ 1|=|x3 y3 1|·A (4)

[0132] |k*x4′ k*y4′ 1|=|x4 y4 1|·A (5)

[0133] Where k is the scaling factor between the pathological slide image and the thumbnail slide image, (x1′, y1′), (x2′, y2′), (x3′, y3′), and (x4′, y4′) are the recorded position information of the thumbnail slide image, and correspondingly, (k*x1′, k*y1′), (k*x2′, k*y2′), (k*x3′, k*y3′), and (k*x4′, k*y4′) are the position information of the pathological slide image after affine transformation from its initial position. (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are the initial position information of the pathological slide image.

[0134] Based on the above formulas (2), (3), (4), and (5), we can obtain the following four systems of equations:

[0135]

[0136]

[0137]

[0138]

[0139] In the above system of equations (6), (7), (8), and (9), a1, a2, b1, b2, c1, and c2 are six unknowns. By solving the eight equations in these four systems of equations, the values ​​of a1, a2, b1, b2, c1, and c2 can be obtained.

[0140] Alternatively, matrix operations can be used to obtain the values ​​of each affine variable parameter. Specifically, the calculation method can be represented by the following formula:

[0141] kP t =A0P s (10)

[0142] Where k is the scaling factor between the pathological slide image and the thumbnail slide image, for example, k is 64. P t The location information of the recorded thumbnail slice image is shown below, P t It is a 2×4 matrix.

[0143]

[0144] A0 is the affine transformation matrix, as shown below. A0 is a 2×3 matrix.

[0145]

[0146] P s The initial position information for the pathological slide image is shown below, P. sIt is a 2×4 matrix.

[0147]

[0148] Transforming the above formula (10) yields the following formula:

[0149]

[0150] Solving formula (14) will yield A0.

[0151] After obtaining the affine transformation parameters corresponding to each pathological slide image, the corresponding pathological slide image is subjected to affine transformation based on the affine transformation parameters.

[0152] Furthermore, based on the recorded positional information of each thumbnail slice image, the dimensions of the stitched thumbnail slice image can be obtained. Specifically, the maximum y-coordinate, maximum x-coordinate, minimum y-coordinate, and minimum x-coordinate of each thumbnail slice image are obtained. The difference between the maximum and minimum y-coordinates is calculated as the width of the stitched thumbnail slice image, and the difference between the maximum and minimum x-coordinates is calculated as the length of the stitched thumbnail slice image. Both the width and length of the stitched thumbnail slice image are multiplied by a scaling factor k to obtain the width and length of the stitched pathological slice image.

[0153] For each pathological slide image, an affine transformation is performed on it, and the transformed image is then converted into an intermediate image of the target size, which is the size of the stitched pathological slide image. This operation is equivalent to performing an affine transformation on the pathological slide image and mapping it onto a blank image of the target size, resulting in an intermediate image corresponding to the pathological slide image, with the target size being the size of the stitched pathological slide image. For example, see... Figure 7 The diagram shows how five pathological slide images are transformed through affine transformation and converted into an intermediate image of the target size. In the intermediate image, all areas except the pathological slide images after affine transformation belong to the background.

[0154] Image fusion:

[0155] Intermediate images corresponding to the pathological slide images are acquired sequentially. The second acquired intermediate image is overlaid on the first acquired intermediate image, and the two intermediate images are fused to obtain a new intermediate image. Then, a third intermediate image is acquired and overlaid on the newly fused intermediate image, and the third intermediate image and the newly fused intermediate image are fused together.

[0156] Merging two intermediate images means merging pixels at the same location in the two intermediate images. The following explanation illustrates the merging of any group of pixels at the same location in two intermediate images:

[0157] If the pixel in the upper middle image belongs to the foreground, then the alpha channel value of the pixel below will be set to 0, while the pixel above will remain unchanged. In this way, the effect is that after fusion, the pixel value at that position is the pixel value of the pixel above.

[0158] If a pixel in the upper middle image belongs to the background, the alpha channel value of the upper pixel will be set to 0, making the upper pixel transparent, while the lower pixel will not be processed. In this way, the effect is that after merging, the pixel value at that position is the pixel value of the lower pixel.

[0159] If a pixel in the upper middle image belongs to the background, and a pixel in the lower middle image also belongs to the background, then the alpha channel value of both pixels is set to 0. This results in a transparent background in the merged image. If a pixel in the upper middle image belongs to the background, and a pixel in the lower middle image belongs to the foreground, then the alpha channel value of the upper pixel is set to 0. This results in the pixel value at that position being the same as the pixel value of the lower pixel after merging.

[0160] Here, the foreground in each intermediate image can be obtained through image standard deviation thresholding, using the same method as the image standard deviation thresholding performed on the thumbnail slice image, which will not be elaborated here. Alternatively, after obtaining the foreground image of the thumbnail slice image, the foreground image can be upsampled to obtain the foreground image in the corresponding pathological slice image (including the pixel position of the foreground image in the pathological slice image). Correspondingly, when saving the thumbnail slice stitched image, the pixel position of the foreground image in each pathological slice image should also be saved so that the foreground and background in the intermediate image can be determined during image fusion.

[0161] In this embodiment, instead of loading and displaying the original pathological slide image, a thumbnail image of the pathological slide is displayed. Since the file size of the thumbnail image is smaller than the pathological slide image, loading and displaying the thumbnail image does not require excessive memory and will not cause computer lag. Furthermore, the user can move the displayed thumbnail images for stitching, resulting in a smoother process. After stitching the thumbnail images, the position information of each thumbnail image is recorded. Then, based on the position information of the thumbnail images, the pathological slide image is stitched together. The entire process does not require loading and displaying the pathological slide image, effectively avoiding the computer lag problem caused by excessive memory consumption due to loading and displaying the pathological slide image, thus allowing the user to complete the stitching of pathological slide images more smoothly and efficiently.

[0162] Based on the same technical concept, this application also provides an apparatus for displaying pathological slide images. This apparatus can be the computer device described in the above embodiments, such as... Figure 8 As shown, the device includes: a display module 810, a manual splicing module 820, a recording module 830, and an automatic splicing module 840.

[0163] Display module 810 is used to display multiple thumbnail slice images; the thumbnail slice images are thumbnails of pathological slice images;

[0164] The manual stitching module 820 is used to move the thumbnail slice image to stitch it in response to the stitching operation of the thumbnail slice image;

[0165] The recording module 830 is used to record the position information of each thumbnail slice image in the plurality of thumbnail slice images in response to the stitching completion command;

[0166] The automatic stitching module 840 is used to stitch together the pathological slide images corresponding to each thumbnail slide image based on the recorded location information to obtain a stitched pathological slide image.

[0167] In one possible implementation, the automatic stitching module 840 is used for:

[0168] For each pathological slide image, based on the initial position information of the pathological slide image, the recorded position information of the corresponding thumbnail slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image, the affine transformation parameters corresponding to the pathological slide image are determined, and the affine transformation is performed on the pathological slide image according to the affine transformation parameters.

[0169] The pathological slide images after each affine transformation are stitched together to obtain a stitched pathological slide image.

[0170] In one possible implementation, the automatic stitching module 840 is used for:

[0171] When multiple pathological slide images after affine transformation have overlapping areas, if the overlapping area of ​​the topmost pathological slide image belongs to the foreground, then the alpha channel value of the overlapping area of ​​the lower pathological slide image is adjusted to 0.

[0172] If the overlapping area of ​​the topmost pathological slide image belongs to the background, then the α channel value of the overlapping area of ​​the topmost pathological slide image is adjusted to 0.

[0173] In one possible implementation, the device further includes a downsampling module for:

[0174] Multiple pathological slide images are downsampled according to a preset downsampling factor to obtain thumbnail slide images corresponding to the pathological slide images.

[0175] In one possible implementation, the display module 810 is used for:

[0176] Displays multiple thumbnail slice images, including:

[0177] Receive a triggered request to acquire pathological slide images;

[0178] Based on the storage path in the pathological slide image acquisition request, read the thumbnail slide images corresponding to the pathological slide images in the same folder under the storage path;

[0179] The retrieved thumbnail slice images are displayed on the interface.

[0180] In one possible implementation, the apparatus further includes a foreground segmentation module for:

[0181] For each thumbnail slice image, determine the foreground image in the thumbnail slice image;

[0182] The α channel value of the images other than the foreground image in the thumbnail slice image is adjusted to 0.

[0183] In one possible implementation, the foreground segmentation module is used for:

[0184] Calculate the standard deviation of the RGB color channels for each pixel in the thumbnail image;

[0185] The image composed of pixels whose standard deviation is greater than the standard deviation threshold is used as the foreground image in the thumbnail slice image.

[0186] In one possible implementation, the device further includes a speckle noise removal module for:

[0187] For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image;

[0188] In the thumbnail image, connected regions with an area smaller than a preset threshold are deleted.

[0189] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0190] It should be noted that the pathological slide image processing device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the pathological slide image processing device and the pathological slide image stitching method embodiment provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiment, which will not be repeated here.

[0191] In this embodiment, instead of loading and displaying the original pathological slide image, a thumbnail image of the pathological slide is displayed. Since the file size of the thumbnail image is smaller than the pathological slide image, loading and displaying the thumbnail image does not require excessive memory and will not cause computer lag. Furthermore, the user can move the displayed thumbnail images for stitching, resulting in a smoother process. After stitching the thumbnail images, the position information of each thumbnail image is recorded. Then, based on the position information of the thumbnail images, the pathological slide image is stitched together. The entire process does not require loading and displaying the pathological slide image, effectively avoiding the computer lag problem caused by excessive memory consumption due to loading and displaying the pathological slide image, thus allowing the user to complete the stitching of pathological slide images more smoothly and efficiently.

[0192] Figure 9 This illustration shows a structural block diagram of a computer device 900 provided in an exemplary embodiment of this application. The computer device 900 may be a portable mobile terminal, such as a smartphone, tablet, laptop, or desktop computer. The computer device 900 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0193] Typically, computer device 900 includes a processor 901 and a memory 902.

[0194] Processor 901 may include one or more processing cores, such as a quad-core processor, a nine-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0195] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one instruction, which is executed by the processor 901 to implement the method for stitching pathological slide images provided in the method embodiments of this application.

[0196] In some embodiments, the computer device 900 may also optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal lines. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least a display screen 904.

[0197] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0198] Display screen 904 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 904 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 904 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 904, disposed on the front panel of computer device 900; in other embodiments, there may be at least two display screens 904, disposed on different surfaces of computer device 900 or in a folded design; in still other embodiments, display screen 904 may be a flexible display screen, disposed on a curved or folded surface of computer device 900. Furthermore, display screen 904 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 904 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0199] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the computer device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0200] In an exemplary embodiment, a computer-readable storage medium is also provided, storing at least one instruction, which is loaded and executed by a processor to implement the method for stitching pathological slide images in the above embodiments. For example, the computer-readable storage medium may be a ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, or optical data storage device, etc.

[0201] In an exemplary embodiment, a computer program product is also provided, which includes at least one instruction that is loaded and executed by a processor to implement the method of stitching pathological slide images as described above.

[0202] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0203] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0204] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals (including but not limited to signals transmitted between the user terminal and other devices) involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the pathological slide images and thumbnail slide images involved in this application were obtained with full authorization.

Claims

1. A method for stitching together pathological slide images, characterized in that, The method includes: Displaying multiple thumbnail slice images; the thumbnail slice images are thumbnails of pathological slice images; In response to the stitching operation of the thumbnail slice image, the thumbnail slice image is moved and stitched together; In response to the stitching completion command, the position information of each thumbnail slice image in the plurality of thumbnail slice images is recorded. The position information of each thumbnail slice image includes the pixel coordinates of the four corner points of each thumbnail slice image in the pixel coordinate system of the thumbnail slice stitched image. The origin of the pixel coordinate system is any corner point of the thumbnail slice stitched image, and the coordinate axes of the pixel coordinate system are the two sides of the thumbnail slice stitched image adjacent to any corner point. The thumbnail slice stitched image is an image obtained by stitching together the plurality of thumbnail slice images. Based on the recorded location information, the pathological slide images corresponding to each thumbnail slide image are stitched together to obtain a stitched pathological slide image.

2. The method according to claim 1, characterized in that, The process of stitching together the pathological slide images corresponding to each thumbnail slide image based on the recorded location information to obtain a stitched pathological slide image includes: For each pathological slide image, based on the initial position information of the pathological slide image, the recorded position information of the corresponding thumbnail slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image, the affine transformation parameters corresponding to the pathological slide image are determined, and the affine transformation is performed on the pathological slide image according to the affine transformation parameters. The pathological slide images after each affine transformation are stitched together to obtain a stitched pathological slide image.

3. The method according to claim 2, characterized in that, The step of stitching together the affine-transformed pathological slide images to obtain a stitched pathological slide image includes: When multiple pathological slide images after affine transformation have overlapping areas, if the overlapping area of ​​the topmost pathological slide image belongs to the foreground, then the alpha channel value of the overlapping area of ​​the lower pathological slide images is adjusted to 0. If the overlapping area of ​​the topmost pathological slide image belongs to the background, then the α channel value of the overlapping area of ​​the topmost pathological slide image is adjusted to 0.

4. The method according to claim 1, characterized in that, Before displaying multiple thumbnail slice images, the method further includes: Multiple pathological slide images are downsampled according to a preset downsampling factor to obtain thumbnail slide images corresponding to the pathological slide images.

5. The method according to claim 1, characterized in that, The display of multiple thumbnail slice images includes: Receive a triggered request to acquire pathological slide images; Based on the storage path in the pathological slide image acquisition request, read the thumbnail slide images corresponding to the pathological slide images in the same folder under the storage path; The retrieved thumbnail slice images are displayed on the interface.

6. The method according to claim 4, characterized in that, After obtaining the thumbnail slice images corresponding to the pathological slice images, the method further includes: For each thumbnail slice image, determine the foreground image in the thumbnail slice image; The α channel value of the images other than the foreground image in the thumbnail slice image is adjusted to 0.

7. The method according to claim 6, characterized in that, Determining the foreground image in the thumbnail slice image includes: Calculate the standard deviation of the RGB color channels for each pixel in the thumbnail slice image; The image composed of pixels whose standard deviation is greater than the standard deviation threshold is used as the foreground image in the thumbnail slice image.

8. The method according to any one of claims 1-7, characterized in that, Before displaying multiple thumbnail slice images, the method further includes: For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image; In the thumbnail slice image, connected regions with an area smaller than a preset threshold are deleted.

9. A device for stitching together pathological slide images, characterized in that, The device includes: The display module is used to display multiple thumbnail slice images; the thumbnail slice images are thumbnails of pathological slice images; A manual stitching module is used to move the thumbnail slice image for stitching in response to a stitching operation on the thumbnail slice image; A recording module is used to record the position information of each thumbnail slice image in the plurality of thumbnail slice images in response to the stitching completion command. The position information of each thumbnail slice image includes the pixel coordinates of the four corner points of each thumbnail slice image in the pixel coordinate system of the thumbnail slice stitched image. The origin of the pixel coordinate system is any corner point of the thumbnail slice stitched image, and the coordinate axes of the pixel coordinate system are the two sides of the thumbnail slice stitched image adjacent to any corner point. The thumbnail slice stitched image is an image obtained by stitching together the plurality of thumbnail slice images. The automatic stitching module is used to stitch together the pathological slide images corresponding to each thumbnail slide image based on the recorded location information to obtain a stitched pathological slide image.

10. The apparatus according to claim 9, characterized in that, The automatic splicing module is used for: For each pathological slide image, based on the initial position information of the pathological slide image, the recorded position information of the corresponding thumbnail slide image, and the scaling factor between the pathological slide image and the corresponding thumbnail slide image, the affine transformation parameters corresponding to the pathological slide image are determined, and the affine transformation is performed on the pathological slide image according to the affine transformation parameters. The pathological slide images after each affine transformation are stitched together to obtain a stitched pathological slide image.

11. The apparatus according to claim 10, characterized in that, The automatic splicing module is used for: When multiple pathological slide images after affine transformation have overlapping areas, if the overlapping area of ​​the topmost pathological slide image belongs to the foreground, then the alpha channel value of the overlapping area of ​​the lower pathological slide images is adjusted to 0. If the overlapping area of ​​the topmost pathological slide image belongs to the background, then the α channel value of the overlapping area of ​​the topmost pathological slide image is adjusted to 0.

12. The apparatus according to claim 9, characterized in that, The device further includes a downsampling module for: Multiple pathological slide images are downsampled according to a preset downsampling factor to obtain thumbnail slide images corresponding to the pathological slide images.

13. The apparatus according to claim 9, characterized in that, The display module is used for: Receive a triggered request to acquire pathological slide images; Based on the storage path in the pathological slide image acquisition request, read the thumbnail slide images corresponding to the pathological slide images in the same folder under the storage path; The retrieved thumbnail slice images are displayed on the interface.

14. The apparatus according to claim 12, characterized in that, The device further includes a foreground segmentation module for: For each thumbnail slice image, determine the foreground image in the thumbnail slice image; The α channel value of the images other than the foreground image in the thumbnail slice image is adjusted to 0.

15. The apparatus according to claim 14, characterized in that, The foreground segmentation module is used for: Calculate the standard deviation of the RGB color channels for each pixel in the thumbnail slice image; The image composed of pixels whose standard deviation is greater than the standard deviation threshold is used as the foreground image in the thumbnail slice image.

16. The apparatus according to any one of claims 9-15, characterized in that, The device also includes a speckle noise removal module for: For each thumbnail slice image, calculate the area of ​​each connected region in the thumbnail slice image; In the thumbnail slice image, connected regions with an area smaller than a preset threshold are deleted.

17. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one instruction, which is loaded and executed by the processor to implement the method for stitching pathological slide images as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the method for stitching pathological slide images as described in any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product stores at least one instruction, which is loaded and executed by a processor to implement the method for stitching pathological slide images as described in any one of claims 1 to 8.

Images