Instance segmentation using overlay tiles and overlay management

Abstract

An image is decomposed into a plurality of tiles (3100, 3200), a tile pair comprising an overlapping region (3110). These tiles can be processed by an instance segmentation algorithm. Techniques are disclosed for overlay management of a plurality of instance boundary lines (3301, 3302, 3303) disposed at least partially within an overlap region (3110).

Description

Technical Field

[0001] Various embodiments relate to instance segmentation techniques for multiple object instances in, for example, an image. Various examples specifically relate to instance segmentation using a tile-based instance segmentation algorithm. In particular, various examples relate to overlap management of object instances segmented in overlapping regions between adjacent tiles. Background Art

[0002] Instance segmentation is a computer vision task that involves identifying and separating individual instances of objects within an image, including detecting the boundaries of each instance of the object and assigning a unique label to each instance of the object. Unlike semantic segmentation, instance segmentation distinguishes between individual instances of objects of the same class.

[0003] Various types of instance segmentation algorithms are known in the art. For example, techniques for performing instance segmentation using machine learning models are known. Machine learning models are used to implement instance segmentation algorithms. Neural networks are examples of machine learning models.

[0004] Figure 1 An exemplary architecture 1000 of a neural network for instance segmentation according to the prior art is schematically shown. The architecture 1000 is introduced in the non-patent literature, "Mask R-CNN" by Kaiming He et al., Proceedings of the IEEE International Conference on Computer Vision, 2017, and is named Mask Region-Based Convolutional Neural Network (MaskR-CNN). The Mask R-CNN, i.e. Figure 1 The architecture 1000 of the invention allows to identify discrete objects, such as football players, in an input image 1010. Mask R-CNN is able to detect the football players and provide masks 1030 that cover each instance of the football players, respectively. These masks 1030 respectively have boundary lines surrounding / enclosing a single instance of the object. Therefore, the masks can be referred to as "instance boundary lines".

[0005] Mask R-CNN has a branch for predicting a segmentation mask 1030 on each region of interest (RoI) 1040, in parallel with branches for classification and bounding box regression, respectively. The mask branch is a fully convolutional network (FCN) applied to each RoI 1040 to predict the segmentation mask 1030 in a pixel-by-pixel manner. In order to establish pixel-to-pixel alignment between the network input (e.g., image 1010) and the network output (e.g., image 1020), Mask R-CNN includes a simple non-quantized layer called "RoIAlign" that is able to preserve the exact spatial location.

[0006] Instance segmentation algorithms such as Mask R-CNN are often developed to operate on datasets of everyday images. Such images can be captured by a cell phone camera and have relatively small file sizes; this limited resolution enables the images to be directly and completely processed by traditional instance segmentation algorithms. This is done by feeding the image through an instance segmentation algorithm, which runs on a computational processor, such as a graphics processing unit (GPU). For example, a single forward pass through an instance segmentation neural network can be performed. An example of a commonly used dataset for developing instance segmentation algorithms is the coco dataset (see: https: / / cocodataset.org / ; retrieved on September 19, 2023 ).

[0007] However, analysis of large images, such as microscope images or satellite images, by conventional instance segmentation algorithms is challenging. Large images can have tens or hundreds of gigapixels. Such large images can be obtained, for example, by stitching together multiple individual images.

[0008] The larger the image being analyzed or processed, the more computational memory is required to run the instance segmentation algorithm. Because computational memory, particularly graphics processing unit (GPU) memory, is both expensive and limited, large images cannot be processed in a conventional manner using existing instance segmentation algorithms, such as in a single forward pass. Techniques for instance segmentation of multiple instances of an object in an image will be described herein in conjunction with a "forward pass" as an exemplary process for performing instance segmentation.

[0009] To alleviate this problem of limited computational resources, images can be broken down into tiles (sometimes also called sub-parts or patches). In instance segmentation algorithms, for example, each tile is processed in a separate forward pass. Then, after instance segmentation is performed, the results of each forward pass are reassembled.

[0010] For example, the non-patent literature Payer, Christian et al., "Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks," Medical Image Analysis 57 (2019): 106–119, proposes a novel recurrent fully convolutional network architecture. Overlap management of instances for predicting objects in overlapping tiles is disclosed. The corresponding merging strategy includes the following steps: instances that touch the border of the tile are removed as long as the border is not the outer boundary of the entire image; and if the instance is predicted by multiple tiles, the instance with the largest area is used. This method faces certain limitations and disadvantages. Using such prior art implementations, it is not possible to accurately and reliably obtain a segmentation map for the entire image. Instance segmentation cannot be achieved for objects of arbitrary size. Summary of the invention

[0011] Advanced instance segmentation techniques are needed. In particular, accurate and / or reliable instance segmentation techniques are needed that allow for large-sized images. Instance segmentation of objects of arbitrary size is needed.

[0012] This need is met by the features of the independent claims. Embodiments are defined by the features of the dependent claims.

[0013] Various examples relate to per-tile processing of images in instance segmentation algorithms. Various examples relate specifically to overlap management: for overlapping tile pairs, there are image regions that are processed multiple times (overlapping regions). These overlapping regions belong to adjacent tile pairs. The results of the instance segmentation algorithms in these overlapping regions are integrated and merged. In these overlapping regions, there may be conflicts, for example, conflicts of different instance segmentation results for different tiles. According to the disclosed techniques, these conflicts are accurately resolved.

[0014] A computer-implemented method includes obtaining an image. The image depicts a plurality of instances of an object.

[0015] For example, a microscope image can be obtained. An optical microscope image or a scanning electron image can be obtained. A satellite image can be obtained. The image can have a particularly large pixel count. For example, an image can have over one million (1E5, i.e., 10 to the fifth power) pixels.

[0016] An object may refer to certain semantic features depicted multiple times in an image, for example, a cell, a specific cell type, a semiconductor structure, a semiconductor structure defect, a house, a house type, a street, a street type, etc. An instance of an object refers to a specific occurrence of the object in an image.

[0017] The method includes decomposing an image into a plurality of tiles. Tiles may also be referred to as patches. Thus, the image may be tiled. Adjacent pairs of tiles in the plurality of tiles include respective overlapping regions. In other words, the tiling process uses a stride size that is smaller than the size of the tile. This creates overlapping regions.

[0018] The method further includes processing the plurality of tiles using an instance segmentation algorithm. Various types of instance segmentation algorithms can be used, including machine learning instance segmentation algorithms. An example is the Mask R-CNN instance segmentation algorithm, see Kaiming He et al., "Mask R-CNN", Proceedings of the IEEE International Conference on Computer Vision, 2017.

[0019] An instance segmentation algorithm provides instance boundaries of multiple instances of an object as output. Each instance of an object is associated with a respective instance boundary line. Thus, different instances are distinguished from each other. The instance boundary lines can be implemented, for example, by bounding boxes, lines (e.g., polygons or splines) that delineate the outer circumference of each instance, or instance segmentation maps, to name just a few examples.

[0020] The method includes, for each adjacent tile pair and for each instance boundary line intercepted by the boundary of a tile of each adjacent tile pair: merging or not merging (selective merging) each instance boundary line intercepted by the boundary of the tile and another instance boundary line located in another tile of the adjacent tile pair. The selective merging is based on the degree of overlap between each instance boundary line intercepted by the boundary of the tile and another instance boundary line.

[0021] As a general rule, different measures of overlap are conceivable. For example, overlap in terms of area can be determined. This means that the size of the intersection of the areas covered by the first instance and the second instance can be proportional to the overlap. In some cases, this can be modified or replaced by a shape-based measure, for example, taking into account the shape of the outline. For example, two circular objects can have a greater overlap than a circular object and a rectangular object.

[0022] Sometimes, the metric for overlap can be user configurable. The metric used to determine overlap can also be selected based on the semantic content of the image. For example, it may be appropriate to use a different metric to determine overlap for a microscopic image depicting cells than for a microscopic image depicting semiconductor structures. This is because semiconductor structures typically exhibit a regular appearance, while cells have arbitrary, irregular shapes. Similar considerations may apply to, for example, satellite images of residential areas (e.g., analyzed to determine building density) if compared to satellite images of forests (e.g., analyzed to quantify vegetation density).

[0023] As a general rule, the overlap can be restricted to the overlapping region. This means that parts of the instance of the object that extend outside the overlapping region may not be considered. This is based on the finding that, in particular for objects of arbitrary shape, the overlap should be determined for regions where multiple metrics for instance segmentation are available, i.e., the overlap is determined only within the overlapping region.

[0024] As a general rule, merging is performed only when the degree of overlap indicates a particularly high degree of overlap.

[0025] Then, a situation may arise in which two overlapping instances are not associated with a particularly high degree of overlap. For example, the two instances may barely overlap and have a very small intersection. In this scenario, one of the two overlapping instances may be discarded / removed, while the other overlapping instance is retained. As a general rule, discarding or removing an instance can involve permanently deleting the instance so that the final segmentation result does not include the removed instance.

[0026] Various examples disclosed herein relate to the selection logic of which of two overlapping instances is removed (e.g., if the two overlapping instances are not merged). Such a decision can be based on various criteria, including but not limited to: random selection; confidence value; the size of the instance boundary line; surrounding environment, etc. The following example illustrates: the instance segmentation algorithm can provide instance boundary lines with different confidence / credibility. These confidence values ​​can then be compared. For example, such instance boundary lines with particularly low confidence values ​​associated with the instance boundary line can be mainly discarded. Sometimes, the area covered by the instance boundary line may be a measure of its confidence. As a general rule, small instance boundary lines, i.e., such instance boundary lines that enclose a small area, can be preferably discarded. Instance boundary lines with a small area to surface ratio can be preferably discarded. The surrounding environment of each instance boundary line can also be taken into account. For example, it is conceivable that such a scene has a densely populated surrounding environment, i.e., the instance boundary lines of many other instances in the immediate vicinity are considered to be less reliable and are therefore preferably discarded.

[0027] For example, when it is determined that the overlap of two instance boundary lines is higher than a first predefined threshold, the two instance boundary lines can be selectively merged with each other; and when it is determined that the overlap is lower than the first predefined threshold but higher than a second predefined threshold, one of the two overlapping instance boundary lines can be removed. Thus, a multi-threshold comparison can be performed.

[0028] For a degree of overlap even below a second predefined threshold, both instances can be retained.

[0029] In some examples, after merging and / or removing all instances with a certain overlap, post-processing for artifact removal can be applied. That is, based on certain predefined criteria, it can be determined whether certain instances correspond to false artifacts. For example, a shape-based artifact detection metric can be employed. It is possible to apply a machine learning model to identify artifacts. The artifacts can then be removed retroactively.

[0030] Above, an overlap management technique for tile processing has been disclosed, which produces tiles with overlapping areas. The overlap management technique produces instance boundary lines for each tile with a merged output in the overlapping area. As a final step, the instance boundary lines of all tiles can be merged to provide a more refined instance segmentation result for the entire image. For example, an instance segmentation map for the final image can be provided.

[0031] Such image-level instance segmentation results can facilitate various use cases. For example, instances can be counted. The fill fraction of instances can be determined. Instances of multiple object types can be compared with each other. Semantic labeling of images is possible. Image-level instance segmentation results can be exported via a graphical user interface. Automated post-processing is also possible.

[0032] Some specific use cases are explained below. Medical Diagnosis: In pathology, instance segmentation can help identify and segment individual cells in tissue samples, which can help diagnose diseases. Drug Discovery: Researchers use instance segmentation to monitor how cells respond to different drug treatments, which is critical for drug development and understanding cellular dynamics. Agriculture: Instance segmentation can be used to identify and count individual trees or plants in a field to assess health, productivity, or monitor growth patterns. Urban Planning and Land Use Classification: Urban planners and environmentalists use instance segmentation to identify individual buildings, roads, parks, and other infrastructure to help with urban planning and environmental impact studies. Defect Detection in Manufacturing: In electronics, SEM images are processed using instance segmentation to identify individual components or defects at a detailed level to ensure product quality. Traffic Management and Monitoring: Instance segmentation can distinguish and count individual vehicles on the road, assisting with traffic flow analysis and congestion prediction. Retail and Inventory Management: Store cameras with instance segmentation can identify and count individual products on shelves, which can help with inventory management and replenishment. Biodiversity Monitoring: Ecologists use instance segmentation in satellite and drone imagery to count individual animals in herds or monitor vegetation types, which is critical for conservation efforts. Materials Science: Researchers use SEM images and instance segmentation to segment individual particles, fibers, or phases in materials, which helps understand material properties and behavior. Archaeology and Heritage Conservation: Archaeologists use instance segmentation in satellite images to identify potential archaeological sites or individual structures and artifacts to guide excavation work or cultural heritage conservation.

[0033] It is understood that the features mentioned above and features not yet explained below can be used not only in the respective combination indicated but also in other combinations or alone, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 An exemplary architecture of a neural network for instance segmentation according to the prior art is schematically shown.

[0035] Figure 2 An exemplary image divided into a plurality of tiles according to various examples is schematically illustrated.

[0036] Figure 3 Exemplary adjacent tile pairs of an image according to various examples are schematically illustrated.

[0037] Figure 4 is a flow chart of a method according to various examples.

[0038] Figure 5 Schematically shows the Figure 3 Respective binary masks of two example boundary lines in the overlapping region of an exemplary pair of adjacent tiles.

[0039] Figure 6 is a block diagram of a computing device according to various examples.

[0040] Figure 7 is a flow chart of a method according to various examples. DETAILED DESCRIPTION

[0041] Some examples of the present disclosure generally provide multiple circuits or other electrical devices. All references to circuits and other electrical devices, and the functions provided by each circuit and other electrical device are intended to be not limited to only cover the contents shown and described herein. Although specific marks can be assigned to the various circuits or other electrical devices disclosed, these marks are not intended to limit the operating range of circuits and other electrical devices. Such circuits and other electrical devices can be combined and / or separated from each other in any way based on the desired specific type of electrical implementation. It should be recognized that any circuit or other electrical device disclosed herein may include any number of microcontrollers, machine learning dedicated hardware (e.g., graphics processor unit (GPU) and / or tensor processing unit (TPU)), integrated circuits, memory devices (e.g., flash memory, random access memory (RAM), read-only memory (ROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other suitable variations of these devices), and software that cooperates with each other to perform (one or more) operations disclosed herein. In addition, any one or more of the electrical devices may be configured to execute a set of program codes, which are implemented in a non-transitory computer-readable medium, and the non-transitory computer-readable medium is programmed to perform any number of functions as disclosed.

[0042] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the following description of the embodiments should not be considered restrictive. The scope of the present invention is not limited by the embodiments or drawings described below, which are only considered illustrative.

[0043] The accompanying drawings should be considered as schematic representations, and the elements shown in the accompanying drawings are not necessarily shown to scale. On the contrary, the various elements are represented so that their functions and general uses become apparent to those skilled in the art. Any connection or coupling between the functional blocks, devices, components or other physical or functional units shown in the accompanying drawings or described herein may also be achieved by indirect connection or coupling. Coupling between components may also be established by wireless connection. Functional blocks may be implemented with hardware, firmware, software or a combination thereof.

[0044] Various techniques described herein generally relate to instance segmentation. Instance segmentation is used to determine instance boundary lines for each instance of an object. Instance boundary lines can be formed by bounding boxes or contours that depict the boundaries of each instance of an object. A segmentation map can be provided per pixel, with each pixel assigned to a specific object instance (or background, or other instances of other object types). Thus, a segmentation map can include a collection of multiple instance boundary lines.

[0045] Instance segmentation is a special form of image segmentation that deals with detecting instances of objects and delineating the boundaries of instances. One way in which the boundaries of instances of objects (instance boundaries) can be described is by assigning pixels of an image to instances of the object.

[0046] Additionally, in some examples, overlapping instances of an object or multiple objects may be determined.

[0047] Unlike semantic segmentation, instance segmentation distinguishes between individual instances of the same class of objects. For example, if there are two instances of an object in an image, instance segmentation will identify these two instances as two separate instances.

[0048] Instance segmentation can optionally distinguish between different types of objects; however, this is not germane to the techniques disclosed herein.

[0049] Prior to the filing of this application, there are various instance segmentation algorithms that can be used herein to process multiple tiles of a single large-size image depicting multiple instances of one or more objects. For example, existing instance segmentation algorithms may include corresponding algorithms disclosed and / or mentioned in the following non-patent literature:

[0050] – Lei Ke, Yurong Dai, and Zhiqiang Deng, “Deepocclusion-aware instance segmentation with overlapping bilayers,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021.

[0051] – Kaiming He et al., “Mask R-CNN,” in Proceedings of the IEEE International Conference on Computer Vision, 2017.

[0052] – Kai Chen et al., “MMDetection: Open mmlabdetection toolbox and benchmark,” arXiv preprint arXiv:1906.07155 (2019).

[0053] –Bolya, Daniel et al., “Yolact: Real-time instances segmentation,” in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019.

[0054] According to various examples, a machine learning model can be used for instance segmentation. The machine learning model can be pre-trained. Mask-R-CNN can be used.

[0055] A specific type of instance segmentation algorithm is not germane to the techniques disclosed herein. The techniques disclosed herein can work with various types of instance segmentation algorithms.

[0056] The techniques disclosed herein enable efficient processing of large-scale images. Tile-wise processing is employed. Tile-wise processing, sometimes also referred to as "patch-based" processing, involves breaking a larger image into smaller tiles (or patches) and processing these tiles independently before combining the results to generate a complete output for the entire image.

[0057] According to various examples, in order to avoid scaling of image data for large-sized images, each such image can be segmented into multiple tiles of appropriate resolution, such as tiles of 256*256 pixels, 480*480 pixels, 512*512 pixels, 720*720 pixels, 1280*720 pixels, etc., which are consistent with the input size of available or existing instance segmentation models or algorithms.

[0058] A large-size image can be divided into multiple tiles using various tiling methods so that adjacent tile pairs include respective overlapping areas. For example, a large-size image can be covered in a sliding window method using a fixed step of a specific pixel count, such as 128, 256, etc. The method places the tile at an overlap with each adjacent tile along the x-axis (or horizontal) direction and the y-axis (or vertical) direction. For example, if the size of the tile is 512*512 pixels and the fixed step is 128 pixels, the method will place the tile at an overlap of 25% with each adjacent tile along the x-axis direction and the y-axis direction.

[0059] Figure 2 Schematically illustrates an exemplary two-dimensional image 2000 that is divided or segmented into a plurality of tiles according to various examples. The image 2000 may be decomposed into a plurality of tiles along the x-axis and the y-axis, respectively. Figure 2As shown, image 2000 is decomposed into 3 rows along the y-axis (also defined as width in this article) and 6 columns along the x-axis (also defined as length in this article), that is, a total of 18 tiles. The 18 tiles have width-length value pairs (w1, l1), (w1, l2), ..., (w3, l5) and (w3, l6), respectively, and will be indicated as tile (w1, l1), tile (w1, l2), ..., tile (w3, l5) and tile (w3, l6) respectively below. Image 2000 includes multiple adjacent tile pairs, and each of the multiple pairs includes an overlapping area. Similar to these tiles, the respective overlapping areas between adjacent tile pairs along the x-axis are also indicated using width-length value pairs, namely, overlapping area (w1, l12), overlapping area (w1, l23),…, overlapping area (w2, l12), overlapping area (w2, l23),…, overlapping area (w3, l45) and overlapping area (w3, l56), and the respective overlapping areas between adjacent tile pairs along the y-axis are also indicated using width-length value pairs, namely, overlapping area (w12, l1), overlapping area (w12, l2),…, overlapping area (w23, l1), overlapping area (w23, l2),… and overlapping area (w23, l6). For example, in the rows of tiles sharing the same width w1, image 2000 includes 5 pairs of adjacent tiles along the x-axis, namely, the pair [tile (w1, l1), tile (w1, l2)] including the overlapping area (w1, l12), the pair [tile (w1, l2), tile (w1, l3)] including the overlapping area (w1, l23), the pair [tile (w1, l3), tile (w1, l4)] including the overlapping area (w1, l34), the pair [tile (w1, l3), tile (w1, l4)] including the overlapping area (w1, l45), and the pair [tile (w1, l5), tile (w1, l6)] including the overlapping area (w1, l56). Similarly, in the columns of tiles sharing the same length l1, image 2000 includes 2 pairs of adjacent tiles along the y-axis, namely, a pair [tile (w1, l1), tile (w2, l1)] including an overlapping region (w12, l1) and a pair [tile (w2, l1), tile (w3, l1)] including an overlapping region (w23, l1). For example, the 18 tiles of image 2000 can be segmented or decomposed from left to right and from top to bottom.

[0060] The various techniques disclosed in this article are generally based on the discovery that overlap management, especially the merging strategy for instance segmentation results in overlapping areas (as disclosed in the non-patent literature - Payer, Christian et al. "Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks", Medical Image Analysis 57 (2019): 106-119), cannot accurately and reliably handle or manage objects whose feature size is larger than the width of the overlapping area. For example, objects that extend through the entire overlapping area and beyond the overlapping area cannot be accurately handled. This limits the ability of prior art solutions to segment instances of objects of arbitrary sizes. It has been found that such inaccuracies in overlap management are caused by larger objects being truncated by the boundaries of two adjacent tiles and therefore removed / discarded. This has a negative impact on the accuracy of instance segmentation. The following will combine Figure 3 This finding is explained in further detail.

[0061] Figure 3 An exemplary adjacent tile pair 3000 of an image according to various examples is schematically shown. For example, the adjacent tile pair 3000 may be Figure 2 Any pair of tiles of the image 2000, for example, a pair [tile (w1, l1), tile (w1, l2)] including an overlapping area (w1, l12), a pair [tile (w1, l2), tile (w1, l3)] including an overlapping area (w1, l23), a pair [tile (w1, l3), tile (w1, l4)] including an overlapping area (w1, l34). The adjacent tile pair 3000 includes a first tile 3100 and a second tile 3200. The first tile 3100 and the second tile 3200 have an overlapping area 3300 between a first boundary 3110 of the first tile 3100 and a second boundary 3210 of the second tile 3200. The first tile 3100 and the second tile 3200 each depict multiple instances of an object, i.e., a whole or partial ellipse filled with a dotted pattern, wherein three instances 3301, 3302, and 3303 of the object are respectively truncated by at least one boundary of two adjacent tiles 3100 and adjacent tiles 3200, i.e., boundary 3110 and / or boundary 3210. For example, instance 3301 is truncated by second boundary 3210, instance 3302 is truncated by both first boundary 3110 and second boundary 3210, and instance 3303 is truncated by first boundary 3110.

[0062] Therefore, instances 3301, 3302, and 3303 extend beyond the overlapping region. Instances 3301, 3302, and 3303 have dimensions comparable to those of overlapping region 3300 (e.g., Figure 3 ) dimensions in the example.

[0063] By utilizing the merging strategy disclosed in the non-patent document Payer, Christian et al., “Segmenting and tracking cell instances with cosine embeddings and recurrent hourglass networks,” Medical Image Analysis 57 (2019): 106–119, respective portions of the instance 3302 depicted in the first tile 3100 and the second tile 3200 are removed, and thus, the instance 3302 of the object is missing from the segmentation result of the pair 3000 obtained by merging the respective segmentation maps of the first tile 3100 and the second tile 3200. This is an undesirable behavior. Therefore, the prior art cannot accurately and reliably merge instance segmentation results from a single tile of a large-sized image back together, especially when the size of the object is larger than, for example, the width or length of the overlapping area.

[0064] The techniques disclosed herein alleviate such limitations and disadvantages.The techniques disclosed herein provide advanced overlap management and detailed merging of matching instances of objects.

[0065] In the following, a technique for facilitating determination of an instance segmentation map for an entire large-scale image is described. The image depicts multiple instances of an object. For example, the image can be obtained during or after a tile experiment. The image is decomposed (or segmented) into multiple tiles, and adjacent tile pairs include respective overlapping regions. The multiple tiles are processed using an instance segmentation algorithm to obtain respective instance boundary lines of multiple instances of the object. For each adjacent tile pair and for each instance boundary line truncated by the boundary of the tile of each adjacent tile pair, each instance boundary line truncated by the boundary of the tile is selectively merged with another instance boundary line based on the degree of overlap between each instance boundary line truncated by the boundary of the tile and another instance boundary line located in another tile of the tile pair.

[0066] According to various examples, the image may include a microscope image or a satellite image. In a microscope image, the object may include various cells, such as muscle cells, neuronal cells, bacteria, viruses, etc. In a satellite image, the object may include trees, buildings, roads, farms, etc. The microscope image may be acquired using an optical microscope. In addition, a charged particle scanning microscope may be used, such as a scanning electron microscope. For example, a multi-field scanning electron microscope imaging device may be used to perform multiple raster scans simultaneously. X-ray images may be studied. Computed tomography images may be studied. The image may be derived from multiple sub-images stitched together, where multiple imaging processes are used to acquire image data, and then the data is stitched together to form an image. The image may be obtained during or after a tile experiment. Processing of the image, for example, segmenting into tiles and performing instance segmentation may occur during or after a tile experiment, i.e., real-time processing or post-processing is possible.

[0067] As will be appreciated, the disclosed techniques are particularly applicable to large-size images, such as images having more than E5, E9, or E12 pixels, and are capable of processing images in a variety of use cases or test scenarios.

[0068] Next, we will combine Figure 4 Details about overlap management and merging of object instances detected in the overlap region of all paths of adjacent tiles are explained.

[0069] Figure 4 is a flow chart of method 4000 according to various examples. For example, according to Figure 4 Method 4000 may be executed by at least one circuit (eg, a central processing unit (CPU) and / or a graphics processing unit (GPU) and / or a tensor processing unit (TPU)) when program code is loaded from a non-volatile memory. Figure 4 The method 4000 facilitates instance segmentation of large-size images, such as microscopy images or satellite images.

[0070] Method 4000 relates to aspects of processing an image depicting multiple instances of an object. The image is decomposed (or segmented) into multiple tiles, and adjacent tile pairs include respective overlapping regions. The multiple tiles are processed using an instance segmentation algorithm to obtain respective instance boundary lines of multiple instances of the object. For each adjacent tile pair and for each instance boundary line truncated by a boundary of a tile of each adjacent tile pair, each instance boundary line truncated by a boundary of the tile is selectively merged with another instance boundary line based on the degree of overlap between each instance boundary line truncated by the boundary of the tile and another instance boundary line located in another tile of the adjacent tile pair. Details of method 4000 are described below.

[0071] Block 4100: Obtain an image depicting multiple instances of an object, e.g. Figure 2 An image 2000, for example, is filled with Figure 2 The oval shown in the dashed pattern.

[0072] For example, the image may be loaded from a memory or obtained from a data acquisition unit via an input interface.The image may be obtained from a microscope, a satellite or a database storing images.

[0073] Block 4100 may optionally include controlling an image acquisition device to acquire an image.

[0074] After acquiring the image, Figure 4 Method 4000 may optionally include image preprocessing, such as noise filtering, cropping boundaries, etc.

[0075] Block 4200: Transmit an image (e.g., Figure 2 The image 2000) is decomposed into a plurality of tiles, wherein adjacent tile pairs include respective overlapping regions. That is, the image is decomposed into a plurality of tiles such that adjacent tile pairs include respective overlapping regions.

[0076] For example, based on Figure 2 The strategy shown in decomposes (or segments) the image into multiple tiles. In addition, various tile methods disclosed in the non-patent literature - Kloster, Michael et al., "Improving deeplearning-based segmentation of diatoms in gigapixel-sized virtual slides by object-based tile positioning and object integrity constraint", Plos one 18.2 (2023): e0272103, can also be applied. Alternatively, the tile overlapping strategy disclosed in the non-patent literature - Ronneberger, Olaf, Philipp Fischer and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation", Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, Springer International Publishing AG, 2015, can also be utilized.

[0077] According to various examples, the image may be decomposed into a plurality of tiles having the same size, for example, 256*256 pixels, 512*512 pixels, etc. Alternatively, the image may be decomposed into a plurality of tiles having different sizes. Figure 2 The tiles in each row of can have different lengths, and Figure 2 The tiles in each column can have different widths.

[0078] After breaking the image into tiles, Figure 4 The method 4000 may optionally include preprocessing each of the plurality of image blocks, such as noise filtering, etc. ( Figure 4 not shown).

[0079] Block 4300: Process the plurality of tiles using an instance segmentation algorithm to obtain instance boundary lines of respective instances of the object.

[0080] For example, multiple forward passes through the instance segmentation algorithm may be performed sequentially or at least partially simultaneously using parallel processing. Each forward pass processes each tile or batch of tiles.

[0081] Advantages of processing by tiles are memory efficiency and the ability to process multiple tiles in parallel, for example, using a GPU or TPU.

[0082] According to the present disclosure, any existing instance segmentation algorithm can be used to process multiple tiles of an image to obtain instance boundary lines within the multiple tiles. Each forward pass of the instance segmentation algorithm can output a respective segmentation map including a set of instance boundary lines. The instance boundary lines can also be output in another data format.

[0083] For example, a trained instance segmentation algorithm can be used to process Figure 2 18 blocks of the image 2000 are obtained to obtain respective instance segmentation maps of the 18 blocks, that is, 18 instance segmentation maps. Respective instance boundary lines of multiple instances of the object, that is, elliptical shapes, can be obtained in the respective instance segmentation maps.

[0084] Instance boundaries can be formed by curves, splines or other 1-D extended structures that depict the outer circumference of each object instance. Instance boundaries surround respective regions of the image accordingly. Another option for implementing instance boundaries is a segmentation map. In a segmentation map, each tile or each pixel of the image is assigned to a specific instance. Yet another further example is a bounding box.

[0085] According to various examples, method 4000 may optionally include determining, based on the obtained instance boundary lines of the plurality of instances of the object, adjacent tile pairs in which at least one instance boundary line is truncated by a boundary of a tile of the adjacent tile pair.

[0086] For example, refer to Figure 2 Since no instance boundary line is truncated by the boundary of the tile pair [tile (w1, l1), tile (w1, l2)], such tile pairs can be ignored when executing box 4400 of method 4000 to save computing time and resources.

[0087] That is, only some overlapping regions that need attention are considered as part of the merging strategy.

[0088] Box 4400 (overlap management): For each adjacent tile pair and for each instance boundary line truncated by the boundary of the tile of each adjacent tile pair (as described above): merge or not merge (selectively merge) each instance boundary line truncated by the boundary of the tile and another instance boundary line based on the degree of overlap between each instance boundary line truncated by the boundary of the tile and another instance boundary line located in another tile of the adjacent tile pair.

[0089] The overlap may be a measure of how well two instance boundaries conform to each other. Thus, the overlap indicates the consistency between two instances that are arranged at approximately the same image location but processed in different forward passes of the instance segmentation algorithm (processing adjacent tiles).

[0090] As a general rule, the degree of overlap can be determined based on an appropriate overlap metric. The overlap metric can be determined based on the respective portion of the instances that are located in the respective overlapping regions of adjacent tile pairs. For example, a shape-based metric can take into account the shape of a curve that depicts the circumference of the respective instance. For example, an area-based metric can take into account the size of the respective instance. For example, a position-based metric can take into account whether the respective instances are located at similar image locations. Combinations of these metrics are possible.

[0091] refer to Figure 3 After performing instance segmentation of the first tile 3100 using an available or existing instance segmentation algorithm, respective instance boundary lines of multiple instances of the object are obtained, i.e., boundary lines of the entire or partial ellipse filled with a dotted pattern. For example, respective entire or partial boundary lines of instance 3301, instance 3302, and instance 3303 can be obtained by processing the first tile 3100. Similarly, respective entire or partial boundary lines of instance 3301, instance 3302, and instance 3303 can be obtained by processing the second tile 3100.

[0092] Since the instances 3301, 3302, and 3303 are located in the overlapping region 3300 of the first tile 3100 and the second tile 3200, all or part of the respective boundary lines of the instances 3301, 3302, and 3303 obtained by processing the first tile 3100 and another all or part of the respective boundary lines of the instances 3301, 3302, and 3303 obtained by processing the second tile 3200 belong to the same instance 3301, 3302, and 3303 of the object, respectively. Therefore, the respective boundary lines depicted in the segmentation maps of the first tile 3100 and the second tile 3200 based on at least one of the instances 3301, 3302, and 3303 can be merged together. The segmentation map of the first tile 3100 and the segmentation map of the second tile 3200 can be merged together after merging the respective instance boundary lines.

[0093] According to various examples, the method 4000 may further include determining a segmentation map for the entire image 2000. Such a segmentation map then depicts respective instance boundary lines of multiple instances of the object based on selective merging of each of the adjacent tile pairs of the multiple tiles (after overlap management). The instance boundary lines can be superimposed on the original image.

[0094] In another variation, automatic post-processing of the segmentation map would be possible. Examples include instance counting (e.g., cell counting), determining the portion of an area covered by instances of an object, e.g., cell confluence determination. When processing images depicting, e.g., semiconductor structures, defects can be detected. Populated areas may be highlighted in satellite images. These are just some specific examples of the countless possible examples of post-processing accurate segmentation maps obtained from the disclosed techniques.

[0095] As can be understood from the above, the merging strategy depends on the overlap degree. Possible implementation details about the overlap degree will be explained below.

[0096] Figure 5 Involving aspects of overlap. Figure 5 Schematically shows Figure 3Two binary masks 3120 and 3220 of the overlapping area 3300 of the adjacent tile pair 3000. The first binary mask 3120 (as an example of an instance boundary line) can be obtained based on the instance segmentation map of the first tile 3100 after removing the instance segmentation of the other two instances 3301 and 3303. Similarly, the second binary mask 3220 can be obtained based on the instance segmentation map of the second tile 3200 after removing the instance segmentation of the other two instances 3301 and 3303. For the first binary mask 3120 and the second binary mask 3220, the pixels belonging to the instance 3302 are true, that is, "1", and the pixels not belonging to the instance 3302 are false, that is, "0". For the instance 3302 of the object, the overlap between the first instance boundary line truncated by the first boundary 3110 of the first tile 3100 and the second instance boundary line truncated by the second boundary 3210 of the second tile 3200 can be calculated or determined according to the following formula:

[0097]

[0098] Among them, the first binary mask 3120 and the second binary mask 3220 can be regarded as two matrices respectively; "AND" and "OR" are element-by-element Boolean operations or entry-by-entry Boolean operations; "SUM" represents the element-by-element or entry-by-entry sum of the matrix.

[0099] Note that this is just one example of a metric for calculating overlap, and other overlap metrics are conceivable.

[0100] In the above example, the overlap is determined within each pair of tiles and the corresponding overlap region 3300 of the other tile. That is, when determining the overlap, the portion of the instance of the object outside the overlap region is ignored. Note that the instance extends from the overlap region into both tiles.

[0101] By considering the degree of overlap - for example, according to the definition provided above by Equation 1, an accurate decision can be made whether certain instance boundary lines belong to the same instance or different instances. Merging instances of an object with artifact instance boundary lines can be avoided. Merging instance boundary lines belonging to different instances can be avoided.

[0102] As a general rule, merging can be performed only for sufficiently large overlaps (here, "large overlap" means significant consistency in the corresponding metrics of instance boundaries obtained by processing adjacent tiles; for example, similar size of contours, similar shapes, and / or similar pixels covered by instance boundaries). Threshold comparisons can be implemented. If the overlap is not large enough, one of the two instances can be retained and the other discarded. In order to determine which of the two instances will be discarded and which of the two instances will be retained, different strategies can be adopted. For example, the confidence of the prediction of the instance segmentation algorithm associated with each of the two instances can be considered.

[0103] In some examples, a multi-threshold comparison of overlap can be implemented. An example is explained below. After determining or calculating the overlap, method 4000 may include comparing the overlap with a first predefined threshold, for example, in the range of 70% to 100%. According to various examples, the selective merging may include merging each instance boundary line truncated by the boundary of the tile with another instance boundary line when it is determined that the overlap is above the first predefined threshold.

[0104] For example, refer to Figure 5 For instance 3302, if the overlap between the first instance boundary line truncated by the first boundary 3110 of the first tile 3100 and the second instance boundary line located in the second tile 3200 is higher than the first predefined threshold, the first instance boundary line truncated by the first boundary 3110 of the first tile 3100 and the second instance boundary line located in the second tile 3200 are merged. This is because the two instances closely correspond to each other. Therefore, it is possible to facilitate a smooth transition of instance boundary lines (or contours) of objects between adjacent tiles.

[0105] If the overlap is below the first predefined threshold, method 4000 may include comparing the overlap with a second predefined threshold, for example, any value between 20% and 40%. The selective merging may include, when it is determined that the overlap is below the first predefined threshold but above the second predefined threshold: determining respective confidence values ​​for each instance boundary line and another instance boundary line truncated by the boundary of the tile, and removing instance boundary lines with lower confidence values ​​from each instance boundary line and another instance boundary line truncated by the boundary of the tile.

[0106] For example, refer to Figure 3, the first boundary 3110 of the first tile 3100 intercepts the instance boundary line of the instance 3303, and a portion of the instance boundary line of the instance 3303 is located in the first tile 3100. Another instance boundary line of the instance 3303, that is, the entire instance boundary line of the instance 3303, is located in the second tile 3200 of the pair 3000. The respective confidence values ​​or confidence levels of both the instance boundary line intercepted by the first boundary 3110 of the first tile 3100 and the other instance boundary line located in the second tile 3200 can be determined based on the respective pixel counts within the respective instance boundary lines or the respective areas of the regions surrounded by the respective instance boundary lines. According to various examples, the respective confidence values ​​can be determined using existing techniques for analyzing predictions of available instance segmentation algorithms.

[0107] like Figure 3 As shown, in the first tile 3100, the pixel count within the instance boundary line of the instance 3303 or the area of ​​the region surrounded by the instance boundary line of the instance 3303 is less than or lower than the pixel count of another instance boundary line of the instance 3303 located in the second tile 3200 or the area of ​​the region surrounded by the another instance boundary line. That is, the another instance boundary line of the instance 3303 located in the second tile 3200 may have a higher confidence value or confidence level. In this way, the instance boundary line of the instance 3303 that is truncated by the first boundary 3110 of the first tile 3100 may be removed.

[0108] Alternatively, when determining that the overlap is above a first predefined threshold, method 4000 may include determining respective instances of the object including each instance boundary line and another instance boundary line as candidate instances to be merged. The selective merging may be based on the respective candidate instances.

[0109] According to various examples, after determining all candidate instances, method 4000 may further include filtering out candidate instances that may generate artifacts after merging and removing such candidate instances. The selective merging may be based on the respective remaining candidate instances after the removal.

[0110] These steps may be needed to account for respective instances of objects that may be detected in only some tiles. The rationale is that the result may look more natural if no instances of the object are detected at all, compared to detecting instances of the object with unnatural artifacts in its boundary lines or contours.

[0111] For example, refer to Figure 3, if the degree of overlap between the first instance boundary line of the instance 3303 truncated by the first boundary 3110 of the first tile 3100 and the second instance boundary line of the instance 3303 located in the second tile 3200 is lower than the first predefined threshold but higher than the second predefined threshold, the first instance boundary line of the instance 3303 truncated by the first boundary 3110 of the first tile 3100 will be removed because the area of ​​the region surrounded by the first instance boundary line is smaller than the area of ​​the region surrounded by the second instance boundary line of the entire instance 3303 of the object. In this way, it is possible to avoid deleting or removing instances of objects that are slightly overlapped accidentally, and at the same time, it is possible to avoid or improve the discontinuity of instance boundary lines of objects after merging.

[0112] Such techniques are based on the following discovery: if an instance boundary line is truncated by the boundary of a tile and does not cross the boundary of the tile to merge with another instance boundary line located in another tile adjacent to the tile, then such instance boundary line is likely to be an artifact at the tile boundary. This artifact may come from the "hallucination" of the instance segmentation algorithm, because such artifacts have very little image context to make accurate predictions in their respective regions. Therefore, such artifacts can be safely removed. Then, discarding the instance boundary lines truncated by the boundary of the tile (the boundary of the tile also corresponds to the boundary of the overlapping area) generally produces better results. This is based on the assumption that the instance of the object extends beyond the boundary of the tile into the other tile in the adjacent tile pair, and therefore the respective instance boundary lines obtained by processing the other tile are more accurately captured.

[0113] Despite this complex logic for determining whether to merge overlapping instance boundaries, it has been observed that artifacts can sometimes be produced when merging instances of objects that overlap across an overlap region. Therefore, after overlap management is completed, post-processing can be used to remove such artifacts.

[0114] According to various examples, method 4000 may further include: when it is determined that one or more artifacts are generated after merging each instance boundary line intercepted by the boundary of the tile and another instance boundary line: removing each merged instance boundary line intercepted by the boundary of the tile.

[0115] According to various examples, method 4000 may further include: determining a confidence value or confidence level for each instance boundary line that is truncated by the boundary of each pair of tiles; for each adjacent tile pair, selecting at least one instance boundary line with the highest confidence value, and the selective merging is further based on the selected at least one instance boundary line.

[0116] For example, refer to Figure 3, the first boundary 3110 of the first block 3100 cuts off the two instance boundary lines of instance 3302 and instance 3303, respectively. The corresponding confidence values ​​or confidence levels of the two instance boundary lines of instance 3302 and instance 3303 can be determined based on the credibility of the instance segmentation algorithm output; for example, for a classification task, the credibility of the classification prediction can be provided based on the distribution of multiple output neurons of the output layer (corresponding to different object types or backgrounds) of a given pixel. In the case where multiple output neurons have similar output probabilities, the credibility is generally low. On the other hand, for significantly different neuron values ​​in the output layer, the credibility is generally high. There are also auxiliary algorithms that can be used to judge the credibility of the prediction of another algorithm that can be adopted in the disclosed examples. A further measure of credibility is the respective count of pixels within the respective instance boundary lines or the respective area of ​​the area surrounded by the respective instance boundary lines. For example, small instance segmentation is generally not as credible as extended instance boundary lines and large instance boundary lines.

[0117] like Figure 3 As shown, in the first block 3100, the pixel count within the instance boundary line of instance 3302 or the area of ​​the region surrounded by the instance boundary line of instance 3302 is greater than the pixel count within the instance boundary line of instance 3303 or the area of ​​the region surrounded by the instance boundary line of instance 3302. That is, the instance boundary line of instance 3302 may have a higher confidence value or confidence level. Therefore, the instance boundary line of instance 3302 may be selected, and the selective merging may be based on the instance boundary line of instance 3302.

[0118] In other words, it is possible to define a priority order between multiple instance boundaries obtained from an instance segmentation algorithm. The priority order is based on a confidence value or confidence level associated with each instance boundary. Different metrics for determining the confidence value or confidence level are conceivable, including, as described above, analysis of the confidence of the inference provided by the instance segmentation algorithm or pixel counting.

[0119] Then, the instance boundary line with a higher priority may be favored, for example, it is less likely to be discarded in the case of limited overlap, or it may be weighted higher when merged with another instance boundary line.

[0120] Figure 69000 according to various examples. The computing device 9000 may include a processor 9020, a memory 9030, and an input / output interface 9010. The processor 9020 is configured to load program code from the memory 9030 and execute the program code. When executing the program code, the processor 9020 performs a method 4000 for processing an image depicting multiple instances of an object, for example, a method 4000 for a microscope image depicting multiple instances of a muscle cell.

[0121] The computing device 9000 may be embedded in or connected to a tissue analyzer or microscope. That is, a tissue analyzer or microscope including the computing device 9000 may be configured to perform the method 4000. The computing device 9000 may perform Figure 7 method.

[0122] Figure 7 is a flow chart of a method according to various examples. The method can be executed by at least one circuit, such as a CPU and / or a GPU and / or a TPU, when loading program code from a non-volatile memory. The method facilitates instance segmentation of large images, such as microscope images or satellite images.

[0123] Figure 7 The method is Figure 4 Specifically, Figure 7 The method is an example implementation of box 4400.

[0124] Block 505: First, all instance boundary lines obtained from multiple forward passes of the instance segmentation algorithm (for each tile) are sorted. That is, the results of all forward passes are sorted.

[0125] This ranking can define an order of priority. The ranking can be based on confidence values. Various metrics can be used to determine confidence values, including the number of pixels covered by the respective instance boundary line or the confidence of the respective prediction of the instance segmentation algorithm.

[0126] Block 510: Select the next instance boundary line according to the priority order, ie, from high priority to low priority. In addition, the integer variable n is set to 0. Block 510 defines an outer loop 598 that iterates through the instance boundary lines from high priority to low priority.

[0127] Block 515 : Check whether the boundary line of the currently selected instance is cut off by the tile boundary.

[0128] If the current instance boundary line is not truncated by a tile boundary, the method starts at block 520. Then, before performing further iterations of block 510, any lower priority instance boundary lines that have any overlap with the current instance boundary line and have a lower priority are removed.

[0129] Block 515 combines block 52 with pre-filtering where the merge strategy does not need to consider the case where the instance boundary line is truncated by the tile boundary. This is equivalent to determining adjacent tile pairs where at least one instance boundary line is truncated by the boundary of each tile of the pair. By discarding an instance boundary line with a smaller confidence value of the current instance boundary line and another instance boundary line in another tile at block 520, the more accurate instance boundary line is retained.

[0130] Next, the case where the boundary line of the current instance is truncated by the tile boundary is discussed. Then, the method starts at block 525 .

[0131] Box 525: Here, it is determined whether the current instance boundary line (truncated by the tile boundary) at least partially overlaps with another instance boundary line. This partial overlap may require a minimum overlap. If there are no overlapping instance boundary lines, it can be determined that the current instance boundary line may be an artifact, and therefore the current instance boundary line is removed at box 530 before further iterations of box 510 begin.

[0132] Thus, the "yes" output of block 525 is accessed by each adjacent tile pair, for each instance boundary line that is truncated by the boundary of each tile, and has one or more overlapping instance boundary lines. The "yes" output triggers one or more iterations of inner loop 599. For each iteration of output loop 598, one or more iterations of inner loop 599 are executed.

[0133] The merging is based on the confidence values ​​or confidence levels associated with the instance boundaries using the inner loop 599 and the outer loop 598 according to a priority order.

[0134] At block 535 , a next one of the one or more overlapping instance boundary lines is selected.

[0135] Box 540, box 550: implement multi-threshold (here including two thresholds T1, T2) comparison. T1>T2. Multi-threshold comparison facilitates selectively merging the current instance boundary line selected at box 510 and the current overlapping instance boundary line 535 according to the overlap between the current instance boundary line selected at box 510 and the current overlapping instance boundary line 535.

[0136] Box 540: When it is determined that the overlap is above the threshold T1, the method starts at box 545 and merges the two instance boundary lines selected in the current iteration of box 510 and box 535. The integer variable n is incremented once. Then, the method starts at box 560.

[0137] Box 550 : When it is determined that the overlap is below the threshold T1 and above the threshold T2 , remove the instance boundary line of lower priority (eg, with lower confidence value) at box 555 . The method then starts at box 560 .

[0138] If the overlap is even lower than the threshold T2, no action is taken (no merging, no removal). Then, the method starts at block 560 .

[0139] Block 560: Determine whether there is another instance boundary line that overlaps with the current instance boundary line selected at block 510 and has a lower priority than the current instance boundary line selected at block 510 and has not been previously processed in the current iteration of the outer loop 598. If so, perform further iterations of block 535; that is, perform further iterations of the inner loop 599 within the current iteration of the outer loop 598. Otherwise, the method starts at block 565, resulting in further iterations of the outer loop 598.

[0140] Block 565: Determine whether the integer variable is greater than 0, i.e., whether one or more merges of the current instance boundary line of the current iteration of the outer loop 598 have been performed by one or more iterations of block 545. If this is not the case, the current instance boundary line selected at the current iteration of block 510 is removed / discarded at block 530 before further iterations of block 510 are started. Otherwise, the current instance boundary line selected at the current iteration of block 510 is retained and further iterations of block 510 are started.

[0141] A further iteration of the outer loop 598 thus begins.

[0142] Once there is no further current instance boundary line to be processed in further iterations of the outer loop 598, artifact filtering may be performed ( Figure 7 Here, additional instance boundary lines that may contain artifacts can be removed. Finally, an instance map depicting all instance boundary lines superimposed on the original image can be generated.

[0143] Figure 7 This is just an example. Various modifications are conceivable. For example, in Figure 7In the example of , the overlap is only used for the "yes" branch of box 515, that is, for instances that are truncated by a tile boundary. Alternatively or additionally, the overlap can be used for the "no" branch of box 515. Any overlapping instances can be selectively merged or discarded based on the overlap, for example, using a multi-threshold comparison as explained above in conjunction with box 540 and box 550. For example, if the overlap is above a (larger) first threshold, the two instances can be merged; if the overlap is below a (smaller) second threshold, one of the two instances can be discarded, and so on.

[0144] In summary, techniques for facilitating the determination of instance segmentation of large-sized images have been described. These techniques allow processing of images depicting multiple instances of objects of any size, i.e., not only objects that are smaller than the width or length of their respective overlapping regions, but also objects that are larger than the width or length of their respective overlapping regions. Therefore, the techniques disclosed herein can facilitate the accurate and reliable determination of the segmentation map of the entire image. An overlap management process for processing overlapping regions between adjacent tiles of a tile image is disclosed. The overlap management process includes determining the degree of overlap between a first instance boundary line associated with a first tile and a second instance boundary line associated with a second tile, both of which are at least partially arranged in the respective overlapping regions of the first tile and the second tile. The overlap management process also includes, based on the degree of overlap, at least selecting between merging the first instance boundary line and the second instance boundary line and removing one of the first instance boundary line and the second instance boundary line. At least one of the first instance boundary line and the second instance boundary line may be truncated by the boundary of the first tile or the second tile. If one of the first instance boundary line and the second instance boundary line is removed, the removed instance boundary line is an instance boundary line associated with a smaller confidence value.

[0145] Although the invention has been shown and described with respect to certain preferred embodiments, equivalents and modifications will occur to others skilled in the art upon the reading and understanding of the specification. The present invention includes all such equivalents and modifications, and is limited only by the scope of the appended claims.

[0146] For illustrative purposes, the above methods of using multiple threshold comparisons to determine whether to merge two overlapping instances of an object or discard one of the two overlapping instances have been adopted. A single threshold comparison can also be adopted, in which two instances are either merged or one instance is discarded depending on whether the overlap is greater than or less than a single threshold.

Claims

1. A computer-implemented method (4000), comprising: – obtaining (4100) images (2000) depicting a plurality of instances of an object; - decomposing (4200) the image (2000) into a plurality of tiles, wherein adjacent tile pairs (3000) of adjacent tiles (3100, 3200) in the plurality of tiles include respective overlapping regions (3300); - processing (4300) the plurality of image tiles using an instance segmentation algorithm (1000) to obtain respective instance boundary lines of the plurality of instances (3301, 3302, 3303) of the object; and -For each adjacent tile pair and for each instance boundary line intercepted by the boundary (3110) of the tile (3100) of said each adjacent tile pair: selectively merge (4400) said each instance boundary line intercepted by the boundary (3110) of the tile (3100) and said another instance boundary line based on the degree of overlap between said each instance boundary line intercepted by the boundary (3110) of the tile (3100) and another instance boundary line in another tile (3200) of said adjacent tile pair (3000).

2. The computer-implemented method (4000) of claim 1, in, The selective merger includes: - when it is determined that the degree of overlap is above a first predefined threshold: merging the each instance boundary line intercepted by the boundary of the tile with the another instance boundary line.

3. The computer-implemented method (4000) of claim 1 or 2, in, The selective merger includes: -When it is determined that the overlap degree is lower than a first predefined threshold and higher than a second predefined threshold: selecting each instance boundary line or the other instance boundary line that is intercepted by the boundary of the tile, and removing the selected one of each instance boundary line and the other instance boundary line that is intercepted by the boundary of the tile.

4. The computer-implemented method (4000) of claim 3, in, The selective merger includes: - determining a first confidence value associated with said each instance boundary line that is intercepted by said boundary of said tile, and determining a second confidence value associated with said another instance boundary line, The selecting of the each instance boundary line or the another instance boundary line that is truncated by the boundary of the tile is based on a comparison of the first confidence value and the second confidence value.

5. The computer-implemented method (4000) of any preceding claim, further comprising: When it is determined that one or more artifacts are generated after the merging of the each instance boundary line intercepted by the boundary of the tile and the another instance boundary line: removing the merged each instance boundary line intercepted by the boundary of the tile.

6. The computer-implemented method (4000) of any preceding claim, further comprising: - based on the instance boundary lines of the plurality of instances of the object, determining the adjacent tile pairs, in which at least one instance boundary line is truncated by the boundary of the tile of the corresponding adjacent tile pair.

7. The computer-implemented method (4000) according to any one of the preceding claims, -in, The degree of overlap is determined within corresponding overlapping regions of the tile and the other tile of each adjacent tile pair.

8. The computer-implemented method (4000) of any preceding claim, further comprising: - for each instance boundary line that is truncated by the boundary of the tile of the each adjacent tile pair: determining a confidence value; - for each adjacent tile pair: selecting at least one instance boundary line with the highest confidence value; Wherein, the selective merging is further based on the selected at least one instance boundary line.

9. The computer-implemented method (4000) of any one of the preceding claims, in, The image is a microscope image or a satellite image.

10. A computing device (9000), comprising a processor (9020) and a memory (9030), wherein: When loading and executing program code from the memory (9030), the processor (9020) is configured to perform a method (4000) according to any one of the preceding claims.

Images