Image processing method, apparatus, device, and medium

Abstract

An image processing method is provided, comprising: obtaining a multi-cell image to be processed, wherein the multi-cell image contains a plurality of cell objects; extracting global image features from the multi-cell image by using an image feature extraction network; determining a plurality of initial edge point sets and a plurality of corresponding center points of the plurality of cell objects from the global image features by using an initial contour extraction network; determining a plurality of fine edge point sets of the plurality of cell objects from the global image features, the plurality of initial edge point sets and the plurality of corresponding center points by using a final contour extraction network, wherein each fine edge point in each fine edge point set defines a contour of a corresponding cell object in the plurality of cell objects; and segmenting a plurality of single-cell image regions from the multi-cell image by using each contour defined by a corresponding fine edge point set.

Description

[0001] This disclosure claims priority to Chinese Patent Application No. CN202211447479.4, filed on November 18, 2022, with the State Intellectual Property Office of the People's Republic of China, entitled "Image Processing Method, Apparatus, Device and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of image processing technology, specifically to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0003] Single-cell segmentation technology can currently be applied to fields such as drug screening and cytopathological diagnosis, which requires segmenting cells from images one by one for subsequent analysis. Drug validation and cytopathological diagnosis mainly focus on morphological changes in cells. However, most current image segmentation technologies are based on pixel-level instance segmentation. Since the features of pixels in edge regions have weak discriminative power, their ability to focus on edges is poor, which affects the accuracy of image segmentation and cannot meet the requirements for subsequent cell morphology analysis. Summary of the Invention

[0004] It would be beneficial to provide a mechanism to alleviate, reduce, or even eliminate one or more of the aforementioned problems.

[0005] According to one aspect of this disclosure, an image processing method is provided, comprising: acquiring a multicellular image to be processed, wherein the multicellular image contains multiple cell objects; extracting global image features from the multicellular image using an image feature extraction network; determining an initial set of edge points and a corresponding center point for each of the multiple cell objects from the global image features using an initial contour extraction network; determining a set of fine edge points for each of the multiple cell objects from the global image features, the corresponding initial set of edge points, and the corresponding center point using a final contour extraction network, wherein each fine edge point in each set of fine edge points defines the contour of a corresponding cell object among the multiple cell objects; and segmenting multiple single-cell image regions from the multicellular image using the contours defined by the corresponding sets of fine edge points, wherein each single-cell image region corresponds to a corresponding cell object among the multiple cell objects.

[0006] According to another aspect of this disclosure, an image filtering method is provided, comprising: presenting a user interface having an interactive interface component for triggering the filtering of single-cell image regions from a multi-cell image containing multiple single-cell image regions; receiving a trigger operation from a user on the interactive interface component, the trigger operation being associated with filtering conditions for filtering single-cell image regions; and, in response to receiving the trigger operation, filtering out single-cell image regions that meet the filtering conditions from the multi-cell image.

[0007] According to another aspect of this disclosure, an image processing apparatus is provided, comprising: a first module for acquiring a multicellular image to be processed, wherein the multicellular image contains multiple cell objects; a second module for extracting global image features from the multicellular image using an image feature extraction network; a third module for determining an initial set of edge points and a corresponding center point for each of the multiple cell objects from the global image features using an initial contour extraction network; a fourth module for determining a set of fine edge points for each of the multiple cell objects from the global image features, the corresponding initial set of edge points, and the corresponding center point using a final contour extraction network, wherein each fine edge point in each set of fine edge points defines the contour of a corresponding cell object among the multiple cell objects; and a fifth module for segmenting multiple single-cell image regions from the multicellular image using the contours defined by the corresponding sets of fine edge points, wherein each single-cell image region corresponds to a corresponding cell object among the multiple cell objects.

[0008] According to another aspect of this disclosure, an image filtering apparatus is provided, comprising: a sixth module for presenting a user interface having an interactive interface component for triggering the filtering of single-cell image regions from a multi-cell image containing multiple single-cell image regions; a seventh module for receiving a trigger operation from a user on the interactive interface component, the trigger operation being associated with filtering conditions for filtering single-cell image regions; and an eighth module for filtering out single-cell image regions that meet the filtering conditions from the multi-cell image in response to receiving the trigger operation.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory storing instructions executable by the processor, the instructions, when executed by the processor, causing the processor to perform the image processing method and the image filtering method described above.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores instructions which, when executed by a processor, cause the processor to perform the image processing method and the image filtering method described above.

[0011] According to another aspect of this disclosure, a computer program product is provided, comprising: instructions, wherein, when executed by a processor, the instructions cause the processor to perform the above-described image processing method and the above-described image filtering method.

[0012] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0013] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0014] Figure 1 This is a flowchart illustrating an example process of acquiring a multicellular image in an image processing method according to an embodiment of the present disclosure;

[0015] Figure 2 These are example diagrams of multi-cell images according to embodiments of the present disclosure;

[0016] Figure 3 This is a flowchart of an image processing method according to an embodiment of the present disclosure;

[0017] Figure 4 This is an example diagram of the center points of multiple cell objects in a multi-cell image according to embodiments of the present disclosure;

[0018] Figure 5 This is an example diagram of the initial set of edge points of multiple cell objects in a multi-cell image according to embodiments of the present disclosure;

[0019] Figure 6 This is an example diagram of a set of rough edge points of multiple cellular objects in a multicellular image according to embodiments of the present disclosure;

[0020] Figure 7 This is an example diagram of a fine set of edge points of multiple cellular objects in a multi-cell image according to embodiments of the present disclosure;

[0021] Figure 8 This is an example diagram of a single-cell image region according to an embodiment of the present disclosure;

[0022] Figure 9 This is a flowchart of an image filtering method according to an embodiment of the present disclosure;

[0023] Figure 10 This is a block diagram of an image processing apparatus according to embodiments of the present disclosure;

[0024] Figure 11 This is a block diagram of an image filtering apparatus according to embodiments of the present disclosure; and

[0025] Figure 12This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0026] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same example of the element, while in other cases, based on the context, they may refer to different examples.

[0027] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0028] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the relevant field and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0029] As used herein, the term "immune cells," commonly known as white blood cells, includes lymphocytes and various phagocytes, and specifically refers to lymphocytes that can recognize antigens and generate specific immune responses. The term "high-content cell imaging system" refers to a high-resolution microscopy imaging system used to capture cell images. Correspondingly, the term "high-content image" refers to a microscopic image acquired using such a microscopy imaging system.

[0030] In related technologies, the procedure for culturing immune cells includes: expanding Jurkat NFAT-Lucia Reporter Cells to a certain scale. After culturing these cells in well plates for a certain period of time (e.g., 12 hours), immunostimulatory factors (compounds at preset concentrations, such as PMA (Phorbol-12-myristate-13-acetate), PHA (phytohemagglutinin), ConA (concanavalin A), Lonomycin, and OKT3 (Anti-Human CD3 Antibody)) are added to the wells containing the Jurkat NFAT-Lucia Reporter Cells and treated for a certain period of time (e.g., 6 hours).

[0031] In related technologies, the procedure for high-content imaging experiments includes: First, staining is performed on different organelles or cellular components using staining methods commonly used in the field. For example, a dye for staining mitochondria can be added to the wells of a plate beforehand and stained for 0.5 hours. Then, paraformaldehyde is added to each well to fix the cells for 10 minutes. Next, most of the culture medium, the mitochondrial dye, and paraformaldehyde are washed away, leaving only a small volume at the bottom to prevent cell aspiration. Then, a buffer solution containing five additional dyes (for staining the endoplasmic reticulum, Golgi apparatus, ribonucleic acid, nucleus, and actin) is added. After washing, the cells in the plate are placed in a high-content cell imaging system for imaging. Multiple high-content images can be captured from each well.

[0032] Figure 1 This is a flowchart illustrating an example process of acquiring a multicellular image in an image processing method according to an embodiment of the present disclosure. Figure 1 As shown, method 100 includes steps 110 to 120.

[0033] In step 110, a high-content image of the immune cell collection is acquired using a high-content cell imaging system. The high-content image contains multiple immune cells and has N color channels. Each color channel corresponds to a type of organelle or cellular component. N is an integer greater than 3. Organelles are microstructures or microorganisms with specific morphology and function distributed within the cytoplasm, such as mitochondria, endoplasmic reticulum, centrosomes, chloroplasts, Golgi apparatus, and ribosomes. Cellular components are the various chemical components that make up the cell, such as myoglobin.

[0034] In one example, an immune cell assembly may be subjected to the same perturbation conditions and then subjected to fluorescent staining. In one example, the perturbation conditions may be exposure to a predetermined concentration of a predetermined compound for a predetermined duration. In one example, the fluorescent staining process may include staining individual organelles and cellular components with dyes of different colors.

[0035] In one example, a high-content cell imaging system is used to capture high-content images of an array of immune cells subjected to the same perturbation conditions and fluorescently stained.

[0036] In one example, the number of color channels N can be 6. Each color channel can be used to observe organelles or cellular components; for example, 6 color channels can be used to observe mitochondria, endoplasmic reticulum, Golgi apparatus, ribonucleic acid, nucleus, and actin, respectively.

[0037] In step 120, the high-content image is preprocessed to generate a multicellular image.

[0038] In one example, a high-content image can have pixel values ​​ranging from 0 to 65535 and 6 color channels. The high-content image needs to be preprocessed to convert it into a multicellular image (color image) with pixel values ​​ranging from 0 to 255 and 3 channels, facilitating direct observation and subsequent algorithmic analysis. An example of such a multicellular image is shown below. Figure 2 As shown (although it is shown as a grayscale image for the sake of illustration).

[0039] In one example, preprocessing may include normalizing the high-content image. For instance, the pixels of the high-content image may be subjected to min-max normalization so that the value of each pixel is in the range of [0, 1], and then each pixel may be multiplied by 255 so that the pixel value range of the normalized high-content image is 0~255.

[0040] Furthermore, preprocessing may also include merging the N color channels of the normalized high-content image into red-green-blue color channels. For example, the N color channels may include a red channel, a green channel, a blue channel, a red-green channel, a red-blue channel, and a green-blue channel. In one example, the average of the red components in the N color channels is calculated as the merged red channel. The average of the green components in the N color channels is calculated as the merged green channel. The average of the blue components in the N color channels is calculated as the merged blue channel.

[0041] In the example, the merged red channel is R, the merged green channel is G, and the merged blue channel is B. R = (R1 + R2 + R3) / 3, G = (G1 + G2 + G3) / 3, and B = (B1 + B2 + B3) / 3. Here, R1 is the red component in the red channel, R2 is the red component in the red-green channel, and R3 is the red component in the red-blue channel. G1 is the green component in the green channel, G2 is the green component in the red-green channel, and G3 is the green component in the green-blue channel. B1 is the blue component in the blue channel, B2 is the blue component in the green-blue channel, and B3 is the blue component in the red-blue channel.

[0042] In another example, merging the N color channels of a normalized high-content image into red, green, and blue color channels includes steps a and b:

[0043] Step a: For each of the N color channels in the normalized high-content image, determine the corresponding red, green and blue color channels by grayscale proportional mapping.

[0044] In one example of grayscale mapping, step a includes:

[0045] For the first color channel out of N color channels

[0046] Step a.1: Determine the proportion of brightness values ​​of each channel in the red, green and blue color channels corresponding to the first color channel based on the color corresponding to the first color channel;

[0047] The first color channel is any one of the N color channels. Each of the N color channels has its corresponding ratio. For example, for the orange channel, the brightness ratio of each channel in its corresponding RGB channel is [74:255:0]; for the green channel, the brightness ratio of each channel in its corresponding RGB channel is [0:255:0].

[0048] The ratio corresponding to the first color channel can be set by the RGB brightness value of the pixel with the largest brightness value in that color channel. For example, for the orange channel, the RGB brightness value of the pixel with the largest brightness value (or the pixel with a brightness value of 255) can be set to [74, 255, 0]. For the green channel, the RGB brightness value of the pixel with the largest brightness value (or the pixel with a brightness value of 255) can be set to [0, 255, 0].

[0049] Step a.2: Determine the brightness value of the first pixel position in the red, green and blue color channels corresponding to the first color channel based on the brightness value of the first pixel position in the first color channel.

[0050] For example, the RGB brightness value corresponding to the maximum brightness value can be proportionally reduced according to the ratio of the brightness value at the first pixel position to the maximum brightness value. For instance, if the maximum brightness value in the orange channel is 255, and the corresponding RGB brightness value is [74, 255, 0], then a pixel with a brightness value of 40 in the orange channel corresponds to an RGB value of 40 / 255. [74,255,0].

[0051] In this way, each color channel corresponds to a set of RGB channels through a grayscale mapping.

[0052] Step b: For each pixel position, take the maximum pixel value corresponding to that pixel position in the N red channels corresponding to the N color channels as the pixel value of that pixel position in the merged red-green-blue color channel; take the maximum pixel value corresponding to that pixel position in the N green channels corresponding to the N color channels as the pixel value of that pixel position in the merged red-green-blue color channel; take the maximum pixel value corresponding to that pixel position in the N blue channels corresponding to the N color channels as the pixel value of that pixel position in the merged red-green-blue color channel.

[0053] For example, each of the 6 color channels corresponds to one set of RGB channels, so the 6 color channels correspond to a total of 6 R channels, 6 G channels, and 6 B channels. If the consecutive brightness values ​​of the 6 R channels at a certain pixel location are [0, 10, 40, 50, 250, 10], then the combined brightness value of the R channel at that pixel location is the maximum value of 250.

[0054] In this way, the RGB channels corresponding to each of the N channels are merged into one RGB channel.

[0055] This embodiment uses maximum pixel multi-channel fusion, which, compared to mean pixel multi-channel fusion, preserves stronger color signals more fully between different channels, and also helps to reduce observation costs and increase semantic content.

[0056] For example, steps a and b may further include at least one of the following steps: saturated pixel overexposure, saturated pixel blur smoothing, brightness adjustment, and sharpness adjustment. It is understood that these steps act on the processing result of step a, i.e., the RGB channels corresponding to the N color channels.

[0057] The following details the steps for overexposure of saturated pixels.

[0058] Saturated pixels (pixels with a brightness value of 255 in a grayscale image; or pixels with a value of 1 when transitioning to the 0-1 range) are crucial for biologists to determine cell state and distinguish between different cells. For example, in a co-culture system of Primary T cells and Tumor cells, the morphological differences between the two are small. The area of ​​the saturated pixel region in RNA or the Golgi apparatus can distinguish between the two types of cells: larger saturated regions indicate Primary T cells, while no saturated regions indicate Tumor cells. Therefore, in visual imaging, it is necessary to enhance the difference in saturated pixels. For example, the specific steps for overexposure of saturated pixels for each RGB channel are as follows:

[0059] Step A: Convert the image color space from RGB to HSV;

[0060] Step B: Determine whether a pixel is a saturated pixel, and process the pixel value of the saturated point as follows: the H channel (Hue) remains unchanged, the S channel (Saturation) is reduced (e.g., reduced to 20%), and the V channel (Value) is increased (e.g., increased to 180%).

[0061] Step C converts pixel values ​​less than 0 to 0 and pixel values ​​greater than 255 to 255, so that the converted pixel values ​​are in the range [0, 255]. This keeps the H channel unchanged, ensuring that the color remains the same. Decreasing the S channel makes the color whiter, and increasing the V channel increases the brightness.

[0062] Step D: Convert the image color space from HSV to RGB.

[0063] The following is an exemplary illustration of the steps for saturated pixel blurring and smoothing. When saturated pixels are overexposed, they all convert to the same pixel value, resulting in harsh and abrupt color areas. Therefore, this system smooths the difference between the saturated pixels and surrounding pixels by applying a Gaussian blur (σ=1) to the entire overexposed image, then replaces the previous saturated pixels with the new ones in the blurred image, leaving other pixels unchanged. The following is an exemplary illustration of the brightness adjustment steps. The image can be converted to HSV color space, the V channel brightness value adjusted to 90%, and then converted back to the RGB channel to display the image more fully.

[0064] The following example illustrates the sharpness adjustment steps. The sharpness can be adjusted to 250% of its previous level using the Laplacian operator to enhance the clarity of local edge details.

[0065] Furthermore, before normalizing the high-content image, preprocessing can also include: performing median filtering on the high-content image to remove a small number of Gaussian noise points present in the high-content image. This filters out noise while preserving edge information, making the high-content image smoother. Median filtering is a non-linear smoothing technique that sets the gray value of each pixel to the median of the gray values ​​of all pixels within a certain neighborhood window of that pixel.

[0066] Furthermore, before normalizing the high-content image, the preprocessing may further include removing overexposed pixels from the high-content image. Overexposed pixels have brightness values ​​exceeding a brightness threshold. In one example, the brightness threshold may be set based on empirical values. For example, when the pixel value range is between 0 and 65535, the brightness threshold is set to "62314 to 65335". In one example, the pixels in the high-content image may be sorted from high to low brightness values ​​first, and then pixels with a preset proportion (e.g., brightness values ​​in the top 3‰) in the high-content image may be removed as overexposed pixels. In another example, the pixels in the high-content image may be sorted from high to low brightness values ​​first, and then pixels with a first preset proportion (e.g., brightness values ​​in the top 3‰) in the high-content image may be removed as overexposed pixels, while pixels with a second preset proportion (e.g., brightness values ​​in the bottom 5%) in the high-content image may also be removed. For example, between the normalization step in step 120 and step a, at least one of adaptive histogram equalization, adaptive contrast adjustment, and adaptive GAMMA correction may also be included.

[0067] The steps of adaptive histogram equalization are illustrated below. The Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm is used. CLAHE is a variant of adaptive histogram equalization that limits contrast amplification to reduce noise amplification. It operates on sub-regions (tiles) of the image instead of the entire image, and then uses bilinear interpolation to combine adjacent tiles to remove the boundaries of small regions. This algorithm can be used to improve image contrast. CLAHE limits amplification by cropping the histogram to a predefined value before calculating the Cumulative Distribution Function (CDF), limiting the slope of the CDF, and therefore also limiting the slope of the transformation function. The value of the histogram that is cropped, the so-called clip limit, depends on the histogram normalization, and thus on the size of the neighborhood region. CLAHE is equivalent to increasing the local contrast of the image to a certain extent while avoiding noise amplification.

[0068] The following example illustrates adaptive contrast adjustment. Its algorithm normalizes the image contrast, calculates the histogram of the input image or mask region, removes the cut-off percentages of the brightest and darkest pixels from the histogram, and then remaps the image proportionally, making the darkest pixels black and the brightest pixels white, while ignoring preset background pixels. In this system, the cut-off percentage parameter and the background pixel parameter are both set to a default value of 0.

[0069] The following example illustrates adaptive GAMMA correction. First, the adaptive function is used to predict... value, The mean value of the image pixels:

[0070]

[0071] Secondly, for each pixel, the output pixel value corresponding to the input pixel value is determined according to the following formula, where Vin and Vout are the input pixel value and the output pixel value, respectively.

[0072]

[0073] For example, in step 120, the high-content image obtained in step 110 is sequentially preprocessed with minimum-maximum regularization, adaptive histogram equalization, adaptive contrast adjustment, adaptive GAMMA correction, grayscale-color proportional mapping, saturated pixel overexposure, saturated pixel blur smoothing, brightness adjustment, sharpness adjustment, and maximum pixel multi-channel fusion to generate a multi-cell image to be processed. Through an adaptive threshold image processing algorithm, highlighting saturated pixels, combined with brightness and sharpness adjustment, and using maximum pixel multi-channel fusion, an enhanced visual image is output. Compared to existing image processing methods, this enhances the distinguishability of different cell types, improves the fluorescence display of organelles to different perturbations, and achieves automated, high-quality analysis of high-content images, greatly improving the analytical capabilities of researchers or artificial intelligence algorithms.

[0074] Figure 3 This is a flowchart of an image processing method 300 according to an embodiment of the present disclosure. Figure 3 As shown, method 300 includes steps 310 to 350.

[0075] In step 310, a multi-cell image to be processed is obtained. The multi-cell image contains multiple cell objects.

[0076] In one example, the multicellular image to be processed can be obtained using method 100. In another example, the pre-obtained multicellular image to be processed can be read from a local storage device or downloaded from a remote storage device.

[0077] In step 320, global image features are extracted from the multicellular image using an image feature extraction network.

[0078] Global image features refer to features that represent the overall properties of a multicellular image. In one example, global image features include the color, texture, and / or shape of individual cellular objects in a multicellular image.

[0079] Image feature extraction networks are used to extract features from cellular objects in images and can be any suitable convolutional neural network. Image feature extraction networks can be pre-trained based on sample images containing cellular objects.

[0080] In one example, step 320 may include the following steps:

[0081] (a) Perform N iterations of dimensionality reduction on the multicellular image to obtain the first feature map. N is an integer greater than or equal to 1, such as 2.

[0082] In one example, each dimensionality reduction operation includes: performing convolution and pooling on the input of the dimensionality reduction operation, and then adding the output of the convolution and the output of the pooling. Taking a single dimensionality reduction operation as an example: performing pooling operations (e.g., average pooling or max pooling) on ​​a multi-cell image to generate features. Figure 1 (feature Figure 1 The size is reduced (e.g., half the size of the multicellular image), thus lowering the dimensionality of the multicellular image. A convolution operation is performed on the multicellular image to generate features. Figure 2 (feature Figure 2 The size is, for example, half the size of the multicellular image, reducing the dimensionality of the multicellular image. Features Figure 1 and characteristics Figure 2 Add them together to generate features. Figure 3 Features Figure 3 This process is repeated N times, serving as the input for the next dimensionality reduction operation.

[0083] Through N iterations of dimensionality reduction, the feature map size is gradually reduced, and high-level semantics are gradually abstracted, outputting the first feature map. On one hand, pooling operations can retain more initial information of the image. On the other hand, convolution operations can provide more semantic information of the image, making the information after dimensionality reduction richer.

[0084] (b) Perform M iterations of aggregation on the first feature map to obtain the second feature map. M is an integer greater than or equal to 1, such as 5.

[0085] In one example, each aggregation operation involves convolving the following aggregates: i) the input to the aggregation operation; ii) the outputs of multiple consecutive convolutions of the input. Taking a single aggregation operation as an example, convolution operation 2 is used to generate features from the first feature map. Figure 4 Convolution operation 3 is used to extract features. Figure 4 Generate features Figure 5 Next, the features Figure 4 and characteristics Figure 5 The features are obtained by performing an addition operation (fusing semantic information). Figure 6 Next, the features Figure 6 The feature is obtained by performing a concatenation operation (fusion of spatial information) with the first feature map. Figure 7 Then, convolution operation 4 is used to extract features. Figure 7 Generate a second feature map.

[0086] By utilizing M aggregation operations, high-level semantic and spatial information can be abstracted. In this example, aggregation operations are used to perform image feature fusion, that is, to aggregate various layers (e.g., the first feature map, feature...). Figure 4 ,feature Figure 5 The feature map information is fused. On the one hand, feature map information from each layer is fused. On the other hand, channel information between different layers is fused, that is, semantic information and spatial information are fused. Semantic information represents the macroscopic level and is used to determine the category of cell objects (e.g., Jurkat cells and primary T cells). Spatial information represents the microscopic level and is used to determine the location of cell objects.

[0087] (c) Convolve the second feature map to obtain global image features.

[0088] In step 330, the initial contour extraction network is used to determine the initial set of edge points and the corresponding center points of multiple cell objects from global image features.

[0089] Examples of center points of multiple cellular objects in a multi-cell image, such as... Figure 4 As shown, an example of the initial set of edge points for multiple cellular objects in a multicellular image is as follows: Figure 5 As shown. For ease of illustration, these images are shown as grayscale images.

[0090] An initial contour extraction network is used to determine the initial set of edge points and corresponding center points for multiple cell objects, and can be any suitable convolutional neural network. The initial contour extraction network can be pre-trained based on sample images containing multiple cell objects.

[0091] In one example, two different convolution operations are performed on the global image features. One operation extracts a center point feature map to obtain the center point, and the other extracts a center offset feature map to obtain the center offset value. For a single-cell object, the coordinates of the center point and each center offset value are added together to obtain the initial set of edge points for the single-cell object.

[0092] In one example, determining the center points of multiple cell objects from global image features using an initial contour extraction network may include: performing a first convolution operation on the global image features to obtain a single-channel feature heatmap. A feature heatmap is a probability map where pixels closer to a peak point have a probability closer to 1, and pixels farther from the center point have a probability closer to 0. A single-channel feature heatmap includes multiple peak points, each representing a center point of a cell object. In this paper, features or feature maps are represented in the format [B, C, H, W], where B represents the BatchSize, C represents the number of channels, H represents the height of the feature or feature map, and W represents the width of the feature or feature map. For example, the global image features are [1, 64, 272, 272]. The first convolution operation transforms the multi-channel (64 in this example) global image features into a single-channel feature heatmap. A single-channel feature heatmap can be made to the same size as the global image features, serving as a center point feature map, i.e., global image features [1, 64, 272, 272] -> center point feature map [1, 1, 272, 272]. In this paper, the symbol "->" is used to connect two objects, indicating that the latter is generated from the former. Each peak point in the center point feature map represents the center of a cell object, i.e., the center point. For example, 72 center points are identified in the center point feature map, meaning that the multi-cell image contains 72 cell objects. Each center point can be represented by 2D coordinates (x, y), i.e., global image features [1, 1, 272, 272] -> center point set [1, 72, 2].

[0093] In one example, using an initial contour extraction network to determine the set of edge points corresponding to multiple cell objects from global image features may include the following steps.

[0094] (a) A second convolution operation is performed on the global image features to obtain a multi-channel feature heatmap. Each channel in the multi-channel feature heatmap includes multiple target points with the same relative position as multiple peak points in the single-channel feature heatmap. The value of each target point represents the offset relative to a corresponding center point among the center points of multiple cell objects. Therefore, such a multi-channel feature heatmap is also called a center offset feature map. Continuing the previous example, the global image features are transformed into a multi-channel feature heatmap with twice the number of preset edge points through the second convolution operation. For example, if the preset number of edge points is 256, the number of channels in the center offset feature map is 256x2=512 (where each edge point has a 2D offset (Δx,Δy) relative to a corresponding center point), i.e., global image features [1, 64, 272, 272] -> center offset feature map [1, 512, 272, 272]. In the center offset feature map, the values ​​of two adjacent channels at each center point (x,y) represent the 2D offset (Δx,Δy) relative to the center point (x,y). This 2D offset (Δx,Δy) represents the corresponding offset of an edge point relative to the center point (x,y) in the x and y directions. Continuing with the previous example where the center point feature map contains 72 center points (x,y), for 72 center points at different locations, the set of center offsets can be represented as [1, 72, 256, 2]. That is, the set of center offsets [1, 72, 256, 2] is obtained through the global image features [1, 512, 272, 272] and the set of center points [1, 72, 2].

[0095] (b) For each cell object among multiple cell objects, calculate the coordinates of each edge point of the cell object based on the coordinates of its center point and the various offsets relative to its center point, and use this as the initial set of edge points. For example, add each 2D offset (Δx, Δy) in the center offset set [1, 72, 256, 2] to the coordinates (x, y) of the corresponding center point in the center point set [1, 72, 2] to obtain a set of edge point coordinates. That is, center point coordinates (x, y) + center offset values ​​(Δx, Δy) = edge point coordinates (x + Δx, y + Δy). Thus, the initial set of edge points [1, 72, 256, 2] is formed. Next, the initial set of edge points [1, 72, 256, 2] and the set of center points [1, 72, 1, 2] are concatenated to obtain the set of "initial edge points + center points" [1, 72, 257, 2] (256 edge points + 1 center point = 257 points).

[0096] Cells typically exhibit different morphologies under different activation states. For example, at lower activation levels, cell edges are more rounded, while at higher activation levels, cell edges are jagged. Therefore, it is difficult to represent all cell objects with a simple preset shape. In the example described above, edge points are obtained by determining the corresponding initial set of edge points and corresponding center points for multiple cell objects, which can adapt to the contour extraction of various cell morphologies.

[0097] In step 340, the final contour extraction network is used to determine the corresponding fine edge point sets for multiple cell objects from global image features, the corresponding initial edge point sets, and the corresponding center points. Each fine edge point in each fine edge point set defines the contour of a corresponding cell object among the multiple cell objects.

[0098] The final contour extraction network is used to determine the fine set of edge points corresponding to multiple cell objects, and can be any suitable convolutional neural network. The final contour extraction network can be pre-trained based on sample images containing multiple cell objects.

[0099] In one example, step 340 may include the following steps.

[0100] (a) Perform a third convolution operation on the global image features to obtain a third feature map. In the example, the third feature map has the same size as the global image features. For example, perform a third convolution operation on the global image features, where the dimensions are first increased and then decreased (i.e., channel fusion is performed to enhance the learning ability of the network), and the output is the third feature map, i.e., global image features [1,64, 272, 272] -> increased to obtain [1, 256, 272, 272] -> decreased to obtain the third feature map [1, 64, 272,272].

[0101] (b) Based on the coordinates of each initial edge point in the corresponding initial edge point set and the coordinates of the corresponding center point, extract the first multi-point feature vector map from the third feature map. The first multi-point feature vector map is formed by concatenating the feature values ​​at the coordinates of each initial edge point and the corresponding center point extracted from the third feature map. For example, map the 256 initial edge points and 1 center point of each cell object to the third feature map, and extract the first multi-point feature vector map, i.e., obtain the first multi-point feature vector map [1, 64, 272, 272] from the third feature map [1, 72, 257, 2]. More specifically, for the 257 coordinate points of each of the 72 single cells (the initial edge point set includes 256 edge point coordinates, and the center point includes 1 center point coordinate), 64 feature vectors can be extracted for each coordinate point in the 64-dimensional channel. 1. Feature vector. Extracting all 64 features from the 257 coordinate points of each of the 72 single cells. The feature vectors of 1 are concatenated to obtain the first multi-point feature vector map [1, 72, 257, 64].

[0102] (c) The first multi-point feature vector map is transformed into a fourth feature map using a multilayer perceptron. The fourth feature map indicates the coarse offset of each edge point of multiple cell objects relative to the corresponding center point of the multiple cell objects. For example, the fourth feature map is obtained by abstracting high-level semantic information from the first multi-point feature vector map through a series of linear transformations. This process can be schematically represented as follows: First multi-point feature vector map [1, 72, 257, 64] -> matrix dimension reduction to obtain [1, 72, 16448] -> dimension reduction to obtain [1, 72, 1024] -> dimension reduction to obtain [1, 72, 512] -> matrix dimension reduction to obtain the fourth feature map [1, 72, 256, 2].

[0103] (d) For each of the multiple cell objects, calculate the coordinates of each edge point of the cell object based on the coordinates of the center point of the cell object and the various rough offsets relative to the center point of the cell object, and use them as a set of rough edge points.

[0104] Continuing with the previous example, each 2D offset in the fourth feature map [1, 72, 256, 2] is added to the coordinates of the corresponding center point in the center point set [1, 72, 2] to obtain the rough edge point set [1, 72, 256, 2].

[0105] Examples of coarse edge point sets of multiple cellular objects in multicellular images include: Figure 6 As shown. For ease of illustration, it is shown as a grayscale image.

[0106] (e) Perform a fine-tuning operation on the coarse edge point set of each cell object for K iterations to obtain the fine edge point set of the cell object. K is an integer greater than or equal to 1, such as 4.

[0107] Examples of fine-grained edge point sets of multiple cellular objects in multicellular images include: Figure 7 As shown. For ease of illustration, it is shown as a grayscale image.

[0108] In one example, each refinement operation may include the following steps.

[0109] (a) Based on the coordinates of each rough edge point in the rough edge point set, a second multi-point feature vector map is extracted from the global image features. The second multi-point feature vector map is constructed by concatenating the feature values ​​at the coordinates of each rough edge point extracted from the global image features. For example, the 256 rough edge points of each cell object are mapped to the global image features, and the second multi-point feature vector map is extracted, i.e., the second multi-point feature vector map [1, 64, 272, 272] is obtained through the global image features [1, 72, 256, 2] and the rough edge point set [1, 72, 256, 64]. More specifically, for the 256 rough edge points of each of the 72 single cells, 64 features can be extracted for each rough edge point in a 64-dimensional channel. A vector of 1. Extracting all 64 of the 256 rough edge points from each of the 72 individual cells. The vectors of 1 are concatenated to obtain the second multi-point feature vector map [1, 72, 256, 64].

[0110] (b) Perform a fourth convolution operation on the second multi-point feature vector map to obtain the third multi-point feature vector. In the example, the fourth convolution operation can be an annular convolution operation. Annular convolution can capture the relationships between all point pairs in the local neighborhood space, thus better capturing local geometric information. More information about annular convolution can be found in A-CNN: Annularly Convolutional Neural Networks on PointClouds, KOMARICHEV A et. al., Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition.

[0111] (c) Perform a fifth convolution operation on the third multi-point feature vector map. For example, performing a fifth convolution operation on the third multi-point feature vector map transforms the width of the third multi-point feature vector (64 in the example) into the dimension of the fine edge point offsets (2 in the example). This process can be schematically represented as: third multi-point feature vector map [1, 72, 256, 64] -> matrix transformation to obtain [1, 64, 256, 72] -> dimensionality reduction to obtain [1, 2, 256, 72] -> matrix transformation to obtain the fine edge point set [1, 72, 256, 2].

[0112] In one example, the fifth convolution operation can be a deformable convolution operation. Deformable convolution adds an extra direction parameter to each element of the convolution kernel, allowing the kernel to expand to a larger range. The direction parameter can be learned for each location in the feature map. Traditional convolution kernels are fixed and have poor adaptability to unknown variations, resulting in weak generalization ability. In the same layer of a convolutional neural network, different locations may correspond to cell objects of different scales or deformations. Deformable convolution can adaptively and automatically adjust its shape or receptive field according to different locations, thus extracting features more accurately.

[0113] In step 350, multiple single-cell image regions are segmented from the multi-cell image using contours defined by corresponding fine edge point sets. Each single-cell image region corresponds to a specific cell object among multiple cell objects.

[0114] In one example, for each set of fine edge points, the individual fine edge points in that set can be sequentially connected to obtain the outline of a single-cell object. Then, based on the outline of each single-cell object, multiple single-immune cell regions are segmented.

[0115] Examples of single-cell image regions are as follows: Figure 8 As shown. For ease of illustration, it is shown as a grayscale image.

[0116] According to method 300, the edge points of cell objects are determined by predicting the center point and center offset value. The edge points are then refined from initial edge points to coarse edge points, and then to fine edge points, until the outline of the cell object is determined. This progressive method can accurately identify the outline of each cell object.

[0117] In applications that predict the activation state of immune cells based on segmented immune cell objects, image filtering can be performed on multiple single-cell image regions to improve the accuracy of activation state prediction. Image filtering can be performed using at least one of the following filtering methods.

[0118] The first method involves image filtering based on the aspect ratio of cells.

[0119] In one example, at least one single-cell image region can be filtered out from multiple single-cell image regions. Each of these at least one single-cell image regions has a maximum aspect ratio greater than an aspect ratio threshold. For example, first, the maximum aspect ratio of the minimum bounding rectangle of the foreground region in each single-cell image region is calculated, which is the ratio of the longer side to the shorter side of the minimum bounding rectangle. Then, the single-cell image regions corresponding to the maximum aspect ratio greater than the aspect ratio threshold (e.g., 1.5) are filtered out. Since multiple single-cell image regions may include images containing adherent cells, this example can filter out images containing adherent cells.

[0120] The second method involves image filtering based on the area ratio of cells.

[0121] In one example, at least one single-cell image region is filtered out from multiple single-cell image regions. The area of ​​each of the at least one single-cell image regions is greater than a first area ratio threshold and / or less than a second area ratio threshold. The first area ratio threshold is greater than the second area ratio threshold. For example, the pixel area of ​​the single-cell image region is calculated and divided by the total area of ​​the multi-cell image. Single-cell image regions corresponding to area ratios greater than the first area ratio threshold (e.g., 1%) and / or less than the second area ratio threshold (e.g., 4‰) are filtered out. This example can filter images containing adherent cells or images containing cell debris.

[0122] The third method involves image filtering based on brightness ratio.

[0123] In one example, at least one single-cell image region is filtered out from multiple single-cell image regions. The ratio of the sum of the pixel luminance values ​​of each single-cell image region to the area of ​​that single-cell image region is greater than a first luminance threshold and / or less than a second luminance threshold. The first luminance threshold is greater than the second luminance threshold. For example, the ratio of the sum of the luminance values ​​of the single-cell image regions (first converting the color single-cell image to a grayscale image of [0,1], then summing all pixel values) to the area of ​​the single-cell image regions is calculated. Single-cell image regions with ratios greater than the first luminance threshold (e.g., 0.8) and / or less than the second luminance threshold (e.g., 0.2) are filtered out. This example can filter images with bright halos or images with dark backgrounds.

[0124] The fourth method involves image screening based on cell integrity.

[0125] In one example, at least one single-cell image region is filtered out from multiple single-cell image regions. The edges of each of the at least one single-cell image regions overlap with the edges of the multi-cell image. For example, it is determined whether the coordinates of at least one edge contour point of the single-cell image region lie on the edge of the multi-cell image; if so, it is filtered out.

[0126] During the process of capturing multicellular images, some cells may be located at the edge of the imaging area, resulting in some cells not being fully captured. This example can filter images containing incomplete cells due to cells being located at the edge of the imaging area.

[0127] It should be noted that the above thresholds can be set according to the morphological characteristics of different types of cells.

[0128] In one example, the image feature extraction network, the initial contour extraction network, and the final contour extraction network can be trained through the following steps.

[0129] (a) Obtain the sample set. The samples in the sample set include multi-cell images of the samples and multiple contour labels corresponding to multiple cells in the multi-cell images of the samples.

[0130] In one example, the sample multi-cell image can be obtained through the operation of method 100.

[0131] In one example, contour labels could be outlines of cells contained in a sample multicellular image labeled by a professional.

[0132] (b) Use the image feature extraction network to be trained to process the multi-cell images of the samples to obtain global image features of the samples.

[0133] (c) Use the initial contour extraction network to be trained to extract global image features from the samples to determine the initial set of edge points and corresponding center points of multiple cell objects.

[0134] (d) Using the final contour extraction network to be trained, determine the corresponding fine edge point sets for multiple cell objects from global image features, the corresponding initial edge point sets, and the corresponding center points. Each fine edge point in each fine edge point set defines the sample contour of one corresponding cell object among the multiple cell objects.

[0135] In one example, the image feature extraction network to be trained, the initial contour extraction network to be trained, and the final contour extraction network to be trained can be various appropriate types of machine learning models, such as convolutional neural networks.

[0136] (e) Based on the difference values ​​between multiple sample contours and their corresponding multiple contour labels, iteratively adjust the parameters of the image feature extraction network to be trained, the initial contour extraction network to be trained, and the final contour extraction network to be trained until the preset training termination condition is met, and obtain the trained image feature extraction network, the initial contour extraction network, and the final contour extraction network.

[0137] In one example, the preset training termination condition may include any of the following: the difference value is less than a preset difference threshold, the number of training sessions reaches a preset number, and the training time reaches a preset duration.

[0138] Figure 9 This is a flowchart of an image filtering method 900 according to an embodiment of the present disclosure. Figure 9 As shown, method 900 includes steps 910 to 930.

[0139] In step 910, a user interface is presented. The user interface has interactive interface components for triggering the selection of single-cell image regions from a multi-cell image containing multiple single-cell image regions.

[0140] In one example, interactive interface components could be action icons or buttons, etc.

[0141] In step 920, a user trigger operation on an interactive interface component is received, and the trigger operation is associated with the filtering conditions for filtering single-cell image regions.

[0142] In one example, the filtering criteria may include at least one of the following: the maximum aspect ratio of the single-cell image region is greater than an aspect ratio threshold; the ratio of the area of ​​the single-cell image region to the area of ​​the multi-cell image is greater than a first area ratio threshold and / or less than a second area ratio threshold, wherein the first area ratio threshold is greater than the second area ratio threshold; the ratio of the sum of the pixel brightness values ​​of the single-cell image region to the area of ​​the single-cell image region is greater than a first brightness threshold and / or less than a second brightness threshold, wherein the first brightness threshold is greater than the second brightness threshold; and the edges of the single-cell image region overlap with the edges of the multi-cell image.

[0143] In one example, the triggering action could be a double-click, swipe, or drag operation, etc.

[0144] In step 930, in response to receiving a trigger operation, single-cell image regions that meet the screening criteria are filtered out from the multi-cell image.

[0145] Figure 10 This is a block diagram of an image processing apparatus 1000 according to an embodiment of the present disclosure. Figure 10 As shown, the image processing device 1000 includes a first module 1010, a second module 1020, a third module 1030, a fourth module 1040, and a fifth module 1050.

[0146] The first module 1010 is used to acquire a multi-cell image to be processed, wherein the multi-cell image contains multiple cell objects.

[0147] The second module 1020 is used to extract global image features from multicellular images using an image feature extraction network.

[0148] The third module 1030 is used to determine the initial set of edge points and the corresponding center points of multiple cell objects from global image features using the initial contour extraction network.

[0149] The fourth module 1040 is used to determine the corresponding fine edge point set of multiple cell objects from global image features, the corresponding initial edge point set and the corresponding center point using the final contour extraction network, wherein each fine edge point in each fine edge point set defines the contour of a corresponding cell object among the multiple cell objects.

[0150] The fifth module 1050 is used to segment multiple single-cell image regions from a multi-cell image using contours defined by corresponding fine edge point sets, wherein each single-cell image region corresponds to a corresponding cell object among multiple cell objects.

[0151] Figure 11 This is a block diagram of an image filtering apparatus 1100 according to an embodiment of the present disclosure. (See diagram for example.) Figure 11 As shown, the image filtering device 1100 includes a sixth module 1110, a seventh module 1120, and an eighth module 1130.

[0152] The sixth module is used to present the user interface, which has interactive interface components for triggering the selection of single-cell image regions from a multi-cell image containing multiple single-cell image regions.

[0153] The seventh module is used to receive user trigger operations for interactive interface components, and the trigger operations are associated with the filtering conditions for filtering single-cell image regions.

[0154] The eighth module is used to filter out single-cell image regions that meet the screening criteria from the multi-cell image in response to a received trigger operation.

[0155] It should be understood that Figure 10 The various modules of the device 1000 shown can be referenced above. Figure 3 The steps in method 300 described correspond to each other. Therefore, the operations, features and advantages described above for method 300 also apply to device 1000 and its included modules. Figure 11 The various modules of the device 1100 shown can be referenced above. Figure 9 The steps in method 900 described correspond to each other. Therefore, the operations, features, and advantages described above for method 900 also apply to apparatus 1100 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0156] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific actions performed by the modules discussed herein include the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that performs the action, called or otherwise accessed by the specific module.

[0157] It should also be understood that various techniques can be described in the general context of software hardware elements or program modules. The various modules described above with respect to Figures 1000 or 1100 can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit.

[0158] According to one aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory communicatively connected to the at least one processor, the at least one memory storing instructions which, when executed by the at least one processor, cause the at least one processor to perform the method described above.

[0159] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided that stores instructions which, when executed by at least one processor of a computer, cause the computer to perform the methods described above.

[0160] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described above.

[0161] Figure 12 An example configuration of an electronic device 1200 that can be used to implement the methods described herein is shown.

[0162] Electronic device 1200 can be a variety of different types of devices. Examples of electronic device 1200 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0163] Electronic device 1200 may include at least one processor 1202, memory 1204, multiple communication interfaces 1206, display device 1208, other input / output (I / O) devices 1210, and one or more mass storage devices 1212 capable of communicating with each other, such as via system bus 1214 or other suitable connections.

[0164] Processor 1202 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 1202 may be configured to acquire and execute computer-readable instructions stored in memory 1204, mass storage device 1212, or other computer-readable media, such as program code of operating system 1216, program code of application program 1218, program code of other program 1220, etc.

[0165] Memory 1204 and mass storage device 1212 are examples of computer-readable storage media for storing instructions executed by processor 1202 to perform the various functions described above. For example, memory 1204 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 1212 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 1204 and mass storage device 1212 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 1202 as a specific machine configured to perform the operations and functions described in the examples herein.

[0166] Multiple programs can be stored on mass storage device 1212. These programs include operating system 1216, one or more application programs 1218, other programs 1220, and program data 1222, and they can be loaded into memory 1204 for execution. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the following components / functions: first module 910, second module 920, third module 930, fourth module 940, fifth module 950, sixth module 960, seventh module 970, eighth module 980, method 3 (including any suitable steps of method 300), method 9 (including any suitable steps of method 900), and / or other embodiments described herein.

[0167] Although Figure 12 The modules 1216, 1218, 1220, and 1222, or portions thereof, are illustrated as being stored in memory 1204 of electronic device 1200; however, modules 1216, 1218, 1220, and 1222, or portions thereof, may be implemented using any form of computer-readable medium accessible by electronic device 1200. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0168] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by electronic devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0169] One or more communication interfaces 1206 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, Near Field Communication (NFC) interface, etc. Communication interface 1206 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 1206 can also provide communication with external storage devices (not shown), such as storage arrays, network-attached storage, storage area networks, etc.

[0170] In some examples, a display device 1208, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 1210 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0171] The technologies described herein can be supported by these various configurations of electronic device 1200, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from electronic device 1200. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect electronic device 1200 to other electronic devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on electronic device 1200 and partly through the platform that abstracts the functionality of the cloud.

[0172] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practice with respect to the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, the indefinite article "a" or "an" does not exclude a plurality, and the term "a plurality" means two or more. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. An image processing method, comprising: Acquire a multi-cell image to be processed, wherein the multi-cell image contains multiple cell objects; Global image features are extracted from the multicellular image using an image feature extraction network; An initial contour extraction network is used to determine the initial set of edge points and the corresponding center points of the multiple cell objects from the global image features; A final contour extraction network is used to determine the corresponding fine edge point sets for the plurality of cell objects from the global image features, the corresponding initial edge point sets, and the corresponding center points, wherein each fine edge point in each fine edge point set defines the contour of a corresponding cell object among the plurality of cell objects; and Multiple single-cell image regions are segmented from the multi-cell image using the respective contours defined by the corresponding fine edge point set, wherein each single-cell image region corresponds to a corresponding cell object among the multiple cell objects; The step of using an initial contour extraction network to determine the initial set of edge points and corresponding center points of the plurality of cell objects from the global image features includes: The global image features are subjected to a first convolution operation to obtain a single-channel feature heatmap, which includes multiple peak points, each peak point representing the coordinates of the corresponding center point of the multiple cell objects. A second convolution operation is performed on the global image features to obtain a multi-channel feature heatmap. Each channel of the multi-channel feature heatmap includes multiple target points with the same relative positions as the multiple peak points in the single-channel feature heatmap. The value of each target point represents an offset relative to a corresponding center point among the center points of the multiple cell objects. For each of the plurality of cell objects, the coordinates of each edge point of the cell object are calculated based on the coordinates of the center point of the cell object and the various offsets relative to the center point of the cell object, and are used as the initial set of edge points.

2. The method as described in claim 1, wherein, The step of extracting global image features from the multicellular image using an image feature extraction network includes: The multi-cell image is subjected to N iterations of dimensionality reduction to obtain a first feature map, where N is an integer greater than or equal to 1; The first feature map is aggregated M times to obtain a second feature map, where M is an integer greater than or equal to 1; and The second feature map is convolved to obtain the global image features.

3. The method as described in claim 2, wherein, Each dimensionality reduction operation includes: The input to this dimensionality reduction operation is subjected to convolution and pooling respectively, and the output of the convolution is added to the output of the pooling.

4. The method of claim 2, wherein, Each aggregation operation includes: Convolution is performed on the following aggregates: i) the input of the aggregation operation; ii) the outputs of multiple consecutive convolutions of the input.

5. The method of claim 1, wherein, The step of determining the corresponding fine edge point set of the multiple cell objects from the global image features, the corresponding initial edge point set, and the corresponding center point using the final contour extraction network includes: A third convolution operation is performed on the global image features to obtain a third feature map; Based on the coordinates of each initial edge point in the corresponding initial edge point set and the coordinates of the corresponding center point, a first multi-point feature vector map is extracted from the third feature map; The first multi-point feature vector map is converted into a fourth feature map by a multilayer perceptron, and the fourth feature map indicates the coarse offset of each edge point of the plurality of cell objects relative to the corresponding center point of the plurality of cell objects. For each of the plurality of cell objects, the coordinates of each edge point of the cell object are calculated based on the coordinates of its center point and various coarse offsets relative to its center point, forming a set of coarse edge points; and For each cell object, the set of coarse edge points is refined K times to obtain the set of fine edge points for that cell object, where K is an integer greater than or equal to 1.

6. The method of claim 5, wherein, Each refinement operation includes: Based on the coordinates of each rough edge point in the set of rough edge points, a second multi-point feature vector map is extracted from the global image features. A fourth convolution operation is performed on the second multi-point feature vector map to obtain a third multi-point feature vector; and A fifth convolution operation is performed on the third multi-point feature vector map.

7. The method of claim 6, wherein, The fourth convolution operation is a circular convolution operation, and the second convolution operation is a deformable convolution operation.

8. The method of claim 1, wherein, The acquisition of the multi-cell image to be processed includes: A high-content image of the cell object set is acquired by a high-content cell object imaging system, wherein the high-content image contains multiple cell objects and has N color channels, each color channel corresponding to a type of organelle or cell component, and N is an integer greater than 3; and The high-content image is preprocessed to generate the multicellular image.

9. The method of claim 8, wherein, The preprocessing of the high-content image includes: Normalize the high-content image; and The N color channels of the normalized high-content image are merged into red, green and blue color channels.

10. The method of claim 9, wherein, The preprocessing of the high-content image further includes: Before normalizing the high-content image, perform at least one of the following: Median filtering is applied to the high-content image; and Remove overexposed pixels from the high-content image, the overexposed pixels having a brightness value exceeding a brightness threshold.

11. The method of claim 9, wherein, The step of merging the N color channels of the normalized high-content image into red, green, and blue color channels includes: Calculate the average value of the red components in the N color channels, and use it as the merged red channel; Calculate the average value of the green components in the N color channels, and use this average value as the merged green channel; and Calculate the average value of the blue components in the N color channels, and use it as the merged blue channel.

12. The method of claim 9, wherein, The N color channels of the normalized high-content image are merged into red, green, and blue color channels, including: For each of the N color channels in the normalized high-content image, the corresponding red, green and blue color channels are determined by grayscale proportional mapping. For each pixel position, the maximum pixel value corresponding to that pixel position in the N red channels of the N color channels is used as the pixel value of that pixel position in the red channel of the merged red-green-blue color channel; the maximum pixel value corresponding to that pixel position in the N green channels of the N color channels is used as the pixel value of that pixel position in the green channel of the merged red-green-blue color channel; and the maximum pixel value corresponding to that pixel position in the N blue channels of the N color channels is used as the pixel value of that pixel position in the blue channel of the merged red-green-blue color channel.

13. The method of claim 12, wherein, For each of the N color channels in the normalized high-content image, the corresponding red, green, and blue color channels are determined through grayscale proportional mapping, including: For the first color channel among N color channels, determine the proportion of brightness values ​​of each channel in the red, green and blue color channels corresponding to the first color channel based on the color corresponding to the first color channel; determine the brightness value of the first pixel position in the red, green and blue color channels corresponding to the first color channel based on the brightness value of the first pixel position in the first color channel.

14. The method according to any one of claims 1-13, further comprising: At least one single-cell image region is filtered out from the plurality of single-cell image regions, wherein the at least one single-cell image region satisfies at least one of the following conditions: The maximum aspect ratio of each of the at least one single-cell image regions is greater than the aspect ratio threshold. The ratio of the area of ​​each of the at least one single-cell image regions to the area of ​​the multi-cell image is greater than a first area ratio threshold and / or less than a second area ratio threshold, wherein the first area ratio threshold is greater than the second area ratio threshold. The ratio of the sum of pixel brightness values ​​of each single-cell image region in the at least one single-cell image region to the area of ​​the single-cell image region is greater than a first brightness threshold and / or less than a second brightness threshold, wherein the first brightness threshold is greater than the second brightness threshold. The edges of the single-cell image regions of each of the at least one single-cell image regions overlap with the edges of the multi-cell image; Each of the at least one single-cell image regions satisfies a filtering condition associated with a user's triggering action on an interactive interface component.

15. The method of claim 14, wherein, The step of filtering out at least one single-cell image region from the plurality of single-cell image regions is performed in response to receiving the trigger operation, and the method further includes: A user interface is presented, which has the interface component for triggering the selection of single-cell image regions from a multi-cell image containing multiple single-cell image regions. Receive the user's trigger operation on the interactive interface component.

16. An image processing apparatus, comprising: The first module is used to acquire a multi-cell image to be processed, wherein the multi-cell image contains multiple cell objects; The second module is used to extract global image features from the multicellular image using an image feature extraction network; The third module is used to determine the corresponding initial edge point set and corresponding center point of the multiple cell objects from the global image features using the initial contour extraction network. The fourth module is used to determine, using a final contour extraction network, a set of fine edge points corresponding to the plurality of cell objects from the global image features, the corresponding initial set of edge points, and the corresponding center points, wherein each fine edge point in each set of fine edge points defines the contour of a corresponding cell object among the plurality of cell objects; and The fifth module is used to segment multiple single-cell image regions from the multi-cell image using the contours defined by the corresponding fine edge point set, wherein each single-cell image region corresponds to a corresponding cell object among the multiple cell objects; The step of determining the initial set of edge points and corresponding center points of the plurality of cell objects from the global image features using an initial contour extraction network includes: The first convolution operation is performed on the global image features to obtain a single-channel feature heatmap, which includes multiple peak points, each peak point representing the coordinates of the corresponding center point of the multiple cell objects. A second convolution operation is performed on the global image features to obtain a multi-channel feature heatmap. Each channel of the multi-channel feature heatmap includes multiple target points with the same relative positions as the multiple peak points in the single-channel feature heatmap. The value of each target point represents an offset relative to a corresponding center point among the center points of the multiple cell objects. For each of the plurality of cell objects, the coordinates of each edge point of the cell object are calculated based on the coordinates of the center point of the cell object and the various offsets relative to the center point of the cell object, and used as the initial set of edge points.

17. An electronic device comprising: processor; as well as A memory storing instructions executable by the processor, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1-15.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-15.

19. A computer program product comprising instructions, wherein, when executed by a processor, the instructions cause the processor to perform the method according to any one of claims 1-15.

Images