Method and system for image segmentation

Abstract

The invention discloses a method and a system for image segmentation. The method for image segmentation comprises the steps of obtaining a to-be-segmented two-dimensional image and a first user prompt corresponding to the to-be-segmented two-dimensional image; using a two-dimensional affinity prediction module to generate a two-dimensional affinity graph based on the two-dimensional image to be segmented; converting the first user prompt into a first prompt graph; generating, using a two-dimensional mask prediction module, a first two-dimensional mask based on the two-dimensional affinity map and the first cue map; and performing connected domain analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different marks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a method and system for image segmentation. Background Art

[0002] In neuroscience research, the reconstruction of neurons is an important means to understand the organization and function of the nervous system. Electron microscopes, due to their ultra-high resolution, can clearly capture the fine structure of neurons and their synapses, making them a key tool for reconstructing dense neuronal connections. However, the huge amount of data brought by ultra-high resolution has greatly increased the burden of manual annotation. At present, the segmentation of neurons mainly uses automated methods based on deep learning. These methods use labeled data to train models to achieve neuron segmentation of specific species. Although existing automated neuron segmentation methods can achieve good segmentation effects on image data of some species, for the processing of non-model organisms and new species, due to the lack of corresponding labeled data in existing models, it is usually difficult to achieve a satisfactory level of segmentation accuracy, and the automated segmentation results often require a lot of manual proofreading. Therefore, there is an urgent need for a fast and accurate image segmentation method and system. Summary of the invention

[0003] It is to be understood that both the foregoing general description and the following detailed description of the present invention are exemplary and explanatory and are intended to provide further explanation of the invention as claimed.

[0004] According to a first aspect of the present invention, a method for image segmentation is provided, comprising: obtaining a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented; using a two-dimensional affinity prediction module to generate a two-dimensional affinity map based on the two-dimensional image to be segmented; converting the first user prompt into a first prompt map; using a two-dimensional mask prediction module to generate a first two-dimensional mask based on the two-dimensional affinity map and the first prompt map; and performing a connected domain analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

[0005] In the above method, the first user prompt includes a positive prompt or a negative prompt for a pixel in the two-dimensional image to be segmented, the positive prompt is used to prompt that the pixel belongs to the foreground, and the negative prompt is used to prompt that the pixel belongs to the background.

[0006] In the above method, the first user prompt is in any one or more of the following forms: point prompt, shape prompt or area prompt.

[0007] In the above method, when the first user prompt is empty, the first prompt map is a default matrix corresponding to the two-dimensional image to be segmented; or when the first user prompt is not empty, the first prompt map includes distribution information obtained by applying a distribution function to the first user prompt.

[0008] In the above method, the method also includes: obtaining a second user prompt based on the first segmented two-dimensional image and corresponding to the two-dimensional image to be segmented; converting the second user prompt into a second prompt map; using the two-dimensional mask prediction module to generate a second two-dimensional mask based on the two-dimensional affinity map and the second prompt map; and performing a connected domain analysis on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0009] In the above method, the two-dimensional mask prediction module is trained through the following steps: obtaining a data set, the data set including a two-dimensional image to be segmented and a user prompt corresponding to the two-dimensional image to be segmented as input information, and including a true value of the two-dimensional mask of the segmented two-dimensional image as output information; using the two-dimensional affinity prediction module to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the data set; converting the user prompt in the data set into a prompt map; using the two-dimensional mask prediction module to generate a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the prompt map; calculating a loss function between the prediction result of the two-dimensional mask and the true value of the two-dimensional mask of the segmented two-dimensional image in the data set; and updating the parameters of the two-dimensional mask prediction module based on the loss function to obtain a trained two-dimensional mask prediction module.

[0010] In the above method, the loss function includes a loss regularization term, which is the product of the true value of the affinity and the predicted value of the mask.

[0011] In the above method, the two-dimensional mask prediction module updates parameters by an online learning method.

[0012] In the above method, the method also includes: acquiring a three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice in the three-dimensional image to be segmented; using a three-dimensional affinity prediction module to generate a three-dimensional affinity map based on the three-dimensional image to be segmented; replacing the two-dimensional image to be segmented in the three-dimensional image to be segmented with the first two-dimensional mask to generate a replaced three-dimensional image; using a three-dimensional mask prediction module to generate a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image; and performing a connected domain analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different areas in the segmented three-dimensional image have different labels.

[0013] In the above method, the two-dimensional image to be segmented is a neuron electron microscope image, and different regions in the first segmented two-dimensional image correspond to different neurons.

[0014] According to a second aspect of the present invention, a method for image segmentation is provided, comprising: acquiring a three-dimensional image to be segmented; using a three-dimensional affinity prediction module to generate a three-dimensional affinity graph based on the three-dimensional image to be segmented; using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; using a three-dimensional mask prediction module to generate a three-dimensional mask based on the three-dimensional affinity graph and the replaced three-dimensional image; and performing a connected domain analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0015] In the above method, using a two-dimensional mask prediction module to generate a two-dimensional mask based on a slice in the three-dimensional image to be segmented includes: obtaining a user prompt corresponding to the slice; and using the two-dimensional mask prediction module to generate the two-dimensional mask based on the slice and the user prompt.

[0016] In the above method, the three-dimensional mask prediction module is trained through the following steps: obtaining a data set, the data set including a three-dimensional image to be segmented as input information, and including a true value of a three-dimensional mask of a segmented three-dimensional image as output information; using the three-dimensional affinity prediction module to generate a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set; using the two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; using the three-dimensional mask prediction module to generate a predicted result of the three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image; calculating a loss function between the predicted result of the three-dimensional mask and the true value of the three-dimensional mask of the segmented three-dimensional image in the data set; and updating the parameters of the three-dimensional mask prediction module based on the loss function to obtain a trained three-dimensional mask prediction module.

[0017] In the above method, the three-dimensional mask prediction module updates parameters through an online learning method.

[0018] In the above method, the three-dimensional image to be segmented is a neuron electron microscope image, and different regions in the segmented three-dimensional image correspond to different neurons.

[0019] According to a third aspect of the present invention, a system for image segmentation is provided, comprising a device for executing any one of the above methods.

[0020] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements any one of the above methods when executed by a processor.

[0021] According to a fifth aspect of the present invention, there is provided a computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein the processor implements a method as described in any one of the above methods when executing the computer program.

[0022] According to a sixth aspect of the present invention, there is provided a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method as described in any one of the above methods is implemented.

[0023] The method and system for image segmentation according to the embodiments of the present invention can achieve fast and accurate image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of the present invention, and are incorporated into and constitute a part of this application. The accompanying drawings illustrate embodiments of the present invention and together with the specification serve to explain the principles of the present invention. In the accompanying drawings:

[0025] Figure 1 is a flow chart of a method for two-dimensional image segmentation according to one embodiment of the present invention;

[0026] Figure 2 is a schematic diagram of a process of two-dimensional image segmentation according to an embodiment of the present invention;

[0027] Figure 3 is a flow chart of a method for two-dimensional image segmentation according to another embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a process of two-dimensional image segmentation and correction according to an embodiment of the present invention;

[0029] Figure 5 is a flowchart of a method for training a two-dimensional mask prediction module according to an embodiment of the present invention;

[0030] Figure 6 is a flow chart of a method for three-dimensional image segmentation according to one embodiment of the present invention;

[0031] Fig. 7A is a schematic diagram of a process of three-dimensional image segmentation according to an embodiment of the present invention;

[0032] Figure 7Bis a schematic diagram of a process of three-dimensional image segmentation according to another embodiment of the present invention;

[0033] Figure 8 is a flowchart of a method for training a three-dimensional mask prediction module according to an embodiment of the present invention;

[0034] Fig. 9 is a flow chart of a method for three-dimensional image segmentation according to another embodiment of the present invention;

[0035] Fig.10 is a block diagram of a system for two-dimensional image segmentation according to an embodiment of the present invention;

[0036] Fig.11 is a block diagram of a system for three-dimensional image segmentation according to an embodiment of the present invention;

[0037] Fig.12 is a block diagram of a system for three-dimensional image segmentation according to another embodiment of the present invention;

[0038] Figures 13A-13D is a result graph of the performance evaluation of two-dimensional image segmentation; and

[0039] Fig.14 It is the result graph of the performance evaluation of 3D image segmentation. DETAILED DESCRIPTION

[0040] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings, but the present invention is not limited thereto but only by the claims. In the accompanying drawings, for illustrative purposes, the sizes of some of the elements may be exaggerated and not drawn to scale. Wherever possible, the same reference numerals will be used in all drawings to represent the same or similar parts.

[0041] Although the terms used in the present invention are selected from well-known and commonly used terms, some of the terms mentioned in the present invention specification may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand the present invention not only by the actual terms used, but also by the meaning contained in each term.

[0042] In the description provided herein, numerous specific details are set forth. However, it should be understood that embodiments of the present invention may be practiced without these specific details. In other examples, well-known methods, structures, and techniques are not shown in detail to avoid obscuring an understanding of the present invention.

[0043] Although existing neuron image segmentation methods can achieve good results on specific species using deep learning technology, they still have some obvious defects. On the one hand, existing methods are highly dependent on large-scale annotated data, which may be difficult to obtain in different species, especially in non-model organisms, making it difficult for segmentation models to generalize to new species. On the other hand, existing segmentation models are usually unable to cope with complex neuronal structural morphology, resulting in a large amount of manual proofreading of segmentation results, which increases the consumption of time and resources.

[0044] In view of the above problems, the present application proposes a method and system for image segmentation. Although for the sake of clarity, the present application mainly describes neuron segmentation in neuron images, the present application can also be applied to other biological data or different types of image data sets, for example, the segmentation of cells in animal and plant tissue sections, microscopic biological images, etc.

[0045] Due to the limitations of imaging technology, three-dimensional images of neurons under electron microscopes are usually generated by superimposing two-dimensional slice images. Figure 1 is a flowchart of a method 100 for two-dimensional image segmentation according to an embodiment of the present invention. Figure 2 It is a schematic diagram of a process of two-dimensional image segmentation according to an embodiment of the present invention.

[0046] At step 102, a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented may be obtained. In one embodiment, the two-dimensional image to be segmented may be a neuron electron microscope image. In one embodiment, the first user prompt may include a positive prompt or a negative prompt for a pixel in the two-dimensional image to be segmented, where a positive prompt is used to prompt that the pixel belongs to the foreground, and a negative prompt is used to prompt that the pixel belongs to the background. A pixel belonging to the foreground means that the pixel belongs to the neuron to be segmented, while a pixel belonging to the background means that the pixel does not belong to a neuron, or belongs to a neuron that does not need to be segmented.

[0047] like Figure 2 As shown, the user can provide the original slice of the electron microscope image as the two-dimensional image to be segmented, and a user prompt can be provided for the original slice. Figure 2 In the example, the user prompt may include a positive prompt represented by a green five-pointed star, indicating that the pixel belongs to the foreground, that is, the neuron to be segmented. The user prompt may also include a negative prompt represented by a red five-pointed star, indicating that the pixel belongs to the background, that is, it cannot be segmented or does not need to be segmented.

[0048] In one embodiment, the first user prompt may be in any one or more of the following forms: a point prompt, a shape prompt, or an area prompt. For example, the user prompt may be provided by clicking a corresponding position on the image through an input device (e.g., a mouse, a keyboard, etc.), thereby providing a prompt for the pixels at the corresponding position. Alternatively, the user prompt may be provided by delimiting a shape or drawing an area at a corresponding position on the image through an input device, thereby providing a prompt for the pixels contained in the shape or area.

[0049] At step 104, a two-dimensional affinity prediction module may be used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented. The two-dimensional affinity prediction module may use a two-dimensional version of a suitable affinity prediction model such as a Local Shape Descriptors (LSD) neural segmentation model (ACRLSD), a MALA (Metropolis-Adjusted Langevin Algorithm) model, etc. Figure 2 As shown, the 2D affinity prediction module can take the original 2D slice as input and predict the affinity between neurons to generate a 2D affinity map.

[0050] At step 106 , the first user prompt may be converted into a first prompt map.

[0051] In one embodiment, when the first user prompt is empty, the first prompt map may be a default matrix corresponding to the two-dimensional image to be segmented. For example, when the user prompt is empty, that is, the user does not provide a prompt, a default matrix corresponding to the original slice size may be used as the prompt map. All values ​​of the default matrix are the same, such as a matrix with all values ​​being one, a matrix with all values ​​being two, and so on.

[0052] In one embodiment, when the first user prompt is not empty, the first prompt map may include distribution information obtained by applying a distribution function to the first user prompt.

[0053] For example, when the user prompts include only positive prompts, or when the user prompts include positive prompts and negative prompts, a positive distribution function can be applied to the positive prompt points, and a negative distribution function can be applied to the negative prompt points, and then a prompt map containing distribution information of all prompt points is obtained.

[0054] by Figure 2 For example, when the user prompts include both positive prompts represented by green five-pointed stars and negative prompts represented by red five-pointed stars, the prompt map includes different types of scattered points corresponding to different types of prompts, and the scattered points have a certain area.

[0055] For example, when the user prompts include only negative prompts, a distribution function is applied to the negative prompt points, and the distribution information of the negative prompt points is subtracted from the default matrix to obtain a prompt map.

[0056] The following is an example of a conversion rule for converting from user prompts to prompt graphs:

[0057]

[0058] PM stands for prompt picture. is a set of point prompts, μ is the position of a point prompt, ρ∈{-1,1} is the type of point prompt, ρ=1 represents a positive prompt, ρ=-1 represents a negative prompt, is the covariance matrix, X represents all pixels on the two-dimensional image, and J is the default matrix (for example, an all-one matrix with the same shape as the two-dimensional image is used here). The value of the covariance matrix can also be set to other values ​​as needed.

[0059] When the user does not provide a hint, the collection of hints When empty, sets the hint map to the default value (J), indicating that masks should be predicted for all neurons.

[0060] When the user gives a positive hint (ρ=1), it includes giving only positive hints or giving both positive hints and negative hints, indicating that only neurons selected by positive hints should be segmented, areas with negative hints should be deleted, and other areas without hints should be ignored, that is, other areas without hints should not be processed.

[0061] When the user gives only negative hints (ρ=-1), it indicates that all neurons not selected by the negative hints should be segmented.

[0062] For example, when the user provides a cue point at the (x, y) coordinate of the image to be segmented through an input device, a Gaussian distribution is generated with the coordinate as the center. If the cue point is a positive cue, the Gaussian distribution is multiplied by a positive coefficient. If the cue point is a negative cue, the Gaussian distribution is multiplied by a negative coefficient, and then the distribution information of all cue points is merged into a cue map.

[0063] It should be understood that although formula (1) uses Gaussian distribution as the distribution function, other distribution functions may be used as needed. In addition, the specific forms and parameters of formulas (1) and (2) may be adjusted as needed without departing from the scope of the present application.

[0064] At step 108, a first two-dimensional mask may be generated based on the two-dimensional affinity map and the first hint map using a two-dimensional mask prediction module. The two-dimensional mask prediction module may use a two-dimensional version of a suitable model such as a model based on a U-Net architecture, a Fully Convolutional Networks (FCN), a SegNet model, or the like.

[0065] like Figure 2 As shown, a two-dimensional mask prediction module can be used to generate a binary two-dimensional mask based on a two-dimensional affinity map and a prompt map converted from a user prompt. The white part in the two-dimensional mask represents the foreground, which refers to all the pixels of the neurons that need to be segmented, and can also be represented by a value of 1; the black part represents the background, which refers to the part that does not need to be segmented, and can also be represented by a value of 0. Although the foreground and background are indicated here with specific colors or values, any other suitable method can be used to indicate the foreground and background as needed.

[0066] At step 110, a connected domain analysis may be performed on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels. In one embodiment, when the two-dimensional image to be segmented is a neuron electron microscopy image, different regions in the first segmented two-dimensional image correspond to different neurons.

[0067] like Figure 2 As shown, by performing connected component analysis (CCA) on the two-dimensional mask, a segmented image can be obtained, in which different neurons (i.e., different regions) are marked with different colors, and the background is marked with other colors (e.g., white). Alternatively, the segmentation result can be indicated by assigning different numbers to pixels belonging to different neurons.

[0068] For example, when performing CCA, Figure 2 Select a seed point in the white foreground area of ​​the two-dimensional mask shown, and then use the seed point as the center to search for points that are also foreground points in its four or eight neighborhoods. If they exist, it means that the two points are connected. Then, diffuse according to the same rules until all connected areas are found. Since different neurons are separated by black background areas, a connected area indicates that it belongs to a neuron, so all neurons in the two-dimensional mask can be found in this way. Finally, different labels can be assigned to different neurons to obtain a segmented image.

[0069] The above interactive image segmentation method based on user prompts can improve the segmentation accuracy. In addition, when applied to new species with a lack of labeled data, high-precision neuron segmentation can still be achieved by providing a small number of prompt points, thus demonstrating excellent cross-species generalization capabilities.

[0070] In traditional methods, different means are usually used to handle the two independent processes of neuron segmentation and proofreading, which not only increases the complexity of the operation, but also limits the overall segmentation efficiency. Therefore, in addition to being used for image segmentation, the image segmentation method of the present application can also be used in the proofreading process to achieve real-time correction through user interaction.

[0071] In one embodiment, method 100 may further include steps 112 to 118, such as Figure 3 After step 110 , method 100 may proceed to step 112 .

[0072] At step 112 , a second user prompt corresponding to the two-dimensional image to be segmented and based on the first segmented two-dimensional image may be obtained.

[0073] Figure 4 FIG. 1 is a flow chart of two-dimensional image segmentation and correction according to an embodiment of the present invention. Figure 4 As shown, the part above the dotted line is the initial segmentation process of the image, and the part below the dotted line is the proofreading process. After the initial segmentation, it can be seen by zooming in that due to the boundary error in the mask, a merging error occurs in the segmentation result where different neurons are merged into one neuron. The user can give positive and / or negative prompts based on the segmentation results. For example, a negative prompt represented by a red five-pointed star can be given to a neuron that is mistakenly merged, indicating that the neuron does not need to be segmented. In addition, a positive prompt represented by a green five-pointed star can be given to the neuron in another process, indicating that the neuron needs to be segmented.

[0074] At step 114, the second user prompt may be converted into a second prompt map. The conversion process is similar to the conversion process discussed above with respect to step 106, and for the sake of clarity, it is not described in detail here.

[0075] At step 116 , a second two-dimensional mask may be generated based on the two-dimensional affinity map and the second hint map using a two-dimensional mask prediction module. The two-dimensional affinity map is the two-dimensional affinity map generated in step 104 .

[0076] At step 118 , a connected component analysis may be performed on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0077] like Figure 4As shown, when only the negative prompt indicated by the red five-pointed star is provided, it means that the neuron indicated by the negative prompt does not need to be segmented, and only the remaining parts are segmented. Therefore, a segmented image in which only the neuron is not segmented can be obtained. When only the positive prompt indicated by the green five-pointed star is provided, it means that only the neuron indicated by the positive prompt needs to be segmented. Therefore, a segmented image in which only the neuron is segmented can be obtained. Then, the image part in which the neuron is correctly segmented is superimposed on the segmented image in which only the neuron is not segmented, and the correct segmented image can be obtained, thereby completing the proofreading process.

[0078] Therefore, by utilizing user prompt information to segment and proofread neurons, the segmentation process and the proofreading process can be unified, thereby significantly reducing the workload of manual labeling and proofreading, and can achieve timely correction of segmentation errors, thereby improving segmentation accuracy and efficiency.

[0079] Figure 5 is a flowchart of a method 500 for training a two-dimensional mask prediction module according to an embodiment of the present invention.

[0080] At step 502, a data set may be acquired. The data set may include a two-dimensional image to be segmented and a user prompt corresponding to the two-dimensional image to be segmented as input information, and include a true value of a two-dimensional mask of the segmented two-dimensional image as output information. The user prompt may include any one or more of the following types: giving a positive prompt to all neurons in the two-dimensional image to be segmented, i.e., a default prompt; giving a positive prompt to a randomly selected portion of neurons in the two-dimensional image to be segmented; giving a negative prompt to a randomly selected portion of neurons in the two-dimensional image to be segmented. The true value of the two-dimensional mask of the segmented two-dimensional image may be obtained by converting the true value of the segmented two-dimensional image into a two-dimensional mask.

[0081] At step 504 , a two-dimensional affinity prediction module may be used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the data set.

[0082] At step 506, the user prompt in the data set may be converted into a prompt map. In one embodiment, when the user prompt is a default prompt, the prompt map may be a default matrix corresponding to the two-dimensional image to be segmented. In one embodiment, when the user prompt includes a positive prompt or a negative prompt, the prompt map may include distribution information obtained by applying a distribution function to the user prompt.

[0083] At step 508 , a two-dimensional mask prediction module may be used to generate a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the hint map.

[0084] At step 510 , a loss function between the predicted result of the two-dimensional mask and the true value of the two-dimensional mask of the segmented two-dimensional image in the dataset may be calculated.

[0085] In one embodiment, the loss function may include a loss regularization term. The loss regularization term may be the product of the true value of the affinity and the predicted value of the mask. For example, the loss regularization term may be represented by the following formula:

[0086] R(g a ,y m ) = g a ·y m (3)

[0087] The loss function may be any one or more suitable functions such as a binary cross entropy loss function, a dice loss function, an MSE (mean square error) loss function, etc. For example, when a binary cross entropy loss function CE (·, ·) and a dice loss function are used, the loss function may be expressed by the following formula:

[0088] L=CE(g m ,y m )+DC(g m ,y m )+R(g a ,y m ) (4)

[0089] Among them, g m ∈{0,1} represents the true value mask of the voxel, y m represents the predicted value of its mask, g a ∈{0,1} represents the true affinity of the voxel. For voxels located on the neuron boundary (g a =1), we expect their y m As close to 0 as possible, that is, for example, when the truth affinity g a = 1, it means that the voxel is on the neuron boundary, so the predicted value y of the expected mask m =0, indicating that it corresponds to the background.

[0090] By utilizing this loss regularization term, the continuity of neuron boundaries within the mask predicted by the 2D mask prediction model can be enhanced.

[0091] At step 512 , parameters of the two-dimensional mask prediction module may be updated based on the loss function to obtain a trained two-dimensional mask prediction module.

[0092] In one embodiment, the two-dimensional mask prediction module can update parameters through an online learning method. For example, the online learning method can be an averaged gradient episodic memory (A-GEM), elastic weight merging, or synaptic intelligence method. By using an online learning method, the two-dimensional mask prediction module can continuously acquire new tasks while retaining previously acquired knowledge, significantly reduce the degree of forgetting previous tasks when gradually learning new species data, and achieve efficient segmentation and tracking of cross-species data.

[0093] By using online learning methods to update the parameters of the 2D mask prediction module, the 2D mask prediction module can gradually optimize itself, improve the segmentation accuracy across species, and reduce the dependence on labeled data, enabling it to have excellent efficiency and accuracy when processing new species data.

[0094] In the image segmentation of electron microscope images, in addition to the segmentation of two-dimensional slices, it is sometimes necessary to segment three-dimensional images across multiple two-dimensional slices, or three-dimensional image tracking, so as to obtain three-dimensional information of each neuron. However, the existing neuron segmentation methods are difficult to effectively cope with the three-dimensional tracking task across slices, resulting in the need to rely on a large amount of manual operations in the processing of three-dimensional data. Therefore, the present application proposes a method for three-dimensional image segmentation.

[0095] Figure 6 is a flowchart of a method 600 for three-dimensional image segmentation according to an embodiment of the present invention. Fig. 7A It is a schematic diagram of a process of three-dimensional image segmentation according to an embodiment of the present invention.

[0096] At step 602, a three-dimensional image to be segmented may be acquired. In one embodiment, the three-dimensional image to be segmented may be a neuron electron microscope image.

[0097] At step 604, a 3D affinity prediction module may be used to generate a 3D affinity map based on the 3D image to be segmented. The 3D affinity prediction module may use a 3D version of a suitable affinity prediction model such as the Local Shape Descriptors (LSD) Neural Segmentation Model (ACRLSD), the MALA (Metropolis-Adjusted Langevin Algorithm) model, etc. Fig. 7A As shown, the 3D affinity prediction module can take the original 3D image as input and predict the affinity between neurons to generate a 3D affinity map.

[0098] At step 606, a two-dimensional mask prediction module may be used to generate a two-dimensional mask based on a slice in the three-dimensional image to be segmented. The slice may be any slice in the original three-dimensional image.

[0099] For example, Fig. 7A As shown, a two-dimensional mask prediction module can be used to generate a binary two-dimensional mask based on the first slice of the original three-dimensional image. The white part in the two-dimensional mask represents the foreground, which refers to all the pixels of the neurons that need to be segmented, and can also be represented by a value of 1; the black part represents the background, which refers to the part that does not need to be segmented, and can also be represented by a value of 0. Although the foreground and background are indicated by specific colors or values ​​here, any other suitable method can be used to indicate the foreground and background as needed. The two-dimensional mask prediction module can use a two-dimensional version of a suitable model such as a model based on a U-Net architecture, a fully convolutional network (FCN), a SegNet model, etc.

[0100] At step 608, the slices in the to-be-segmented three-dimensional image may be replaced with a two-dimensional mask to generate a replaced three-dimensional image. Fig. 7A For example, the first slice of the original three-dimensional image may be replaced with the two-dimensional mask generated by the two-dimensional mask prediction module, thereby generating a replaced three-dimensional image.

[0101] At step 610, a 3D mask prediction module may be used to generate a 3D mask based on the 3D affinity map and the replaced 3D image. The 3D mask prediction module may use a 3D version of a suitable model such as a U-Net architecture-based model, a Fully Convolutional Networks (FCN), a SegNet model, etc.

[0102] like Fig. 7A As shown, a three-dimensional mask prediction module can be used to generate a binary three-dimensional mask based on a three-dimensional affinity map and a replaced three-dimensional image. The white part in the three-dimensional mask represents the foreground, which refers to all the pixels of the neurons that need to be segmented, and can also be represented by a value of 1; the black part represents the background, which refers to the part that does not need to be segmented, and can also be represented by a value of 0. Although the foreground and background are indicated here with specific colors or values, any other suitable method can be used to indicate the foreground and background as needed.

[0103] At step 612, a connected domain analysis may be performed on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels. In one embodiment, when the three-dimensional image to be segmented is a neuron electron microscopy image, different regions in the segmented three-dimensional image correspond to different neurons.

[0104] like Fig. 7A As shown, by performing connected component analysis (CCA) on the three-dimensional mask, a segmented image can be obtained, in which different neurons (i.e., different regions) are marked with different colors, and the background is marked with other colors (e.g., white). Alternatively, the segmentation result can be indicated by assigning different numbers to pixels belonging to different neurons.

[0105] For example, when performing CCA, Fig. 7A Select a seed point in the white foreground area of ​​the three-dimensional mask shown, and then use the seed point as the center to search for points that are also foreground points in its four or eight neighborhoods. If they exist, it means that the two points are connected. Then, diffuse according to the same rules until all connected areas are found. Since different neurons are separated by black background areas, a connected area indicates that it belongs to a neuron, so all neurons in the three-dimensional mask can be found in this way. Finally, different labels can be assigned to different neurons to obtain a segmented image.

[0106] Compared with the traditional three-dimensional image tracking method that directly generates a three-dimensional mask based on a three-dimensional image, the method of the present application replaces the slices in the three-dimensional image with a two-dimensional mask of the slice, and then generates a three-dimensional mask based on the replaced three-dimensional image. This can improve the segmentation accuracy, improve the efficiency and accuracy of three-dimensional data processing, and ensure the feasibility of large-scale three-dimensional reconstruction.

[0107] In one embodiment, when the two-dimensional mask prediction module is used to generate the two-dimensional mask based on the slices in the three-dimensional image to be segmented at step 606, a user prompt corresponding to the slice may also be obtained. The user prompt may include a positive prompt or a negative prompt for the pixel in the two-dimensional slice, where a positive prompt is used to prompt that the pixel belongs to the foreground, and a negative prompt is used to prompt that the pixel belongs to the background. Figure 7B It is a schematic diagram of a process of three-dimensional image segmentation according to another embodiment of the present invention.

[0108] exist Figure 7B In the example, the user can provide user prompts for the original slice. The user prompts can include positive prompts represented by green five-pointed stars, indicating that the pixel belongs to the foreground, that is, the neuron to be segmented. The user prompts can also include negative prompts represented by red five-pointed stars, indicating that the pixel belongs to the background, that is, it cannot be segmented or does not need to be segmented.

[0109] Then, a 2D mask prediction module can be used to generate a 2D mask based on the slices and user hints. The 2D mask prediction module can be a reference Figure 2 The described 2D mask prediction module may also employ any suitable model that can generate a 2D mask based on user prompts.

[0110] By providing user prompts for the mask prediction process of the two-dimensional slices, the prediction accuracy of the two-dimensional mask can be improved, thereby further improving the prediction accuracy of the three-dimensional mask, and enabling selective segmentation of the image according to the user prompts.

[0111] Figure 8 8 is a flowchart of a method 800 for training a 3D mask prediction module according to an embodiment of the present invention.

[0112] At step 802, a data set may be obtained. The data set may include a 3D image to be segmented as input information, and may include a true value of a 3D mask of the segmented 3D image as output information. The true value of the 3D mask of the segmented 3D image may be obtained by converting the true value of the segmented 3D image into the 3D mask.

[0113] At step 804 , a three-dimensional affinity prediction module may be used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set.

[0114] At step 806 , a two-dimensional mask prediction module may be used to generate a two-dimensional mask based on the slices in the three-dimensional image to be segmented.

[0115] At step 808 , the slices in the three-dimensional image to be segmented may be replaced with the two-dimensional mask to generate a replaced three-dimensional image.

[0116] At step 810 , a three-dimensional mask prediction module may be used to generate a prediction result of a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image.

[0117] At step 812 , a loss function between the predicted result of the three-dimensional mask and the true value of the three-dimensional mask of the segmented three-dimensional image in the dataset may be calculated.

[0118] At step 814 , parameters of the 3D mask prediction module may be updated based on the loss function to obtain a trained 3D mask prediction module.

[0119] In one embodiment, the 3D mask prediction module can update parameters through an online learning method. For example, the online learning method can be a method such as average gradient segment memory (A-GEM), elastic weight merging, or synaptic intelligence. By using an online learning method to semi-automatically annotate new species data, the 3D mask prediction module can continuously acquire new tasks while retaining previously acquired knowledge, significantly reducing the degree of forgetting previous tasks when gradually learning new species data, and achieving efficient segmentation and tracking of cross-species data.

[0120] By using online learning methods to update the parameters of the 3D mask prediction module, the 3D mask prediction module can gradually optimize itself, improve the segmentation accuracy across species, and reduce the dependence on labeled data, enabling it to have excellent efficiency and accuracy when processing new species data.

[0121] Fig. 9 is a flowchart of a method 900 for three-dimensional image segmentation according to another embodiment of the present invention.

[0122] Steps 902 to 910 and reference Figure 1 Steps 102-110 in the described method 100 are the same. To avoid redundancy, the specific details of steps 902-910 are not described here. After completing step 910, it is known that the predicted two-dimensional mask can produce a segmented two-dimensional image that meets the requirements. The two-dimensional mask can include all neurons or only the required part of neurons.

[0123] At step 912, a three-dimensional image to be segmented may be acquired. The two-dimensional image to be segmented, which has been previously segmented, is a slice in the three-dimensional image to be segmented.

[0124] At step 914, a 3D affinity prediction module may be used to generate a 3D affinity map based on the 3D image to be segmented. The 3D affinity prediction module may use a 3D version of a suitable affinity prediction model such as a Local Shape Descriptors (LSD) based neuron segmentation model (ACRLSD), a MALA (Metropolis-Adjusted Langevin Algorithm) model, etc.

[0125] At step 916 , the two-dimensional image to be segmented in the three-dimensional image to be segmented may be replaced with the first two-dimensional mask to generate a replaced three-dimensional image.

[0126] At step 918, a 3D mask prediction module may be used to generate a 3D mask based on the 3D affinity map and the replaced 3D image. The 3D mask prediction module may use a 3D version of a suitable model such as a U-Net architecture-based model, a Fully Convolutional Networks (FCN), a SegNet model, etc.

[0127] At step 920 , a connected component analysis may be performed on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0128] In one embodiment, a three-dimensional mask prediction module can be trained by the following steps: obtaining a data set, the data set including a three-dimensional image to be segmented as input information, and including a true value of a three-dimensional mask of the segmented three-dimensional image as output information; using a three-dimensional affinity prediction module to generate a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set; using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; using the three-dimensional mask prediction module to generate a predicted result of the three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image; calculating a loss function between the predicted result of the three-dimensional mask and the true value of the three-dimensional mask of the segmented three-dimensional image in the data set; and updating the parameters of the three-dimensional mask prediction module based on the loss function to obtain a trained three-dimensional mask prediction module.

[0129] In one embodiment, the 3D mask prediction module may update parameters by an online learning method.

[0130] Therefore, by performing interactive segmentation based on user prompts only on representative 2D images and then using 2D masks instead of 2D images for 3D image tracking, the interaction and annotation workload can be reduced exponentially.

[0131] Fig.10 1 is a block diagram of a system 1000 for two-dimensional image segmentation according to an embodiment of the present invention. The system 1000 may include an acquisition module 1002, a two-dimensional affinity prediction module 1004, a prompt conversion module 1006, a two-dimensional mask prediction module 1008, and a connected domain analysis module 1010. The acquisition module 1002 may be used to acquire a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented. The two-dimensional affinity prediction module 1004 may be used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented. The prompt conversion module 1006 may be used to convert the first user prompt into a first prompt map. The two-dimensional mask prediction module 1008 may be used to generate a first two-dimensional mask based on the two-dimensional affinity map and the first prompt map. The connected domain analysis module 1010 may be used to perform a connected domain analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

[0132] In one embodiment, the first user prompt may include a positive prompt or a negative prompt for a pixel in the two-dimensional image to be segmented, wherein a positive prompt is used to prompt that the pixel belongs to the foreground, and a negative prompt is used to prompt that the pixel belongs to the background.

[0133] In one embodiment, the first user prompt may be in any one or more of the following forms: a point prompt, a shape prompt, or an area prompt.

[0134] In one embodiment, when the first user prompt is empty, the first prompt map may be a default matrix corresponding to the two-dimensional image to be segmented; or when the first user prompt is not empty, the first prompt map may include distribution information obtained by applying a distribution function to the first user prompt.

[0135] In one embodiment, the acquisition module 1002 may also be used to acquire a second user prompt corresponding to the to-be-segmented two-dimensional image based on the first segmented two-dimensional image. The prompt conversion module 1006 may also be used to convert the second user prompt into a second prompt map. The two-dimensional mask prediction module 1008 may also be used to generate a second two-dimensional mask based on the two-dimensional affinity map and the second prompt map. The connected domain analysis module 1010 may also be used to perform a connected domain analysis on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0136] In one embodiment, the two-dimensional mask prediction module 1008 can be trained by the following steps: obtaining a data set, the data set including the two-dimensional image to be segmented and the user prompt corresponding to the two-dimensional image to be segmented as input information, and including the true value of the two-dimensional mask of the segmented two-dimensional image as output information; using the two-dimensional affinity prediction module to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the data set; converting the user prompt in the data set into a prompt map; using the two-dimensional mask prediction module to generate a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the prompt map; calculating the loss function between the prediction result of the two-dimensional mask and the true value of the two-dimensional mask of the segmented two-dimensional image in the data set; and updating the parameters of the two-dimensional mask prediction module based on the loss function to obtain a trained two-dimensional mask prediction module.

[0137] In one embodiment, the loss function may include a loss regularization term, which is the product of the true value of the affinity and the predicted value of the mask.

[0138] In one embodiment, the two-dimensional mask prediction module 1008 may update parameters by an online learning method.

[0139] Fig.111 is a block diagram of a system 1100 for three-dimensional image segmentation according to an embodiment of the present invention. The system 1100 may include an acquisition module 1102, a three-dimensional affinity prediction module 1104, a two-dimensional mask prediction module 1106, a slice replacement module 1108, a three-dimensional mask prediction module 1110, and a connected domain analysis module 1112. The acquisition module 1102 may be used to acquire a three-dimensional image to be segmented. The three-dimensional affinity prediction module 1104 may be used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented. The two-dimensional mask prediction module 1106 may be used to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented. The slice replacement module 1108 may be used to replace the slices in the three-dimensional image to be segmented with a two-dimensional mask to generate a replaced three-dimensional image. The three-dimensional mask prediction module 1110 may be used to generate a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image. The connected domain analysis module 1112 may be used to perform a connected domain analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0140] In one embodiment, the acquisition module 1102 may also be used to acquire a user prompt corresponding to the slice. The two-dimensional mask prediction module 1106 may also be used to generate a two-dimensional mask based on the slice and the user prompt.

[0141] In one embodiment, the three-dimensional mask prediction module 1110 can be trained by the following steps: obtaining a data set, the data set including a three-dimensional image to be segmented as input information, and including a true value of a three-dimensional mask of the segmented three-dimensional image as output information; using a three-dimensional affinity prediction module to generate a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set; using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; using the three-dimensional mask prediction module to generate a predicted result of the three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image; calculating a loss function between the predicted result of the three-dimensional mask and the true value of the three-dimensional mask of the segmented three-dimensional image in the data set; and updating the parameters of the three-dimensional mask prediction module based on the loss function to obtain a trained three-dimensional mask prediction module.

[0142] In one embodiment, the 3D mask prediction module 1110 may update parameters by an online learning method.

[0143] Fig.121 is a block diagram of a system 1200 for 3D image segmentation according to another embodiment of the present invention. The system 1200 may include an acquisition module 1202, a 2D affinity prediction module 1204, a hint conversion module 1206, a 2D mask prediction module 1208, a connected domain analysis module 1210, a 3D affinity prediction module 1212, a slice replacement module 1214, and a 3D mask prediction module 1216. The acquisition modules 1202 to the connected domain analysis module 1210 are connected to the reference Fig.10 The acquisition module 1002 to the connected domain analysis module 1010 in the described system 1000 are the same. To avoid redundancy, the specific details of the acquisition module 1202 to the connected domain analysis module 1210 are not described here. The acquisition module 1202 can also be used to acquire a three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice in the three-dimensional image to be segmented. The three-dimensional affinity prediction module 1212 can be used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented. The slice replacement module 1214 can be used to replace the two-dimensional image to be segmented in the three-dimensional image to be segmented with a first two-dimensional mask to generate a replaced three-dimensional image. The three-dimensional mask prediction module 1216 can be used to generate a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image. The connected domain analysis module 1210 can also be used to perform a connected domain analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0144] In one embodiment, the three-dimensional mask prediction module 1216 can be trained by the following steps: obtaining a data set, the data set including a three-dimensional image to be segmented as input information, and including a true value of a three-dimensional mask of the segmented three-dimensional image as output information; using a three-dimensional affinity prediction module to generate a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set; using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; using a three-dimensional mask prediction module to generate a predicted result of the three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image; calculating a loss function between the predicted result of the three-dimensional mask and the true value of the three-dimensional mask of the segmented three-dimensional image in the data set; and updating the parameters of the three-dimensional mask prediction module based on the loss function to obtain a trained three-dimensional mask prediction module.

[0145] In one embodiment, the 3D mask prediction module 1216 may update parameters by an online learning method.

[0146] Figure 13A-13BThis is a performance comparison of ACRLSD-2D, SegNeuro-2D, and the improved version of SegNeuro-2D*. ACRLSD-2D is a two-dimensional version of the ACRLSD model. The SegNeuro-2D model is an example of the two-dimensional image segmentation method of the present application, which is trained using cross entropy and dice loss functions. The improved SegNeuro-2D* model is another example of the two-dimensional image segmentation method of the present application. Compared with the SegNeuro-2D model, a loss term related to neuron boundaries is added on the basis of the cross entropy and dice loss functions. The above three models are trained here using fruit fly data and tested on zebra finch data for cross-species evaluation. Fig.13A This is a performance comparison of ACRLSD-2D, SegNeuro-2D, and the improved version of SegNeuro-2D* on a slice selected from the zebra finch data. In the zebra finch test case, the variation of information (VOI) of ACRLSD-2D without prompts is 2.1093, while SegNeuro-2D and the improved version of SegNeuro-2D* are 2.2587 and 1.9752 respectively. The smaller the VOI value, the closer the result is to the true value. Although ACRLSD-2D, SegNeuro-2D, and SegNeuro-2D* have relatively poor boundary predictions for some zebra finch neurons without prompts, as shown on the right, SegNeuro-2D* can significantly improve the mask prediction accuracy of neurons with poor boundaries through a small number of manual proofreading prompt points, and finally achieve a VOI of 1.7734. Fig. 13B The results are a quantitative comparison of the performance of the three models on all 33 zebra finch data volumes. The results show that SegNeuro-2D* performs best in all 33 data volumes, especially in the cued default mode, showing excellent cross-species generalization performance, and performs even better when there are artificial cues.

[0147] It can be seen that the interactive image segmentation method of the present application can improve the segmentation accuracy of the model across species.

[0148] The experiment further simulated the use of online learning methods to train the SegNeuro-2D model. The model was initially trained on fruit fly data, and online learning was performed on the first six zebra finch data volumes in sequence, while the subsequent 25 zebra finch data volumes were used for testing and evaluation. The experimental results show that as the amount of zebra finch data increases, the VOI of the SegNeuro-2D model shows a significant downward trend. When the A-GEM strategy is used for online learning, the performance is very close to that of training using all zebra finch data volumes, and the training time of online learning is only 58.2% of the original training method. Therefore, the use of online learning methods enables the SegNeuro-2D model to achieve segmentation performance close to that of manual annotation on new species data.

[0149] Figure 13C-Figure 13D This is a performance comparison between the existing SAM (Segment Anything Model) model and the two-dimensional image segmentation method of this application. Fig. 13C As shown in FIG, the SAM model can segment a single neuron based on user prompts, but when it is provided with prompts about multiple neurons, the segmentation results are significantly wrong. However, the two-dimensional image segmentation method of the present application can achieve accurate segmentation of the corresponding neurons when provided with prompts about any number of neurons. Fig.13D As shown, the SAM model can only segment one neuron at a time when a cue is provided to the neuron each time, while the two-dimensional image segmentation method of the present application can achieve complete segmentation in the case of default cues, and can achieve selective segmentation in the case of non-default cues including positive cues and negative cues.

[0150] In addition, the two-dimensional image segmentation method of the present application requires significantly fewer interactions than the SAM model. In the experiment, the SAM model required an average of 95 interactions to segment a slice of 2 cubic micrometer fruit fly data, while the two-dimensional image segmentation method of the present application only required two interactions, greatly reducing manual intervention.

[0151] Fig.14 The figure is the result of the performance evaluation of 3D image segmentation. SegNeuro-3D is an example of the 3D image segmentation method of this application. In the experiment, the neuron tracking model SegNeuro-3D was trained using the HEMI-BRAIN and FIB-25 datasets, and the three neurons with the largest volume were selected in the HEMI-BRAIN ROI-1 dataset for segmentation accuracy testing. Fig.14As shown in the figure, the variation of information (VOI) of SegNeuro-3D for these three neurons is close to 0. This shows that the SegNeuro-3D model can achieve the same accuracy as the true value in three-dimensional neuron tracking, showing excellent automatic tracking ability.

[0152] The image segmentation method implemented by the present application using artificial intelligence technology can greatly improve the image segmentation speed and segmentation accuracy, and reduce manual participation. The two-dimensional image segmentation method of the present application can improve data processing efficiency and segmentation accuracy by utilizing user prompts. By integrating segmentation and proofreading in the same framework, real-time error correction can be achieved and the proofreading workload can be reduced. The three-dimensional image tracking method of the present application is based on the two-dimensional mask of the two-dimensional slice, which realizes the automatic tracking of three-dimensional neurons across slices, improves the efficiency and accuracy of three-dimensional data processing, reduces the need for manual intervention, and ensures the feasibility of large-scale three-dimensional reconstruction. In addition, by combining the image segmentation method with the online learning method, it can adapt to the data of new species, and improves the universal ability of the image segmentation method between different species.

[0153] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0154] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0155] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0156] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0157] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0159] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0160] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0161] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for image segmentation, comprising: Acquire a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented; Using a two-dimensional affinity prediction module, generating a two-dimensional affinity map based on the two-dimensional image to be segmented; converting the first user prompt into a first prompt map; generating a first two-dimensional mask based on the two-dimensional affinity map and the first hint map using a two-dimensional mask prediction module; as well as A connected component analysis is performed on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

2. The method of claim 1, wherein: The first user prompt includes a positive prompt or a negative prompt for a pixel in the two-dimensional image to be segmented, wherein the positive prompt is used to prompt that the pixel belongs to the foreground, and the negative prompt is used to prompt that the pixel belongs to the background.

3. The method of claim 1, wherein: The first user prompt is in any one or more of the following forms: a point prompt, a shape prompt or an area prompt.

4. The method of claim 1, wherein: When the first user prompt is empty, the first prompt image is a default matrix corresponding to the two-dimensional image to be segmented; or When the first user prompt is not empty, the first prompt map includes distribution information obtained by applying a distribution function to the first user prompt.

5. The method of claim 1, wherein: The method further comprises: Acquire a second user prompt based on the first segmented two-dimensional image and corresponding to the two-dimensional image to be segmented; converting the second user prompt into a second prompt map; generating a second two-dimensional mask based on the two-dimensional affinity map and the second hint map using the two-dimensional mask prediction module; and A connected component analysis is performed on the second two-dimensional mask to generate a second segmented two-dimensional image.

6. The method of claim 1, wherein: The two-dimensional mask prediction module is trained by the following steps: Acquire a data set, the data set including a two-dimensional image to be segmented and a user prompt corresponding to the two-dimensional image to be segmented as input information, and including a true value of a two-dimensional mask of the segmented two-dimensional image as output information; Using the two-dimensional affinity prediction module, generating a two-dimensional affinity map based on the two-dimensional image to be segmented in the data set; converting user prompts in the data set into a prompt map; Using the two-dimensional mask prediction module, generating a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the hint map; Calculating a loss function between a prediction result of the two-dimensional mask and a true value of the two-dimensional mask of the segmented two-dimensional image in the data set; as well as The parameters of the two-dimensional mask prediction module are updated based on the loss function to obtain a trained two-dimensional mask prediction module.

7. The method of claim 6, wherein: The loss function includes a loss regularization term, which is the product of the true value of the affinity and the predicted value of the mask.

8. The method of claim 1, wherein: The two-dimensional mask prediction module updates parameters through an online learning method.

9. The method of claim 1, wherein: The method further comprises: Acquire a three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice of the three-dimensional image to be segmented; Using a three-dimensional affinity prediction module, generating a three-dimensional affinity map based on the three-dimensional image to be segmented; replacing the two-dimensional image to be segmented in the three-dimensional image to be segmented with the first two-dimensional mask to generate a replaced three-dimensional image; generating a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image using a three-dimensional mask prediction module; and A connected component analysis is performed on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

10. The method of claim 1, wherein: The two-dimensional image to be segmented is a neuron electron microscope image, and different regions in the first segmented two-dimensional image correspond to different neurons.

11. A method for image segmentation, comprising: Acquire a three-dimensional image to be segmented; Using a three-dimensional affinity prediction module, generating a three-dimensional affinity map based on the three-dimensional image to be segmented; Using a two-dimensional mask prediction module, generating a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slice in the to-be-segmented three-dimensional image with the two-dimensional mask to generate a replaced three-dimensional image; generating a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image using a three-dimensional mask prediction module; as well as A connected component analysis is performed on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

12. The method of claim 11, wherein: Using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented includes: Obtaining a user prompt corresponding to the slice; and The two-dimensional mask is generated based on the slice and the user hint using the two-dimensional mask prediction module.

13. The method of claim 11, wherein: The 3D mask prediction module is trained by the following steps: Acquire a data set, wherein the data set includes a three-dimensional image to be segmented as input information, and includes a true value of a three-dimensional mask of the segmented three-dimensional image as output information; Using the three-dimensional affinity prediction module, generating a three-dimensional affinity map based on the three-dimensional image to be segmented in the data set; Using the two-dimensional mask prediction module, generating a two-dimensional mask based on slices in the three-dimensional image to be segmented; replacing the slice in the to-be-segmented three-dimensional image with the two-dimensional mask to generate a replaced three-dimensional image; generating a prediction result of a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image using the three-dimensional mask prediction module; Calculating a loss function between a prediction result of the three-dimensional mask and a true value of the three-dimensional mask of the segmented three-dimensional image in the data set; as well as The parameters of the 3D mask prediction module are updated based on the loss function to obtain a trained 3D mask prediction module.

14. The method of claim 11, wherein: The three-dimensional mask prediction module updates parameters through an online learning method.

15. The method of claim 11, wherein: The three-dimensional image to be segmented is a neuron electron microscope image, and different regions in the segmented three-dimensional image correspond to different neurons.

16. A system for image segmentation, comprising means for performing the method according to any one of claims 1-15.

17. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

18. A computer device comprising a memory and a processor, wherein a computer program capable of being executed on the processor is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 15 is implemented.

19. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 15 is implemented.

Images