Methods and systems for image segmentation

Abstract

This application discloses a method and system for image segmentation. One method for image segmentation includes: acquiring a two-dimensional image to be segmented and a first user cue corresponding to the two-dimensional image to be segmented; generating a two-dimensional affinity map based on the two-dimensional image to be segmented using a two-dimensional affinity prediction module; converting the first user cue into a first cue map; generating a first two-dimensional mask based on the two-dimensional affinity map and the first cue map using a two-dimensional mask prediction module; and performing connected component analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to methods and systems for image segmentation. Background Technology

[0002] In neuroscience research, neuronal reconstruction is a crucial means of understanding the organization and function of the nervous system. Electron microscopy, with its ultra-high resolution, can clearly capture the fine structure of neurons and their synapses, making it a key tool for reconstructing dense neuronal connections. However, the massive data volume resulting from ultra-high resolution significantly increases the burden of manual annotation. Currently, neuron segmentation mainly utilizes automated methods based on deep learning, which train models using labeled data to segment neurons in specific species. While existing automated neuron segmentation methods can achieve good segmentation results on image data of some species, for non-model organisms and new species, the lack of corresponding labeled data for existing models often results in unsatisfactory segmentation accuracy, and automated segmentation results often require extensive manual verification. Therefore, a fast and accurate image segmentation method and system are urgently needed. Summary of the Invention

[0003] It should be understood that the above general description and the following detailed description of the invention are exemplary and illustrative, and are intended to provide further explanation of the invention as described in the claims.

[0004] According to a first aspect of the present invention, a method for image segmentation is provided, comprising: acquiring a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented; generating a two-dimensional affinity map based on the two-dimensional image to be segmented using a two-dimensional affinity prediction module; converting the first user prompt into a first prompt map; generating a first two-dimensional mask based on the two-dimensional affinity map and the first prompt map using a two-dimensional mask prediction module; and performing connected component analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

[0005] In the above method, the first user prompt includes a positive or negative prompt for the pixels in the two-dimensional image to be segmented, wherein the positive prompt is used to indicate that the pixel belongs to the foreground and the negative prompt is used to indicate that the pixel belongs to the background.

[0006] In the above method, the first user prompt takes any one or more of the following forms: dot prompt, shape prompt, or area prompt.

[0007] In the above method, when the first user prompt is empty, the first prompt image is a default matrix corresponding to the two-dimensional image to be segmented; or when the first user prompt is not empty, the first prompt image includes distribution information obtained by applying a distribution function to the first user prompt.

[0008] In the above method, the method further includes: obtaining a second user prompt based on the first segmented two-dimensional image and corresponding to the two-dimensional image to be segmented; converting the second user prompt into a second prompt image; using the two-dimensional mask prediction module, based on the two-dimensional affinity map and the second prompt image to generate a second two-dimensional mask; and performing connected component analysis on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0009] In the above method, the two-dimensional mask prediction module is trained through the following steps: acquiring a dataset, the dataset including a two-dimensional image to be segmented and user prompts corresponding to the two-dimensional image to be segmented as input information, and including the true values ​​of the two-dimensional masks of the segmented two-dimensional image as output information; using the two-dimensional affinity prediction module to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the dataset; converting the user prompts in the dataset into prompt maps; using the two-dimensional mask prediction module to generate a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the prompt maps; calculating a loss function between the prediction result of the two-dimensional mask and the true values ​​of the two-dimensional masks of the segmented two-dimensional image in the dataset; and updating the parameters of the two-dimensional mask prediction module based on the loss function to obtain a trained two-dimensional mask prediction module.

[0010] In the above method, the loss function includes a loss regularization term, which is the product of the true value of affinity and the predicted value of the mask.

[0011] In the above method, the two-dimensional mask prediction module updates its parameters through an online learning method.

[0012] In the above method, the method further includes: acquiring a three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice of the three-dimensional image to be segmented; generating a three-dimensional affinity map based on the three-dimensional image to be segmented using a three-dimensional affinity prediction module; replacing the two-dimensional image to be segmented in the three-dimensional image to be segmented with the first two-dimensional mask to generate a replaced three-dimensional image; generating a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image using a three-dimensional mask prediction module; and performing connected component analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0013] In the above method, the two-dimensional image to be segmented is an electron micrograph of a neuron, and different regions in the first segmented two-dimensional image correspond to different neurons.

[0014] According to a second aspect of the present invention, a method for image segmentation is provided, comprising: acquiring a three-dimensional image to be segmented; generating a three-dimensional affinity map based on the three-dimensional image to be segmented using a three-dimensional affinity prediction module; generating a two-dimensional mask based on slices in the three-dimensional image to be segmented using a two-dimensional mask prediction module; replacing the slices in the three-dimensional image to be segmented with the two-dimensional mask to generate a replaced three-dimensional image; generating a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image using a three-dimensional mask prediction module; and performing connected component analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0015] In the above method, generating a two-dimensional mask using a two-dimensional mask prediction module based on slices in the three-dimensional image to be segmented includes: obtaining a user prompt corresponding to the slice; and generating the two-dimensional mask using the two-dimensional mask prediction module based on the slice and the user prompt.

[0016] In the above method, the 3D mask prediction module is trained through the following steps: acquiring a dataset, the dataset including a 3D image to be segmented as input information and the true values ​​of the 3D masks of the segmented 3D image as output information; using the 3D affinity prediction module to generate a 3D affinity map based on the 3D image to be segmented in the dataset; using the 2D mask prediction module to generate a 2D mask based on slices in the 3D image to be segmented; replacing the slices in the 3D image to be segmented with the 2D mask to generate a replaced 3D image; using the 3D mask prediction module to generate a prediction result of the 3D mask based on the 3D affinity map and the replaced 3D image; calculating a loss function between the prediction result of the 3D mask and the true values ​​of the 3D masks of the segmented 3D image in the dataset; and updating the parameters of the 3D mask prediction module based on the loss function to obtain a trained 3D mask prediction module.

[0017] In the above method, the three-dimensional mask prediction module updates its parameters through an online learning method.

[0018] In the above method, the three-dimensional image to be segmented is an electron micrograph of a neuron, and different regions in the segmented three-dimensional image correspond to different neurons.

[0019] According to a third aspect of the present invention, a system for image segmentation is provided, comprising means for performing the method as described in any of the above methods.

[0020] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any of the above methods.

[0021] According to a fifth aspect of the present invention, a computer device is provided, including a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, wherein the processor executes the computer program to implement the method described in any of the above methods.

[0022] According to a sixth aspect of the present invention, a computer program product is provided, comprising a computer program, wherein the computer program, when executed by a processor, implements the method described in any of the above methods.

[0023] The image segmentation method and system according to embodiments of the present invention can achieve fast and accurate image segmentation. Attached Figure Description

[0024] The accompanying drawings are included to provide a further understanding of the invention; they are incorporated into and constitute a part of this application. The drawings illustrate embodiments of the invention and, together with this specification, serve to explain the principles of the invention. In the drawings:

[0025] Figure 1 This is a flowchart of a method for two-dimensional image segmentation according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of a two-dimensional image segmentation process according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of a method for two-dimensional image segmentation according to another embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a two-dimensional image segmentation and correction process according to an embodiment of the present invention;

[0029] Figure 5 This is a flowchart of a method for training a two-dimensional mask prediction module according to an embodiment of the present invention;

[0030] Figure 6 This is a flowchart of a method for three-dimensional image segmentation according to an embodiment of the present invention;

[0031] Figure 7A This is a schematic flowchart of three-dimensional image segmentation according to an embodiment of the present invention;

[0032] Figure 7BThis is a schematic flowchart of three-dimensional image segmentation according to another embodiment of the present invention;

[0033] Figure 8 This is a flowchart of a method for training a 3D mask prediction module according to an embodiment of the present invention;

[0034] Figure 9 This is a flowchart of a method for three-dimensional image segmentation according to another embodiment of the present invention;

[0035] Figure 10 This is a block diagram of a system for two-dimensional image segmentation according to an embodiment of the present invention;

[0036] Figure 11 This is a block diagram of a system for three-dimensional image segmentation according to an embodiment of the present invention;

[0037] Figure 12 This is a block diagram of a system for three-dimensional image segmentation according to another embodiment of the present invention;

[0038] Figures 13A-13D This is a performance evaluation result diagram for 2D image segmentation; and

[0039] Figure 14 This is a graph showing the performance evaluation results of 3D image segmentation. Detailed Implementation

[0040] Embodiments of the invention will now be described in detail with reference to the accompanying drawings, but the invention is not limited thereto but is defined solely by the claims. In the drawings, some elements may be enlarged and drawn out of scale for illustrative purposes. Wherever possible, the same reference numerals will be used in all drawings to denote the same or similar parts.

[0041] Although the terminology used in this invention is selected from commonly known and used terms, some terms mentioned in this specification may have been chosen by the applicant in his or her judgment, and their detailed meanings are explained in the relevant sections of the description herein. Furthermore, the invention should be understood not only by the actual terms used, but also by the meaning implied by each term.

[0042] Numerous specific details are set forth in the description provided herein. However, it should be understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of the invention.

[0043] While existing neuronal image segmentation methods can achieve good results in specific species using deep learning techniques, they still have some significant drawbacks. Firstly, existing methods are heavily reliant on large-scale labeled data, which may be difficult to obtain in different species, especially in non-model organisms, making it difficult for segmentation models to generalize to new species. Secondly, existing segmentation models often struggle to handle complex neuronal structures, leading to the need for extensive manual verification of the segmentation results, increasing time and resource consumption.

[0044] In view of the above problems, this application proposes a method and system for image segmentation. Although for clarity, this application mainly describes neuron segmentation in neuron images, it can also be applied to other biological data or different types of image datasets, such as cell segmentation in animal and plant tissue sections, microscopic biological images, etc.

[0045] Due to limitations in imaging technology, three-dimensional images of neurons under an electron microscope are usually generated by superimposing two-dimensional slice images. Figure 1 This is a flowchart of a method 100 for two-dimensional image segmentation according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a two-dimensional image segmentation process according to an embodiment of the present invention.

[0046] At step 102, a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented can be obtained. In one embodiment, the two-dimensional image to be segmented can be an electron microscope image of a neuron. In one embodiment, the first user prompt can include positive or negative prompts for pixels in the two-dimensional image to be segmented, where a positive prompt indicates that the pixel belongs to the foreground and a negative prompt indicates that the pixel belongs to the background. A pixel belonging to the foreground means that the pixel belongs to the neuron to be segmented, while a pixel belonging to the background means that the pixel does not belong to the neuron, or belongs to the neuron that does not need to be segmented.

[0047] like Figure 2 As shown, users can provide raw slices of electron microscope images as 2D images to be segmented, and user prompts can be provided for the raw slices. Figure 2 In the process, user prompts can include positive prompts represented by green pentagrams, indicating that the pixel belongs to the foreground, i.e., the neuron to be segmented. User prompts can also include negative prompts represented by red pentagrams, indicating that the pixel belongs to the background, i.e., it cannot be segmented or does not need to be segmented.

[0048] In one embodiment, the first user prompt may take any one or more of the following forms: a point prompt, a shape prompt, or a region prompt. For example, the user prompt may be provided by clicking a corresponding location on the image using an input device (e.g., a mouse, keyboard, etc.), thereby providing a prompt for the pixels at that corresponding location. Alternatively, the user prompt may be provided by defining a shape or drawing a region at a corresponding location on the image using an input device, thereby providing a prompt for the pixels contained in the shape or region.

[0049] In step 104, a two-dimensional affinity prediction module can be used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented. The two-dimensional affinity prediction module can employ two-dimensional versions of suitable affinity prediction models, such as the Local Shape Descriptors (LSD) neuron segmentation model (ACRLSD) or the MALA (Metropolis-Adjusted Langevin Algorithm) model. Figure 2 As shown, the two-dimensional affinity prediction module can take the original two-dimensional slice as input and predict the affinity between neurons to generate a two-dimensional affinity map.

[0050] At step 106, the first user prompt can be converted into a first prompt image.

[0051] In one embodiment, when the first user prompt is empty, the first prompt image can be a default matrix corresponding to the two-dimensional image to be segmented. For example, when the user prompt is empty, i.e., the user has not provided a prompt, a default matrix corresponding to the original slice size can be used as the prompt image. All values ​​of the default matrix are the same, such as a matrix with all values ​​of one, a matrix with all values ​​of two, etc.

[0052] In one embodiment, when the first user prompt is not empty, the first prompt graph may include distribution information obtained by applying a distribution function to the first user prompt.

[0053] For example, when user prompts include only positive prompts, or when user prompts include both positive and negative prompts, a positive distribution function can be applied to the positive prompt points and a negative distribution function can be applied to the negative prompt points, thus obtaining a prompt map containing the distribution information of all prompt points.

[0054] by Figure 2 For example, when a user prompt includes both positive prompts represented by green pentagrams and negative prompts represented by red pentagrams, the prompt image includes scatter points of different types corresponding to different types of prompts, and the scatter points have a certain area.

[0055] For example, when user prompts only include negative prompts, a distribution function is applied to the negative prompt points, and the distribution information of the negative prompt points is subtracted from the default matrix to obtain the prompt map.

[0056] The following is an example of the conversion rules used to transform user prompts into prompt images:

[0057]

[0058] PM represents the prompt image. Let μ be the set of tooltips, μ be the position of a tooltip, and ρ ∈ {-1, 1} be the type of tooltip, where ρ = 1 represents a positive tooltip and ρ = -1 represents a negative tooltip. X is the covariance matrix, where X represents all pixels in the 2D image, and J is the default matrix (e.g., a matrix of all ones with the same shape as the 2D image is used here). The value of the covariance matrix can also be set to other values ​​as needed.

[0059] When the user does not provide a prompt, i.e., a set of prompts. If empty, the hint graph is set to the default value (J), indicating that a mask for all neurons should be predicted.

[0060] When the user provides a positive prompt (ρ=1), including providing only a positive prompt or providing both a positive and a negative prompt, it indicates that only the neurons selected by the positive prompt should be segmented, the regions with negative prompts should be deleted, and other regions without prompts should be ignored, that is, other regions without prompts should not be processed.

[0061] When the user provides only a negative cue (ρ = -1), the instruction is to split all neurons that were not selected by the negative cue.

[0062] For example, when a user provides a cue point at the (x,y) coordinates of the image to be segmented via an input device, a Gaussian distribution is generated centered on that coordinate. If the cue point is a positive cue, the Gaussian distribution is multiplied by a positive coefficient; if the cue point is a negative cue, the Gaussian distribution is multiplied by a negative coefficient. Then, the distribution information of all cue points is merged into a cue map.

[0063] It should be understood that although formula (1) uses a Gaussian distribution as the distribution function, other distribution functions can be used as needed. In addition, the specific forms and parameters of formulas (1) and (2) can be adjusted as needed without departing from the scope of this application.

[0064] At step 108, a first two-dimensional mask can be generated using a two-dimensional mask prediction module based on a two-dimensional affinity map and a first cue map. The two-dimensional mask prediction module can employ a two-dimensional version of a suitable model, such as a U-Net-based model, a Fully Convolutional Network (FCN), or a SegNet model.

[0065] like Figure 2 As shown, a binary 2D mask can be generated using a 2D mask prediction module, based on a 2D affinity map, and a cue map derived from user cues. In the 2D mask, the white portion represents the foreground, referring to all pixels of the neurons to be segmented, and can be represented by a value of 1; the black portion represents the background, referring to the parts that do not need to be segmented, and can be represented by a value of 0. Although specific colors or values ​​are used here to indicate the foreground and background, any other suitable method can be used as needed.

[0066] At step 110, connected component analysis can be performed on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels. In one embodiment, when the two-dimensional image to be segmented is an electron micrograph of a neuron, different regions in the first segmented two-dimensional image correspond to different neurons.

[0067] like Figure 2 As shown, by performing Connected Component Analysis (CCA) on a two-dimensional mask, a segmented image can be obtained, where different neurons (i.e., different regions) are marked with different colors, and the background is marked with another color (e.g., white). Alternatively, the segmentation result can be indicated by assigning different numbers to pixels belonging to different neurons.

[0068] For example, when executing CCA, it is possible to Figure 2 In the white foreground region of the 2D mask shown, a seed point is selected. Then, using this seed point as the center, points in its four or eight neighborhoods that are also foreground are searched. If such points exist, it means the two points are connected. This process is then repeated until all connected regions are found. Since different neurons are separated by the black background region, a connected region represents that it belongs to a neuron, thus allowing all neurons in the 2D mask to be found in this way. Finally, different labels can be assigned to different neurons to obtain the segmented image.

[0069] The interactive image segmentation method based on user prompts described above can improve segmentation accuracy. Furthermore, when applied to new species lacking labeled data, it can still achieve high-precision neuron segmentation by providing only a few cue points, thus demonstrating excellent cross-species generalization ability.

[0070] Traditional methods typically employ different approaches to handle the two independent processes of neuron segmentation and proofreading, which not only increases operational complexity but also limits overall segmentation efficiency. Therefore, in addition to image segmentation, the image segmentation method of this application can also be used for the proofreading process, enabling real-time correction through user interaction.

[0071] In one embodiment, method 100 may further include steps 112-118, such as... Figure 3 As shown. After step 110, method 100 can proceed to step 112.

[0072] At step 112, a second user prompt corresponding to the two-dimensional image to be segmented can be obtained based on the first segmented two-dimensional image.

[0073] Figure 4 This is a schematic diagram illustrating a two-dimensional image segmentation and correction process according to an embodiment of the present invention. Figure 4 As shown, the area above the dashed line represents the initial image segmentation process, while the area below the dashed line represents the verification process. After the initial segmentation, magnification reveals that boundary errors in the mask lead to merging errors, where different neurons are combined into a single neuron. Users can provide positive and / or negative feedback based on the segmentation results. For example, a negative feedback symbol (represented by a red pentagram) can be given for an incorrectly merged neuron, indicating that the neuron does not need to be segmented. Conversely, a positive feedback symbol (represented by a green pentagram) can be given for the same neuron in another process, indicating that the neuron needs to be segmented.

[0074] At step 114, the second user prompt can be converted into a second prompt image. This conversion process is similar to the conversion process discussed above regarding step 106, and will not be repeated here for clarity.

[0075] At step 116, a second two-dimensional mask can be generated using the two-dimensional mask prediction module, based on the two-dimensional affinity map and the second cue map. This two-dimensional affinity map is the same as the one generated in step 104.

[0076] At step 118, connected component analysis can be performed on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0077] like Figure 4As shown, when only negative cues (indicated by red pentagrams) are provided, it means the neuron indicated by the negative cue does not need to be segmented, and only the remaining parts are segmented. Therefore, a segmented image is obtained where only that neuron is not segmented. When only positive cues (indicated by green pentagrams) are provided, it means only the neuron indicated by the positive cue needs to be segmented. Therefore, a segmented image is obtained where only that neuron is segmented. Then, by overlaying the correctly segmented portion of the image onto the segmented image where only that neuron is not segmented, a correct segmented image is obtained, thus completing the calibration process.

[0078] Therefore, by utilizing user prompts for neuron segmentation and verification, the segmentation and verification processes can be unified, significantly reducing the workload of manual annotation and verification. Furthermore, it enables timely correction of segmentation errors, thereby improving segmentation accuracy and efficiency.

[0079] Figure 5 This is a flowchart of a method 500 for training a two-dimensional mask prediction module according to an embodiment of the present invention.

[0080] At step 502, a dataset can be obtained. The dataset may include the 2D image to be segmented and user prompts corresponding to the 2D image as input information, and the ground truth value of the 2D mask of the segmented 2D image as output information. The user prompts may include any one or more of the following types: providing positive prompts for all neurons in the 2D image to be segmented (i.e., the default prompt); providing positive prompts for a randomly selected subset of neurons in the 2D image to be segmented; providing negative prompts for a randomly selected subset of neurons in the 2D image to be segmented. The ground truth value of the 2D mask of the segmented 2D image can be obtained by converting the ground truth value of the segmented 2D image into a 2D mask.

[0081] At step 504, a two-dimensional affinity map can be generated using the two-dimensional affinity prediction module based on the two-dimensional image to be segmented in the dataset.

[0082] At step 506, user cues in the dataset can be converted into a cue graph. In one embodiment, when the user cues are default cues, the cue graph can be a default matrix corresponding to the two-dimensional image to be segmented. In one embodiment, when the user cues include positive or negative cues, the cue graph can include distribution information obtained by applying a distribution function to the user cues.

[0083] At step 508, a two-dimensional mask prediction module can be used to generate a prediction result for the two-dimensional mask based on a two-dimensional affinity map and a cue map.

[0084] At step 510, a loss function can be calculated between the predicted result of the two-dimensional mask and the true value of the two-dimensional mask of the segmented two-dimensional image in the dataset.

[0085] In one embodiment, the loss function may include a loss regularization term. The loss regularization term can be the product of the ground truth of affinity and the predicted value of the mask. For example, this loss regularization term can be expressed by the following formula:

[0086] R(g a ,y m ) = g a ·y m (3)

[0087] The loss function can be any one or more suitable functions such as the binary cross-entropy loss function, the DICE loss function, or the MSE (mean square error) loss function. For example, when using the binary cross-entropy loss function CE(·,·) and the DICE loss function, the loss function can be expressed by the following formula:

[0088] L=CE(g m ,y m )+DC(g m ,y m )+R(g a ,y m (4)

[0089] Among them, g m ∈{0,1} represents the truth mask of a voxel, y m This represents the predicted value of its mask, g. a ∈{0,1} represents the truth affinity of a voxel. For a voxel (g) located on the boundary of a neuron... a =1), expect their y m As close to 0 as possible, that is, for example, when the truth value affinity g a When = 1, it means that the voxel is on the boundary of the neuron, therefore the expected mask prediction value y is... m =0 means that it corresponds to the background.

[0090] By utilizing this loss regularization term, the continuity of the neuron boundaries within the mask predicted by the two-dimensional mask prediction model can be enhanced.

[0091] At step 512, the parameters of the two-dimensional mask prediction module can be updated based on the loss function to obtain the trained two-dimensional mask prediction module.

[0092] In one embodiment, the two-dimensional mask prediction module can update its parameters using an online learning method. For example, online learning methods could include Averaged Gradient Episodic Memory (A-GEM), elastic weight merging, or synaptic intelligence. By using online learning methods, the two-dimensional mask prediction module can continuously acquire new tasks while retaining previously acquired knowledge. This significantly reduces the forgetting of previous tasks when progressively learning new species data, enabling efficient segmentation and tracking of cross-species data.

[0093] By using online learning methods to update the parameters of the 2D mask prediction module, the module can gradually optimize itself, improve cross-species segmentation accuracy, and reduce dependence on labeled data, thus achieving excellent efficiency and accuracy when processing new species data.

[0094] In electron microscope image segmentation, besides segmenting two-dimensional slices, it is sometimes necessary to segment three-dimensional images spanning multiple two-dimensional slices, or to perform three-dimensional image tracking, in order to obtain the three-dimensional information of each neuron. However, existing neuron segmentation methods are difficult to effectively handle three-dimensional tracking tasks across slices, resulting in a continued reliance on extensive manual operations in the processing of three-dimensional data. Therefore, this application proposes a method for three-dimensional image segmentation.

[0095] Figure 6 This is a flowchart of a method 600 for three-dimensional image segmentation according to an embodiment of the present invention. Figure 7A This is a schematic diagram of a three-dimensional image segmentation process according to an embodiment of the present invention.

[0096] At step 602, the three-dimensional image to be segmented can be acquired. In one embodiment, the three-dimensional image to be segmented can be an electron microscope image of a neuron.

[0097] At step 604, a 3D affinity prediction module can be used to generate a 3D affinity map based on the 3D image to be segmented. The 3D affinity prediction module can employ a 3D version of a suitable affinity prediction model, such as the Local Shape Descriptors (LSD) neuron segmentation model (ACRLSD) or the MALA (Metropolis-Adjusted Langevin Algorithm) model. Figure 7A As shown, the three-dimensional affinity prediction module can take the original three-dimensional image as input and predict the affinity between neurons to generate a three-dimensional affinity map.

[0098] At step 606, a two-dimensional mask can be generated using the two-dimensional mask prediction module based on slices in the three-dimensional image to be segmented. These slices can be any slice from the original three-dimensional image.

[0099] For example, such as Figure 7A As shown, a 2D mask prediction module can be used to generate a binarized 2D mask based on the first slice of the original 3D image. In the 2D mask, the white portion represents the foreground, referring to all pixels of the neurons to be segmented, and can be represented by a value of 1; the black portion represents the background, referring to the parts that do not need to be segmented, and can be represented by a value of 0. Although specific colors or values ​​are used here to indicate the foreground and background, any other suitable method can be used as needed. The 2D mask prediction module can employ 2D versions of suitable models such as those based on the U-Net architecture, Fully Convolutional Networks (FCNs), or SegNet.

[0100] At step 608, slices in the 3D image to be segmented can be replaced with 2D masks to generate a replaced 3D image. Figure 7A For example, the first slice of the original 3D image can be replaced with a 2D mask generated by the 2D mask prediction module to generate a replaced 3D image.

[0101] At step 610, a 3D mask can be generated using a 3D mask prediction module based on a 3D affinity map and a replaced 3D image. The 3D mask prediction module can employ a 3D version of a suitable model, such as a U-Net-based model, a Fully Convolutional Network (FCN), or a SegNet model.

[0102] like Figure 7A As shown, a binarized 3D mask can be generated using a 3D mask prediction module based on a 3D affinity map and a replaced 3D image. In the 3D mask, the white portion represents the foreground, referring to all pixels of the neurons to be segmented, and can be represented by a value of 1; the black portion represents the background, referring to the parts that do not need to be segmented, and can be represented by a value of 0. Although specific colors or values ​​are used here to indicate the foreground and background, any other suitable method can be used as needed.

[0103] At step 612, connected component analysis can be performed on the 3D mask to generate a segmented 3D image, wherein different regions in the segmented 3D image have different labels. In one embodiment, when the 3D image to be segmented is an electron micrograph of a neuron, different regions in the segmented 3D image correspond to different neurons.

[0104] like Figure 7A As shown, a segmented image can be obtained by performing connected component analysis (CCA) on a 3D mask, where different neurons (i.e., different regions) are marked with different colors, and the background is marked with other colors (e.g., white). Alternatively, the segmentation result can be indicated by assigning different numbers to pixels belonging to different neurons.

[0105] For example, when executing CCA, it is possible to Figure 7A In the 3D mask shown, a seed point is selected within the white foreground region. Then, using this seed point as the center, points in its four or eight neighborhoods that are also foreground are searched. If such points exist, the two points are considered connected. This process is repeated until all connected regions are found. Since different neurons are separated by the black background region, a connected region represents that it belongs to a neuron, thus allowing all neurons in the 3D mask to be found in this way. Finally, different labels can be assigned to different neurons to obtain the segmented image.

[0106] Compared with traditional 3D image tracking methods that directly generate 3D masks based on 3D images, the method in this application replaces slices in the 3D image with 2D masks of those slices, and then generates 3D masks based on the replaced 3D image. This can improve segmentation accuracy, increase the efficiency and accuracy of 3D data processing, and ensure the feasibility of large-scale 3D reconstruction.

[0107] In one embodiment, when the two-dimensional mask prediction module is used in step 606 to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented, user prompts corresponding to the slices can also be obtained. User prompts can include positive or negative prompts for pixels in the two-dimensional slice; positive prompts indicate that the pixel belongs to the foreground, and negative prompts indicate that the pixel belongs to the background. Figure 7B This is a schematic diagram of a three-dimensional image segmentation process according to another embodiment of the present invention.

[0108] exist Figure 7B In this process, users can provide user prompts for the original slices. User prompts can include positive prompts, represented by green pentagrams, indicating that the pixel belongs to the foreground, i.e., the neuron to be segmented. User prompts can also include negative prompts, represented by red pentagrams, indicating that the pixel belongs to the background, i.e., it cannot be segmented or does not need to be segmented.

[0109] Then, a two-dimensional mask can be generated using a two-dimensional mask prediction module, based on slicing and user prompts. The two-dimensional mask prediction module can be a reference... Figure 2 The described two-dimensional mask prediction module can also employ any suitable model that can generate two-dimensional masks based on user prompts.

[0110] By providing user prompts during the mask prediction process for 2D slices, the prediction accuracy of 2D masks can be improved, thereby further improving the prediction accuracy of 3D masks, and enabling selective image segmentation based on user prompts.

[0111] Figure 8 This is a flowchart of a method 800 for training a 3D mask prediction module according to an embodiment of the present invention.

[0112] At step 802, a dataset can be obtained. The dataset may include the 3D image to be segmented as input information and may include the ground truth value of the 3D mask of the segmented 3D image as output information. The ground truth value of the 3D mask of the segmented 3D image can be obtained by converting the ground truth value of the segmented 3D image into a 3D mask.

[0113] At step 804, a three-dimensional affinity map can be generated using the three-dimensional affinity prediction module based on the three-dimensional images to be segmented in the dataset.

[0114] At step 806, a two-dimensional mask can be generated using a two-dimensional mask prediction module based on slices in the three-dimensional image to be segmented.

[0115] At step 808, slices in the three-dimensional image to be segmented can be replaced with two-dimensional masks to generate a replaced three-dimensional image.

[0116] At step 810, a 3D mask prediction module can be used to generate a prediction result for the 3D mask based on a 3D affinity map and a replaced 3D image.

[0117] At step 812, a loss function can be calculated between the predicted result of the 3D mask and the true value of the 3D mask of the segmented 3D image in the dataset.

[0118] At step 814, the parameters of the 3D mask prediction module can be updated based on the loss function to obtain the trained 3D mask prediction module.

[0119] In one embodiment, the 3D mask prediction module can update its parameters using an online learning method. For example, the online learning method could be average gradient fragment memory (A-GEM), elastic weight merging, or synaptic intelligence. By using online learning methods to semi-automatically annotate new species data, the 3D mask prediction module can continuously acquire new tasks while retaining previously acquired knowledge. This significantly reduces the forgetting of previous tasks as it progressively learns from new species data, achieving efficient segmentation and tracking of cross-species data.

[0120] By using online learning methods to update the parameters of the 3D mask prediction module, the module can gradually optimize itself, improve cross-species segmentation accuracy, and reduce dependence on labeled data, thus achieving excellent efficiency and accuracy when processing new species data.

[0121] Figure 9 This is a flowchart of a method 900 for three-dimensional image segmentation according to another embodiment of the present invention.

[0122] Steps 902-910 and references Figure 1 Steps 102-110 in the described method 100 are the same. To avoid redundancy, the specific details of steps 902-910 are not described here. After completing step 910, it is known that the predicted two-dimensional mask can generate a segmented two-dimensional image that meets the requirements. The two-dimensional mask can contain all neurons or only the required portion of neurons.

[0123] At step 912, the three-dimensional image to be segmented can be obtained. The two-dimensional image to be segmented, which was previously segmented into a two-dimensional image, is a slice of the three-dimensional image to be segmented.

[0124] At step 914, a 3D affinity prediction module can be used to generate a 3D affinity map based on the 3D image to be segmented. The 3D affinity prediction module can employ a 3D version of a suitable affinity prediction model, such as the Local Shape Descriptors (LSD) neuron segmentation model (ACRLSD) or the MALA (Metropolis-Adjusted Langevin Algorithm) model.

[0125] At step 916, the two-dimensional image to be segmented in the three-dimensional image to be segmented can be replaced with the first two-dimensional mask to generate the replaced three-dimensional image.

[0126] At step 918, a 3D mask can be generated using a 3D mask prediction module based on a 3D affinity map and a replaced 3D image. The 3D mask prediction module can employ a 3D version of a suitable model, such as a U-Net-based model, a Fully Convolutional Network (FCN), or a SegNet model.

[0127] At step 920, connected component analysis can be performed on the 3D mask to generate a segmented 3D image, wherein different regions in the segmented 3D image have different labels.

[0128] In one embodiment, the 3D mask prediction module can be trained through the following steps: acquiring a dataset that includes a 3D image to be segmented as input information and ground truth values ​​of 3D masks of the segmented 3D image as output information; using a 3D affinity prediction module to generate a 3D affinity map based on the 3D image to be segmented in the dataset; using a 2D mask prediction module to generate a 2D mask based on slices in the 3D image to be segmented; replacing the slices in the 3D image to be segmented with the 2D mask to generate a replaced 3D image; using the 3D mask prediction module to generate a prediction result of the 3D mask based on the 3D affinity map and the replaced 3D image; calculating a loss function between the prediction result of the 3D mask and the ground truth values ​​of the 3D masks of the segmented 3D image in the dataset; and updating the parameters of the 3D mask prediction module based on the loss function to obtain a trained 3D mask prediction module.

[0129] In one embodiment, the 3D mask prediction module can update its parameters using an online learning method.

[0130] Therefore, by performing interactive segmentation based on user prompts on only representative 2D images, and then using 2D masks instead of 2D images for 3D image tracking, the workload of interaction and annotation can be reduced exponentially.

[0131] Figure 10 This is a block diagram of a system 1000 for two-dimensional image segmentation according to an embodiment of the present invention. System 1000 may include an acquisition module 1002, a two-dimensional affinity prediction module 1004, a cue conversion module 1006, a two-dimensional mask prediction module 1008, and a connected component analysis module 1010. The acquisition module 1002 can be used to acquire a two-dimensional image to be segmented and a first user cue corresponding to the two-dimensional image to be segmented. The two-dimensional affinity prediction module 1004 can be used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented. The cue conversion module 1006 can be used to convert the first user cue into a first cue map. The two-dimensional mask prediction module 1008 can be used to generate a first two-dimensional mask based on the two-dimensional affinity map and the first cue map. The connected component analysis module 1010 can be used to perform connected component analysis on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

[0132] In one embodiment, the first user prompt may include a positive or negative prompt for a pixel in the two-dimensional image to be segmented, wherein a positive prompt indicates that the pixel belongs to the foreground and a negative prompt indicates that the pixel belongs to the background.

[0133] In one embodiment, the first user prompt may take any one or more of the following forms: dot prompt, shape prompt, or area prompt.

[0134] In one embodiment, when the first user prompt is empty, the first prompt graph can be a default matrix corresponding to the two-dimensional image to be segmented; or when the first user prompt is not empty, the first prompt graph can include distribution information obtained by applying a distribution function to the first user prompt.

[0135] In one embodiment, the acquisition module 1002 can also be used to acquire a second user cue corresponding to the two-dimensional image to be segmented, based on the first segmented two-dimensional image. The cue conversion module 1006 can also be used to convert the second user cue into a second cue map. The two-dimensional mask prediction module 1008 can also be used to generate a second two-dimensional mask based on the two-dimensional affinity map and the second cue map. The connected component analysis module 1010 can also be used to perform connected component analysis on the second two-dimensional mask to generate a second segmented two-dimensional image.

[0136] In one embodiment, the two-dimensional mask prediction module 1008 can be trained through the following steps: acquiring a dataset, which includes a two-dimensional image to be segmented and user prompts corresponding to the two-dimensional image to be segmented as input information, and includes the true values ​​of the two-dimensional masks of the segmented two-dimensional image as output information; using the two-dimensional affinity prediction module to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the dataset; converting the user prompts in the dataset into prompt maps; using the two-dimensional mask prediction module to generate a prediction result of the two-dimensional mask based on the two-dimensional affinity map and the prompt maps; calculating a loss function between the prediction result of the two-dimensional mask and the true values ​​of the two-dimensional masks of the segmented two-dimensional image in the dataset; and updating the parameters of the two-dimensional mask prediction module based on the loss function to obtain the trained two-dimensional mask prediction module.

[0137] In one embodiment, the loss function may include a loss regularization term, which is the product of the true value of affinity and the predicted value of the mask.

[0138] In one embodiment, the two-dimensional mask prediction module 1008 can update its parameters through an online learning method.

[0139] Figure 11This is a block diagram of a system 1100 for three-dimensional image segmentation according to an embodiment of the present invention. System 1100 may include an acquisition module 1102, a three-dimensional affinity prediction module 1104, a two-dimensional mask prediction module 1106, a slice replacement module 1108, a three-dimensional mask prediction module 1110, and a connected component analysis module 1112. The acquisition module 1102 can be used to acquire a three-dimensional image to be segmented. The three-dimensional affinity prediction module 1104 can be used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented. The two-dimensional mask prediction module 1106 can be used to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented. The slice replacement module 1108 can be used to replace slices in the three-dimensional image to be segmented with two-dimensional masks to generate a replaced three-dimensional image. The three-dimensional mask prediction module 1110 can be used to generate a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image. The connected component analysis module 1112 can be used to perform connected component analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0140] In one embodiment, the acquisition module 1102 can also be used to acquire user prompts corresponding to the slices. The two-dimensional mask prediction module 1106 can also be used to generate a two-dimensional mask based on the slices and user prompts.

[0141] In one embodiment, the 3D mask prediction module 1110 can be trained through the following steps: acquiring a dataset, which includes a 3D image to be segmented as input information and the true values ​​of the 3D masks of the segmented 3D image as output information; using a 3D affinity prediction module to generate a 3D affinity map based on the 3D image to be segmented in the dataset; using a 2D mask prediction module to generate a 2D mask based on slices in the 3D image to be segmented; replacing the slices in the 3D image to be segmented with the 2D mask to generate a replaced 3D image; using a 3D mask prediction module to generate a prediction result of the 3D mask based on the 3D affinity map and the replaced 3D image; calculating a loss function between the prediction result of the 3D mask and the true values ​​of the 3D masks of the segmented 3D image in the dataset; and updating the parameters of the 3D mask prediction module based on the loss function to obtain a trained 3D mask prediction module.

[0142] In one embodiment, the 3D mask prediction module 1110 can update its parameters using an online learning method.

[0143] Figure 12This is a block diagram of a system 1200 for three-dimensional image segmentation according to another embodiment of the present invention. System 1200 may include an acquisition module 1202, a two-dimensional affinity prediction module 1204, a cue conversion module 1206, a two-dimensional mask prediction module 1208, a connected component analysis module 1210, a three-dimensional affinity prediction module 1212, a slice replacement module 1214, and a three-dimensional mask prediction module 1216. The acquisition module 1202 to the connected component analysis module 1210 are related to a reference... Figure 10 The acquisition module 1002 to the connected component analysis module 1010 in the described system 1000 are identical. To avoid redundancy, the specific details of the acquisition module 1202 to the connected component analysis module 1210 are not described here. The acquisition module 1202 can also be used to acquire the three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice of the three-dimensional image to be segmented. The three-dimensional affinity prediction module 1212 can be used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented. The slice replacement module 1214 can be used to replace the two-dimensional image to be segmented in the three-dimensional image to be segmented with a first two-dimensional mask to generate a replaced three-dimensional image. The three-dimensional mask prediction module 1216 can be used to generate a three-dimensional mask based on the three-dimensional affinity map and the replaced three-dimensional image. The connected component analysis module 1210 can also be used to perform connected component analysis on the three-dimensional mask to generate a segmented three-dimensional image, wherein different regions in the segmented three-dimensional image have different labels.

[0144] In one embodiment, the 3D mask prediction module 1216 can be trained through the following steps: acquiring a dataset, which includes a 3D image to be segmented as input information and the true values ​​of the 3D masks of the segmented 3D image as output information; using a 3D affinity prediction module to generate a 3D affinity map based on the 3D image to be segmented in the dataset; using a 2D mask prediction module to generate a 2D mask based on slices in the 3D image to be segmented; replacing the slices in the 3D image to be segmented with the 2D mask to generate a replaced 3D image; using a 3D mask prediction module to generate a prediction result of the 3D mask based on the 3D affinity map and the replaced 3D image; calculating a loss function between the prediction result of the 3D mask and the true values ​​of the 3D masks of the segmented 3D image in the dataset; and updating the parameters of the 3D mask prediction module based on the loss function to obtain a trained 3D mask prediction module.

[0145] In one embodiment, the 3D mask prediction module 1216 can update its parameters using an online learning method.

[0146] Figures 13A-13BThis is a performance comparison of ACRLSD-2D, SegNeuro-2D, and the improved SegNeuro-2D*. ACRLSD-2D is a two-dimensional version of the ACRLSD model. The SegNeuro-2D model is an example of the two-dimensional image segmentation method proposed in this application, trained using cross-entropy and the Dice loss function. The improved SegNeuro-2D* model is another example of the two-dimensional image segmentation method proposed in this application, which, compared to the SegNeuro-2D model, adds a loss term related to neuron boundaries to the cross-entropy and Dice loss functions. The three models were trained on Drosophila data and tested on Zebra finches for cross-species evaluation. Figure 13A This comparison examines the performance of ACRLSD-2D, SegNeuro-2D, and the improved SegNeuro-2D* on a selected slice of data from the zebra finch dataset. In the zebra finch test cases, ACRLSD-2D achieved a variation of information (VOI) of 2.1093 without prompts, while SegNeuro-2D and the improved SegNeuro-2D* achieved 2.2587 and 1.9752, respectively. A lower VOI value indicates a result closer to the true value. Although ACRLSD-2D, SegNeuro-2D, and SegNeuro-2D* performed relatively poorly in predicting the boundaries of some zebra finch neurons without prompts, as shown on the right, SegNeuro-2D* significantly improved the mask prediction accuracy for neurons with poor boundaries through minimal manual correction of prompt points, ultimately achieving a VOI of 1.7734. Figure 13B This is a quantitative comparison of the performance of the three models across all 33 zebra finch datasets. The results show that SegNeuro-2D* exhibits the best VOI across all 33 datasets, particularly demonstrating excellent cross-species generalization performance in the cue-based default mode, and performing even better with human prompts.

[0147] Therefore, the interactive image segmentation method of this application can improve the segmentation accuracy of the model across species.

[0148] The experiment further simulated training the SegNeuro-2D model using an online learning method. The model was initially trained on Drosophila data, and then trained online sequentially on the first six Zebra Finch datasets, with the subsequent 25 Zebra Finch datasets used for testing and evaluation. Experimental results showed that the VOI of the SegNeuro-2D model significantly decreased with the increase in the amount of Zebra Finch data. When using the A-GEM strategy for online learning, the performance was very close to that of training using all Zebra Finch datasets, and the training time for online learning was only 58.2% of the original training method. Therefore, using an online learning method enables the SegNeuro-2D model to achieve segmentation performance close to that of manually labeled data on new species.

[0149] Figures 13C-13D This is a performance comparison between the existing SAM (Segment Anything Model) model and the two-dimensional image segmentation method proposed in this application. Figure 13C As shown, the SAM model can segment individual neurons based on user prompts, but significant errors occur when prompts about multiple neurons are provided. However, the two-dimensional image segmentation method of this application can achieve accurate segmentation of the corresponding neurons when prompted about any number of neurons. Figure 13D As shown, the SAM model can only segment one neuron at a time when a cue is provided, while the two-dimensional image segmentation method of this application can achieve complete segmentation with the default cue and selective segmentation with non-default cue including positive and negative cue.

[0150] Furthermore, the two-dimensional image segmentation method of this application requires significantly fewer interactions compared to the SAM model. In experiments, the SAM model required an average of 95 interactions to segment a slice of 2 cubic micrometer fruit fly data, while the two-dimensional image segmentation method of this application only required two interactions, greatly reducing manual intervention.

[0151] Figure 14 This is a performance evaluation result of 3D image segmentation. SegNeuro-3D is an example of the 3D image segmentation method proposed in this application. In the experiments, the neuron tracking model SegNeuro-3D was trained using the HEMI-BRAIN and FIB-25 datasets, and the three largest neurons in the HEMI-BRAIN ROI-1 dataset were selected for segmentation accuracy testing. Figure 14As shown, the variation of information (VOI) for these three neurons in SegNeuro-3D is close to 0. This indicates that the SegNeuro-3D model can achieve the same accuracy as the true value in 3D neuron tracking, demonstrating excellent automated tracking capabilities.

[0152] This application utilizes an image segmentation method based on artificial intelligence technology, which can significantly improve image segmentation speed and accuracy while reducing manual intervention. The two-dimensional image segmentation method of this application improves data processing efficiency and segmentation accuracy by leveraging user prompts. By integrating segmentation and verification within the same framework, real-time error correction can be achieved, reducing the workload of verification. The three-dimensional image tracking method of this application, based on two-dimensional masks of two-dimensional slices, achieves automatic tracking of three-dimensional neurons across slices, improving the efficiency and accuracy of three-dimensional data processing, reducing the need for manual intervention, and ensuring the feasibility of large-scale three-dimensional reconstruction. Furthermore, by combining the image segmentation method with online learning methods, it can adapt to data from new species, improving the universality of the image segmentation method across different species.

[0153] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0154] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0155] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0156] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0157] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0158] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0159] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0161] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for neuronal image segmentation, comprising: Obtain a two-dimensional image to be segmented and a first user prompt corresponding to the two-dimensional image to be segmented. The first user prompt includes a positive prompt or a negative prompt for a pixel in the two-dimensional image to be segmented. The positive prompt is used to indicate that the pixel belongs to the foreground, and the negative prompt is used to indicate that the pixel belongs to the background. A two-dimensional affinity prediction module is used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented; The first user prompt is converted into a first prompt image, wherein... When the first user prompt is empty, the first prompt image is a default matrix corresponding to the two-dimensional image to be segmented, and the first prompt image indicates the segmentation of all regions; or When the first user prompt is not empty, the first prompt graph includes distribution information obtained by applying a distribution function to the first user prompt, wherein... When the first user prompt includes a positive prompt, the first prompt diagram indicates the segmentation of the area selected by the positive prompt, the deletion of the area with a negative prompt, and the ignoring of other areas without prompts, or When the first user prompt only includes negative prompts, the first prompt map indicates the segmentation of all areas not selected by negative prompts; A first two-dimensional mask is generated using a two-dimensional mask prediction module, based on the two-dimensional affinity map and the first cue map; and Connectivity analysis is performed on the first two-dimensional mask to generate a first segmented two-dimensional image, wherein different regions in the first segmented two-dimensional image have different labels.

2. The method as described in claim 1, wherein, The first user prompt takes any one or more of the following forms: dot prompt, shape prompt, or area prompt.

3. The method as described in claim 1, wherein, The method further includes: Obtain a second user prompt based on the first segmented two-dimensional image, which corresponds to the two-dimensional image to be segmented; Convert the second user prompt into a second prompt image; The second two-dimensional mask is generated using the two-dimensional mask prediction module, based on the two-dimensional affinity map and the second cue map; and Connectivity analysis is performed on the second two-dimensional mask to generate a second segmented two-dimensional image.

4. The method of claim 1, wherein, The two-dimensional mask prediction module is trained through the following steps: Obtain a dataset, which includes a two-dimensional image to be segmented and user prompts corresponding to the two-dimensional image to be segmented as input information, and includes the true value of the two-dimensional mask of the segmented two-dimensional image as output information; The two-dimensional affinity prediction module is used to generate a two-dimensional affinity map based on the two-dimensional image to be segmented in the dataset. Convert the user prompts in the dataset into prompt graphs; The two-dimensional mask prediction module is used to generate the prediction result of the two-dimensional mask based on the two-dimensional affinity map and the cue map; Calculate the loss function between the predicted result of the two-dimensional mask and the true value of the two-dimensional mask of the segmented two-dimensional image in the dataset; as well as The parameters of the two-dimensional mask prediction module are updated based on the loss function to obtain a trained two-dimensional mask prediction module.

5. The method of claim 4, wherein, The loss function includes a loss regularization term, which is the product of the true value of affinity and the predicted value of the mask.

6. The method of claim 1, wherein, The two-dimensional mask prediction module updates its parameters through an online learning method.

7. The method of claim 1, wherein, The method further includes: Obtain a three-dimensional image to be segmented, wherein the two-dimensional image to be segmented is a slice of the three-dimensional image to be segmented; A three-dimensional affinity prediction module is used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented; The two-dimensional image to be segmented in the three-dimensional image to be segmented is replaced with the first two-dimensional mask to generate a replaced three-dimensional image; A 3D mask is generated using a 3D mask prediction module, based on the 3D affinity map and the replaced 3D image; and Connectivity analysis is performed on the 3D mask to generate a segmented 3D image, wherein different regions in the segmented 3D image have different labels.

8. The method of claim 1, wherein, The two-dimensional image to be segmented is an electron microscope image of a neuron, and different regions in the first segmented two-dimensional image correspond to different neurons.

9. A method for neuronal image segmentation, comprising: Obtain the 3D image to be segmented; A three-dimensional affinity prediction module is used to generate a three-dimensional affinity map based on the three-dimensional image to be segmented; A two-dimensional mask is generated based on slices in the three-dimensional image to be segmented using a two-dimensional mask prediction module. The slices in the three-dimensional image to be segmented are replaced with the two-dimensional mask to generate a replaced three-dimensional image; A 3D mask is generated using a 3D mask prediction module based on the 3D affinity map and the replaced 3D image; as well as Connectivity analysis is performed on the 3D mask to generate a segmented 3D image, wherein different regions in the segmented 3D image have different labels.

10. The method of claim 9, wherein, Using a two-dimensional mask prediction module to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented includes: Obtain user prompts corresponding to the slice, the user prompts including positive or negative prompts for pixels in the slice, the positive prompts indicating that the pixel belongs to the foreground, and the negative prompts indicating that the pixel belongs to the background; and The two-dimensional mask is generated using the two-dimensional mask prediction module, based on the slice and the user prompt.

11. The method of claim 9, wherein, The 3D mask prediction module is trained through the following steps: Obtain a dataset, which includes a 3D image to be segmented as input information and the true value of the 3D mask of the segmented 3D image as output information; The three-dimensional affinity prediction module is used to generate a three-dimensional affinity map based on the three-dimensional images to be segmented in the dataset. The two-dimensional mask prediction module is used to generate a two-dimensional mask based on slices in the three-dimensional image to be segmented. The slices in the three-dimensional image to be segmented are replaced with the two-dimensional mask to generate a replaced three-dimensional image; The 3D mask prediction module is used to generate a prediction result of the 3D mask based on the 3D affinity map and the replaced 3D image; Calculate the loss function between the predicted result of the 3D mask and the true value of the 3D mask of the segmented 3D image in the dataset; as well as The parameters of the 3D mask prediction module are updated based on the loss function to obtain a trained 3D mask prediction module.

12. The method of claim 9, wherein, The 3D mask prediction module updates its parameters through an online learning method.

13. The method of claim 9, wherein, The three-dimensional image to be segmented is an electron microscope image of a neuron, and different regions in the segmented three-dimensional image correspond to different neurons.

14. A system for image segmentation, comprising means for performing the method as described in any one of claims 1-13.

15. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-13.

16. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, wherein, When the processor executes the computer program, it implements the method as described in any one of claims 1-13.

17. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-13.

Images