Three-dimensional interactive segmentation method and device, computer equipment and storage medium

Abstract

The invention discloses a three-dimensional interactive segmentation method and device, computer equipment and a storage medium, and is applied to the technical field of computers, and the method comprises the steps: obtaining a three-dimensional fluorescent microscopic image and interactive prompt information; wherein the interactive prompt information comprises a positive sample point, a negative sample point and a preorder segmentation mask; inputting the three-dimensional fluorescent microscopic image and the interactive prompt information into a constructed interactive segmentation model, and outputting a segmentation result; based on feedback of a user on the segmentation result, performing iterative optimization on the interactive segmentation model until the segmentation result meets preset precision; according to the invention, based on deep learning, the interaction prompt information and the three-dimensional fluorescence microscopic image are fused, accurate segmentation of the neuron structure is realized, and the precision and speed of neuron image labeling are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a three-dimensional interactive segmentation method, apparatus, computer device, and storage medium. Background Art

[0002] The image annotation task refers to the process of attaching semantic information to objects, regions, or scenes in a static image, thereby providing supervision signals and training data for computer vision tasks to help the model understand and recognize the image content. In the field of life sciences, the neuron annotation task in three-dimensional fluorescence microscopy images is an important part of mapping the whole-brain mesoscopic connection map, and its goal is to extract the signals of neurons in the image to help researchers determine the positions and connection information of neurons. Due to the limitations of optical imaging, the quality of fluorescence microscopy images is often unsatisfactory, and the volume and quantity of images are huge, resulting in extremely difficult neuron annotation tasks. Due to the high difficulty of the annotation task and the lack of effective annotation data for neuron images, some deep learning algorithms are difficult to have good data for training.

[0003] To overcome these defects, the present application proposes a three-dimensional interactive segmentation method, apparatus, computer device, and storage medium. Summary of the Invention

[0004] The purpose of the present application is to provide a three-dimensional interactive segmentation method, apparatus, computer device, and storage medium, aiming to solve the above problems.

[0005] To achieve the above purpose, the present application provides the following technical solutions:

[0006] In the first aspect, the present application provides a three-dimensional interactive segmentation method, including:

[0007] Obtain a three-dimensional fluorescence microscopy image and interactive prompt information; wherein, the interactive prompt information includes positive sample points, negative sample points, and a previous segmentation mask;

[0008] Input the three-dimensional fluorescence microscopy image and the interactive prompt information into a constructed interactive segmentation model to output a segmentation result;

[0009] Based on the user's feedback on the segmentation result, iteratively optimize the interactive segmentation model until the segmentation result meets a preset accuracy.

[0010] In the second aspect, the present application provides a three-dimensional interactive segmentation apparatus, including:

[0011] An acquisition module: acquire a three-dimensional fluorescence microscopy image and interactive prompt information; wherein, the interactive prompt information includes positive sample points, negative sample points, and a previous segmentation mask;

[0012] Model processing module: Input the three-dimensional fluorescence microscopic image and the interactive prompt information into the constructed interactive segmentation model, and output the segmentation result;

[0013] Iterative optimization module: Based on the user's feedback on the segmentation result, iteratively optimize the interactive segmentation model until the segmentation result meets the preset accuracy.

[0014] In a third aspect, the present application provides a computer device, which includes a processor and a memory coupled to the processor. Among them, the memory stores program instructions for implementing a three-dimensional interactive segmentation method; the processor is used to execute the program instructions stored in the memory to implement a three-dimensional interactive segmentation.

[0015] In a fourth aspect, the present application provides a storage medium storing program instructions that can be run by a processor, and the program instructions are used to execute a three-dimensional interactive segmentation method.

[0016] The present application provides a three-dimensional interactive segmentation method, device, computer device and storage medium, which have the following beneficial effects:

[0017] (1) Process the three-dimensional fluorescence microscopic image through the interactive segmentation model, fully capture the spatial topological structure of neurons, and solve the problem of losing structural information when synthesizing three-dimensional results from two-dimensional sections; at the same time, experiments show that the recall rate of the final segmentation mask reaches 92%, the accuracy rate is above 90%, and the single running speed is in milliseconds, meeting the real-time interaction requirements.

[0018] (2) Prepare the gold-labeled data set based on the A* optimal path algorithm, that is, the instance-level neuron skeleton map. Through the centerline consistency constraint, ensure the quality of the training data and reduce the workload of manual correction.

[0019] (3) The interactive visualization program proposed in the present application supports the rotation, scaling and point annotation operations of three-dimensional images. Users only need to click on the positive and negative sample points to complete the annotation, improving the annotation efficiency.

[0020] (4) Through data iterative optimization, use the interactive segmentation model to label data to feed back the training set, iterate the model to improve performance, and form a positive cycle of "model optimization - data expansion"; at the same time, fill the gap in the lack of three-dimensional interaction models in the field of biological microscopic images. Description of the drawings

[0021] Figure 1 It is a schematic flowchart of a three-dimensional interactive segmentation method according to Embodiment 1 of the present application;

[0022] Figure 2 It is a flowchart of a three-dimensional interactive segmentation method according to Embodiment 1 of the present application;

[0023] Figure 3 It is an example diagram of the training set images of Embodiment 1 of the present application;

[0024] Figure 4 It is a schematic structural diagram of the interactive segmentation model of Embodiment 1 of the present application;

[0025] Figure 5 It is a schematic diagram of the interactive segmentation model of Embodiment 1 of the present application during the training process;

[0026] Figure 6 It is a result comparison diagram of a three-dimensional interactive segmentation method and a traditional method of Embodiment 1 of the present application;

[0027] Figure 7 It is an example diagram of an interaction program applying a three-dimensional interactive segmentation method of Embodiment 1 of the present application;

[0028] Figure 8 It is a schematic structural diagram of a three-dimensional interactive segmentation device of Embodiment 2 of the present application;

[0029] Figure 9 It is a schematic structural diagram of a computer device of Embodiment 3 of the present application;

[0030] Figure 10 It is a schematic structural diagram of a storage medium of Embodiment 4 of the present application. Detailed implementation manners

[0031] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The following analyzes the solutions in the prior art in combination with related technologies.

[0033] Traditional image annotation methods directly manually box and draw the objects to be annotated on the image. The basic method of neuron annotation is: in three-dimensional visualization applications, manually trace the foreground signal along the neuron fibers and perform saturation annotation on the data within the image block. In the early years, some traditional image processing algorithms, such as threshold segmentation, could automatically extract neuron signals in the image, but such methods often only used simple brightness information. There are also methods that use the brightest feature of the center line of the neuron image and use the brightest path tracking algorithm based on the A* pathfinding algorithm to annotate the neuron structure. In recent years, the rise of deep learning methods has brought new methods to the industry. Scholars have trained various neural network models to perform semantic and even instance segmentation on images. The introduction of segmentation models has also brought great convenience to the neuron annotation task.

[0034] There is also an interactive segmentation method that allows the model to utilize manually or automatically provided hint information and combine it with the original image to obtain a preliminary segmentation result. Subsequently, the user provides further hint information to assist the model in refining the result to a more precise one. This process is iterated until the user is satisfied. The application prospects of interactive segmentation algorithms are very broad, and significant progress has been made in both natural images and medical images. The annotation tasks for natural images and medical images have also improved in efficiency with the advancement of interactive segmentation algorithms. However, the promotion in biological microscopic images, especially in three-dimensional neuron images, is still far from sufficient.

[0035] In three-dimensional biological microscopic images, especially under optical imaging systems, the data modality has become three-dimensional, making existing technologies inapplicable directly. Due to the different data modalities, the interactive segmentation models on two-dimensional images cannot be directly used on three-dimensional biological microscopic images. Some scholars have explored applying two-dimensional interactive segmentation models to two-dimensional sections of biological images and then synthesizing three-dimensional results, but the effects are not good. Because neuron images place more emphasis on the topological structure in three-dimensional space, and lack three-dimensional structure information in two-dimensional sections. There has been little exploration of the integration of "interactive" and "three-dimensional data". The lack of a gold-standard dataset has led to a shortage of sufficient data for training algorithm models with strong generalization ability.

[0036] Aiming at the shortcomings of the existing technologies, this application proposes a three-dimensional interactive segmentation method, device, computer device, and storage medium for three-dimensional fluorescence microscopic images of neurons.

[0037] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0038] Embodiment 1

[0039] Please refer to Figure 1 , which is a schematic flowchart of a three-dimensional interactive segmentation method according to Embodiment 1 of the present application. The steps include:

[0040] S1: Obtain a three-dimensional fluorescence microscopic image and interactive hint information. Among them, the interactive hint information includes positive sample points, negative sample points, and a previous segmentation mask.

[0041] In this embodiment, specifically, three-dimensional fluorescence microscopic image data of macaque brain axons are collected, including 20 groups of axon trunk data and 10 groups of axon terminal data. The image resolution is 128×128×128 voxels and is stored in a 16-bit grayscale format. Among them, the positive sample points in the interactive prompt information represent the position information of the target object, the negative sample points represent the position information of non-target areas and interference items, and the previous segmentation mask represents the existing segmentation result.

[0042] S2: Input the three-dimensional fluorescence microscopic image and the interactive prompt information into the constructed interactive segmentation model, and output the segmentation result.

[0043] In this embodiment, the positive sample points, negative sample points, and the previous segmentation mask are represented as three-dimensional grayscale images and fused in the channel dimension. Features are extracted through a three-dimensional convolutional layer to obtain a first feature map; features of the three-dimensional fluorescence microscopic image are extracted through a three-dimensional convolutional layer to obtain a second feature map. The first feature map and the second feature map are added and then input into the backbone network to output the predicted segmentation result.

[0044] The training process of the interactive segmentation model includes:

[0045] Please refer to Figure 3 , which is an example diagram of the training set images of Embodiment 1 of this application.

[0046] Using the feature that the center line of neurons has the highest brightness, the brightest path tracking algorithm of the A* optimal path algorithm is used to perform skeleton annotation on the neuron fibers in the three-dimensional image, which is used as the training set. During the annotation process, the reciprocal of the brightness value of the voxel is used as the movement cost, that is, the brighter the voxel, the smaller the movement cost; the distance from the current position to the key position is used as the target cost, and the total cost is the movement cost plus the target cost. Therefore, starting from the starting point and moving in the direction of the key point, walking along the brightest path can minimize the cost of the traced path. The neuron segments in the training set are instance annotation data, and the annotation results are verified by the consistency of the neuron center lines, and the mask of each segment can be accessed.

[0047] Please refer to Figure 4 , which is a schematic structural diagram of the interactive segmentation model of Embodiment 1 of this application. Specifically, it includes:

[0048] Input layer: Receive the three-dimensional fluorescence microscopic image (size H×W×D), the previous segmentation mask (H×W×D), and the positive and negative sample point coordinates (converted into a three-dimensional grayscale image, size H×W×D×2).

[0049] Feature extraction: Interactive prompt information (mask + positive and negative points) extracts features through a 3D convolutional layer (kernel 3×3×3, output channels 16); the 3D fluorescence microscopic image extracts features through a 3D convolutional layer with the same parameters; the feature maps are stacked in the channel dimension and then input into the 3D U-Net backbone network.

[0050] The U-Net structure includes:

[0051] Encoder: 3 downsampling layers, with feature channels being 32, 64, and 128 in sequence, stride 1, convolutional kernel 3×3×3, activation function ReLU; Decoder: 3 upsampling layers, fusing encoder features through skip connections, with output channels being the same as the input size; Output layer: 1×1×1 convolution + Sigmoid activation, generating a probability mask (H×W×D).

[0052] Please refer to Figure 5 , which is a schematic diagram of the interactive segmentation model in the training process of Embodiment 1 of this application.

[0053] In the training stage, the behavior of the user is simulated through random sampling for end-to-end training. The predicted mask output by the interactive segmentation model is compared with the ground truth mask to divide the true positive region, false positive region, and false negative region.

[0054] Among them, the last layer of the interactive segmentation model outputs a predicted mask after Sigmoid activation. The predicted mask is binarized with a threshold of 0.5 to obtain a binarized predicted mask; the ground truth mask is also a binary image. The false positive region and false negative region are obtained by performing a difference set operation on the predicted mask and the ground truth mask. The set of all pixels that are 1 in the predicted mask but 0 in the ground truth mask forms the false positive region, indicating the region where the model's prediction is correct but the actual situation is wrong. The pixels in the false positive region are originally background but are predicted and classified as foreground by the model. Therefore, the false positive region should be background, and negative sample points can be sampled from it to prompt the model not to segment these regions.

[0055] The set of all pixels that are 0 in the predicted mask but 1 in the ground truth mask forms the false negative region, indicating the region where the model's prediction is wrong but the actual situation is correct. The pixels in the false negative region are originally foreground but are predicted and classified as background by the model. Therefore, the false negative region should be foreground, and positive sample points can be sampled from it to prompt the model to segment these regions.

[0056] In the initial training, the initial segmentation mask is set to be blank, and based on the sampled points as prompt information, the parameters of the interactive segmentation model are updated in combination with the 3D fluorescence microscopic image.

[0057] In iterative training, a combined loss function is used for training, and the parameters of the interactive segmentation model are updated through backpropagation. The combined loss function is the weighted sum of Dice Loss and Focal Loss, and the specific expression is:

[0058]

[0059] where y is the ground truth mask; is the predicted mask; γ is a hyperparameter, which is set to 2 in this embodiment;

[0060]

[0061] S3: Based on the user's feedback on the segmentation result, the interactive segmentation model is iteratively optimized until the segmentation result meets the preset accuracy.

[0062] In this embodiment, for the output segmentation result, the user observes the correct area and the incorrect area and gives new positive and negative sample points. In the next iteration, the interactive segmentation model uses the three-dimensional fluorescence microscopy image, this segmentation result, and the new prompt information to output a new segmentation result. Through such iterative cycles, prompt information is continuously provided to the interactive segmentation model to obtain a more refined and accurate segmentation result.

[0063] Please refer to Figure 6 , which is a result comparison diagram between a three-dimensional interactive segmentation method of Embodiment 1 of this application and a traditional method. In an image scene with dense neurons, the A* algorithm needs to place a starting point and an ending point and perform path search based on the brightest information. However, due to the complex scene, the path search is incorrect, and the neuron fiber pointed by the arrow in the figure is not found, and the entire search time cost is 35 seconds. By using the interactive segmentation model proposed in the present invention, after placing prompt points on the fiber, the model can identify the fiber structure and segment the specified fiber according to the prompt points, and can continue to place prompt points to optimize the segmentation result.

[0064] Please refer to Figure 7 , which is an example diagram of an interactive program applying a three-dimensional interactive segmentation method of this application. Based on the Napari visualization program, an interactive program equipped with this interactive segmentation model is developed. The user inputs an image, rotates and drags the mouse to observe the image, and places positive and negative sample points, and runs the model inference through a button to obtain the segmentation result and save it. Specifically, it includes:

[0065] Image import: Support the loading of three-dimensional images in.tiff format and render them as three-dimensional images in real time.

[0066] Interactive annotation: The user places positive and negative sample points in the three-dimensional view by clicking the mouse, and supports the operations of undoing or redoing.

[0067] Interactive segmentation model inference: After clicking the "Run" button, the model outputs the segmentation mask in real time, which is superimposed and displayed on the original image.

[0068] Result export: The segmentation mask can be saved as a TIFF-format image and supports import and observation in the napari application. It is also possible to save only the voxel coordinates as a JSON file for reading through a text editor.

[0069] Example of the operation process: The user imports a set of axon fiber images and clicks 2 positive sample points on the upper and lower parts of the target fiber. After the first inference of the model, the fiber backbone is segmented, but it is discontinuous in the middle, and the mask also segments parts that are not the current fiber. The user adds 1 positive sample point in the middle of the fiber and 2 negative sample points in the non-fiber parts where the segmentation mask prediction is incorrect. After the second inference of the model, the target fiber is basically completely segmented, and the user can place positive sample points again to prompt the model to make a more accurate prediction. Finally, the mask is exported and used for connection map analysis. In summary, Example 1 of the present application is based on deep learning, integrates interactive prompt information and three-dimensional fluorescence microscopy images, and realizes the accurate segmentation of neuron structures. Specifically, by constructing a three-dimensional interactive segmentation model, the three-dimensional spatial structure information of neurons is fully utilized. Among them, by simulating user behavior, an end-to-end training process is realized, and the segmentation performance of the model is improved. At the same time, the present application provides a powerful tool for neuroscience research, efficiently processes a large amount of neuron image data, improves the accuracy and efficiency of the annotation task, and is applicable to different types of neuron image scenarios.

[0070] Example 2

[0071] Please refer to Figure 8 , which is a schematic structural diagram of a three-dimensional interactive segmentation device according to Example 2 of the present application; the specific content includes:

[0072] Acquisition module: Acquire three-dimensional fluorescence microscopy images and interactive prompt information; wherein, the interactive prompt information includes positive sample points, negative sample points, and the previous segmentation mask.

[0073] Model processing module: Input the three-dimensional fluorescence microscopy image and the interactive prompt information into the constructed interactive segmentation model, and output the segmentation result.

[0074] Iterative optimization module: Based on the user's feedback on the segmentation result, iteratively optimize the interactive segmentation model until the segmentation result meets the preset accuracy.

[0075] Example 3

[0076] Please refer to Figure 9, which is a schematic structural diagram of the computer device according to Embodiment 3 of the present application. The computer device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0077] The memory 52 stores program instructions for implementing the above-mentioned three-dimensional interactive segmentation method.

[0078] The processor 51 is configured to execute the program instructions stored in the memory 52 to implement a three-dimensional interactive segmentation.

[0079] Among them, the processor 51 can also be referred to as a CPU (Central Processing Unit, central processing unit).

[0080] The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0081] Embodiment 4

[0082] Please refer to Figure 10 , which is a schematic structural diagram of the storage medium according to Embodiment 4 of the present application. The storage medium of the embodiment of the present application stores a program file 61 capable of implementing all the above methods. Among them, the program file 61 can be stored in the above storage medium in the form of a software product, including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods according to various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc that can store program codes, or devices such as a computer, a server, a mobile phone, a tablet, etc.

[0083] It should be noted that in this article, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, apparatus, article or method comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, apparatus, article or method. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, apparatus, article or method comprising that element.

[0084] The above are only the preferred embodiments of the present application, and do not limit the patent scope of the present application accordingly. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present application.

[0085] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principle and spirit of the present application. The scope of the present application is defined by the appended claims and their equivalents.

[0086] Certainly, the present invention can also have other various implementation manners. Based on this implementation manner, other implementation manners obtained by those of ordinary skill in the art without any creative work belong to the scope protected by the present invention.

Claims

1. A three-dimensional interactive segmentation method, characterized in that: include: Acquire a three-dimensional fluorescence microscopy image and interactive prompt information; wherein the interactive prompt information includes positive sample points, negative sample points, and a preceding segmentation mask; Inputting the three-dimensional fluorescence microscopic image and the interactive prompt information into the constructed interactive segmentation model, and outputting the segmentation result; Based on user feedback on the segmentation result, the interactive segmentation model is iteratively optimized until the segmentation result meets a preset accuracy.

2. A three-dimensional interactive segmentation method according to claim 1, characterized in that: The step of inputting the three-dimensional fluorescence microscopic image and the interactive prompt information into the constructed interactive segmentation model and outputting the segmentation result specifically includes the following steps: In the interactive segmentation model, a three-dimensional U-Net structure is used as the backbone network, the hierarchical feature channels of the U-Net are 32, 64 and 128 respectively, the convolution kernel size is 3×3×3, the intermediate layer activation function is ReLU, and the output layer activation function is Sigmoid; The positive sample points, negative sample points and previous segmentation masks are represented as three-dimensional grayscale images and fused in the channel dimension; features are extracted through a three-dimensional convolutional layer to obtain a first feature map; Extracting features of the three-dimensional fluorescence microscopy image through a three-dimensional convolutional layer to obtain a second feature map; The first feature map and the second feature map are added and input into the backbone network, and the predicted segmentation result is output.

3. A three-dimensional interactive segmentation method according to claim 2, characterized in that: The training process of the interactive segmentation model includes: The neuron data annotated by the brightest path tracing algorithm of the A* optimal path algorithm is used as a training set, the neuron fragments in the training set are instance annotated data, and the annotation results are verified by the consistency of the neuron centerline; Perform end-to-end training by simulating user behavior through random sampling; A combined loss function is used for training, and back propagation is used to update the parameters of the interactive segmentation model.

4. A three-dimensional interactive segmentation method according to claim 3, characterized in that: The step of performing end-to-end training by simulating user behavior through random sampling specifically includes the following steps: Compare the predicted mask output by the interactive segmentation model with the true value mask to divide the true positive area, false positive area and false negative area; The true positive region is a region that is predicted correctly; the false positive region is a set of pixels that are 1 in the prediction mask but 0 in the true value mask, that is, a region that is predicted correctly but actually wrong; the false negative region is a set of pixels that are 0 in the prediction mask but 1 in the true value mask, that is, a region that is predicted wrongly but actually correct; Sampling negative sample points from the false positive area, and sampling positive sample points from the false negative area; In the first training, the initial segmentation mask was set to blank, and the parameters of the interactive segmentation model were updated based on the sampling points as prompt information combined with the 3D fluorescence microscopy images.

5. A three-dimensional interactive segmentation method according to claim 3, characterized in that: The combined loss function is the weighted sum of Dice Loss and Focal Loss, and the specific expression is: Among them, y is the true value mask; is the prediction mask; γ is a hyperparameter; 6. A three-dimensional interactive segmentation method according to claim 1, characterized in that: The positive sample points represent the position information of the target object, and the negative sample points represent the position information of the non-target area and the interference items.

7. A three-dimensional interactive segmentation method according to claim 1, characterized in that: Deploy the three-dimensional interactive segmentation method through a visual interactive application; The application is used for importing, rotating, scaling, labeling positive and negative sample points, model reasoning, and saving segmentation results of three-dimensional fluorescence microscopy images; The application is developed based on the Napari open source application.

8. A three-dimensional interactive segmentation device, characterized in that: include: Acquisition module: acquiring a three-dimensional fluorescence microscopic image and interactive prompt information; wherein the interactive prompt information includes positive sample points, negative sample points and a preceding segmentation mask; Model processing module: inputting the three-dimensional fluorescence microscopic image and the interactive prompt information into the constructed interactive segmentation model, and outputting the segmentation result; Iterative optimization module: based on the user's feedback on the segmentation result, iteratively optimize the interactive segmentation model until the segmentation result meets the preset accuracy.

9. A computer device, characterized in that: The computer device includes a processor and a memory coupled to the processor, wherein the memory stores program instructions for implementing a three-dimensional interactive segmentation method as described in any one of claims 1-7; and the processor is used to execute the program instructions stored in the memory to implement a three-dimensional interactive segmentation.

10. A storage medium, characterized in that: Program instructions executable by a processor are stored, and the program instructions are used to execute a three-dimensional interactive segmentation method as described in any one of claims 1-7.

Images