Network for dense instance segmentation supervised by physics simulation

Abstract

The application discloses a network dense instance segmentation method based on physical simulation supervision, and comprises the following steps: acquiring a bacterial image dataset, dividing the bacterial image dataset into a training set and a test set; constructing an instance segmentation model; training the instance segmentation model by using the training set, obtaining a trained instance segmentation model; based on the trained instance segmentation model, performing instance segmentation on the bacterial image by using the test set, and obtaining an instance segmentation result. The application realizes accurate segmentation of densely distributed cell nuclei in biological images by combining physical simulation and deep learning technology.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and biomedical image processing, and particularly relates to a network pair dense instance segmentation method based on physical simulation supervision. BACKGROUND

[0002] In pathology and biology research, cell nucleus instance segmentation is a crucial task. However, due to the dense distribution and severe overlap of cell nuclei, accurate labeling and automatic identification of each cell nucleus instance face great challenges. Existing common methods include two-stage and end-to-end frameworks. The former, such as the classic watershed operator and convex hull detection operator, is simple and easy to use, but has obvious cross-domain limitations for pathological images with complex morphology and varied styles. End-to-end methods such as cellpose and hovernet achieve better segmentation accuracy based on gradient map regression, but are limited to single-pixel-level perception and cannot globally model the entire target, so the separation ability of a single target is still challenged. SUMMARY

[0003] To solve the above technical problems, the present application provides a network pair dense instance segmentation method based on physical simulation supervision, which realizes accurate segmentation of densely distributed cell nuclei in biological images by combining physical simulation and deep learning technology.

[0004] To achieve the above purpose, the present application provides a network pair dense instance segmentation method based on physical simulation supervision, comprising:

[0005] Obtain a bacterial image dataset, and divide the bacterial image dataset into a training set and a test set;

[0006] Construct an instance segmentation model;

[0007] Train the instance segmentation model using the training set to obtain a trained instance segmentation model;

[0008] Based on the trained instance segmentation model, perform instance segmentation on the bacterial image using the test set to obtain an instance segmentation result.

[0009] Optionally, the instance segmentation model comprises a neural network model of a semantic segmentation branch and a physical simulation branch, wherein the semantic segmentation branch and the physical simulation branch share the same encoder, the encoder adopts Unet network coding, and the semantic segmentation branch and the physical simulation branch adopt Unet network decoding.

[0010] Optionally, the process of constructing the instance segmentation model comprises:

[0011] The semantic segmentation branch receives a bacterial image dataset, generates a corresponding semantic segmentation mask, and the semantic segmentation mask includes foreground pixels and background pixels.

[0012] The physical simulation branch receives the semantic segmentation mask, converts the semantic segmentation mask into a virtual elastic material, applies a force on the virtual elastic material, supervises the output of the physical simulation branch according to a simulation equation loss derived from an equation of elastic mechanics, and generates a displacement map.

[0013] The watershed method is used to infer an instance region from the displacement map, and instance segmentation is completed.

[0014] Optionally, the equation of elastic mechanics is Ka=F.

[0015] Wherein, K is a stiffness matrix, F is an applied external force, and a is a node displacement.

[0016] Optionally, the equation of elastic mechanics is used to describe the displacement field generated after applying an external force to the virtual elastic material under fixed boundary conditions.

[0017] Optionally, the displacement map includes multiple displacement components, each component corresponding to the displacement of a different layer node in the virtual material, and each component is represented in the x, y and z directions.

[0018] Optionally, the construction process of the training set includes:

[0019] According to the bacterial image dataset, the foreground region and the background region of the image are identified, and three layers are expanded along the z-axis to obtain a three-dimensional binary image.

[0020] Optionally, identifying the foreground region includes:

[0021] According to the foreground region, the boundary and background of the foreground region are detected, three layers are expanded along the z-axis, and a three-dimensional binary image of the foreground region is generated.

[0022] The technical effect of the present application: the present application discloses a network dense instance segmentation method based on physical simulation supervision, which realizes the accurate segmentation of densely distributed cell nuclei in biological images by combining physical simulation and deep learning technology; the present application introduces the physical field simulation process based on finite elements as supervision into the segmentation network, and gets rid of the dependence of the training process on the instance segmentation label, and at the same time, the modeling ability of the complete target form is obtained, and finally the effect of completing high-precision segmentation at low cost is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which form a part of this application, are intended to provide further understanding of the application and serve to explain the exemplary embodiments of the present application and their descriptions, and do not constitute improper limitations on the present application. In the drawings:

[0024] Figure 1 A flowchart of a method for dense instance segmentation of a network supervised by physical simulation is shown in the embodiments of the present application. DETAILED DESCRIPTION

[0025] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0027] As shown in Figure 1 The network proposed in the present application is composed of an encoder and a decoder. The encoder part adopts the encoder structure of UNet. The decoder part is composed of two parallel branches. The two branches are structurally identical and both adopt the decoder structure of UNet, but the difference lies in the use of different supervision information. The first branch is a semantic segmentation branch, which outputs a two-channel semantic segmentation result, and uses a binary semantic label as supervision. The second branch is a physical simulation branch, which outputs a nine-channel feature map. There is also a displacement mapping unit at the output of this branch, where the binary semantic label is converted into a set of boundary conditions as supervision information, and the output feature map is converted into a deformed three-layer mesh surface. Here, the foreground part in the semantic label corresponds to a pixel region subjected to a uniform vertically downward force, while the background part has no external force. The nodes corresponding to the foreground edge pixels are subjected to a fixed constraint, limiting the displacement to 0. For the output feature map, the pixel values at the same position in each channel are extracted and combined into a nine-dimensional vector. This nine-dimensional vector is the displacement amount of the three nodes corresponding to the position of the three-layer mesh surface in the three-dimensional space. The resultant force between the internal and external forces of the mesh surface in the physical field is calculated as a loss. During inference, the first layer of the mesh surface is extracted as a two-dimensional energy map and subjected to a watershed operation to obtain the instance segmentation result.

[0028] The method for dense instance segmentation of a network supervised by physical simulation provided in the embodiments includes:

[0029] An image dataset of bacteria is obtained, and the image dataset of bacteria is divided into a training set and a test set;

[0030] An instance segmentation model is constructed;

[0031] The training set is used to train the instance segmentation model, and a trained instance segmentation model is obtained;

[0032] Based on the trained instance segmentation model, the test set is used for instance segmentation of bacterial images to obtain an instance segmentation result.

[0033] Specifically, the application constructs a neural network framework comprising a semantic segmentation branch and a physical simulation branch (PS branch);

[0034] In the training phase, the foreground region of the image is converted into a virtual elastic material through the semantic segmentation branch, and a force is applied to the virtual material to generate an elastic mechanics equation describing the relationship between the force and the displacement;

[0035] The physical simulation branch generates a displacement map according to the elastic mechanics equation, and the displacement map is composed of the displacement of the nodes and reflects the deformation of the object inside;

[0036] The output of the physical simulation branch is supervised by minimizing the simulation equation loss (SES loss) derived from the elastic mechanics equation to ensure that the generated displacement map is consistent with the true physical deformation;

[0037] In the prediction phase, the generated displacement map and the watershed method are used to infer the instance region from the displacement map to realize dense instance segmentation;

[0038] The semantic segmentation branch and the physical simulation branch share the same encoder and only need to distinguish the pixel labels of the foreground and the background.

[0039] The elastic mechanics equation is used to describe the displacement field generated after applying an external force to the virtual elastic material under fixed boundary conditions.

[0040] The displacement map comprises a plurality of displacement components, each component corresponding to the displacement of a different layer node in the virtual material, and being represented in the x, y and z directions respectively.

[0041] Further, the instance segmentation model comprises a neural network model of the semantic segmentation branch and the physical simulation branch, wherein the semantic segmentation branch and the physical simulation branch share the same encoder, the encoder adopts Unet network encoding, and the semantic segmentation branch and the physical simulation branch adopt Unet network decoding.

[0042] Further, the process of constructing the instance segmentation model comprises:

[0043] The semantic segmentation branch receives a bacterial image dataset to generate a corresponding semantic segmentation mask, and the semantic segmentation mask comprises foreground pixels and background pixels.

[0044] The physical simulation branch receives the semantic segmentation mask, converts the semantic segmentation mask into a virtual elastic material, applies a force on the virtual elastic material, supervises the output of the physical simulation branch according to a simulation equation loss derived from an equation of elastic mechanics, and generates a displacement map;

[0045] The watershed method is used to infer an instance region from the displacement map, and instance segmentation is completed.

[0046] Further, the equation of elastic mechanics is Ka=F.

[0047] Wherein, K is a stiffness matrix; F is an applied external force; and a is the displacement of a node.

[0048] Further, the equation of elastic mechanics is used to describe the displacement field generated after an external force is applied to the virtual elastic material under fixed boundary conditions.

[0049] Further, the displacement map includes a plurality of displacement components, each component corresponding to the displacement of a different layer node in the virtual material, and each component is represented in the x, y and z directions.

[0050] Further, the construction process of the training set includes:

[0051] According to the bacterial image data set, the foreground region and the background region are identified, and three layers are expanded along the z-axis to obtain a three-dimensional binary graph.

[0052] Further, the foreground region includes:

[0053] According to the foreground region, the boundary and background of the foreground region are detected, three layers are expanded along the z-axis, and a three-dimensional binary graph of the foreground region is generated.

[0054] As shown in Figure 1 The simulation equation loss (SES loss) is constructed by considering the balance of the system stiffness matrix, internal force, external force and residual force to ensure that the output of the physical simulation branch is close to the true displacement field derived from the equation of elastic mechanics. The SES loss directly uses the physical equation corresponding to the deformation as supervision, instead of the traditional pixel-level regression. This loss function can more accurately reflect the results of physical simulation and help improve segmentation accuracy.

[0055] The present application only uses semantic segmentation annotations during training, reducing the dependence on dense instance labels, thereby reducing the labeling cost and improving the applicability of the method.

[0056] The present application is applicable to the field of biomedical image processing, particularly to the task of nuclear instance segmentation, and can accurately segment small target objects arranged densely.

[0057] The present application provides a specific application example, in which a bacterial semantic segmentation dataset containing 1402 images is used for training and validation. During the training process, the semantic segmentation branch and the PS branch are jointly optimized to minimize the total loss function, which includes the following steps:

[0058] Step 1, constructing the neural network framework: construct a neural network framework containing semantic segmentation branch and physical simulation branch (PS branch). The two branches share the same encoder, but have different outputs and processing procedures. The encoder uses the Unet network encoding part. The semantic segmentation branch and the physical simulation branch also use the Unet decoding part.

[0059] Step 2, processing of semantic segmentation branch: the semantic segmentation branch receives the input image and generates the corresponding semantic segmentation mask. These masks distinguish foreground and background pixels, providing a basis for subsequent physical simulation.

[0060] Step 3, processing of physical simulation branch: the physical simulation branch receives the semantic segmentation mask as input and converts it into a virtual elastic material. Apply force on the virtual material, causing elastic deformation, and then generate displacement map. This displacement map describes the deformation of each pixel relative to its corresponding finite element before force is applied. In this embodiment, each pixel corresponds to a finite element.

[0061] Step 4, generation and use of displacement map: in the prediction stage, according to the correspondence between displacement map and foreground pixels, assign displacement node values to foreground regions. Generate displacement map, in which pixel values decrease with the deviation from the instance center.

[0062] Step 5, instance segmentation: use the watershed method to infer the instance region from the displacement map, realizing instance-level segmentation.

[0063] In the completed implementation, constructing the training set is a key link.

[0064] In the process of constructing the training set, for a given image, first identify its foreground and background regions, and expand three layers along the z-axis to obtain a three-dimensional binary image. For the foreground region, detect its boundary and background, and then expand three layers along the z-axis to generate a three-dimensional binary image. Each voxel in the three-dimensional binary image is regarded as a finite element, and the voxel node is regarded as a finite element node. In this way, the binary image is converted into a linear elastic material, and the displacement field of the voxel node is obtained by applying force to supervise the branch structure of the physical simulation. Specifically, among all the finite elements, the finite element nodes corresponding to the image background and the foreground region boundary are fixed, the finite element nodes corresponding to the interior of the foreground region are subjected to a force along the Z direction, and the Young's modulus and Poisson's ratio required for calculating the linear elastic model are given. Then according to the balance between the external stress and the internal stress suffered by the node, the following equation Ka=F is satisfied. Where K is the stiffness matrix, F is the external force applied, and a is the displacement of the node. Because the model is for linear elastic material, the node is fixed during the movement process, so a and F are linearly related.

[0065] The application discloses a network dense instance segmentation method based on physical simulation supervision, which realizes accurate segmentation of densely distributed cell nuclei in biological images by combining physical simulation and deep learning technology; the application introduces a physical field simulation process based on finite elements into the segmentation network as supervision, breaks away from the dependence of the training process on instance segmentation labels, and simultaneously obtains modeling capability for complete target morphology, so that the effect of completing high-precision segmentation at low cost is finally achieved.

[0066] The above is only a preferred specific embodiment of the application, but the protection scope of the application is not limited to this, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the application, which should be covered in the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A network-based dense instance segmentation method based on physical simulation supervision, characterized by: include: Obtaining a bacterial image dataset, and dividing the bacterial image dataset into a training set and a test set; Build an instance segmentation model; Using the training set to train the instance segmentation model to obtain a trained instance segmentation model; Based on the trained instance segmentation model, the test set is used to perform instance segmentation of the bacterial image to obtain an instance segmentation result; The instance segmentation model includes: a neural network model of a semantic segmentation branch and a physical simulation branch, wherein the semantic segmentation branch and the physical simulation branch share the same encoder, the encoder adopts Unet network encoding, and the semantic segmentation branch and the physical simulation branch adopt Unet network decoding; The process of building an instance segmentation model includes: The semantic segmentation branch receives a bacterial image dataset and generates a corresponding semantic segmentation mask, where the semantic segmentation mask includes foreground pixels and background pixels; The physical simulation branch receives the semantic segmentation mask, converts the semantic segmentation mask into a virtual elastic material, applies a force to the virtual elastic material, and supervises the output of the physical simulation branch according to the simulation equation loss derived from the elastic mechanics equation to generate a displacement map; Using the watershed method to infer the instance region from the displacement map, and completing instance segmentation; The elastic mechanics equation is Ka=F; Where K is the stiffness matrix; F is the applied external force; a is the displacement of the node; The elastic mechanics equation is used to describe the displacement field generated when an external force is applied to a virtual elastic material under fixed boundary conditions.

2. The method for dense instance segmentation based on network based on physical simulation supervision according to claim 1, characterized in that The displacement map includes multiple displacement components, each component corresponds to the displacement of a node in a different layer of the virtual material, and is represented in the x, y, and z directions respectively.

3. The network-based dense instance segmentation method based on physical simulation supervision according to claim 1, characterized in that The construction process of the training set includes: According to the bacterial image dataset, the foreground and background areas of the image are identified, and three layers are expanded along the z-axis to obtain a three-dimensional binary image.

4. The method for dense instance segmentation based on network based on physical simulation supervision according to claim 3, characterized in that Identify foreground areas including: According to the foreground area, the boundary and background of the foreground area are detected, and three layers are expanded along the z-axis to generate a three-dimensional binary image of the foreground area.

Images