Segmentation method and training method of cascade segmentation network and readable storage medium

Abstract

The invention provides a segmentation method of a cascade segmentation network, a training method and a readable storage medium. The segmentation method comprises the steps of obtaining a first annotation image; segmenting the training image by using a first segmentation network to obtain a first segmentation result; determining a segmentation mask from the first segmentation result based on the first annotation image; removing the segmentation mask from the first annotation image to obtain a second annotation image; determining a first loss value according to the plurality of first annotation images and the plurality of corresponding first segmentation results; segmenting the training image by using a second segmentation network to obtain a second segmentation result; determining a second loss value according to the plurality of second annotation images and the plurality of corrected second segmentation results; determining a total loss value according to the first loss value and the second loss value; and determining a loss function according to the total loss value, and training the cascade segmentation network according to the loss function to obtain the trained cascade segmentation network, thereby improving the training efficiency and segmentation precision of the cascade segmentation network.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing, and in particular, relates to a segmentation method, a training method and a readable storage medium of a cascade segmentation network. Background Art

[0002] By segmenting medical images, the anatomical structure of the human body can be reconstructed. Therefore, it plays a vital role in image analysis and disease diagnosis. It is not only often used as a pre-processing for medical image analysis, but also can be used by doctors to diagnose and formulate treatment plans for patients. It is of great significance for early detection and treatment of diseases. With the development of technology, the diameter of the branches of the anatomical structure of the human body that can be entered by the catheter, such as the bronchi or blood vessels, is getting smaller and smaller, so the requirements for segmentation are getting higher and higher.

[0003] The most mainstream segmentation method currently is the deep learning method. However, due to the imbalanced distribution inside the bronchi, that is, the diameter of the tracheal tree becomes smaller and smaller as the number of levels of the tree structure increases, resulting in the small diameter portion accounting for a very small proportion of the overall bronchial volume. Therefore, when using conventional deep learning methods, it is generally found that small bronchi are missing in the segmentation results of the tracheal tree. In the prior art, this problem can be solved by training multiple segmentation networks separately, such as one segmentation network to segment the entire bronchus and another segmentation network to segment the small bronchi. Although this method can solve the problem caused by the imbalanced distribution inside the bronchi to a certain extent, the process of training multiple different segmentation networks separately is cumbersome and time-consuming.

[0004] Therefore, how to provide an efficient segmentation method is the problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present application provide a segmentation method, a training method and a readable storage medium of a cascade segmentation network, which can improve the training efficiency and segmentation accuracy of the segmentation network.

[0006] In a first aspect, an embodiment of the present application provides a segmentation method of a cascade segmentation network, comprising:

[0007] The cascade segmentation network includes a first segmentation network and a second segmentation network, and the method includes:

[0008] Acquire multiple medical images of anatomical structures as training images;

[0009] Acquire a first annotated image of the training image;

[0010] Segment the training image using a first segmentation network to obtain a first segmentation result;

[0011] Based on the first annotated image, determining a segmentation mask from the first segmentation result, wherein the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image;

[0012] removing the segmentation mask from the first annotated image to obtain a second annotated image;

[0013] determining a first loss value according to the plurality of first annotated images and the plurality of corresponding first segmentation results;

[0014] Segment the training image using a second segmentation network to obtain a second segmentation result;

[0015] removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result;

[0016] determining a second loss value according to the plurality of second annotated images and the plurality of corresponding corrected second segmentation results;

[0017] Determining a total loss value according to the first loss value and the second loss value;

[0018] Determining a loss function according to the total loss value, and training the cascade segmentation network according to the loss function to obtain a trained cascade segmentation network;

[0019] Acquire a medical image to be segmented;

[0020] The trained cascade segmentation network is used to segment the medical image to obtain a target segmentation result.

[0021] Optionally, segmenting the training image according to the trained cascade segmentation network to obtain a segmentation result of the training image;

[0022] The first annotated image is updated according to the segmentation result of the training image.

[0023] Optionally, updating the first annotated image according to the segmentation result of the training image includes:

[0024] Merging the segmentation result of the training image and the first annotated image to obtain a merged result;

[0025] Determining a first maximum connected domain in the first annotated image;

[0026] Determine the second largest connected domain in the merging result;

[0027] Determine an additional part of the maximum connected domain according to the first maximum connected domain and the second maximum connected domain;

[0028] It is determined whether the tracheal radius and / or tracheal direction of the added part in the maximum connected domain meets the preset requirements. If so, the first annotated image is updated according to the merging result.

[0029] Optionally, determining a bifurcation point in the training image according to the first annotated image;

[0030] Changing the current position of the bifurcation point within a preset position range;

[0031] Perform spline interpolation according to the changed current position to obtain an enhanced image;

[0032] The enhanced image is used as a training image.

[0033] Optionally, the target segmentation result includes multiple main bronchi and multiple broken bronchi, and the method further includes:

[0034] determining a first endpoint of the subject's bronchus;

[0035] Determine a second endpoint of a broken bronchus that matches the main bronchus from the multiple broken bronchus, wherein the matched broken bronchus and the main bronchus belong to the same tracheal branch;

[0036] Determine a splicing point according to the first endpoint and the second endpoint, wherein the splicing point is used to splice the main bronchus and the broken bronchus;

[0037] determining a first trachea radius at the first endpoint;

[0038] determining a second trachea radius at the second end point;

[0039] Determine the trachea radius at the splicing point according to the splicing point, the first trachea radius and the second trachea radius;

[0040] Based on the splicing point and the trachea radius at the splicing point, the main bronchus and the matching broken bronchus are spliced.

[0041] Optionally, the centerline of the main bronchus includes a plurality of first skeleton points, the centerline of the matched broken bronchus includes a plurality of second skeleton points, and determining the splicing point according to the first endpoint and the second endpoint includes:

[0042] Acquire a plurality of first skeleton points from the first endpoint along the centerline of the subject's bronchus;

[0043] Acquire a plurality of second skeleton points from the second endpoint along the center line of the broken bronchus;

[0044] Determine a fitting function using a least square method according to the obtained multiple first skeleton points and the obtained multiple second skeleton points, wherein the fitting function is used to characterize a center line of a spliced ​​bronchus of the main bronchus and the matched broken bronchus;

[0045] The splicing point is determined according to the first endpoint, the second endpoint and the fitting function.

[0046] Optionally, determining the first trachea radius at the first endpoint includes:

[0047] Determine the direction of the first trachea at the first end point;

[0048] Determine a first trachea cross section perpendicular to the direction of movement of the first trachea;

[0049] determining a first center of the first tracheal cross-section;

[0050] Establishing a pixel space coordinate system with the first center as the origin, wherein the pixel space includes multiple layers, and each layer includes multiple pixels;

[0051] Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer;

[0052] Using the radius corresponding to the outermost trachea layer that reaches the preset ratio as the first trachea radius at the first end point;

[0053] or,

[0054] Determine the direction of the first trachea at the first end point;

[0055] Determine a first trachea cross section perpendicular to the direction of movement of the first trachea;

[0056] determining a first centroid of the first tracheal cross section;

[0057] respectively determining a plurality of first distances between a plurality of boundary points of the first trachea cross section and the first centroid;

[0058] A first average value of the plurality of first distances is determined, and the first average value is used as a first trachea radius at the first endpoint.

[0059] Optionally, before determining the first average value of the first distance, the method further includes:

[0060] determining a first mean and a first variance according to the plurality of first distances;

[0061] Determine a first concentrated distribution range of the plurality of first distances according to the first mean and the first variance;

[0062] The first distance within the distribution range of the first set is screened.

[0063] Optionally, determining the second trachea radius at the second endpoint includes:

[0064] Determining the direction of the second air pipe at the second end point;

[0065] Determine a second trachea cross section perpendicular to the direction of movement of the second trachea;

[0066] determining a second center of the second tracheal cross-section;

[0067] Establishing a pixel space coordinate system with the second center as the origin, wherein the pixel space includes multiple layers, and each layer includes multiple pixels;

[0068] Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer;

[0069] Using the radius corresponding to the outermost trachea layer that reaches the preset ratio as the second trachea radius at the second end point;

[0070] or,

[0071] Determining the direction of the second air pipe at the second end point;

[0072] Determine a second trachea cross section perpendicular to the direction of movement of the second trachea;

[0073] determining a second centroid of the second tracheal cross section;

[0074] respectively determining a plurality of second distances between a plurality of boundary points of the second trachea cross section and the second centroid;

[0075] A second average value of the plurality of second distances is determined, and the second average value is used as the second trachea radius at the second endpoint.

[0076] Optionally, splicing the main bronchus and the matching broken bronchus based on the splicing point and the trachea radius at the splicing point includes:

[0077] Based on the splicing point and the trachea radius at the splicing point, setting a voxel screening range;

[0078] Within the voxel screening range, selecting voxels whose distance from the splicing point does not exceed the trachea radius at the splicing point;

[0079] The selected voxels are marked as spliced ​​bronchus to splice the main bronchus and the matched broken bronchus.

[0080] Optionally, the centerline of the main bronchus includes a plurality of first skeleton points, and the step of determining the second endpoint of the broken bronchus matching the main bronchus includes:

[0081] Determine endpoints of the multiple broken bronchi, and use the endpoints of the multiple broken bronchi as first candidate endpoints;

[0082] Calculating a third distance between the plurality of first candidate endpoints and the first endpoint;

[0083] Determine a first candidate endpoint whose third distance is within a preset distance threshold as a second candidate endpoint;

[0084] Acquire a plurality of first skeleton points from the first endpoint along the centerline of the subject's bronchus;

[0085] Acquire a plurality of second candidate skeleton points along the center line of the broken bronchus from the second candidate endpoint;

[0086] Randomly select two first skeleton points to form a first vector, randomly select two second candidate skeleton points to form a second vector, and randomly select one first skeleton point and one second candidate skeleton point to form a third vector;

[0087] Determine whether the angle between the first vector and the second vector is less than a first angle threshold, and whether the angle between the first vector and the third vector is less than a second angle threshold; or, determine whether the angle between the first vector and the second vector is less than a first angle threshold, and whether the angle between the second vector and the third vector is less than a third angle threshold;

[0088] If both are true, the second candidate endpoint is used as the second endpoint.

[0089] In a second aspect, an embodiment of the present application provides a segmentation method of a cascade segmentation network, comprising:

[0090] determining a first endpoint of the subject's bronchus;

[0091] Determining a second end point of the broken bronchus that matches the main bronchus from a plurality of broken bronchus;

[0092] Determine a splicing point according to the first endpoint and the second endpoint, wherein the splicing point is used to splice the main bronchus and the matching broken bronchus;

[0093] determining a first trachea radius at the first endpoint;

[0094] determining a second trachea radius at the second end point;

[0095] Determine the trachea radius at the splicing point according to the splicing point, the first trachea radius and the second trachea radius;

[0096] splicing the main bronchus and the matching broken bronchus based on the splicing point and the trachea radius at the splicing point;

[0097] Among them, the main bronchus and the matching broken bronchus belong to the same tracheal branch.

[0098] In a third aspect, an embodiment of the present application provides a training method for a cascade segmentation network, wherein the cascade segmentation network includes a first segmentation network and a second segmentation network, and the method includes:

[0099] Acquire multiple medical images of anatomical structures as training images;

[0100] Acquire a first annotated image of the training image;

[0101] Segment the training image using a first segmentation network to obtain a first segmentation result;

[0102] Based on the first annotated image, determining a segmentation mask from the first segmentation result, wherein the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image;

[0103] removing the segmentation mask from the first annotated image to obtain a second annotated image;

[0104] Determining a first loss value according to the first annotated image and the first segmentation result;

[0105] Segment the training image using a second segmentation network to obtain a second segmentation result;

[0106] removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result;

[0107] determining a second loss value according to the second annotated image and the corrected second segmentation result;

[0108] Determining a total loss value according to the first loss value and the second loss value;

[0109] A loss function is determined according to the total loss value, and the cascade segmentation network is trained according to the loss function to obtain a trained cascade segmentation network.

[0110] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which may store a program, which, when executed by a processor, implements part or all of the steps of the segmentation method of the cascade segmentation network of the first and second aspects or the training method of the third aspect.

[0111] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed, executes part or all of the steps of the segmentation method of the cascade segmentation network of any one of the first and second aspects or the training method of the third aspect.

[0112] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0113] The beneficial effect of the embodiment of the present application compared with the prior art is: the embodiment of the present application trains the first segmentation network and the second segmentation network uniformly, and removes the anatomical structure that has been correctly segmented by the first segmentation network when training the second segmentation network, thereby more accurately reflecting the segmentation result of the second segmentation network, which is beneficial to improving the training efficiency and segmentation accuracy of the cascade segmentation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0114] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0115] Figure 1 It is a flowchart of a segmentation method of a cascade segmentation network provided in an embodiment of the present application;

[0116] Figure 2 is a flow chart of a segmentation method of a cascade segmentation network according to another embodiment of the present application;

[0117] Figure 3A is a schematic diagram before spline interpolation according to an embodiment of the present application;

[0118] Figure 3B is a schematic diagram after spline interpolation in one embodiment of the present application;

[0119] Figure 4 This is a pixel statistics diagram of an embodiment of the present application;

[0120] Figure 5 This is a schematic diagram of angle criterion judgment according to an embodiment of the present application;

[0121] Figure 6is a schematic diagram of angle criterion judgment according to another embodiment of the present application;

[0122] Figure 7 Schematic diagram of the principle of a control device according to an embodiment of the present application. DETAILED DESCRIPTION

[0123] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0124] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0125] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0126] The training of the cascade segmentation network provided in the embodiment of the present application may include the following steps: 1. data preparation, 2. model structure design, 3. loss function design, and 4. iterative training.

[0127] Among them, the data in step 1 is sample data, including medical images used for training and manually annotated label files. Data preparation includes collecting these medical images, and optionally, preprocessing can be performed. Preprocessing generally includes normalizing the image, expanding the number of samples, etc. The label file is obtained after annotating the medical image. The label file reflects the expected segmentation result of the medical image, so the label file is recorded as the segmentation expectation or segmentation target Y. After the medical image and the label file are preprocessed and the model is trained, the segmentation result X of the medical image will be output. The label file is usually saved in the form of an image, so the label file in the embodiment of the present application is also called an annotated image.

[0128] Step 2 is the structural design of a specific neural network model, such as the number of layers and the type of each layer.

[0129] Step 3 is the design of the loss function to accurately calculate the gap between the segmentation result X and the segmentation target Y.

[0130] Step 4 is to input the medical image into the cascade segmentation network, calculate the loss function according to the output segmentation result X and segmentation target Y of the cascade segmentation network, and then adjust and optimize the parameters of the cascade segmentation network according to the loss function. Repeat the above process to iteratively train the cascade segmentation network until the condition for stopping training is met.

[0131] like Figure 1 As shown, a flow chart of a segmentation method of a cascade segmentation network provided by an embodiment of the present application, wherein the cascade segmentation network includes a first segmentation network and a second segmentation network, and the method includes:

[0132] Step S10: Acquire multiple medical images of anatomical structures as training images;

[0133] Usually, multiple medical images are obtained as training images for training the cascade segmentation network. Among them, medical images include medical image pictures collected by CT scan, MRI, ultrasound, etc., which are generally presented as grayscale images without color information. Medical images can be images of various anatomical structures of the human body, for example, anatomical structures can be natural tubes such as trachea, blood vessels, ureters or intestines. The medical images used for training in the embodiments of the present application are also referred to as training images.

[0134] Step S20: Acquire a first annotated image of the training image;

[0135] The first annotated image is an image obtained after annotating the training image. The first annotated image reflects the expected segmentation target of the first segmentation network for the training image. The first annotated image is a form of expression of the segmentation target of the first segmentation network. In the embodiment of the present application, the segmentation target of the first segmentation network can be referred to as the first segmentation target. In the embodiment of the present application, the first annotated image is used to represent the first segmentation target.

[0136] Optionally, the annotated image is obtained by manually annotating the medical image according to the manual annotation results. Among them, manual annotation is to manually perform some annotations on the medical image, such as marking in red, drawing circles, drawing dots, etc., to mark the segmentation object. Taking bronchial segmentation as an example, the segmentation object is the bronchus. In the lung CT image, the location of the bronchus is manually marked. Among them, the bronchus generally has hole characteristics. The manual annotation method may include selecting a point in the bronchus, drawing circles / boxes and other patterns in the bronchus, and framing the bronchus with broken lines / curves. Since the manually given annotation only indicates the location of the bronchus, it is often impossible to accurately give the range of each bronchus. Therefore, the medical image is segmented according to the manual annotation results to obtain the annotated image. The segmentation refers to the binary segmentation of the medical image. The segmentation result can be represented by a binary image (i.e., annotated image), the size of which is the same as that of the medical image, wherein the pixel value of each pixel is used to indicate whether the corresponding pixel in the medical image belongs to the segmentation object. If it belongs, the pixel is called the foreground, and if it does not belong, the pixel is called the background. The embodiments of the present application are described by taking the foreground pixel value as 1 and the background pixel value as 0. In fact, in other embodiments, the values ​​may be opposite, for example, the foreground pixel value is 0 and the background pixel value is 1, and the embodiments of the present application are not limited to this.

[0137] Furthermore, when displaying, since the pixel value of a grayscale image generally ranges from 0 to 255, rather than simply 0 and 1, in order to intuitively see the annotated image in the image, the pixel point with a pixel value of 1 is generally reassigned to 255 when displayed. It should be noted that the actual value of the pixel value of the annotated image in this method is still 0 or 1. In this application, when displaying a binary image, 1 corresponds to white and 0 corresponds to black; in actual applications, it can also be reversed, that is, 1 corresponds to black and 0 corresponds to white.

[0138] Step S30: Use a first segmentation network to segment the training image to obtain a first segmentation result.

[0139] Step S40: Based on the first annotated image, determine a segmentation mask from the first segmentation result, where the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image.

[0140] The segmentation mask is used to characterize the correct anatomical structure in the first segmentation result, that is, the segmentation mask reflects the correct part of the anatomical structure predicted by the first segmentation network when segmenting the training image. Optionally, by comparing the first segmentation result and the first annotated image, the anatomical structure they both include is found to be the segmentation mask.

[0141] Step S50: removing the segmentation mask from the first annotated image to obtain a second annotated image.

[0142] The second annotated image is an image obtained after annotating the training image. The second annotated image reflects the desired segmentation target of the second segmentation network for the training image. The second annotated image is a representation of the segmentation target of the second segmentation network. In the embodiment of the present application, the segmentation target of the second segmentation network can be referred to as the second segmentation target. In the embodiment of the present application, the second annotated image is used to represent the second segmentation target.

[0143] In the embodiment of the present application, the second annotated image is different from the first annotated image, and the segmentation mask needs to be removed from the first annotated image to obtain the second annotated image, that is, the second segmentation target of the second segmentation network needs to remove the anatomical structure that has been correctly segmented by the first segmentation network, so as to more accurately reflect the segmentation target of the second segmentation network, which is beneficial to network training.

[0144] Step S60: Determine a first loss value according to the first annotated image and the first segmentation result.

[0145] The first loss value is used to reflect the difference between the first segmentation result predicted by the first segmentation network and the first segmentation target. The smaller the first loss value, the closer the predicted segmentation result is to the segmentation target, that is, the closer it is to the truth. The loss function includes a variety of methods, which can be selected according to the situation. For example, the loss function Loss1 = W1'﹡dice_loss1+W2'﹡focal_loss1, where W1', W2' are weight coefficients, which can be flexibly set as long as W1'+W2'=1 is satisfied.

[0146] Optionally, a difference between the first annotated image and the first segmentation result is first determined, and the difference is used as the first loss value.

[0147] Step S70: Use a second segmentation network to segment the training image to obtain a second segmentation result.

[0148] Optionally, during the training process, in order to optimize the segmentation effect of the second segmentation network, the first segmentation result may be input into the second segmentation network.

[0149] Step S80: removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result.

[0150] In order to accurately reflect the segmentation effect of the second segmentation network in the embodiment of the present application, the second segmentation result is corrected. Optionally, the anatomical structure that has been correctly segmented by the first segmentation network is removed from the second segmentation result predicted by the second segmentation network, that is, the segmentation mask is removed from the second segmentation result to obtain a corrected second segmentation result, so that the segmentation result of the second segmentation network can be more accurately reflected, which is beneficial to network training.

[0151] Step S90: Determine a second loss value according to the second annotated image and the corrected second segmentation result.

[0152] The second loss value reflects the difference between the second segmentation result predicted by the second segmentation network and the second segmentation target. The second loss value can be calculated according to the loss function. Similarly, the loss function Loss2 can also be selected according to the situation. For example, Loss2 = W1″﹡dice_loss2+W2″﹡focal_loss2, where W1″, W2″ are weight coefficients, which can be flexibly set as long as W1″+W2″=1 is satisfied.

[0153] Step S100: Determine a total loss value according to the first loss value and the second loss value.

[0154] Step S110: determining a loss function according to the total loss value, and training the cascade segmentation network according to the loss function to obtain a trained cascade segmentation network.

[0155] The training process can be a repeated and continuous training. According to the continuous training, observe the training effect. If you are satisfied, you can stop the training. If you are still not satisfied, you can continue the training. In the embodiment of the present application, this repeated training is also referred to as multiple rounds of training. After a round of training, if you are not satisfied, you can continue to the next round of training. In a round of training, multiple training images are usually trained, and each training image will have its corresponding first / second segmentation result, first / second labeled image, and first / second loss value. Optionally, the total loss value of multiple training images is determined based on the first loss values ​​of multiple training images and the second loss values ​​of multiple training images. Then, it is determined whether to end the training based on the total loss value of each training image. When the training is ended, the loss function corresponding to this round of training is determined.

[0156] For example, in the embodiment of the present application, the first loss value is recorded as Loss1, the second loss value is recorded as Loss2, the loss function is recorded as Total_loss, the first segmentation network is recorded as model1, and the second segmentation network is recorded as model2. Figure 2 FIG. 4 is a schematic diagram of a cascade segmentation network provided in accordance with an embodiment of the present application.

[0157] Total_loss=W1﹡Loss1+W2﹡Loss2

[0158] The weight coefficients of loss1 and loss2 are usually pre-set using empirical values, such as (0.5, 0.5) or (0.6, 0.4) and other combinations. In order to make the segmentation network move in the direction we expect, we first train model1 to basic convergence, and then train model2. This process is completed automatically. For example, the specific steps are as follows:

[0159] Loss1=W1′﹡dice_loss1+W2′﹡focal_loss1

[0160] Loss2=W1″﹡dice_loss2+W2″﹡focal_loss2

[0161] During training, Loss1 and Loss2 can be constructed based on dice_loss1 and focal_loss2. Of course, other methods can also be used, and the present application embodiment does not impose specific restrictions on this. W1′, W2′, W1″ and W2″ can be set according to specific circumstances.

[0162] When the dice_loss1 of model1 is ≥ 0.3, w1 = 1, w2 = 0, which allows model1 to converge quickly, so that model1 mainly completes the segmentation of the bronchial trunk area, and the segmentation target of model2 is the boundary area and small bronchial area that are not well segmented by model1.

[0163] When dice_loss1 of model1 is less than 0.3, w1=w2=0.5, and when model1 is able to complete the initial backbone segmentation, loss2 is calculated into the total Total_loss.

[0164] During the continuous iterative training process, the weights of loss1 and loss2 are dynamically adjusted until the optimal weights w1 and w2 are obtained, thereby finally determining the loss function Total_loss.

[0165] According to the aforementioned steps S10 to S110 , the training of the trained cascade segmentation network can be obtained.

[0166] Step S120: Acquire the medical image to be segmented.

[0167] The medical images to be segmented may include medical images of patients who need to undergo medical operations, wherein the medical operations may include surgery or biopsy, etc. The medical images of patients may include intraoperative images or preoperative images of patients.

[0168] Step S130: Use the trained cascade segmentation network to segment the medical image to be segmented to obtain a target segmentation result. The target segmentation result can reflect the anatomical structure of the patient, and further anatomical structure modeling, registration or surgical navigation can be performed based on the target segmentation result.

[0169] The target segmentation result can have a variety of forms of expression, for example, it can be the probability that each voxel is a bronchus in the voxel space reflecting the anatomical structure; or further identify whether the corresponding voxel is a bronchus based on the probability, for example, voxel 1 is a bronchus, and voxel 0 is not a bronchus; or further include a center line, and only the anatomical structure exists on the center line, for example, the center line of the bronchus includes multiple skeleton points, the skeleton point corresponds to voxel 1, and other points on the non-center line correspond to voxel 0. In the application, the target segmentation results in different expressions can be selected as needed. In the embodiment of the present application, voxels will also be referred to as points.

[0170] The embodiment of the present application trains the first segmentation network and the second segmentation network uniformly, and removes the anatomical structure that has been correctly segmented by the first segmentation network when training the second segmentation network, thereby more accurately reflecting the segmentation result of the second segmentation network, which is beneficial to improving the training efficiency and segmentation accuracy of the cascade segmentation network.

[0171] In one embodiment, the segmentation method of the cascade segmentation network provided in the embodiment of the present application may further include:

[0172] Segmenting the training image according to the trained cascade segmentation network to obtain a segmentation result of the training image;

[0173] The first annotated image is updated according to the segmentation result of the training image.

[0174] Further, updating the first annotated image according to the segmentation result of the training image specifically includes:

[0175] Merging the segmentation result of the training image and the first annotated image to obtain a merged result;

[0176] Determining a first maximum connected domain in the first annotated image;

[0177] Determine the second largest connected domain in the merging result;

[0178] Determine an additional part of the maximum connected domain according to the first maximum connected domain and the second maximum connected domain;

[0179] Determine whether the tracheal radius and / or tracheal direction of the added part in the maximum connected domain meets the preset requirements. If so, update the first annotated image according to the merged result. The tracheal direction refers to the direction in which the trachea extends backward. For example, when determining the tracheal direction at the skeleton point A in the trachea, the skeleton point B adjacent to the skeleton point A can be obtained first, and the direction of the vector formed by AB is used as the tracheal direction at the skeleton point A in the trachea. The skeleton point B can be closely adjacent to the skeleton point A or separated by other skeleton points.

[0180] The preset requirements can be flexibly set as needed, for example, the radius of the added part of the trachea does not exceed the radius of the trachea adjacent to the added part of the trachea, and the direction of the trachea meets the angle criteria of the trachea.

[0181] After the first annotated image is updated, the second annotated image can be updated at the same time, and then new training is performed based on the updated first annotated image and the second annotated image, so as to continuously improve the cascade segmentation network.

[0182] In one embodiment, in order to increase the learning samples and improve the effect of segmentation network training, the segmentation method of the cascade segmentation network provided in the embodiment of the present application may also include: enhancing the medical image. When the original medical image has been annotated, the enhanced medical image also has the corresponding annotation information, so it can also be used as a learning sample for network training. In the embodiment of the present application, the enhanced medical image is also referred to as an enhanced image. Among them, the enhanced processing of the medical image specifically includes:

[0183] Step S101: According to the first annotated image, determine the bifurcation point in the training image. Optionally, traverse each skeleton point on the center line to determine the number of skeleton points within the 26 neighborhoods of each skeleton point. If there are 3 or more, it is a bifurcation point. Other methods in the prior art can also be used to determine the bifurcation point in the training image. The embodiments of the present application do not impose any restrictions on this. For example, reference can be made to 202310026323.7 Catheter robot and its alignment method, and a method for determining a bifurcation point in a readable storage medium.

[0184] Step S102: changing the current position of the bifurcation point within a preset position range.

[0185] When there are multiple bifurcation points, each bifurcation point can change its current position within its preset position range. The preset position ranges of each bifurcation point can be the same or different. The preset position range is the range within which the position of the bifurcation point can be changed. For example, if the preset position range is a range of N voxels, then the bifurcation point can move within the range of N voxels. For example, if N is 10 in the preset position range of a bifurcation point, then the bifurcation point can move up 8 voxels or down 7 voxels or to the left 5 voxels or to the right 3 voxels; if N is 20 in the preset position range of another bifurcation point, then the other bifurcation point can move up 12 voxels or down 14 voxels or to the left 5 voxels or to the right 9 voxels.

[0186] The embodiment of the present application selects the bifurcation point as the control point for position change, which can avoid the distortion of the anatomical structure and the deformation that does not conform to the actual shape.

[0187] Step S103: performing spline interpolation according to the changed current position to obtain an enhanced image.

[0188] The position of the bifurcation point has been changed, so other points besides the bifurcation point, such as the skeleton points between multiple bifurcation points and other points outside the center line, need to be changed, that is, the positions that are not bifurcation points need to be filled, so as to ensure that the anatomical structure in the enhanced image is not deformed or distorted. Optionally, the positions of other points after the change can be determined by spline interpolation, such as Figure 3A Schematic diagram before spline interpolation, as shown Figure 3B This is a schematic diagram after spline interpolation. Figure 3A The cross 301 in the figure indicates a bifurcation point. Figure 3A The circle 302 in the figure indicates the bifurcation point after the position change. Figure 3B The circle 302 with a cross in the figure indicates the bifurcation point after the position change. Figure 3A Point 303 in represents other points except the bifurcation point. Figure 3B Point 304 in the figure represents other points that have changed their positions except for the bifurcation point. The enhanced medical image can be determined based on the changed positions of the bifurcation point and the changed positions of the skeleton points. Compared with the previous medical image, the overall anatomical structure of the enhanced medical image has been deformed, thereby providing more possibilities of anatomical structures. For example, when the anatomical structure is a bronchus, the bronchus is affected by respiratory movement and may undergo various deformations. Therefore, the embodiment of the present application can better simulate bronchi with different deformations by providing more possibilities of bronchus, which is beneficial to improving the segmentation accuracy of the cascade segmentation network.

[0189] Step S104: using the enhanced image as a training image.

[0190] It is understandable that after the training image is enhanced, its corresponding annotated image also needs to be enhanced accordingly, and then the network training is performed based on the enhanced image and the enhanced annotated image. Of course, the network training can also be performed based on the enhanced image and the enhanced annotated image, the training image before the enhancement and the corresponding annotated image.

[0191] The embodiment of the present application obtains an enhanced image by changing the position of the bifurcation point, thereby achieving various deformations of the anatomical structure, and then trains based on the enhanced image to obtain the cascade segmentation network, thereby improving the segmentation accuracy of the cascade segmentation network.

[0192] In one embodiment, the target segmentation result in the segmentation method of the cascade segmentation network provided in the embodiment of the present application includes multiple main bronchi and multiple broken bronchi, multiple main bronchi refer to multiple end bronchi in the relatively coherent and complete part of the segmentation result, and multiple broken bronchi refer to multiple bronchioles in the smaller part of the segmentation result. There are gaps, i.e., breaks, between the multiple main bronchi and the multiple broken bronchi. In order to better achieve the segmentation effect of the target segmentation result, the main bronchi and the broken bronchi need to be spliced ​​together, but the main bronchi and the broken bronchi are not spliced ​​randomly, but the main bronchi and the broken bronchi belonging to the same tracheal branch need to be found and spliced ​​together. In the embodiment of the present application, the main bronchi and the broken bronchi belonging to the same tracheal branch are also referred to as matching main bronchi and matching broken bronchi, that is, each main bronchus may have a matching broken bronchus, and a broken bronchus may have a matching main bronchus.

[0193] The main bronchus and the broken bronchus both include a centerline, the centerline of the main bronchus includes a plurality of first skeleton points, and the centerline of the matched broken bronchus includes a plurality of second skeleton points. The segmentation method may also include splicing the main bronchus and the matched broken bronchus, specifically including:

[0194] Step S150: determining the first endpoint of the subject's bronchus;

[0195] A second endpoint of the severed bronchus is determined that matches the subject bronchus.

[0196] When the target segmentation result includes multiple main bronchi, the endpoints of each main bronchi can be found respectively. The endpoint refers to the outermost skeleton point on the centerline of the main bronchus, and the first endpoint is also the skeleton point closest to the centerline of the matching broken bronchus among the multiple first skeleton points. The second endpoint is the skeleton point closest to the centerline of the matching main bronchus among the multiple second skeleton points.

[0197] Step S1501: Traverse each skeleton point on the center line of the main bronchus and the broken bronchus respectively, and determine the number of skeleton points in its 26 neighborhoods. If there is only one point, it is considered to be an endpoint and recorded; if there are two points, execute step S1502; if there are 3 or more points, no processing is performed.

[0198] Step S1502: When there are two skeleton points in the 26-neighborhood of the skeleton point on the center line, set the skeleton point to point A, set the two adjacent points to B1 and B2 respectively, and perform a depth-first search with point A as the seed point. If the growth result is A-B1 and then to a series of other points, and then jump to A-B2 to another series of points, it means that point A is a normal point on the center line; if a depth-first search is performed with point A as the seed point, if the growth result is A-B1-B2 or A-B2-B1 or A-B2 to a series of other points and then to B1, it means that the current point A is the endpoint, which can be recorded.

[0199] According to steps S1501 and 1502, the endpoints of multiple main bronchi and multiple broken bronchi can be found; after the matching broken bronchi are determined, the endpoints of the matching broken bronchi can also be found. The specific implementation principle is the same and will not be repeated here.

[0200] Optionally, the method for determining the endpoint can also refer to 202310026323.7 catheter robot and its alignment method, and the method for determining the end point in the bifurcation point in the readable storage medium.

[0201] Step S160: determining a splicing point according to the first endpoint and the second endpoint, the splicing point being used to splice the main bronchus and the matching broken bronchus, specifically comprising:

[0202] Step S1601: From the first endpoint, multiple first skeleton points are obtained along the center line of the main bronchus; from the second endpoint, multiple second skeleton points are obtained along the center line of the broken bronchus. The acquisition of the first skeleton point and the second skeleton point can be achieved in a regional growing manner. For example, taking the first endpoint as the seed point, if the pixel value in the 26 neighborhoods is 1 (corresponding to the skeleton point on the center line), one is found to grow once, and it can be stopped when a bifurcation point is encountered or the number of skeleton points meets the requirement; or taking the second endpoint as the seed point, if the pixel value in the 26 neighborhoods is 1 (corresponding to the skeleton point on the center line), one is found to grow once, and it can be stopped when there is no new growable point or the number of skeleton points meets the requirement.

[0203] Step S1602: Determine a fitting function using the least squares method based on the acquired multiple first skeleton points and the acquired multiple second skeleton points, wherein the fitting function is used to characterize the center line of the spliced ​​bronchus of the main bronchus and the matching broken bronchus.

[0204] Step S1603: Determine the splicing point according to the first endpoint, the second endpoint and the fitting function.

[0205] Exemplary: the first skeleton points of the main bronchus include the first endpoint [179,242,536], other first skeletons [179,242,535], [180,242,534], [181,243,533], [182,242,533] and [182,242,532]; the second skeleton points of the broken bronchus include the second endpoint [176,242,543], other second skeleton points [176,242,543], [176,242,544], [175,242,545].

[0207] Sorting: [182,242,532], [182,242,533], [181,243,533], [180,242,534], [179,242,535], [179,242,536], [176,242,543], [176,242,543], [176,242,544], [175,242,545].

[0209] There are 10 points in total, so t is [0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. By using the latest square method to calculate the fitting function x = f(t), y = f(t), z = f(t), the corresponding t value can be obtained according to the positions of the two end points, which is 0.5 and 0.6. Therefore, the points of the unconnected part can be calculated in the range of 0.5-0.6.

[0210] Because the length of the unconnected part is uncertain, in order to ensure that all points are found, it is assumed that there are 30 t in the range of 0.5-0.6, so the t values ​​in this example are: 0.5, 0.504, 0.508, 0.512... 0.58, 0.584, 0.588, 0.592, 0.596. The number of t can be set flexibly, for example, it can be 10 or 50, etc. Substitute t into the fitting function x=f(t), y=f(t), z=f(t), and the corresponding x, y and z can be calculated. For example, the corresponding splicing points are [179,242,536][179,242,535][180,242,534][178,242,537]…[176,242,541][176,242,542][176,242,543][176,242,544].

[0211] Furthermore, if there are duplicate points in the fitting points, the duplicates can be removed to obtain the final splicing points.

[0212] In one embodiment, determining the splicing point according to the first endpoint and the second endpoint may include:

[0213] The first endpoint and the second endpoint are evenly divided and then fine-tuned as the first splicing point, and then the first endpoint and the first splicing point are evenly divided and then fine-tuned as the second splicing point. The second endpoint and the first splicing point are evenly divided and then fine-tuned as the third splicing point. The new splicing point, the first endpoint and the second endpoint are evenly divided and then fine-tuned as the new splicing point. The iteration may be stopped after a certain number of times, wherein the number of iterations may be set according to experience or according to the difference between the first endpoint and the second endpoint.

[0214] Exemplary: the number of iterations is determined according to the positions of the first endpoint and the second endpoint, for example, the average value of the first endpoint and the second endpoint is taken and then rounded to an integer as the number of iterations.

[0215] The position of the splicing point is determined according to the positions of the first endpoint and the second endpoint, for example, the midpoint of the first endpoint and the second endpoint is taken as the splicing point, or the intersection of a line passing through the trachea direction at the first endpoint and a line passing through the trachea direction at the second endpoint is taken as the splicing point.

[0216] The first endpoint and the splicing point form a first connecting line, and the second endpoint and the splicing point form a second connecting line;

[0217] The first connection line is angle-corrected, and the second connection line is angle-corrected. The angles of the first connection line angle correction and the second connection line angle correction can be the same or different, and can usually be set within 30 degrees according to the morphology of the bronchus.

[0218] Determine an intersection point of the corrected first connecting line and the corrected second connecting line, use the intersection point to update the splicing point, and reduce the number of iterations by 1;

[0219] The first endpoint is updated using the updated splicing point, and the step is returned to determine the position of the splicing point according to the positions of the first endpoint and the second endpoint until the number of iterations is 0; or the second endpoint is updated using the updated splicing point, and the step is returned to determine the position of the splicing point according to the positions of the first endpoint and the second endpoint until the number of iterations is 0.

[0220] The first endpoint, the second endpoint and all the splicing points are taken as known endpoints through continuous cycles, and new splicing points are determined according to the aforementioned method until the number of iterations reaches 0, thereby obtaining the splicing point.

[0221] Step S170: Determine a first trachea radius at the first endpoint; determine a second trachea radius at the second endpoint.

[0222] This application embodiment provides two methods for determining the tracheal radius:

[0223] The first method: determining the first trachea radius at the first endpoint comprises:

[0224] Determine the direction of the first trachea at the first end point;

[0225] Determine a first tracheal section perpendicular to the direction of the first tracheal movement. The outermost layer of the first tracheal section includes multiple boundary points. There must be a point (pixel) that is not a bronchus within the 8-neighborhood of the boundary point, such as a point of pixel 0, so the boundary point can be determined by traversing each point on the tracheal section.

[0226] Determine the first center of the first tracheal section. Optionally, the first center is determined according to a K-MEANS algorithm or a priest-villager model, that is, the sum of the distances from the boundary points of the first tracheal section to the first center is minimized.

[0227] A pixel space coordinate system is established with the first center as the origin, wherein the pixel space includes multiple layers, for example, i layers, each layer includes multiple pixels, for example, the number of pixels is 8i, where i is a natural number greater than 0;

[0228] Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer. First, determine whether each pixel is a bronchus based on the target segmentation result. Then calculate whether the ratio of the pixels of the bronchus to all the pixels reaches the preset ratio of the layer. If the preset ratio is reached, the layer can be considered to be a bronchus. If the preset ratio cannot be reached, the layer can be considered not to be a bronchus. The preset ratios of each layer can be the same or different. Usually the ratio is set to be higher than 30%. For example, the number of pixels for the first layer of bronchus is not less than 3, and the number of pixels for the second layer of bronchus is not less than 6.

[0229] The radius corresponding to the outermost tracheal layer that reaches the preset ratio is used as the first tracheal radius at the first end point. The distribution of the pixels of the bronchus in the outermost layer is counted, and the distance from the most concentrated distribution position to the first center is optionally used as the radius. Figure 4 The pixel statistics diagram in the embodiment of the present application is shown. The outermost layer is assumed to be the second layer. The second layer has 16 pixels in total. The pixels of the bronchus with a distance of 2 include 2 pixels. The bronchus pixels include 3 pixels with a distance of The pixels of the bronchus include 2, and since the distance is The bronchus has the largest number of pixels, so we select The radius of the bronchus at is selected as the radius, for example, the radius of the bronchus at [2,1] is selected as the radius.

[0230] Furthermore, the coordinates of the three-dimensional voxel space are determined according to the coordinates of the pixel space coordinate system of the outermost trachea layer. Specifically, the change relationship between the coordinates [x, y, z] of the three-dimensional voxel space and the coordinates [x′, y′] of the pixel space coordinate system is determined according to the coordinates [x0, y0, z0] of the three-dimensional voxel space of the origin, that is:

[0231] xx′=x0

[0232] yy′=y0

[0233] Substituting x and y into the equation for the tracheal cross section will yield z.

[0234] Then the distance d between [x, y, z] and the origin can be calculated, including:

[0235] d 2 =(x-x0) 2 ﹡spacing 2 [x] +(y-y0) 2 ﹡spacing 2 [y] +(z-z0) 2 ﹡spacing2 [z]

[0236] The distance d can be taken as the trachea radius.

[0237] The pixel spacing is the distance between two pixels. Spacing defines the physical size of the image pixel and ensures the accuracy of the actual distance measurement. The actual size of the object can be calculated through spacing. [x] ,spacing [y] ,spacing [z] Represents the voxel spacing in the x, y, and z directions respectively.

[0238] Similarly, determining the second trachea radius at the second endpoint includes:

[0239] Determining the direction of the second air pipe at the second end point;

[0240] Determine a second trachea cross section perpendicular to the direction of movement of the second trachea;

[0241] determining a second center of the second tracheal cross-section;

[0242] A pixel space coordinate system is established with the second center as the origin, wherein the pixel space includes i layers, the number of pixels in each layer is 8i, and i is a natural number greater than 0;

[0243] Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer;

[0244] The radius corresponding to the outermost trachea layer that reaches the preset ratio is used as the second trachea radius at the second end point.

[0245] The second method: determining the first trachea radius at the first endpoint includes:

[0246] Determine the direction of the first trachea at the first end point;

[0247] Determine a first trachea cross section perpendicular to the direction of movement of the first trachea;

[0248] Determine the first centroid of the first tracheal section. The centroid is the geometric center of the tracheal section, and its coordinates may be the average of the coordinates of all boundary points constituting the first tracheal section. If the density of the bronchus can be considered uniform, the centroid of the bronchus is also the center of mass.

[0249] respectively determining a plurality of first distances between a plurality of boundary points of the first trachea cross section and the formed first centroid;

[0250] A first average value of the plurality of first distances is determined, and the first average value is used as a first trachea radius at the first endpoint.

[0251] Similarly, determining the second trachea radius at the second endpoint includes:

[0252] Determining the direction of the second air pipe at the second end point;

[0253] Determine a second trachea cross section perpendicular to the direction of movement of the second trachea;

[0254] determining a second centroid of the second tracheal cross section;

[0255] respectively determining a plurality of second distances between a plurality of boundary points of the second trachea cross section and the formed second center;

[0256] A second average value of the plurality of second distances is determined, and the second average value is used as the second trachea radius at the second endpoint.

[0257] Furthermore, before determining the first average value of the first distance, the method further includes:

[0258] determining a first mean and a first variance according to the plurality of first distances;

[0259] Determine a first concentrated distribution range of the plurality of first distances according to the first mean and the first variance;

[0260] The first distance within the distribution range of the first set is screened.

[0261] Furthermore, before determining the second average value of the second distance, the method further includes:

[0262] determining a second mean and a second variance based on the plurality of second distances;

[0263] Determine a second concentrated distribution range of the plurality of second distances according to the second mean and the second variance;

[0264] The second set of distributions within the second distance is filtered.

[0265] For example, the distances from the boundary points of the trachea cross section to the centroid are [1, 1.1, 1.15, 1.33, 1.40, 1.48, 1.53, 1.62, 1.7, 1.8, 1.95, 2] respectively.

[0266] The mean is 1.505

[0267] The variance is 0.312

[0268] The minimum value of the concentrated distribution range is the mean minus the variance, that is, 1.505-0.312=1.193, and the maximum value of the concentrated distribution range is the mean plus the variance, that is, 1.505+0.312=1.817, so the concentrated distribution range is [1.193~1.817].

[0269] Therefore, the values ​​within the concentrated distribution range [1.193~1.817] are selected, including: 1.33, 1.40, 1.48, 1.53, 1.62, 1.7, 1.8.

[0270] The average of the seven distances within the concentrated distribution range is calculated to be 1.55, so the tracheal radius is 1.55.

[0271] Furthermore, to ensure the accuracy of the radius result, the radii of the endpoint and several points related to the endpoint may be calculated and the average value may be taken as the radius of the endpoint.

[0272] Step S180: Determine the trachea radius at the splicing point according to the splicing point, the first trachea radius and the second trachea radius, including two methods:

[0273] The first method: Based on the three coordinate directions of x, y, and z, select the appropriate direction for calculation.

[0274] The principle is to calculate the difference of the x, y, and z coordinates of the first endpoint and the second endpoint respectively, and select the direction with the largest difference for calculation. Among them, the direction with the largest difference is considered to be the direction of tracheal rupture.

[0275] For example, select the x direction as the calculation direction and start interpolation from the main bronchus:

[0276] sizeX=|x1-x2|

[0277] step=|x3-x1|

[0278] Δr=|r1-r2| / sizeX

[0279] r=r1-Δr*step

[0280] Among them, x1 is the x-coordinate of the first endpoint; x2 is the x-coordinate of the second endpoint; x3 is the x-coordinate of the splicing point, r1 is the first tracheal radius, r2 is the second tracheal radius, Δr is the radius growth rate, and r is the tracheal radius at the splicing point.

[0281] The second method: directly calculate based on the distance between the first endpoint and the second endpoint.

[0282] First, find the Euclidean distance d between the first endpoint and the second endpoint;

[0283] The growth rate of the radius is Δr

[0284] Δr=|r1-r2| / d

[0285] Then, the distance d1 between the splicing point and the first endpoint is calculated;

[0286] Then the trachea radius r at the current splicing point is:

[0287] r=r1-Δr*d1

[0288] Step S190: splicing the main bronchus and the broken bronchus based on the splicing point and the trachea radius at the splicing point, specifically comprising:

[0289] Based on the splicing point and the tracheal radius at the splicing point, a voxel screening range is set. Optionally, a cube range is set with the splicing point as the center, and a value slightly larger than half of the tracheal radius at the splicing point is selected as half of the edge length of the cube. Among them, how much larger half of the edge length is than half of the tracheal radius at the splicing point can be flexibly set, for example, half of the edge length can be determined by rounding up.

[0290] Within the voxel screening range, voxels whose distance from the splicing point does not exceed the tracheal radius at the splicing point are selected. By setting the voxel screening range, the number of voxels for judging whether the distance does not exceed the tracheal radius at the splicing point can be reduced, thereby improving the splicing efficiency.

[0291] The selected voxels are marked as spliced ​​bronchus to splice the main bronchus and the matching broken bronchus. The spliced ​​bronchus is used to represent the originally blank bronchus between the main bronchus and the matching broken bronchus obtained through the aforementioned splicing process.

[0292] The embodiment of the present application realizes splicing the main bronchus and the matching broken bronchus by simulating the shape of the originally blank spliced ​​trachea at the fracture as realistically as possible based on the splicing point and the trachea radius at the splicing point.

[0293] In one embodiment, in the segmentation method of the cascade segmentation network provided in the embodiment of the present application, the second endpoint of the broken bronchus matching the main bronchus is determined, and the centerline of the main bronchus includes multiple first skeleton points, specifically including:

[0294] Acquire a plurality of first skeleton points from the first endpoint along the centerline of the subject's bronchus;

[0295] The endpoints of the multiple broken bronchi are determined, and the endpoints of the multiple broken bronchi are used as first candidate endpoints.

[0296] A third distance between the plurality of first candidate endpoints and the first endpoint is calculated.

[0297] The first candidate endpoint whose third distance is within the preset distance threshold is determined as the second candidate endpoint. The preset distance threshold can be flexibly determined according to the specific situation. For example, when the number of second candidate endpoints is large, the preset distance threshold can be reduced, and when the number of second candidate endpoints is small, the preset distance threshold can be increased. The first candidate endpoint whose third distance is less than the preset distance threshold can be regarded as having the possibility of matching with the main bronchus, that is, belonging to the same tracheal branch as the main bronchus.

[0298] Acquire a plurality of second candidate skeleton points along the center line of the broken bronchus from the second candidate endpoint;

[0299] Randomly select two first skeleton points to form a first vector, randomly select two second candidate skeleton points to form a second vector, and randomly select one first skeleton point and one second candidate skeleton point to form a third vector;

[0300] According to the angle criterion, whether the second candidate endpoint can be used as the second endpoint is judged, specifically including: judging whether the angle between the first vector and the second vector is less than the first angle threshold, and whether the angle between the first vector and the third vector is less than the second angle threshold; or, judging whether the angle between the first vector and the second vector is less than the first angle threshold, and whether the angle between the second vector and the third vector is less than the third angle threshold. Among them, the angle threshold is determined according to the morphology of the bronchus, and the first angle threshold and the second angle threshold can be the same or different. Optionally, the second angle threshold is less than the first angle threshold. For example, the first angle threshold can be set to 30°, the second angle threshold to 25°, etc. Similarly, the third angle threshold can be the same or different from the first angle threshold, for example, the third angle threshold is less than the first angle threshold. The second angle threshold and the third angle threshold can be the same or different. Optionally, the third angle threshold is less than the second angle threshold.

[0301] If so, the second candidate endpoint is used as the second endpoint. The broken bronchus where the second endpoint is located is the matching broken bronchus.

[0302] If there are multiple second candidate endpoints, the same process may be performed on the multiple second candidate endpoints in sequence.

[0303] Exemplarily, the first skeleton point is obtained from the first end point of the main bronchus in a direction opposite to the direction of the trachea. For example, the process may stop when a bifurcation point is encountered, or when the number of skeleton points reaches a preset number, such as 5.

[0304] Similarly, from the second end point of the broken bronchus, a second skeleton point is obtained along the direction of the trachea. For example, the number of skeleton points can be stopped when it reaches a preset number, such as 5, or when there are no more skeleton points. Figure 5 Schematic diagram of angle criterion judgment, A1 and A2 skeleton points are selected from the skeleton points of the main bronchus, and the skeleton points A1 and A2 are used to form a first vector A1A2; B1 and B2 skeleton points are selected from the skeleton points of the broken bronchus; the skeleton points B1 and B2 are used to form a second vector B1B2, and the skeleton points A2 and B1 are used to form a third vector A2B1;

[0305] Determine whether the angle between the first vector A1A2 and the second vector B1B2 is less than a first angle threshold, and whether the angle between the first vector A1A2 and the third vector A2B1 is less than a second angle threshold; or determine whether the angle between the first vector A1A2 and the second vector B1B2 is less than a first angle threshold, and whether the angle between the second vector B1B2 and the third vector A2B1 is less than a third angle threshold. Figure 6 Schematic diagram of angle criterion judgment, through two angle judgments, the judgment accuracy is improved, only the broken bronchus 601 is spliced ​​with the main bronchus 603, and the broken bronchus 602 is avoided from being spliced ​​with the main bronchus 603. The broken bronchus 601 is a broken bronchus that matches the main bronchus 603.

[0306] If both are true, then the second candidate endpoint is used as the second endpoint.

[0307] Furthermore, if there are multiple second candidate endpoints that simultaneously satisfy the aforementioned angle criterion, one of them may be selected as the second endpoint, for example, the one that is closest to the first endpoint.

[0308] When the target segmentation result includes multiple main bronchi, the same processing as above can be performed on each main bronchus to find the broken bronchi matching each main bronchus and then splice them respectively.

[0309] In the embodiment of the present application, the broken bronchus is considered to match the main bronchus only when the main bronchus and the broken bronchus meet the angle criterion, and then the splicing is performed, thereby avoiding splicing that does not conform to the bronchial morphology.

[0310] In one embodiment, a training method for a cascade segmentation network is further provided, wherein the cascade segmentation network includes a first segmentation network and a second segmentation network, and the training method includes:

[0311] Acquire multiple medical images of anatomical structures as training images;

[0312] Acquire a first annotated image of the training image;

[0313] Segment the training image using a first segmentation network to obtain a first segmentation result;

[0314] Based on the first annotated image, determining a segmentation mask from the first segmentation result, wherein the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image;

[0315] removing the segmentation mask from the first annotated image to obtain a second annotated image;

[0316] Determining a first loss value according to the first annotated image and the first segmentation result;

[0317] Segment the training image using a second segmentation network to obtain a second segmentation result;

[0318] removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result;

[0319] determining a second loss value according to the second annotated image and the corrected second segmentation result;

[0320] Determining a total loss value according to the first loss value and the second loss value;

[0321] A loss function is determined according to the plurality of total loss values, and the cascade segmentation network is trained according to the loss function to obtain a trained cascade segmentation network.

[0322] The embodiment of the present application trains the first segmentation network and the second segmentation network uniformly, and removes the anatomical structure that has been correctly segmented by the first segmentation network when training the second segmentation network, thereby more accurately reflecting the segmentation result of the second segmentation network, which is beneficial to improving the training efficiency of the cascade segmentation network.

[0323] In one embodiment, a segmentation method for a segmented network is also provided, comprising:

[0324] determining a first endpoint of the subject's bronchus;

[0325] determining a second endpoint of the broken bronchus that matches the subject bronchus;

[0326] Determine a splicing point according to the first endpoint and the second endpoint, wherein the splicing point is used to splice the main bronchus and the matching broken bronchus;

[0327] determining a first trachea radius at the first endpoint;

[0328] determining a second trachea radius at the second end point;

[0329] Determine the trachea radius at the splicing point according to the splicing point, the first trachea radius and the second trachea radius;

[0330] Based on the splicing point and the trachea radius at the splicing point, the main bronchus and the matching broken bronchus are spliced.

[0331] Among them, the main bronchus and the matching broken bronchus belong to the same tracheal branch. The main bronchus and the broken bronchus include multiple, and the center lines of multiple main bronchus and multiple broken bronchus constitute the target segmentation result. The target segmentation result can be obtained by segmentation by the segmentation network in the prior art, or by segmentation by the aforementioned cascade segmentation network. The center line of the main bronchus includes multiple first skeleton points, and the center line of the matching broken bronchus includes multiple second skeleton points.

[0332] Optionally, how to determine the splicing point, the tracheal radius, the tracheal radius at the splicing point, and splicing the main bronchus and the matching broken bronchus based on the splicing point and the tracheal radius at the splicing point is the same as the aforementioned implementation principle and will not be repeated here for the sake of brevity of the entire text.

[0333] The present application also provides a control device. Figure 7 As shown, the control device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503, and a communication bus 504.

[0334] The processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0335] The communication interface 502 is used to communicate with other devices such as various sensors, rotating motors, solenoid valves, or network elements of other clients or servers.

[0336] The processor 501 is used to execute the program 505, and specifically can execute the relevant steps in the above method embodiment.

[0337] Specifically, the program 505 may include program codes, which include computer operation instructions.

[0338] The processor 505 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, or an FPGA, or a graphics processing unit (GPU). The one or more processors included in the control device may be processors of the same type, such as one or more CPUs, or one or more GPUs; or may be processors of different types, such as one or more CPUs and one or more GPUs.

[0339] The memory 503 is used to store the program 505. The memory 503 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0340] The program 505 can be specifically used to enable the processor 501 to execute the steps of the method described in any of the above embodiments.

[0341] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0342] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0343] It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should also be within the protection scope of the present invention and will not be described in detail here.

[0344] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0345] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0346] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if the described condition or event is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce the described condition or event is detected" or "in response to detecting the described condition or event" depending on the context.

[0347] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0348] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0349] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0350] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0351] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0352] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0353] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0354] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A segmentation method of a cascade segmentation network, characterized in that: The cascade segmentation network includes a first segmentation network and a second segmentation network, and the method includes: Acquire multiple medical images of anatomical structures as training images; Acquire a first annotated image of the training image; Segment the training image using a first segmentation network to obtain a first segmentation result; Based on the first annotated image, determining a segmentation mask from the first segmentation result, wherein the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image; removing the segmentation mask from the first annotated image to obtain a second annotated image; Determining a first loss value according to the first annotated image and the first segmentation result; Segment the training image using a second segmentation network to obtain a second segmentation result; removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result; determining a second loss value according to the second annotated image and the corrected second segmentation result; Determining a total loss value according to the first loss value and the second loss value; Determining a loss function according to the plurality of total loss values, and training the cascade segmentation network according to the loss function to obtain a trained cascade segmentation network; Acquire a medical image to be segmented; The trained cascade segmentation network is used to segment the medical image to obtain a target segmentation result.

2. The segmentation method according to claim 1, characterized in that: The method further comprises: Segmenting the training image according to the trained cascade segmentation network to obtain a segmentation result of the training image; The first annotated image is updated according to the segmentation result of the training image.

3. The segmentation method according to claim 2, characterized in that: The updating of the first annotated image according to the segmentation result of the training image includes: Merging the segmentation result of the training image and the first annotated image to obtain a merged result; Determining a first maximum connected domain in the first annotated image; Determine the second largest connected domain in the merging result; Determine an additional part of the maximum connected domain according to the first maximum connected domain and the second maximum connected domain; It is determined whether the tracheal radius and / or tracheal direction of the added part in the maximum connected domain meets the preset requirements. If so, the first annotated image is updated according to the merging result.

4. The segmentation method according to claim 1, characterized in that: The method further comprises: determining a bifurcation point in the training image according to the first annotated image; Changing the current position of the bifurcation point within a preset position range; Perform spline interpolation according to the changed current position to obtain an enhanced image; The enhanced image is used as a training image.

5. The segmentation method according to claim 1, characterized in that: The target segmentation result includes a plurality of main bronchi and a plurality of broken bronchi, and the method further includes: determining a first endpoint of the subject's bronchus; Determine a second endpoint of a broken bronchus that matches the main bronchus from the multiple broken bronchus, wherein the matched broken bronchus and the main bronchus belong to the same tracheal branch; Determine a splicing point according to the first endpoint and the second endpoint, wherein the splicing point is used to splice the main bronchus and the broken bronchus; determining a first trachea radius at the first endpoint; determining a second trachea radius at the second end point; Determine the trachea radius at the splicing point according to the splicing point, the first trachea radius and the second trachea radius; Based on the splicing point and the trachea radius at the splicing point, the main bronchus and the matching broken bronchus are spliced.

6. The segmentation method according to claim 5, characterized in that: The centerline of the main bronchus includes a plurality of first skeleton points, the centerline of the matched broken bronchus includes a plurality of second skeleton points, and determining the splicing point according to the first endpoint and the second endpoint includes: Acquire a plurality of first skeleton points from the first endpoint along the centerline of the subject's bronchus; Acquire a plurality of second skeleton points from the second endpoint along the center line of the broken bronchus; Determine a fitting function using a least square method according to the obtained multiple first skeleton points and the obtained multiple second skeleton points, wherein the fitting function is used to characterize a center line of a spliced ​​bronchus of the main bronchus and the matched broken bronchus; The splicing point is determined according to the first endpoint, the second endpoint and the fitting function.

7. The segmentation method according to claim 5, characterized in that: The determining of the first trachea radius at the first endpoint comprises: Determine the direction of the first trachea at the first end point; Determine a first trachea cross section perpendicular to the direction of movement of the first trachea; determining a first center of the first tracheal cross-section; Establishing a pixel space coordinate system with the first center as the origin, wherein the pixel space includes multiple layers, and each layer includes multiple pixels; Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer; Using the radius corresponding to the outermost trachea layer that reaches the preset ratio as the first trachea radius at the first end point; or, Determine the direction of the first trachea at the first end point; Determine a first trachea cross section perpendicular to the direction of movement of the first trachea; determining a first centroid of the first tracheal cross section; respectively determining a plurality of first distances between a plurality of boundary points of the first trachea cross section and the first centroid; A first average value of the plurality of first distances is determined, and the first average value is used as a first trachea radius at the first endpoint.

8. The segmentation method according to claim 7, characterized in that: Before determining the first average value of the first distance, the method further includes: determining a first mean and a first variance according to the plurality of first distances; Determine a first concentrated distribution range of the plurality of first distances according to the first mean and the first variance; The first distance within the distribution range of the first set is screened.

9. The segmentation method according to claim 5, characterized in that: The determining of the second trachea radius at the second endpoint comprises: Determining the direction of the second air pipe at the second end point; Determine a second trachea cross section perpendicular to the direction of movement of the second trachea; determining a second center of the second tracheal cross-section; Establishing a pixel space coordinate system with the second center as the origin, wherein the pixel space includes multiple layers, and each layer includes multiple pixels; Determine whether the ratio of the pixels of the bronchus in each layer to all the pixels in the layer reaches the preset ratio of the layer; Using the radius corresponding to the outermost trachea layer that reaches the preset ratio as the second trachea radius at the second end point; or, Determining the direction of the second air pipe at the second end point; Determine a second trachea cross section perpendicular to the direction of movement of the second trachea; determining a second centroid of the second tracheal cross section; respectively determining a plurality of second distances between a plurality of boundary points of the second trachea cross section and the second centroid; A second average value of the plurality of second distances is determined, and the second average value is used as the second trachea radius at the second endpoint.

10. The segmentation method according to claim 5, characterized in that: The step of splicing the main bronchus and the matching broken bronchus based on the splicing point and the trachea radius at the splicing point comprises: Based on the splicing point and the trachea radius at the splicing point, setting a voxel screening range; Within the voxel screening range, selecting voxels whose distance from the splicing point does not exceed the trachea radius at the splicing point; The selected voxels are marked as spliced ​​bronchus to splice the main bronchus and the matched broken bronchus.

11. The segmentation method according to claim 5, characterized in that: The centerline of the main bronchus includes a plurality of first skeleton points, and the second endpoint of the broken bronchus matching the main bronchus is determined, including: Determine endpoints of the multiple broken bronchi, and use the endpoints of the multiple broken bronchi as first candidate endpoints; Calculating a third distance between the plurality of first candidate endpoints and the first endpoint; Determine a first candidate endpoint whose third distance is within a preset distance threshold as a second candidate endpoint; Acquire a plurality of first skeleton points from the first endpoint along the centerline of the subject's bronchus; Acquire a plurality of second candidate skeleton points along the center line of the broken bronchus from the second candidate endpoint; Randomly select two first skeleton points to form a first vector, randomly select two second candidate skeleton points to form a second vector, and randomly select one first skeleton point and one second candidate skeleton point to form a third vector; Determine whether the angle between the first vector and the second vector is less than a first angle threshold, and whether the angle between the first vector and the third vector is less than a second angle threshold; or, determine whether the angle between the first vector and the second vector is less than a first angle threshold, and whether the angle between the second vector and the third vector is less than a third angle threshold; If both are true, the second candidate endpoint is used as the second endpoint.

12. A training method for a cascade segmentation network, characterized in that: The cascade segmentation network includes a first segmentation network and a second segmentation network, and the method includes: Acquire multiple medical images of anatomical structures as training images; Acquire a first annotated image of the training image; Segment the training image using a first segmentation network to obtain a first segmentation result; Based on the first annotated image, determining a segmentation mask from the first segmentation result, wherein the segmentation mask is used to characterize the same anatomical structure in the first segmentation result and the first annotated image; removing the segmentation mask from the first annotated image to obtain a second annotated image; Determining a first loss value according to the first annotated image and the first segmentation result; Segment the training image using a second segmentation network to obtain a second segmentation result; removing the segmentation mask from the second segmentation result to obtain a modified second segmentation result; determining a second loss value according to the second annotated image and the corrected second segmentation result; Determining a total loss value according to the first loss value and the second loss value; A loss function is determined according to the total loss value, and the cascade segmentation network is trained according to the loss function to obtain a trained cascade segmentation network.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the segmentation method according to any one of claims 1 to 11, or implements the training method according to claim 12.

Images