Image segmentation method, device, electronic device and storage medium

Abstract

The present invention discloses an image segmentation method, device, electronic device, and storage medium. The method includes: obtaining an original image, and respectively obtaining an initial segmentation image and a skeleton segmentation image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model; performing a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image; performing a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image; generating a target segmentation image of the original image based on the expanded connected regions after the expansion process based on the skeleton connected line segments and the first processed image. Through the technical solution disclosed in the embodiments of the present invention, the problem that there is a phenomenon of regional interruption in the segmentation result obtained based on the existing segmentation method, resulting in poor accuracy of the segmentation result, is solved, and the integrity of the segmentation result is improved, thereby improving the accuracy of the segmentation result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular, to an image segmentation method, apparatus, electronic device, and storage medium. Background Art

[0002] In the field of modern medical technology, the automated coronary artery reconstruction technology has important clinical value and practical significance for doctors. To perform automated coronary artery reconstruction, it is first necessary to solve the problem of automated coronary artery segmentation in this process. Different from traditional coronary artery segmentation methods, due to many advantages of neural networks, there is an increasing tendency to use neural networks to complete coronary artery segmentation work.

[0003] In the process of coronary artery segmentation based on neural networks in the embodiments of the present invention, it is found that there is a phenomenon of regional interruption in the segmentation results, resulting in poor accuracy of the segmentation results. Summary of the Invention

[0004] The present invention provides an image segmentation method, apparatus, electronic device, and storage medium to solve the problem that there is a phenomenon of regional interruption in the segmentation results obtained by the existing segmentation method, resulting in poor accuracy of the segmentation results, so as to improve the integrity of the segmentation results and thus improve the accuracy of the segmentation results.

[0005] In a first aspect, an embodiment of the present invention provides an image segmentation method, which includes:

[0006] Obtain an original image, and respectively obtain an initial segmentation image and a skeleton segmentation image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model;

[0007] Perform a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image;

[0008] Perform a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image;

[0009] Generate a target segmentation image of the original image based on the extended connected region after the expansion process of the skeleton connected line segments and the first processed image.

[0010] Optionally, the performing a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image includes:

[0011] Perform a subtraction process on the initial segmentation image and the skeleton segmentation image to determine at least one candidate skeleton line segment in the skeleton segmentation image;

[0012] Determine at least one skeleton connected line segment in the candidate region based on the line segment endpoints of the candidate skeleton line segments and the region boundary of the first processed image.

[0013] Optionally, generating the target segmentation image of the original image based on the expanded connected region after the expansion processing of the skeleton connected line segment and the first processed image includes:

[0014] For any skeleton connected line segment, determine at least one expansion parameter of the current skeleton connected line segment based on the region boundary of the first processed image;

[0015] Perform expansion processing on the skeleton connected line segment based on the expansion parameter to obtain the expanded connected region of the skeleton connected line segment;

[0016] Perform fusion processing on the first processed image and the expanded connected region to generate the target segmentation image of the original image.

[0017] Optionally, the expansion parameter includes an endpoint expansion parameter and a center expansion parameter of the skeleton connected line segment; for any skeleton connected line segment, the endpoint expansion parameter is determined based on the line segment endpoints of the current skeleton connected line segment and the boundary shapes of the region boundaries connected to the line segment endpoints;

[0018] Correspondingly, performing expansion processing on the skeleton connected line segment based on the expansion parameter to obtain the expanded connected region of the skeleton connected line segment includes:

[0019] Perform linear fitting processing on each of the endpoint expansion data to generate at least one center point expansion parameter of the current skeleton connected line segment;

[0020] Generate the expanded connected region of the skeleton connected line segment based on the endpoint expansion parameter and each of the center expansion parameters.

[0021] Optionally, the training method of the skeleton segmentation model includes:

[0022] Obtain a sample image for training the skeleton segmentation model, and the target region after annotation in the sample image, and perform skeletonization processing on the target region to obtain the skeleton line segments of the target region;

[0023] Input the sample image into the skeleton segmentation model to be trained to obtain the sample prediction result output by the skeleton segmentation model;

[0024] Perform iterative training on the skeleton segmentation model based on the target region, skeleton line segments, and the sample prediction result to obtain the skeleton segmentation model.

[0025] Optionally, the sample prediction result includes the region prediction result corresponding to the target region and the skeleton prediction result corresponding to the skeleton line segment;

[0026] Correspondingly, the iterative training of the skeleton segmentation model based on the target region, the skeleton line segment, and the sample prediction result includes:

[0027] For any round of iteration, based on the skeleton prediction result, the target region, and the skeleton line segment, determine the first sample prediction probability of the skeleton segmentation model in the current iteration round;

[0028] Based on the skeleton line segment, the region prediction result, and the skeleton prediction result, determine the second sample prediction probability of the skeleton segmentation model in the current iteration round;

[0029] Generate the model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability, and adjust the model parameters of the skeleton segmentation model based on the model loss function.

[0030] Optionally, the skeletonization process of the target region to obtain the skeleton line segment of the target region includes:

[0031] The determination of the first sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton prediction result, the target region, and the skeleton line segment includes:

[0032]

[0033] Where Recall represents the first sample prediction probability; S P represents the skeleton prediction result; V L represents the target region; S L represents the skeleton line segment;

[0034] Correspondingly, the determination of the second sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton line segment, the region prediction result, and the skeleton prediction result includes:

[0035]

[0036] Where Precision represents the second sample prediction probability; S L represents the skeleton line segment; V P represents the region prediction result; S P represents the skeleton prediction result;

[0037] Correspondingly, generating the model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability includes:

[0038]

[0039] Among them, skeletonDice loss represents the model loss function; Recall represents the first sample prediction probability; Precision represents the second sample prediction probability.

[0040] In a second aspect, an embodiment of the present invention further provides an image segmentation device, and the device includes:

[0041] A segmented image acquisition module, configured to acquire an original image, and respectively obtain an initial segmented image and a skeleton segmented image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model;

[0042] A first processed image generation module, configured to perform a fusion process on the initial segmented image and the skeleton segmented image to generate a first processed image;

[0043] A skeleton connected line segment acquisition module, configured to perform a subtraction process on the initial segmented image and the skeleton segmented image to generate skeleton connected line segments in the skeleton segmented image;

[0044] A target segmented image generation module, configured to generate a target segmented image of the original image based on the extended connected region after the extension process of the skeleton connected line segment and the first processed image.

[0045] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0046] At least one processor; and

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor can execute the image segmentation method according to any embodiment of the present invention.

[0049] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image segmentation method according to any embodiment of the present invention when executed.

[0050] In the technical solution of the embodiment of the present invention, an initial segmentation image is obtained through an initial segmentation model to acquire the branch detail information of the original image. Meanwhile, a skeleton segmentation image is obtained based on the skeleton segmentation model to acquire the topological information of the connection of each branch in the original image. Furthermore, fusion processing is performed based on the initial segmentation image and the skeleton segmentation image, so that the generated target segmentation result contains both branch detail information and branch connection information, that is, it is closer to the original image, thereby improving the accuracy of the segmentation result.

[0051] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. Brief Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0053] Figure 1 It is a flowchart of an image segmentation method provided in Embodiment 1 of the present invention;

[0054] Figure 2 It is a schematic diagram of the principle for generating the first processed image involved in Embodiment 1 of the present invention;

[0055] Figure 3 It is a schematic diagram of the principle for generating the skeleton connection line segments involved in Embodiment 1 of the present invention;

[0056] Figure 4 It is a schematic diagram of the principle for generating the target segmentation image involved in Embodiment 1 of the present invention;

[0057] Figure 5 It is a flowchart of an image segmentation method provided in Embodiment 2 of the present invention;

[0058] Figure 6 It is a schematic structural diagram of an image segmentation device provided in Embodiment 3 of the present invention;

[0059] Figure 7 It is a schematic structural diagram of an electronic device for implementing the image segmentation method of the embodiment of the present invention. Detailed Embodiments

[0060] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here.

[0062] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of these messages or information.

[0063] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0064] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that executes the operation of the technical solution of the present disclosure according to the prompt message.

[0065] As an optional but non-limiting implementation manner, the manner of sending a prompt message to the user in response to receiving an active request from the user can be, for example, in the form of a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0066] It can be understood that the above process of notifying and obtaining the user's authorization is only illustrative and does not constitute a limitation on the implementation manner of the present disclosure. Other manners that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0067] It can be understood that the data involved in the technical solution of the present invention (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the corresponding laws, regulations and related regulations.

[0068] Example 1

[0069] Figure 1 The following is a flowchart of an image segmentation method provided by Example 1 of the present invention. This example is applicable to the situation of segmenting an image. This method can be executed by an image segmentation device, which can be implemented in the form of hardware and / or software, and can be configured in an intelligent terminal and a cloud server.

[0070] In some other embodiments, during the process of segmenting an image with many branches, when performing segmentation processing based on the segmentation method of the prior art, the segmentation results of each branch can be obtained, but it is easy to have an interruption phenomenon between the branches, resulting in inaccurate segmentation results. To solve the above technical problems, this example proposes an image segmentation algorithm. Specifically, an initial segmentation image of the original image is obtained based on an initial segmentation model, the branch detail information of the original image is acquired, and at the same time, a skeleton segmentation image is obtained based on a skeleton segmentation model, and the topological information of the connection of each branch in the original image is acquired. Then, fusion processing is performed based on the initial segmentation image and the skeleton segmentation image, so that the generated target segmentation result contains both branch detail information and branch connection information, that is, the target segmentation image is closer to the real original image, thereby improving the accuracy of the segmentation result.

[0071] As Figure 1 shown, the method includes:

[0072] S110. Obtain the original image, and respectively obtain the initial segmentation image and the skeleton segmentation image of the original image based on the pre-trained initial segmentation model and skeleton segmentation model.

[0073] In the embodiment of the present invention, the original image can be understood as an image containing many branches, such as an image containing coronary blood vessels. Optionally, it can also be other images with branches, such as a flower branch image, etc. The content of the image in this example is not limited.

[0074] Specifically, if the original image is an image containing coronary blood vessels, the acquisition method may include scanning the target object based on a medical imaging device to obtain the original image, or reading the image data in a local database or the database of a cloud server to obtain the original image; optionally, if the original image is an image containing flower branches, the acquisition method may include taking a picture based on a shooting device to obtain the original image, or scanning based on a preset scanning device to obtain the original image; in this example, different acquisition methods can be adopted according to the type of the original image, and this example does not limit this.

[0075] Optionally, when the original image is obtained, image segmentation processing is performed on the original image. Specifically, a pre-trained segmentation model can be used to process the original image to obtain the corresponding segmented image. Optionally, the method for obtaining the initial segmented image of the original image based on the pre-trained initial segmentation model includes: inputting the original image into the initial segmentation model to obtain the initial segmented image output by the initial segmentation model; Optionally, the method for obtaining the skeleton segmented image of the original image based on the pre-trained skeleton segmentation model includes: inputting the original image into the skeleton segmentation model to obtain the skeleton segmented image output by the skeleton segmentation model.

[0076] It should be noted that in this embodiment, the order of obtaining the initial segmented image and the skeleton segmented image is not limited. They can be obtained simultaneously or sequentially. This embodiment does not limit this.

[0077] S120. Perform fusion processing on the initial segmented image and the skeleton segmented image to generate a first processed image.

[0078] In the embodiment of the present invention, the first processed image can be understood as the fusion image of the initial segmented image and the skeleton segmented image. The fusion processing can be understood as performing a union operation on the initial segmented image and the skeleton segmented image. Specifically, see Figure 2 , Pn represents the initial segmented image, Ps represents the skeleton segmented image; Pn and Ps are merged and then the maximum connected component processing is performed to obtain the fusion result Pc, that is, the first processed image. Optionally, through the merging process, the obtained segmented image contains both the branch detail information of the original image and the branch connectivity information of the original image.

[0079] S130. Perform a subtraction operation on the initial segmented image and the skeleton segmented image to generate the skeleton connected line segments in the skeleton segmented image.

[0080] In the embodiment of the present invention, the subtraction operation can be understood as determining the different regions between the two images.

[0081] Optionally, the method for generating the skeleton connected line segments in the skeleton segmented image based on the subtraction operation may include: performing a subtraction operation on the initial segmented image and the skeleton segmented image to determine at least one candidate skeleton line segment in the skeleton segmented image; based on the line segment endpoints of the candidate skeleton line segment and the region boundary of the first processed image, determining at least one skeleton connected line segment in the candidate region.

[0082] Specifically, the skeleton segmented image is compared with the initial segmented image for coverage to determine the skeleton line segments in the skeleton segmented image that are not covered by the initial segmented image, and the above non-covered skeleton line segments are determined as candidate skeleton line segments.

[0083] Optionally, for any of the candidate skeleton line segments described above, determine whether the two endpoints of the skeleton line segment are connected to the break points in the initial segmented image. Optionally, if only one end is connected or neither end is connected, remove the skeleton line segment to avoid introducing incorrect candidate line segments, which may cause inaccurate segmentation results. Conversely, if both ends are connected, determine that the skeleton line segment is a skeleton connected line segment in the skeleton segmented image. Specifically, refer to Figure 3 , in the figure, Ps represents the skeleton segmented image, Pn represents the initial segmented image, and the candidate skeleton line segment Lc is obtained based on the difference set between Ps and Pn (the set of pixel points that belong to Ps and do not belong to Pn); and the candidate skeleton line segment is screened to obtain the skeleton connected line segment Ls.

[0084] S140. Generate the target segmented image of the original image based on the extended connected region and the first processed image after the extension processing of the skeleton connected line segment.

[0085] In the embodiment of the present invention, in the case of obtaining the skeleton connected line segment of the skeleton segmented image, the skeleton connected line segment is extended based on the first processed image to generate an extended connected region, and then the first processed image and the extended connected region are fused to generate the target segmented image of the original image.

[0086] Optionally, for any skeleton connected line segment, the method of extending the skeleton connected line segment to generate an extended connected region may include: determining at least one extension parameter of the current skeleton connected line segment based on the first processed image; and extending the skeleton connected line segment based on the extension parameter to obtain the extended connected region of the skeleton connected line segment.

[0087] It should be noted that the extension parameter can be understood as the diameter parameter by which the skeleton of the coronary lumen needs to be extended. The extension parameter includes the endpoint extension parameter and the center extension parameter of the skeleton connected line segment; for any skeleton connected line segment, the endpoint extension parameter is determined based on the line segment endpoints of the current skeleton connected line segment and the boundary shapes of the regions connected to the line segment endpoints; in other words, the endpoint extension parameter can be determined based on the lumen diameter at the break points of the segmented image in the first processed image. Correspondingly, extending the skeleton connected line segment based on the extension parameter to obtain the extended connected region of the skeleton connected line segment includes: performing linear fitting processing on each endpoint extension data to generate at least one center extension parameter of the current skeleton connected line segment; and generating the extended connected region of the skeleton connected line segment based on the endpoint extension parameter and each center extension parameter.

[0088] Specifically, based on determining the endpoint expansion parameters of the two endpoints of the current skeleton connected line segment, the expansion parameter of the midpoint of the current skeleton connected line segment is fitted by using the two endpoint expansion parameters to determine at least one center point expansion parameter of the current skeleton connected line segment; furthermore, the endpoint expansion parameter and each center expansion parameter are used as expansion targets to perform expansion processing on the current skeleton connected line segment to obtain the expanded connected region of the current skeleton connected line segment. Optionally, based on the above-described embodiment, all the skeleton connected line segments are traversed to obtain each expanded connected region.

[0089] Optionally, in this embodiment, the expansion processing methods may include, but are not limited to, local expansion, global expansion, and gradual expansion and other expansion methods. Specifically, local expansion can be understood as expanding the stenotic part of the lumen to the normal size, and the rest of the normal lumen remains unchanged; global expansion can be understood as uniformly expanding the entire coronary lumen to a unified scale size to ensure that the lumen diameter at the stenotic part reaches the normal size; gradual expansion can be understood as expanding the coronary lumen to a preset size. For example, the lumen shape gradually becomes smaller from the proximal end to the distal end. Of course, the above-described expansion methods are only optional expansion methods, and other expansion methods may also be adopted in this embodiment, and no further examples are given here.

[0090] Of course, it is also possible to perform expansion processing on the skeleton connected line segment based on other morphological expansion methods, i.e., the dilation method, to obtain the expanded connected region in this embodiment, and this embodiment does not limit this.

[0091] Optionally, in the case of obtaining the expanded connected region of the skeleton connected line segment, the expanded connected region and the first processed image are fused to generate the target segmentation image of the original image. Specifically, see Figure 4 , Pc represents the first processed image, Rc represents the expanded connected region, and Pc and Rc are spliced to obtain the target segmentation image P corresponding to the original image.

[0092] In this embodiment, due to the diversity of the original images, there may be a phenomenon of coronary lumen stenosis caused by atherosclerosis or other reasons, that is, the thickness of the coronary lumen in the original images is inconsistent. Furthermore, during the process of obtaining the initial segmentation image based on the initial segmentation model, there will be a problem of interrupted branches in the segmentation result. Based on the above technical problems, in this embodiment, the coronary lumen skeleton line segments in the original images are marked, and the skeleton segmentation model is trained based on the skeleton line segments to segment the skeleton line segments with consistent lumen thickness in the original images, thereby solving the problem of interrupted branches in the segmentation result and obtaining a complete skeleton segmentation image with connected branches. It should be noted that although the obtained skeleton segmentation image has good branch connectivity, its lumen diameter is small (i.e., the lumen is not full). Therefore, it is necessary to perform an expansion process on the skeleton connected line segments in the skeleton segmentation image based on the initial segmentation image, and perform a fusion process on the expanded image, so that the obtained target segmentation image has higher accuracy and better quality.

[0093] Embodiment 2

[0094] Figure 5 As shown in the flowchart of an image segmentation method provided in Embodiment 2 of the present invention. On the basis of the above embodiment, optionally, before obtaining the initial segmentation image and the skeleton segmentation image of the original image based on the pre-trained initial segmentation model and the skeleton segmentation model respectively, the initial segmentation model and the skeleton segmentation model are pre-trained.

[0095] As Figure 5 shown, the method includes:

[0096] S210. Pre-train the initial segmentation model and the skeleton segmentation model.

[0097] Before inputting the original image into the initial segmentation model in the technical solution of this embodiment, the initial segmentation model is pre-trained to perform image segmentation on the original image through the trained segmentation model to ensure the accuracy of the segmentation result of the initial segmentation image. If the initial segmentation model is a blood vessel segmentation model, the training process for the initial segmentation model may include: obtaining coronary blood vessel images for model training, and performing manual annotation on the coronary blood vessel images; inputting the annotated coronary blood vessel images into a pre-determined deep neural network to obtain a segmentation prediction result; comparing the segmentation prediction results of each coronary blood vessel image with the manual annotation, generating a loss function of the model based on the comparison result, and updating the network structure parameters of the deep neural network based on the loss function to evolve it in the direction of reducing the prediction error; further, using a large number of coronary blood vessel images as training samples, repeating the above iterative process thousands of times, and finally stopping the training to obtain the trained initial segmentation model when the comparison result between the segmentation prediction result and the manual annotation reaches a preset threshold.

[0098] In this embodiment, an initial segmentation image obtained based on an initial segmentation model can yield a segmentation result with rich branch details. It should be noted that due to the diversity of original images, some samples may exhibit phenomena such as coronary lumen stenosis caused by atherosclerosis, etc. That is, sample images with inconsistent lumen thickness will reduce the learning ability of the initial segmentation model, resulting in the problem of interrupted branches in the segmentation result of the original image during the process of image segmentation of the original image with inconsistent lumen thickness. Based on the above technical problems, in this embodiment, the coronary lumen skeleton line segments in the original image are annotated, and a skeleton segmentation model is trained based on the skeleton line segments to segment the skeleton line segments with consistent lumen thickness in the original image, thereby solving the problem of interrupted branches in the segmentation result of the original image during the process of image segmentation of the original image.

[0099] Optionally, the training process of the skeleton segmentation model in this embodiment includes: obtaining a sample image for training the skeleton segmentation model and the annotated target region in the sample image, and performing skeletonization processing on the target region to obtain the skeleton line segments of the target region; inputting the sample image into the skeleton segmentation model to be trained to obtain the sample prediction result output by the skeleton segmentation model; and performing iterative training on the skeleton segmentation model based on the target region, the skeleton line segments, and the sample prediction result to obtain the skeleton segmentation model.

[0100] Specifically, a sample image for training the skeleton segmentation model is obtained. This sample image can include sample images with normal coronary lumens and sample images with coronary lumen stenosis; the coronary lumen in the sample image is annotated, and the annotated coronary lumen is used as the target region in the sample image. On this basis, skeletonization processing is performed on the target region, that is, the coronary lumen in the sample image, to obtain the processed skeleton line segments, and the extended segmentation model is trained based on the obtained target region and the skeleton line segments. Optionally, the method for performing skeletonization processing on the target region to obtain the skeleton line segments of the target region can include: determining the topological structure of the target region and determining the skeleton line segments of the target region based on the topological structure. Specifically for coronary blood vessels, on the premise of maintaining the connectivity of the foreground region of the binary image of the coronary blood vessels unchanged, the foreground pixels at the center edge are reduced as much as possible, and finally the topological structure of the coronary blood vessels, that is, the skeleton line segments of the coronary blood vessels, is obtained.

[0101] It should be noted that the effect of using the skeleton line segments after skeletonization processing for training in this embodiment is that sample images with the same thickness can be input into the model for learning, enhancing the learning of the connectivity at the lumen stenosis, thereby improving the learning ability of the skeleton segmentation model for image segmentation.

[0102] Optionally, input the sample image into the skeleton segmentation model to be trained, and obtain the sample prediction result output by the skeleton segmentation model. The sample prediction result includes the region prediction result corresponding to the target region and the skeleton prediction result corresponding to the skeleton line segment. Correspondingly, iteratively training the skeleton segmentation model based on the target region, the skeleton line segment, and the sample prediction result includes: for any round of iteration, determining the first sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton prediction result, the target region, and the skeleton line segment; determining the second sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton line segment, the region prediction result, and the skeleton prediction result; generating the model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability, and adjusting the model parameters of the skeleton segmentation model based on the model loss function.

[0103] It should be noted that the first sample prediction probability can be understood as the topological recall rate of the skeleton segmentation model, which is used to express the probability that the center line correctly predicted by the gold standard center line is segmented, that is, it focuses on evaluating the integrity of the coronary artery gold standard center line being segmented. In this embodiment, the larger the first sample prediction value, the better. Optionally, the method for determining the first sample prediction value can be determined by the following expression. Exemplarily, the expression may include:

[0104]

[0105] where Recall represents the first sample prediction probability; S P represents the skeleton prediction result; V L represents the target region; S L represents the skeleton line segment.

[0106] It should be noted that the second sample prediction probability can be understood as the topological precision rate of the skeleton segmentation model, which is used to express the probability that the result of the predicted segmentation center line is correctly predicted; that is, it focuses on evaluating the accuracy of the predicted segmentation center line itself. In this embodiment, the larger the second sample prediction value, the better. Optionally, the method for determining the second sample prediction value can be determined by the following expression. Exemplarily, the expression may include:

[0107]

[0108] where Precision represents the second sample prediction probability; S L represents the skeleton line segment; V P represents the region prediction result; S P represents the skeleton prediction result.

[0109] Optionally, when determining the first sample prediction probability and the second sample prediction probability, a model loss function of the skeleton segmentation model in the current round is generated based on the first sample prediction probability and the second sample prediction probability. Optionally, the following expression can be used to determine the model loss function of the skeleton segmentation model. Exemplarily, the expression may include:

[0110]

[0111] Among them, skeletonDice loss represents the model loss function; Recall represents the first sample prediction probability; Precision represents the second sample prediction probability.

[0112] Furthermore, a large number of coronary artery images are used as training samples, and the above iterative process is repeated thousands of times. Finally, when the comparison result between the segmentation prediction result and the manual annotation reaches a preset threshold, the training is stopped to obtain a trained skeleton segmentation model.

[0113] Compared with the commonly used Dice loss function, which only calculates at the pixel level, does not consider the connectivity between pixels, and is insensitive to the calculation of interruption phenomena, the segmentation model trained based on the Dice loss results in inaccurate segmentation results when segmenting images. The beneficial effect of training the skeleton segmentation model in the above manner in this embodiment is that: a model loss is generated based on the connected pixels of each branch after skeletonization, and training is performed based on the model loss, which improves the model's learning of connectivity. Therefore, the trained skeleton segmentation model has better connectivity in the segmentation result when segmenting images, thereby improving the accuracy of the segmentation result.

[0114] S220. Obtain the original image, and respectively obtain the initial segmentation image and the skeleton segmentation image of the original image based on the pre-trained initial segmentation model and the skeleton segmentation model.

[0115] S230. Perform a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image.

[0116] S240. Perform a subtraction process on the initial segmentation image and the skeleton segmentation image to generate the skeleton connected line segments in the skeleton segmentation image.

[0117] S250. Generate the target segmentation image of the original image based on the extended connected region after the expansion process of the skeleton connected line segments and the first processed image.

[0118] In the embodiments of the present invention, by determining the loss function for training the skeleton segmentation model based on the recall rate and precision rate of the samples, the learning ability of the skeleton segmentation model is improved, and further, the segmentation accuracy of the trained skeleton segmentation model during the segmentation process is improved. Optionally, an initial segmentation image is obtained based on the initial segmentation model, and the branch detail information of the original image is acquired. At the same time, a skeleton segmentation image is obtained based on the skeleton segmentation model, and the topological information of the connectivity of each branch in the original image is acquired. Then, based on the initial segmentation image and the skeleton segmentation image, a fusion process is performed, so that the generated target segmentation result contains both branch detail information and branch connectivity information, that is, it is closer to the original image, thereby realizing the improvement of the accuracy of the segmentation result.

[0119] Embodiment III

[0120] Figure 6 FIG. is a schematic structural diagram of an image segmentation device provided in Embodiment III of the present invention. As Figure 6 shown, the device includes: a segmentation image acquisition module 310, a first processed image generation module 320, a skeleton connected line segment acquisition module 330, and a target segmentation image generation module 340; wherein,

[0121] The segmentation image acquisition module 310 is configured to acquire an original image, and respectively obtain an initial segmentation image and a skeleton segmentation image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model;

[0122] The first processed image generation module 320 is configured to perform a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image;

[0123] The skeleton connected line segment acquisition module 330 is configured to perform a subtraction process on the initial segmentation image and the skeleton segmentation image to generate the skeleton connected line segments in the skeleton segmentation image;

[0124] The target segmentation image generation module 340 is configured to generate a target segmentation image of the original image based on the expanded connected region after the expansion process of the skeleton connected line segments and the first processed image.

[0125] On the basis of the above embodiment, optionally, the skeleton connected line segment acquisition module 330 includes:

[0126] A candidate skeleton line segment generation sub-module, configured to perform a subtraction process on the initial segmentation image and the skeleton segmentation image to determine at least one candidate skeleton line segment in the skeleton segmentation image;

[0127] A skeleton connected line segment generation sub-module, configured to determine at least one skeleton connected line segment in the candidate region based on the line segment endpoints of the candidate skeleton line segments and the region boundary of the first processed image.

[0128] Based on the above embodiments, optionally, the target segmentation image generation module 340 includes:

[0129] An extended parameter determination sub-module, configured to determine at least one extended parameter of the current skeleton connected line segment based on the first processed image for any skeleton connected line segment;

[0130] An extended connected region generation sub-module, configured to perform an extension process on the skeleton connected line segment based on the extended parameter to obtain an extended connected region of the skeleton connected line segment;

[0131] A target segmentation image generation sub-module, configured to perform a fusion process on the first processed image and the extended connected region to generate a target segmentation image of the original image.

[0132] Based on the above embodiments, optionally, the extended parameter includes an endpoint extension parameter and a center extension parameter of the skeleton connected line segment; for any skeleton connected line segment, the endpoint extension parameter is determined based on the line segment endpoints of the current skeleton connected line segment and the boundary shapes of the boundaries of the regions connected to the line segment endpoints;

[0133] Correspondingly, the extended connected region generation sub-module includes:

[0134] A center extension parameter determination unit, configured to perform a linear fitting process on each of the endpoint extension data to generate at least one center point extension parameter of the current skeleton connected line segment;

[0135] An extended connected region generation unit, configured to generate an extended connected region of the skeleton connected line segment based on the endpoint extension parameter and each of the center extension parameters.

[0136] Based on the above embodiments, optionally, the device further includes: a model training module, configured to train the skeleton segmentation model;

[0137] Optionally, the model training module includes:

[0138] A region and line segment acquisition sub-module, configured to acquire a sample image for training the skeleton segmentation model, and a target region after annotation in the sample image, and perform a skeletonization process on the target region to obtain a skeleton line segment of the target region;

[0139] A sample prediction result acquisition sub-module, configured to input the sample image into the skeleton segmentation model to be trained to obtain a sample prediction result output by the skeleton segmentation model;

[0140] A skeleton segmentation model training sub-module, which is used to iteratively train the skeleton segmentation model based on the target region, the skeleton line segments, and the sample prediction results to obtain the skeleton segmentation model.

[0141] Based on the above embodiments, optionally, the sample prediction results include a region prediction result corresponding to the target region and a skeleton prediction result corresponding to the skeleton line segments;

[0142] Correspondingly, the skeleton segmentation model training sub-module includes:

[0143] A first sample prediction probability determination unit, which is used to determine the first sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton prediction result, the target region, and the skeleton line segments for any round of iteration;

[0144] A second sample prediction probability determination unit, which is used to determine the second sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton line segments, the region prediction result, and the skeleton prediction result;

[0145] A skeleton segmentation model training unit, which is used to generate a model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability, and adjust the model parameters of the skeleton segmentation model based on the model loss function.

[0146] Based on the above embodiments, optionally, the first sample prediction probability determination unit includes:

[0147] A first sample prediction probability determination sub-unit, which is used to determine the first sample prediction probability based on the following expression; wherein, the expression includes:

[0148]

[0149] wherein, Recall represents the first sample prediction probability; S P represents the skeleton prediction result; V L represents the target region; S L represents the skeleton line segments;

[0150] Correspondingly, the second sample prediction probability determination unit includes:

[0151] A second sample prediction probability determination sub-unit, which is used to determine the second sample prediction probability based on the following expression; wherein, the expression includes:

[0152]

[0153] where Precision represents the predicted probability of the second sample; S L represents the skeleton line segment; V P represents the predicted result of the region; S P represents the predicted result of the skeleton;

[0154] Correspondingly, the model loss function determination unit includes:

[0155] A model loss function determination subunit, configured to determine the model loss function based on the following expression; wherein, the expression includes:

[0156]

[0157] where skeletonDice loss represents the model loss function; Recall represents the predicted probability of the first sample; Precision represents the predicted probability of the second sample.

[0158] The image segmentation device provided by the embodiments of the present invention can execute the image segmentation method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method.

[0159] Embodiment 4

[0160] Figure 7 FIG. shows a schematic structural diagram of an electronic device 10 that can be used to implement the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0161] As Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. The memory stores a computer program executable by the at least one processor. The processor 11 can execute various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0162] Multiple components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0163] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the image segmentation method.

[0164] In some embodiments, the image segmentation method can be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image segmentation method described above can be executed. Alternatively, in other embodiments, the processor can be configured to execute the image segmentation method in any other appropriate manner (e.g., by means of firmware).

[0165] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0166] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0167] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0168] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0169] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0170] A computing system can include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The client-server relationship is created by computer programs running on respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0171] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0172] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An image segmentation method, characterized in that, Including: Obtain an original image, and respectively obtain an initial segmentation image and a skeleton segmentation image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model; Perform a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image; Perform a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image; Generate a target segmentation image of the original image based on the extended connected region after the extension process of the skeleton connected line segments and the first processed image; The generating the target segmentation image of the original image based on the extended connected region after the extension process of the skeleton connected line segments and the first processed image includes: For any skeleton connected line segment, determine at least one extension parameter of the current skeleton connected line segment based on the region boundary of the first processed image; Perform an extension process on the skeleton connected line segment based on the extension parameter to obtain an extended connected region of the skeleton connected line segment; Perform a fusion process on the first processed image and the extended connected region to generate a target segmentation image of the original image; The extension parameter includes an endpoint extension parameter and a center extension parameter of the skeleton connected line segment; for any skeleton connected line segment, the endpoint extension parameter is determined based on the line segment endpoints of the current skeleton connected line segment and the boundary shapes of the respective region boundaries connected to the line segment endpoints; Correspondingly, the performing an extension process on the skeleton connected line segment based on the extension parameter to obtain an extended connected region of the skeleton connected line segment includes: Perform a linear fitting process on each of the endpoint extension data to generate at least one center point extension parameter of the current skeleton connected line segment; Generate an extended connected region of the skeleton connected line segment based on the endpoint extension parameter and each of the center extension parameters.

2. The method according to claim 1, wherein The performing a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image includes: Perform a subtraction process on the initial segmentation image and the skeleton segmentation image to determine at least one candidate skeleton line segment in the skeleton segmentation image; Based on the line segment endpoints of the segment candidate skeleton line segments and the region boundary of the first processed image, determine at least one skeleton connected line segment in the candidate region.

3. The method according to claim 1, wherein The training method of the skeleton segmentation model includes: Obtain a sample image for training the skeleton segmentation model, and the target region after annotation in the sample image, and perform a skeletonization process on the target region to obtain skeleton line segments of the target region; Input the sample image into the skeleton segmentation model to be trained to obtain a sample prediction result output by the skeleton segmentation model; Perform iterative training on the skeleton segmentation model based on the target region, the skeleton line segments, and the sample prediction result to obtain a skeleton segmentation model.

4. The method according to claim 3, wherein The sample prediction result includes a region prediction result corresponding to the target region and a skeleton prediction result corresponding to the skeleton line segments. Correspondingly, the iterative training of the skeleton segmentation model based on the target region, the skeleton line segments, and the sample prediction results includes: For any round of iteration, based on the skeleton prediction results, the target region, and the skeleton line segments, determine the first sample prediction probability of the skeleton segmentation model in the current iteration round; Based on the skeleton line segments, the region prediction results, and the skeleton prediction results, determine the second sample prediction probability of the skeleton segmentation model in the current iteration round; Generate the model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability, and adjust the model parameters of the skeleton segmentation model based on the model loss function.

5. The method according to claim 4, characterized in that The determination of the first sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton prediction results, the target region, and the skeleton line segments includes: Among them, Recall represents the predicted probability of the first sample; S P represents the predicted result of the skeleton; V L represents the target area; S L represents the skeleton line segment; Correspondingly, the determination of the second sample prediction probability of the skeleton segmentation model in the current iteration round based on the skeleton line segments, the region prediction results, and the skeleton prediction results includes: Among them, Precision represents the predicted probability of the second sample; S L represents the skeleton line segment; V P represents the region prediction result; S P represents the skeleton prediction result; Correspondingly, the generation of the model loss function of the skeleton segmentation model in the current round based on the first sample prediction probability and the second sample prediction probability includes: Wherein, skeletonDice loss represents the model loss function; Recall represents the first sample prediction probability; Precision represents the second sample prediction probability.

6. An image segmentation device, characterized in that, Includes: A segmentation image acquisition module, configured to acquire an original image, and respectively obtain an initial segmentation image and a skeleton segmentation image of the original image based on a pre-trained initial segmentation model and a skeleton segmentation model; A first processed image generation module, configured to perform a fusion process on the initial segmentation image and the skeleton segmentation image to generate a first processed image; A skeleton connected line segment acquisition module, configured to perform a subtraction process on the initial segmentation image and the skeleton segmentation image to generate skeleton connected line segments in the skeleton segmentation image; A target segmentation image generation module, configured to generate a target segmentation image of the original image based on the expanded connected region after the expansion process of the skeleton connected line segments and the first processed image; The target segmentation image generation module includes: An expansion parameter determination sub-module, configured to, for any skeleton connected line segment, determine at least one expansion parameter of the current skeleton connected line segment based on the first processed image; An expanded connected region generation sub-module, configured to perform an expansion process on the skeleton connected line segment based on the expansion parameter to obtain an expanded connected region of the skeleton connected line segment; A target segmentation image generation sub-module, configured to perform a fusion process on the first processed image and the expanded connected region to generate a target segmentation image of the original image; The expansion parameter includes an endpoint expansion parameter and a center expansion parameter of the skeleton connected line segment; for any skeleton connected line segment, the endpoint expansion parameter is determined based on the line segment endpoints of the current skeleton connected line segment and the boundary shapes of the respective region boundaries connected to the line segment endpoints; Correspondingly, the extended connected region generation sub-module includes: A central extension parameter determination unit, configured to perform linear fitting processing on each of the endpoint extension data to generate at least one central extension parameter of the current skeleton connected line segment; An extended connected region generation unit, configured to generate an extended connected region of the skeleton connected line segment based on the endpoint extension parameters and each of the central extension parameters.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image segmentation method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the image segmentation method according to any one of claims 1-5 when executed by a processor.

Images