Multi-model segmentation method, surgical robot and related products

Abstract

The application discloses a multi-model segmentation method, a surgical robot and related products. The method comprises the following steps: based on a first model and a second model, a first segmentation result of a first tissue of a target object and a second segmentation result of a second tissue of the target object are segmented from a first three-dimensional electronic computed tomography image; and then, based on the positions of the first segmentation result and the second segmentation result in the first three-dimensional CT image, the first segmentation result and the second segmentation result are combined to obtain a target segmentation result comprising the first tissue and the second tissue of the target object. The confidence of the target segmentation result can be improved by the method.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to a multi-model segmentation method, a surgical robot, and related products. Background Technology

[0002] After obtaining a three-dimensional CT image of the target object through computed tomography (CT), the tissue in the three-dimensional CT image can be segmented to obtain a three-dimensional segmentation result of the target object's tissue. However, the confidence level of the three-dimensional segmentation result obtained by current technology is low. Summary of the Invention

[0003] This application provides a multi-model segmentation method, a surgical robot, and related products. The related products include a multi-model segmentation device, electronic equipment, and computer-readable storage media to improve the confidence level of tissue segmentation of a target object.

[0004] Firstly, a multi-model segmentation method is provided, which is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target segmentation result includes a first tissue and a second tissue of the target object, and the method includes:

[0005] Acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object;

[0006] Extract the first region of interest (ROI) including the first tissue from the first 3D CT image.

[0007] Extract a second region of interest (ROI) including the second tissue from the first 3D CT image;

[0008] The first tissue in the first ROI is segmented using the first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue;

[0009] The second model is used to segment the second tissue in the second ROI to obtain a second segmentation result. The second model is used to segment the second tissue.

[0010] Based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first 3D CT image, the third position of the first segmentation result in the first 3D CT image is determined;

[0011] Based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first three-dimensional CT image, the sixth position of the second segmentation result in the first three-dimensional CT image is determined;

[0012] The target segmentation result is obtained based on the third position, the sixth position, the first segmentation result, and the second segmentation result. The position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0013] In one optional implementation, obtaining the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result includes:

[0014] Determine the first region corresponding to the third location from the first three-dimensional CT image;

[0015] Determine the second region corresponding to the sixth position from the first three-dimensional CT image;

[0016] In the case where there is a first intersection between the first region and the second region, a third segmentation result of the first intersection is determined based on the first segmentation result, and a fourth segmentation result of the first intersection is determined based on the second segmentation result;

[0017] If the third segmentation result is different from the fourth segmentation result, a first distance between the first intersection region and the center of the first region is determined, and a second distance between the first intersection region and the center of the second region is determined.

[0018] Based on the segmentation result corresponding to the minimum value of the first distance and the second distance, a fifth segmentation result of the first intersection region is determined;

[0019] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the segmentation results other than the segmentation results of the second segmentation result.

[0020] In one optional implementation, obtaining the target segmentation result based on the segmentation results in the first segmentation result excluding the segmentation results in the first intersection region, the fifth segmentation result, and the segmentation results in the second segmentation result excluding the segmentation results in the first intersection region includes:

[0021] If the fifth segmentation result is the third segmentation result, the fifth segmentation result and the second ROI are input into the second model, so that the second model segments the region in the second ROI other than the first intersection region based on the fifth segmentation result, and obtains the sixth segmentation result;

[0022] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result.

[0023] In one alternative implementation, the target object further includes a third organization;

[0024] Before obtaining the target segmentation result based on the segmentation results other than the segmentation results of the first segmentation result excluding the first intersection region, the fifth segmentation result, and the sixth segmentation result, the method further includes:

[0025] The seventh segmentation result is obtained at the seventh position in the first three-dimensional CT image. The seventh segmentation result is the segmentation result obtained by segmenting the third tissue in the first three-dimensional CT image using the third model.

[0026] Determine the third region corresponding to the seventh position from the first three-dimensional CT image;

[0027] In the case where there is a second intersection between the third region and the fourth region, an eighth segmentation result of the second intersection region is determined based on the second segmentation result, and a ninth segmentation result of the second intersection region is determined based on the seventh segmentation result, wherein the fourth region is the region in the second region other than the first intersection region;

[0028] If the eighth segmentation result differs from the ninth segmentation result, the tenth segmentation result of the second intersection region is determined based on the eighth segmentation result and the ninth segmentation result.

[0029] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result, including:

[0030] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, the tenth segmentation result, and the segmentation results other than the segmentation results of the fourth region in the sixth segmentation result.

[0031] In one optional implementation, determining the tenth segmentation result of the second intersection region based on the eighth segmentation result and the ninth segmentation result includes:

[0032] Determine a third distance between the second intersection region and the center of the second region, and determine a fourth distance between the second intersection region and the center of the third region;

[0033] If the difference between the third distance and the fourth distance is greater than a first threshold, the segmentation result corresponding to the minimum value of the third distance and the fourth distance is determined as the tenth segmentation result;

[0034] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, then the ninth segmentation result is determined to be the tenth segmentation result.

[0035] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, then the eighth segmentation result is determined to be the tenth segmentation result.

[0036] In one optional implementation, the step of extracting a first ROI including the first tissue from the first three-dimensional CT image includes:

[0037] The probability that the semantics of a pixel in the first 3D CT image is the first tissue is determined, thus obtaining at least one first probability;

[0038] At least one first candidate region is determined from the first three-dimensional CT image based on the at least one first probability, wherein the sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold.

[0039] Determine a first number of pixels in the at least one first candidate region;

[0040] The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and the reference size of the first tissue.

[0041] Secondly, a multi-model segmentation apparatus is provided, which is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target object includes a first tissue and a second tissue, and the multi-model segmentation apparatus includes:

[0042] The acquisition unit is used to acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object;

[0043] The cropping unit is used to crop a first ROI including the first tissue from the first three-dimensional CT image;

[0044] The cropping unit is further configured to crop a second ROI including the second tissue from the first three-dimensional CT image;

[0045] A segmentation unit is used to segment the first tissue in the first ROI using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue;

[0046] The segmentation unit is further configured to segment the second tissue in the second ROI using a second model to obtain a second segmentation result, wherein the second model is used to segment the second tissue;

[0047] The determining unit is configured to determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first ROI and a second position of the first ROI in the first three-dimensional CT image.

[0048] The determining unit is further configured to determine the sixth position of the second segmentation result in the first three-dimensional CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first three-dimensional CT image;

[0049] The processing unit is configured to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0050] In an optional implementation, the processing unit is further configured to:

[0051] If the fifth segmentation result is the third segmentation result, the fifth segmentation result and the second ROI are input into the second model, so that the second model segments the region in the second ROI other than the first intersection region based on the fifth segmentation result, and obtains the sixth segmentation result;

[0052] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result.

[0053] In one alternative implementation, the target object further includes a third organization;

[0054] The acquisition unit is further configured to acquire the seventh position of the seventh segmentation result in the first three-dimensional CT image, wherein the seventh segmentation result is a segmentation result obtained by segmenting the third tissue in the first three-dimensional CT image using the third model.

[0055] The determining unit is further configured to determine a third region corresponding to the seventh position from the first three-dimensional CT image;

[0056] The determining unit is further configured to, in the case that there is a second intersection region between the third region and the fourth region, determine an eighth segmentation result of the second intersection region based on the second segmentation result, and determine a ninth segmentation result of the second intersection region based on the seventh segmentation result, wherein the fourth region is the region in the second region other than the first intersection region;

[0057] The determining unit is further configured to determine the tenth segmentation result of the second intersection region based on the eighth segmentation result and the ninth segmentation result when the eighth segmentation result is different from the ninth segmentation result;

[0058] The processing unit is further configured to obtain the target segmentation result based on the segmentation results in the first segmentation result excluding the segmentation result of the first intersection region, the fifth segmentation result, the tenth segmentation result, and the sixth segmentation result excluding the segmentation result of the fourth region.

[0059] In an optional implementation, the determining unit is further configured to:

[0060] Determine a third distance between the second intersection region and the center of the second region, and determine a fourth distance between the second intersection region and the center of the third region;

[0061] If the difference between the third distance and the fourth distance is greater than a first threshold, the segmentation result corresponding to the minimum value of the third distance and the fourth distance is determined as the tenth segmentation result;

[0062] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, then the ninth segmentation result is determined to be the tenth segmentation result.

[0063] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, then the eighth segmentation result is determined to be the tenth segmentation result.

[0064] In an optional implementation, the interception unit is further configured to:

[0065] The probability that a pixel in the first 3D CT image represents the first tissue is determined, thus obtaining at least one first probability;

[0066] At least one first candidate region is determined from the first three-dimensional CT image based on the at least one first probability, wherein the sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold.

[0067] Determine a first number of pixels in the at least one first candidate region;

[0068] The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and the reference size of the first tissue.

[0069] Thirdly, a surgical robot is provided, including a multi-model segmentation device as described in the second aspect. In this third aspect, the surgical robot can perform a multi-model segmentation method using the multi-model segmentation device, thereby improving the confidence level of the target segmentation result.

[0070] Fourthly, an electronic device is provided, comprising: a processor and a memory, the memory for storing computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0071] Fifthly, another electronic device is provided, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method as described in the first aspect above and any possible implementation thereof.

[0072] In a sixth aspect, a computer-readable storage medium is provided, wherein a computer program is stored therein, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method as described in the first aspect above and any possible implementation thereof.

[0073] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, wherein, when the computer program or instructions are executed on a computer, the computer performs the method described in the first aspect and any possible implementation thereof.

[0074] In this embodiment, the first 3D CT image includes the target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce the interference of this image content on the segmentation of the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce the interference of this image content on the segmentation of the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue. Because the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, which can improve the confidence level of the first segmentation result. Furthermore, because the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, which can improve the confidence level of the second segmentation result.

[0075] Because the first segmentation result is obtained by segmenting the first Region of Interest (ROI), the position indicated by the first segmentation result is not the position in the first 3D CT image. Therefore, the segmentation device determines the third position of the first segmentation result in the first 3D CT image based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first 3D CT image. Similarly, the segmentation device determines the sixth position of the second segmentation result in the first 3D CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first 3D CT image.

[0076] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, the target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows for the segmentation of local regions in the first 3D CT image using the first and second models respectively, resulting in the first and second segmentation results, thus improving the confidence levels of both. Then, the positions of the first and second segmentation results within the first 3D CT image are determined so that they can be fused to obtain the target segmentation result. Finally, fusing the first and second segmentation results to obtain the target segmentation result further improves the confidence level of the target segmentation result. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0078] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0079] Figure 1 A flowchart illustrating a multi-model segmentation method provided in an embodiment of this application;

[0080] Figure 2 A flowchart illustrating another multi-model segmentation method provided in an embodiment of this application;

[0081] Figure 3 A schematic diagram of a target segmentation result provided in an embodiment of this application;

[0082] Figure 4 A schematic diagram illustrating another target segmentation result provided in an embodiment of this application;

[0083] Figure 5 This is a schematic diagram of the structure of a multi-model segmentation device provided in an embodiment of this application;

[0084] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0085] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0086] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0087] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. It should be understood that in this application, "at least one" means one or more, "more" means two or more, and "at least two" means two or three or more.

[0088] In the medical field, it is often necessary to use models to segment tissues in 3D CT images of target objects (such as humans) so that relevant personnel can perform corresponding processing based on the segmentation results. These models can be deep learning models or machine learning models, such as neural networks. However, there are many types of tissues within a target object. Using a single model to segment different types of tissues can easily lead to low confidence in the segmentation results. Therefore, this application provides a multi-model segmentation method that segments different types of tissues using different models. Then, based on the segmentation results of all models, the target segmentation result for the tissues within the target object is determined, which can improve the confidence of the target segmentation result.

[0089] In one alternative implementation, the target object includes a first tissue and a second tissue, for example, the first tissue is a lung and the second tissue is a kidney. A multi-model segmentation method is used to segment the target from a three-dimensional CT image to obtain a target segmentation result, wherein the target segmentation result includes the first tissue and the second tissue.

[0090] The multi-model segmentation method in this application embodiment is executed by a multi-model segmentation device (hereinafter referred to as the segmentation device). The segmentation device can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the segmentation device can be one of the following: a computer or a server.

[0091] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a multi-model segmentation method provided in an embodiment of this application.

[0092] 101. Acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object.

[0093] In this embodiment of the application, the target object can be a human being. The first three-dimensional CT image is a three-dimensional image obtained by performing a CT scan on the target object. The first three-dimensional CT image includes multiple tissues within the target object; for example, the target object is a human being. The first three-dimensional CT image includes the kidneys, lungs, blood vessels, bones, etc., within the human body.

[0094] In one implementation of acquiring a first three-dimensional CT image, the segmentation device receives the first three-dimensional CT image input by a user through input components. These input components include: a keyboard, a mouse, a touchscreen, a touchpad, and an audio input device.

[0095] In another implementation of acquiring the first 3D CT image, the segmentation device receives the first 3D CT image sent by the terminal. Optionally, the terminal can be any of the following: a mobile phone, a computer, a tablet computer, a server, or a wearable device.

[0096] 102. Extract a first ROI that includes the first tissue from the first three-dimensional CT image.

[0097] The first ROI is a portion of the first 3D CT image, i.e., the first ROI is an image region in the 3D CT image. In one possible implementation, the segmentation device determines the probability that a pixel in the first 3D CT image semantically represents a first tissue, obtaining at least one first probability. Based on the at least one first probability, at least one first candidate region is determined from the first 3D CT image, wherein the sum of the first probabilities of pixels in the at least one first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. A first number of pixels in the at least one first candidate region is determined. Based on the reference size of the first candidate region corresponding to the minimum value of the first number and the first tissue, the first ROI is cropped from the first 3D CT image.

[0098] In this implementation, the first probability in the at least one first probability corresponds one-to-one with a pixel in the first 3D CT image. For example, the first 3D CT image includes pixel a and pixel b, and the first tissue is the lung. The segmentation device determines the probability that pixel a is the lung as a first probability p1, and determines the probability that pixel b is the lung as a first probability p2. Therefore, at least one first probability includes both first probability p1 and first probability p2. The higher the first probability of a pixel, the higher the probability that the pixel is the first tissue, that is, the higher the probability that the pixel belongs to the region corresponding to the first tissue.

[0099] After obtaining at least one first probability, at least one first candidate region can be determined from the first 3D CT image based on the at least one first probability. Each of the at least one first candidate region is a portion of the first 3D CT image, i.e., each first candidate region is an image region within the 3D CT image. The sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. The fact that the sum of the first probabilities of the pixels in the first candidate region is greater than or equal to the second threshold indicates that the sum of the probabilities of the pixels in the first candidate region representing the first tissue is high, meaning that the confidence level of the first candidate region corresponding to the first tissue is high. The fact that the sum of the first probabilities of the pixels in the first candidate region is less than or equal to the third threshold indicates that the sum of the probabilities of the pixels in the first candidate region representing the first tissue is limited to not exceeding the third threshold, thus avoiding an excessive number of pixels in the first candidate region.

[0100] Optionally, the segmentation device randomly selects a pixel from the first 3D CT image as an initial region. If the sum of the first probabilities of the pixels in the initial region and the first probabilities of the pixels adjacent to the initial region is less than a second threshold, the pixels adjacent to the initial region are added to the initial region. If the sum of the first probabilities of the pixels in the initial region is greater than or equal to the second threshold and less than or equal to a third threshold, the initial region is determined as a first candidate region. If the sum of the first probabilities of the pixels in the initial region is greater than or equal to the third threshold, pixels at the edges are removed from the initial region until the sum of the first probabilities of the pixels in the initial region is greater than or equal to the second threshold and less than or equal to the third threshold. In this way, a first candidate region can be determined from the first 3D CT image, and by using different pixels in the first 3D CT image as different initial regions, multiple different first candidate regions can be obtained.

[0101] After determining at least one first candidate region, a first number of pixels within that first candidate region is determined. A smaller first number indicates a higher average first probability for the pixels within the first candidate region, thus indicating a higher confidence level that the first candidate region corresponds to the first tissue. Therefore, the segmentation device can improve the confidence level of the first ROI by extracting a first ROI from the first 3D CT image based on the first candidate region corresponding to the minimum first number and the reference size of the first tissue.

[0102] Optionally, the segmentation device determines a first ROI from the first three-dimensional CT image based on a reference size of a first candidate region and a first tissue, wherein the center of the first ROI is the center of the first candidate region, and the size of the first ROI is larger than the reference size.

[0103] In this implementation, after obtaining at least one first probability, the segmentation device first roughly screens and obtains at least one first candidate region with high confidence, where high confidence in the first candidate region means that the first candidate region is a region corresponding to the first tissue. Then, based on the first number of pixels in the first candidate regions, the region with the highest confidence (i.e., the first candidate region corresponding to the minimum value of the first number) is determined from the at least one first candidate region. Finally, based on the region with the highest confidence and the size of the first tissue, a first ROI is cropped from the first 3D CT image, which can improve the confidence of the first ROI.

[0104] In another possible implementation, the segmentation device extracts a first ROI from a first 3D CT image based on an algorithm for detecting ROIs. For example, the algorithm for detecting ROIs includes a first neural network, wherein the first neural network has the ability to detect the location of a first tissue in the 3D CT image.

[0105] 103. Extract a second ROI that includes the second tissue from the first three-dimensional CT image.

[0106] Similarly, the second probability in at least one of the above-mentioned second probabilities corresponds one-to-one with a pixel in the first 3D CT image. For example, the first 3D CT image includes pixel a, pixel b, and the second tissue is a kidney. The segmentation device determines the probability that pixel a's semantics are kidney as a second probability p3, and determines the probability that pixel b's semantics are kidney as a second probability p4. Then, at least one second probability includes the second probability p3 and the first probability p4. The higher the second probability of a pixel, the higher the probability that the pixel's semantics are the second tissue, that is, the higher the probability that the pixel belongs to the region corresponding to the second tissue.

[0107] In one possible implementation, the segmentation device determines the probability that a pixel in a first 3D CT image is semantically a second tissue, obtaining at least one second probability. Based on the at least one second probability, at least one second candidate region is determined from the first 3D CT image, wherein the sum of the second probabilities of pixels in the at least one second candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. A second number of pixels in the at least one second candidate region is determined. Based on the reference size of the second candidate region and the second tissue corresponding to the minimum value of the second number, a second ROI is cropped from the first 3D CT image.

[0108] In another possible implementation, the segmentation device extracts a second ROI from the first 3D CT image based on an algorithm for detecting ROIs. For example, the algorithm for detecting ROIs includes a second neural network, wherein the second neural network has the ability to detect the location of a second tissue in the 3D CT image.

[0109] 104. Use the first model to segment the first tissue in the first ROI to obtain the first segmentation result, wherein the first model is used to segment the first tissue.

[0110] In this embodiment, the first model is trained based on a first training dataset, wherein the training data in the first training dataset includes three-dimensional CT images, and the labeled data of the training data in the first training dataset includes the segmentation results of the first tissue in the three-dimensional CT images. That is to say, the first model has high accuracy in segmenting the first tissue.

[0111] The first segmentation result is used to indicate the location of the first tissue within the first Region of Interest (ROI). That is, the location indicated by the first segmentation result is the location within the first ROI. For example, if the location indicated by the first segmentation result is w1, then the region at location w1 within the first ROI is the first tissue. It should be understood that because the first segmentation result is obtained by segmenting the first ROI, the location indicated by the first segmentation result is not the location in the first 3D CT image. Therefore, based on the location indicated by the first segmentation result, the location of the first tissue in the first 3D CT image cannot be directly determined.

[0112] 105. Use the second model to segment the second tissue in the second ROI to obtain the second segmentation result, wherein the second model is used to segment the second tissue.

[0113] In this embodiment, the second model is trained based on a second training dataset, wherein the training data in the second training dataset includes 3D CT images, and the labeled data of the training data in the second training dataset includes the segmentation results of the second tissue in the 3D CT images. In other words, the second model has high accuracy in segmenting the second tissue.

[0114] The second segmentation result is used to indicate the location of the second tissue within the second ROI. That is, the location indicated by the second segmentation result is the location within the second ROI. For example, if the location indicated by the second segmentation result is w2, then the region at location w2 within the second ROI is the second tissue. It should be understood that because the second segmentation result is obtained by segmenting the second ROI, the location indicated by the second segmentation result is not the location in the first 3D CT image. Therefore, based on the location indicated by the second segmentation result, the location of the second tissue in the first 3D CT image cannot be directly determined.

[0115] 106. Based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first three-dimensional CT image, determine the third position of the first segmentation result in the first three-dimensional CT image.

[0116] As described in step 104, the location of the first tissue in the first 3D CT image cannot be directly determined based on the location indicated by the first segmentation result. Therefore, to determine the location of the first tissue in the first 3D CT image, it is necessary to determine the location of the first segmentation result in the first 3D CT image. Thus, the segmentation device determines the third location of the first segmentation result in the first 3D CT image by executing step 106. Specifically, the location of the first ROI in the first 3D CT image can be determined based on the second location, and the location of the first segmentation result in the first ROI can be determined based on the first location. Therefore, the location of the first segmentation result in the first 3D CT image, i.e., the third location, can be determined based on both the first and second locations.

[0117] 107. Based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first three-dimensional CT image, determine the sixth position of the second segmentation result in the first three-dimensional CT image.

[0118] As described in step 105, the location of the second tissue in the first 3D CT image cannot be directly determined based on the location indicated by the second segmentation result. Therefore, to determine the location of the second tissue in the first 3D CT image, it is necessary to determine the location of the second segmentation result in the first 3D CT image. Thus, the segmentation device determines the sixth position of the second segmentation result in the first 3D CT image by executing step 107. Specifically, the location of the second ROI in the first 3D CT image can be determined based on the fifth position, and the location of the second segmentation result in the second ROI can be determined based on the fourth position. Therefore, the location of the second segmentation result in the first 3D CT image, i.e., the sixth position, can be determined based on the fourth and fifth positions.

[0119] 108. Based on the third position, the sixth position, the first segmentation result, and the second segmentation result, the target segmentation result is obtained. The position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0120] In one possible implementation, the segmentation device merges the first segmentation result and the second segmentation result based on the third position and the sixth position to obtain the target segmentation result, wherein the position of the first segmentation result in the target segmentation result is the third position, and the position of the second segmentation result in the target segmentation result is the sixth position.

[0121] In this embodiment, the first 3D CT image includes the target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce the interference of this image content on the segmentation of the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce the interference of this image content on the segmentation of the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue. Because the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, which can improve the confidence level of the first segmentation result. Furthermore, because the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, which can improve the confidence level of the second segmentation result.

[0122] Because the first segmentation result is obtained by segmenting the first Region of Interest (ROI), the position indicated by the first segmentation result is not the position in the first 3D CT image. Therefore, the segmentation device determines the third position of the first segmentation result in the first 3D CT image based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first 3D CT image. Similarly, the segmentation device determines the sixth position of the second segmentation result in the first 3D CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first 3D CT image.

[0123] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, the target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows for the segmentation of local regions in the first 3D CT image using the first and second models respectively, resulting in the first and second segmentation results, thus improving the confidence levels of both. Then, the positions of the first and second segmentation results within the first 3D CT image are determined so that they can be fused to obtain the target segmentation result. Finally, fusing the first and second segmentation results to obtain the target segmentation result further improves the confidence level of the target segmentation result.

[0124] As an optional implementation, the segmentation device performs the following steps during step 108:

[0125] 201. Determine the first region corresponding to the third position from the first three-dimensional CT image.

[0126] The aforementioned first region is located at the third position in the first three-dimensional CT image.

[0127] 202. Determine the second region corresponding to the sixth position from the first three-dimensional CT image.

[0128] The aforementioned second region is located at the sixth position in the first three-dimensional CT image.

[0129] 203. In the case where there is a first intersection region between the first region and the second region, a third segmentation result of the first intersection region is determined based on the first segmentation result, and a fourth segmentation result of the first intersection region is determined based on the second segmentation result.

[0130] The first intersection region is the intersection region of the first region and the second region. The existence of the first intersection region indicates that the first segmentation result and the second segmentation result have an intersection. Specifically, both the first segmentation result and the second segmentation result include the segmentation result of the first intersection region. Therefore, the segmentation device can determine the third segmentation result of the first intersection region based on the first segmentation result, and determine the fourth segmentation result of the first intersection region based on the second segmentation result.

[0131] 204. In the case that the third segmentation result is different from the fourth segmentation result, determine the first distance between the center of the first intersection region and the center of the first region, and determine the second distance between the center of the first intersection region and the center of the second region.

[0132] The center of the first region can be the geometric center of the first region, and the center of the second region can be the geometric center of the second region. The first distance can be the distance between the geometric center of the first intersection region and the center of the first region, and the second distance can be the distance between the geometric center of the first intersection region and the center of the second region.

[0133] 205. Based on the segmentation result corresponding to the minimum value between the first distance and the second distance, determine the fifth segmentation result of the first intersection region.

[0134] Because the semantics of different pixels in the first region are correlated, the first model utilizes the semantics of other pixels in the first region when determining the segmentation result of a pixel in the first region. In other words, the confidence level of the semantics of other pixels affects the confidence level of the segmentation result of that pixel. For example, if the first region includes pixels s1 and s2, the first model utilizes the semantics of pixel s2 when determining the segmentation result of pixel s1 (i.e., determining the semantics of pixel s1). In this case, the higher the confidence level of the semantics of pixel s2, the higher the confidence level of the segmentation result of pixel s1 (i.e., the higher the confidence level of the semantics of pixel s1). Similarly, when determining the segmentation result of a pixel in the second region, the first model utilizes the semantics of other pixels in the second region. Furthermore, because the smaller the distance between two pixels, the higher the semantic correlation between the two pixels, and the higher the confidence level of the segmentation result at the center of the segmentation result, the closer a pixel in the first region is to the center of the first region, the higher the confidence level of the segmentation result of that pixel. Based on this, the segmentation device determines the fifth segmentation result for the first intersection region based on the segmentation result corresponding to the minimum value of the first region and the second distance, which can improve the confidence of the fifth segmentation result. Optionally, the segmentation device determines the segmentation result corresponding to the minimum value of the first distance and the second distance as the fifth segmentation result. For example, if the minimum value of the first distance and the second distance is the first distance, and the segmentation result corresponding to the first distance is the third segmentation result, then the fifth segmentation result is the third segmentation result. If the minimum value of the first distance and the second distance is the second distance, and the segmentation result corresponding to the second distance is the fourth segmentation result, then the fifth segmentation result is the fourth segmentation result.

[0135] In steps 205 and 206, the third segmentation result is different from the fourth segmentation result, indicating that there is a discrepancy between the third segmentation result and the fourth segmentation result. Therefore, the segmentation device determines the fifth segmentation result of the first intersection region by executing steps 205 and 206, thereby eliminating the discrepancy.

[0136] 206. Based on the segmentation results in the first segmentation result excluding the segmentation results in the first intersection region, the fifth segmentation result, and the second segmentation result excluding the segmentation results in the first intersection region, the target segmentation result is obtained.

[0137] In one possible implementation, the segmentation device merges the segmentation results in the first segmentation result excluding the segmentation results in the first intersection region, the fifth segmentation result, and the second segmentation result excluding the segmentation results in the first intersection region to obtain the target segmentation result.

[0138] In another possible implementation, if the fifth segmentation result is the same as the third segmentation result, the segmentation device inputs the fifth segmentation result and the second ROI into the second model. This allows the second model to segment the regions within the second ROI, excluding the first intersection region, based on the fifth segmentation result, thus obtaining a sixth segmentation result. The target segmentation result is then obtained based on the segmentation results from the first segmentation result (excluding the first intersection region), the fifth segmentation result, and the sixth segmentation result.

[0139] In this implementation, the fifth segmentation result is the same as the third segmentation result, indicating that the confidence level of the fourth segmentation result is low, which also means that the confidence level of the second segmentation result is low. Therefore, when the fifth segmentation result is the same as the third segmentation result, the segmentation device inputs the fifth segmentation result and the second ROI into the second model, so that the second model corrects the segmentation results of the regions in the second region other than the first intersection region based on the fifth segmentation result, to obtain the sixth segmentation result. Then, based on the segmentation results other than the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result, the target segmentation result is obtained, which can improve the confidence level of the target segmentation result.

[0140] Optionally, the target object may also include a third tissue, which is different from both the first and second tissues. For example, the first tissue is the kidney, the second tissue is the lung, and the third tissue is the bone.

[0141] Before performing the step of "obtaining the target segmentation result based on the segmentation results other than the segmentation results of the first segmentation result excluding the first intersection region, the fifth segmentation result, and the sixth segmentation result," the segmentation device further performs the following steps: Obtaining the seventh position of the seventh segmentation result in the first three-dimensional CT image, wherein the seventh segmentation result is obtained by segmenting the third tissue in the first three-dimensional CT image using the third model. Determining the third region corresponding to the seventh position from the first three-dimensional CT image. If the third region and the fourth region have a second intersection region, determining the eighth segmentation result of the second intersection region based on the second segmentation result, and determining the ninth segmentation result of the second intersection region based on the seventh segmentation result, wherein the fourth region is the region in the second region excluding the first intersection region. If the eighth segmentation result and the ninth segmentation result are different, determining the tenth segmentation result of the second intersection region based on the eighth and ninth segmentation results.

[0142] The seventh segmentation result can be obtained through the following steps: Extract a third region of interest (ROI) including the third tissue from the first 3D CT image. Segment the third tissue within the third ROI using the third model to obtain the seventh segmentation result. The position of the seventh segmentation result in the first 3D CT image is the seventh position, and the position of the third region in the first 3D CT image is also the seventh position. In other words, the seventh segmentation result is the segmentation result of the third region in the first 3D CT image.

[0143] The area outside the first intersection region in the second region is the fourth region. The third region and the fourth region have a second intersection region, indicating that the segmentation results corresponding to the fourth region in the seventh and second segmentation results intersect. Therefore, the segmentation device can determine the eighth segmentation result of the second intersection region based on the second segmentation result, and the ninth segmentation result of the second intersection region based on the seventh segmentation result. It should be understood that the second segmentation result includes the segmentation result of the fourth region, and the fourth region includes the second intersection region; therefore, the eighth segmentation result of the second intersection region can be determined based on the second segmentation result. The eighth segmentation result differs from the ninth segmentation result, indicating a discrepancy between them. Therefore, the segmentation device determines the tenth segmentation result of the second intersection region based on the eighth and ninth segmentation results, thereby eliminating this discrepancy.

[0144] After determining the tenth segmentation result of the second intersection region, the segmentation device performs the following steps during the execution of the step "obtain the target segmentation result based on the segmentation results other than the segmentation result of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result": Based on the segmentation results other than the segmentation result of the first intersection region in the first segmentation result, the fifth segmentation result, the tenth segmentation result, and the sixth segmentation result other than the segmentation result of the fourth region in the sixth segmentation result, the target segmentation result is obtained. This improves the confidence level of the target segmentation result.

[0145] Optionally, during the execution of step "determining the tenth segmentation result of the second intersection region based on the eighth and ninth segmentation results", the segmentation device performs the following steps: determining a third distance between the center of the second intersection region and the center of the second region, and determining a fourth distance between the center of the second intersection region and the center of the third region. If the difference between the third and fourth distances is greater than a first threshold, the segmentation result corresponding to the minimum of the third and fourth distances is determined as the tenth segmentation result. If the difference between the third and fourth distances is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, the ninth segmentation result is determined as the tenth segmentation result. If the difference between the third and fourth distances is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, the eighth segmentation result is determined as the tenth segmentation result.

[0146] In this step, if the difference between the third and fourth distances is greater than the first threshold, it indicates a significant difference between them. In this case, the confidence levels of the segmentation results corresponding to the third and fourth distances differ considerably. Therefore, the segmentation device determines the segmentation result corresponding to the minimum of the third and fourth distances as the tenth segmentation result, which improves the confidence level of the tenth segmentation result.

[0147] If the difference between the third and fourth distances is less than or equal to the first threshold, it indicates that the difference between the third and fourth distances is small. In this case, the difference in confidence between the segmentation results corresponding to the third and fourth distances is small. If the tenth segmentation result is determined based on the segmentation result corresponding to the minimum of the third and fourth distances, it is likely to result in a low confidence level for the tenth segmentation result. Therefore, the segmentation device determines the tenth segmentation result based on the fifth segmentation result.

[0148] Specifically, the third segmentation result is determined based on the first segmentation result, and the fourth segmentation result is determined based on the second segmentation result. Therefore, if the fifth segmentation result is the third segmentation result, it indicates that the confidence level of the first segmentation result is high, and correspondingly, the confidence level of the second segmentation result is low. Conversely, if the fifth segmentation result is the fourth segmentation result, it indicates that the confidence level of the second segmentation result is high, and correspondingly, the confidence level of the first segmentation result is low. Thus, when the fifth segmentation result is the third segmentation result, the segmentation device determines the tenth segmentation result based on the seventh segmentation result instead of the second, which can improve the confidence level of the tenth segmentation result. Conversely, when the fifth segmentation result is the fourth segmentation result, the segmentation device determines the tenth segmentation result based on the second segmentation result instead of the seventh, which can also improve the confidence level of the tenth segmentation result.

[0149] Since the eighth segmentation result is determined based on the second segmentation result, and the ninth segmentation result is determined based on the seventh segmentation result, the segmentation device determines the ninth segmentation result as the tenth segmentation result when the difference between the third and fourth distances is less than or equal to the first threshold and the fifth segmentation result is the third segmentation result. It also determines the eighth segmentation result as the tenth segmentation result when the difference between the third and fourth distances is less than or equal to the first threshold and the fifth segmentation result is the fourth segmentation result. This can improve the confidence of the tenth segmentation result.

[0150] For a better understanding of the multi-model segmentation method described above, please refer to [link / reference]. Figure 2 , Figure 2 This is a flowchart illustrating another multi-model segmentation method provided in an embodiment of this application. Figure 2As shown, a first 3D CT image is first acquired, and then input into an image preprocessing module to preprocess the image. Specifically, the image preprocessing module performs at least one of the following processes on the first 3D CT image: image resampling, window width and level adjustment, normalization, and cropping. Image resampling adjusts the size of the first 3D CT image to a preset size. Window width and level adjustment adjusts the grayscale values ​​of the pixels in the first 3D CT image to a first preset range. Normalization adjusts the pixel values ​​in the first 3D CT image to a second preset range. Cropping extracts Regions of Interest (ROIs) from the first 3D CT image, such as the first ROI, second ROI, and third ROI mentioned above. Optionally, the image preprocessing module can also convert the orientation of the first 3D CT image to a preset orientation. Optionally, the orientation information of the first 3D CT image is stored in either of the following formats: Digital Imaging and Communications in Medicine (DICOM) tag, or Neuroimaging Informatics Technology Initiative (NIfTI) header.

[0151] The image preprocessing module processes the first 3D CT image to obtain a preprocessed first 3D CT image. This preprocessed image is then input to the task configuration and scheduling module. Based on a preset multi-model collaborative execution process, the task configuration and scheduling module deploys the multi-model segmentation module to the processing unit of the segmentation device. The processing unit then runs the model used for tissue segmentation within the segmentation module. The processing unit includes a central processing unit (CPU) or a graphics processing unit (GPU). Optionally, the task configuration and scheduling module can dynamically deploy models from the multi-model segmentation module according to the available space in the processing unit of the segmentation device. This improves the utilization of the segmentation device's hardware resources and the running efficiency of the models in the multi-model segmentation module. For example, if the available memory on the GPU is sufficient to run the model, deploying the model on the GPU improves its running efficiency. If the available memory on the GPU is insufficient, deploying the model on the CPU fully utilizes the hardware resources of the segmentation device and improves segmentation efficiency.

[0152] Optionally, the preset multi-model collaborative execution flow includes an execution graph (EG), which includes the loading order of models in the multi-model segmentation module, the dependencies between models in the multi-model segmentation module, and the calling logic of models in the multi-model segmentation module. The dependencies between models in the multi-model segmentation module include serial and parallel relationships. For example, the preset multi-model collaborative execution flow includes: Step 1: Using a model for segmenting whole-body organs, segment the lungs from the 3D CT image. Step 2: Based on the lung segmentation results, extract the lung region from the 3D CT image, and call a model for segmenting blood vessels and organs in the lung to segment the blood vessels and trachea from the lung region.

[0153] exist Figure 2 The multi-model segmentation module includes multiple models for segmentation, such as the first, second, and third models mentioned earlier. Optionally, the multi-model segmentation module runs models based on the NVIDIA TensorRT engine. This leverages the NVIDIA TensorRT engine's dynamic batch processing mechanism and memory reuse strategy to reduce runtime latency and GPU memory usage, thereby meeting the requirements for high throughput and low latency. Optionally, when loading a model for the first time based on the NVIDIA TensorRT engine, the deep learning model in Open Neural Network Exchange (ONNX) format can be converted to a model in TensorRTengine format. Optionally, the files in the multi-model segmentation module are written in the C++ programming language.

[0154] After obtaining the segmentation results of different tissues of the target object based on the multi-model segmentation module, the segmentation results of different tissues can be aligned by the spatial resampling module to determine the position of the segmentation results of different tissues in the first three-dimensional CT image, which is beneficial for subsequent fusion of the segmentation results of different tissues. For example, the positions of the segmentation results of the first tissue and the segmentation results of the second tissue in the first three-dimensional CT image can be determined by steps 106 and 107.

[0155] After alignment by the spatial resampling module, the segmentation results of different tissues can be processed by the result fusion and discrepancy handling module to eliminate discrepancies. After discrepancy elimination, the segmentation results of different tissues are fused to obtain the target segmentation result. The discrepancy elimination process can be found in steps 201 to 206 described above.

[0156] Optionally, the result fusion and discrepancy handling module can also eliminate discrepancies through any of the following methods: priority control and confidence-weighted fusion, where priority control refers to eliminating discrepancies based on the priority of the organization. For example, if the priority of the first organization is higher than that of the second organization, and there is a discrepancy between the segmentation results of the first organization and the segmentation results of the second organization, then the segmentation result of the discrepancy region (such as the first intersection region and the second intersection region mentioned above) is determined based on the segmentation result of the first organization. For another example, the weight of the first organization is z1, and the weight of the second organization is z2. The semantics of determining the discrepancy region based on the segmentation result of the first organization is that the confidence level of the first organization is p5. The semantics of determining the discrepancy region based on the segmentation result of the second organization is that the confidence level of the second organization is p6. Then, the product of z1 and p5 can be determined to be c1, and the product of z2 and p6 can be determined to be c2. If c1 is greater than or equal to c2, then the segmentation result of the discrepancy region is determined based on the segmentation result of the first organization; if c1 is less than c2, then the segmentation result of the discrepancy region is determined based on the segmentation result of the second organization.

[0157] After obtaining the target segmentation result through the result fusion and divergence processing module, the format of the target segmentation result can be adjusted through the structured output module. Optionally, the structured output module adjusts the format of the target segmentation result to one of the following: NIfTI, DICOM-Segmentation (SEG), JSON, or eXtensible Markup Language (XML). Optionally, the structured output module adds at least one of the following information to the target segmentation result: the name of the tissue in the target segmentation result, the location of the tissue in the target segmentation result in the first 3D CT image, the volume of the tissue in the target segmentation result, and the tissue number in the target segmentation result.

[0158] Optionally, Figure 3 This is a schematic diagram illustrating a target segmentation result provided in an embodiment of this application. Figure 3 As shown, the target segmentation result includes multiple tissues within the target object. Figure 4 This is a schematic diagram illustrating another target segmentation result provided in an embodiment of this application. For example... Figure 4 As shown, the target segmentation results include the segmentation results of the lungs, trachea, blood vessels and tissues inside the lungs within the target object.

[0159] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0160] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, while using clear signs / information to inform users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0161] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0162] Please see Figure 5 , Figure 5 This is a schematic diagram of a multi-model segmentation device provided in an embodiment of this application. The multi-model segmentation device 1 is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target object includes a first tissue and a second tissue. The multi-model segmentation device 1 includes: an acquisition unit 11, a cropping unit 12, a segmentation unit 13, a determination unit 14, and a processing unit 15, wherein:

[0163] Acquisition unit 11 is used to acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object;

[0164] The cropping unit 12 is used to crop a first ROI including the first tissue from the first three-dimensional CT image;

[0165] The cropping unit 12 is further configured to crop a second ROI including the second tissue from the first three-dimensional CT image;

[0166] Segmentation unit 13 is used to segment the first tissue in the first ROI using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue;

[0167] The segmentation unit 13 is further configured to segment the second tissue in the second ROI using a second model to obtain a second segmentation result, wherein the second model is used to segment the second tissue;

[0168] The determining unit 14 is used to determine the third position of the first segmentation result in the first three-dimensional CT image based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first three-dimensional CT image.

[0169] The determining unit 14 is further configured to determine the sixth position of the second segmentation result in the first three-dimensional CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first three-dimensional CT image;

[0170] Processing unit 15 is configured to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0171] In an optional implementation, the processing unit 15 is further configured to:

[0172] If the fifth segmentation result is the third segmentation result, the fifth segmentation result and the second ROI are input into the second model, so that the second model segments the region in the second ROI other than the first intersection region based on the fifth segmentation result, and obtains the sixth segmentation result;

[0173] The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result.

[0174] In one alternative implementation, the target object further includes a third organization;

[0175] The acquisition unit 11 is further configured to acquire the seventh position of the seventh segmentation result in the first three-dimensional CT image, wherein the seventh segmentation result is a segmentation result obtained by segmenting the third tissue in the first three-dimensional CT image using the third model.

[0176] The determining unit 14 is further configured to determine a third region corresponding to the seventh position from the first three-dimensional CT image;

[0177] The determining unit 14 is further configured to, in the case that there is a second intersection region between the third region and the fourth region, determine an eighth segmentation result of the second intersection region based on the second segmentation result, and determine a ninth segmentation result of the second intersection region based on the seventh segmentation result, wherein the fourth region is the region in the second region other than the first intersection region;

[0178] The determining unit 14 is further configured to determine the tenth segmentation result of the second intersection region based on the eighth segmentation result and the ninth segmentation result when the eighth segmentation result is different from the ninth segmentation result;

[0179] The processing unit 15 is further configured to obtain the target segmentation result based on the segmentation results in the first segmentation result excluding the segmentation result of the first intersection region, the fifth segmentation result, the tenth segmentation result, and the sixth segmentation result excluding the segmentation result of the fourth region.

[0180] In an optional implementation, the determining unit 14 is further configured to:

[0181] Determine a third distance between the second intersection region and the center of the second region, and determine a fourth distance between the second intersection region and the center of the third region;

[0182] If the difference between the third distance and the fourth distance is greater than a first threshold, the segmentation result corresponding to the minimum value of the third distance and the fourth distance is determined as the tenth segmentation result;

[0183] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, then the ninth segmentation result is determined to be the tenth segmentation result.

[0184] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, then the eighth segmentation result is determined to be the tenth segmentation result.

[0185] In an optional implementation, the interception unit 12 is further configured to:

[0186] The probability that the semantics of a pixel in the first 3D CT image is the first tissue is determined, thus obtaining at least one first probability;

[0187] At least one first candidate region is determined from the first three-dimensional CT image based on the at least one first probability, wherein the sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold.

[0188] Determine a first number of pixels in the at least one first candidate region;

[0189] The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and the reference size of the first tissue.

[0190] In this embodiment, the first 3D CT image includes the target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce the interference of this image content on the segmentation of the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce the interference of this image content on the segmentation of the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue. Because the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, which can improve the confidence level of the first segmentation result. Furthermore, because the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, which can improve the confidence level of the second segmentation result.

[0191] Because the first segmentation result is obtained by segmenting the first Region of Interest (ROI), the position indicated by the first segmentation result is not the position in the first 3D CT image. Therefore, the segmentation device determines the third position of the first segmentation result in the first 3D CT image based on the first position of the first segmentation result in the first ROI and the second position of the first ROI in the first 3D CT image. Similarly, the segmentation device determines the sixth position of the second segmentation result in the first 3D CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first 3D CT image.

[0192] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, the target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows for the segmentation of local regions in the first 3D CT image using the first and second models respectively, resulting in the first and second segmentation results, thus improving the confidence levels of both. Then, the positions of the first and second segmentation results within the first 3D CT image are determined so that they can be fused to obtain the target segmentation result. Finally, fusing the first and second segmentation results to obtain the target segmentation result further improves the confidence level of the target segmentation result.

[0193] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0194] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0195] The processor 21 can be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, the processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0196] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0197] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0198] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. This embodiment of the application does not limit the specific data stored in the memory.

[0199] Understandable, Figure 6 This is merely a simplified design of an electronic device. In practical applications, the electronic device may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of this application are within the protection scope of this application.

[0200] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0201] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0203] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0205] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0206] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-model segmentation method, characterized in that, The multi-model segmentation method is used to segment a target object from a 3D CT image to obtain a target segmentation result, wherein the target segmentation result includes a first tissue and a second tissue of the target object, and the method includes: Acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object; A first region of interest, including the first tissue, is extracted from the first three-dimensional CT image; A second region of interest, including the second tissue, is extracted from the first three-dimensional CT image; The first tissue in the first region of interest is segmented using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; The second tissue in the second region of interest is segmented using a second model to obtain a second segmentation result. The second model is used to segment the second tissue. Based on the first position of the first segmentation result in the first region of interest and the second position of the first region of interest in the first 3D CT image, the third position of the first segmentation result in the first 3D CT image is determined; Based on the fourth position of the second segmentation result in the second region of interest and the fifth position of the second region of interest in the first 3D CT image, the sixth position of the second segmentation result in the first 3D CT image is determined; The target segmentation result is obtained based on the third position, the sixth position, the first segmentation result, and the second segmentation result. The position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

2. The multi-model segmentation method according to claim 1, characterized in that, The process of obtaining the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result includes: Determine the first region corresponding to the third location from the first three-dimensional CT image; Determine the second region corresponding to the sixth position from the first three-dimensional CT image; In the case where there is a first intersection between the first region and the second region, a third segmentation result of the first intersection is determined based on the first segmentation result, and a fourth segmentation result of the first intersection is determined based on the second segmentation result; If the third segmentation result is different from the fourth segmentation result, a first distance between the first intersection region and the center of the first region is determined, and a second distance between the first intersection region and the center of the second region is determined. Based on the segmentation result corresponding to the minimum value of the first distance and the second distance, a fifth segmentation result of the first intersection region is determined; The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the segmentation results other than the segmentation results of the second segmentation result.

3. The multi-model segmentation method according to claim 2, characterized in that, The step of obtaining the target segmentation result based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the segmentation results other than the segmentation results of the second segmentation result in the second segmentation result includes: If the fifth segmentation result is the third segmentation result, the fifth segmentation result and the second region of interest are input into the second model, so that the second model segments the region of interest other than the first intersection region based on the fifth segmentation result, and obtains the sixth segmentation result; The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result.

4. The multi-model segmentation method according to claim 3, characterized in that, The target also includes third-party organizations; Before obtaining the target segmentation result based on the segmentation results other than the segmentation results of the first segmentation result excluding the first intersection region, the fifth segmentation result, and the sixth segmentation result, the method further includes: The seventh segmentation result is obtained at the seventh position in the first three-dimensional CT image. The seventh segmentation result is the segmentation result obtained by segmenting the third tissue in the first three-dimensional CT image using the third model. Determine the third region corresponding to the seventh position from the first three-dimensional CT image; In the case where there is a second intersection between the third region and the fourth region, an eighth segmentation result of the second intersection region is determined based on the second segmentation result, and a ninth segmentation result of the second intersection region is determined based on the seventh segmentation result, wherein the fourth region is the region in the second region other than the first intersection region; If the eighth segmentation result differs from the ninth segmentation result, the tenth segmentation result of the second intersection region is determined based on the eighth segmentation result and the ninth segmentation result. The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, and the sixth segmentation result, including: The target segmentation result is obtained based on the segmentation results other than the segmentation results of the first intersection region in the first segmentation result, the fifth segmentation result, the tenth segmentation result, and the segmentation results other than the segmentation results of the fourth region in the sixth segmentation result.

5. The multi-model segmentation method according to claim 4, characterized in that, The step of determining the tenth segmentation result of the second intersection region based on the eighth segmentation result and the ninth segmentation result includes: Determine a third distance between the second intersection region and the center of the second region, and determine a fourth distance between the second intersection region and the center of the third region; If the difference between the third distance and the fourth distance is greater than a first threshold, the segmentation result corresponding to the minimum value of the third distance and the fourth distance is determined as the tenth segmentation result; If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, then the ninth segmentation result is determined to be the tenth segmentation result. If the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, then the eighth segmentation result is determined to be the tenth segmentation result.

6. The method according to any one of claims 1 to 5, characterized in that, Extracting a first region of interest, including the first tissue, from the first 3D CT image includes: The probability that the semantics of a pixel in the first 3D CT image is the first tissue is determined, thus obtaining at least one first probability; At least one first candidate region is determined from the first three-dimensional CT image based on the at least one first probability, wherein the sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. Determine a first number of pixels in the at least one first candidate region; Based on the first candidate region corresponding to the minimum value of the first number and the reference size of the first tissue, the first region of interest is extracted from the first three-dimensional CT image.

7. A multi-model segmentation device, characterized in that, The multi-model segmentation device is used to segment a target object from a three-dimensional CT image to obtain the target segmentation result. The target object includes a first tissue and a second tissue. The multi-model segmentation device includes: An acquisition unit is used to acquire a first three-dimensional CT image, wherein the first three-dimensional CT image includes the target object; The cropping unit is used to crop a first region of interest, including the first tissue, from the first three-dimensional CT image; The cropping unit is further configured to crop a second region of interest, including the second tissue, from the first three-dimensional CT image; A segmentation unit is used to segment the first tissue in the first region of interest using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; The segmentation unit is further configured to segment the second tissue in the second region of interest using a second model to obtain a second segmentation result, wherein the second model is used to segment the second tissue; The determining unit is configured to determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first region of interest and a second position of the first region of interest in the first three-dimensional CT image; The determining unit is further configured to determine the sixth position of the second segmentation result in the first three-dimensional CT image based on the fourth position of the second segmentation result in the second region of interest and the fifth position of the second region of interest in the first three-dimensional CT image; The processing unit is configured to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

8. A surgical robot, characterized in that, The surgical robot includes the multi-model segmentation device as described in claim 7.

9. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

Images