Multi-model segmentation method, surgical robot and related products

Abstract

The invention discloses a multi-model segmentation method, a surgical robot and a related product. The method comprises the following steps: based on a first model and a second model, segmenting a first three-dimensional computed tomography image to obtain a first segmentation result of a first tissue of a target object and a second segmentation result of a second tissue of the target object, and combining the first segmentation result and the second segmentation result based on the position of the first segmentation result in the first three-dimensional electronic computed tomography image and the position of the second segmentation result in the first three-dimensional CT image to obtain a target segmentation result comprising the first tissue and the second tissue of the target object. Through the method, the confidence of the target segmentation result can be improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a multi-model segmentation method, a surgical robot and related products. Background Art

[0002] After obtaining a 3D CT image of a target object through computed tomography (CT), tissue segmentation within the 3D CT image can be performed to obtain a 3D segmentation result of the target tissue. However, current technologies have low confidence levels in the 3D segmentation results. Summary of the Invention

[0003] The present application provides a multi-model segmentation method, a surgical robot and related products, wherein the related products include a multi-model segmentation device, an electronic device, and a computer-readable storage medium to improve the confidence of the segmentation of the target object's tissue.

[0004] In a first aspect, a multi-model segmentation method is provided. The multi-model segmentation method is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target segmentation result includes a first tissue of the target object and a second tissue of the target object. The method includes: Acquiring a first three-dimensional CT image, where the first three-dimensional CT image includes the target object; extracting a first region of interest (ROI) including the first tissue from the first three-dimensional CT image; extracting a second ROI including the second tissue from the first three-dimensional CT image; Segmenting the first tissue in the first ROI using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; Segmenting the second tissue in the second ROI using a second model to obtain a second segmentation result, wherein the second model is used to segment the second tissue; determining a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first ROI and a second position of the first ROI in the first three-dimensional CT image; determining a sixth position of the second segmentation result in the first three-dimensional CT image based on a fourth position of the second segmentation result in the second ROI and a fifth position of the second ROI in the first three-dimensional CT image; Based on the third position, the sixth position, the first segmentation result and the second segmentation result, the target segmentation result is obtained, the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0005] In an optional implementation, obtaining the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result includes: determining a first region corresponding to the third position from the first three-dimensional CT image; determining a second region corresponding to the sixth position from the first three-dimensional CT image; In a case where there is a first intersection area between the first area and the second area, determining a third segmentation result of the first intersection area based on the first segmentation result, and determining a fourth segmentation result of the first intersection area based on the second segmentation result; When the third segmentation result is different from the fourth segmentation result, determining a first distance between the first intersection area and the center of the first area, and determining a second distance between the first intersection area and the center of the second area; determining a fifth segmentation result of the first intersection area based on a segmentation result corresponding to a minimum value of the first distance and the second distance; The target segmentation result is obtained based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, and the segmentation results in the second segmentation results excluding the segmentation result of the first intersection area.

[0006] In an optional embodiment, obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the segmentation results other than the segmentation result of the first intersection area in the second segmentation results includes: When the fifth segmentation result is the third segmentation result, inputting the fifth segmentation result and the second ROI into the second model, so that the second model segments the region of the second ROI excluding the first intersection region based on the fifth segmentation result to obtain a sixth segmentation result; The target segmentation result is obtained based on the segmentation results in the first segmentation results except the segmentation result of the first intersection area, the fifth segmentation result, and the sixth segmentation result.

[0007] In an optional embodiment, the target object further includes a third tissue; Before obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the sixth segmentation result, the method further includes: obtaining a seventh segmentation result at a seventh position in the first three-dimensional CT image, where the seventh segmentation result is a segmentation result obtained by segmenting a third tissue in the first three-dimensional CT image using a third model; determining a third region corresponding to the seventh position from the first three-dimensional CT image; When there is a second intersection area between the third area and the fourth area, determining an eighth segmentation result of the second intersection area based on the second segmentation result, and determining a ninth segmentation result of the second intersection area based on the seventh segmentation result, wherein the fourth area is an area of ​​the second area excluding the first intersection area; When the eighth segmentation result is different from the ninth segmentation result, determining a tenth segmentation result for the second intersection area based on the eighth segmentation result and the ninth segmentation result; The obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the sixth segmentation result includes: The target segmentation result is obtained based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, the tenth segmentation result, and the segmentation results in the sixth segmentation result excluding the segmentation result of the fourth area.

[0008] In an optional implementation, determining a tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result includes: determining a third distance between the second intersection area and the center of the second area, and determining a fourth distance between the second intersection area and the center of the third area; When the difference between the third distance and the fourth distance is greater than a first threshold, determining a segmentation result corresponding to a minimum value between the third distance and the fourth distance as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, determining the ninth segmentation result as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, the eighth segmentation result is determined to be the tenth segmentation result.

[0009] In an optional embodiment, extracting a first ROI including the first tissue from the first three-dimensional CT image includes: Determining a probability that the semantics of a pixel in the first three-dimensional CT image is the first tissue, to obtain at least one first probability; determining at least one first candidate region from the first three-dimensional CT image based on the at least one first probability, wherein a sum of the first probabilities of pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold; determining a first number of pixels in the at least one first candidate region; The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and a reference size of the first tissue.

[0010] In a second aspect, a multi-model segmentation device is provided, wherein the multi-model segmentation device is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target object includes a first tissue and a second tissue, and the multi-model segmentation device includes: an acquiring unit, configured to acquire a first three-dimensional CT image, where the first three-dimensional CT image includes the target object; a cutting unit, configured to cut out a first ROI including the first tissue from the first three-dimensional CT image; The interception unit is further configured to intercept a second ROI including the second tissue from the first three-dimensional CT image; a segmentation unit, configured to segment the first tissue in the first ROI using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; The segmentation unit is further configured to segment the second tissue in the second ROI using a second model to obtain a second segmentation result, wherein the second model is configured to segment the second tissue; a determining unit, configured to determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first ROI and a second position of the first ROI in the first three-dimensional CT image; The determining unit is further configured to determine a sixth position of the second segmentation result in the first three-dimensional CT image based on a fourth position of the second segmentation result in the second ROI and a fifth position of the second ROI in the first three-dimensional CT image; A processing unit is used to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0011] In an optional implementation, the processing unit is further configured to: When the fifth segmentation result is the third segmentation result, inputting the fifth segmentation result and the second ROI into the second model, so that the second model segments the region of the second ROI excluding the first intersection region based on the fifth segmentation result to obtain a sixth segmentation result; The target segmentation result is obtained based on the segmentation results in the first segmentation results except the segmentation result of the first intersection area, the fifth segmentation result, and the sixth segmentation result.

[0012] In an optional embodiment, the target object further includes a third tissue; The acquiring unit is further configured to acquire a seventh position of a seventh segmentation result in the first three-dimensional CT image, where the seventh segmentation result is a segmentation result obtained by segmenting a third tissue in the first three-dimensional CT image using a third model; The determining unit is further configured to determine a third region corresponding to the seventh position from the first three-dimensional CT image; The determining unit is further configured to, when a second intersection area exists between the third area and the fourth area, determine an eighth segmentation result of the second intersection area based on the second segmentation result, and determine a ninth segmentation result of the second intersection area based on the seventh segmentation result, wherein the fourth area is an area of ​​the second area excluding the first intersection area; The determining unit is further configured to determine, when the eighth segmentation result is different from the ninth segmentation result, a tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result; The processing unit is further configured to obtain the target segmentation result based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, the tenth segmentation result, and the segmentation results in the sixth segmentation results excluding the segmentation result of the fourth area.

[0013] In an optional implementation, the determining unit is further configured to: determining a third distance between the second intersection area and the center of the second area, and determining a fourth distance between the second intersection area and the center of the third area; When the difference between the third distance and the fourth distance is greater than a first threshold, determining a segmentation result corresponding to a minimum value between the third distance and the fourth distance as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, determining the ninth segmentation result as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, the eighth segmentation result is determined to be the tenth segmentation result.

[0014] In an optional embodiment, the interception unit is further configured to: Determining a probability that the semantics of a pixel in the first three-dimensional CT image is the first tissue, to obtain at least one first probability; determining at least one first candidate region from the first three-dimensional CT image based on the at least one first probability, wherein a sum of the first probabilities of pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold; determining a first number of pixels in the at least one first candidate region; The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and a reference size of the first tissue.

[0015] In a third aspect, a surgical robot is provided, comprising the multi-model segmentation apparatus according to the second aspect. In the third aspect, the surgical robot can execute a multi-model segmentation method through the multi-model segmentation apparatus to improve the confidence of the target segmentation result.

[0016] In a fourth aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0017] In a fifth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.

[0018] In a sixth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any one of its possible implementation methods.

[0019] In a seventh aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.

[0020] In an embodiment of the present application, a first 3D CT image includes a target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce interference from this image content in segmenting the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue from the first 3D CT image. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce interference from this image content in segmenting the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue from the first 3D CT image. Since the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, thereby increasing the confidence level of the first segmentation result. Furthermore, since the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, thereby increasing the confidence level of the second segmentation result.

[0021] Because the first segmentation result is obtained by segmenting the first ROI, the location indicated by the first segmentation result is not a location in the first 3D CT image. Therefore, the segmentation device determines a third location of the first segmentation result in the first 3D CT image based on the first location of the first segmentation result in the first ROI and the second location of the first ROI in the first 3D CT image. Similarly, the segmentation device can determine a sixth location of the second segmentation result in the first 3D CT image based on the fourth location of the second segmentation result in the second ROI and the fifth location of the second ROI in the first 3D CT image.

[0022] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, a target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows the local region in the first 3D CT image to be segmented based on the first model and the second model, respectively, to obtain the first segmentation result and the second segmentation result, thereby improving the confidence of the first segmentation result and the confidence of the second segmentation result. The positions of the first and second segmentation results within the first 3D CT image are then determined, so that the first and second segmentation results can be fused to obtain the target segmentation result. Finally, the target segmentation result is obtained by fusing the first and second segmentation results, thereby improving the confidence of the target segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0024] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0025] Figure 1 A schematic diagram of a multi-model segmentation method provided in an embodiment of the present application; Figure 2 A schematic diagram of a flow chart of another multi-model segmentation method provided in an embodiment of the present application; Figure 3 A schematic diagram of a target segmentation result provided in an embodiment of the present application; Figure 4 A schematic diagram of another target segmentation result provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a multi-model segmentation device provided in an embodiment of the present application; Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0028] References to "embodiments" herein mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, that the embodiments described herein may be combined with other embodiments. It should be understood that, in this application, "at least one (item)" means one or more, "a plurality" means two or more, and "at least two (items)" means two or three or more.

[0029] In the medical field, it is usually necessary to use a model to segment the tissues in the three-dimensional CT image of the target object (such as a person) so that relevant personnel can perform corresponding processing based on the segmentation results. The model can be a deep learning model or a machine learning model, for example, the model is a neural network. However, there are many types of tissues in the target object. If one model is used to segment different types of models, it is easy to lead to low confidence in the segmentation results. Based on this, an embodiment of the present application provides a multi-model segmentation method, which segments different types of tissues through different models, and then determines the target segmentation results of the tissues in the target object based on the segmentation results of all models, which can improve the confidence of the target segmentation results.

[0030] In an optional embodiment, the target object includes a first tissue and a second tissue, for example, the first tissue is a lung and the second tissue is a kidney. The multi-model segmentation method is used to segment the target object from the three-dimensional CT image to obtain a target segmentation result, wherein the target segmentation result includes the first tissue and the second tissue.

[0031] The multi-model segmentation method in the embodiments of this application is performed by a multi-model segmentation device (hereinafter referred to as the segmentation device). The segmentation device can be any electronic device capable of executing the technical solutions disclosed in the embodiments of this application. Optionally, the segmentation device can be one of the following: a computer or a server.

[0032] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a multi-model segmentation method provided in an embodiment of the present application.

[0033] 101. Acquire a first three-dimensional CT image, where the first three-dimensional CT image includes a target object.

[0034] In the embodiment of the present application, the target object may be a human. The first 3D CT image is a 3D image obtained by performing a CT scan on the target object. The first 3D CT image includes multiple tissues within the target object. For example, the target object is a human. The first 3D CT image includes kidneys, lungs, blood vessels, bones, etc. within the human body.

[0035] In one implementation of acquiring the first three-dimensional CT image, the segmentation device receives the first three-dimensional CT image input by the user through an input component, which includes a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0036] In another implementation of acquiring the first three-dimensional CT image, the segmentation device receives the first three-dimensional CT image sent by a terminal. Optionally, the terminal can be any of the following: a mobile phone, a computer, a tablet computer, a server, or a wearable device.

[0037] 102. Extract a first ROI including a first tissue from the first three-dimensional CT image.

[0038] The first ROI is a portion of the first three-dimensional CT image, that is, the first ROI is an image region in the three-dimensional CT image. In one possible implementation, the segmentation device determines the probability that the semantics of a pixel in the first three-dimensional CT image is a first tissue, and obtains at least one first probability. Based on the at least one first probability, at least one first candidate region is determined from the first three-dimensional CT image, wherein the sum of the first probabilities of the pixels in the at least one first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. A first number of pixels in the at least one first candidate region is determined. Based on the reference size of the first candidate region and the first tissue corresponding to the minimum value of the first number, the first ROI is intercepted from the first three-dimensional CT image.

[0039] In this implementation, the first probability in the at least one first probability corresponds one-to-one to a pixel in the first three-dimensional CT image. For example, the first three-dimensional CT image includes pixel a and pixel b, and the first tissue is the lung. The segmentation device determines the probability that the semantic meaning of pixel a is lung as the first probability p1, and determines the probability that the semantic meaning of pixel b is lung as the first probability p2. Then, the at least one first probability includes the first probability p1 and the first probability p2. The higher the first probability of a pixel, the higher the probability that the semantic meaning of the pixel is the first tissue, that is, the higher the probability that the pixel belongs to the area corresponding to the first tissue.

[0040] After obtaining at least one first probability, at least one first candidate region can be determined from the first three-dimensional CT image based on the at least one first probability. Each of the at least one first candidate region is a portion of the first three-dimensional CT image, i.e., each first candidate region is an image region within the three-dimensional CT image. The sum of the first probabilities of the pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. If the sum of the first probabilities of the pixels in the first candidate region is greater than or equal to the second threshold, it indicates that the sum of the probabilities that the pixels in the first candidate region are semantically associated with the first tissue is high, which means that the confidence level that the first candidate region corresponds to the first tissue is high. If the sum of the first probabilities of the pixels in the first candidate region is less than or equal to the third threshold, it indicates that the sum of the probabilities that the pixels in the first candidate region are semantically associated with the first tissue is limited to not exceed the third threshold. This prevents an excessive number of pixels in the first candidate region.

[0041] Optionally, the segmentation device selects any pixel from the first three-dimensional CT image as the initial region. If the sum of the first probabilities of pixels in the initial region and the first probabilities of pixels adjacent to the initial region is less than a second threshold, the pixels adjacent to the initial region are added to the initial region. If the sum of the first probabilities of pixels in the initial region is greater than or equal to the second threshold and less than or equal to a third threshold, the initial region is determined to be a first candidate region. If the sum of the first probabilities of pixels in the initial region is greater than or equal to a third threshold, pixels at the edge are removed from the initial region until the sum of the first probabilities of pixels in the initial region is greater than or equal to the second threshold and less than or equal to the third threshold. In this way, the first candidate region can be determined from the first three-dimensional CT image, and by selecting different pixels in the first three-dimensional CT image as different initial regions, multiple different first candidate regions can be obtained.

[0042] After determining at least one first candidate region, a first number of pixels in the at least one first candidate region is determined. A smaller first number indicates a higher average first probability of the pixels in the first candidate region, and therefore a higher confidence level that the first candidate region corresponds to the first tissue. Therefore, the segmentation device extracts a first ROI from the first three-dimensional CT image based on the first candidate region corresponding to the minimum first number and the reference size of the first tissue, thereby improving the confidence level of the first ROI.

[0043] Optionally, the segmentation device determines a first ROI from the first three-dimensional CT image based on the first candidate region and a reference size of the first tissue, wherein the center of the first ROI is the center of the first candidate region and the size of the first ROI is larger than the reference size.

[0044] In this implementation, after obtaining at least one first probability, the segmentation device first performs a rough screening to obtain at least one first candidate region with a high confidence level. The high confidence level of the first candidate region means that the confidence level is high that the first candidate region corresponds to the first tissue. Then, based on the first number of pixels in the first candidate region, the region with the highest confidence level (i.e., the first candidate region corresponding to the minimum value of the first number) is determined from the at least one first candidate region. Finally, based on the region with the highest confidence level and the size of the first tissue, a first ROI is extracted from the first 3D CT image, thereby improving the confidence level of the first ROI.

[0045] In another possible implementation, the segmentation device extracts a first ROI from the first three-dimensional CT image based on an algorithm for detecting the ROI. For example, the algorithm for detecting the ROI includes a first neural network, wherein the first neural network has the ability to detect the position of the first tissue in the three-dimensional CT image.

[0046] 103. Extract a second ROI including a second tissue from the first three-dimensional CT image.

[0047] Similarly, the second probability in the at least one second probability corresponds one-to-one with a pixel in the first three-dimensional CT image. For example, the first three-dimensional CT image includes pixel a and pixel b, and the second tissue is the kidney. The segmentation device determines the probability that pixel a has the semantic meaning of kidney as second probability p3, and determines the probability that pixel b has the semantic meaning of kidney as second probability p4. Then, the at least one second probability includes second probability p3 and first probability p4. The higher the second probability of a pixel, the higher the probability that the pixel has the semantic meaning of the second tissue, that is, the higher the probability that the pixel belongs to the region corresponding to the second tissue.

[0048] In one possible implementation, a segmentation device determines a probability that a pixel in a first 3D CT image is semantically a second tissue, obtaining at least one second probability. Based on the at least one second probability, at least one second candidate region is determined from the first 3D CT image, wherein the sum of the second probabilities of pixels in the at least one second candidate region is greater than or equal to a second threshold and less than or equal to a third threshold. A second number of pixels in the at least one second candidate region is determined. Based on the second candidate region corresponding to the minimum value of the second number and a reference size of the second tissue, a second ROI is extracted from the first 3D CT image.

[0049] In another possible implementation, the segmentation device extracts the second ROI from the first three-dimensional CT image based on an algorithm for detecting the ROI. For example, the algorithm for detecting the ROI includes a second neural network, wherein the second neural network has the ability to detect the position of the second tissue in the three-dimensional CT image.

[0050] 104. Segment the first tissue in the first ROI using the first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue.

[0051] In this embodiment of the present application, the first model is trained based on a first training dataset, wherein the training data in the first training dataset includes 3D CT images, and the annotation data of the training data in the first training dataset includes segmentation results of the first tissue in the 3D CT images. In other words, the first model achieves a high level of segmentation accuracy for the first tissue.

[0052] The first segmentation result is used to indicate the location of the first tissue within the first ROI. That is, the location indicated by the first segmentation result is a location within the first ROI. For example, if the first segmentation result indicates location w1, then the region within the first ROI at location w1 is the first tissue. It should be understood that because the first segmentation result is obtained by segmenting the first ROI, the location indicated by the first segmentation result is not a location within the first 3D CT image. In other words, the location of the first tissue within the first 3D CT image cannot be directly determined based on the location indicated by the first segmentation result.

[0053] 105. Use the second model to segment the second tissue in the second ROI to obtain a second segmentation result, wherein the second model is used to segment the second tissue.

[0054] In this embodiment of the present application, the second model is trained based on a second training dataset, wherein the training data in the second training dataset includes 3D CT images, and the annotation data of the training data in the second training dataset includes segmentation results of the second tissue in the 3D CT images. In other words, the second model achieves a high level of segmentation accuracy for the second tissue.

[0055] The second segmentation result is used to indicate the location of the second tissue within the second ROI. That is, the location indicated by the second segmentation result is the location within the second ROI. For example, if the second segmentation result indicates location w2, then the region within the second ROI at location w2 is the second tissue. It should be understood that because the second segmentation result is obtained by segmenting the second ROI, the location indicated by the second segmentation result is not the location within the first 3D CT image. In other words, the location of the second tissue within the first 3D CT image cannot be directly determined based on the location indicated by the second segmentation result.

[0056] 106. Determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first ROI and a second position of the first ROI in the first three-dimensional CT image.

[0057] As described in step 104, the position of the first tissue in the first 3D CT image cannot be directly determined based on the position indicated by the first segmentation result. Therefore, in order to determine the position of the first tissue in the first 3D CT image, it is necessary to determine the position of the first segmentation result in the first 3D CT image. The segmentation device then performs step 106 to determine a third position of the first segmentation result in the first 3D CT image. Specifically, the position of the first ROI in the first 3D CT image can be determined based on the second position, while the position of the first segmentation result in the first ROI can be determined based on the first position. Therefore, the position of the first segmentation result in the first 3D CT image, i.e., the third position, can be determined based on the first and second positions.

[0058] 107. Determine a sixth position of the second segmentation result in the first three-dimensional CT image based on the fourth position of the second segmentation result in the second ROI and the fifth position of the second ROI in the first three-dimensional CT image.

[0059] As described in step 105, the position of the second tissue in the first 3D CT image cannot be directly determined based on the position indicated by the second segmentation result. Therefore, to determine the position of the second tissue in the first 3D CT image, it is necessary to determine the position of the second segmentation result in the first 3D CT image. The segmentation device then performs step 107 to determine the sixth position of the second segmentation result in the first 3D CT image. Specifically, the position of the second ROI in the first 3D CT image can be determined based on the fifth position, while the position of the second segmentation result in the second ROI can be determined based on the fourth position. Therefore, based on the fourth and fifth positions, the position of the second segmentation result in the first 3D CT image, i.e., the sixth position, can be determined.

[0060] 108. Obtain a target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0061] In one possible implementation, the segmentation device merges the first segmentation result and the second segmentation result based on the third position and the sixth position to obtain a target segmentation result, wherein the position of the first segmentation result in the target segmentation result is the third position, and the position of the second segmentation result in the target segmentation result is the sixth position.

[0062] In an embodiment of the present application, a first 3D CT image includes a target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce interference from this image content in segmenting the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue from the first 3D CT image. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce interference from this image content in segmenting the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue from the first 3D CT image. Since the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, thereby increasing the confidence level of the first segmentation result. Furthermore, since the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, thereby increasing the confidence level of the second segmentation result.

[0063] Because the first segmentation result is obtained by segmenting the first ROI, the location indicated by the first segmentation result is not a location in the first 3D CT image. Therefore, the segmentation device determines a third location of the first segmentation result in the first 3D CT image based on the first location of the first segmentation result in the first ROI and the second location of the first ROI in the first 3D CT image. Similarly, the segmentation device can determine a sixth location of the second segmentation result in the first 3D CT image based on the fourth location of the second segmentation result in the second ROI and the fifth location of the second ROI in the first 3D CT image.

[0064] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, a target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows the local region in the first 3D CT image to be segmented based on the first model and the second model, respectively, to obtain the first segmentation result and the second segmentation result, thereby improving the confidence of the first segmentation result and the confidence of the second segmentation result. The positions of the first and second segmentation results within the first 3D CT image are then determined, so that the first and second segmentation results can be fused to obtain the target segmentation result. Finally, the target segmentation result is obtained by fusing the first and second segmentation results, thereby improving the confidence of the target segmentation result.

[0065] As an optional implementation manner, the segmentation device performs the following steps during the execution of step 108: 201. Determine a first region corresponding to a third position from a first three-dimensional CT image.

[0066] The position of the first region in the first three-dimensional CT image is a third position.

[0067] 202. Determine a second region corresponding to a sixth position from the first three-dimensional CT image.

[0068] The position of the second region in the first three-dimensional CT image is the sixth position.

[0069] 203. When there is a first intersection area between the first area and the second area, determine a third segmentation result of the first intersection area based on the first segmentation result, and determine a fourth segmentation result of the first intersection area based on the second segmentation result.

[0070] The first intersection area is the intersection area of ​​the first area and the second area, and the existence of the first intersection area indicates that the first segmentation result and the second segmentation result have an intersection. Specifically, the first segmentation result and the second segmentation result both include the segmentation result of the first intersection area. Therefore, the segmentation device can determine the third segmentation result of the first intersection area based on the first segmentation result, and determine the fourth segmentation result of the first intersection area based on the second segmentation result.

[0071] 204. When the third segmentation result is different from the fourth segmentation result, determine a first distance between the first intersection area and the center of the first area, and determine a second distance between the first intersection area and the center of the second area.

[0072] The center of the first region may be the geometric center of the first region, the center of the second region may be the geometric center of the second region, the first distance may be the distance between the geometric center of the first intersection region and the center of the first region, and the second distance may be the distance between the geometric center of the first intersection region and the center of the second region.

[0073] 205. Determine a fifth segmentation result of the first intersection area based on the segmentation result corresponding to the minimum value of the first distance and the second distance.

[0074] Because the semantics of different pixels in the first region are correlated, the first model leverages the semantics of other pixels in the first region when determining the segmentation result for a pixel in the first region. That is, the confidence of the semantics of other pixels influences the confidence of the segmentation result for that pixel. For example, if the first region includes pixels s1 and s2, the first model leverages the semantics of pixel s2 when determining the segmentation result for pixel s1 (i.e., determining the semantics of pixel s1). In this case, the higher the confidence of the semantics of pixel s2, the higher the confidence of the segmentation result for pixel s1 (i.e., the higher the confidence of the semantics of pixel s1). Similarly, the first model leverages the semantics of other pixels in the second region when determining the segmentation result for a pixel in the second region. Furthermore, because the smaller the distance between two pixels, the higher the semantic correlation between the two pixels, and the confidence of the segmentation result at the center of the segmentation result is generally higher, the closer a pixel in a region is to the center of the first region, the higher the confidence of the segmentation result for that pixel. Based on this, the segmentation device determines the fifth segmentation result for the first intersection area based on the segmentation result corresponding to the minimum value between the first area and the second distance, thereby improving the confidence of the fifth segmentation result. Optionally, the segmentation device determines the segmentation result corresponding to the minimum value between the first distance and the second distance as the fifth segmentation result. For example, if the minimum value between the first distance and the second distance is the first distance, and the segmentation result corresponding to the first distance is the third segmentation result, the fifth segmentation result is the third segmentation result. If the minimum value between the first distance and the second distance is the second distance, and the segmentation result corresponding to the second distance is the fourth segmentation result, the fifth segmentation result is the fourth segmentation result.

[0075] In step 205 and step 206, the third segmentation result is different from the fourth segmentation result, indicating that there is a disagreement between the third segmentation result and the fourth segmentation result. Therefore, the segmentation device determines the fifth segmentation result of the first intersection area by executing step 205 and step 206, thereby eliminating the disagreement.

[0076] 206 . Obtain a target segmentation result based on the segmentation results in the first segmentation result excluding the segmentation result of the first intersection area, the fifth segmentation result, and the segmentation results in the second segmentation result excluding the segmentation result of the first intersection area.

[0077] In one possible implementation, the segmentation device merges the segmentation results in the first segmentation result except the segmentation result of the first intersection area, the fifth segmentation result, and the segmentation results in the second segmentation result except the segmentation result of the first intersection area to obtain a target segmentation result.

[0078] In another possible implementation, when the fifth segmentation result is the same as the third segmentation result, the segmentation device inputs the fifth segmentation result and the second ROI into the second model, so that the second model segments the region of the second ROI excluding the first intersection region based on the fifth segmentation result to obtain a sixth segmentation result. A target segmentation result is obtained based on the segmentation results of the first segmentation result excluding the segmentation result of the first intersection region, the fifth segmentation result, and the sixth segmentation result.

[0079] In this implementation, the fifth segmentation result is the third segmentation result, indicating that the confidence level of the fourth segmentation result is low, which in turn indicates that the confidence level of the second segmentation result is low. Therefore, if the fifth segmentation result is the third segmentation result, the segmentation device inputs the fifth segmentation result and the second ROI into the second model, so that the second model corrects the segmentation results of the second region excluding the first intersection region based on the fifth segmentation result, thereby obtaining a sixth segmentation result. The target segmentation result is then obtained based on the segmentation results of the first segmentation result excluding the segmentation result of the first intersection region, the fifth segmentation result, and the sixth segmentation result, thereby increasing the confidence level of the target segmentation result.

[0080] Optionally, the target object further includes a third tissue, wherein the third tissue is different from the first tissue and the second tissue, for example, the first tissue is the kidney, the second tissue is the lung, and the third tissue is the bone.

[0081] Before executing the step of "obtaining a target segmentation result based on the segmentation results of the first segmentation result other than the segmentation result of the first intersection region, the fifth segmentation result, and the sixth segmentation result," the segmentation device further executes the following steps: obtaining a seventh segmentation result at the seventh position in the first three-dimensional CT image, wherein the seventh segmentation result is a segmentation result obtained by segmenting a third tissue in the first three-dimensional CT image using a third model. Determining a third region corresponding to the seventh position in the first three-dimensional CT image. If a second intersection region exists between the third region and the fourth region, determining an eighth segmentation result for the second intersection region based on the second segmentation result, and determining a ninth segmentation result for the second intersection region based on the seventh segmentation result, wherein the fourth region is the region of the second region other than the first intersection region. If the eighth segmentation result differs from the ninth segmentation result, determining a tenth segmentation result for the second intersection region based on the eighth and ninth segmentation results.

[0082] The seventh segmentation result can be obtained by the following steps: extracting a third ROI including the third tissue from the first 3D CT image. Segmenting the third tissue in the third ROI using the third model to obtain a seventh segmentation result. The seventh segmentation result is located at the seventh position in the first 3D CT image, and the third region is located at the seventh position in the first 3D CT image. That is, the seventh segmentation result is a segmentation result of the third region in the first 3D CT image.

[0083] The area other than the first intersection area in the second area is the fourth area. There is a second intersection area between the third area and the fourth area, which means that there is an intersection between the seventh segmentation result and the segmentation result corresponding to the fourth area in the second segmentation result. Therefore, the segmentation device can determine the eighth segmentation result of the second intersection area based on the second segmentation result, and determine the ninth segmentation result of the second intersection area based on the seventh segmentation result. It should be understood that the second segmentation result includes the segmentation result of the fourth area, and the fourth area includes the second intersection area, so the eighth segmentation result of the second intersection area can be determined based on the second segmentation result. The eighth segmentation result is different from the ninth segmentation result, which means that there is a disagreement between the eighth segmentation result and the ninth segmentation result. Therefore, the segmentation device determines the tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result, thereby eliminating the disagreement.

[0084] After determining the tenth segmentation result for the second intersection region, the segmentation device, during the step of "obtaining a target segmentation result based on the segmentation results in the first segmentation results excluding the segmentation result for the first intersection region, the fifth segmentation result, and the sixth segmentation result," performs the following steps: obtaining the target segmentation result based on the segmentation results in the first segmentation results excluding the segmentation result for the first intersection region, the fifth segmentation result, the tenth segmentation result, and the segmentation results in the sixth segmentation results excluding the segmentation result for the fourth region. This improves the confidence level of the target segmentation result.

[0085] Optionally, the segmentation device performs the following steps during the process of executing the step of "determining the tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result": determining a third distance between the second intersection area and the center of the second area, and determining a fourth distance between the second intersection area and the center of the third area. When the difference between the third distance and the fourth distance is greater than a first threshold, determining the segmentation result corresponding to the minimum of the third distance and the fourth distance as the tenth segmentation result. When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, determining the ninth segmentation result as the tenth segmentation result. When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, determining the eighth segmentation result as the tenth segmentation result.

[0086] In this step, the difference between the third distance and the fourth distance is greater than the first threshold, indicating that the difference between the third distance and the fourth distance is significant. In this case, the confidence level of the segmentation result corresponding to the third distance is significantly different from the confidence level of the segmentation result corresponding to the fourth distance. Therefore, the segmentation device determines the segmentation result corresponding to the minimum of the third and fourth distances as the tenth segmentation result, thereby increasing the confidence level of the tenth segmentation result.

[0087] If the difference between the third distance and the fourth distance is less than or equal to the first threshold, it indicates that the difference between the third distance and the fourth distance is small. In this case, the difference between the confidence level of the segmentation result corresponding to the third distance and the confidence level of the segmentation result corresponding to the fourth distance is small. If the tenth segmentation result is determined based on the segmentation result corresponding to the minimum of the third and fourth distances, the confidence level of the tenth segmentation result is likely to be low. Therefore, the segmentation device determines the tenth segmentation result based on the fifth segmentation result.

[0088] Specifically, the third segmentation result is determined based on the first segmentation result, and the fourth segmentation result is determined based on the second segmentation result. Therefore, if the fifth segmentation result is the third segmentation result, the confidence of the first segmentation result is high, and accordingly, the confidence of the second segmentation result is low. Conversely, if the fifth segmentation result is the fourth segmentation result, the confidence of the second segmentation result is high, and accordingly, the confidence of the first segmentation result is low. Therefore, if the fifth segmentation result is the third segmentation result, the segmentation device determines the tenth segmentation result based on the seventh segmentation result instead of the second segmentation result, thereby improving the confidence of the tenth segmentation result. Conversely, if the fifth segmentation result is the fourth segmentation result, the segmentation device determines the tenth segmentation result based on the second segmentation result instead of the seventh segmentation result, thereby improving the confidence of the tenth segmentation result.

[0089] Because the eighth segmentation result is determined based on the second segmentation result and the ninth segmentation result is determined based on the seventh segmentation result, the segmentation device determines the ninth segmentation result as the tenth segmentation result when the difference between the third distance and the fourth distance is less than or equal to the first threshold and the fifth segmentation result is the third segmentation result. When the difference between the third distance and the fourth distance is less than or equal to the first threshold and the fifth segmentation result is the fourth segmentation result, the segmentation device determines the eighth segmentation result as the tenth segmentation result, thereby improving the confidence of the tenth segmentation result.

[0090] To better understand the multi-model segmentation method described above, please refer to Figure 2 , Figure 2 This is a flow chart of another multi-model segmentation method provided in an embodiment of the present application. Figure 2As shown, a first three-dimensional CT image is first acquired, and then the first three-dimensional CT image is input into an image preprocessing module so that the image preprocessing module preprocesses the first three-dimensional CT image. Specifically, the image preprocessing module is used to perform at least one of the following processing on the first three-dimensional CT image: image resampling, window width and window position adjustment, normalization, and cropping. Image resampling is used to adjust the size of the first three-dimensional CT image to a preset size. Window width and window position adjustment is used to adjust the grayscale values ​​of pixels in the first three-dimensional CT image to within a first preset range. Normalization is used to adjust the pixel values ​​in the first three-dimensional CT image to within a second preset range. Cropping is used to intercept ROIs from the first three-dimensional CT image, such as the first ROI, second ROI, and third ROI described above. Optionally, the image preprocessing module can also be used to convert the direction of the first three-dimensional CT image to a preset direction. Optionally, the orientation information of the first three-dimensional CT image is stored in any one of the following formats: a Digital Imaging and Communications in Medicine (DICOM) tag or a Neuroimaging Informatics Technology Initiative (NIfTI) header.

[0091] After the image preprocessing module processes the first three-dimensional CT image, a preprocessed first three-dimensional CT image is obtained. The preprocessed first three-dimensional CT image is then input into the task configuration and scheduling module, so that the task configuration and scheduling module deploys the multi-model segmentation module to a processing unit of the segmentation device based on a preset multi-model collaborative execution process, so that the processing unit executes the model used for tissue segmentation in the segmentation module. The processing unit includes a central processing unit (CPU) or a graphics processing unit (GPU). Optionally, the task configuration and scheduling module can dynamically deploy the models in the multi-model segmentation module based on the available space in the processing unit of the segmentation device, thereby improving hardware resource utilization of the segmentation device and improving the operating efficiency of the models in the multi-model segmentation module. For example, if the available memory of the GPU is sufficient to run the model, deploying the model on the GPU can improve the operating efficiency of the model. If the available memory of the GPU is insufficient to run the model, deploying the model on the CPU can both fully utilize the hardware resources of the segmentation device and improve segmentation efficiency.

[0092] Optionally, the preset multi-model collaborative execution process includes an execution graph (EG), which includes the loading order of the models in the multi-model segmentation module, the dependency relationship of the models in the multi-model segmentation module, and the calling logic of the models in the multi-model segmentation module, wherein the dependency relationship of the models in the multi-model segmentation module includes a series relationship and a parallel relationship. For example, the preset multi-model collaborative execution process includes: Step 1: Use the model for segmenting whole body organs to segment the lung segmentation results from the three-dimensional CT image. Step 2: Based on the lung segmentation results, cut out the lung area from the three-dimensional CT image, and call the model for segmenting the blood vessels and organs of the lung to segment the blood vessels and trachea of ​​the lung from the lung area.

[0093] exist Figure 2 In [1], the multi-model segmentation module includes multiple models for segmenting tissues, such as the first, second, and third models mentioned above. Optionally, the multi-model segmentation module runs models based on the NVIDIA TensorRT engine. This allows the NVIDIA TensorRT engine's dynamic batching mechanism and memory reuse strategy to reduce runtime latency and video memory usage, thereby meeting high throughput and low latency requirements. Optionally, when the model is first loaded based on the NVIDIA TensorRT engine, a deep learning model in the Open Neural Network Exchange (ONNX) format can be converted to a model in the TensorRT engine format. Optionally, the files in the multi-model segmentation module are written in the C++ programming language.

[0094] After obtaining segmentation results for different tissues of the target object using the multi-model segmentation module, the spatial resampling module can be used to align the segmentation results of the different tissues to determine their positions within the first 3D CT image, thereby facilitating the subsequent fusion of the segmentation results of the different tissues. For example, steps 106 and 107 can determine the positions of the segmentation results of the first tissue and the segmentation results of the second tissue within the first 3D CT image.

[0095] After alignment by the spatial resampling module, the results fusion and divergence processing module can be used to eliminate the differences in the segmentation results of different tissues. After the differences are eliminated, the segmentation results of different tissues are fused to obtain the target segmentation result. The divergence elimination process can be seen in steps 201 to 206 described above.

[0096] Optionally, the result fusion and divergence handling module can also resolve divergences through any of the following methods: priority control or confidence-weighted fusion. Priority control refers to resolving divergences based on the priorities of the tissues. For example, if the priority of the first tissue is higher than that of the second tissue, and the segmentation results of the first tissue and the second tissue diverge, the segmentation results of the diverging regions (such as the first intersection region and the second intersection region described above) are determined based on the segmentation results of the first tissue. For another example, the weight of the first tissue is z1, and the weight of the second tissue is z2. The confidence level p5 for the semantics of the diverging regions determined based on the segmentation results of the first tissue is determined to be the first tissue. The confidence level p6 for the semantics of the diverging regions determined based on the segmentation results of the second tissue is determined to be the second tissue. The product of z1 and p5 is determined to be c1, and the product of z2 and p6 is determined to be c2. If c1 is greater than or equal to c2, the segmentation results of the diverging regions are determined based on the segmentation results of the first tissue. If c1 is less than c2, the segmentation results of the diverging regions are determined based on the segmentation results of the second tissue.

[0097] After obtaining the target segmentation result through the result fusion and divergence processing module, the structured output module can adjust the format of the target segmentation result. Optionally, the structured output module adjusts the format of the target segmentation result to one of the following: NIfTI, DICOM Segmentation (SEG), JSON, or eXtensible Markup Language (XML). Optionally, the structured output module adds at least one of the following information to the target segmentation result: the name of the tissue in the target segmentation result, the location of the tissue in the target segmentation result in the first three-dimensional CT image, the volume of the tissue in the target segmentation result, and the number of the tissue in the target segmentation result.

[0098] Optionally, Figure 3 This is a schematic diagram of a target segmentation result provided in an embodiment of the present application. Figure 3 As shown, the target segmentation result includes multiple tissues in the target object. Figure 4 This is a schematic diagram of another target segmentation result provided in an embodiment of the present application. Figure 4 As shown, the target segmentation results include the segmentation results of the lungs, trachea, blood vessels and tissues inside the lungs in the target object.

[0099] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0100] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0101] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.

[0102] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of a multi-model segmentation device provided in an embodiment of the present application. The multi-model segmentation device 1 is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target object includes a first tissue and a second tissue. The multi-model segmentation device 1 includes: an acquisition unit 11, a truncation unit 12, a segmentation unit 13, a determination unit 14, and a processing unit 15, wherein: An acquiring unit 11 is configured to acquire a first three-dimensional CT image, where the first three-dimensional CT image includes the target object; a cutting unit 12, configured to cut out a first ROI including the first tissue from the first three-dimensional CT image; The interception unit 12 is further configured to intercept a second ROI including the second tissue from the first three-dimensional CT image; a segmentation unit 13, configured to segment the first tissue in the first ROI using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; The segmentation unit 13 is further configured to segment the second tissue in the second ROI using a second model to obtain a second segmentation result, wherein the second model is configured to segment the second tissue; a determining unit 14, configured to determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first ROI and a second position of the first ROI in the first three-dimensional CT image; The determining unit 14 is further configured to determine a sixth position of the second segmentation result in the first three-dimensional CT image based on a fourth position of the second segmentation result in the second ROI and a fifth position of the second ROI in the first three-dimensional CT image; The processing unit 15 is used to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

[0103] In an optional implementation, the processing unit 15 is further configured to: When the fifth segmentation result is the third segmentation result, inputting the fifth segmentation result and the second ROI into the second model, so that the second model segments the region of the second ROI excluding the first intersection region based on the fifth segmentation result to obtain a sixth segmentation result; The target segmentation result is obtained based on the segmentation results in the first segmentation results except the segmentation result of the first intersection area, the fifth segmentation result, and the sixth segmentation result.

[0104] In an optional embodiment, the target object further includes a third tissue; The acquiring unit 11 is further configured to acquire a seventh position of a seventh segmentation result in the first three-dimensional CT image, where the seventh segmentation result is a segmentation result obtained by segmenting a third tissue in the first three-dimensional CT image using a third model; The determining unit 14 is further configured to determine a third region corresponding to the seventh position from the first three-dimensional CT image; The determining unit 14 is further configured to, when a second intersection area exists between the third area and the fourth area, determine an eighth segmentation result of the second intersection area based on the second segmentation result, and determine a ninth segmentation result of the second intersection area based on the seventh segmentation result, wherein the fourth area is an area of ​​the second area excluding the first intersection area; The determining unit 14 is further configured to determine a tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result when the eighth segmentation result is different from the ninth segmentation result; The processing unit 15 is further configured to obtain the target segmentation result based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, the tenth segmentation result, and the segmentation results in the sixth segmentation results excluding the segmentation result of the fourth area.

[0105] In an optional implementation, the determining unit 14 is further configured to: determining a third distance between the second intersection area and the center of the second area, and determining a fourth distance between the second intersection area and the center of the third area; When the difference between the third distance and the fourth distance is greater than a first threshold, determining a segmentation result corresponding to a minimum value between the third distance and the fourth distance as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, determining the ninth segmentation result as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, the eighth segmentation result is determined to be the tenth segmentation result.

[0106] In an optional embodiment, the interception unit 12 is further configured to: Determining a probability that the semantics of a pixel in the first three-dimensional CT image is the first tissue, to obtain at least one first probability; determining at least one first candidate region from the first three-dimensional CT image based on the at least one first probability, wherein a sum of the first probabilities of pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold; determining a first number of pixels in the at least one first candidate region; The first ROI is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and a reference size of the first tissue.

[0107] In an embodiment of the present application, a first 3D CT image includes a target object. Since the first 3D CT image includes a significant amount of image content other than the first tissue, to reduce interference from this image content in segmenting the first tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a first ROI including the first tissue from the first 3D CT image. Similarly, since the first 3D CT image includes a significant amount of image content other than the second tissue, to reduce interference from this image content in segmenting the second tissue in the first 3D CT image, the segmentation device, after acquiring the first 3D CT image, extracts a second ROI including the second tissue from the first 3D CT image. Since the first model is used to segment the first tissue, the segmentation device uses the first model to segment the first tissue in the first ROI, obtaining a first segmentation result, thereby increasing the confidence level of the first segmentation result. Furthermore, since the second model is used to segment the second tissue, the segmentation device uses the second model to segment the second tissue in the second ROI, obtaining a second segmentation result, thereby increasing the confidence level of the second segmentation result.

[0108] Because the first segmentation result is obtained by segmenting the first ROI, the location indicated by the first segmentation result is not a location in the first 3D CT image. Therefore, the segmentation device determines a third location of the first segmentation result in the first 3D CT image based on the first location of the first segmentation result in the first ROI and the second location of the first ROI in the first 3D CT image. Similarly, the segmentation device can determine a sixth location of the second segmentation result in the first 3D CT image based on the fourth location of the second segmentation result in the second ROI and the fifth location of the second ROI in the first 3D CT image.

[0109] Finally, based on the third position, the sixth position, the first segmentation result, and the second segmentation result, a target segmentation result can be obtained. The position of the first segmentation result within the target segmentation result is determined based on the third position, and the position of the second segmentation result within the target segmentation result is determined based on the sixth position. This allows the local region in the first 3D CT image to be segmented based on the first model and the second model, respectively, to obtain the first segmentation result and the second segmentation result, thereby improving the confidence of the first segmentation result and the confidence of the second segmentation result. The positions of the first and second segmentation results within the first 3D CT image are then determined, so that the first and second segmentation results can be fused to obtain the target segmentation result. Finally, the target segmentation result is obtained by fusing the first and second segmentation results, thereby improving the confidence of the target segmentation result.

[0110] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0111] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, connection through various interfaces, transmission lines, buses, etc.

[0112] The processor 21 may be one or more graphics processing units (GPUs). If the processor 21 is a GPU, the GPU may be a single-core GPU or a multi-core GPU. Alternatively, the processor 21 may be a processor group consisting of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Alternatively, the processor may be another type of processor, and the present embodiment is not limiting.

[0113] The memory 22 can be used to store computer program instructions and various computer program codes, including program codes for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used for related instructions and data.

[0114] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0115] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.

[0116] It is understandable that Figure 6 Only a simplified design of an electronic device is shown. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the scope of protection of the present application.

[0117] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of this application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0120] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0122] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0123] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium. When executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A multi-model segmentation method, characterized in that: The multi-model segmentation method is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target segmentation result includes a first tissue of the target object and a second tissue of the target object. The method includes: Acquiring a first three-dimensional CT image, where the first three-dimensional CT image includes the target object; intercepting a first region of interest including the first tissue from the first three-dimensional CT image; intercepting a second region of interest including the second tissue from the first three-dimensional CT image; Segmenting the first tissue in the first region of interest using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; segmenting the second tissue in the second region of interest using a second model to obtain a second segmentation result, wherein the second model is used to segment the second tissue; determining a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first region of interest and a second position of the first region of interest in the first three-dimensional CT image; determining a sixth position of the second segmentation result in the first three-dimensional CT image based on a fourth position of the second segmentation result in the second region of interest and a fifth position of the second region of interest in the first three-dimensional CT image; Based on the third position, the sixth position, the first segmentation result and the second segmentation result, the target segmentation result is obtained, the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

2. The multi-model segmentation method according to claim 1, characterized in that The obtaining the target segmentation result based on the third position, the sixth position, the first segmentation result, and the second segmentation result includes: determining a first region corresponding to the third position from the first three-dimensional CT image; determining a second region corresponding to the sixth position from the first three-dimensional CT image; In a case where there is a first intersection area between the first area and the second area, determining a third segmentation result of the first intersection area based on the first segmentation result, and determining a fourth segmentation result of the first intersection area based on the second segmentation result; When the third segmentation result is different from the fourth segmentation result, determining a first distance between the first intersection area and the center of the first area, and determining a second distance between the first intersection area and the center of the second area; determining a fifth segmentation result of the first intersection area based on a segmentation result corresponding to a minimum value of the first distance and the second distance; The target segmentation result is obtained based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, and the segmentation results in the second segmentation results excluding the segmentation result of the first intersection area.

3. The multi-model segmentation method according to claim 2, characterized in that The obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the segmentation results other than the segmentation result of the first intersection area in the second segmentation results includes: When the fifth segmentation result is the third segmentation result, inputting the fifth segmentation result and the second region of interest into the second model, so that the second model segments the region of the second region of interest excluding the first intersection region based on the fifth segmentation result to obtain a sixth segmentation result; The target segmentation result is obtained based on the segmentation results in the first segmentation results except the segmentation result of the first intersection area, the fifth segmentation result, and the sixth segmentation result.

4. The multi-model segmentation method according to claim 3, characterized in that The target objects also include third organizations; Before obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the sixth segmentation result, the method further includes: obtaining a seventh segmentation result at a seventh position in the first three-dimensional CT image, where the seventh segmentation result is a segmentation result obtained by segmenting a third tissue in the first three-dimensional CT image using a third model; determining a third region corresponding to the seventh position from the first three-dimensional CT image; When there is a second intersection area between the third area and the fourth area, determining an eighth segmentation result of the second intersection area based on the second segmentation result, and determining a ninth segmentation result of the second intersection area based on the seventh segmentation result, wherein the fourth area is an area of ​​the second area excluding the first intersection area; When the eighth segmentation result is different from the ninth segmentation result, determining a tenth segmentation result for the second intersection area based on the eighth segmentation result and the ninth segmentation result; The obtaining the target segmentation result based on the segmentation results other than the segmentation result of the first intersection area in the first segmentation results, the fifth segmentation result, and the sixth segmentation result includes: The target segmentation result is obtained based on the segmentation results in the first segmentation results excluding the segmentation result of the first intersection area, the fifth segmentation result, the tenth segmentation result, and the segmentation results in the sixth segmentation result excluding the segmentation result of the fourth area.

5. The multi-model segmentation method according to claim 4, characterized in that: Determining a tenth segmentation result of the second intersection area based on the eighth segmentation result and the ninth segmentation result includes: determining a third distance between the second intersection area and the center of the second area, and determining a fourth distance between the second intersection area and the center of the third area; When the difference between the third distance and the fourth distance is greater than a first threshold, determining a segmentation result corresponding to a minimum value between the third distance and the fourth distance as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the third segmentation result, determining the ninth segmentation result as the tenth segmentation result; When the difference between the third distance and the fourth distance is less than or equal to the first threshold, and the fifth segmentation result is the fourth segmentation result, the eighth segmentation result is determined to be the tenth segmentation result.

6. The method according to any one of claims 1 to 5, characterized in that The step of intercepting a first region of interest including the first tissue from the first three-dimensional CT image comprises: Determining a probability that the semantics of a pixel in the first three-dimensional CT image is the first tissue, to obtain at least one first probability; determining at least one first candidate region from the first three-dimensional CT image based on the at least one first probability, wherein a sum of the first probabilities of pixels in each first candidate region is greater than or equal to a second threshold and less than or equal to a third threshold; determining a first number of pixels in the at least one first candidate region; The first region of interest is extracted from the first three-dimensional CT image based on the first candidate region corresponding to the minimum value of the first number and a reference size of the first tissue.

7. A multi-model segmentation device, characterized in that: The multi-model segmentation device is used to segment a target object from a three-dimensional CT image to obtain a target segmentation result, wherein the target object includes a first tissue and a second tissue. The multi-model segmentation device includes: an acquiring unit, configured to acquire a first three-dimensional CT image, where the first three-dimensional CT image includes the target object; a cutting unit, configured to cut out a first region of interest including the first tissue from the first three-dimensional CT image; The interception unit is further configured to intercept a second region of interest including the second tissue from the first three-dimensional CT image; a segmentation unit, configured to segment the first tissue in the first region of interest using a first model to obtain a first segmentation result, wherein the first model is used to segment the first tissue; The segmentation unit is further configured to segment the second tissue in the second region of interest using a second model to obtain a second segmentation result, wherein the second model is configured to segment the second tissue; a determining unit, configured to determine a third position of the first segmentation result in the first three-dimensional CT image based on a first position of the first segmentation result in the first region of interest and a second position of the first region of interest in the first three-dimensional CT image; The determining unit is further configured to determine a sixth position of the second segmentation result in the first three-dimensional CT image based on a fourth position of the second segmentation result in the second region of interest and a fifth position of the second region of interest in the first three-dimensional CT image; A processing unit is used to obtain the target segmentation result based on the third position, the sixth position, the first segmentation result and the second segmentation result, wherein the position of the first segmentation result in the target segmentation result is determined based on the third position, and the position of the second segmentation result in the target segmentation result is determined based on the sixth position.

8. A surgical robot, characterized in that: The surgical robot includes the multi-model segmentation device as described in claim 7.

9. An electronic device, characterized in that: include: A processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images