Liver and gall part segmentation method and device based on deep learning, equipment and medium

Abstract

The invention discloses a liver and gall part segmentation method and device based on deep learning, equipment and a medium, and belongs to the technical field of artificial intelligence. In the embodiment of the invention, the three deep learning models are used to carry out three-stage image segmentation on the original image containing the liver and gall part, and the position of each human tissue structure of the liver and gall part in the medical image is automatically identified and segmented. Surgical navigation is provided for clinicians during hepatobiliary disease and surgical planning, the deep learning technology is applied, the method is more accurate and faster, the method can be suitable for disease diagnosis, surgical formulation and radiotherapy plans of clinicians in more departments, the applicability of the method is improved, and the method has general clinical value and is worthy of popularization and application. The three-stage segmentation module can realize more accurate segmentation, can give consideration to global information and local information, better conforms to the reading habit of clinicians, and improves the segmentation precision and accuracy.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for liver and gallbladder segmentation based on deep learning. Background Art

[0002] Liver cancer, also referred to as "primary liver cancer", mainly includes three different pathological types: hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC) and combined hepatocellular-cholangiocarcinoma (cHCC-CCA). It can be seen that liver cancer seriously affects people's lives and health, but most of the liver cancer patients in my country are in the middle and late stages (70%) when they seek medical treatment, showing symptoms such as large intrahepatic tumor load, high probability of portal vein cancer thrombus, and poor liver function. Therefore, early detection and early treatment are the key to improving the treatment of liver cancer.

[0003] The continuous development of medical imaging technology has greatly improved the diagnosis and treatment of liver cancer. Traditional 3D reconstruction technology mainly relies on manual segmentation or semi-automatic segmentation methods, which are not only time-consuming and labor-intensive, but also may have inaccurate segmentation problems for complex hepatobiliary system structures. Summary of the invention

[0004] The embodiment of the present application provides a method, device, equipment and medium for segmenting the liver and gallbladder based on deep learning, which can achieve the effect of improving segmentation accuracy, fast and accurate. The technical solution is as follows:

[0005] On the one hand, a method for segmenting liver and gallbladder based on deep learning is provided, the method comprising:

[0006] Based on the original image including the liver and gallbladder, a first image is acquired;

[0007] Inputting the first image into a multi-organ segmentation model, and segmenting the first image by the multi-organ segmentation model to obtain a first segmentation result, wherein the first segmentation result is used to indicate the positions of the liver and gallbladder and the surrounding multiple organs, bones, and tubular structures in the first image;

[0008] inputting the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, and the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, the hepatic vein, the portal vein, and the dilated bile duct in the liver region image;

[0009] inputting the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, and having the liver tumor segmentation model perform liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, wherein the third segmentation result is used to indicate a position of the liver tumor in the first image;

[0010] Three-dimensional reconstruction is performed based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, and the standard template library file supports rendering and display in multiple software.

[0011] In some embodiments, performing three-dimensional reconstruction based on the first segmentation result, the second segmentation result, and the third segmentation result includes:

[0012] In response to the original image comprising multiple multi-modal and / or multi-phase medical images, the multiple medical images and each organ tissue in the liver and gallbladder region in each segmentation result of the multiple medical images are registered, and each registered segmentation result is three-dimensionally reconstructed.

[0013] In some embodiments, the registering of each organ tissue in the liver and gallbladder region in the plurality of medical images and each segmentation result of the plurality of medical images comprises:

[0014] rigidly registering a plurality of medical images with a target image;

[0015] Perform deformation registration on multiple rigidly registered medical images and the target image to obtain multiple registered images;

[0016] The respective segmentation results of the multiple medical images are rigidly registered and deformably registered according to the registration data of the multiple medical images to serve as the segmentation results of the multiple registered images.

[0017] In some embodiments, the method further comprises:

[0018] In response to the original images comprising a plurality of multi-modality and / or multi-phase medical images, displaying the plurality of original images;

[0019] In response to receiving a selection instruction for any original image, the original image is used as a target image to perform registration on other original images.

[0020] In some embodiments, the registering of each organ tissue in the liver and gallbladder region in the plurality of medical images and each segmentation result of the plurality of medical images comprises:

[0021] The multiple medical images and the segmentation results of the multiple medical images are input into a medical image registration and fusion module, and the medical image registration and fusion module registers each organ tissue in the liver and gallbladder area in the multiple medical images and the segmentation results of the multiple medical images.

[0022] In some embodiments, the three-dimensional reconstruction is performed based on the first segmentation result, the second segmentation result, and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder, including:

[0023] generating a surface mesh based on voxel data of images in the first segmentation result, the second segmentation result, and the third segmentation result;

[0024] Smoothing the surface mesh;

[0025] The hole filling algorithm is used to fill the smoothed surface mesh;

[0026] Save the filled surface mesh as a standard template library file.

[0027] In some embodiments, the training process of the multi-organ segmentation model is implemented based on a first loss function, and the training process of the intra-liver segmentation model is implemented based on a second loss function, and the first loss function and / or the second loss function are obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation;

[0028] The training process of the liver tumor segmentation model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a third loss function, wherein the third loss function is obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation.

[0029] In some embodiments, an attention mechanism is introduced into the multi-organ segmentation model and / or the intra-liver segmentation model, and the attention mechanism is used to segment any image in an input image sequence based on data of other images in the image sequence, and / or the attention mechanism is used to segment any pixel in any image based on data of other pixels in the image.

[0030] On the one hand, a deep learning-based liver and gallbladder segmentation device is provided, the device comprising:

[0031] An acquisition module, used for acquiring a first image based on an original image containing a liver and gallbladder portion;

[0032] A first segmentation module, used for inputting the first image into a multi-organ segmentation model, and having the multi-organ segmentation model segment the first image to obtain a first segmentation result, wherein the first segmentation result is used for indicating the positions of the liver and gallbladder and the surrounding multiple organs, bones and tubular structures in the first image;

[0033] a second segmentation module, configured to input the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, wherein the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, the hepatic vein, the portal vein, and the dilated bile duct in the liver region image;

[0034] a third segmentation module, configured to input the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, wherein the liver tumor segmentation model performs liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, wherein the third segmentation result is used to indicate a position of the liver tumor in the first image;

[0035] A reconstruction module is used to perform three-dimensional reconstruction based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, wherein the standard template library file supports rendering and display in multiple software.

[0036] In some embodiments, the reconstruction module is used to align each organ tissue in the liver and gallbladder region in the multiple medical images and each segmentation result of the multiple medical images in response to the original image including multiple multi-modal and / or multi-phase medical images, and perform three-dimensional reconstruction of each segmentation result after the alignment.

[0037] In some embodiments, the reconstruction module is used to:

[0038] rigidly registering a plurality of medical images with a target image;

[0039] Perform deformation registration on the multiple rigidly registered medical images and the target image to obtain multiple registered images;

[0040] The respective segmentation results of the multiple medical images are rigidly registered and deformably registered according to the registration data of the multiple medical images to serve as the segmentation results of the multiple registered images.

[0041] In some embodiments, the apparatus further comprises:

[0042] A display module, configured to display the plurality of original images in response to the original images comprising a plurality of multi-modality and / or multi-phase medical images;

[0043] The reconstruction module is used for responding to receiving a selection instruction for any original image, taking the original image as a target image, and registering other original images.

[0044] In some embodiments, the reconstruction module is used to input multiple medical images and the respective segmentation results of the multiple medical images into a medical image registration and fusion module, and the medical image registration and fusion module registers each organ tissue in the liver and gallbladder area in the multiple medical images and the respective segmentation results of the multiple medical images.

[0045] In some embodiments, the reconstruction module is used to:

[0046] generating a surface mesh based on voxel data of images in the first segmentation result, the second segmentation result, and the third segmentation result;

[0047] Smoothing the surface mesh;

[0048] The hole filling algorithm is used to fill the smoothed surface mesh;

[0049] Save the filled surface mesh as a standard template library file.

[0050] In some embodiments, the training process of the multi-organ segmentation model is implemented based on a first loss function, and the training process of the intra-liver segmentation model is implemented based on a second loss function, and the first loss function and / or the second loss function are obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation;

[0051] The training process of the liver tumor segmentation model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a third loss function, wherein the third loss function is obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation.

[0052] In some embodiments, an attention mechanism is introduced into the multi-organ segmentation model and / or the intra-liver segmentation model, and the attention mechanism is used to segment any image in an input image sequence based on data of other images in the image sequence, and / or the attention mechanism is used to segment any pixel in any image based on data of other pixels in the image.

[0053] On the one hand, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement various optional implementations of the above-mentioned deep learning-based liver and gallbladder segmentation method.

[0054] On the one hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement various optional implementations of the above-mentioned deep learning-based liver and gallbladder segmentation method.

[0055] On the one hand, a computer program product or a computer program is provided, wherein the computer program product or the computer program comprises one or more program codes, wherein the one or more program codes are stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the electronic device can perform the hepatobiliary segmentation method based on deep learning in any possible implementation manner described above.

[0056] In the embodiment of the present application, three deep learning models are used to perform three-stage image segmentation on the original image containing the liver and gallbladder, and the liver and gallbladder and its surrounding organs, bones and tubular structures, the location of the hepatic artery, hepatic vein, portal vein and dilated bile duct in the medical image and the location of the liver tumor are automatically identified and segmented, providing a surgical navigation for clinicians when performing liver and gallbladder diseases and surgical planning, and applying deep learning technology, which is more accurate and faster, and not only the liver and gallbladder are segmented, but also the surrounding organs and other human structures are segmented, which can be applied to disease diagnosis, surgical planning and radiotherapy planning of clinicians in more departments, improving the applicability of the method and having universal clinical value. In addition, the segmentation module of the above three stages can achieve more accurate segmentation, can take into account both global information and local information, and is more in line with the reading habits of clinicians, thereby further improving the segmentation precision and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 It is a schematic diagram of an implementation environment of a liver and gallbladder segmentation method based on deep learning provided in an embodiment of the present application;

[0059] Figure 2 is a flow chart of a method for segmenting liver and gallbladder based on deep learning provided in an embodiment of the present application;

[0060] Figure 3 is a flow chart of a method for segmenting liver and gallbladder based on deep learning provided in an embodiment of the present application;

[0061] Figure 4 It is a structural schematic diagram of a liver and gallbladder segmentation device based on deep learning provided in an embodiment of the present application;

[0062] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] In the present application, the terms "first", "second", etc. are used to distinguish between identical or similar items having substantially the same effects and functions. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor is the quantity and execution order limited. It should also be understood that although the following description uses the terms first, second, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, can be separate and different images.

[0065] The term "at least one" in this application means one or more, and the term "plurality" in this application means two or more, for example, a plurality of data packets means two or more data packets.

[0066] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0067] It should also be understood that the term "and / or" used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. The term "and / or" is a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0068] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0069] It should also be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0070] It should also be understood that the term “comprising” (also known as “inCludes”, “inCluding”, “Comprises” and / or “Comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0071] It should also be understood that the term "if" may be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if it is determined that ..." or "if [a stated condition or event] is detected" may be interpreted to mean "upon determining that ..." or "in response to determining that ..." or "upon detecting [a stated condition or event]" or "in response to detecting [a stated condition or event]," depending on the context.

[0072] At least some functions of the device or electronic device provided in the embodiments of the present application can be implemented by an AI model, such as at least one module among multiple modules of the device or electronic device can be implemented by an AI model. Functions associated with AI can be performed by non-volatile memory, volatile memory and processor.

[0073] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0074] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0075] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0076] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0077] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0078] According to the present application, at least one step in the method performed in the electronic device, such as identifying architectural elements, classifying floor plan areas, etc., can be implemented using an artificial intelligence model. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means obtaining a predefined operating rule or artificial intelligence model configured to perform a desired feature (or purpose) by training a basic artificial intelligence model with multiple training data through a training algorithm.

[0079] The nouns involved in this application are explained below.

[0080] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0081] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0082] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0083] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence.

[0084] Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0085] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0086] The solutions provided in the embodiments of the present application involve artificial intelligence computer vision technology, machine learning and other technologies, which are specifically explained through the following embodiments.

[0087] The implementation environment of this application is described below.

[0088] Figure 1 1 is a schematic diagram of an implementation environment of a deep learning-based hepatobiliary segmentation method provided in an embodiment of the present application. The implementation environment includes a terminal 101, or the implementation environment includes a terminal 101 and a deep learning-based hepatobiliary segmentation platform 102. The terminal 101 is connected to the deep learning-based hepatobiliary segmentation platform 102 via a wireless network or a wired network.

[0089] The terminal 101 is at least one of a smart phone, a game console, a desktop computer, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player or an MP4 (Moving Picture Experts Group Audio Layer IV) player, and a laptop computer. The terminal 101 is installed and runs an application that supports deep learning-based hepatobiliary segmentation.

[0090] Exemplarily, the terminal 101 has an image segmentation function. After acquiring the original image containing the liver and gallbladder, three deep learning models can be used to perform three-stage image segmentation to obtain the positions of various tissue structures of the liver and gallbladder in the image. The terminal 101 completes this work independently, or after the terminal 101 acquires the original image, it can transmit it to the liver and gallbladder segmentation platform 102 based on deep learning to provide image segmentation services for it. This embodiment of the application is not limited to this.

[0091] The deep learning-based hepatobiliary segmentation platform 102 includes at least one of a server, multiple servers, a cloud computing platform, and a virtualization center. The deep learning-based hepatobiliary segmentation platform 102 is used to provide background services for applications that support deep learning-based hepatobiliary segmentation. Optionally, the deep learning-based hepatobiliary segmentation platform 102 undertakes the main processing work, and the terminal 101 undertakes the secondary processing work; or, the deep learning-based hepatobiliary segmentation platform 102 undertakes the secondary processing work, and the terminal 101 undertakes the main processing work; or, the deep learning-based hepatobiliary segmentation platform 102 or the terminal 101 undertakes the processing work separately. Alternatively, the deep learning-based hepatobiliary segmentation platform 102 and the terminal 101 use a distributed computing architecture for collaborative computing.

[0092] Optionally, the deep learning-based hepatobiliary segmentation platform 102 includes at least one server 1021 and a database 1022. The database 1022 is used to store data. In an embodiment of the present application, the database 1022 stores sample input data and sample annotation data, and can also store user input data to provide data services for at least one server 1021.

[0093] A server is an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminals are smartphones, tablets, laptops, desktop computers, smart speakers, smart watches, etc., but are not limited to these.

[0094] Those skilled in the art know that the number of the above-mentioned terminals 101 and servers 1021 is more or less. For example, the above-mentioned terminal 101 and server 1021 are only one, or the above-mentioned terminals 101 and servers 1021 are dozens or hundreds, or more. The embodiment of the present application does not limit the number and device type of the terminals or servers.

[0095] The continuous development of medical imaging technology has greatly improved the diagnosis and treatment of liver cancer. 3D visualization images based on computer post-processing medical image feature extraction and reconstruction can provide intuitive and three-dimensional anatomical structures before surgery, thereby evaluating and optimizing surgical plans, including measuring the potential residual liver volume and structure, determining the optional resection range, simulating the best section, selecting blood vessels to be resected and reconstructed, and predicting intraoperative blood loss. In particular, when the lesion involves important anatomical structures and requires complex liver resection, regional metastatic lymph node resection, resection of malignant tumors involving large hepatic vessels, and liver resection combined with portal vein and hepatic artery resection and reconstruction, 3D visualization technology can help improve the controllability and safety of liver surgery. [3] In radiotherapy, the delineation of the tumor target area in the entire abdomen and the surrounding tissues and organs (or organs at risk) is the first step in the formulation of the entire treatment plan.

[0096] Traditional 3D reconstruction technology mainly relies on manual segmentation or semi-automatic segmentation methods, which are not only time-consuming and labor-intensive, but also may have inaccurate segmentation problems for complex hepatobiliary system structures.

[0097] In recent years, with the development of deep learning technology, automated medical image processing has gradually become possible. Deep learning, especially convolutional neural networks (CNN), has demonstrated excellent performance in medical image segmentation and reconstruction. Through deep learning models, more accurate liver and gallbladder segmentation can be achieved, and it helps to automate the three-dimensional reconstruction process, assisting clinical diagnostic decisions and precision treatment.

[0098] In the treatment of liver cancer, accurate segmentation and identification of the liver, gallbladder, intrahepatic dilated bile ducts, hepatic bile ducts, portal vein, hepatic artery, and inferior vena cava are intuitively important. Hepatobiliary tumors (such as liver cancer, cholangiocarcinoma) may spread to surrounding organs, especially the pancreas and duodenum. By three-dimensionally reconstructing the spleen, stomach, kidneys, pancreas, and duodenum, doctors can better assess whether the tumor has invaded these adjacent organs, determine the extent of tumor spread, and help develop a more comprehensive treatment plan. In radiotherapy for hepatobiliary tumors, the segmentation and reconstruction of surrounding organs (such as the stomach, pancreas, and kidneys) helps to develop accurate dose distribution plans to avoid excessive radiation to these sensitive organs, thereby protecting their functions. And in image-guided minimally invasive surgical navigation, the segmentation of the above organs can help doctors accurately develop efficient surgical pathways and resection plans.

[0099] In the embodiment of the present application, these are taken into consideration, and a three-stage automatic hepatobiliary segmentation and 3D reconstruction method based on deep learning is provided. The method takes into account the role of the surrounding organs (pancreas, stomach, etc.) in image-guided surgery and radiotherapy, and the segmentation accuracy and speed are effectively improved by applying deep learning technology, that is, a complete reconstruction solution and visualization system for clinicians is provided. Figure 2 The illustrated embodiment provides a detailed description of the above-mentioned three-stage image segmentation and three-dimensional reconstruction method based on deep learning.

[0100] Figure 2 is a flowchart of a method for segmenting the liver and gallbladder based on deep learning provided in an embodiment of the present application. The method is applied to an electronic device, which is a server or a terminal. Figure 2 , the method comprises the following steps.

[0101] 201. The electronic device obtains a first image based on an original image including a liver and gallbladder area.

[0102] The present invention proposes a three-stage automatic hepatobiliary segmentation and three-dimensional reconstruction method based on deep learning, hoping to provide a new method for clinicians in various departments when conducting hepatobiliary disease and surgical planning, which is more accurate and faster.

[0103] The embodiment of the present application aims to fully automatically identify the liver and gallbladder areas in medical images, and to perform three-dimensional reconstruction based on the segmentation results, thereby providing important imaging information support for subsequent diagnosis, surgery, and other work.

[0104] The hepatobiliary region includes multiple organs (liver, stomach, spleen, pancreas, duodenum, gallbladder, kidney, skin), bones (thoracic and lumbar vertebrae), tubular structures (hepatic artery, hepatic vein, portal vein, inferior vena cava, hepatobiliary duct, dilated bile duct,) and liver tumors. Of course, the hepatobiliary region may also include more human tissue structures in the region, which can be set by relevant technicians when training the model according to needs, and the embodiments of the present application are not limited to this.

[0105] In the embodiment of the present application, the original image containing the liver and gallbladder may be a medical image containing the liver and gallbladder. When a patient visits a doctor, a medical image containing the liver and gallbladder may be taken so that the doctor can judge the patient's condition based on the medical image. The embodiment of the present application does not limit the specific shooting instrument used to shoot the original image containing the liver and gallbladder, such as CT, MRI, etc.

[0106] In some embodiments, the original image containing the liver and gallbladder is generally a three-dimensional (3D) image. Therefore, after the electronic device obtains the original image containing the liver and gallbladder, it can process to obtain multiple slice images or a sequence of slice images, and then perform image segmentation on the slice images. It should be noted that the original image containing the liver and gallbladder is generally a 3D image. Each image is generally called a slice in medicine.

[0107] In some embodiments, after the electronic device acquires the original image containing the liver and gallbladder, it can first pre-process it to obtain a first image, so as to obtain an image suitable for model processing.

[0108] In some embodiments, the electronic device may perform standardization processing on the original image to obtain the first image. The standardization processing includes operations such as denoising, contrast enhancement, and resolution unification to ensure that the input data information features are consistent with those in the model training stage so that they can be input into the deep learning model to improve the accuracy of subsequent image segmentation.

[0109] In some embodiments, the preprocessing may include cropping in addition to the above-mentioned standardization processing, and the cropping is to crop the background area outside the body in the original image according to the grayscale information and / or position information. The above-mentioned original image containing the liver and gallbladder may contain parts other than the user's body, such as the environment when the user takes the photo. Considering that the purpose of our image segmentation is to identify the liver and gallbladder of the user, the parts outside the user's body are noise interference for image segmentation. Therefore, in the embodiment of the present application, the electronic device can first crop the original image, cut off the parts outside the user's body, and only the image area of ​​the user's body remains. Specifically, the electronic device obtains the original image data containing the patient's spine and crops the background area outside the body. Because in the image acquisition stage, the background area outside the body is basically air and is not imaged; in addition, the scanning bed that may appear in the image is a fixed mechanical device, which can be removed by simple grayscale information and position information.

[0110] 202. The electronic device inputs the first image into a multi-organ segmentation model, and the multi-organ segmentation model segments the first image to obtain a first segmentation result, where the first segmentation result is used to indicate the positions of the liver and gallbladder and the surrounding multiple organs, bones and tubular structures in the first image.

[0111] The first-stage model (also known as the multi-organ segmentation model) is used to segment the liver and gallbladder and the surrounding organs, bones, and tubular structures. This process uses the first image for segmentation, and can accurately locate each organ, bone, and tubular structure in order to fully extract the characteristics of each tissue and organ in the liver and gallbladder. This process not only identifies the organs, but also segments the bones and tubular structures, and can effectively perform surgical navigation, thereby obtaining global image information and avoiding missed judgments due to the lack of important information at certain levels.

[0112] The segmentation module of the liver and gallbladder is divided into three stages, and step 202 is the first stage. In the first stage, the standardized image data obtained in the previous step is first input into the multi-organ segmentation model. This model is mainly trained to segment multiple organs, bones and extrahepatic tubular structures. The model also outputs the tubular structure inside the liver, but it is not constrained to be correct because the foreground area of ​​these tissue structures is relatively large and the boundaries are relatively clear. In addition, multi-target segmentation can constrain multiple adjacent tissue structures to avoid mis-segmentation.

[0113] The multi-organ segmentation model may be a pre-trained model for image segmentation, and the multi-organ segmentation model may be trained based on a large number of sample original images and target segmentation results of the sample original images.

[0114] In some embodiments, the training process of the multi-organ segmentation model can be: obtaining a sample original image and a target prediction result of the sample original image; obtaining a sample first image based on the sample original image, and then inputting the sample first image into the initial multi-organ segmentation model, and the initial multi-organ segmentation model segments the sample first image, outputting the segmentation result of the sample first image, and then obtaining a first loss function based on the segmentation result of the sample first image and its target segmentation result, and then adjusting the model parameters of the initial multi-organ segmentation model based on the first loss function, and then repeating the above steps of inputting the sample original image, image segmentation, and adjusting the model parameters until the first target condition is reached and the multi-organ segmentation model is obtained.

[0115] Among them, the first target condition can be set by relevant technical personnel based on experience or needs. For example, the first target condition can be that the number of iterations reaches a first number. The specific value of the first number can be set by relevant technical personnel based on needs or experience, or the first target condition can be that the first loss function converges. The embodiments of the present application are not limited to this.

[0116] For the multi-organ segmentation model, in step 202, the electronic device inputs the standardized data into the model of the first stage. The neural network of the multi-organ segmentation model can select various suitable deep neural network models, such as Unet, TransUnet, SwinUNETR, etc., and the model structure can also be designed by relevant technical personnel, or any model with image segmentation function can be directly used. Of course, it is also possible to further increase or reduce the intermediate operation layer in combination with personal experience, such as introducing other advanced technologies such as attention mechanisms to improve the prediction accuracy and generalization ability of the model. When the model architecture design is large enough, the learning ability of the model can learn accurate spinal segmentation and labeling, so the embodiment of the present application does not constrain the multi-organ segmentation model to a certain model architecture.

[0117] For the first loss function, the first loss function can also be called an accuracy evaluation function, and the first loss function can be designed by relevant technical personnel according to needs or experience. In some embodiments, the training process of the multi-organ segmentation model is implemented based on a first loss function, and the first loss function is obtained based on the integration of multiple loss functions using multiple scales, wherein the weights of multiple loss functions are adaptively adjusted according to the training situation. Exemplarily, the multiple loss functions can consider but not only loss functions such as MSE (mean-square error), Dice coefficient, and boundary loss.

[0118] Among them, mean square error, Dice coefficient is a measurement method for comparing the similarity of two samples. It can measure the degree of overlap between two sets and is often used in text similarity calculation, image processing, bioinformatics and other fields. The value range of the Dice coefficient is, the closer the value is to 1, the more similar the two sets are. Boundary loss aims to improve the model's detection and prediction accuracy of target edges. In this way, the mean square error can ensure the kernel loss, and the boundary loss can ensure the accuracy of the target edge. The accuracy of the model can be effectively improved through multiple loss functions.

[0119] 203. The electronic device inputs the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, and the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, hepatic vein, portal vein and dilated bile duct in the liver region image.

[0120] The segmentation module of the liver and gallbladder is divided into three stages, and step 203 is the second stage. Based on the liver and tubular structures segmented in the first stage, the second stage of fine segmentation of the liver tissue is performed around the liver. The original image of the liver area, as well as the liver and tubular structure result images segmented in the first stage, are input into the second stage network together, and the finely segmented hepatic artery, hepatic vein, portal vein and dilated bile duct in the liver are output. Considering that the first stage is not detailed enough for the segmentation of the fine tissue structure in the liver, the liver area is separately input into the second stage model for fine segmentation, so that both global and local information can be taken into account to obtain more accurate image segmentation results.

[0121] The input of the second-stage model includes both the original image and the segmentation results of the first stage. The advantage is that the model can correct the results of the first stage based on local image details. This optimization method is also in line with the clinicians' own reading habits, analyzing from the whole to the part, from coarse to fine.

[0122] In some embodiments, the training process of the intrahepatic segmentation model can be: obtaining a sample original image and a target segmentation result of the sample original image; then inputting the liver region image in the target segmentation result of the sample original image into the three-dimensional initial intrahepatic segmentation model, and the initial intrahepatic segmentation model segments the liver region image, and outputs the segmentation result of the liver region image. Then, based on the segmentation result of the liver region image and its target segmentation result, a second loss function is obtained, and then the model parameters of the initial intrahepatic segmentation model are adjusted based on the second loss function, and then the above steps of inputting the sample original image, image segmentation, and adjusting the model parameters are repeated until the second target condition is reached, and the intrahepatic segmentation model is obtained.

[0123] Among them, the second target condition can be set by relevant technical personnel based on experience or needs. For example, the second target condition can be that the number of iterations reaches a second number. The specific value of the second number can be set by relevant technical personnel based on needs or experience, or the second target condition can be that the second loss function converges. The embodiments of the present application are not limited to this.

[0124] In some embodiments, the multi-organ segmentation model and the intrahepatic segmentation model can be jointly trained, so that the joint training of the two-stage model can make the two models combine to segment the image more accurately. Specifically, the segmentation result output by the initial multi-organ segmentation model and the image of the liver area in the first image of the sample can be input into the initial intrahepatic segmentation model together, and this is performed in each iteration. After the intrahepatic segmentation model outputs the segmentation result, the second loss function is obtained by the segmentation result and the target segmentation result, and the first loss function and the second loss function are integrated into the target loss function. The two model parameters are adjusted and optimized through the target loss function, and the subsequent iterative process is carried out to realize model training. For example, the integration method can be a weighted summation, or it can be other combination methods, which is not limited in the embodiments of the present application.

[0125] For the second loss function, the second loss function can also be called an accuracy evaluation function, and the second loss function can be designed by relevant technical personnel according to needs or experience. In some embodiments, the training process of the intrahepatic segmentation model is implemented based on a second loss function, and the second loss function is obtained based on the integration of multiple loss functions using multiple scales, wherein the weights of multiple loss functions are adaptively adjusted according to the training situation. Exemplarily, the multiple loss functions can consider but not only loss functions such as MSE (mean-square error), Dice coefficient, and boundary loss.

[0126] An attention mechanism is introduced into the above-mentioned multi-organ segmentation model and / or intra-liver segmentation model, and the attention mechanism is used to segment any image in an input image sequence based on data of other images in the image sequence, and / or the attention mechanism is used to segment any pixel in any image based on data of other pixels in the image.

[0127] 204. The electronic device inputs the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, and the liver tumor segmentation model performs liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, where the third segmentation result is used to indicate a location of the liver tumor in the first image.

[0128] The segmentation module of the liver and gallbladder is divided into three stages, and step 204 is the third stage. Finally, the result images segmented in the first two stages are input into the third liver tumor detection and segmentation neural network (that is, the liver tumor segmentation model) together with the original image. The reason for segmenting the liver tumor separately here is that, first of all, liver tumors may be multiple, of different sizes, with diverse and complex shapes and textures, and there is also the possibility of metastasis. Inputting the surrounding tissue structure as an anatomical prior into the network can serve as an attention mechanism, allowing the neural network to pay attention to the vicinity of blood vessels, the overlapping areas of the bile duct and the liver, etc., and may even achieve the prediction of liver cancer staging and classification.

[0129] For the prediction of liver cancer staging and classification, in some embodiments, the liver tumor segmentation model can be trained in advance by using the sample original image and the target segmentation result, so that the liver tumor segmentation model can realize the prediction of liver cancer staging and classification. Furthermore, in step 204, the third segmentation result can be obtained without output, and the staging or classification of the liver tumor is predicted according to at least one of the size, infiltration depth, infiltration range, adjacent organ involvement, lymph node metastasis, and hematogenous or other distant metastasis of the liver tumor in the third segmentation result to obtain a prediction result, which is used to indicate the tumor staging and / or classification of the liver tumor. Among them, the adjacent organ involvement may include the involvement of adjacent organs or the non-involvement of adjacent organs, or may include the number, size or range of adjacent organs involved. The lymph node metastasis may include whether there is metastasis of local and / or distant lymph nodes.

[0130] In some embodiments, the training process of the liver tumor segmentation model may be: obtaining a sample original image and a target segmentation result of the sample original image; then inputting the sample original image into a three-dimensional initial liver tumor segmentation model, and having the initial liver tumor segmentation model perform liver tumor segmentation on the sample original image, and outputting the segmentation result of the sample original image. Then, based on the segmentation result of the sample original image and its target segmentation result, a third loss function is obtained, and then the model parameters of the initial sample original image are adjusted based on the third loss function, and then the above steps of inputting the sample original image, image segmentation, and adjusting the model parameters are repeated until the third target condition is reached, and the sample original image is obtained.

[0131] Among them, the third target condition can be set by relevant technical personnel based on experience or needs. For example, the third target condition can be that the number of iterations reaches the second number. The specific value of the third number can be set by relevant technical personnel based on needs or experience, or the third target condition can be that the third loss function converges. The embodiments of the present application are not limited to this.

[0132] In some embodiments, the multi-organ segmentation model, the intrahepatic segmentation model and the liver tumor segmentation model can be jointly trained, so that the joint training of the three-stage model can make the three models combine to segment the image more accurately. Specifically, the segmentation result output by the initial multi-organ segmentation model and the image of the liver area in the sample first image can be input into the initial intrahepatic segmentation model together, and this is performed in each iteration. After the intrahepatic segmentation model outputs the segmentation result, the segmentation results output by the first two models and the sample first image are input into the initial liver tumor segmentation model for liver tumor segmentation to obtain the segmentation result, and then the target loss function integrating the first loss function, the second loss function and the third loss function is obtained through the three segmentation results and the target segmentation result. The model parameters are adjusted and the subsequent iterative process is carried out through the target loss function to realize model training. For example, the integration method can be weighted summation or other combination methods, which is not limited in the embodiments of the present application.

[0133] In some embodiments, the training process of the liver tumor segmentation model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a third loss function is implemented, and the third loss function is obtained based on the integration of multiple loss functions using multiple scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation. That is, in addition to the loss function consistent with the first stage, the loss function (or accuracy assessment function) of the third stage also needs to increase the constraints of the false positive rate of the classification label and the boundary loss. Thereby strengthening the model's constraints on the correctness of the boundaries and classifications, and obtaining an accurate segmentation result.

[0134] 205. The electronic device performs three-dimensional reconstruction based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, wherein the standard template library file supports rendering and display in multiple software.

[0135] For the three-dimensional reconstruction, a variety of three-dimensional reconstruction methods can be used. The embodiment of the present application provides a possible method as follows, but is not limited to the following method. Specifically, the electronic device can generate a surface mesh based on the voxel data of the image in the first segmentation result, the second segmentation result, and the third segmentation result, and then smooth the surface mesh. The electronic device then uses a hole filling algorithm to fill the smoothed surface mesh, and saves the filled surface mesh as a standard template library (Standard Template Library, STL) file. The process of generating the surface mesh is the process of extracting isosurfaces. Among them, the surface mesh filling can ensure the continuity of the final rendering.

[0136] In some embodiments, the smoothing process may adopt any smoothing method, for example, the Laplacian smoothing method, which is not limited in the embodiments of the present application.

[0137] In some embodiments, the electronic device may also use a mesh simplification algorithm to reduce the number of meshes in the generated surface mesh to improve transmission and rendering efficiency.

[0138] The above STL files can be loaded and rendered on various software, platforms and AR / VR devices. Clinicians can visualize and edit the three-dimensional organs and tumors in the liver and gallbladder, as well as make surgical plans.

[0139] In some embodiments, different file contents may be adaptively displayed for different departments. For example, the radiotherapy department only needs to render and display the output file of the third module (ie, the liver tumor segmentation model).

[0140] In some embodiments, because the medical images used for diagnosis may be multi-modal and multi-phase, and because different organs and tissue structures have different imaging effects on different images, for example, the hepatic artery is more obvious on the arterial phase CT, and the hepatic vein is more obvious on the venous phase CT. The segmentation module can act on multiple medical images of the hepatobiliary part, and by registering and fusing the multi-modal and multi-phase images and the segmentation results, a more accurate fused image information can be obtained, which is conducive to clinicians making more accurate judgments. Therefore, we can apply the above segmentation module to multiple medical images of the hepatobiliary part, and then register the multiple images with their segmentation results.

[0141] Specifically, in response to the original image including multiple multi-modal and / or multi-phase medical images, the electronic device aligns the multiple medical images and each organ tissue in the liver and gallbladder area in each segmentation result of the multiple medical images, and performs three-dimensional reconstruction on each segmentation result after the alignment.

[0142] In some embodiments, the registration process can be implemented through deep learning. Specifically, the electronic device inputs multiple medical images and the segmentation results of the multiple medical images into a medical image registration and fusion module, and the medical image registration and fusion module registers each organ tissue in the liver and gallbladder region in the multiple medical images and the segmentation results of the multiple medical images. This module registers multi-modal and multi-phase medical images one by one.

[0143] In some embodiments, the registration may include rigid registration and deformable registration. The electronic device may take multiple medical images, and then rigidly register the multiple medical images with the target image. Then the electronic device performs deformable registration on the rigidly registered multiple medical images with the target image to obtain multiple registered images. Finally, the electronic device performs rigid registration and deformable registration on each segmentation result of the multiple medical images according to the registration data of the multiple medical images as the segmentation results of the multiple registered images.

[0144] That is, first, the image to be transformed and the target image are rigidly registered, that is, the image to be transformed is moved and rotated to the same physical space position as the target image, and most of the tissues and organs are aligned, for the accuracy and speed of subsequent deformation registration. This rigid registration method can be a traditional algorithm or a method based on deep learning. Then, the image to be transformed and the template image are deformably registered so that the corresponding tissue and organ shapes and positions of the image to be transformed and the target image are consistent. Finally, the segmentation result of the image to be transformed (the tissue structure that clinicians believe is more accurate) is subjected to the rigid plus deformation registration just now, so that the transformed segmentation result can be seen on the template image, and the result replaces the original segmentation result.

[0145] In some embodiments, a clinician may be asked to select a template image, and other images are transformed into the template image one by one. In response to the original image including multiple multi-modal and / or multi-phase medical images, the electronic device displays the multiple original images. In response to receiving a selection instruction for any original image, the electronic device uses the original image as a target image and registers the other original images.

[0146] In other embodiments, the target image may be a pre-selected image, and each image to be registered may be registered using the target image.

[0147] In a specific example, if Figure 3 As shown, the original image data of the patient's liver and gallbladder can be obtained and standardized according to the multimodal and multi-phase data that may be input. Then the deep learning segmentation module is input, and the neural network is used to obtain all organs and tubular structures in the liver, gallbladder, liver tumor and other parts. Then the above results are input into the 3D reconstruction module to obtain the STL files of various organs and tumors. The original image data and the reconstructed STL files can be imported into various software, platforms and even AR / VR platforms for rendering, allowing clinicians to browse and edit these 3D structures.

[0148] Through the above embodiments, the embodiments of the present application clearly provide an automatic segmentation and 3D reconstruction system for the liver and gallbladder, that is, the images obtained by clinical scanning are segmented to obtain the liver and gallbladder parts and then reconstructed in 3D, providing diagnosis and surgical guidance for clinicians and improving work efficiency. Furthermore, we propose a three-stage segmentation method based on deep learning, which can take advantage of each, take into account global and local information, and obtain the best results.

[0149] The advantages of the embodiments of the present application are as follows:

[0150] (1) The present invention is the first to propose a method and system for 3D reconstruction and visualization of liver and gallbladder based on deep learning; the method for realizing 3D reconstruction of liver and gallbladder using deep learning is the key point;

[0151] (2) The first proposed a complete three-dimensional reconstruction of the organ tissue structure of the hepatobiliary region, including multiple organs (liver, stomach, spleen, pancreas, duodenum, gallbladder, kidney, skin), bones (thoracic and lumbar vertebrae), tubular structures (hepatic artery, hepatic vein, portal vein, inferior vena cava, hepatobiliary duct, dilated bile duct), and liver tumors.

[0152] (3) In the first stage, a three-stage hepatobiliary segmentation module was proposed. In the first stage, multiple organs and extrahepatic tubular structures are clearly segmented in the image. In the second stage, the focus is on segmenting the tubular structures inside the liver that are difficult to identify with the naked eye. Finally, a detection combined with segmentation model is used for multiple and complex liver tumors, taking advantage of the strengths of each to obtain the optimal hepatobiliary 3D reconstruction results.

[0153] (4) Based on the multi-modal and multi-phase images collected clinically, a registration and fusion module is provided so that doctors can fuse the segmentation results obtained from the multi-modal images into the same image to obtain a more accurate reconstruction result.

[0154] In the embodiment of the present application, three deep learning models are used to perform three-stage image segmentation on the original image containing the liver and gallbladder, and the liver and gallbladder and its surrounding organs, bones and tubular structures, the location of the hepatic artery, hepatic vein, portal vein and dilated bile duct in the medical image and the location of the liver tumor are automatically identified and segmented, providing a surgical navigation for clinicians when performing liver and gallbladder diseases and surgical planning, and applying deep learning technology, which is more accurate and faster, and not only the liver and gallbladder are segmented, but also the surrounding organs and other human structures are segmented, which can be applied to disease diagnosis, surgical planning and radiotherapy planning of clinicians in more departments, improving the applicability of the method and having universal clinical value. In addition, the segmentation module of the above three stages can achieve more accurate segmentation, can take into account both global information and local information, and is more in line with the reading habits of clinicians, thereby further improving the segmentation precision and accuracy.

[0155] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described in detail here.

[0156] The present application embodiment provides a liver and gallbladder segmentation device based on deep learning, such as Figure 4 As shown, the device comprises:

[0157] An acquisition module 401 is used to acquire a first image based on an original image including a liver and gallbladder region;

[0158] A first segmentation module 402 is used to input the first image into a multi-organ segmentation model, and the multi-organ segmentation model segments the first image to obtain a first segmentation result, wherein the first segmentation result is used to indicate the positions of the liver and gallbladder and the surrounding multiple organs, bones and tubular structures in the first image;

[0159] A second segmentation module 403 is used to input the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, and the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, hepatic vein, portal vein and dilated bile duct in the liver region image;

[0160] A third segmentation module 404 is used to input the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, and the liver tumor segmentation model performs liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, wherein the third segmentation result is used to indicate a location of the liver tumor in the first image;

[0161] The reconstruction module 405 is used to perform three-dimensional reconstruction based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, and the standard template library file supports rendering and display in multiple software.

[0162] In some embodiments, the reconstruction module 405 is used to align each organ tissue in the liver and gallbladder region in the multiple medical images and each segmentation result of the multiple medical images in response to the original image including multiple multi-modal and / or multi-phase medical images, and perform three-dimensional reconstruction of each segmentation result after the alignment.

[0163] In some embodiments, the reconstruction module 405 is used to:

[0164] rigidly registering a plurality of medical images with a target image;

[0165] Perform deformation registration on multiple rigidly registered medical images and the target image to obtain multiple registered images;

[0166] The respective segmentation results of the multiple medical images are rigidly registered and deformably registered according to the registration data of the multiple medical images to serve as the segmentation results of the multiple registered images.

[0167] In some embodiments, the apparatus further comprises:

[0168] A display module, configured to display the plurality of original images in response to the original images comprising a plurality of multi-modality and / or multi-phase medical images;

[0169] The reconstruction module 405 is used for responding to receiving a selection instruction for any original image, taking the original image as a target image, and registering other original images.

[0170] In some embodiments, the reconstruction module 405 is used to input multiple medical images and the respective segmentation results of the multiple medical images into a medical image registration and fusion module, and the medical image registration and fusion module performs registration on each organ tissue in the liver and gallbladder region in the multiple medical images and the respective segmentation results of the multiple medical images.

[0171] In some embodiments, the reconstruction module 405 is used to:

[0172] generating a surface mesh based on voxel data of images in the first segmentation result, the second segmentation result, and the third segmentation result;

[0173] Smoothing the surface mesh;

[0174] The hole filling algorithm is used to fill the smoothed surface mesh;

[0175] Save the filled surface mesh as a standard template library file.

[0176] In some embodiments, the training process of the multi-organ segmentation model is implemented based on a first loss function, and the training process of the intra-liver segmentation model is implemented based on a second loss function, and the first loss function and / or the second loss function are obtained based on the integration of multiple loss functions using multiple scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation;

[0177] The training process of the liver tumor segmentation model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a third loss function, wherein the third loss function is obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation.

[0178] In some embodiments, an attention mechanism is introduced into the multi-organ segmentation model and / or the intra-liver segmentation model, and the attention mechanism is used to segment any image in an input image sequence based on data of other images in the image sequence, and / or the attention mechanism is used to segment any pixel in any image based on data of other pixels in the image.

[0179] In the embodiment of the present application, three deep learning models are used to perform three-stage image segmentation on the original image containing the liver and gallbladder, and the liver and gallbladder and its surrounding organs, bones and tubular structures, the location of the hepatic artery, hepatic vein, portal vein and dilated bile duct in the medical image and the location of the liver tumor are automatically identified and segmented, providing a surgical navigation for clinicians when performing liver and gallbladder diseases and surgical planning, and applying deep learning technology, which is more accurate and faster, and not only the liver and gallbladder are segmented, but also the surrounding organs and other human structures are segmented, which can be applied to disease diagnosis, surgical planning and radiotherapy planning of clinicians in more departments, improving the applicability of the method and having universal clinical value. In addition, the segmentation module of the above three stages can achieve more accurate segmentation, can take into account both global information and local information, and is more in line with the reading habits of clinicians, thereby further improving the segmentation precision and accuracy.

[0180] It should be noted that: the liver and gallbladder part segmentation device based on deep learning provided in the above embodiment only uses the division of the above functional modules as an example when performing liver and gallbladder part segmentation based on deep learning. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the liver and gallbladder part segmentation device based on deep learning is divided into different functional modules to complete all or part of the functions described above. In addition, the liver and gallbladder part segmentation device based on deep learning provided in the above embodiment and the liver and gallbladder part segmentation method embodiment based on deep learning belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0181] Figure 5It is a structural diagram of an electronic device provided in an embodiment of the present application. The electronic device 500 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 501 and one or more memories 502, wherein the memory 502 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 5501 to implement the liver and gallbladder segmentation method based on deep learning provided in the above-mentioned various method embodiments. The electronic device may also include other components for realizing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input and output interface for input and output. The embodiments of the present application are not described in detail here.

[0182] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, wherein the at least one computer program is executable by a processor to complete the deep learning-based hepatobiliary segmentation method in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0183] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising one or more program codes, the one or more program codes being stored in a computer-readable storage medium. One or more processors of an electronic device can read the one or more program codes from the computer-readable storage medium, and the one or more processors execute the one or more program codes, so that the electronic device can perform the above-mentioned hepatobiliary segmentation method based on deep learning.

[0184] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0185] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0186] A person of ordinary skill in the art will understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.

[0187] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for liver and gallbladder segmentation based on deep learning, characterized in that: The method comprises: Based on the original image including the liver and gallbladder, a first image is acquired; Inputting the first image into a multi-organ segmentation model, and segmenting the first image by the multi-organ segmentation model to obtain a first segmentation result, wherein the first segmentation result is used to indicate the positions of the liver and gallbladder and the surrounding multiple organs, bones, and tubular structures in the first image; inputting the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, and the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, the hepatic vein, the portal vein, and the dilated bile duct in the liver region image; inputting the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, and having the liver tumor segmentation model perform liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, wherein the third segmentation result is used to indicate a position of the liver tumor in the first image; Three-dimensional reconstruction is performed based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, and the standard template library file supports rendering and display in multiple software.

2. The method for liver and gallbladder segmentation based on deep learning according to claim 1, characterized in that: The performing three-dimensional reconstruction based on the first segmentation result, the second segmentation result and the third segmentation result includes: In response to the original image comprising multiple multi-modal and / or multi-phase medical images, the multiple medical images and each organ tissue in the liver and gallbladder region in each segmentation result of the multiple medical images are registered, and each registered segmentation result is three-dimensionally reconstructed.

3. The method for liver and gallbladder segmentation based on deep learning according to claim 2, characterized in that: The registering of each organ tissue in the liver and gallbladder region of the plurality of medical images and each segmentation result of the plurality of medical images comprises: rigidly registering a plurality of medical images with a target image; Perform deformation registration on the multiple rigidly registered medical images and the target image to obtain multiple registered images; The respective segmentation results of the multiple medical images are rigidly registered and deformably registered according to the registration data of the multiple medical images to serve as the segmentation results of the multiple registered images.

4. The method for liver and gallbladder segmentation based on deep learning according to claim 3, characterized in that: The method further comprises: In response to the original images comprising a plurality of multi-modality and / or multi-phase medical images, displaying the plurality of original images; In response to receiving a selection instruction for any original image, the original image is used as a target image to perform registration on other original images.

5. The method for liver and gallbladder segmentation based on deep learning according to claim 2 or 3, characterized in that: The registering of each organ tissue in the liver and gallbladder region of the plurality of medical images and each segmentation result of the plurality of medical images comprises: The multiple medical images and the segmentation results of the multiple medical images are input into a medical image registration and fusion module, and the medical image registration and fusion module registers each organ tissue in the liver and gallbladder area in the multiple medical images and the segmentation results of the multiple medical images.

6. The method for liver and gallbladder segmentation based on deep learning according to claim 1, characterized in that: The three-dimensional reconstruction is performed based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder, including: generating a surface mesh based on voxel data of images in the first segmentation result, the second segmentation result, and the third segmentation result; Smoothing the surface mesh; The hole filling algorithm is used to fill the smoothed surface mesh; Save the filled surface mesh as a standard template library file.

7. The method for liver and gallbladder segmentation based on deep learning according to claim 1, characterized in that: The training process of the multi-organ segmentation model is implemented based on a first loss function, and the training process of the intra-liver segmentation model is implemented based on a second loss function, wherein the first loss function and / or the second loss function are obtained based on an integration of multiple loss functions using multiple scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation; The training process of the liver tumor segmentation model is based on at least one of the false positive rate of the segmentation result and the boundary loss function, and a third loss function, wherein the third loss function is obtained based on the integration of multiple loss functions using multi-scales, wherein the weights of the multiple loss functions are adaptively adjusted according to the training situation.

8. A liver and gallbladder segmentation device based on deep learning, characterized in that: The device comprises: An acquisition module, used for acquiring a first image based on an original image containing a liver and gallbladder portion; A first segmentation module, used for inputting the first image into a multi-organ segmentation model, and having the multi-organ segmentation model segment the first image to obtain a first segmentation result, wherein the first segmentation result is used for indicating the positions of the liver and gallbladder and the surrounding multiple organs, bones and tubular structures in the first image; a second segmentation module, configured to input the liver region image in the first image and the first segmentation result into an intrahepatic segmentation model, wherein the intrahepatic segmentation model segments the liver region image based on the liver region image and the first segmentation result to obtain a second segmentation result, wherein the second segmentation result is used to indicate the positions of the hepatic artery, the hepatic vein, the portal vein, and the dilated bile duct in the liver region image; a third segmentation module, configured to input the first image, the first segmentation result, and the second segmentation result into a liver tumor segmentation model, wherein the liver tumor segmentation model performs liver tumor segmentation on the first image based on the first segmentation result and the second segmentation result to obtain a third segmentation result, wherein the third segmentation result is used to indicate a position of the liver tumor in the first image; A reconstruction module is used to perform three-dimensional reconstruction based on the first segmentation result, the second segmentation result and the third segmentation result to obtain a standard template library file of each organ tissue structure in the liver and gallbladder area, wherein the standard template library file supports rendering and display in multiple software.

9. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the deep learning-based hepatobiliary area segmentation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the deep learning-based hepatobiliary segmentation method as described in any one of claims 1 to 7.

Images