Image processing method and device, electronic equipment and storage medium
By combining registration and segmentation models between cortical and medullary images, the problem of insufficient segmentation accuracy caused by indistinct renal vein visualization in the cortical phase was solved, thus achieving accurate three-dimensional reconstruction of the renal vein.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INFERVISION MEDICAL TECH CO LTD
- Filing Date
- 2022-09-07
- Publication Date
- 2026-08-04
AI Technical Summary
Existing segmentation algorithms struggle to achieve good segmentation results when renal vein visualization is not obvious during the cortical phase, resulting in insufficient accuracy in three-dimensional reconstruction of the renal vein.
By acquiring images of the medullary and cortical phases, image registration is performed using rigid and non-rigid registration modules to obtain the registration deformation field. Based on this deformation field, the vein segmentation results of the medullary phase are transformed to the cortical phase. Further corrections are made by combining the midline and the segmentation model to improve the segmentation accuracy.
This improved the segmentation accuracy of renal veins in the cortical phase, ensured the accuracy of three-dimensional reconstruction of renal veins, and solved the problem of poor results from direct segmentation in the cortical phase.
Smart Images
Figure CN116309244B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image analysis technology, specifically to an image processing method and apparatus, electronic device and storage medium. Background Technology
[0002] Currently, to achieve better 3D reconstruction of the renal cortical phase, in addition to 3D reconstruction of the renal tumor and renal artery, 3D reconstruction of the renal vein is also required. However, since the renal vein is not as clearly visualized in the cortical phase as the renal tumor and renal artery (i.e., the renal vein is not clearly visualized in the cortical phase), commonly used segmentation algorithms (such as ResUnet) often fail to achieve good results.
[0003] Therefore, improving the accuracy of renal vein segmentation during the cortical phase has become an urgent technical problem to be solved. Summary of the Invention
[0004] In view of this, embodiments of this application provide an image processing method and apparatus, an electronic device and a storage medium, which can improve the accuracy of vein segmentation in the cortical phase.
[0005] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring a cortical phase image and a first medullary phase image; inputting the first medullary phase image into a first segmentation model to obtain a vein segmentation result in the medullary phase; registering the cortical phase image and the first medullary phase image to obtain a first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image and the cortical phase image are registered; and transforming the vein segmentation result in the medullary phase based on the first registration deformation field to obtain a vein segmentation result in the first cortical phase.
[0006] In some embodiments of this application, registering a cortical phase image and a first medullary phase image to obtain a first registration deformation field includes: inputting the cortical phase image and the first medullary phase image into a registration model to obtain a first registration deformation field. The registration model includes a rigid registration module and a non-rigid registration module. The rigid registration module is used to obtain multiple affine transformation parameters, and the non-rigid registration module is used to obtain a second registration deformation field.
[0007] In some embodiments of this application, inputting a cortical phase image and a first medullary phase image into a registration model to obtain a first registration deformation field includes: inputting the cortical phase image and the first medullary phase image into a rigid registration module to obtain multiple affine transformation parameters; performing an affine transformation on the first medullary phase image based on the multiple affine transformation parameters to obtain a second medullary phase image; inputting the second medullary phase image and the cortical phase image into a non-rigid registration module to obtain a second registration deformation field; and obtaining a first registration deformation field based on the multiple affine transformation parameters and the second registration deformation field.
[0008] In some embodiments of this application, obtaining a first registration deformation field based on multiple affine transformation parameters and a second registration deformation field includes: obtaining a third registration deformation field based on multiple affine transformation parameters using a specified formula; and merging the second registration deformation field and the third registration deformation field to obtain the first registration deformation field.
[0009] In some embodiments of this application, the method further includes: training the registration model using multiple loss functions, wherein the multiple loss functions include a first loss function for training rigid registration modules and a second loss function for training non-rigid registration modules.
[0010] In some embodiments of this application, after transforming the vein segmentation result of the medullary phase based on the first registration deformation field to obtain the vein segmentation result of the first cortical phase, the method further includes: calculating the midline of the vein segmentation result of the first cortical phase; and obtaining the vein segmentation result of the second cortical phase based on the midline and the cortical phase image.
[0011] In some embodiments of this application, obtaining vein segmentation results in the second cortical phase based on midline and cortical phase images includes: inputting midline and cortical phase images into a second segmentation model to obtain vein segmentation results in the second cortical phase, wherein the second segmentation model is used to calculate the boundary of the vein segmentation results in the second cortical phase based on the midline.
[0012] Secondly, embodiments of this application provide an image processing apparatus, comprising: a first acquisition module for acquiring a cortical phase image and a first medullary phase image; a second acquisition module for inputting the first medullary phase image into a first segmentation model to acquire a vein segmentation result in the medullary phase; a third acquisition module for registering the cortical phase image and the first medullary phase image to acquire a first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image and the cortical phase image are registered; and a fourth acquisition module for transforming the vein segmentation result in the medullary phase based on the first registration deformation field to acquire a vein segmentation result in the first cortical phase.
[0013] Thirdly, embodiments of this application provide a computationally readable storage medium storing a computer program for performing the image processing method described in the first aspect.
[0014] Fourthly, embodiments of this application provide an electronic device, including: a processor; and a memory for storing processor-executable instructions, wherein the processor is used to execute the image processing method described in the first aspect above.
[0015] This application provides an image processing method, apparatus, electronic device, and storage medium. By acquiring medullary phase images, cortical phase images, and vein segmentation results in the medullary phase, and registering the medullary phase images with the cortical phase images to obtain a registration deformation field, the vein segmentation results in the medullary phase are transformed using the registration deformation field to obtain vein segmentation results in the cortical phase. This allows the embodiments of this application to improve the segmentation accuracy of veins (i.e., renal veins) in the cortical phase, avoiding the problem of poor vein segmentation results when directly performing vein segmentation on the cortical phase images. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of an implementation environment provided in the embodiments of this application.
[0017] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application.
[0018] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application.
[0019] Figure 4 This is a flowchart illustrating the registration method of the registration model provided in an exemplary embodiment of this application.
[0020] Figure 5 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application.
[0021] Figure 6 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application.
[0022] Figure 7 This is a schematic diagram of the structure of an image processing apparatus provided in an exemplary embodiment of this application.
[0023] Figure 8 This is a block diagram of an electronic device for image processing provided in an exemplary embodiment of this application. Detailed Implementation
[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Application Overview
[0026] Computed Tomography (CT) uses precisely collimated X-ray beams, gamma rays, ultrasound waves, etc., along with highly sensitive detectors, to scan a specific part of the human body one section after another. It features fast scanning time and clear images, and can be used to examine a variety of diseases.
[0027] When performing enhanced CT scans on patients with kidney disease, doctors first need to differentiate between the three stages: the cortical stage, the medullary stage, and the excretory stage. Currently, enhanced CT imaging is commonly used to clearly define the characteristics of these three stages. The scanning time for the cortical stage is delayed, approximately 35-30 seconds; for the medullary stage, it is delayed by about 60-70 seconds; and for the excretory stage, the enhanced CT scan can take 2-3 minutes. Therefore, the scan duration can serve as a standard for distinguishing between the three stages. Furthermore, since the symptoms differ in each stage, doctors can use the contrast-enhanced CT scans from different stages to locate tumors and lesions during treatment.
[0028] When treating kidney diseases, doctors need to compare information about kidney tumors, as well as examine information about the renal veins and renal arteries. The renal vein is a tributary of the inferior vena cava and lies anterior to the renal artery, running horizontally inward to drain into the inferior vena cava. The renal vein is more clearly visualized in the medullary phase, and segmentation algorithms (such as ResUnet) can usually achieve good segmentation results. However, kidney tumors and renal arteries are more clearly visualized in the cortical phase. Therefore, to better perform three-dimensional reconstruction in the cortical phase and assist doctors in diagnosing and treating kidney diseases, it is also necessary to reconstruct the renal vein in the cortical phase. However, because the renal vein is not clearly visualized in the cortical phase, directly segmenting the renal vein in the cortical phase usually does not yield good results.
[0029] This invention proposes an image processing method that acquires medullary phase images, cortical phase images, and vein segmentation results in the medullary phase. The medullary phase images are then registered with the cortical phase images to obtain a registration deformation field. This registration deformation field is then used to transform the vein segmentation results in the medullary phase to obtain the vein segmentation results in the cortical phase. This allows the embodiments of this application to improve the segmentation accuracy of veins (i.e., renal veins) in the cortical phase, avoiding the problem of poor vein segmentation results when directly performing vein segmentation on the cortical phase images.
[0030] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.
[0031] Exemplary System
[0032] Figure 1This is a schematic diagram of an implementation environment provided in an embodiment of this application. The implementation environment includes a CT scanner 130, a server 120, and a computer device 110. The computer device 110 can acquire multiple sets of medical images from the CT scanner 130 used for X-ray scanning of human tissue. Simultaneously, the computer device 110 can be connected to the server 120 via a communication network. Optionally, the communication network can be a wired network or a wireless network.
[0033] The computer device 110 can be a general-purpose computer or a computer device composed of dedicated integrated circuits, etc., and this application embodiment does not limit this. For example, the computer device 110 can be a mobile terminal device such as a tablet computer, or it can be a personal computer (PC), such as a laptop computer and a desktop computer, etc. Those skilled in the art will know that the number of the above-mentioned computer devices 110 can be one or more, and their types can be the same or different. For example, there can be one computer device 110, or there can be dozens or hundreds of computer devices 110, or more. This application embodiment does not limit the number and type of computer devices 110.
[0034] In some alternative embodiments, computer device 110 acquires multiple sets of medical sample images from CT scanner 130, each set of medical sample images including multiple two-dimensional medical sample images. Computer device 110 trains a neural network using the multiple sets of medical sample images to obtain a network model for classifying veins (e.g., renal veins) in a set of medical sample images.
[0035] Server 120 is a single server, or a combination of several servers, or a virtualization platform, or a cloud computing service center.
[0036] In some alternative embodiments, computer device 110 sends multiple sets of medical sample images acquired from CT scanner 130 to server 120. A set of medical sample images includes multiple two-dimensional medical sample images. Server 120 trains a neural network using the multiple sets of medical sample images to obtain a network model for classifying veins in a set of medical sample images.
[0037] Exemplary methods
[0038] Figure 2 This is a schematic flowchart of an image processing method provided in an exemplary embodiment of this application. Figure 2 The method is executed by a computing device, such as a server. Figure 2 As shown, the image processing method includes the following:
[0039] S210: Acquire images of the cortical phase and the first medullary phase.
[0040] Specifically, both the cortical phase image and the first medullary phase image can be medical images containing the renal vein. These medical images can be obtained directly using techniques such as computed tomography (CT), computed radiography (CR), digital radiography (DR), magnetic resonance imaging (MRI), or ultrasound. Furthermore, both the cortical phase image and the first medullary phase image can be in DICOM format; this application does not impose specific limitations on this.
[0041] S220: Input the first medullary phase image into the first segmentation model to obtain the vein segmentation results of the medullary phase.
[0042] Specifically, the first segmentation model can be a ResUnet network, and this application embodiment does not specifically limit the type of segmentation model. The first segmentation model can be a model used to segment veins (e.g., renal veins) in images of the first medullary phase.
[0043] In one embodiment, after the first medullary phase image is input into the first segmentation model, the first segmentation model segments the veins in the first medullary phase image to obtain the vein segmentation results of the medullary phase.
[0044] It should be noted that the first segmentation model can be a classification network. Furthermore, step S220 can be understood as the process of segmenting the renal vein from the first medullary phase image using a commonly used segmentation algorithm, as described in this embodiment.
[0045] S230: Register the cortical phase image and the first medullary phase image to obtain the first registration deformation field.
[0046] In one embodiment, the first registration deformation field is the deformation path when registering a first medullary phase image with a first cortical phase image.
[0047] Specifically, the cortical phase image and the first medullary phase image are input into the registration model for registration. The cortical phase image can be used as a fixed image, and the first medullary phase image can be used as a moving image to obtain a first registration deformation field. This first registration deformation field can be the deformation path of the first medullary phase image (as the moving image) and the cortical phase image (as the fixed image) during registration.
[0048] In one embodiment, the registration model may include a rigid registration module and a non-rigid registration module. First, the cortical phase image and the first medullary phase image are input into the rigid registration module to obtain multiple affine transformation parameters. Then, an affine transformation is performed on the first medullary phase image based on these parameters to obtain a second medullary phase image. Next, the second medullary phase image and the cortical phase image are input into the non-rigid registration module to obtain a second registration deformation field. Based on the multiple affine transformation parameters and the second registration deformation field, a first registration deformation field is then obtained.
[0049] It should be noted that image registration is the process of superimposing two or more images from different sources, taken at different times and angles. The image registration process is an automatic or manual operation that attempts to find matching points between two images and align them spatially to minimize the required error, i.e., a uniform proximity measurement between the two images.
[0050] S240: Transform the vein segmentation results of the medullary phase based on the first registration deformation field to obtain the vein segmentation results of the first cortical phase.
[0051] Specifically, when the computing device inputs the cortical phase image and the first medullary phase image into the registration model for registration, it can also simultaneously input the vein segmentation result of the medullary phase into the registration model. The vein segmentation result of the medullary phase can be transformed according to the first registration deformation field to obtain the vein segmentation result of the first cortical phase in the cortical phase image, that is, the obtained vein segmentation result of the first cortical phase is used as the output of the registration model.
[0052] In one embodiment, the cortical phase image, the first medullary phase image, and the vein segmentation results of the medullary phase are input into the registration model to obtain the vein segmentation results of the first cortical phase.
[0053] It should be noted that the total deformation field (i.e., the first registration deformation field) can be obtained through the embodiments of this application, so the vein segmentation result in the medullary stage can be transformed based on the obtained first registration deformation field to obtain the vein segmentation result in the cortical stage (i.e., the vein segmentation result in the first cortical stage).
[0054] S250: The midline for calculating the venous segmentation results in the first cortical phase.
[0055] S260: Based on midline and cortical phase images, obtain the vein segmentation results for the second cortical phase.
[0056] It should be noted that for a detailed description of steps S250 and S260, please refer to [link / reference needed]. Figure 5 The specific description of steps S550 and S560 in the embodiment.
[0057] Therefore, this embodiment of the application obtains medullary phase images, cortical phase images, and vein segmentation results in the medullary phase, and registers the medullary phase images with the cortical phase images to obtain a registration deformation field. Then, it uses the registration deformation field to transform the vein segmentation results in the medullary phase to obtain the vein segmentation results in the cortical phase. This allows the embodiment of the application to improve the segmentation accuracy of veins (i.e., renal veins) in the cortical phase and avoids the problem of poor vein segmentation results when directly performing vein segmentation on the cortical phase images.
[0058] In one embodiment of this application, registering a cortical phase image and a first medullary phase image to obtain a first registration deformation field includes: inputting the cortical phase image and the first medullary phase image into a registration model to obtain a first registration deformation field. The registration model includes a rigid registration module and a non-rigid registration module. The rigid registration module is used to obtain multiple affine transformation parameters, and the non-rigid registration module is used to obtain a second registration deformation field.
[0059] Specifically, the registration module may include a rigid registration module and a non-rigid registration module. The rigid registration module may be a rigid registration network used to obtain multiple affine transformation parameters to facilitate affine transformation of the first medullary phase image. Furthermore, the network structure of the rigid registration network may resemble a classification network; that is, the rigid registration network may be replaced by a 3D classification network.
[0060] The non-rigid registration module can be a non-rigid registration network used to obtain the second registration deformation field (or non-rigid deformation field), and the network structure of the non-rigid registration network can be similar to the network structure of a UNet, that is, the non-rigid registration network can be replaced by a 3D segmentation network, such as ResUnet or Deeplab segmentation network. The embodiments of this application do not specifically limit the rigid registration module and the non-rigid registration module.
[0061] In one embodiment, the registration network is an unsupervised training network, meaning that the registration network can be trained without labeled data.
[0062] It should be noted that the registration model can be understood as an end-to-end registration network that combines rigid and non-rigid registration. That is, the rigid registration module and the non-rigid registration module can be two modules of a registration model.
[0063] It should also be noted that for a detailed description of the embodiments of this application, please refer to [link to relevant documentation]. Figure 3 To avoid repetition, the embodiments described herein will not be repeated here.
[0064] Therefore, the embodiments of this application provide a guarantee for obtaining accurate venous segmentation results in the cortical phase by setting up a registration network that combines rigid and non-rigid registration.
[0065] Figure 3 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application. Figure 3 The example is Figure 2 Further limitations on step S230. For example... Figure 3 As shown, step S230 further includes the following:
[0066] S310: Input the cortical phase image and the first medullary phase image into the rigid registration module to obtain multiple affine transformation parameters.
[0067] Specifically, the cortical phase image and the first medullary phase image are first concatenated along the channel dimension to obtain the concatenated first image. Then, the concatenated first image is input into the rigid registration module (i.e., the rigid registration network) to obtain multiple affine transformation parameters. The number of affine transformation parameters can be 12 or 16, etc., and the embodiments of this application do not specifically limit the number of affine transformation parameters.
[0068] Preferably, in this embodiment of the application, the number of affine transformation parameters is set to 12. That is, the output of the rigid registration network can be a 12-dimensional vector, and this 12-dimensional vector can correspond to the 12 parameters of the affine transformation, that is, each dimension of the 12-dimensional vector corresponds to one affine transformation parameter.
[0069] In one embodiment, multiple affine transformation parameters can be represented in matrix form, for example, as a 3×4 affine matrix.
[0070] It should be noted that stitching along the channel dimension can be understood as adding two images of the same size together along the channel dimension. For example, if the size of both images is "1×1×512×512×300", then the stitched image will be "1×2×512×512×300".
[0071] S320: Perform an affine transformation on the first medullary phase image based on multiple affine transformation parameters to obtain the second medullary phase image.
[0072] Specifically, the first medullary phase image (i.e., the moving image) is affinely transformed using 12 affine transformation parameters (i.e., a 3×4 affine matrix) to obtain the second medullary phase image, i.e., the transformed image.
[0073] It should be noted that an affine transformation (also known as an affine mapping) refers to the process in geometry of transforming a vector space into another vector space by performing a linear transformation followed by a translation.
[0074] S330: Input the second medullary phase image and the cortical phase image into the non-rigid registration module to obtain the second registration deformation field.
[0075] Specifically, the second medullary phase image and the cortical phase image are first stitched together along the channel dimension to obtain the stitched second image. Then, the stitched second image is input into a non-rigid registration module (i.e., a non-rigid registration network) to obtain the second registration deformation field (i.e., the non-rigid deformation field). The second registration deformation field can be a feature map with the same size as the input image and three channels, representing the offset of the deformation field in the x, y, and z directions.
[0076] S340: Obtain the first registration deformation field based on multiple affine transformation parameters and the second registration deformation field.
[0077] Specifically, by merging multiple affine transformation parameters and the second registration deformation field, a total deformation field after merging is obtained, namely the first registration deformation field, so that the first medullary phase image can be transformed based on the first registration deformation field to obtain the registered image.
[0078] In one embodiment, see Figure 4 First, the cortical phase image 410 and the first medullary phase image 420 are stitched together and input into the rigid registration module 430 to obtain an affine matrix 440. Then, the first medullary phase image 420 is subjected to an affine transformation (i.e., spatial transformation) based on the affine matrix 440 to obtain the second medullary phase image 450. Next, the second medullary phase image 450 is stitched together with the cortical phase image 410 and input into the non-rigid registration module 460 to obtain a second registration deformation field 470. Finally, the affine matrix 440 and the second registration deformation field 470 are merged to obtain a first registration deformation field 480, and the first medullary phase image 420 is transformed based on this first registration deformation field 480 to obtain the registered image 490.
[0079] Therefore, it can be seen that the embodiments of this application utilize rigid registration modules and non-rigid registration modules to obtain the registration deformation field, thereby ensuring the accuracy of the registration deformation field and further improving the registration accuracy between cortical and medullary phase images.
[0080] In one embodiment of this application, obtaining a first registration deformation field based on multiple affine transformation parameters and a second registration deformation field includes: obtaining a third registration deformation field based on multiple affine transformation parameters using a specified formula; and merging the second registration deformation field and the third registration deformation field to obtain the first registration deformation field.
[0081] Specifically, an affine matrix can be obtained based on multiple affine transformation parameters, and then the affine matrix is converted into a third registration deformation field using a specified formula. In this application embodiment, the specified formula is not specifically limited. As long as the formula can convert the affine matrix into a feature map with 3 channels (that is, representing the offset of the deformation field in the x, y and z directions), it is within the protection scope of this application embodiment.
[0082] Furthermore, the third registration deformation field and the second registration deformation field (i.e., the non-rigid deformation field) are combined to obtain the first registration deformation field (i.e., the total registration deformation field). For example, the third registration deformation field is added to the second registration deformation field to obtain the total registration deformation field, i.e., the first registration deformation field.
[0083] Therefore, this embodiment of the application improves the accuracy of image registration by converting affine transformation parameters into a deformation field and combining the deformation field with a non-rigid deformation field, thus avoiding the problem of poor image registration effect and providing a guarantee for obtaining accurate vein segmentation results in the cortical phase.
[0084] In one embodiment of this application, the method further includes: training the registration model using multiple loss functions, wherein the multiple loss functions include a first loss function for training rigid registration modules and a second loss function for training non-rigid registration modules.
[0085] Specifically, when training the registration network, multiple loss functions can be used to backpropagate the model until the desired registration network is obtained. The number of loss functions can be one or more; the number of loss functions is not specifically limited in this embodiment.
[0086] In one embodiment, the number of loss functions can be two, meaning multiple loss functions including a first loss function and a second loss function. The first loss function is used to train the rigid registration module, i.e., to constrain the affine transformation parameters.
[0087] For example, the difference between the determinant of the affine transformation matrix and 1 should be minimized to ensure that the affine transformation does not undergo excessive deformation. The second loss function is used to train the non-rigid registration module, that is, to constrain the non-rigid registration network so that the gradient of the non-rigid deformation field (i.e., the second registration deformation field) is minimized to ensure that the deformation field is not excessively distorted.
[0088] In one embodiment, the number of loss functions can be three, i.e., multiple loss functions including a first loss function, a second loss function, and a third loss function. The third loss function is used to calculate the similarity between the registered image and the original image. The first and second loss functions are essentially the same as described in the above embodiments.
[0089] It should be noted that the first, second, and third loss functions can be loss functions of the same or different types. They can also be loss functions that are partially the same or partially different. For example, the first, second, and third loss functions can all be cross-entropy loss functions, or the first and second loss functions can be cross-entropy loss functions, and the third loss function can be a mean squared error loss function, etc.
[0090] Therefore, it can be seen that the embodiments of this application utilize at least one loss function to perform backpropagation on the model in order to better obtain the required registration model.
[0091] Figure 5 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application. Figure 5 The example is Figure 2 The further limitations of step S240 are similar and will not be repeated here; the focus is on describing the differences. For example... Figure 5 As shown, the image processing method includes the following:
[0092] S510: Acquire images of the cortical phase and the first medullary phase.
[0093] S520: Input the first medullary phase image into the first segmentation model to obtain the vein segmentation results of the medullary phase.
[0094] S530: Register the cortical phase image and the first medullary phase image to obtain the first registration deformation field.
[0095] In one embodiment, the first registration deformation field is the deformation path when registering a first medullary phase image with a first cortical phase image.
[0096] S540: Transform the vein segmentation results of the medullary phase based on the first registration deformation field to obtain the vein segmentation results of the first cortical phase.
[0097] It should be noted that although step S540 has obtained the vein segmentation result in the cortical phase (i.e., the vein segmentation result in the first cortical phase), there is usually a problem that the edges of the vein segmentation result do not fit the cortical phase image. Therefore, this embodiment introduces a second segmentation model to further correct the vein segmentation result in the first cortical phase.
[0098] S550: Calculate the midline of the venous segmentation results in the first cortical phase.
[0099] Specifically, the midline can be the center line of the vein segmentation result in the first cortical phase. The embodiments of this application do not specifically limit the method of calculating the midline. For example, firstly, the region of the vein segmentation result in the first cortical phase is determined, and then the midline of the region (i.e., the vein segmentation result in the first cortical phase) is determined using a midline recognition algorithm; or the vein segmentation result in the first cortical phase is input into a network model for extracting the midline to directly obtain the midline of the vein segmentation result in the first cortical phase.
[0100] S560: Based on midline and cortical phase images, obtain the vein segmentation results for the second cortical phase.
[0101] Specifically, the midline of the vein segmentation result in the first cortical phase and the cortical phase image are input into the second segmentation model to obtain the vein segmentation result in the second cortical phase. The second segmentation model can calculate the boundary of the vein segmentation result in the second cortical phase based on the midline of the vein segmentation result in the first cortical phase, that is, to determine the edge of the vein segmentation result in the second cortical phase and locate the vein segmentation result in the cortical phase image.
[0102] It should be noted that, in this embodiment, the segmented veins of the medullary phase (i.e., the renal veins of the medullary phase) are first registered onto the cortical phase image using a registration technique. While registration can achieve roughly accurate localization, the edges of the first cortical phase vein segmentation result obtained after registration are usually not very accurate. Therefore, this embodiment uses a segmentation network (i.e., a second segmentation model) after registration to completely align the edges of the registered first cortical phase vein segmentation result with the veins in the cortical phase image, thus solving the problem of inaccurate edges caused by registration. In other words, this embodiment proposes a "two-stage (i.e., medullary and cortical phase) registration plus segmentation" technical solution, solving the problem of three-dimensional reconstruction of the renal veins in the cortical phase image.
[0103] Therefore, this embodiment of the application eliminates the edge inaccuracies caused by registration by utilizing the midline of the vein segmentation result (rather than directly using the vein segmentation result) to obtain the vein segmentation result in the second cortical phase. Furthermore, this embodiment of the application retains the approximate positional information of the vein segmentation result provided after registration based only on the midline of the vein segmentation result, and further corrects the edges of the vein segmentation result using a segmentation model, thereby improving the accuracy of the vein segmentation result in the cortical phase image.
[0104] In one embodiment of this application, obtaining the vein segmentation result of the second cortical phase based on the midline and cortical phase images includes: inputting the midline and cortical phase images into a second segmentation model to obtain the vein segmentation result of the second cortical phase, wherein the second segmentation model is used to calculate the boundary of the vein segmentation result of the second cortical phase based on the midline.
[0105] Specifically, the second segmentation model can be a network model used to calculate the vein boundary in the cortical phase image based on the vein midline; that is, a model used to calculate the boundary of the vein segmentation result to be obtained in the second cortical phase image based on the midline of the vein segmentation result in the first cortical phase. It should be noted that the second segmentation model can be a simple segmentation network, such as the UNet network.
[0106] In one embodiment, the input of the second segmentation model is the midline of the vein segmentation result of the first cortical phase image and the registered first cortical phase image, and the output is the vein segmentation result of the second cortical phase after eliminating the influence of the boundary.
[0107] Therefore, it can be seen that the embodiments of this application obtain the second cortical phase vein segmentation result by setting a segmentation model (i.e., the second segmentation model) and based on the midline of the vein segmentation result, the vein segmentation result in the second cortical phase is obtained without the influence of the boundary, thereby making the segmentation model further correct the edge of the vein segmentation result in the cortical phase.
[0108] Figure 6 This is a schematic flowchart of an image processing method provided in another exemplary embodiment of this application. The image processing method includes the following.
[0109] The first medullary phase image 620 is input into the first segmentation model 630 for segmenting the veins to obtain the vein segmentation result 640 of the medullary phase. Then, the cortical phase image 610, the first medullary phase image 620 and the vein segmentation result 640 of the medullary phase are input into the registration model 650 to obtain the vein segmentation result 660 of the first cortical phase.
[0110] Furthermore, the midline of the vein segmentation result 660 in the first cortical phase is calculated to obtain the midline 670 of the vein segmentation result in the first cortical phase. Then, the midline 670 of the vein segmentation result in the first cortical phase and the cortical phase image 610 are input into the second segmentation model 680 to obtain the vein segmentation result 690 in the second cortical phase with higher accuracy.
[0111] Exemplary device
[0112] Figure 7 This is a schematic diagram of the structure of an image processing apparatus 700 provided in an exemplary embodiment of this application. Figure 7 As shown, the image processing device 700 includes: a first acquisition module 710, a second acquisition module 720, a third acquisition module 730, a fourth acquisition module 740, a calculation module 750, and a sixth acquisition module 760.
[0113] The first acquisition module 710 is used to acquire the cortical phase image and the first medullary phase image; the second acquisition module 720 is used to input the first medullary phase image into the first segmentation model to acquire the vein segmentation result of the medullary phase; the third acquisition module 730 is used to register the cortical phase image and the first medullary phase image to acquire the first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image is registered with the cortical phase image; the fourth acquisition module 740 is used to transform the vein segmentation result of the medullary phase based on the first registration deformation field to acquire the vein segmentation result of the first cortical phase.
[0114] This application provides an image processing apparatus that acquires medullary phase images, cortical phase images, and medullary phase vein segmentation results, and registers the medullary phase images with the cortical phase images to obtain a registration deformation field. Then, the registration deformation field is used to transform the medullary phase vein segmentation results to obtain the cortical phase vein segmentation results. This allows the embodiment of this application to improve the segmentation accuracy of veins (i.e., renal veins) in the cortical phase, avoiding the problem of poor vein segmentation results when directly performing vein segmentation on the cortical phase images.
[0115] According to one embodiment of this application, the third acquisition module 730 is used to input the cortical phase image and the first medullary phase image into the registration model to obtain the first registration deformation field. The registration model includes a rigid registration module and a non-rigid registration module. The rigid registration module is used to acquire multiple affine transformation parameters, and the non-rigid registration module is used to acquire the second registration deformation field.
[0116] According to one embodiment of this application, the third acquisition module 730 is used to input the cortical phase image and the first medullary phase image into the rigid registration module to acquire multiple affine transformation parameters; perform an affine transformation on the first medullary phase image based on the multiple affine transformation parameters to acquire the second medullary phase image; input the second medullary phase image and the cortical phase image into the non-rigid registration module to acquire the second registration deformation field; and acquire the first registration deformation field based on the multiple affine transformation parameters and the second registration deformation field.
[0117] According to one embodiment of this application, the third acquisition module 730 is used to acquire a third registration deformation field based on multiple affine transformation parameters using a specified formula; and to merge the second registration deformation field and the third registration deformation field to acquire a first registration deformation field.
[0118] According to one embodiment of this application, the third acquisition module 730 is used to train the registration model using multiple loss functions, wherein the multiple loss functions include a first loss function for training a rigid registration module and a second loss function for training a non-rigid registration module.
[0119] According to one embodiment of this application, the calculation module 750 is used to calculate the midline of the vein segmentation result in the first cortical phase; the sixth acquisition module 760 is used to acquire the vein segmentation result in the second cortical phase based on the midline and the cortical phase image.
[0120] According to one embodiment of this application, the sixth acquisition module 760 is used to input the midline and cortical phase images into the second segmentation model to acquire the vein segmentation result of the second cortical phase, wherein the second segmentation model is used to calculate the boundary of the vein segmentation result of the second cortical phase based on the midline.
[0121] It should be understood that the specific working process and functions of the first acquisition module 710, the second acquisition module 720, the third acquisition module 730, the fourth acquisition module 740, the calculation module 750, and the sixth acquisition module 760 in the above embodiments can be referred to the above. Figures 1 to 6 The description of the image processing method provided in the embodiments will not be repeated here to avoid repetition.
[0122] Exemplary electronic devices and computer-readable storage media
[0123] Figure 8 This is a block diagram of an electronic device for image processing provided in an exemplary embodiment of this application.
[0124] Reference Figure 8 The electronic device 800 includes a processing component 810, which further includes one or more processors, and memory resources represented by memory 820 for storing instructions, such as application programs, that can be executed by the processing component 810. The application programs stored in memory 820 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 810 is configured to execute instructions to perform the aforementioned image processing method.
[0125] Electronic device 800 may also include a power supply component configured to perform power management of electronic device 800, a wired or wireless network interface configured to connect electronic device 800 to a network, and an input / output (I / O) interface. Electronic device 800 can be operated based on an operating system stored in memory 820, such as Windows Server. TM Mac OSX TM Unix TM Linux TM FreeBSD TM Or similar.
[0126] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the aforementioned electronic device 800, enables the electronic device 800 to perform an image processing method, comprising: acquiring a cortical phase image and a first medullary phase image; inputting the first medullary phase image into a first segmentation model to obtain a vein segmentation result in the medullary phase; registering the cortical phase image and the first medullary phase image to obtain a first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image and the cortical phase image are registered; transforming the vein segmentation result in the medullary phase based on the first registration deformation field to obtain a vein segmentation result in the first cortical phase.
[0127] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.
[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0129] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0132] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0133] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program verification codes, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0134] It should be noted that in the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0135] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.
Claims
1. An image processing method, characterized in that, include: Acquire images of the cortical phase and the first medullary phase; The first medullary phase image is input into the first segmentation model to obtain the vein segmentation result of the medullary phase; The cortical phase image and the first medullary phase image are registered to obtain a first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image and the cortical phase image are registered; The vein segmentation results of the medullary phase are transformed based on the first registration deformation field to obtain the vein segmentation results of the first cortical phase. Calculate the midline of the venous segmentation results in the first cortical phase; Based on the midline and the cortical phase image, the vein segmentation result of the second cortical phase is obtained, wherein obtaining the vein segmentation result of the second cortical phase based on the midline and the cortical phase image includes: The midline and the cortical phase image are input into the second segmentation model to obtain the vein segmentation result of the second cortical phase, wherein the second segmentation model is used to calculate the boundary of the vein segmentation result of the second cortical phase based on the midline.
2. The image processing method according to claim 1, characterized in that, The step of registering the cortical phase image and the first medullary phase image to obtain a first registration deformation field includes: The cortical phase image and the first medullary phase image are input into the registration model to obtain the first registration deformation field. The registration model includes a rigid registration module and a non-rigid registration module. The rigid registration module is used to obtain multiple affine transformation parameters, and the non-rigid registration module is used to obtain a second registration deformation field.
3. The image processing method according to claim 2, characterized in that, The step of inputting the cortical phase image and the first medullary phase image into the registration model to obtain the first registration deformation field includes: The cortical phase image and the first medullary phase image are input into the rigid registration module to obtain the plurality of affine transformation parameters; Based on the multiple affine transformation parameters, the first medullary phase image is subjected to an affine transformation to obtain the second medullary phase image; The second medullary phase image and the cortical phase image are input into the non-rigid registration module to obtain the second registration deformation field; The first registration deformation field is obtained based on the plurality of affine transformation parameters and the second registration deformation field.
4. The image processing method according to claim 3, characterized in that, The step of obtaining the first registration deformation field based on the plurality of affine transformation parameters and the second registration deformation field includes: The third registration deformation field is obtained using a specified formula based on the multiple affine transformation parameters. The second registration deformation field is merged with the third registration deformation field to obtain the first registration deformation field.
5. The image processing method according to claim 2, characterized in that, Also includes: The registration model is trained using multiple loss functions, wherein the multiple loss functions include a first loss function for training the rigid registration module and a second loss function for training the non-rigid registration module.
6. An image processing apparatus, characterized in that, include: The first acquisition module is used to acquire images of the cortical phase and images of the first medullary phase. The second acquisition module is used to input the first medullary phase image into the first segmentation model to obtain the vein segmentation result of the medullary phase; The third acquisition module is used to register the cortical phase image and the first medullary phase image to acquire a first registration deformation field, wherein the first registration deformation field is the deformation path when the first medullary phase image is registered with the cortical phase image; The fourth acquisition module is used to transform the vein segmentation result of the medullary phase based on the first registration deformation field to obtain the vein segmentation result of the first cortical phase. The calculation module is used to calculate the midline of the vein segmentation results in the first cortical phase; The sixth acquisition module is used to acquire the vein segmentation result of the second cortical phase based on the midline and the cortical phase image, wherein acquiring the vein segmentation result of the second cortical phase based on the midline and the cortical phase image includes: The midline and the cortical phase image are input into the second segmentation model to obtain the vein segmentation result of the second cortical phase, wherein the second segmentation model is used to calculate the boundary of the vein segmentation result of the second cortical phase based on the midline.
7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image processing method according to any one of claims 1 to 5.
8. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions. The processor is used to execute the image processing method according to any one of claims 1 to 5.