Medical image processing method, system and storage medium

Abstract

The application discloses a medical image processing method and system and a storage medium. The medical image processing method comprises the following steps: acquiring a to-be-processed medical image and image information of the to-be-processed medical image; acquiring target feature information under different preset resolutions based on the image information; inputting the target feature information into a preset guide neural network to respectively output guide maps of the target feature information under different preset resolutions; generating a first target grid of the target feature information under a corresponding preset resolution based on the guide maps; and acquiring a target medical image obtained by performing denoising processing on the to-be-processed medical image based on a plurality of first target grids corresponding to different preset resolutions. The medical image is subjected to splatting layering, the layered grid can quickly complete filtering, the filtered grid is subjected to slicing combination, a denoised medical image is obtained, the medical image can be quickly subjected to denoising processing, and real-time volume rendering is realized.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and particularly relates to a medical image processing method, system and storage medium. Background Technology

[0002] In medical image visualization engineering, one of the most fundamental tasks is rendering medical volume data. Unlike surface rendering, volume rendering encompasses various techniques for generating images from three-dimensional scalar data. Among volume rendering techniques, PBVR (Physically-Based Volumetric Rendering) is currently a relatively cutting-edge rendering method. PBVR renders images by simulating the optical flow in the real world, resulting in strong sense of depth, rich texture, and realistic lighting effects; however, it places high demands on computing power. To reduce the computational cost of PBVR and achieve real-time rendering, statistical sampling methods such as Monte Carlo sampling have been proposed. Monte Carlo sampling reduces the computational demands by tracking only a finite number of light rays and their reflections and refractions. However, this method inevitably introduces noise, leading to image distortion in the rendered image.

[0003] To address this problem, two approaches exist: improving the sampling algorithm and applying denoising algorithms. For the second approach, denoising algorithms such as bilateral filtering and temporal variance filtering have been proposed. In recent years, with the development of deep learning technology based on neural networks, more and more people are using deep learning to guide filtering and denoising algorithms or directly predict the true pixel values, thereby achieving rendering effects similar to high sampling rates under low sampling rate conditions. Among these, bilateral mesh filtering, developed from bilateral filtering, reduces computation by employing the concept of upscaling grayscale information, filtering, and then reconstructing, achieving fast filtering and denoising effects.

[0004] However, most current deep learning denoising methods based on neural networks, such as bilateral meshes, are used for offline surface rendering and image dehazing. They do not take into account the special characteristics of the computational equations for volume data rendering when designing the input features and network architecture of the neural network. In addition, the computational load of volume rendering is huge, which leads to the problem of difficulty in real-time rendering and denoising. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of existing neural network-based rendering and denoising methods that do not take into account the special characteristics of volume data rendering, resulting in difficulties in real-time rendering and denoising. The present invention provides a medical image processing method, system and storage medium.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution:

[0007] This invention provides a medical image processing method, the medical image processing method comprising:

[0008] Acquire the medical image to be processed, and the image information of the medical image to be processed;

[0009] Based on the image information, target feature information at different preset resolutions is obtained;

[0010] The target feature information is input into a preset guidance neural network to output guidance maps of the target feature information at different preset resolutions.

[0011] Based on the guiding map, a first target grid is generated at the corresponding preset resolution for the target feature information;

[0012] Based on multiple first target grids corresponding to different preset resolutions, the target medical image obtained by denoising the medical image to be processed is acquired.

[0013] In this scheme, the rendered medical images are splatting layered by guiding a neural network to generate multiple first target meshes corresponding to different resolution channels. The layered meshes can be filtered quickly, and then the filtered meshes are combined by slicing to obtain the target medical image. This scheme can quickly and efficiently denoise medical images rendered at low sampling rates, thereby achieving real-time physically based volume rendering and ensuring the accuracy and efficiency of medical image rendering.

[0014] Preferably, the step of obtaining the target medical image obtained by denoising the medical image to be processed based on multiple first target grids corresponding to different preset resolutions includes:

[0015] The first target mesh is filtered to obtain the second target mesh;

[0016] The second target grid is sliced ​​to obtain denoised medical images at different preset resolutions;

[0017] The target medical image is generated based on the different denoised medical images.

[0018] In this scheme, filtering the first target grid after layering can improve the efficiency of denoising medical images.

[0019] Preferably, the step of generating the target medical image based on different denoised medical images includes:

[0020] Obtain the denoising weights corresponding to different preset resolutions;

[0021] Based on the denoising weights, different denoised medical images are fused to obtain the target medical image.

[0022] In this solution, by weighting and combining denoised medical images, the proportion of images with different resolutions can be flexibly changed to obtain the final target medical image as needed.

[0023] Preferably, the target feature information is radiance information;

[0024] The steps for obtaining target feature information at different preset resolutions include:

[0025] The radiance information is downsampled to obtain the radiance information at different preset resolutions.

[0026] In this scheme, radiance information at different resolutions is obtained by downsampling the radiance information, and then a grid is generated with multiple resolution channels, which can better denoise the local and overall medical images separately.

[0027] Preferably, the medical image processing method further includes:

[0028] Based on the image information, other feature information besides the radiance information is obtained;

[0029] The other feature information includes at least one of albedo information, normal information, and depth information;

[0030] The other feature information and the radiance information are input into the preset guidance neural network to output the guidance map of the radiance information at different preset resolutions.

[0031] In this scheme, the radiance information is guided by a pre-defined neural network through other feature information to generate grid values ​​on a bilateral grid, which can improve the accuracy of grid generation and thus improve the accuracy and effectiveness of medical image processing.

[0032] Preferably, the medical image processing method further includes:

[0033] The average value of the actual radiance information of the current frame medical image and the first radiance information obtained after denoising of the previous frame medical image is calculated and used as the target radiance information of the current frame medical image.

[0034] In this scheme, by using temporal filtering of radiance information, flickering caused by inconsistent brightness in local areas between consecutive frames can be avoided, thereby improving the quality of medical image rendering.

[0035] Preferably, the step of constructing the preset guiding neural network includes:

[0036] Acquire several sample medical images;

[0037] The volume data of each of the sample medical images is rendered to obtain the sample image at the first sampling rate and the ground truth image at the second sampling rate.

[0038] Wherein, the first sampling rate is less than the second sampling rate;

[0039] The sample image and the truth image are divided into several sample image blocks and truth image blocks at the same location;

[0040] The sample image patches are input into the neural network to obtain the denoised image patches;

[0041] The loss value between the denoised patch and the corresponding ground truth patch is calculated according to a preset loss function, and the parameters of the neural network are updated based on the loss value to obtain the preset guided neural network.

[0042] In this scheme, the accuracy of the parameters in the preset guided neural network can be improved by training the preset guided neural network in the form of dividing the sample map and the ground map corresponding to the sample medical image into several patch blocks.

[0043] Preferably, the step of dividing the sample image and the truth image into several sample image blocks and truth image blocks at the same location includes:

[0044] Obtain the sample center map and the truth center map at the center positions of the sample map and the truth map;

[0045] The sample center map and the truth center map are divided into several sample map blocks and truth map blocks at the same location.

[0046] In this scheme, by acquiring sample patches and ground truth patches at the center location for training, effective patches can be selected, improving the effectiveness of the training data and thus improving the accuracy of the parameters in the pre-defined guided neural network.

[0047] The present invention also provides a medical image processing system, the medical image processing system comprising:

[0048] The image information acquisition module is used to acquire the medical image to be processed, and the image information of the medical image to be processed;

[0049] The target feature acquisition module is used to acquire target feature information at different preset resolutions based on the image information;

[0050] The guide graph generation module is used to input the target feature information into a preset guide neural network to output guide graphs of the target feature information at different preset resolutions.

[0051] The first mesh generation module is used to generate a first target mesh of the target feature information at the corresponding preset resolution based on the guide image;

[0052] The target image acquisition module is used to acquire the target medical image obtained by denoising the medical image to be processed based on multiple first target grids corresponding to different preset resolutions.

[0053] In this scheme, the rendered medical images are splatting and layered by a guided neural network to generate multiple first target meshes corresponding to different resolution channels. The layered meshes can be filtered quickly, and then the filtered meshes are combined by slicing and other operations to obtain the target medical image. This can quickly and efficiently denoise the medical images to be processed rendered at low sampling rates, thereby realizing real-time physically based volume rendering and ensuring the accuracy and efficiency of medical image rendering.

[0054] Preferably, the target image acquisition module further includes:

[0055] The second mesh generation unit is used to filter the first target mesh to obtain the second target mesh;

[0056] A denoised image acquisition unit is used to slice the second target grid to obtain denoised medical images at different preset resolutions;

[0057] The target image acquisition unit is used to generate the target medical image based on different denoised medical images.

[0058] In this scheme, filtering the first target grid after layering can improve the efficiency of denoising medical images.

[0059] Preferably, the target image acquisition unit is further configured to:

[0060] Obtain the denoising weights corresponding to different preset resolutions;

[0061] Based on the denoising weights, different denoised medical images are fused to obtain the target medical image.

[0062] In this solution, by weighting and combining denoised medical images, the proportion of images with different resolutions can be flexibly changed to obtain the final target medical image as needed.

[0063] Preferably, the target feature information is radiance information;

[0064] The target feature acquisition module includes:

[0065] A radiance acquisition unit is used to downsample the radiance information to acquire the radiance information at different preset resolutions.

[0066] In this scheme, radiance information at different resolutions is obtained by downsampling the radiance information, and then a grid is generated with multiple resolution channels, which can better denoise the local and overall medical images separately.

[0067] Preferably, the medical image processing system further includes:

[0068] Other feature acquisition modules are used to acquire other feature information besides the radiance information based on the image information;

[0069] The other feature information includes at least one of albedo information, normal information, and depth information;

[0070] The guide map generation module is also used to input the other feature information and the radiance information into the preset guide neural network to output the guide map of the radiance information at different preset resolutions.

[0071] In this scheme, the radiance information is guided by a pre-defined neural network through other feature information to generate grid values ​​on a bilateral grid, which can improve the accuracy of grid generation and thus improve the accuracy and effectiveness of medical image processing.

[0072] Preferably, the medical image processing system further includes:

[0073] The temporal filtering module is used to calculate the average value of the actual radiance information of the current frame medical image and the first radiance information obtained after denoising the previous frame medical image, so as to use it as the target radiance information of the current frame medical image.

[0074] In this scheme, by using temporal filtering of radiance information, flickering caused by inconsistent brightness in local areas between consecutive frames can be avoided, thereby improving the quality of medical image rendering.

[0075] Preferably, the medical image processing system further includes a network construction module for constructing the preset guided neural network;

[0076] The network construction module includes:

[0077] The sample image acquisition unit is used to acquire several sample medical images.

[0078] The rendering unit is used to render the volume data of each of the sample medical images, and to obtain the sample image at the first sampling rate and the ground truth image at the second sampling rate respectively.

[0079] Wherein, the first sampling rate is less than the second sampling rate;

[0080] A segmentation unit is used to segment the sample image and the truth image into several sample image blocks and truth image blocks at the same location;

[0081] The denoising unit is used to input sample image patches into the neural network and obtain denoised image patches.

[0082] The network training unit is used to calculate the loss value between the denoised patch and the corresponding ground truth patch according to a preset loss function, and update the parameters of the neural network based on the loss value to obtain the preset guided neural network.

[0083] In this scheme, the accuracy of the parameters in the preset guided neural network can be improved by training the preset guided neural network in the form of dividing the sample map and the ground map corresponding to the sample medical image into several patch blocks.

[0084] Preferably, the segmentation unit is further configured to:

[0085] Obtain the sample center map and the truth center map at the center positions of the sample map and the truth map;

[0086] The sample center map and the truth center map are divided into several sample map blocks and truth map blocks at the same location.

[0087] In this scheme, by acquiring sample patches and ground truth patches at the center location for training, effective patches can be selected, improving the effectiveness of the training data and thus improving the accuracy of the parameters in the pre-defined guided neural network.

[0088] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and used to run on the processor, wherein the processor executes the computer program to implement the above-described medical image processing method.

[0089] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described medical image processing method.

[0090] Based on common knowledge in the field, the preferred conditions can be combined arbitrarily to obtain various preferred embodiments of the present invention.

[0091] The positive and progressive effects of this invention are as follows: by guiding a neural network to perform splatting layering on the rendered medical images, multiple first target meshes corresponding to different resolution channels are generated. The layered meshes can be quickly filtered, and then the filtered meshes are combined by slicing and other operations to obtain the target medical image. This can quickly and efficiently denoise the medical images to be processed rendered at low sampling rates, thereby ensuring real-time physically based volume rendering and guaranteeing the accuracy and efficiency of medical image rendering. Attached Figure Description

[0092] Figure 1 This is a first flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0093] Figure 2 This is a second flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0094] Figure 3 This is the third flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0095] Figure 4 This is the fourth flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0096] Figure 5 This is the fifth flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0097] Figure 6 This is the sixth flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0098] Figure 7 This is the seventh flowchart of the medical image processing method according to Embodiment 1 of the present invention;

[0099] Figure 8 This is a schematic diagram of the first module of the medical image processing system according to Embodiment 2 of the present invention;

[0100] Figure 9 This is a schematic diagram of the second module of the medical image processing system according to Embodiment 2 of the present invention;

[0101] Figure 10 This is a schematic diagram of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0102] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0103] Example 1

[0104] This embodiment provides a medical image processing method; see [link to relevant documentation]. Figure 1 The medical image processing method includes:

[0105] S1. Acquire the medical image to be processed, and the image information of the medical image to be processed;

[0106] The main improvement of this invention lies in the denoising process after volume data rendering. The medical image to be processed in step S1 is the image obtained after rendering. Unlike surface rendering, since the radiative transfer equation of volume rendering needs to be sampled and integrated within the volume data, if it is not enough to extract only the information of the first collision of light rays, the image information of the medical image to be processed needs to be obtained, including the radiance information obtained from the first and second collisions of light rays, as well as other feature information such as normal, depth, and albedo.

[0107] S2. Based on image information, obtain target feature information at different preset resolutions;

[0108] Step S2, based on image information, acquires target feature information at different resolutions, such as radiance information. High-resolution information can be used to depict precise features of local areas in medical images, while low-resolution information can be used to depict the overall contour features of medical images.

[0109] S3. Input the target feature information into the preset guidance neural network to output guidance maps of the target feature information at different preset resolutions;

[0110] Step S3 inputs target feature information of different resolutions into the guiding neural network through multiple channels to generate corresponding guiding maps. For example, guiding maps e0, e1, and e2 can be generated through three resolution channels.

[0111] S4. Based on the guiding image, generate the first target mesh with target feature information at the corresponding preset resolution;

[0112] Step S4, based on the aforementioned guiding graph, splatters the input target feature information under the guidance of the guiding graph to generate corresponding first target meshes, such as mesh 0, mesh 1, and mesh 2. The mesh generation uses a Bilateral Grid algorithm.

[0113] S5. Based on multiple first target grids corresponding to different preset resolutions, obtain the target medical image obtained by denoising the medical image to be processed.

[0114] Step S5 performs filtering, slicing, and weighted combination operations on the first target grid to obtain the denoised target medical image.

[0115] In this scheme, the rendered medical images are splatting and layered by a guided neural network to generate multiple first target meshes corresponding to different resolution channels. The layered meshes can be filtered quickly, and then the filtered meshes are combined by slicing and other operations to obtain the target medical image. This can quickly and efficiently denoise the medical images to be processed rendered at low sampling rates, thereby realizing real-time physically based volume rendering and ensuring the accuracy and efficiency of medical image rendering.

[0116] In a feasible solution, such as Figure 2 As shown, step S5 includes:

[0117] S51. Filter the first target mesh to obtain the second target mesh;

[0118] S52. Slice the second target grid to obtain denoised medical images at different preset resolutions;

[0119] S53. Generate target medical images based on different denoised medical images.

[0120] Specifically, a kernel filtering operation can be performed on the first target mesh to obtain the second target mesh; the filtering kernel can be a tent filter. Then, slicing is performed on the second target mesh, which is the reverse process of guided mesh generation, reconstructing the filtered second target mesh into a denoised medical image. Finally, the target medical image is generated based on the denoised medical images at different resolutions.

[0121] In this scheme, filtering the first target grid after layering can improve the efficiency of denoising medical images.

[0122] In a feasible solution, such as Figure 3 As shown, step S53 includes:

[0123] S531. Obtain the denoising weights corresponding to different preset resolutions;

[0124] S532. Based on the denoising weights, different denoised medical images are fused to obtain the target medical image.

[0125] For example, for denoised medical images 0, 1, and 2 at three resolutions, with weights w0, w1, and w2 respectively, the denoised medical images are combined by weighted average according to their corresponding weights to obtain the target medical image.

[0126] In this solution, by weighting and combining denoised medical images, the proportion of images with different resolutions can be flexibly changed to obtain the final target medical image as needed.

[0127] In a feasible solution, the target feature information is radiance information;

[0128] like Figure 4 The step S2 includes:

[0129] S21. Based on the image information, downsample the radiance information to obtain radiance information at different preset resolutions.

[0130] Specifically, radiance information is downsampled according to different downsampling ratios to obtain radiance information at different preset resolutions.

[0131] In this scheme, radiance information at different resolutions is obtained by downsampling the radiance information, and then a grid is generated with multiple resolution channels, which can better denoise the local and overall medical images separately.

[0132] In a feasible solution, such as Figure 5 As shown, step S2 includes:

[0133] S22. Based on image information, obtain other feature information besides radiance information;

[0134] Other feature information includes at least one of albedo information, normal information, and depth information;

[0135] Step S3 includes:

[0136] S31. Input other feature information and radiance information into the preset guidance neural network to output guidance maps of radiance information at different preset resolutions.

[0137] Specifically, albedo information, normal information, depth information, and other feature information are input into a preset guiding neural network to generate a guiding map and help the preset guiding neural network determine how the radiance information is mapped to the grid value on the first target grid.

[0138] In this scheme, the radiance information is guided by a pre-defined neural network through other feature information to generate grid values ​​on a bilateral grid, which can improve the accuracy of grid generation and thus improve the accuracy and effectiveness of medical image processing.

[0139] In a feasible approach, medical image processing methods also include:

[0140] The average value of the actual radiance information of the current frame medical image and the first radiance information obtained after denoising of the previous frame medical image is calculated and used as the target radiance information of the current frame medical image.

[0141] Specifically, if the radiance of consecutive frames of medical images is inconsistent in the local area after denoising, it will cause flickering. Therefore, the actual radiance information of the current frame of medical image and the first radiance information of the previous frame of medical image after denoising should be fused by temporal filtering. That is, the average value of the actual radiance information of the current frame of medical image and the first radiance information of the previous frame of medical image after denoising should be calculated as the target radiance information of the current frame of medical image.

[0142] In this scheme, by using temporal filtering of radiance information, flickering caused by inconsistent brightness in local areas between consecutive frames can be avoided, thereby improving the quality of medical image rendering.

[0143] In one feasible approach, the steps for constructing a pre-defined guiding neural network include:

[0144] Acquire several sample medical images;

[0145] Render the volume data of each sample medical image to obtain the sample image at the first sampling rate and the ground truth image at the second sampling rate.

[0146] The first sampling rate is less than the second sampling rate;

[0147] The sample map and the truth map are divided into several sample map blocks and truth map blocks at the same location;

[0148] The sample image patches are input into the neural network to obtain the denoised image patches;

[0149] The loss value between the denoised patch and the corresponding ground truth patch is calculated based on the preset loss function, and the parameters of the neural network are updated based on the loss value to obtain the preset guided neural network.

[0150] Specifically, the volume data of the sample medical images is rendered at a low sampling rate (e.g., 32 times / pixel) to generate information such as radiance, albedo, normal, and depth for the first and second collisions under the low sampling rate. All of this information is saved as a high dynamic range image file as a sample image. Then, the same volume data of the sample medical images is rendered at a high sampling rate (e.g., 4096 times / pixel) to generate information such as radiance, albedo, normal, and depth for the first and second collisions under the high sampling rate. All of this information is saved as a high dynamic range image file as a ground truth image.

[0151] Each pair of sample and ground truth images is divided into several sample and ground truth patches at the same location. These sample and ground truth patches are used for training the pre-defined guided neural network. The sample patches are input into the neural network through a multi-resolution channel to generate a bilateral grid. After filtering, slicing, and weighted combination, denoised patches are obtained. Based on a pre-defined loss function, the loss value between the corresponding ground truth patch and the denoised patch is calculated, and the difference is backpropagated back into the neural network to update the weights, thus obtaining the pre-defined guided neural network.

[0152] In this scheme, the accuracy of the parameters in the preset guided neural network can be improved by training the preset guided neural network in the form of dividing the sample map and the ground map corresponding to the sample medical image into several patch blocks.

[0153] In a feasible approach, the steps of dividing the sample map and the truth map into several sample map patches and truth map patches at the same location include:

[0154] Obtain the sample center map and the truth center map at the center position of the sample map and the truth map;

[0155] The sample center map and the truth center map are divided into several sample map blocks and truth map blocks at the same location.

[0156] Specifically, since the effective part of medical images is often concentrated in the middle of the screen, with a large amount of black background around it, in order to prevent too many pure black patches from affecting the training effect of the neural network, sample patches and ground truth patches should be obtained from the center of the image.

[0157] In this scheme, by acquiring sample patches and ground truth patches at the center location for training, effective patches can be selected, improving the effectiveness of the training data and thus improving the accuracy of the parameters in the pre-defined guided neural network.

[0158] The following is combined Figure 6 The overall flowchart of medical image processing and Figure 7 The denoising flowchart illustrates the complete implementation process of the medical image processing method provided in this embodiment using a specific implementation method:

[0159] (I) Importing Patient Data

[0160] Read medical imaging data such as CT (Computed Tomography) and MR (Magnetic Resonance Imaging) from the hard drive and import them into the visualization engine.

[0161] (II) Rendering

[0162] The visualization engine rasterizes the screen space. Then, to map the volume data to pixel values ​​in screen space, it emits n (n<64) rays from each rasterized pixel to sample the pixel values ​​of the volume data. After path tracing and calculation of the radiative transfer equation, it obtains n-spp (sampling per pixel) low-sampling feature information, including radiance r. i Other feature information f i .

[0163] Unlike surface rendering, volume rendering requires distance sampling and integration within the volume data for its radiative transfer equation. Extracting only the initial collision information of rays is insufficient. Therefore, we extract the radiance r... i This includes the radiance obtained from the first and second collisions of light, and similarly, other feature information f i This includes the normals, depth, and albedo obtained from the initial collision of light rays, as well as the normals, depth, and albedo from the second collision. The features of the second collision retain more rendering information specific to volume rendering, allowing the neural network to better learn its noise characteristics. All of the above information is generated by the visualization engine during the rendering stage and stored in the G-Buffer (graphics buffer) for later use.

[0164] (III) Noise Reduction

[0165] 1)Reference Figure 3 The radiance of image i obtained through the rendering stage is r i Other feature information is f i The radiance r of the previous frame i-1 Radiance r of the current frame i Perform time-lapse filtering and fusion to obtain a new r i The purpose of this step is to prevent flickering caused by inconsistent brightness in the local area between consecutive frames after denoising.

[0166] 2) Radiance r i The data is passed to the downsampling module for downsampling.

[0167]

[0168] Where H, W, and D are the height, width, and depth of the projected bilateral grid, respectively, meaning the resolution of the bilateral grid is H*W*D, and η h η w η d These represent the minimum downsampling factor for the height, width, and pixel depth of the original medical image, respectively. h and w represent the height and width of the original medical image, d represents the range of values ​​for the guide image (default d = 255), and m represents the downsampling factor. For grid 0 (original image), m = 1; for grid 1, m = 2; and for grid 2, m = 4.

[0169] 3) r i with f i Simultaneously, the input is fed into the guiding neural network, which uses a CNN (Convolutional Neural Network) architecture. For real-time performance, a lightweight guiding neural network with only two convolutional layers (128*128*20 + 128*128*5) is used to guide r. i Perform splatting. Guide the neural network to generate guide maps e0, e1, and e2 in the three resolution channels respectively, such that the input radiance r i Guided by the master graph, grids 0, 1, and 2 are generated respectively. The grid coordinates are...

[0170]

[0171] Where u and v are the coordinates of the width and height axes of the original, undenoised medical image, respectively. For example, if the original image has a resolution of 720*1280, then h = 720, w = 1280, u ∈ [0, 1279], v ∈ [0, 719], and u and v are both positive integers; j = (u, v) is the pixel coordinate of the noisy image i. That is, the feature (r) at j = (u, v) i f i Neural network with input parameter θ The output value is the guided bilateral grid value g(j).

[0172] 4) Perform kernel filtering operations on grids 0, 1, and 2 respectively, using tent filtering kernels:

[0173] T(x,y,z)=max(1-|x|,0)·max(1-|y|,0)·max(1-|z|,0)

[0174] Where x, y, and z are the coordinates of the bilateral grid reference system.

[0175] The filtered grid is

[0176]

[0177] w q＝(x，y，z)，j＝(u，v) =T(g) i (j)-q)

[0178] Where q = (x, y, z) are the coordinates of pixel j = (u, v) of the undenoised medical image projected onto the bilateral grid.

[0179] 5) Perform the inverse process of guided mesh generation to reconstruct the filtered mesh into image i, i.e.

[0180]

[0181] Among them, G′ i Mesh grids with a value of 0 contribute nothing to slicing and therefore do not need to be considered.

[0182] 6) The denoised images 0, 1, and 2 are combined by weighted average, with weights w0, w1, and w2 respectively.

[0183] 7) Output the final denoised image i (i.e. the target medical image), and copy it as the input of the previous frame for the next frame i+1 time filtering processing.

[0184] (iv) Post-processing

[0185] The obtained high dynamic range image is converted into a low dynamic range image by tone mapping and gamma correction and then displayed on the screen.

[0186] In addition, the training process for the pre-defined guided neural network is as follows:

[0187] 1) Dataset generation

[0188] Training a neural network requires pairs of sample images and ground truth images. For sample image generation, the collected sample medical body data is input into the volume rendering engine to generate radiance, albedo, normal, and depth for the first and second collisions of light rays at a low sampling rate. Radiance, albedo, and normal are represented by RGB (red, green, blue) three channels, while depth is represented by a single channel. All of these are saved as high dynamic range (HDR) image files as sample images. For ground truth image generation, the rendering engine performs high sampling rate (e.g., 4096 samples / pixel) rendering on the same sample medical body data as the sample images, and the final rendered image is saved as an HDR image file.

[0189] In addition, the validity of the data needs to be considered. Since the selected network convolutional layer is 128*128*20, each pair of sample images and ground truth images needs to be randomly cropped to n at the same position. p share w p ×h p ×c p w p <w,h p <h, where w p =128,h p =128, c p n is the number of channels. p The value can be customized based on the image size; the default value is 50 for 1280*720 resolution images. This patch is used for training on each frame during training.

[0190] Furthermore, due to the unique nature of medical data, the data objects are often displayed in the center of the screen, surrounded by a large black background. To prevent excessive pure black sample image-ground image patch pairs from being cropped out, which could negatively impact training performance, the patch position will be determined by randomly offsetting the image from its center position by a certain amount. Let x... p y p These are the starting coordinates of the patch, and the four corner coordinates of the patch are (x, y, y). p y p ), (x p +w p y p ), (x p y p +h p ), (x p +w p y p +h p ),but:

[0191] x p =(ww p ) / 2+x offset

[0192] y p =(hh) p ) / 2+y offset

[0193]

[0194]

[0195] Where N(0, 0.1) represents a value with an expected value of 0 and a standard deviation of 0. The normal distribution indicates that the offset of the patch is within the range of the patch edge and the screen edge and follows a normal distribution, mostly within the range of the volumetric data display in the center of the screen.

[0196] 2) Training process

[0197] The sample image patch is input into the neural network through a multi-resolution channel to generate a two-sided grid. After filtering, slicing, and weighted combination, a denoised image patch is obtained. According to the preset loss function, the loss value between the corresponding ground truth image patch and the denoised image patch is calculated, and the difference is backpropagated back to the neural network to update the weight values ​​in the neural network, so as to obtain the preset guided neural network.

[0198] The loss function chosen is the L1-loss (a loss function) which is the difference between the radiance per pixel in the ground truth image and the radiance per pixel in the sample image.

[0199]

[0200] Among them, R i For the radiance of the reconstructed denoised patch, t i This represents the radiance of the pixel corresponding to the ground truth patch. Since the guiding neural network is only at the front end of the overall pipeline, the backpropagation calculation of the loss function needs to be divided into three steps:

[0201] i) Calculate the gradient from the reconstructed, denoised patch to the mesh.

[0202] ii) Calculate the gradient from the mesh to the guiding network

[0203] iii) The final gradient is obtained by multiplying the gradients in three steps according to the chain rule:

[0204]

[0205] The derivation process is too complex to be detailed here.

[0206] It should be noted that the volume data used in this invention can be any renderable data generated by any medical device; the number of layers and size of the neural network in this invention can be modified according to requirements. For example, if it is not used for real-time rendering, i.e., the time requirement is not high, a neural network with 5-7 convolutional layers can be used to increase the denoising effect. Any modification to the network structure will affect its computation time; the rendering features and their number of channels used in this invention are not limited to those mentioned in the text and can be changed according to requirements; the loss function used in this invention is not limited to L1-loss, but can also be L2-loss (a loss function), cross-entropy, etc.

[0207] The medical image processing method provided in this embodiment uses a guided neural network to perform splatting layering on the rendered medical image, generating multiple first target meshes corresponding to different resolution channels. The layered meshes can quickly complete filtering processing, and then the filtered meshes are combined by slicing and other operations to obtain the target medical image. This method can quickly and efficiently denoise medical images rendered at low sampling rates, thereby achieving real-time physically based volume rendering. In addition, the method trains a preset guided neural network by patching the sample image and the ground truth image and selecting patches at the center of the image, which improves the overall performance of the neural network and ensures the accuracy and efficiency of medical image rendering and denoising.

[0208] Example 2

[0209] This embodiment provides a medical image processing system, referring to... Figure 8 The medical image processing system includes:

[0210] Image information acquisition module 1 is used to acquire the medical image to be processed and the image information of the medical image to be processed;

[0211] Target feature acquisition module 2 is used to acquire target feature information at different preset resolutions based on image information;

[0212] The guide graph generation module 3 is used to input the target feature information into the preset guide neural network to output guide graphs of the target feature information at different preset resolutions.

[0213] The first mesh generation module 4 is used to generate a first target mesh with target feature information at a corresponding preset resolution based on the guide image;

[0214] The target image acquisition module 5 is used to acquire the target medical image obtained by denoising the medical image to be processed based on multiple first target grids corresponding to different preset resolutions.

[0215] In a feasible solution, refer to Figure 9 The target image acquisition module 5 also includes:

[0216] The second mesh generation unit 51 is used to filter the first target mesh to obtain the second target mesh;

[0217] The denoised image acquisition unit 52 is used to slice the second target grid to obtain denoised medical images at different preset resolutions.

[0218] The target image acquisition unit 53 is used to generate target medical images based on different denoised medical images.

[0219] In one feasible embodiment, the target image acquisition unit 53 is further configured to:

[0220] Obtain the denoising weights corresponding to different preset resolutions;

[0221] Based on the denoising weights, different denoised medical images are fused together to obtain the target medical image.

[0222] In a feasible solution, the target feature information is radiance information;

[0223] Target feature acquisition module 2 includes:

[0224] The radiance acquisition unit 21 is used to downsample the radiance information and acquire radiance information at different preset resolutions.

[0225] In one feasible solution, the medical image processing system also includes:

[0226] Other feature acquisition module 6 is used to acquire feature information other than radiance information based on image information;

[0227] Other feature information includes at least one of albedo information, normal information, and depth information;

[0228] The guide map generation module 3 is also used to input other feature information and radiance information into a preset guide neural network to output guide maps of radiance information at different preset resolutions.

[0229] In one feasible solution, the medical image processing system also includes:

[0230] The temporal filtering module 7 is used to calculate the average value of the actual radiance information of the current frame medical image and the first radiance information obtained after denoising the previous frame medical image, so as to use it as the target radiance information of the current frame medical image.

[0231] In one feasible embodiment, the medical image processing system also includes a network construction module 8 for constructing a pre-defined guided neural network;

[0232] Network building block 8 includes:

[0233] Sample image acquisition unit 81 is used to acquire several sample medical images;

[0234] The rendering unit 82 is used to render the volume data of each sample medical image and obtain the sample image at the first sampling rate and the ground truth image at the second sampling rate respectively.

[0235] The first sampling rate is less than the second sampling rate;

[0236] The segmentation unit 83 is used to segment the sample map and the truth map into several sample map blocks and truth map blocks at the same location;

[0237] The denoising unit 84 is used to input sample image patches into the neural network to obtain denoised image patches;

[0238] The network training unit 85 is used to calculate the loss value between the denoised patch and the corresponding ground truth patch according to the preset loss function, and update the parameters of the neural network based on the loss value to obtain the preset guided neural network.

[0239] In one feasible embodiment, the segmentation unit 83 is further used for:

[0240] Obtain the sample center map and the truth center map at the center position of the sample map and the truth map;

[0241] The sample center map and the truth center map are divided into several sample map blocks and truth map blocks at the same location.

[0242] Since the medical image processing system provided in this embodiment is based on the same principle as the medical image processing method provided in Embodiment 1, it will not be described again here.

[0243] The medical image processing system provided in this embodiment uses a guided neural network to splatter and layer rendered medical images, generating multiple first target meshes corresponding to different resolution channels. The layered meshes can be quickly filtered, and then the filtered meshes are combined using slicing and other operations to obtain the target medical image. This system can quickly and efficiently denoise medical images rendered at low sampling rates, thereby achieving real-time physically based volume rendering. In addition, the system trains a preset guided neural network by patching the sample image and the ground truth image and selecting patches at the center of the image, which improves the overall performance of the neural network and ensures the accuracy and efficiency of medical image rendering and denoising.

[0244] Example 3

[0245] This embodiment provides an electronic device. Figure 10 This is a schematic diagram of the electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the medical image processing method of Embodiment 1. Figure 10 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0246] like Figure 10 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0247] Bus 33 includes a data bus, an address bus, and a control bus.

[0248] The memory 32 may include volatile memory, such as random access memory (RAM) 321 and / or cache memory 322, and may further include read-only memory (ROM) 323.

[0249] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0250] The processor 31 executes various functional applications and data processing, such as the medical image processing method of Embodiment 1 of this disclosure, by running computer programs stored in the memory 32.

[0251] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 35. Furthermore, the model-generating device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 36. Figure 10 As shown, network adapter 36 communicates with other modules of the model-generated device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 30, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0252] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0253] Example 4

[0254] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the program module initialization method of Embodiment 1.

[0255] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0256] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to execute the initialization method of the program module implementing Embodiment 1.

[0257] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0258] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. A medical image processing method, characterized in that, The medical image processing method includes: Acquire the medical image to be processed, and the image information of the medical image to be processed; Based on the image information, target feature information at different preset resolutions is obtained; The target feature information is input into a preset guiding neural network to output guiding maps of the target feature information at different preset resolutions; based on the guiding maps, a first target grid of the target feature information at the corresponding preset resolution is generated; Based on multiple first target grids corresponding to different preset resolutions, obtain the target medical image obtained by denoising the medical image to be processed; The step of obtaining the target medical image obtained by denoising the medical image to be processed based on multiple first target grids corresponding to different preset resolutions includes: The first target mesh is filtered to obtain the second target mesh; The second target grid is sliced ​​to obtain denoised medical images at different preset resolutions; The target medical image is generated based on the different denoised medical images; The steps for constructing the preset guiding neural network include: Acquire several sample medical images; The volume data of each of the sample medical images is rendered to obtain the sample image at the first sampling rate and the ground truth image at the second sampling rate. Wherein, the first sampling rate is less than the second sampling rate; The sample image and the truth image are divided into several sample image blocks and truth image blocks at the same location; The sample image patches are input into the neural network to obtain the denoised image patches; The loss value between the denoised patch and the corresponding ground truth patch is calculated according to a preset loss function, and the parameters of the neural network are updated based on the loss value to obtain the preset guided neural network.

2. The medical image processing method as described in claim 1, characterized in that, The step of generating the target medical image based on the different denoised medical images includes: Obtain the denoising weights corresponding to different preset resolutions; Based on the denoising weights, different denoised medical images are fused to obtain the target medical image.

3. The medical image processing method as described in claim 1, characterized in that, The target feature information is radiance information; The steps for obtaining target feature information at different preset resolutions include: The radiance information is downsampled to obtain the radiance information at different preset resolutions.

4. The medical image processing method as described in claim 3, characterized in that, The medical image processing method further includes: Based on the image information, other feature information besides the radiance information is obtained; The other feature information includes at least one of albedo information, normal information, and depth information; The other feature information and the radiance information are input into the preset guidance neural network to output the guidance map of the radiance information at different preset resolutions.

5. The medical image processing method as described in claim 3 or 4, characterized in that, The medical image processing method further includes: The average value of the actual radiance information of the current frame medical image and the first radiance information obtained after denoising of the previous frame medical image is calculated and used as the target radiance information of the current frame medical image.

6. The medical image processing method as described in claim 1, characterized in that, The step of dividing the sample image and the truth image into several sample image blocks and truth image blocks at the same location includes: Obtain the sample center map and the truth center map at the center positions of the sample map and the truth map; The sample center map and the truth center map are divided into several sample map blocks and truth map blocks at the same location.

7. A medical image processing system, characterized in that, The medical image processing system includes: The image information acquisition module is used to acquire the medical image to be processed, and the image information of the medical image to be processed; The target feature acquisition module is used to acquire target feature information at different preset resolutions based on the image information; The guide graph generation module is used to input the target feature information into a preset guide neural network to output guide graphs of the target feature information at different preset resolutions. The first mesh generation module is used to generate a first target mesh of the target feature information at the corresponding preset resolution based on the guide image; The target image acquisition module is used to acquire the target medical image obtained by denoising the medical image to be processed based on multiple first target grids corresponding to different preset resolutions. The target image acquisition module includes: The second mesh generation unit is used to filter the first target mesh to obtain the second target mesh; A denoised image acquisition unit is used to slice the second target grid to obtain denoised medical images at different preset resolutions; The target image acquisition unit is used to generate the target medical image based on different denoised medical images; The medical image processing system also includes: The network construction module is used to construct the preset guided neural network; The network construction module includes: The sample image acquisition unit is used to acquire several sample medical images. The rendering unit is used to render the volume data of each of the sample medical images, and to obtain the sample image at a first sampling rate and the ground truth image at a second sampling rate, respectively; wherein the first sampling rate is less than the second sampling rate; A segmentation unit is used to segment the sample image and the truth image into several sample image blocks and truth image blocks at the same location; The denoising unit is used to input sample image patches into the neural network and obtain denoised image patches. The network training unit is used to calculate the loss value between the denoised patch and the corresponding ground truth patch according to a preset loss function, and update the parameters of the neural network based on the loss value to obtain the preset guided neural network.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the medical image processing method according to any one of claims 1-6.

Images