A deep learning-based automatic visualization three-dimensional image correction and reconstruction method

Abstract

This invention relates to the field of image correction technology and discloses an automatic visualization method for 3D image correction and reconstruction based on deep learning. By processing and analyzing the original 3D medical image and the type of medical department, a target 3D medical image and image correction content are generated. Several target correction models are matched. Considering the concurrent image processing resources of the 3D image correction server, a set of target correction models selected from the target correction model library and deployed on the 3D image correction server is solved to ensure that the 3D image correction server meets the limitations of video memory capacity, the requirements of candidate correction subjects requested by the user, and the optimal medical experience within the target time period. A correction model selection strategy and an image correction content selection strategy are generated, thereby controlling the deployment of correction models and the process of image correction, reconstruction, and output on the 3D image correction server. This achieves automated and visualized correction and reconstruction of 3D medical images based on deep learning in clinical diagnosis and treatment scenarios.

Description

TECHNICAL FIELD

[0001] The application relates to the field of image correction, in particular to an automatic visual three-dimensional image correction and reconstruction method based on deep learning. BACKGROUND

[0002] Three-dimensional medical images (such as CT and MRI) reflect the structure and spatial relationship of human organs and tissues, and are important basis for clinical diagnosis, surgical planning and efficacy evaluation. However, during the acquisition process, image defects or blurring often occur due to factors such as motion and artifacts, affecting the judgment of doctors.

[0003] Traditional three-dimensional medical image repair methods mainly include gray scale interpolation method, image registration method and edge extension technology based on morphology. These methods are widely used in early medical image processing, and have the characteristics of simple calculation and low implementation cost. The gray scale interpolation method (such as nearest neighbor interpolation, linear interpolation and spline interpolation) calculates the gray scale value of the missing area by using the pixel gray scale information of the existing slices, and completes the image completion. The image registration method uses the spatial correspondence between multiple images to align the images acquired at different times or different angles, and then fills the missing area. The morphological method based on edge extension attempts to extend the edge structure of the image to restore the boundary information of the missing part. These traditional technologies are commonly used in early CT image reconstruction and partial MRI scan repair tasks.

[0004] Although the traditional methods have the advantages of simple implementation and high computational efficiency, their effect is obviously limited when facing medical images with complex anatomical structure and high three-dimensional continuity requirements. The interpolation method cannot restore the structural details, which is easy to cause image blur and artifact; the image registration depends on the quality of the reference image, and is not sensitive to displacement error and organ deformation, which is difficult to accurately repair; the morphological method relies too much on local edge features, and it is difficult to restore the anatomical true form in the missing area. In addition, these methods generally lack understanding of the context semantic information of the image, and it is difficult to infer reasonable completion results according to the surrounding structure, which cannot meet the clinical needs of high-quality repair of modern medical images.

[0005] In recent years, deep learning technology has developed rapidly in the field of medical image processing, especially in the task of three-dimensional image repair and reconstruction, which has shown strong modeling ability and expression ability. The three-dimensional convolutional neural network model represented by UNet3D and V-Net can automatically learn anatomical structure and spatial features from a large number of medical images through end-to-end training, and realize high-precision prediction and repair of image missing areas. Existing research shows that the model based on UNet3D has achieved better performance than traditional methods in the repair of three-dimensional images such as lung CT and brain MRI, and has significantly improved the continuity and visual quality of the image.

[0006] Although the deep learning-based three-dimensional medical image restoration method has made significant progress in structural restoration accuracy, current technologies mostly focus on the optimization of network structure, and there are limitations of imbalance between memory occupation and processing efficiency in the actual application level. In the diagnosis and treatment scene, the correction, reconstruction and visualization of three-dimensional medical images are important support for clinical diagnosis, especially during the peak period of diagnosis and treatment, doctors in different departments (such as radiology, orthopedics, neurology, etc.) need to rely on the correction server to quickly process image data to provide accurate basis for disease diagnosis and treatment plan. However, the deep learning-based three-dimensional medical image correction model faces severe hardware resource limitations when deployed:

[0007] (1) If a special small area correction model (each model is only optimized for 1-2 specific structures) is deployed for different organs or local structures (such as lung nodules, brain hippocampus, lumbar intervertebral disc, etc.), although it can improve the correction accuracy by focusing on the target area, it will lead to a sharp increase in the number of models. The three-dimensional model itself handles three-dimensional pixel (voxel) data, and the memory occupation of a single model is significantly higher than that of a two-dimensional model (such as a UNet3D model for lung sub-section, the memory occupation may reach 8-12GB), and when dozens of small area models run concurrently, the total memory demand is likely to exceed the hardware limit of the server (such as a GPU with 24GB of memory on a single card, which is difficult to support more than 4 such models), causing memory overflow, model loading failure, and other problems, directly interrupting the clinical image processing process.

[0008] (2) If a large area correction model (such as a V-Net model covering the whole chest or whole head) is used, multiple organs (such as the chest model processing lung, heart, mediastinum) can be compatible with a single model, which can reduce the number of models and reduce the total memory occupation (the memory of a single large model may be controlled at 15-20GB), but it will cause a sharp decline in processing efficiency due to indiscriminate correction. Large area models need to calculate all voxels in the input image (including organs or background areas that are not the target of diagnosis and treatment), resulting in a large number of unnecessary correction operations, which increases the processing delay of a single image from hundreds of milliseconds to 2-3 seconds, causing task queue congestion during peak periods (such as processing thousands of three-dimensional images per day), which seriously affects the clinical diagnosis and treatment rhythm.

[0009] Therefore, how to realize the automatic and visual correction and reconstruction of three-dimensional medical images based on deep learning in the clinical diagnosis and treatment scene, alleviate the hardware resource limitations of imbalance between memory occupation and processing efficiency when facing a large number of three-dimensional medical image correction demands, and improve the correction efficiency of three-dimensional medical image correction and the clinical diagnosis and treatment experience is a technical problem that needs to be solved. SUMMARY

[0010] The main purpose of the present application is to provide an automatic visualization three-dimensional image correction and reconstruction method based on deep learning, aiming to solve at least one of the above technical problems.

[0011] To achieve the above purpose, the present application provides an automatic visualization three-dimensional image correction and reconstruction method based on deep learning, which comprises the following steps:

[0012] The original three-dimensional medical image in each user request in the target period is preprocessed to generate a target three-dimensional medical image containing a region to be repaired and a normal region;

[0013] Based on the type of diagnosis and treatment department in each user request, the image correction content is determined; wherein the image correction content includes a plurality of candidate correction subjects and a correction priority list;

[0014] Using the plurality of candidate correction subjects of the image correction content, match the plurality of target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library; wherein each target correction model is configured to perform three-dimensional medical image correction on at least one candidate correction subject;

[0015] According to the correction priority list and the plurality of target correction models corresponding to each user request in the target period, considering the image concurrent processing resources of the three-dimensional image correction server, generate the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period;

[0016] According to the correction model selection strategy and the image correction content selection strategy, control the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period, and send the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal.

[0017] Optionally, the pre-processing step of the original three-dimensional medical image in each user request to generate a target three-dimensional medical image containing a region to be repaired and a normal region, specifically includes:

[0018] Performing a first preprocessing action on the original three-dimensional medical image in each user request, including resampling, direction alignment, intensity normalization and outlier removal, to obtain a preliminary preprocessed image;

[0019] Performing a second preprocessing action on the preliminary preprocessed image, including mask extraction and repair region division, to generate a target three-dimensional medical image containing a region to be repaired and a normal region.

[0020] Optionally, a first preprocessing action including resampling, direction alignment, intensity normalization and outlier removal is performed on the original three-dimensional medical image in each user request to obtain a preliminary preprocessed image, specifically including:

[0021] Resampling is configured to resample the original three-dimensional medical image in each user request to a uniform voxel spacing in different resolutions or anisotropy;

[0022] Direction alignment is configured to align the image coordinate system direction of the original three-dimensional medical image in each user request by the SetDirection method;

[0023] Intensity normalization is configured to eliminate inter-device gray scale differences by using window width and window level adjustment for the original three-dimensional medical image in each user request;

[0024] Outlier removal is configured to remove extreme values caused by air or metal contrast by threshold method for the original three-dimensional medical image in each user request.

[0025] Optionally, a second preprocessing action including mask extraction and repair region division is performed on the preliminary preprocessed image to generate a target three-dimensional medical image containing a repair region and a normal region, specifically including:

[0026] The gray scale segmentation threshold is determined by the Otsu adaptive threshold segmentation algorithm to extract the tissue region; the BinaryFillhole operation is performed on the initial binary mask to fill the holes;

[0027] The maximum connected domain is extracted by ConnectedComponent analysis to exclude isolated blocks and noise interference; the morphological smoothing processing is applied to the mask edge to obtain the final mask by closing operation;

[0028] On the basis of the final mask, the Subtract operation is performed to identify the areas below the preset intensity, structure fracture or shedding from the original image and divide them into the repair region.

[0029] Optionally, based on the type of the diagnosis and treatment department in each user request, the image correction content is determined, specifically including:

[0030] The diagnosis and treatment department category in each user request is extracted, the diagnosis and treatment disease list corresponding to the diagnosis and treatment department category is queried, and the associated organ tissue structure of each diagnosis and treatment disease in the diagnosis and treatment disease list is analyzed;

[0031] The associated organ tissue structure of each diagnosis and treatment disease is summarized, and the image correction content requested by each user is generated according to the preset importance of each associated organ tissue structure in each diagnosis and treatment department category. Several candidate correction subjects and correction priority lists are generated.

[0032] Optionally, the several candidate correction subjects of the image correction content are used to match the several target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library, specifically including:

[0033] Accessing the three-dimensional medical image correction model library; wherein the three-dimensional medical image correction model library stores several candidate correction models, and each candidate correction model is configured to perform three-dimensional medical image correction on different numbers of candidate correction subjects;

[0034] Extracting the several candidate correction subjects in the image correction content, and querying the several candidate correction models containing at least all candidate correction subjects as the several target correction models corresponding to the user request in the three-dimensional medical image correction model library according to the several candidate correction subjects.

[0035] Optionally, the construction of the several candidate correction models in the three-dimensional medical image correction model library specifically includes:

[0036] Collecting three-dimensional medical images with different numbers of candidate correction subjects, randomly generating several cubic occlusion regions in each three-dimensional medical image, and generating pseudo-defect input images as three-dimensional medical image test samples of the candidate correction model;

[0037] The UNet3D network structure in the form of encoder-decoder symmetry is constructed, and the original three-dimensional medical image is used as the learning target. The reconstruction error of the output of the UNet3D network structure and the original three-dimensional medical image is optimized by the L1 loss function, and the learning context structure completion ability is learned.

[0038] The self-supervised learning strategy is used for model training to obtain the candidate correction model for image correction of three-dimensional medical images with different numbers of candidate correction subjects.

[0039] Optionally, according to the correction priority list and the several target correction models corresponding to each user request in the target period, considering the image concurrent processing resources of the three-dimensional image correction server, the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period are generated, specifically including:

[0040] query the image concurrent processing resource of the three-dimensional image correction server; wherein the image concurrent processing resource comprises server video memory capacity and test output delay of the image correction model of the server when performing image correction test on the three-dimensional medical image with different candidate correction subjects;

[0041] construct a target correction model library according to a plurality of target correction models corresponding to each user request, and solve the target correction model set selected from the target correction model library and deployed on the three-dimensional image correction server by using an optimization algorithm, so that the three-dimensional image correction server satisfies a first constraint condition that the sum of the video memory occupation amounts of all the target correction models deployed thereon is less than the server video memory capacity, a second constraint condition that the candidate correction subjects of the top preset proportion of priority in the correction priority list corresponding to each user request are covered in at least one target correction model in all the target correction models, and an optimization target that the minimum test output delay cumulative sum of the three-dimensional medical image of each user request in the target period when performing image correction on the candidate correction subjects of the top preset proportion of priority by using the target correction model set is minimum;

[0042] generate the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period according to the optimization algorithm solution result;

[0043] control the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period according to the correction model selection strategy and the image correction content selection strategy, and send the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal.

[0044] Optionally, the step of generating the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period according to the optimization algorithm solution result specifically comprises:

[0045] generate the correction model selection strategy of the three-dimensional image correction server in the target period according to the target correction model set;

[0046] generate the image correction content selection strategy based on the candidate correction subject of the target correction model corresponding to the minimum test output delay of the three-dimensional medical image of each user request.

[0047] Optionally, the step of controlling the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period according to the correction model selection strategy and the image correction content selection strategy specifically comprises:

[0048] According to the correction model selection strategy, the target correction model set is deployed in the three-dimensional image correction server, the target three-dimensional medical image requested by each user is input into the three-dimensional image correction server, the corresponding target correction model is driven according to the image correction content selection strategy, the three-dimensional image correction is performed on the image correction content in the repair area of the target three-dimensional medical image, and a corrected three-dimensional medical image is obtained;

[0049] The corrected three-dimensional medical image is subjected to visual rendering, the correction and reconstruction of the original three-dimensional medical image are completed, and the corrected three-dimensional medical image obtained by reconstruction is sent to a diagnosis and treatment terminal.

[0050] The application has the beneficial effects that an automatic visual three-dimensional image correction and reconstruction method based on deep learning is provided, the original three-dimensional medical image and the diagnosis and treatment department type are processed and analyzed, the target three-dimensional medical image and the image correction content are generated, a plurality of target correction models are matched, the image concurrent processing resources of the three-dimensional image correction server are considered, the target correction model set selected from the target correction model library and deployed in the three-dimensional image correction server is solved, so that the three-dimensional image correction server meets the video memory capacity limit, the candidate correction subject requirement of the user request and the best diagnosis and treatment experience within the target period, the correction model selection strategy and the image correction content selection strategy are generated, and then the process of the correction model deployment and the image correction, reconstruction and output in the three-dimensional image correction server are controlled, and the automatic and visual correction and reconstruction of the three-dimensional medical image based on deep learning in the clinical diagnosis and treatment scene are realized. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the automatic visual three-dimensional image correction and reconstruction method based on deep learning in the embodiment of the application;

[0052] Figure 2 The structure diagram of the improved lightweight UNet3D network structure in the embodiment of the application;

[0053] Figure 3 The three-dimensional surface rendering diagram of the repaired image obtained by using the image correction model to perform image correction in the embodiment of the application;

[0054] Figure 4 The volume rendering diagram of the repaired image obtained by using the image correction model to perform image correction in the embodiment of the application;

[0055] Figure 5 The axial section view in the axial view of the repaired image obtained by using the image correction model to perform image correction in the embodiment of the application;

[0056] Figure 6A coronal section view in a three-dimensional graph of a repaired image obtained by using an image correction model to correct images in the embodiment of the present application;

[0057] Figure 7 A sagittal section view in a three-dimensional graph of a repaired image obtained by using an image correction model to correct images in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0059] The embodiment of the present application provides an automatic visual three-dimensional image correction and reconstruction method based on deep learning, referring to Figure 1 , Figure 1 The flowchart of the embodiment of the automatic visual three-dimensional image correction and reconstruction method based on deep learning of the present application.

[0060] In the embodiment, an automatic visual three-dimensional image correction and reconstruction method based on deep learning, the method comprises the following steps:

[0061] S1: pre-processing the original three-dimensional medical image in each user request in a target period to generate a target three-dimensional medical image containing a to-be-repaired area and a normal area;

[0062] S2: determining the image correction content based on the type of the diagnosis and treatment department in each user request; wherein the image correction content comprises a plurality of candidate correction subjects and a correction priority list;

[0063] S3: using the plurality of candidate correction subjects of the image correction content to match a plurality of target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library; wherein each target correction model is configured to perform three-dimensional medical image correction on at least one candidate correction subject;

[0064] S4: according to the correction priority list and the plurality of target correction models corresponding to each user request in the target period, considering the image concurrent processing resources of the three-dimensional image correction server, generating a correction model selection strategy and an image correction content selection strategy of the three-dimensional image correction server in the target period;

[0065] S5: according to the correction model selection strategy and the image correction content selection strategy, controlling the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period, and sending the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal.

[0066] It should be noted that the three-dimensional medical image correction model based on deep learning faces serious hardware resource limitation problem when deployed:

[0067] (1) If a special small area correction model (each model is optimized for 1-2 specific structures) is deployed for different organs or local structures (such as lung nodules, brain hippocampus, lumbar intervertebral disc, etc.), although the correction accuracy can be improved by focusing on the target area, the number of models will increase significantly. The memory occupation of a single model of a three-dimensional model is significantly higher than that of a two-dimensional model (for example, the memory occupation of a UNet3D model for lung sub-sections may reach 8-12 GB), and when dozens of small area models run concurrently, the total memory demand is likely to exceed the hardware limit of the server (such as a GPU with 24 GB of memory per card, which is difficult to support more than 4 such models), causing memory overflow, model loading failure and other problems, directly interrupting the clinical image processing process.

[0068] (2) If a large area correction model (such as a V-Net model covering the whole chest or whole head) is used, multiple organs (such as the chest model processing lung, heart, mediastinum) can be compatible with a single model, which can reduce the number of models and reduce the total memory occupation (the memory of a single large model may be controlled in 15-20 GB), but it will cause a sharp decline in processing efficiency due to indiscriminate correction. The large area model needs to calculate all the voxels in the input image (including organs or background areas that are not the target of diagnosis and treatment), which results in a large number of unnecessary correction operations, causing the processing delay of a single image to increase from hundreds of milliseconds to 2-3 seconds, which will cause task queue congestion in peak periods (such as processing thousands of three-dimensional images per day), and will seriously affect the clinical diagnosis and treatment rhythm.

[0069] In order to solve the above problems, the embodiment generates target three-dimensional medical images and image correction content by processing and analyzing the original three-dimensional medical images and the type of diagnosis and treatment department, matches several target correction models, considers the image concurrent processing resources of the three-dimensional image correction server, solves the target correction model set selected from the target correction model library and deployed in the three-dimensional image correction server, so that the three-dimensional image correction server meets the memory capacity limit, the candidate correction subject requirements of the user request and the best diagnosis and treatment experience in the target period, generates the correction model selection strategy and the image correction content selection strategy, and then controls the process of correction model deployment, image correction, reconstruction and output in the three-dimensional image correction server, realizes the automatic and visual correction reconstruction of three-dimensional medical images based on deep learning in the clinical diagnosis and treatment scene.

[0070] In the preferred embodiment, the original three-dimensional medical image in each user request is preprocessed to generate a target three-dimensional medical image containing the area to be repaired and the normal area, which specifically includes:

[0071] S11 performing first preprocessing actions including resampling, direction alignment, intensity normalization and outlier rejection on the original three-dimensional medical image in each user request to obtain a preliminary preprocessed image;

[0072] S12 performing second preprocessing actions including mask extraction and repair region division on the preliminary preprocessed image to generate a target three-dimensional medical image containing a region to be repaired and a normal region.

[0073] Further, the step of performing first preprocessing actions including resampling, direction alignment, intensity normalization and outlier rejection on the original three-dimensional medical image in each user request to obtain a preliminary preprocessed image specifically comprises:

[0074] S111: resampling, configured to resample the original three-dimensional medical image in each user request to a uniform voxel spacing for different resolutions or anisotropy;

[0075] S112: direction alignment, configured to unify the image coordinate system direction by the SetDirection method for the original three-dimensional medical image in each user request;

[0076] S113: intensity normalization, configured to eliminate device-to-device grayscale differences by using window width and window level adjustment for the original three-dimensional medical image in each user request;

[0077] S114: outlier rejection, configured to remove extreme values caused by air or metal contrast by threshold method for the original three-dimensional medical image in each user request.

[0078] Further, the step of performing second preprocessing actions including mask extraction and repair region division on the preliminary preprocessed image to generate a target three-dimensional medical image containing a region to be repaired and a normal region specifically comprises:

[0079] S121: determining a gray scale segmentation threshold by an Otsu adaptive threshold segmentation algorithm to extract a tissue region; performing a BinaryFillhole operation on the initial binary mask to fill holes;

[0080] S122: extracting a maximum connected domain by ConnectedComponent analysis to exclude isolated blocks and noise interference; applying a closing operation to perform morphological smoothing processing on the mask edge to obtain a final mask;

[0081] S123: on the basis of the final mask, identifying a region below a preset intensity, a structure fracture or a shedding region from the original image by a Subtract operation and dividing it as a region to be repaired.

[0082] In practical applications, for the processing of the original three-dimensional medical image to the target three-dimensional medical image, the following execution processes are included: resampling medical images of different resolutions or anisotropies to a uniform voxel spacing (such as 1x1x1 mm3), eliminating the effects brought by the differences of acquisition devices. The SetDirection method is used to unify the image coordinate system direction, ensuring the spatial consistency of subsequent processing. Window width and window level adjustment (such as CT value mapping to [-1000, 2000]) is used to eliminate the gray difference between devices. Threshold method (such as CT value truncation to [-1024, 3071]) is used to remove extreme values caused by air or metal artifacts. After that, the Otsu adaptive threshold segmentation algorithm is used to automatically determine the gray scale segmentation threshold, and the main tissue region (such as the cardiac cavity or muscle structure in the heart CT) is extracted; then, the BinaryFillhole operation is performed on the initial binary mask to fill the holes and ensure the region connectivity; then, the ConnectedComponent analysis is used to extract the maximum connected domain, and the isolated small blocks and noise interference are excluded; finally, the closing operation (i.e., dilation followed by erosion) is applied to the mask edge for morphological smoothing processing, and the final mask with complete boundary and clear structure is obtained. On the basis of this mask, the system identifies the low-intensity, structure broken or detached region from the original image through the Subtract operation, and delimits it as the repair region to be repaired.

[0083] In a preferred embodiment, based on the type of diagnosis and treatment department in each user request, the image correction content determination step specifically includes:

[0084] S21: Extract the diagnosis and treatment department category in each user request, query the diagnosis and treatment disease list corresponding to the diagnosis and treatment department category, and analyze the associated organ tissue structure of each diagnosis and treatment disease in the diagnosis and treatment disease list;

[0085] S22: The associated organ tissue structure of each diagnosis and treatment disease is summarized, and a list of several candidate correction subjects and correction priorities in the image correction content of each user request is generated according to the preset importance of each associated organ tissue structure in each diagnosis and treatment department category.

[0086] On this basis, the several candidate correction subjects of the image correction content are used to match the several target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library step, specifically including:

[0087] S31: Access the three-dimensional medical image correction model library; wherein the three-dimensional medical image correction model library stores several candidate correction models, and each candidate correction model is configured to perform three-dimensional medical image correction on different numbers of candidate correction subjects;

[0088] S32: extract a plurality of candidate correction subjects in the image correction content, and according to the plurality of candidate correction subjects, query a plurality of candidate correction models for at least containing all candidate correction subjects in the three-dimensional medical image correction model library as a plurality of target correction models corresponding to the user request.

[0089] In the embodiment, each user request further contains a diagnosis and treatment department type. According to the diagnosis and treatment department type, a plurality of candidate correction subjects and a correction priority list of each user request can be generated. Then, by accessing the three-dimensional medical image correction model library, a plurality of candidate correction models for at least containing all candidate correction subjects are extracted and queried as a plurality of target correction models corresponding to the user request.

[0090] For example, the diagnosis and treatment department type of the user request can be orthopedics, and the target three-dimensional medical image is a waist image. Then, the plurality of target correction models corresponding to the user request obtained by extraction and query can be a first correction model for image correction of the lumbar vertebrae and the intestinal tract, a second correction model for image correction of the lumbar vertebrae and the kidney, and a third correction model for image correction of the lumbar vertebrae, the intestinal tract and the kidney. Since all of them can correct the lumbar vertebrae of the user request (although it is the correction of the same area image, but the parts trained in the early stage of the model are different, resulting in the generation of the model for image correction of different parts), the three correction models can be determined as a plurality of target correction models based on deep learning corresponding to the user request, and then used for the establishment of the target correction model library and the solution of the target correction model set.

[0091] In the preferred embodiment, the construction of the plurality of candidate correction models in the three-dimensional medical image correction model library specifically includes:

[0092] S311: collect three-dimensional medical images with different numbers of candidate correction subjects, randomly generate a plurality of cubic occlusion regions in each three-dimensional medical image, and generate pseudo-defect input images as three-dimensional medical image test samples of the candidate correction model;

[0093] S312: construct a UNet3D network structure in the form of an encoder-decoder pair, use the original three-dimensional medical image as the learning target, and optimize the reconstruction error of the original three-dimensional medical image and the output of the UNet3D network structure through an L1 loss function to learn the context structure completion ability.

[0094] S313: adopt a self-supervised learning strategy to train the model, and obtain a candidate correction model for image correction of three-dimensional medical images with different numbers of candidate correction subjects.

[0095] In this embodiment, for the construction of several candidate correction models in the three-dimensional medical image correction model library, an improved lightweight UNet3D network structure is adopted, as shown in Figure 2 The overall architecture is in the form of an encoder-decoder symmetry, with strong multi-scale feature extraction and spatial context modeling capabilities. The encoding path extracts high-dimensional features of the three-dimensional medical image layer by layer, the decoding path gradually recovers the image structure, and the shallow texture and deep semantic information are fused through the jump connection, effectively enhancing the structure recovery ability of the missing area. In the training stage, a self-supervised learning strategy is adopted, a plurality of cubic occlusion regions (such as 16x16x16 size) are randomly generated in the original volume image, a pseudo-defect input image is constructed, and the original image is used as the learning target. The reconstruction error between the network output and the original image is optimized by the L1 loss function, guiding the model to learn the context structure completion ability. In the inference stage, a sliding window strategy is adopted to repair and reconstruct the three-dimensional volume image block by block, and finally output the repair image and its corresponding difference highlight map, realizing the high-quality three-dimensional medical image repair with consideration of structure continuity, texture consistency and spatial rationality.

[0096] In the preferred embodiment, according to the correction priority list corresponding to each user request and several target correction models within the target period, considering the image concurrent processing resources of the three-dimensional image correction server, the correction model selection strategy and image correction content selection strategy of the three-dimensional image correction server within the target period are generated, which specifically includes:

[0097] S41: Query the image concurrent processing resources of the three-dimensional image correction server; wherein the image concurrent processing resources include server video memory capacity and test output delay when the image correction model for performing image correction on three-dimensional medical images with different candidate correction subjects is used for image correction test;

[0098] S42: According to the several target correction models corresponding to each user request, a target correction model library is constructed, and an optimization algorithm is used to select and deploy a target correction model set from the target correction model library on the three-dimensional image correction server, so that the three-dimensional image correction server meets the first constraint condition that the sum of the memory occupancy of all target correction models deployed is less than the server video memory capacity, the second constraint condition that the candidate correction subject of the top pre-set proportion priority in the correction priority list corresponding to each user request is covered in at least one target correction model in all target correction models, and the optimization target that the minimum test output delay cumulative sum of each user request three-dimensional medical image in the target period when the target correction model set is used for image correction containing the candidate correction subject of the top pre-set proportion priority is the minimum;

[0099] S43: generating a correction model selection strategy and an image correction content selection strategy of the three-dimensional image correction server in the target period according to the solving result of the optimization algorithm;

[0100] S44: controlling a target correction model deployed in the three-dimensional image correction server to correct and reconstruct a target three-dimensional medical image input by each user in the target period according to the correction model selection strategy and the image correction content selection strategy, and sending the reconstructed corrected three-dimensional medical image to a diagnosis and treatment terminal.

[0101] On this basis, the step of generating a correction model selection strategy and an image correction content selection strategy of the three-dimensional image correction server in the target period according to the solving result of the optimization algorithm specifically includes:

[0102] S431: generating a correction model selection strategy of the three-dimensional image correction server in the target period according to the target correction model set;

[0103] S432: generating an image correction content selection strategy based on a candidate correction subject of a target correction model corresponding to the minimum test output delay of the three-dimensional medical image requested by each user.

[0104] In this embodiment, considering the image concurrent processing resources of the three-dimensional image correction server, a target correction model set is selected from the target correction model library and deployed in the three-dimensional image correction server to make the three-dimensional image correction server meet the video memory capacity limit, the deployed model meet the candidate correction subject requirements of the user request, and the diagnosis and treatment experience in the target period be optimal, a correction model selection strategy and an image correction content selection strategy are generated, and then the process of correction model deployment and image correction, reconstruction and output in the three-dimensional image correction server is controlled. Therefore, the present application realizes the automatic and visual correction reconstruction of three-dimensional medical images based on deep learning in a clinical diagnosis and treatment scene, alleviates the hardware resource limitation of unbalanced memory occupation and processing efficiency when facing a large number of three-dimensional medical image correction requirements, and improves the correction efficiency of three-dimensional medical image correction and the clinical diagnosis and treatment experience.

[0105] In the preferred embodiment, the step of controlling a target correction model deployed in the three-dimensional image correction server to correct and reconstruct a target three-dimensional medical image input by each user in the target period according to the correction model selection strategy and the image correction content selection strategy specifically includes:

[0106] S51: According to the correction model selection strategy, a set of target correction models is deployed on the three-dimensional image correction server. The target three-dimensional medical image requested by each user is input into the three-dimensional image correction server. According to the image correction content selection strategy, the corresponding target correction model is driven to perform three-dimensional image correction on the image correction content in the area to be repaired in the target three-dimensional medical image to obtain the corrected three-dimensional medical image.

[0107] S52: Perform visualization rendering on the corrected three-dimensional medical image to complete the correction and reconstruction of the original three-dimensional medical image, and send the reconstructed corrected three-dimensional medical image to the diagnostic terminal.

[0108] like Figures 3-4 As shown, Figure 3 To create a 3D surface map for the repaired image, Figure 4 To repair the image, volume rendering was used. While surface rendering showed the integrity of the image's external morphology, volume rendering further revealed the hierarchy and distribution of the internal tissue structure, providing important reference for medical image analysis and diagnosis. Figures 5-7 As shown, cross-sectional views of the repaired image in axial, coronal, and sagittal planes (i.e., "three-view images") are presented sequentially. It can be observed that the repaired image exhibits good structural continuity and texture consistency in all three directions, demonstrating the system's ability to reconstruct three-dimensional images with consistent texture. These results demonstrate that the present invention, based on modified deep learning technology, achieves high-quality correction and reconstruction of three-dimensional images, and provides a system that automates the process with one click and visualizes the results.

[0109] In this embodiment, the solution-generated correction model selection strategy and image correction content selection strategy are used to control the 3D image correction server to correct and reconstruct the repairable areas of the target 3D medical image input by each user within the target time period. (Since the image defects or blurred areas are random and uncertain, the model deployment and correction time consumption is based on the estimation of full-area correction. In reality, the repairable area is often only a part of the area. Correcting only a part of the image will result in better system status and efficiency.) When the server's video memory allows, the deployment and execution of small-area image correction models are adopted as much as possible to reduce invalid area correction and improve server response and processing speed. This provides accurate basis for disease diagnosis and treatment plan formulation after being pushed to the doctor's diagnosis terminal.

[0110] It is to be understood that the terms "one embodiment", "another embodiment", "other embodiments", "first embodiment", "second embodiment", etc. as may be found in the specification and / or in the claims, indicate that the alternative is included in at least one embodiment. These terms only specify the scope of claimable subject matter; and do not necessarily affect the scope of the application. Further, these terms only indicate particular embodiments of the applications. Other embodiments of the present application can be derived from the description, experimental results and / or the claims, without departing from the scope of the present application.

[0111] It is to be understood that the terms "including", "comprising", "consisting" and "consisting essentially of" or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0112] The above only describes the preferred embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, which is made according to the content of the specification and drawings of the present application, is also included in the patent protection scope of the present application.

Claims

1. A deep learning-based automatic visualization three-dimensional image rectification and reconstruction method, characterized in that, The method comprises the following steps: Preprocessing the original three-dimensional medical image in each user request in a target period to generate a target three-dimensional medical image containing a region to be repaired and a normal region; Based on the type of the diagnosis and treatment department in each user request, determine the image correction content; wherein the image correction content includes a plurality of candidate correction subjects and a correction priority list; Using the plurality of candidate correction subjects of the image correction content, match the plurality of target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library; wherein each target correction model is configured to perform three-dimensional medical image correction on at least one candidate correction subject; According to the correction priority list and the plurality of target correction models corresponding to each user request in the target period, considering the image concurrent processing resources of the three-dimensional image correction server, generate the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period; specifically including: Query the image concurrent processing resources of the three-dimensional image correction server; wherein the image concurrent processing resources include server video memory capacity and test output delay when the image correction model used for image correction of three-dimensional medical images with different candidate correction subjects performs image correction test; According to the plurality of target correction models corresponding to each user request, construct a target correction model library, and use an optimization algorithm to solve the target correction model set selected from the target correction model library and deployed in the three-dimensional image correction server, so that the three-dimensional image correction server satisfies the first constraint condition that the sum of the memory occupancy of all deployed target correction models is less than the server video memory capacity, the second constraint condition that the candidate correction subject of the front preset proportion priority in the correction priority list corresponding to each user request is covered in at least one target correction model, and the optimization target that the minimum test output delay cumulative sum of the three-dimensional medical image of each user request in the target period when the target correction model set is used to perform image correction of the candidate correction subject of the front preset proportion priority is the minimum; According to the optimization algorithm solving result, generate the correction model selection strategy and the image correction content selection strategy of the three-dimensional image correction server in the target period; According to the correction model selection strategy and the image correction content selection strategy, control the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period, and send the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal; According to the correction model selection strategy and the image correction content selection strategy, control the target correction model deployed in the three-dimensional image correction server to correct and reconstruct the target three-dimensional medical image input by each user request in the target period, and send the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal.

2. The deep learning based automatic visualization three-dimensional image rectification and reconstruction method of claim 1, wherein, The preprocessing step of the original three-dimensional medical image in each user request to generate a target three-dimensional medical image containing a region to be repaired and a normal region, specifically includes: Performing a first preprocessing action including resampling, direction alignment, intensity normalization and outlier removal on the original three-dimensional medical image in each user request to obtain a preliminary preprocessed image; Performing a second preprocessing action including mask extraction and repair region division on the preliminary preprocessed image to generate a target three-dimensional medical image containing a repair region and a normal region. 3.The deep learning based automatic visualization 3D image rectification and reconstruction method of claim 2, wherein, The preliminary preprocessed image step specifically includes: Resampling, configured to resample the original three-dimensional medical image in each user request to a uniform voxel spacing for different resolutions or anisotropy; Direction alignment, configured to align the image coordinate system direction of the original three-dimensional medical image in each user request by the SetDirection method; Intensity normalization, configured to eliminate device-to-device grayscale differences by adjusting the window width and window level for the original three-dimensional medical image in each user request; Outlier removal, configured to remove extreme values caused by air or metal contrast by threshold method for the original three-dimensional medical image in each user request. 4.The deep learning based automatic visualization 3D image rectification and reconstruction method of claim 2, wherein, The second preprocessing action of the preliminary preprocessed image specifically includes: Determine the gray scale segmentation threshold by Otsu adaptive threshold segmentation algorithm to extract the tissue region; perform BinaryFillhole operation on the initial binary mask to fill the holes; Extract the maximum connected domain by ConnectedComponent analysis to exclude isolated blocks and noise interference; apply closing operation for morphological smoothing processing on the mask edge to obtain the final mask; On the basis of the final mask, identify the low-intensity, structure fracture or shedding area from the original image by Subtract operation and divide it into the repair region.

5. The deep learning based automatic visualization three-dimensional image rectification and reconstruction method of claim 1, wherein, The image correction content determination step based on the type of each user request specifically includes: Extract the diagnosis and treatment department category in each user request, query the diagnosis and treatment disease list corresponding to the diagnosis and treatment department category, and analyze the associated organ tissue structure of each diagnosis and treatment disease in the diagnosis and treatment disease list; Summarize the associated organ tissue structure of each diagnosis and treatment disease, sort each associated organ tissue structure in each diagnosis and treatment department category according to the preset importance, and generate a list of several candidate correction subjects and correction priorities in the image correction content of each user request.

6. The deep learning based automatic visualization three-dimensional image rectification and reconstruction method of claim 1, wherein, The several candidate correction subjects of the image correction content are matched with several target correction models based on deep learning corresponding to the user request in the three-dimensional medical image correction model library step, specifically including: Access the three-dimensional medical image correction model library; wherein the three-dimensional medical image correction model library stores several candidate correction models, and each candidate correction model is configured to perform three-dimensional medical image correction on different numbers of candidate correction subjects; extracting a plurality of candidate correction subjects in the image correction content, and querying, according to the plurality of candidate correction subjects, a three-dimensional medical image correction model library to obtain a plurality of candidate correction models containing all the candidate correction subjects as a plurality of target correction models corresponding to the user request.

7. The automatic visualization three-dimensional image rectification and reconstruction method based on deep learning according to claim 6, wherein, The construction of the plurality of candidate correction models in the three-dimensional medical image correction model library specifically includes: collecting three-dimensional medical images with different numbers of candidate correction subjects, randomly generating a plurality of cuboid occlusion regions in each three-dimensional medical image, and generating pseudo-defect input images as three-dimensional medical image test samples of the candidate correction models; constructing a UNet3D network structure in the form of an encoder-decoder pair, taking the original three-dimensional medical image as a learning target, and optimizing the reconstruction error between the output of the UNet3D network structure and the original three-dimensional medical image through an L1 loss function to learn the context structure completion capability; using a self-supervised learning strategy to train the model and obtain candidate correction models for three-dimensional medical images with different numbers of candidate correction subjects.

8. The deep learning based automatic visualization three-dimensional image rectification and reconstruction method of claim 1, wherein, According to the optimization algorithm solution result, the three-dimensional image correction server generates a correction model selection strategy and an image correction content selection strategy in the target period, specifically including: According to the target correction model set, the three-dimensional image correction server generates a correction model selection strategy in the target period; based on the candidate correction subjects of the target correction model corresponding to the minimum test output delay of each user request three-dimensional medical image, an image correction content selection strategy is generated.

9. The deep learning based automatic visualization three-dimensional image rectification and reconstruction method of claim 1, wherein, According to the correction model selection strategy and the image correction content selection strategy, the target correction model deployed in the three-dimensional image correction server corrects and reconstructs the target three-dimensional medical image input by each user request in the target period, and sends the reconstructed correction three-dimensional medical image to the diagnosis and treatment terminal. Specifically includes: According to the correction model selection strategy, the three-dimensional image correction server deploys a target correction model set, inputs the target three-dimensional medical image of each user request into the three-dimensional image correction server, and drives the corresponding target correction model according to the image correction content selection strategy to perform three-dimensional image correction on the image correction content in the target three-dimensional medical image. The repair area obtains a corrected three-dimensional medical image; performing visual rendering on the corrected three-dimensional medical image, completing the correction and reconstruction of the original three-dimensional medical image, and sending the reconstructed corrected three-dimensional medical image to the diagnosis and treatment terminal.

Images