Lightweight 3D medical image real-time reasoning method and system based on edge computing

Abstract

The present invention relates to the field of image data processing technology, and specifically discloses a lightweight 3D medical image real-time reasoning method and system based on edge computing, including the construction of a C++ framework model, anisotropy detection and processing, separation of low-resolution axis resampling, 3D resampling, determining the sliding window step size and generating a weight map, using a lightweight student model for prediction, sliding window reasoning, image post-processing, performing size adjustment, voxel connectivity and integrity filling on the predicted segmentation map obtained by reasoning, and outputting the final global segmentation map. The present invention uses a lightweight student model to replace the traditional nnUNet, which requires 5 model reasonings, reducing the number of reasonings by 80% and significantly shortening the single reasoning time; at the same time, C++ language reconstruction is used to achieve CUDA parallelization, and through GPU acceleration of data block extraction, mirror transformation and weighted aggregation operators, the performance of the entire process is improved by more than 10 times compared with Python implementation.

Description

Technical Field

[0001] The present invention relates to the field of image data processing technology, in particular to the field of real-time reasoning processing technology such as 3D medical image segmentation and labeling, and specifically to a lightweight 3D medical image real-time reasoning method and system based on edge computing. Background Art

[0002] The core of current medical image segmentation technology has shifted from traditional methods to deep learning-driven approaches. While traditional techniques such as threshold segmentation, region growing, and atlas-based registration remain practical in specific scenarios (such as high-contrast organ segmentation), they generally suffer from complex artificial feature design and poor generalization capabilities. In recent years, encoder-decoder architectures, exemplified by U-Net, have become mainstream, achieving breakthroughs in tumor and organ segmentation tasks by integrating multi-scale features through skip connections. Improved versions such as the Attention U-Net with an attention mechanism, the three-dimensionally expanded V-Net, and the Transformer-based TransUNet have significantly improved the localization accuracy of complex lesions.

[0003] Technological developments are showing three major trends: First, multimodal fusion technology integrates data from multiple sources, such as CT, MRI, and PET, leveraging cross-modal feature complementarity to improve segmentation robustness. Second, self-supervised / weakly supervised learning strategies, such as contrastive learning and label propagation, effectively alleviate the pain point of scarce annotated medical imaging data. Furthermore, real-time interactive segmentation systems are gradually being integrated into clinical workflows. Current challenges focus on generalization capabilities from small samples, model interpretability verification, and domain adaptation for different devices and protocols. Federated learning, domain adaptation, and explainable AI are key breakthrough areas.

[0004] Currently, the most widely used framework for medical image reasoning is the general-purpose medical processing framework nnUNet. As a commonly used framework for medical processing, nnUNet has been widely recognized within the industry for its flexible and simple API. However, in terms of engineering and efficiency, native nnUNet is not developer-friendly due to limitations in its overall architecture and Python API, sometimes sacrificing time for accuracy. Its main limitation is the high GPU memory requirements for 3D model training, which significantly limits its application to large-scale data. Furthermore, it requires additional fine-tuning for extreme data distributions. Summary of the Invention

[0005] In order to solve the problems of the existing nnUNet framework model, such as large model size, high requirements for GPU computing, low upper limit for large-scale data processing, and low processing efficiency, this application has specifically developed a set of lightweight 3D medical image real-time inference methods and systems based on edge computing to replace the existing nnUNet framework model. While basically retaining the accuracy of the existing nnUNet framework model, the size of the model is reduced from the existing 100MB-200MB to 1MB-3MB, and the volume is basically reduced to about 1%-2% of the existing nnUNet framework model, reducing the model occupancy by more than 98%. The existing nnUNet framework model is subject to the limitation of the original 5-fold model to infer separately and then make aggregated predictions at the post-processing level, resulting in 5 times of hardware time loss in order to obtain higher prediction accuracy, which poses great difficulties for medical-related real-time auxiliary tasks.

[0006] This application is based on the general framework of nnUNet, combined with nnUNet's own rich training framework and strategy to realize a new nnUNet knowledge distillation technology based on a layer-by-layer progressive feature map. This enables the present invention to seamlessly connect the nnUNet framework, which makes the operation very easy. After testing, the student model after distillation and the prediction results of the original nnUNet's 5 models generally have a Dice coefficient loss of less than 5%. If some hyperparameters of the distillation architecture are changed, the accuracy of the model is even better than the effect of the original 5 models predicting and re-aggregating separately. At the same time, this is also the fundamental reason why the model volume of the present invention is much smaller than the existing nnUNet framework model.

[0007] In order to achieve the above objectives, the technical solutions adopted in this application are:

[0008] The lightweight 3D medical image real-time reasoning method based on edge computing provided by the present invention includes the following steps:

[0009] Step STP100: Build a C++ framework model. Build a 5-fold teacher model and a lightweight student model based on the nnUNet model. Use the 5-fold teacher model to predict sample image data. Use the simultaneously output Logits-P and feature maps to provide global guidance to the lightweight student model. This process is completed when the lightweight student model's prediction accuracy reaches the preset value of the teacher model's prediction accuracy.

[0010] Step STP200, anisotropy detection and processing, anisotropy detection is performed on the input 3D medical image, by determining the ratio of the extreme value of the resolution of any axis c Whether the difference meets the threshold γ0, when the ratio c > γ0, then proceed to step STP300; when the ratio cWhen ≤γ0, the resampling strategy of step STP400 is performed; wherein, the ratio c It is the ratio of the maximum value to the minimum value of the image voxel on the corresponding axis;

[0011] Step STP300 , separating the low-resolution axis resampling, checking whether the original 3D medical image shape and the target 3D medical image shape are consistent. If they are consistent, executing step STP500 . If they are inconsistent, performing independent interpolation 2D resampling of the 2D plane perpendicular to the low-resolution axis using OpenCV, and then performing 1D resampling of the 2D resampling result in the low-resolution axis direction, completing 2D+1D resampling.

[0012] Step STP400, 3D resampling, first use OpenCV functions to perform 2D resampling on each XY plane perpendicular to the low-resolution axis, then collect low-resolution axis pixels at the same (y, x) position to form 1D lines, and then use OpenCV's resize function to resample these 1D lines in the low-resolution axis direction;

[0013] Step STP500: Determine the sliding window step size and generate a weight map. Use the window step size calculator to automatically determine the overlap step size based on the input volume size and block size, and use the Gaussian weighted aggregator to generate a weight map using the pre-calculated 3D Gaussian kernel for weight distribution. ;

[0014] Step STP600, using a lightweight student model to predict the probability distribution Logits-C of each voxel point in the 3D medical image;

[0015] Step STP700, sliding window reasoning, uses the CUDA operator module to perform block extraction, overlapping sliding, enhancement, aggregation and normalization on the input 3D medical image to obtain a predicted global predicted segmentation atlas;

[0016] Step STP800, image post-processing, performs size adjustment, voxel connectivity and integrity filling on the predicted segmentation map obtained by reasoning in step STP700, and outputs the final global segmentation map.

[0017] Preferably, the step of performing block extraction overlapping sliding on the input 3D medical image by the CUDA operator module in step STP700 includes step STP710: converting the volume of the large-size 3D medical image into Decomposed into a series of fixed-size window blocks by CUDA kernel function , the possibly overlapping image blocks are recorded as , slide the window in the form of a matrix in the Z, Y, and X dimensions on the GPU with a predefined step size according to the CUDA stream to generate the starting coordinates of each block ( d,h,w ), expressed as:

[0018]

[0019] in, , , ,and , An image block representing the size of the sliding window, It is the stream form of CUDA kernel.

[0020] Further preferably, the step of performing enhancement processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP720: performing enhancement processing on each extracted image block Apply a set of predefined spatial transformations after CUDA kernel processing on the GPU, perform inference on each transformed block according to the CUDA stream, and then perform the corresponding inverse transformation on the CUDA kernel on the obtained prediction results. Finally, all the transformed versions of the prediction results are averaged on the GPU, which can be expressed as:

[0021]

[0022] in, is an image block The average predicted Logits after TTA, Represents the depth, height and width dimension indexes respectively, with sizes of , is the number of categories, It is the stream form of CUDA kernel; represents a set of predefined spatial transformations, Representative Set A specific transform index in ; Representative A spatial transformation function, It will The transformed prediction result is restored to the original block direction, that is, the The inverse function of a transformation; Representing the inference function, the spatial transformation includes a combination of mirror flipping and / or non-flipping along the Z, Y, and X axes.

[0023] Further preferably, the step of performing aggregation processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP730: specifically using a Gaussian weighted aggregation method, specifically including introducing a pre-calculated Gaussian weight map with the same size as the block in the GPU cache area , and then each image block The prediction results after step STP720 average When added to the global prediction map, it is weighted element by element using the CUDA Gaussian kernel function on the GPU to obtain the global cumulative probability distribution, which is specifically expressed as:

[0024]

[0025]

[0026] in, is a Gaussian weighted graph, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel, It is the GPU cache of CUDA kernel functions.

[0027] Further preferably, the step of normalizing the input 3D medical image by the CUDA operator module in step STP700 includes step STP740: normalizing all image blocks After the weighted predictions are accumulated in the global GPU cache, the final prediction of each voxel is obtained by dividing the weighted Logits accumulated for each voxel by the total weight accumulated for each voxel. , specifically expressed as:

[0028]

[0029] in, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel.

[0030] The present invention also provides a lightweight 3D medical imaging real-time reasoning system based on edge computing, which is used to execute the real-time reasoning method provided above, including a C++-based feature distillation module and an inference engine module. The feature distillation module includes distilling the 5-fold teacher model set of the original nnUNet into a single lightweight student model. The student model has a volume of 1MB-3MB and an accuracy loss of no more than 5%; the inference engine module is used to realize full-process acceleration from model conversion, preprocessing, sliding window reasoning to post-processing, and supports CUDA parallel computing; the sliding window optimization module adopts GPU-driven overlapping window reasoning technology to achieve smooth output of prediction results through Gaussian weighted aggregation; the edge deployment interface module adapts to the personalized deployment requirements of different hardware devices.

[0031] Preferably, the feature distillation module also includes a progressive knowledge transfer unit for optimizing student model parameters through layer-by-layer feature map comparison and global guidance during the student model training stage. The progressive knowledge transfer unit includes an adaptive parameter module for dynamically adjusting the knowledge transfer path and weights, an intermediate semantic expression feature map distillation module for aligning the teacher model and the student model, and a self-supervised comparison module for learning enhanced representations through inter-sample relationship learning.

[0032] Preferably, the inference engine module includes an anisotropy processor for detecting the physical spacing ratio of the medical image and starting the separate Z-axis processing when the following formula is satisfied:

[0033]

[0034] in, Represents the ratio of the extreme values of resolution of any axis, Resolution extreme value ratio threshold, is the maximum resolution, is the minimum resolution;

[0035] The sliding window optimization module includes a window step size calculator that automatically determines the overlap step size based on the input volume size and block size:

[0036]

[0037] in, is the image block size, =0.5, indicating 50% overlap;

[0038] CUDA stream controller, which manages asynchronous pipeline execution of data block fetch, image enhancement, inference, and aggregation;

[0039] Gaussian weighted aggregator, using precomputed 3D Gaussian kernel for weight distribution:

[0040]

[0041]

[0042] in, is a Gaussian weighted graph, are the coordinates within the block, are the block center coordinates, It is a block in dimension i The size, standard deviation Set proportional to the block size, =0.5.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above-mentioned real-time reasoning method when executed by a processor.

[0044] The present invention also provides a medical image processing device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the steps of the above-mentioned real-time reasoning method when executing the program.

[0045] Beneficial effects:

[0046] 1. This invention uses a single lightweight student model to replace the traditional nnUNet, which requires 5 model inferences. The number of inferences is reduced by 80%, and the single inference time is significantly shortened. At the same time, it uses C++ language reconstruction to achieve CUDA parallelization. Through GPU acceleration of data block extraction, mirror transformation and weighted aggregation operators, the performance of the entire process is improved by more than 10 times compared with Python implementation.

[0047] 2. The present invention adopts lightweight models and adapts to edge device deployment, overcoming the difficulties of existing large-volume models of hundreds of MB in deploying on edge devices with limited video memory. The present invention compresses the model size to 1-3MB through feature map migration and channel pruning. At the same time, it removes Python dependencies and reduces video memory usage by 70%, making it deployable on embedded GPUs or medical terminal devices.

[0048] 3. The present invention dynamically calibrates the intermediate layer features of the student model through the teacher model, retains the dark knowledge, controls the Dice coefficient loss to ≤5%, and the accuracy loss is controllable; furthermore, the prediction results of the overlapping area of the sliding window are smoothed and weighted to avoid the accumulation of edge prediction errors, improve the continuity of the segmentation boundary, and make the segmentation results reliable.

[0049] 4. Compared with the existing nnUNet framework model, which easily leads to the problem of low-resolution axis information loss or artifacts due to direct 3D resampling, the present invention dynamically selects a 2D+1D resampling strategy based on the physical spacing ratio to retain the details of each axis; after interpolation, it is cropped to the original data range to avoid interference from outliers.

[0050] 5. This invention is refactored using a new C++ language and can be compiled for ARM / x86 architectures, adapting to full-scenario deployments from cloud servers to portable ultrasound devices. It automatically adjusts the sliding window batch size based on the device's computing power, maintaining real-time performance even on low-end devices. A modular CUDA operator library supports user-defined processes.

[0051] 6. The present invention can be expanded to 4D data processing in 3D space + time dimensions, supporting dynamic organ motion analysis. The preprocessing module automatically adapts to the physical characteristics of different modal images such as CT, MRI, and PET, specifically solving the problem of insufficient processing capabilities of existing general frameworks for 4D images, such as dynamic cardiac MRI. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative labor.

[0053] Figure 1 It is a schematic block diagram of system data flow during the system training and use phases of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0055] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without making any creative efforts shall fall within the scope of protection of the present application.

[0056] Example 1:

[0057] The lightweight 3D medical image real-time reasoning method based on edge computing provided by the present invention includes the following steps:

[0058] Step STP100: Build a C++ framework model. Build a 5-fold teacher model and a lightweight student model based on the nnUNet model. Use the 5-fold teacher model to predict sample image data. Use the simultaneously output Logits-P and feature maps to provide global guidance to the lightweight student model. This process is completed when the lightweight student model's prediction accuracy reaches the preset value of the teacher model's prediction accuracy.

[0059] Step STP200, anisotropy detection and processing, anisotropy detection is performed on the input 3D medical image, by determining the ratio of the extreme value of the resolution of any axis c Whether the difference meets the threshold γ0, when the ratio c >γ0, then proceed to step STP300; when the ratio c When ≤γ0, the resampling strategy of step STP400 is performed; wherein, the ratio c It is the ratio of the maximum value to the minimum value of the image voxel on the corresponding axis;

[0060] Step STP300 , separating the low-resolution axis resampling, checking whether the original 3D medical image shape and the target 3D medical image shape are consistent. If they are consistent, executing step STP500 . If they are inconsistent, performing independent interpolation 2D resampling of the 2D plane perpendicular to the low-resolution axis using OpenCV, and then performing 1D resampling of the 2D resampling result in the low-resolution axis direction, completing 2D+1D resampling.

[0061] Step STP400, 3D resampling, first use OpenCV functions to perform 2D resampling on each XY plane perpendicular to the low-resolution axis, then collect low-resolution axis pixels at the same (y, x) position to form 1D lines, and then use OpenCV's resize function to resample these 1D lines in the low-resolution axis direction;

[0062] Step STP500: Determine the sliding window step size and generate a weight map. Use the window step size calculator to automatically determine the overlap step size based on the input volume size and block size, and use the Gaussian weighted aggregator to generate a weight map using the pre-calculated 3D Gaussian kernel for weight distribution. ;

[0063] Step STP600, using a lightweight student model to predict the probability distribution Logits-C of each voxel point in the 3D medical image;

[0064] Step STP700, sliding window reasoning, uses the CUDA operator module to perform block extraction, overlapping sliding, enhancement, aggregation and normalization on the input 3D medical image to obtain a predicted global predicted segmentation atlas;

[0065] Step STP800, image post-processing, performs size adjustment, voxel connectivity and integrity filling on the predicted segmentation map obtained by reasoning in step STP700, and outputs the final global segmentation map.

[0066] Example 2:

[0067] This embodiment is based on the embodiment 1, and further optimizes step STP700. Specifically, the step of performing block extraction and overlapping sliding on the input 3D medical image by the CUDA operator module in this embodiment includes step STP710: converting the volume of the large-size 3D medical image into Decomposed into a series of fixed-size window blocks by CUDA kernel function , the possibly overlapping image blocks are recorded as , slide the window in the form of a matrix in the Z, Y, and X dimensions on the GPU with a predefined step size according to the CUDA stream to generate the starting coordinates of each block ( d,h,w ), expressed as:

[0068]

[0069] in, , , ,and , An image block representing the size of the sliding window, It is the stream form of CUDA kernel.

[0070] In this embodiment, the step of performing enhancement processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP720: performing enhancement processing on each extracted image block. Apply a set of predefined spatial transformations after CUDA kernel processing on the GPU, perform inference on each transformed block according to the CUDA stream, and then perform the corresponding inverse transformation on the CUDA kernel on the obtained prediction results. Finally, all the transformed versions of the prediction results are averaged on the GPU, which can be expressed as:

[0071]

[0072] in, is an image block The average predicted Logits after TTA, Represents the depth, height and width dimension indexes respectively, with sizes of , is the number of categories, It is the stream form of CUDA kernel; represents a set of predefined spatial transformations, Representative Set A specific transform index in ; Representative A spatial transformation function, It will The transformed prediction result is restored to the original block direction, that is, the The inverse function of a transformation; Representing the inference function, the spatial transformation includes a combination of mirror flipping and / or non-flipping along the Z, Y, and X axes.

[0073] In this embodiment, the step of performing aggregation processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP730: specifically using a Gaussian weighted aggregation method, specifically including introducing a pre-calculated Gaussian weight map of the same size as the block into the GPU cache area. , and then each image block The prediction results after step STP720 average When added to the global prediction map, it is weighted element by element using the CUDA Gaussian kernel function on the GPU to obtain the global cumulative probability distribution, which is specifically expressed as:

[0074]

[0075]

[0076] in, is a Gaussian weighted graph, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel, It is the GPU cache of CUDA kernel functions.

[0077] In this embodiment, the step of normalizing the input 3D medical image by the CUDA operator module in step STP700 includes step STP740: After the weighted predictions are accumulated in the global GPU cache, the final prediction of each voxel is obtained by dividing the weighted Logits accumulated for each voxel by the total weight accumulated for each voxel. , specifically expressed as:

[0078]

[0079] in, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel.

[0080] Example 3:

[0081] The present embodiment provides a lightweight 3D medical image real-time reasoning system based on edge computing, which is used to execute the real-time reasoning method provided by any of the above embodiments, including a feature distillation module and an inference engine module based on C++. The feature distillation module includes distilling the 5-fold teacher model set of the original nnUNet into a single lightweight student model. The student model has a volume of 1MB-3MB and an accuracy loss of no more than 5%; the inference engine module is used to realize full-process acceleration from model conversion, preprocessing, sliding window reasoning to post-processing, and supports CUDA parallel computing; the sliding window optimization module adopts GPU-driven overlapping window reasoning technology to achieve smooth output of prediction results through Gaussian weighted aggregation; the edge deployment interface module adapts to the personalized deployment requirements of different hardware devices. See the system framework principle for details. Figure 1 As shown in the figure, the native nnUNet includes a pre-processing module-P, an empirical parameter module, a cross-validation module, a fixed parameter module, and a post-processing module-P, which is then output from the native nnUNet framework to a 5-fold teacher model.

[0082] In this embodiment, the feature distillation module also includes a progressive knowledge transfer unit for optimizing student model parameters through layer-by-layer feature map comparison and global guidance during the student model training stage. The progressive knowledge transfer unit includes an adaptive parameter module for dynamically adjusting the knowledge transfer path and weights, an intermediate semantic expression feature map distillation module for aligning the teacher model and the student model, and a self-supervised comparison module for learning enhanced representation through inter-sample relationship learning.

[0083] In this embodiment, the inference engine module includes an anisotropic processor for detecting the physical spacing ratio of the medical image and starting the separate Z-axis processing when the following equation is satisfied:

[0084]

[0085] in, Represents the ratio of the extreme values of resolution of any axis, Resolution extreme value ratio threshold, is the maximum resolution, is the minimum resolution;

[0086] The sliding window optimization module includes a window step size calculator that automatically determines the overlap step size based on the input volume size and block size:

[0087]

[0088] in, is the image block size, =0.5, indicating 50% overlap;

[0089] CUDA stream controller, which manages asynchronous pipeline execution of data block fetch, image enhancement, inference, and aggregation;

[0090] Gaussian weighted aggregator, using precomputed 3D Gaussian kernel for weight distribution:

[0091]

[0092]

[0093] in, is a Gaussian weighted graph, are the coordinates within the block, are the block center coordinates, It is a block in dimension i The size, standard deviation Set proportional to the block size, =0.5.

[0094] Regarding the implementation of CUDA operators in the C++ framework, the following description is given in this embodiment:

[0095] First, we leverage the parallel computing capabilities of graphics processing units (GPUs) to efficiently implement CUDA operators for key computational steps in the sliding window inference process. These operators are designed to accelerate intensive prediction tasks on large-scale volumetric data, such as 3D medical images. By offloading computationally intensive operations from the CPU to the GPU and leveraging the CUDA architecture for massive parallelization, they significantly improve processing speed and efficiency. These operators cover the entire process, from data block extraction, spatial transformation, weighted aggregation, to final result integration, and offer a variety of optimized implementation options, including single-block, streaming asynchronous, and batch processing.

[0096] Data Block Extraction Operator: Efficiently copies a rectangular sub-region (Patch) of specified starting coordinates and size from a large 3D input image tensor stored in GPU memory as input for subsequent model inference.

[0097] Using CUDA threads, each thread is responsible for calculating and copying one or more voxel values in the target patch buffer. For each thread, a one-dimensional index in the target patch buffer is calculated based on its unique thread ID. The coordinates are decomposed into three-dimensional coordinates within the patch. Combined with the starting coordinates (d, h, w) of the patch in the original image, calculate the three-dimensional coordinates of the voxel in the source image buffer, convert the three-dimensional coordinates of the source image into a one-dimensional index of the source image buffer, and then calculate the voxel in the source image buffer. Read the value and write to middle.

[0098]

[0099] The kernel is designed to process multiple patches in a single batch. The thread index is further broken down to include the index of the item within the batch. The starting coordinates (d, h, w) are obtained from the coordinate array in the GPU memory. Each batch item corresponds to a different coordinate. The kernel handles potential padding and controls the effective range by the current batch size when the batch size is not divisible by the actual workload.

[0100] Mirror transformation operator: performs a mirror operation on the data block in the GPU memory along one or more specified coordinate axes, depth Z=0, height Y=1, width X=2, as part of test-time data augmentation TTA or part of the result inverse transformation.

[0101] Using a massively parallel strategy, each thread is responsible for processing an element in the source data block and writing it to the mirrored location in the target data block. The thread calculates the one-dimensional index of the source element in the data block and decomposes it into three-dimensional coordinates. For mirror output pointers, also contains the category index Calculate the target 3D coordinates according to the axis to be mirrored For example, if axis=0, then , other coordinates remain unchanged. Read the value and write it to .

[0102] For host-side logic, since mirroring operations are sequence-dependent, the host-side wrapper functions mirror*Cuda and mirrorBatch*Cuda typically employ a "ping-pong buffer" strategy, utilizing a temporary GPU buffer to sequentially apply the mirror kernel function to each axis to ensure correct multi-axis mirroring. Simultaneously, the kernel processes batches of data, performing mirroring operations independently on each patch within the batch.

[0103] Weighted Accumulator: Multiplies the Logits of a single or batch of predicted patches after inference by the corresponding pre-computed Gaussian weight map, and atomically accumulates the weighted results into the global (covering the entire input volume size) predicted Logits buffer and weight accumulation buffer.

[0104] Start threads covering all elements of the prediction patch, including the category dimension. Each thread calculates the category it is responsible for and the spatial position pos within the Patch (decomposed into ). Combined with the starting coordinates d, h, w of the Patch, calculate the coordinates of the voxel in the global volume . Read the predicted value P=pred[thread globalidx ] and the corresponding Gaussian weights , calculate the weighted value .

[0105] Calculate the voxel predicted in the global Logits buffer logits and the global weight buffer n predictions One-dimensional index in global logitsidx and globa lspatialidx .

[0106] Will Safely add to predicted logits [global logitsidx ], and Safely accumulate to n predictions [global spatialidx ]. atomicAdd ensures that even if multiple threads, i.e. threads processing different patches of overlapping areas, try to update the same memory location at the same time, the result is correct and serialized access. Usually, only c=0 thread accumulates once to avoid redundancy. The kernel processes batch data, and the thread calculates batch idx and read from the GPU coordinate array (d,h, w) . Optimize memory access patterns.

[0107] Fusion operator: Reduces CUDA kernel startup overhead and GPU memory bandwidth usage by merging / fusing multiple logically consecutive steps in the sliding window process into a single kernel function for execution.

[0108] Data extraction and mirroring are performed within a single kernel function. A thread reads the source image data, directly calculates the coordinates within the mirrored target patch, and writes the data to the final location in the target patch buffer. Reverse mirroring and weighted accumulation of the model output are performed within a single kernel function. The thread reads the unreverse mirrored model output value, calculates the global target coordinate after reverse mirroring, reads the corresponding Gaussian weight, calculates the weighted value, and then atomically accumulates it to the target location in the global buffer. This reduces the number of intermediate data reads and writes, reduces kernel startup latency, and potentially improves data locality, thereby improving overall performance.

[0109] Final aggregation operator: After the weighted predictions of all patches are accumulated, the accumulated Logits are divided by the accumulated weights for final normalization to obtain the average predicted Logits of each voxel, and the layout of the data in the video memory may be adjusted.

[0110] Start threads covering the entire volume space dimension. Each thread is responsible for a spatial position. Read the accumulated weight W=n predictions [spatial idx ].if is greater than a small threshold to avoid division by zero, then calculate Then, for all categories , read the cumulative LogitL = predicted logits [c×total voxels +spatial idx ], calculate the average , and write the result back to predicted logits Buffer, to achieve in-place modification.

[0111] If the final output memory layout needs to be changed, for example from C, D, H, W to D, H, W, C, a parallel copy kernel is started to calculate indices and move data based on the source and target layouts. If the layout remains unchanged, no operation is required.

[0112] Asynchronous stream operators: implement overlapping execution between computing tasks and data transmission or other computing tasks, improving GPU utilization. The *Parallel version of the above operators may also include cudaStream t The parameter version allows kernel function launch and memory operations to be submitted to a specific CUDA stream. Combined with CUDA event cudaEvent t Synchronous control allows you to build complex execution pipelines, for example, executing inference for the current batch on one stream while performing patch extraction and mirroring for the next batch on another stream, thereby hiding some of the latency.

[0113] In general, the comparison of the effects of this embodiment and the existing native nnUNet is shown in Table 1 below:

[0114]

[0115] Table 1 Comparison of the effects of this embodiment and the native nnUNet framework model

[0116] Example 4:

[0117] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the real-time reasoning method described in Examples 1 and 2 above are implemented.

[0118] Example 5:

[0119] The present invention also provides a medical image processing device, including a memory, a processor and a computer program stored in the memory. When the processor executes the program, it implements the lightweight 3D medical image real-time inference method based on edge computing described in the above-mentioned embodiments 1-2.

[0120] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A lightweight 3D medical image real-time inference method based on edge computing, characterized by: The following steps are involved: Step STP100: Build a C++ framework model. Build a 5-fold teacher model and a lightweight student model based on the nnUNet model. Use the 5-fold teacher model to predict sample image data. Use the simultaneously output Logits-P and feature maps to provide global guidance to the lightweight student model. This process is completed when the lightweight student model's prediction accuracy reaches the preset value of the teacher model's prediction accuracy. Step STP200, anisotropy detection and processing, anisotropy detection is performed on the input 3D medical image, by determining the ratio of the extreme value of the resolution of any axis γ Does the difference meet the threshold? , when the ratio γ > When the ratio γ ≤ When , the resampling strategy of step STP400 is performed; wherein, the ratio γ It is the ratio of the maximum value to the minimum value of the image voxel on the corresponding axis; Step STP300 , separating the low-resolution axis resampling, checking whether the original 3D medical image shape and the target 3D medical image shape are consistent. If they are consistent, executing step STP500 . If they are inconsistent, performing independent interpolation 2D resampling of the 2D plane perpendicular to the low-resolution axis using OpenCV, and then performing 1D resampling of the 2D resampling result in the low-resolution axis direction, completing 2D+1D resampling. Step STP400, 3D resampling, first use OpenCV functions to perform 2D resampling on each XY plane perpendicular to the low-resolution axis, then collect low-resolution axis pixels at the same (y, x) position to form 1D lines, and then use OpenCV's resize function to resample these 1D lines in the low-resolution axis direction; Step STP500: Determine the sliding window step size and generate a weight map. Use the window step size calculator to automatically determine the overlap step size based on the input volume size and block size, and use the Gaussian weighted aggregator to generate a weight map using the pre-calculated 3D Gaussian kernel for weight distribution. ; Step STP600, using a lightweight student model to predict the probability distribution Logits-C of each voxel point in the 3D medical image; Step STP700, sliding window reasoning, uses the CUDA operator module to perform block extraction, overlapping sliding, enhancement, aggregation and normalization on the input 3D medical image to obtain a predicted global predicted segmentation atlas; Step STP800, image post-processing, performs size adjustment, voxel connectivity and integrity filling on the predicted segmentation map obtained by reasoning in step STP700, and outputs the final global segmentation map.

2. The lightweight 3D medical image real-time reasoning method based on edge computing according to claim 1 is characterized in that: The step of performing block extraction and overlapping sliding of the input 3D medical image by the CUDA operator module in step STP700 includes step STP710: converting the volume of the large-size 3D medical image into Decomposed into a series of fixed-size window blocks by CUDA kernel function , the possible overlapping image blocks are recorded as , slide the window in the form of a matrix in the Z, Y, and X dimensions on the GPU with a predefined step size according to the CUDA stream to generate the starting coordinates of each block ( d,h,w ), expressed as: in, , , ,and , An image block representing the size of the sliding window, It is the stream form of CUDA kernel.

3. The lightweight 3D medical image real-time reasoning method based on edge computing according to claim 2 is characterized in that: The step of performing enhancement processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP720: performing enhancement processing on each extracted image block Apply a set of predefined spatial transformations after CUDA kernel processing on the GPU, perform inference on each transformed block according to the CUDA stream, and then perform the corresponding inverse transformation on the CUDA kernel on the obtained prediction results. Finally, all the transformed versions of the prediction results are averaged on the GPU, which can be expressed as: in, is an image block The average predicted Logits after TTA, Represents the depth, height and width dimension indexes respectively, with sizes of , is the number of categories, It is the stream form of CUDA kernel; represents a set of predefined spatial transformations, Representative Set A specific transform index in ; Representative A spatial transformation function, It will The transformed prediction result is restored to the original block direction, that is, the The inverse function of a transformation; Representing the inference function, the spatial transformation includes a combination of mirror flipping and / or non-flipping along the Z, Y, and X axes.

4. The lightweight 3D medical image real-time reasoning method based on edge computing according to claim 3 is characterized in that: The step of performing aggregation processing on the input 3D medical image by the CUDA operator module in step STP700 includes step STP730: specifically using a Gaussian weighted aggregation method, specifically including introducing a pre-calculated Gaussian weight map of the same size as the block into the GPU cache area , and then each image block The prediction results after step STP720 average When added to the global prediction map, it is weighted element by element using the CUDA Gaussian kernel function on the GPU to obtain the global cumulative probability distribution, which is specifically expressed as: in, is a Gaussian weighted graph, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel, It is the GPU cache of CUDA kernel functions.

5. The lightweight 3D medical image real-time reasoning method based on edge computing according to claim 3 is characterized in that: The step of normalizing the input 3D medical image by the CUDA operator module in step STP700 includes step STP740: normalizing all image blocks After the weighted predictions are accumulated in the global GPU cache, the final prediction of each voxel is obtained by dividing the weighted Logits accumulated for each voxel by the total weight accumulated for each voxel. , specifically expressed as: in, is the global cumulative probability distribution, is the global cumulative weight, initialized to 0; It is the stream form of CUDA kernel.

6. A lightweight 3D medical image real-time reasoning system based on edge computing, used to execute the method according to any one of claims 1 to 5, characterized in that: It includes a C++-based feature distillation module and an inference engine module. The feature distillation module distills the original nnUNet's 5-fold teacher model set into a single lightweight student model. The lightweight student model has a volume of 1MB-3MB and an accuracy loss of no more than 5%. The inference engine module is used to accelerate the entire process from model conversion, preprocessing, sliding window inference to post-processing, and supports CUDA parallel computing. The sliding window optimization module uses GPU-driven overlapping window inference technology to achieve smooth output of prediction results through Gaussian weighted aggregation. The edge deployment interface module adapts to the personalized deployment requirements of different hardware devices.

7. The lightweight 3D medical image real-time reasoning system based on edge computing according to claim 6, characterized in that: The feature distillation module also includes a progressive knowledge transfer unit for optimizing the parameters of the lightweight student model through layer-by-layer feature map comparison and global guidance during the lightweight student model training phase. The progressive knowledge transfer unit includes an adaptive parameter module for dynamically adjusting the knowledge transfer path and weights, an intermediate semantic expression feature map distillation module for aligning the teacher model and the lightweight student model, and a self-supervised comparison module for learning enhanced representations through inter-sample relationship learning.

8. The lightweight 3D medical image real-time reasoning system based on edge computing according to claim 6, characterized in that: The inference engine module includes an anisotropic processor for detecting the physical spacing ratio of the medical image and starting the separation Z-axis processing when the following formula is satisfied: in, Represents the ratio of the extreme values of resolution of any axis, Resolution extreme value ratio threshold, is the maximum resolution, is the minimum resolution; The sliding window optimization module includes a window step size calculator that automatically determines the overlap step size based on the input volume size and block size: in, is the image block size, =0.5, indicating 50% overlap; CUDA stream controller, which manages asynchronous pipeline execution of data block fetch, image enhancement, inference, and aggregation; Gaussian weighted aggregator, using precomputed 3D Gaussian kernel for weight distribution: in, is a Gaussian weighted graph, are the coordinates within the block, are the block center coordinates, It is a block in dimension i The size, standard deviation Set proportional to the block size, =0.5.

Images