Medical image processing method and system based on visual large model

Abstract

The application relates to the technical field of image processing, in particular to a medical image processing method and system based on a visual large model, which can significantly improve the medical image processing efficiency and accuracy, and is based on a Medficient SAM image segmentation large model; the Medficient SAM model is used for high-precision segmentation of nine different modal medical images including CT, MR, ultrasound and the like; the system realizes automatic lesion region image data quantization analysis, helps doctors to carry out rapid preliminary screening, and improves the work efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a medical image processing method and system based on a visual large model capable of significantly improving the efficiency and accuracy of medical image processing based on a Medficient SAM image segmentation large model. BACKGROUND

[0002] Traditional medical expert models can achieve accurate segmentation of lesion characteristics for a certain disease. In comparison, visual large models have achieved good segmentation effect and generalization ability on natural images and remote sensing images after large-scale data training. Current medical diagnosis strategies require professional physicians to manually determine the location of the lesion, and medical expert models can only segment single modal data, making it difficult to further promote related applications. SUMMARY

[0003] The present application proposes a medical image processing method and system based on a visual large model capable of realizing multiple medical image processing, aiming at the shortcomings and deficiencies in the prior art.

[0004] The present application achieves the following measures:

[0005] A medical image processing method based on a visual large model, characterized by a Medficient SAM image segmentation large model, which is optimized for lightweight to realize the segmentation processing of multiple modal medical images, comprising the following steps:

[0006] Step 1: Convert the trained PyTorch model to ONNX format to allow interoperability between different deep learning frameworks, thereby simplifying the model deployment process;

[0007] Step 2: Perform ONNX model quantization: Before converting the ONNX model to the OM model, use the Ascend Model Compression Tool (AMCT) to quantize the model from FP32 to INT8 format, reducing the model calculation complexity and speeding up the inference speed. During quantization, the data is converted from floating-point numbers to fixed-point numbers, and the process is represented by the following mathematical formula:

[0008]

[0009] Where Δ represents the scaling factor of quantization, min(x) represents the minimum value of the input data, and y is the quantized fixed-point value.

[0010] Step 3: Convert the ONNX model to the OM model: use the Ascend Tensor Compiler tool to convert the ONNX format model to the OM format suitable for Ascend AI chips;

[0011] Step 4: Deploy the OM model on Ascend 310 using ACL technology to realize real-time inference: Step 4-1: Initialize the inference environment, initialize the model inference environment using the Python API interface pyACL of Ascend AI, initialize the environment through the acl.init() function, ensure the correct allocation of device resources, and provide a stable hardware and software foundation for subsequent model loading and inference;

[0012] Step 4-2: Load the OM model: Load the OM model onto the Ascend 310B inference device, load the model into memory by calling acl.mdl.load_from_file(), the loaded model will be inferred by NPU (Neural Processing Unit), when loading the model, the shape and data type of the input / output tensor must be considered and necessary conversion must be performed;

[0013] Step 4-3: Inference execution: First, load the input data (such as medical image data after DICOM file parsing) into memory, call the model for inference through the acl.mdl.execute() interface, during the inference process, NPU uses a highly parallel computing architecture to accelerate convolution layer computation through matrix multiplication:

[0014] Z = X * W + B,

[0015] Where X is the input feature matrix, W is the weight matrix, B is the bias term, and Z is the result of the convolution operation, Step 4-4: Post-processing and result output: After inference is completed, the output result is the segmented medical image, saved in a Numpy array, through post-processing on the output Numpy array, it is converted to DICOM or PNG format for display on the front end.

[0016] The application also provides a medical image processing system based on a visual large model, which is characterized by comprising a PACS (Picture Archiving and Communication System, a system integrating image storage, transmission, management and diagnosis in the medical field) system and a model deployed and accessed to the PACS system to realize convenient interaction of an operator.

[0017] In the application, the inference result is stored as a DICOM or other standard medical image format after processing, for a doctor to view, and the segmented result after inference can be used to label a lesion area and generate an auxiliary diagnosis report, and the output data result is returned to the doctor through the PACS system and visualized through a web front end, helping the doctor to quickly locate a lesion position.

[0018] Compared with the prior art, the application optimizes the lightweight SAM with a large number of parameters, further improves the model inference speed, and provides a good prerequisite for the deployment of the visual large model on a medical device. BRIEF DESCRIPTION OF DRAWINGS

[0019] ATTACHMENT Figure 1 is a flowchart of the application. DETAILED DESCRIPTION

[0020] The application will be further described below in combination with the drawings and examples.

[0021] Example:

[0022] This example aims to provide a medical image data processing method and system based on a visual large model, which uses a Medficient SAM model to realize high-precision segmentation of nine different modal medical images including CT, MR and ultrasound, and the system helps doctors to quickly perform preliminary screening through automatic lesion area image data quantification analysis, thereby improving work efficiency and accuracy.

[0023] The example converts the deep learning model into an OM format file, and performs model quantization and tensor decomposition based on the Ascend Model Compression Tool (AMCT), to finally realize the miniaturization and lightness of the model. By deploying the model on the Huawei Ascend 310B series inference card, with the help of the Da Vinci architecture independently developed by Huawei, the system significantly reduces power consumption while ensuring high throughput and high computing power. Compared with the traditional GPU inference platform, the inference speed is significantly improved, greatly improving the timeliness of doctor-assisted diagnosis. The specific performance data is shown in Table 1:

[0024] Table 1 System performance data

[0025]

[0026] In addition, the system simulates the PACS data system commonly used in hospitals by combining the front-end and back-end architectures with database technology, integrates the Medficient SAM model deployed on the Ascend device into the PACS system, and realizes fully automated, multi-modal, and high-precision medical image-assisted diagnosis.

[0027] The example is implemented by the following steps:

[0028] Step (1): Convert the model to ONNX format: First, convert the trained PyTorch model to ONNX format. ONNX is an intermediate representation for representing deep learning models, which allows interoperability between different deep learning frameworks, simplifying the deployment process of models, and ensuring cross-platform compatibility of AI models. The specific implementation is as follows: torch.onnx.export(model, input_tensor, "model.onnx", export_params=True);

[0029] This process ensures the complete export of model parameters and structure for subsequent deployment;

[0030] Step (2): Quantization of ONNX model: Before converting the ONNX model to the OM model, use the Ascend Model Compression Tool (AMCT) to quantize the model. AMCT uses Post-Training Quantization (PTQ) to quantize the model from FP32 to INT8 format without sacrificing too much accuracy, reducing model computation complexity and speeding up inference. When quantizing, data is converted from floating-point numbers to fixed-point numbers, and the core process can be represented by the following mathematical formula:

[0031]

[0032] Where Δ represents the scaling factor of quantization, min(x) represents the minimum value of input data, and y is the quantized fixed-point value;

[0033] Step (3): Convert the ONNX model to an OM model. Use the Ascend Tensor Compiler (ATC) tool to convert the ONNX format model to an OM format adapted to the Ascend AI chip. ATC will optimize for the Ascend 310 AI chip, for example, by adjusting the model structure to adapt to specific hardware acceleration features. ATC maps the operators in the model to the hardware's computing resources through static compilation to fully leverage the computing advantages of the Ascend 310B hardware;

[0034] Step (4): Deploy the om model on Ascend310 using ACL technology to achieve real-time inference, specifically including:

[0035] Step 4-1: Initialize the inference environment: Initialize the model inference environment using Ascend AI's Python API interface pyACL. The environment is initialized using the acl.init() function to ensure proper allocation of device resources, providing a stable hardware and software foundation for subsequent model loading and inference.

[0036] Step 4-2: Load the OM model. Load the OM model onto the Ascend 310B inference device by calling acl.mdl.load_from_file() to load the model into memory. The loaded model will be inferenced by the NPU (Neural Processing Unit). When loading the model, the shape and data type of the model's input / output tensors must be considered and necessary conversions must be performed.

[0037] Step (4-3): Inference execution. Model inference consists of two steps: First, the input data (such as medical images after parsing DICOM files) is loaded into memory. Then, the model is called for inference through the acl.mdl.execute() interface. During inference, the NPU uses a highly parallel computing architecture and accelerates the calculation of convolutional layers through matrix multiplication.

[0038] Z = X·W + B,

[0039] Where X is the input feature matrix, W is the weight matrix, B is the bias term, and Z is the result of the convolution operation; Step (5): Post-processing and result output: After the inference is completed, the output result is the segmented medical image, which is stored in a NumPy array. The output NumPy array is post-processed and converted to DICOM or PNG format for display on the front end. The inference framework used is AscendCL (Ascend Computing Language) based on Ascend, which provides efficient computing resource scheduling for inference tasks;

[0040] This example also builds a PACS system and deploys the model into the system, including: building a backend system, using Flask framework to build the backend, which is used to receive DICOM images in the hospital PACS system. The backend will parse the received DICOM file through the Orthanc server, extract the image data for inference; receive the file through the POST request of Flask and store it to the server, and Orthanc is responsible for handling the access of DICOM files.

[0041] In this example, the inference result is processed and stored as DICOM or other standard medical image format for doctors to view. The segmentation result after inference can be used to label the lesion area and generate an auxiliary diagnosis report; the output data result will be returned to the doctor through the PACS system and visualized through the web front end, helping the doctor quickly locate the lesion position.

[0042] In this example, the front end uses a JavaScript framework (such as Vue.js or React) to implement the interactive interface. Doctors can upload DICOM files and view the segmentation results, and also generate preliminary diagnosis reports based on the automatic labeling results of the system. The generation of the diagnosis report is based on the results of the model segmentation, combined with the size, location, shape and other parameters of the specific lesion.

Claims

1. A medical image processing method based on a large visual model, characterized in that, Based on the MedficientSAM image segmentation model, a lightweight optimization is performed to achieve segmentation processing of medical images with multiple modalities. The specific steps include: Step 1: Convert the trained PyTorch model to ONNX format to allow interoperability between different deep learning frameworks, thereby simplifying the model deployment process; Step 2: Perform quantization of the ONNX model: Before converting the ONNX model to the OM model, use the Ascend model compression tool AMCT to quantize the model, converting it from FP32 to INT8 format. This reduces the computational complexity of the model and speeds up inference. During quantization, the data is converted from floating-point numbers to fixed-point numbers. The process is represented by the following mathematical formula: ， in, This represents the scaling factor for quantization. This represents the minimum value of the input data. It is a quantized fixed-point value; Step 3: Convert the ONNX model to an OM model: Use the Ascend Tensor Compiler tool to convert the ONNX format model to the OM format adapted to the Ascend AI chip; Step 4: Deploy the OM model on Ascend310 using ACL technology to achieve real-time inference: Step 4-1: Initialize the inference environment. Use Ascend AI's Python API interface pyACL to initialize the model inference environment. Use the acl.init() function to initialize the environment and ensure the correct allocation of device resources, providing a stable hardware and software foundation for subsequent model loading and inference. Step 4-2: Load the OM model: Load the OM model onto the Ascend 310B inference device by calling acl.mdl.load_from_file() to load the model into memory. The loaded model will be used for inference computation by the NPU. When loading the model, the shape and data type of the model's input / output tensors must be considered and converted. Step 4-3: Inference Execution: First, the input data, the medical image data parsed from the DICOM file, is loaded into memory. The model is then invoked for inference via the acl.mdl.execute() interface. During inference, the NPU utilizes a highly parallel computing architecture, accelerating convolutional layer computation through matrix multiplication. ， in, It is the input feature matrix. It is a weight matrix. It is a bias term. It is the result of the convolution operation. Step 4-4: Post-processing and result output: After inference is completed, the output result is the segmented medical image, which is stored in a NumPy array. The output NumPy array is post-processed and converted to DICOM or PNG format for display on the front end.

2. A system for performing the medical image processing method based on a large visual model as described in claim 1, characterized in that, This includes setting up a PACS system and deploying the model into the PACS system to enable convenient interaction for operators. The backend is built using the Flask framework to receive DICOM images from the hospital's PACS system. The backend parses the received DICOM files through the Orthanc server, extracts the image data for inference, and receives and stores the files to the server via Flask POST requests. Orthanc is responsible for handling the storage and retrieval of DICOM files. There is also a frontend module, which uses the JavaScript framework to implement the interactive interface.

Images