Medical image processing method and system based on visual large model

Abstract

The invention relates to the technical field of image processing, in particular to a medical image processing method and system based on a large visual model, which are based on a large MedficientSAM image segmentation model and can remarkably improve the medical image processing efficiency and accuracy, and the MedficientSAM model is utilized to realize high-precision segmentation on medical images of nine different modalities including CT, MR, ultrasound and the like, so that the medical image processing efficiency and accuracy are improved. According to the system, through automatic focus area image data quantitative analysis, doctors are helped to carry out rapid preliminary screening, and the working efficiency and precision are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a medical image processing method and system based on a visual large model, which is based on the MedficientSAM image segmentation large model and can significantly improve the efficiency and accuracy of medical image processing. Background Art

[0002] Traditional medical expert models can achieve accurate segmentation for the lesion characteristics of a certain disease. In comparison, the visual large model has achieved good segmentation effect and generalization ability on natural images and remote sensing images after training with large-scale data. The current medical diagnosis strategy requires professional doctors to manually determine the location of the lesion. The medical expert model can only segment a single modality of data, making it difficult to further promote related applications. Summary of the invention

[0003] In view of the shortcomings and deficiencies in the prior art, the present invention proposes a medical image processing method and system based on a visual large model that can realize multiple medical image processing.

[0004] The present invention is achieved by the following measures:

[0005] A medical image processing method based on a visual large model is characterized by performing lightweight optimization based on the MedficientSAM image segmentation large model to achieve segmentation processing of multiple modal medical images, specifically comprising the following steps:

[0006] Step 1: Convert the trained PyTorch model to ONNX format to allow interoperability between different deep learning frameworks, thereby simplifying the model deployment process;

[0007] Step 2: Quantize the ONNX model: Before converting the ONNX model to the OM model, use the Ascend model compression tool AMCT to quantize the model from FP32 to INT8 format to reduce the model calculation complexity and speed up the inference speed. During quantization, the data is converted from floating point numbers to fixed point numbers. The process is expressed by the following mathematical formula:

[0008]

[0009] Where Δ represents the quantized scaling factor, min(x) represents the minimum value of the input data, and y is the fixed-point value after quantization;

[0010] Step 3: Convert the ONNX model to the OM model: Use the Ascend Tensor Compiler tool to convert the ONNX model to the OM format that is compatible with the Ascend AI chip.

[0011] Step 4: Use ACL technology to deploy the OM model on Ascend310 to achieve real-time reasoning: Step 4-1: Initialize the reasoning environment. Use the Python API interface pyACL of Ascend AI to initialize the model reasoning environment. Use the acl.init() function to initialize the environment to ensure the correct allocation of device resources, providing a stable hardware and software foundation for subsequent model loading and reasoning.

[0012] Step 4-2: Load the OM model: Load the OM model onto the Ascend 310B inference device. Call acl.mdl.load_from_file() to load the model into memory. The loaded model will be used by the NPU (Neural Processing Unit) for inference calculations. When loading the model, the shape and data type of the model's input / output tensors must be considered and necessary conversions must be performed.

[0013] Step 4-3: Inference execution: First, load the input data (such as medical imaging data after parsing the DICOM file) into the memory, and call the model for inference through the acl.mdl.execute() interface. During the inference process, the NPU uses a highly parallel computing architecture to accelerate the convolution layer calculation through matrix multiplication:

[0014] Z=X·W+B,

[0015] Among them, X is the input feature matrix, W is the weight matrix, B is the bias term, and Z is the result of the convolution operation. Step 4-4: Post-processing and result output: After the inference is completed, the output result is the segmented medical image, which is saved in the Numpy array. By post-processing the output Numpy array, it is converted into DICOM or PNG format for front-end display.

[0016] The present invention also proposes a medical image processing system based on a visual large model, which is characterized by including building a PACS (Picture Archiving and Communication System, a system in the medical field that integrates image storage, transmission, management and diagnosis) system and deploying the model to connect to the PACS system to achieve convenient interaction for operators. The backend is constructed using the Flask framework to receive DICOM medical images in the hospital PACS system. The backend parses the received DICOM medical images through the Orthanc server (an open source, lightweight DICOM server), extracts image data for reasoning, receives files through Flask's POST request and stores them in the server, and the Orthanc system is responsible for processing the access of DICOM medical images. A front-end module is also provided, and the front-end uses the JavaScript framework to implement the interactive interface.

[0017] After the inference results in the present invention are processed, they are stored in DICOM or other standard medical imaging formats for doctors to view. The segmentation results after inference can be used to mark the lesion area and generate auxiliary diagnosis reports. The output data results will be returned to the doctor through the PACS system and visualized through the web front end to help doctors quickly locate the lesion.

[0018] Compared with the prior art, the present invention further improves the model reasoning speed by performing lightweight optimization on the SAM with a huge number of parameters, thus providing a good prerequisite for the deployment of large visual models on medical equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Attached Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] Example:

[0022] This example aims to provide a medical image data processing method and system based on a visual big model, using the MedficientSAM model to achieve high-precision segmentation of nine different modalities of medical images including CT, MR, and ultrasound. The system helps doctors perform rapid preliminary screening and improve work efficiency and accuracy through automated quantitative analysis of image data in the lesion area.

[0023] In this example, the deep learning model is converted into an OM format file, and the model is quantized and tensor decomposed based on the Ascend Model Compression Tool (AMCT), ultimately achieving miniaturization and lightweighting of the model. By deploying the model on the Huawei Ascend 310B series inference card, with the help of Huawei's self-developed Da Vinci architecture, the system significantly reduces power consumption while ensuring high throughput and high computing power, and compared with the traditional GPU inference platform, the inference speed is significantly improved, greatly improving the timeliness of doctor-assisted diagnosis, as shown in Table 1:

[0024] Table 1 System performance data

[0025]

[0026] In addition, the system simulates the PACS data system commonly used in hospitals through the front-end and back-end architecture combined with database technology, and integrates the MedficientSAM model deployed on the Ascend device into the PACS system, thereby realizing fully automated, multi-modal, and high-precision medical image-assisted diagnosis.

[0027] This example is implemented through the following steps:

[0028] Step (1): Convert the model to ONNX format: First, convert the trained PyTorch model to ONNX format. ONNX is an intermediate representation for deep learning models. It allows interoperability between different deep learning frameworks, thereby simplifying the model deployment process and is used for cross-platform compatibility of AI models. The specific implementation is as follows: torch.onnx.export(model, input_tensor,"model.onnx", export_params = True);

[0029] This process ensures the complete export of model parameters and structure for subsequent deployment;

[0030] Step (2): Quantization of ONNX model: Before converting the ONNX model to the OM model, use the Ascend Model Compression Tool (AMCT) to quantize the model. AMCT uses Post-Training Quantization (PTQ), which can quantize the model from FP32 to INT8 format without sacrificing too much accuracy, reducing the model calculation complexity and speeding up the inference speed. During quantization, the data is converted from floating point numbers to fixed point numbers. The core process can be expressed by the following mathematical formula:

[0031]

[0032] Where Δ represents the quantized scaling factor, min(x) represents the minimum value of the input data, and y is the fixed-point value after quantization;

[0033] Step (3): Convert the ONNX model to an OM model. Use the Ascend Tensor Compiler (ATC) tool to convert the ONNX model to an OM format that is compatible with the Ascend AI chip. ATC will optimize the Ascend 310AI chip, such as adjusting the model structure to adapt to specific hardware acceleration features. ATC maps the operators in the model to the computing resources of the hardware through static compilation to give full play to the computing advantages of the Ascend 310B hardware.

[0034] Step (4): Use ACL technology to deploy the om model on Ascend310 to achieve real-time reasoning, including:

[0035] Step 4-1: Initialize the inference environment: Use the Python API interface pyACL of Ascend AI to initialize the model inference environment. Use the acl.init() function to initialize the environment to ensure the correct allocation of device resources, providing a stable hardware and software foundation for subsequent model loading and inference;

[0036] Step 4-2: Load the OM model. Load the OM model onto the Ascend 310B inference device. Call acl.mdl.load_from_file() to load the model into memory. The loaded model will be used by the NPU (Neural Processing Unit) for inference calculations. When loading the model, the shape and data type of the model's input / output tensors must be considered and necessary conversions must be performed.

[0037] Step (4-3): Inference execution. Model inference is divided into two steps: first, the input data (such as medical images after DICOM file parsing) is loaded into the memory, and the model is called for inference through the acl.mdl.execute() interface. During the inference process, the NPU uses a highly parallel computing architecture to accelerate the convolution layer calculation through matrix multiplication:

[0038] Z=X·W+B,

[0039] Where X is the input feature matrix, W is the weight matrix, B is the bias term, and Z is the result of the convolution operation; Step (5): Post-processing and result output: After the inference is completed, the output result is the segmented medical image, which is saved in a Numpy array. The output Numpy array is post-processed and converted into DICOM or PNG format for front-end display. The inference framework used is AscendCL (Ascend Computing Language) based on Ascend, which provides efficient computing resource scheduling for inference tasks;

[0040] This example also builds a PACS system and deploys the model to the system, including: building a backend system, using the Flask framework to build the backend to receive DICOM images from the hospital PACS system. The backend parses the received DICOM files through the Orthanc server and extracts the image data for reasoning; receives files through Flask's POST request and stores them on the server, and Orthanc is responsible for handling the storage and access of DICOM files.

[0041] In this example, after the inference results are processed, they are stored in DICOM or other standard medical image formats for doctors to view. The segmentation results after inference can be used to mark the lesion area and generate auxiliary diagnosis reports; the output data results will be returned to the doctor through the PACS system and visualized through the web front end to help doctors quickly locate the lesion.

[0042] In this example, the front-end uses a JavaScript framework (such as Vue.js or React) to implement the interactive interface. Doctors can upload DICOM files and view the segmentation results, and can also generate a preliminary diagnosis report based on the results of the system's automatic annotation. The generation of the diagnosis report is based on the results of the model segmentation, combined with the size, location, morphology and other parameters of the specific lesion.

Claims

1. A medical image processing method based on a visual large model, characterized in that: Based on the MedficientSAM image segmentation model, lightweight optimization is performed to achieve segmentation processing of multi-modal medical images, which specifically includes the following steps: Step 1: Convert the trained PyTorch model to ONNX format to allow interoperability between different deep learning frameworks, thereby simplifying the model deployment process; Step 2: Quantize the ONNX model: Before converting the ONNX model to the OM model, use the Ascend model compression tool AMCT to quantize the model from FP32 to INT8 format to reduce the model calculation complexity and speed up the inference speed. During quantization, the data is converted from floating point numbers to fixed point numbers. The process is expressed by the following mathematical formula: Where Δ represents the quantized scaling factor, min(x) represents the minimum value of the input data, and y is the fixed-point value after quantization; Step 3: Convert the ONNX model to the OM model: Use the Ascend Tensor Compiler tool to convert the ONNX model to the OM format that is compatible with the Ascend AI chip. Step 4: Use ACL technology to deploy the OM model on Ascend310 to achieve real-time reasoning: Step 4-1: Initialize the reasoning environment. Use the Python API interface pyACL of Ascend AI to initialize the model reasoning environment. Use the acl.init() function to initialize the environment to ensure the correct allocation of device resources, providing a stable hardware and software foundation for subsequent model loading and reasoning. Step 4-2: Load the OM model: Load the OM model onto the Ascend 310B inference device. Call acl.mdl.load_from_file() to load the model into memory. The loaded model will be used by the NPU (Neural Processing Unit) for inference calculations. When loading the model, the shape and data type of the model's input / output tensors must be considered and necessary conversions must be performed. Step 4-3: Inference execution: First, load the input data (such as medical imaging data after parsing the DICOM file) into the memory, and call the model for inference through the acl.mdl.execute() interface. During the inference process, the NPU uses a highly parallel computing architecture to accelerate the convolution layer calculation through matrix multiplication: Z=X·W+B, Among them, X is the input feature matrix, W is the weight matrix, B is the bias term, and Z is the result of the convolution operation. Step 4-4: Post-processing and result output: After the inference is completed, the output result is the segmented medical image, which is saved in the Numpy array. By post-processing the output Numpy array, it is converted into DICOM or PNG format for front-end display.

2. A medical image processing system based on a visual large model, characterized in that: It includes building a PACS system and deploying the model to the PACS system to achieve convenient interaction for operators. The Flask framework is used to build the backend to receive DICOM images from the hospital PACS system. The backend parses the received DICOM files through the Orthanc server, extracts the image data for reasoning, and receives the files through Flask's POST request and stores them on the server. Orthanc is responsible for handling the storage and access of DICOM files. There is also a front-end module, and the front-end uses the JavaScript framework to implement the interactive interface.

Images