A cross-modal medical image quality assessment and body position detection model and method

Abstract

The present invention discloses a cross-modal medical image quality assessment and body position detection model and method, the model includes an information encoding module, a hybrid convolution model using super convolution and a Swin-transformer model; the information encoding module can generate corresponding medical image type codes according to the modal information, part information and type information of the data set; the hybrid convolution model can use different modal data sets and medical image type codes to perform quality assessment prediction training, and can perform quality assessment on predicted CT or MRI images; the Swin-transformer model can use different modal data sets and medical image type codes to perform body position prediction training, and can perform body position detection on predicted CT or MRI images. The present invention can accurately and automatically perform quality assessment and body position detection on medical images of multiple modes and anatomical parts, with high versatility, high speed, high accuracy and good robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a cross-modal medical image quality assessment and body position detection model and method. Background Art

[0002] CT (computed tomography) and MRI (magnetic resonance imaging) are two important medical imaging technologies. They can provide doctors with detailed information about tissue structures and have been widely used in the clinical diagnosis and treatment of various diseases. CT and MRI have their own advantages and complement each other. Clinicians need to choose according to specific needs. However, for a large number of CT images and MRI images, ensuring their quality is the basis for ensuring patient safety, accurate diagnosis and effective treatment. In clinical practice, image clarity and integrity are important factors affecting image quality. High-quality images provide clearer and more detailed anatomical information, which helps doctors accurately diagnose diseases and formulate treatment plans. Low-quality images may lead to misdiagnosis or missed diagnosis, affecting the patient's diagnosis and treatment results. Therefore, medical image quality assessment and body position detection are of great significance.

[0003] Image Quality Assessment (IQA) is a method to measure the quality of an image, which aims to simulate the perception of the human eye on an image. According to the subject of the assessment, image quality assessment is mainly divided into subjective assessment and objective assessment. Among them, subjective assessment relies on medical imaging professionals or clinicians to observe the image through human eyes to assess its quality, and the assessment results are based on visual perception and subjective consciousness. At present. The three most commonly used subjective assessment methods include double stimulus damage grading method, double stimulus continuous quality grading method and single stimulus continuous quality grading method. Objective assessment can be further divided into full reference image quality assessment, semi-reference image quality assessment and no reference image quality assessment based on whether there is a reference image. Due to its particularity, medical images do not have effective reference images. Body position detection is to analyze the image to detect whether the image includes complete anatomical structures to meet clinical examination needs.

[0004] At present, the image quality assessment of CT and MRI is still carried out in traditional ways such as manual investigation, supervision and reporting. However, subjective assessment is not only time-consuming and laborious, but different professionals will produce different assessment results, resulting in inconsistent assessment results. Therefore, researchers have begun to use computer-aided tools or software to objectively and quantitatively evaluate image quality, by calculating and measuring parameters such as noise level, contrast, and resolution to provide objective assessment results. Objective assessment is automatically completed through algorithms, which can provide consistent and repeatable results, and convert image quality into measurable values ​​for easy comparison and analysis. At present, various indicators and methods such as signal-to-noise ratio, structural similarity index, and mean square error have been developed for objective assessment. However, objective assessment algorithms usually need to process large amounts of data and complex calculations. At the same time, as a special imaging mode, objective assessment of medical images often lacks clinical relevance and cannot effectively capture subjective factors.

[0005] In recent years, with the rise of deep learning, researchers have begun to use deep learning methods for automated image quality assessment. By training artificial intelligence models, automatic evaluation of medical image quality can be achieved, which improves the objectivity and efficiency of the evaluation. At the same time, artificial intelligence continuously optimizes and improves its evaluation model through learning, making its evaluation results closer to the actual situation. With the development of artificial intelligence, Lei et al. used deep learning and machine learning methods to evaluate the quality of medical images of different modalities. In order to explore the effectiveness of IQA methods on medical images, Mason et al. evaluated 10 full-reference image quality assessment methods, and Stępień et al. evaluated the no-reference image quality assessment method of MRI. In addition, Kastryulin et al. evaluated the application of 35 different image quality assessment methods (including full reference and no-reference) in MRI using signal-to-noise ratio, CNR and the presence of artifacts as evaluation criteria.

[0006] In these studies, different deep learning methods were used to assess the image quality of medical images. However, most of the existing AI-based models use synthetic data for inference, and their accuracy remains to be verified. At the same time, the models have poor robustness and generalization, and cannot be applied to the quality assessment of medical images of different modalities. As for body position detection, there is still no research on the detection of related missing conditions. Summary of the invention

[0007] Based on this, the present invention provides a cross-modal medical image quality assessment and body position detection model and method to achieve accurate and automated cross-modal medical image quality assessment and body position detection, which is conducive to solving the problems of existing medical image assessment, such as being time-consuming and labor-intensive, having low accuracy and poor robustness.

[0008] In order to solve the above technical problems and achieve the above technical effects, the present invention is implemented through the following technical solutions:

[0009] A cross-modal medical image quality assessment and body position detection model, including: an information encoding module, a hybrid convolution (CNN) model using hyper-convolution (Hyper-conv) and a Swin-transformer model; wherein,

[0010] The information encoding module is responsible for generating corresponding medical image type codes according to the modality information, part information and type information of the collected CT or MRI image data set, so as to serve as guidance information for training the hybrid convolutional model and the Swin-transformer model;

[0011] The hybrid convolutional model is responsible for using the collected CT or MRI image data set and the corresponding medical image type code to perform regression training on predicting the quality scores and radiation dose / magnetic field strength scores of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, automatically predicting and generating the quality score and / or radiation dose / magnetic field strength score of the CT or MRI image to be predicted;

[0012] The Swin-transformer model is responsible for using the collected CT or MRI image data set and the corresponding medical image type coding to perform regression training to predict the body position missing ratio of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, predicting and generating the body position missing ratio of the CT or MRI image to be predicted.

[0013] Furthermore, the architecture of the hybrid convolutional model consists of a first feature extraction module, an output prompt module and a first fully connected module;

[0014] The first feature extraction module is responsible for extracting features from the input CT or MRI image;

[0015] The first feature extraction module includes 5 feature extraction units arranged in sequence, each of which is composed of a convolution layer, a batch normalization layer, a linear activation layer and a maximum pooling layer arranged in sequence; wherein the convolution layers in the first three feature extraction units are ordinary convolution layers, and the convolution layers in the last two feature extraction units are super convolution layers;

[0016] The output prompt module is responsible for generating corresponding output prompt information according to the prediction result type selected by the user, and splicing the output prompt information with the feature vector generated by the first feature extraction module to generate a feature with prompt information;

[0017] The first fully connected module is responsible for predicting and generating a quality score and / or a radiation dose / magnetic field strength score of an input CT or MRI image according to the feature with prompt information generated by the output prompt module;

[0018] The first fully connected module includes three linear layers and one softmax layer, wherein a dropout layer with a parameter of 0.5 and a linear activation layer are added after the first two linear layers.

[0019] Furthermore, the architecture of the Swin-transformer model consists of a second feature extraction module and a second fully connected module;

[0020] The second feature extraction module is responsible for extracting features from the input CT or MRI image;

[0021] The second feature extraction module includes a patch partitioning unit and four sequentially arranged stage units, each of which is composed of a patch merging layer and a Swin-transformer module;

[0022] The second fully connected module is responsible for predicting and generating the body rotation angle and body position missing status of the input CT or MRI image according to the features extracted by the second feature extraction module;

[0023] The second fully connected module includes three linear layers and one softmax layer, wherein a dropout layer with a parameter of 0.5 and a linear activation layer are added after the first two linear layers.

[0024] A training method for the above cross-modal medical image quality assessment and body position detection model comprises the following steps:

[0025] Step 1) Collect CT images with different radiation doses and MRI images with different magnetic field strengths to construct a cross-modal medical quality assessment dataset;

[0026] Step 2) Collect a certain number of CT images and MRI images and simulate different body position missing and angle errors to construct a body position detection dataset;

[0027] Step 3) the information encoding module respectively obtains the modality information, location information and type information of each medical image in the cross-modality medical quality assessment dataset and the body position detection dataset, and converts the information into a medical image type code corresponding to each medical image;

[0028] Step 4) inputting the cross-modal medical quality assessment dataset into the hybrid convolutional model, and adding the medical image type code corresponding to the cross-modal medical quality assessment dataset as guidance information into the hybrid convolutional model, and performing quality score and / or radiation dose / magnetic field strength score regression training on the hybrid convolutional model, so that the hybrid convolutional model can automatically predict the quality scores and / or radiation dose / magnetic field strength scores of medical images of different modalities;

[0029] Step 5) inputting the body position detection data set into the Swin-transformer model, and adding the medical image type code corresponding to the body position detection data set as guidance information into the Swin-transformer model, and performing body position missing ratio regression training on the Swin-transformer model, so that the Swin-transformer model can automatically predict the body position missing ratio of medical images of different modalities;

[0030] Step 6) Optimize the model parameters of the hybrid convolutional model and the Swin-transformer model using a mean square error loss function.

[0031] Furthermore, the cross-modal medical quality assessment data set includes CT data consisting of CT images with different radiation doses and MRI data consisting of MRI images with different magnetic field strengths; wherein,

[0032] The CT data is composed of a certain number of CT image groups, each of which contains at least four CT images with different radiation doses, and each of the CT images is labeled with the radiation dose corresponding to the image;

[0033] The MRI data consists of a certain number of MRI image groups, each of which contains at least two MRI images with different magnetic field strengths, and each of the MRI images is annotated with the corresponding magnetic field strength as a training label.

[0034] Furthermore, the method for simulating the treatment of different body position loss and angle malalignment is:

[0035] A certain number of CT images are collected, and they are randomly rotated within a certain angle range and randomly translated within a certain pixel range to obtain simulated body position CT images of different body positions, and the volume calculation missing ratio of the anatomical structure before and after simulation of each simulated body position CT image is calculated. Each simulated body position CT image is annotated with the calculated volume calculation missing ratio as a training label.

[0036] Furthermore, the information encoding module uses corresponding one-hot encoding (One-Hot Encoding) to respectively represent the modality information, part information and type information of each medical image in the cross-modal medical quality assessment dataset and the body position detection dataset, so as to facilitate the training of the hybrid convolutional model and the Swin-transformer model.

[0037] Furthermore, when training the hybrid convolutional model, the medical image type encoding corresponding to the cross-modal medical quality assessment dataset is first used as guidance information and added to the super-convolutional layer of the first feature extraction module of the hybrid convolutional model. Then, the first feature extraction module in the hybrid convolutional model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the user randomly determines that one or two types of prediction results, including quality score and radiation dose / magnetic field strength score, need to be output. Then, the output prompt module in the hybrid convolutional model generates corresponding output prompt information according to the prediction result type selected by the user, and concatenates the output prompt information with the feature vector generated by the first feature extraction module to generate features with prompt information. Finally, the first fully connected module in the hybrid convolutional model learns to predict quality scores and / or radiation dose / magnetic field strength scores for medical images of different modalities according to the features with prompt information and the corresponding quality score training labels and / or radiation dose / magnetic field strength training labels, thereby completing the corresponding quality and / or radiation dose / magnetic field strength regression training.

[0038] Furthermore, the method of adding the medical image type encoding corresponding to the cross-modal medical quality assessment dataset to the super-convolutional layer is:

[0039] Firstly, the corresponding medical image type code is input into the multi-layer perceptron in the super-convolutional layer to generate a mapping function of the modality information;

[0040] Then, the generated mapping function is used to perform matrix multiplication with the training weight library, so that the convolution kernel of the super convolution layer is represented as a function of the kernel coordinates;

[0041] Finally, the kernel size and its receptive field are decoupled from the number of learnable parameters by obtaining the convolution kernel coordinates and directly generating the kernel weights.

[0042] Furthermore, when training the Swin-transformer model, the medical image type encoding corresponding to the body position detection data set is first used as guidance information and directly added to the second feature extraction module of the Swin-transformer model. Then, the second feature extraction module in the Swin-transformer model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the second fully connected module in the Swin-transformer model learns to predict the volume calculation missing ratio of medical images of different modalities based on the features and the volume calculation missing ratio as training labels, thereby completing the corresponding body position missing ratio regression training.

[0043] A cross-modality medical image quality assessment and body position detection method comprises the following steps:

[0044] Step 1) obtaining the CT or MRI image to be predicted;

[0045] Step 2) inputting the acquired CT or MRI image to be predicted into the cross-modal medical image quality assessment and body position detection model as described in any one of claims 1 to 3 or the cross-modal medical image quality assessment and body position detection model trained by the training method as described in any one of claims 4 to 9;

[0046] Step 3) The user sets the type of prediction result to be output as a quality score or a radiation dose / magnetic field strength score or both, and generates corresponding output prompt information accordingly;

[0047] Step 4) the cross-modality medical image quality assessment and body position detection model processes the input CT or MRI image, and generates corresponding quality score prediction results and / or radiation dose / magnetic field intensity prediction results, as well as body position loss ratio prediction results according to the output prompt information;

[0048] Step 5) outputting the generated quality score prediction result and / or radiation dose / magnetic field strength prediction result, and the body position loss ratio prediction result.

[0049] A computer device comprises: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus, and the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform operations corresponding to the above-mentioned cross-modal medical image quality assessment and body position detection method.

[0050] A computer-readable storage medium stores at least one executable instruction, wherein the executable instruction enables a processor to perform operations corresponding to the above-mentioned cross-modality medical image quality assessment and body position detection method.

[0051] The beneficial effects of the present invention are:

[0052] 1. When constructing the model, the present invention combines a hybrid convolution model using hyper-conv and a Swin-transformer model, so that the model of the present invention has quality assessment capabilities, radiation dose / magnetic field strength assessment capabilities, and body position detection capabilities. Among them, the hyper-conv layer replaces part of the ordinary convolution layer in the original convolutional neural network. The use of Hyper-conv can increase the number of convolution kernels when the parameters are fixed, and reduce the learning parameters when the convolution kernels are fixed, effectively improving the accuracy and generalization of the model.

[0053] 2. When training the model, the present invention not only constructs a cross-modal medical image quality assessment dataset and a body position detection dataset to cope with the training of the hybrid convolutional model and the Swin-transformer model respectively, but also adds medical image type encoding to guide the training of the hybrid convolutional model and the Swin-transformer model, thereby further improving the model performance.

[0054] 3. The model of the present invention can perform accurate and automatic quality assessment, radiation dose / magnetic field strength assessment and body position detection on medical images of various modes and anatomical parts. It is a universal cross-modal medical image quality assessment and body position detection model with the characteristics of fast prediction speed, high accuracy and good robustness.

[0055] 4. The model of the present invention is used to perform image quality assessment, radiation dose / magnetic field strength assessment and body position detection on medical images of different modalities, which helps to ensure key features of medical images such as clarity, contrast and spatial resolution, and evaluate image integrity through body position, thereby improving doctors' reading efficiency, which is crucial for early diagnosis and treatment planning of diseases.

[0056] 5. At the same time, the model and method of the present invention provide effective assistance for the fusion, registration and segmentation of cross-modal medical images, ensuring the consistency and accuracy of various tasks in cross-modal medical images. Therefore, the model of the present invention is expected to be used as an effective tool for evaluating medical image quality and body position detection.

[0057] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0059] Figure 1 The model architecture diagram of the cross-modal medical image quality assessment and body position detection model of the present invention;

[0060] Figure 2 It is a flowchart of the cross-modality medical image quality assessment and body position detection model training method of the present invention;

[0061] Figure 3 is a schematic diagram of cross-modal medical quality assessment data for training a cross-modal medical image quality assessment model in an embodiment of the present invention;

[0062] Figure 4 Schematic diagram of body position detection data used for training a cross-modal medical image quality assessment model in an embodiment of the present invention;

[0063] Figure 5 Schematic diagram of the process of the cross-modal medical image quality assessment and body position detection method of the present invention. DETAILED DESCRIPTION

[0064] The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings so that the purpose, features and advantages of the invention can be more clearly understood. It should be understood that the embodiments shown in the accompanying drawings are not intended to limit the scope of the present invention, but are only intended to illustrate the essential spirit of the technical solution of the present invention.

[0065] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the relevant art will recognize that the embodiments may be practiced without one or more of these specific details. In other cases, well-known devices, structures, and techniques associated with the present application may not be shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0066] Unless the context requires otherwise, throughout the specification and claims, the word "comprise" and variations such as "include" and "have" should be construed in an open, inclusive sense, ie, should be interpreted as "including, but not limited to."

[0067] References throughout the specification to "one embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of "in one embodiment" or "in an embodiment" in various places throughout the specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any manner in one or more embodiments.

[0068] As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It should be noted that the term "or" is generally employed in its sense including "and / or" unless the context clearly dictates otherwise.

[0069] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0070] See also Figure 1 As shown, the present invention first proposes a cross-modal medical image quality assessment and body position detection model. The architecture of the model mainly includes an information encoding module, an improved convolutional neural network (CNN) model using hyper-convolution (Hyper-conv), and a Swin-transformer model.

[0071] The information encoding module can generate corresponding medical image type codes according to the modality information, part information and type information of the collected CT or MRI image data set, so as to serve as guiding information during the training of the hybrid convolutional model and the Swin-transformer model.

[0072] The hybrid convolution model is responsible for using the collected CT or MRI image data set and the corresponding medical image type code to perform regression training to predict the quality scores and radiation dose / magnetic field intensity scores of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, automatically predicting and generating the quality score and / or radiation dose / magnetic field intensity score of the CT or MRI image to be predicted.

[0073] The architecture of the hybrid convolutional model consists of a first feature extraction module, an output prompt module and a first fully connected module.

[0074] The first feature extraction module is responsible for extracting features from the input CT or MRI image.

[0075] The first feature extraction module includes 5 feature extraction units arranged in sequence, each of which is composed of a convolutional layer, a batch normalization layer, a linear activation layer and a maximum pooling layer arranged in sequence; wherein the convolutional layers in the first 3 feature extraction units are ordinary convolutional layers, and the convolutional layers in the last 2 feature extraction units are hyper-convolutional layers.

[0076] The output prompt module is responsible for generating corresponding output prompt information according to the prediction result type selected by the user, and concatenating the output prompt information with the feature vector generated by the first feature extraction module to generate features with prompt information.

[0077] In order to enhance the versatility of the model and enable the hybrid convolutional model to better perform different types of quality score and / or radiation dose / magnetic field intensity score regression tasks after feature extraction, the present invention designs an output prompt module between the first feature extraction module and the first fully connected module. The output prompt module is to adapt to the migration to human-scored data, that is, the hybrid convolutional model of the present invention can train data sets labeled by dose and intensity, data labeled by manual scoring, and mixed data sets containing the above two types of data. In order to help the hybrid convolutional model identify the output type required by the user, the present invention adds this module again.

[0078] The first fully connected module is responsible for predicting and generating a quality score and / or a radiation dose / magnetic field strength score of an input CT or MRI image according to the feature with prompt information generated by the output prompt module.

[0079] The first fully connected module includes three linear layers and one softmax layer, wherein a dropout layer with a parameter of 0.5 and a linear activation layer are added after the first two linear layers.

[0080] In order to unify the output results of images of different modalities within the same range, the present invention adds a softmax layer at the end of the first fully connected module. The function of the softmax layer is to exponentially process the output of the hybrid convolutional model (quality scores and / or radiation dose / magnetic field strength scores of medical images of different modalities), and then normalize the processed results so that the value of each output node is between 0 and 1.

[0081] The Swin-transformer model is responsible for using the collected CT or MRI image data set and the corresponding medical image type code to perform regression training to predict the body position missing ratio of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, predicting and generating the body position missing ratio (volume calculation missing ratio) of the CT or MRI image to be predicted.

[0082] The architecture of the Swin-transformer model consists of a second feature extraction module and a second fully connected module.

[0083] The second feature extraction module is responsible for extracting features from the input CT or MRI image.

[0084] The second feature extraction module includes a patch partitioning unit and four stage units arranged in sequence, and each of the stage units is composed of a patch merging layer and a Swin-transformer module.

[0085] The second fully connected module is responsible for predicting and generating the body rotation angle and body position missing status of the input CT or MRI image based on the features extracted by the second feature extraction module.

[0086] The second fully connected module includes three linear layers and one softmax layer, wherein a dropout layer with a parameter of 0.5 and a linear activation layer are added after the first two linear layers.

[0087] In order to unify the output results of images of different modalities within the same range, the present invention adds a softmax layer at the end of the second fully connected module. The function of the softmax layer is to index the output of the Swin-transformer model (the body position loss ratio of medical images of different modalities) and then normalize the processed results so that the value of each output node is between 0 and 1.

[0088] See also Figure 2 As shown, after the above-mentioned cross-modal medical image quality assessment and body position detection model is constructed, the present invention further provides a method for training the cross-modal medical image quality assessment and body position detection model, comprising the following steps:

[0089] 1. Construct a cross-modal medical quality assessment dataset;

[0090] CT images with different radiation doses and MRI images with different magnetic field strengths are collected to construct a cross-modal medical quality assessment dataset.

[0091] The cross-modality medical quality assessment dataset includes CT data consisting of CT images with different radiation doses and MRI data consisting of MRI images with different magnetic field strengths.

[0092] The CT data is composed of a certain number of CT image groups, each of which contains at least four CT images with different radiation doses, and each of the CT images is labeled with the radiation dose corresponding to the image;

[0093] The MRI data consists of a certain number of MRI image groups, each of which contains at least two MRI images with different magnetic field strengths, and each of the MRI images is annotated with the corresponding magnetic field strength as a training label.

[0094] The cross-modal medical image quality assessment and body position detection model of the present invention can theoretically be applied to the quality assessment of CT or MRI images of all parts, organs or tissues. By selecting a CT or MRI image dataset of a specific part, organ or tissue, the cross-modal medical image quality assessment and body position detection model of the present invention can be specially trained to predict the quality score of the CT or MRI image of the desired part, organ or tissue.

[0095] See also Figure 3 As shown in FIG. 1 , as an embodiment of the present invention, the cross-modal medical quality assessment dataset of this embodiment is collected from public datasets and data from medical institutions. As far as we know, the cross-modal medical quality assessment dataset of this embodiment is the first medical image dataset constructed using real data rather than synthetic data. The specific construction method is as follows:

[0096] First, chest lung window CT images and chest soft tissue window CT images with four different radiation doses of 30mAs (milliampere-seconds), 50mAs, 150mAs, and 200mAs were collected from corresponding medical institutions, and 200 cases of CT images of each type and dose were collected, for a total of 1,600 CT data. The higher the radiation dose, the higher the corresponding CT image quality;

[0097] Then, we selected 200 breast MRI images with magnetic field strength of 1.5T and 3T from the Duke University breast MRI public dataset, and 200 brain MRI images with magnetic field strength of 1.5T and 3T from the Alzheimer's disease standardized MRI public dataset, for a total of 800 MRI data.

[0098] Finally, 1,600 CT data and 800 MRI data were annotated separately, with their corresponding different doses or magnetic field strengths as labels, forming a complete cross-modal medical quality assessment dataset, with a total of 2,400 image data.

[0099] 2. Construct a body position detection dataset;

[0100] A certain number of CT images and MRI images are collected and simulated with different body position missing and angle errors to construct a body position detection dataset.

[0101] The method for simulating the treatment of different body position loss and angle malalignment is:

[0102] A certain number of CT images are collected, and they are randomly rotated within a certain angle range and randomly translated within a certain pixel range to obtain simulated body position CT images of different body positions, and the volume calculation missing ratio of the simulated anterior and posterior structures of each simulated body position CT image is calculated. Each simulated body position CT image is annotated with the calculated volume calculation missing ratio as a training label.

[0103] The cross-modal medical image quality assessment and body position detection model of the present invention can theoretically be applied to the body position detection of CT or MRI images of all parts, organs or tissues. The cross-modal medical image quality assessment and body position detection model of the present invention can be specially trained to predict the body position rotation angle and body position loss of CT or MRI images of the required parts, organs or tissues by selecting CT or MRI images of specific parts, organs or tissues and then forming a data set through specific different body position simulation processing.

[0104] See also Figure 4 As shown, as an embodiment of the present invention, the body position detection data set of this embodiment is collected from data of medical institutions. The specific construction method is:

[0105] First, 475 chest lung window CT images were collected from corresponding medical institutions; since the purpose of the next model training is to detect the missing state of the lungs, the embodiment of the present invention uses lung window CT images;

[0106] Then, the 475 lung window CT images collected were randomly rotated at an angle of (-70°, 70°) and randomly translated at a pixel of (50, 200) to simulate different body positions, with a simulation probability of 0.6;

[0107] Finally, the missing ratio is calculated by calculating the volume of the lung structure before and after simulation as the training label to form a complete body position detection dataset.

[0108] 3. Edit medical image type codes;

[0109] The information encoding module respectively obtains the modality information, location information and type information of each medical image in the cross-modality medical quality assessment dataset and the body position detection dataset, and converts the information into a medical image type code corresponding to each medical image.

[0110] The information encoding module uses corresponding one-hot encoding (One-Hot Encoding) to respectively represent the modality information, part information and type information of each medical image in the cross-modal medical quality assessment dataset and the body position detection dataset, so as to facilitate the training of the hybrid convolution model and the Swin-transformer model.

[0111] See also Figure 1 As shown, as an embodiment of the present invention, this embodiment uses four images: chest CT lung window image, chest CT soft tissue window, breast MRI T1 sequence image, and brain MRI T1 sequence. First, according to the modality information, location information, and type information, single-hot encoding is used to represent different image modalities, and it is used to control the output of the convolution layer. Specifically, this embodiment designs CT encoding as 01, MRI encoding as 10, chest scan as 001, breast scan as 010, brain scan as 100, T1 modality image as 001, lung window image as 010, and soft tissue window image as 100.

[0112] 4. Quality assessment training;

[0113] The cross-modal medical quality assessment dataset is input into the hybrid convolutional model, and the medical image type code corresponding to the cross-modal medical quality assessment dataset is added to the hybrid convolutional model as guidance information, and the hybrid convolutional model is trained on quality scores and / or radiation dose / magnetic field strength scores, so that the hybrid convolutional model can automatically predict the quality scores and / or radiation dose / magnetic field strength scores of medical images of different modalities.

[0114] When training the hybrid convolutional model, the medical image type encoding corresponding to the cross-modal medical quality assessment dataset is first used as guidance information and added to the super-convolutional layer of the first feature extraction module of the hybrid convolutional model. Then, the first feature extraction module in the hybrid convolutional model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the user randomly determines that one or two types of prediction results of the quality score and the radiation dose / magnetic field strength score need to be output. Then, the output prompt module in the hybrid convolutional model generates corresponding output prompt information according to the prediction result type selected by the user, and splices the output prompt information with the feature vector generated by the first feature extraction module to generate features with prompt information. Finally, the first fully connected module in the hybrid convolutional model learns to predict the quality score and / or radiation dose / magnetic field strength score of medical images of different modalities according to the features with prompt information and the corresponding quality score training labels and / or radiation dose / magnetic field strength training labels, thereby completing the corresponding quality and / or radiation dose / magnetic field strength regression training.

[0115] The output prompt information may include three types, namely, outputting the quality score alone, outputting the radiation dose / magnetic field strength score alone, and outputting the quality score and the radiation dose / magnetic field strength score at the same time. Through these three different types of output prompt information, the hybrid convolution model is helped to understand the results to be output. The output prompt module maps the output prompt information into a feature vector with a length of 128, and splices this feature vector with the feature vector extracted by the hybrid convolution model to achieve the purpose of the prompt.

[0116] See also Figure 1 As shown, in the hybrid convolution model, the present invention uses hyper-convolution to replace the original ordinary convolution layer in CNN. The present invention first inputs the corresponding medical image type code into the multi-layer perceptron in the hyper-convolution layer to generate a mapping function of the modal information; then uses the generated mapping function to perform matrix multiplication with the training weight library, so as to express the convolution kernel of the hyper-convolution layer as a function of the kernel coordinates; finally, the kernel size and its receptive field are decoupled from the number of learnable parameters, and the kernel weights are directly generated by obtaining the convolution kernel coordinates. The use of hyper-convolution can increase the number of convolution kernels when the parameters are fixed; when the convolution kernel is fixed, the learning parameters are reduced, effectively improving the accuracy and generalization of the model.

[0117] 5. Posture detection training;

[0118] The body position detection dataset is input into the Swin-transformer model, and the medical image type code corresponding to the body position detection dataset is added to the Swin-transformer model as guidance information, and the Swin-transformer model is trained for body position missing ratio regression, so that the Swin-transformer model can automatically predict the body position missing ratio of medical images of different modalities.

[0119] When training the Swin-transformer model, the medical image type code corresponding to the body position detection data set is first used as guidance information and directly added into the second feature extraction module of the Swin-transformer model. Then, the second feature extraction module in the Swin-transformer model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the second fully connected module in the Swin-transformer model learns to predict the volume calculation missing ratio of medical images of different modalities based on the features and the volume calculation missing ratio as training labels, thereby completing the corresponding body position missing ratio regression training.

[0120] The Swin-transformer model uses a hierarchical structure to provide feature information of various scales. The sliding window operation includes non-overlapping local windows and overlapping cross windows, and the moving window operation enables adjacent windows to interact with each other. The hierarchical design of the Swin-transformer model limits the focus on a single window, on the one hand, introduces the locality of the convolution operation of the convolutional neural network model (CNN), and on the other hand, saves the amount of calculation. The present invention uses Swin-transformer to predict the body position loss and body position rotation angle of medical images, thereby knowing the body position detection situation.

[0121] 6. Model optimization;

[0122] The learning of the cross-modal medical image quality assessment and body position detection model of the present invention uses the MSE (mean square error) loss function to optimize the model parameters of the hybrid convolution model and the Swin-transformer model. MSE measures the mean square error between the model prediction value and the actual value. The smaller the MSE value, the smaller the difference between the model's prediction value and the true value, and the better the performance of the model. The calculation formula of MSE is as follows:

[0123] (1);

[0124] In formula (1), n ​​is the number of samples, x i is the true value of the model, yi is the predicted value of the model.

[0125] The present invention builds a model based on Python 3.9 environment and PyTorch framework. The optimizer uses the Adam optimizer with an initial learning rate of 0.0001. The regularization step size is set to 1 and the gamma is set to 0.75. The number of training rounds is set to 50 and the batch size is set to 1. The present invention trains and tests the cross-modal medical image quality assessment and body position detection model on an Intel Xeon W2245 CPU @ 3.90GHz and an NVIDIA RTX A6000 computer.

[0126] See also Figure 5 As shown, based on the above-trained cross-modal medical image quality assessment and body position detection model, the present invention also proposes a cross-modal medical image quality assessment and body position detection method, which specifically includes the following steps:

[0127] Step 1) obtaining the CT or MRI image to be predicted;

[0128] Step 2) inputting the acquired CT or MRI image to be predicted into the trained cross-modal medical image quality assessment and body position detection model;

[0129] Step 3) The user sets the type of prediction result to be output as a quality score or a radiation dose / magnetic field strength score or both, and generates corresponding output prompt information accordingly;

[0130] Step 4) the cross-modality medical image quality assessment and body position detection model processes the input CT or MRI image, and generates corresponding quality score prediction results and / or radiation dose / magnetic field intensity prediction results, as well as body position loss ratio prediction results according to the output prompt information;

[0131] Step 5) outputting the generated quality score prediction result and / or radiation dose / magnetic field strength prediction result, and the body position loss ratio prediction result.

[0132] Next, we evaluate the results of the proposed method to verify the effectiveness of the method. Our idea is that this method can be used as an effective tool for evaluating medical image quality and body position detection.

[0133] In order to measure the consistency between the model verification results and subjective evaluation, we used three indicators that can verify the evaluation results: SRCC (Spearman rank correlation coefficient), PLCC (Pearson linear correlation coefficient) and RMSE (root mean square error). They are also the most commonly used performance evaluation methods in image quality assessment methods.

[0134] 1. SRCC uses linear correlation analysis of the ranks of two target arrays to quantify the monotonicity of image quality assessment prediction. The value range is 0~1. When the performance value is 1, the two sets of data are consistent. The calculation formula of SRCC is as follows:

[0135] (2);

[0136] In formula (2), N represents the number of samples, d i It represents the difference between the subjective score ranking and the objective score ranking of image i.

[0137] 2. PLCC evaluates the accuracy of the image quality assessment model by calculating the mean opinion score MOS and the linear correlation with the objective score after regression. The value range is -1 to 1. When the PLCC value is zero, the two sets of data have no relationship at all; on the other hand, a PLCC value of 1 or -1 indicates a complete positive or negative correlation. The function for calculating the nonlinear regression target score is as follows:

[0138] (3);

[0139] In formula (3), Q is the original objective quality score, β is the fitting parameter, and p is the objective score after regression.

[0140] The formula for calculating PLCC is as follows:

[0141] (4);

[0142] In formula (4), s i represents the MOS of image i, p i represents the objective score of image i, represents the average subjective rating and, represents the average objective rating.

[0143] 3. RMSE evaluates the consistency of the image quality assessment model prediction by calculating the absolute error between the objective score and the subjective score. Values ​​closer to 0 indicate better model performance. The formula for calculating RMSE is as follows:

[0144] (5);

[0145] In formula (5), s i represents the MOS of image i, p i Represents the objective score of image i.

[0146] In order to verify the effectiveness of our proposed method, we first adjusted some network layers and training parameters of the traditional CNN to adapt to our dataset. Secondly, we used the improved network to train the model. Then, we tested the trained model separately on six sub-datasets (lung window dataset, soft tissue window dataset, CT mixed dataset, brain dataset, breast dataset, and MRI mixed dataset). The test results of each dataset are shown in Table 1 below:

[0147] Table 1

[0148]

[0149] The experimental results show that the model proposed in the present invention has good test results in various sub-datasets and the model test correlation is high.

[0150] Furthermore, in order to verify the advantages of the model of the present invention compared with other neural network models and other image quality assessment methods, we conducted a comparative experiment. First, we compared the experimental results of the model of the present invention with those of other neural network models. The experimental results and the ranking of the model of the present invention are shown in Table 2 below:

[0151] Table 2

[0152]

[0153] For the body position detection task, we used the proposed model to compare with two other classic convolutional neural networks and obtained the best results. The experimental results and the ranking of the proposed model are shown in Table 3 below:

[0154] Table 3

[0155]

[0156] The experimental results show that the model of the present invention achieves the best results in all indicators when compared with existing models and methods. The model of the present invention can automatically perform quality assessment and position detection on medical images of multiple modes and anatomical parts. This will be the first universal model for quality assessment and position detection of medical images of different modalities.

[0157] Most existing medical image quality assessment (IQA) methods are trained on two-dimensional synthetic data. To further verify the effectiveness of the proposed model, we retrained the network using a publicly available synthetic dataset. The dataset consists of 1,000 abdominal soft tissue window images labeled by five professional radiologists. In order to accept two-dimensional images as input, the maxpool layer in the proposed model is adjusted synchronously with the network depth modification. For effective comparison, we selected existing medical image IQA methods that were trained using the same type of images. As shown in Table 4, the experimental results show that the proposed model is also competitive in terms of manually annotated synthetic two-dimensional datasets.

[0158] Table 4

[0159]

[0160] By performing image quality assessment and body position detection on medical images of different modalities, it helps to ensure key features of medical images such as clarity, contrast and spatial resolution, and evaluate image integrity through body position, thereby improving the efficiency of doctors' reading. This is crucial for early diagnosis and treatment planning of diseases. At the same time, the model and method of the present invention provide effective assistance for the fusion, registration and segmentation of cross-modal medical images, ensuring the consistency and accuracy of various tasks in cross-modal medical images.

[0161] The present invention also provides a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus, and the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned cross-modal medical image quality assessment and body position detection method.

[0162] The present invention also provides a computer-readable storage medium, in which at least one executable instruction is stored, and the executable instruction enables a processor to perform operations corresponding to the above-mentioned cross-modal medical image quality assessment and body position detection method.

[0163] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A cross-modal medical image quality assessment and body position detection model, characterized in that: It includes an information encoding module, a hybrid convolution model using super convolution, and a Swin-transformer model; among them, The information encoding module is responsible for generating corresponding medical image type codes according to the modality information, part information and type information of the collected CT or MRI image data set, so as to serve as guidance information for training the hybrid convolutional model and the Swin-transformer model; The hybrid convolutional model is responsible for using the collected CT or MRI image data set and the corresponding medical image type code to perform regression training on predicting the quality scores and radiation dose / magnetic field strength scores of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, automatically predicting and generating the quality score and / or radiation dose / magnetic field strength score of the CT or MRI image to be predicted; The Swin-transformer model is responsible for using the collected CT or MRI image data set and the corresponding medical image type coding to perform regression training to predict the body position missing ratio of medical images of different modalities, and is responsible for processing the input CT or MRI image to be predicted, predicting and generating the body position missing ratio of the CT or MRI image to be predicted.

2. The cross-modality medical image quality assessment and body position detection model according to claim 1, characterized in that: The architecture of the hybrid convolutional model consists of a first feature extraction module, an output prompt module and a first fully connected module; The first feature extraction module is responsible for extracting features from the input CT or MRI image; The first feature extraction module includes 5 feature extraction units arranged in sequence, each of which is composed of a convolution layer, a batch normalization layer, a linear activation layer and a maximum pooling layer arranged in sequence; wherein the convolution layers in the first three feature extraction units are ordinary convolution layers, and the convolution layers in the last two feature extraction units are super convolution layers; The output prompt module is responsible for generating corresponding output prompt information according to the prediction result type selected by the user, and splicing the output prompt information with the feature vector generated by the first feature extraction module to generate a feature with prompt information; The first fully connected module is responsible for predicting and generating a quality score and / or a radiation dose / magnetic field strength score of an input CT or MRI image according to the feature with prompt information generated by the output prompt module; The first fully connected module includes three linear layers and one softmax layer, wherein a dropout layer and a linear activation layer are added after the first two linear layers.

3. The cross-modality medical image quality assessment and body position detection model according to claim 1, characterized in that: The architecture of the Swin-transformer model consists of a second feature extraction module and a second fully connected module; The second feature extraction module is responsible for extracting features from the input CT or MRI image; The second feature extraction module includes a patch partitioning unit and four sequentially arranged stage units, each of which is composed of a patch merging layer and a Swin-transformer module; The second fully connected module is responsible for predicting and generating the body rotation angle and body position missing status of the input CT or MRI image according to the features extracted by the second feature extraction module; The second fully connected module includes three linear layers and one softmax layer, wherein a dropout layer and a linear activation layer are added after the first two linear layers.

4. A method for training a cross-modal medical image quality assessment and body position detection model as claimed in any one of claims 1 to 3, characterized in that: The following steps are involved: Step 1) Collect CT images with different radiation doses and MRI images with different magnetic field strengths to construct a cross-modal medical quality assessment dataset; Step 2) Collect a certain number of CT images and MRI images and simulate different body position missing and angle errors to construct a body position detection dataset; Step 3) the information encoding module respectively obtains the modality information, location information and type information of each medical image in the cross-modality medical quality assessment dataset and the body position detection dataset, and converts the information into a medical image type code corresponding to each medical image; Step 4) inputting the cross-modal medical quality assessment dataset into the hybrid convolutional model, and adding the medical image type code corresponding to the cross-modal medical quality assessment dataset as guidance information into the hybrid convolutional model, and performing quality score and / or radiation dose / magnetic field strength score regression training on the hybrid convolutional model, so that the hybrid convolutional model can automatically predict the quality scores and / or radiation dose / magnetic field strength scores of medical images of different modalities; Step 5) inputting the body position detection data set into the Swin-transformer model, and adding the medical image type code corresponding to the body position detection data set as guidance information into the Swin-transformer model, and performing body position missing ratio regression training on the Swin-transformer model, so that the Swin-transformer model can automatically predict the body position missing ratio of medical images of different modalities; Step 6) Optimize the model parameters of the hybrid convolutional model and the Swin-transformer model using a mean square error loss function.

5. The training method according to claim 4, characterized in that: The cross-modal medical quality assessment data set includes CT data consisting of CT images with different radiation doses and MRI data consisting of MRI images with different magnetic field strengths; wherein, The CT data is composed of a certain number of CT image groups, each of which contains at least four CT images with different radiation doses, and each of the CT images is labeled with the radiation dose corresponding to the image; The MRI data consists of a certain number of MRI image groups, each of which contains at least two MRI images with different magnetic field strengths, and each of the MRI images is annotated with the corresponding magnetic field strength as a training label.

6. The training method according to claim 4, characterized in that: The method for simulating the treatment of different body position loss and angle malalignment is: A certain number of CT images are collected, and they are randomly rotated within a certain angle range and randomly translated within a certain pixel range to obtain simulated body position CT images of different body positions, and the volume calculation missing ratio of the anatomical structure before and after simulation of each simulated body position CT image is calculated. Each simulated body position CT image is annotated with the calculated volume calculation missing ratio as a training label.

7. The training method according to claim 4, characterized in that: The information encoding module uses corresponding single-hot encoding to respectively represent the modality information, part information and type information of each medical image in the cross-modal medical quality assessment dataset and the body position detection dataset, so as to facilitate the training of the hybrid convolutional model and the Swin-transformer model.

8. The training method according to claim 4, characterized in that: When training the hybrid convolutional model, the medical image type encoding corresponding to the cross-modal medical quality assessment dataset is first used as guidance information and added to the super-convolutional layer of the first feature extraction module of the hybrid convolutional model. Then, the first feature extraction module in the hybrid convolutional model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the user randomly determines that one or two types of prediction results of the quality score and the radiation dose / magnetic field strength score need to be output. Then, the output prompt module in the hybrid convolutional model generates corresponding output prompt information according to the prediction result type selected by the user, and splices the output prompt information with the feature vector generated by the first feature extraction module to generate features with prompt information. Finally, the first fully connected module in the hybrid convolutional model learns to predict the quality score and / or radiation dose / magnetic field strength score of medical images of different modalities according to the features with prompt information and the corresponding quality score training labels and / or radiation dose / magnetic field strength training labels, thereby completing the corresponding quality and / or radiation dose / magnetic field strength regression training.

9. The training method according to claim 8, characterized in that: When training the Swin-transformer model, the medical image type code corresponding to the body position detection data set is first used as guidance information and directly added into the second feature extraction module of the Swin-transformer model. Then, the second feature extraction module in the Swin-transformer model learns to extract features of medical images of different modalities under the guidance of the guidance information. Then, the second fully connected module in the Swin-transformer model learns to predict the volume calculation missing ratio of medical images of different modalities based on the features and the volume calculation missing ratio as training labels, thereby completing the corresponding body position missing ratio regression training.

10. A cross-modality medical image quality assessment and body position detection method, characterized in that: include: Step 1) obtaining the CT or MRI image to be predicted; Step 2) inputting the acquired CT or MRI image to be predicted into the cross-modal medical image quality assessment and body position detection model as described in any one of claims 1 to 3 or the cross-modal medical image quality assessment and body position detection model trained by the training method as described in any one of claims 4 to 9; Step 3) The user sets the type of prediction result to be output as a quality score or a radiation dose / magnetic field strength score or both, and generates corresponding output prompt information accordingly; Step 4) the cross-modality medical image quality assessment and body position detection model processes the input CT or MRI image, and generates corresponding quality score prediction results and / or radiation dose / magnetic field intensity prediction results, as well as body position loss ratio prediction results according to the output prompt information; Step 5) outputting the generated quality score prediction result and / or radiation dose / magnetic field strength prediction result, and the body position loss ratio prediction result.

Images