An intra-frame weighted medical image quality control method and system with class-specific attention

Abstract

The application discloses an intra-frame weighting medical image quality control method and system, and relates to the technical field of medical images, which solves the technical problem of weak recognition ability of the existing medical image processing method in dealing with 3D medical image processing, and low sensitivity to subtle artifacts or body position deviation, and improves the recognition ability of the model. The method comprises the following steps: acquiring medical image data, wherein the medical image data comprises 3D medical images and corresponding evaluation systems; performing data labeling on the 3D medical images based on the evaluation systems to obtain a plurality of sequence data; dividing the plurality of sequence data into a training set, a verification set and a test set; and training a medical image quality control model based on the training set, the verification set and the test set. The training process comprises data preprocessing, spatial feature extraction, time sequence enhanced feature extraction, class-guided intra-frame weighting score calculation, model optimization and class activation map generation. The method enhances the recognition ability of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical imaging, and in particular to an intra-frame weighted medical image quality control method and system with specific attention. BACKGROUND

[0002] Medical image quality control is an important part of modern medical imaging clinical work. It provides key quality control basis for medical imaging through automatic analysis of CT, MRI, X-ray and other images, and lays a foundation for subsequent high-quality diagnosis. With the development of artificial intelligence technology, medical image quality control methods have experienced an evolution from traditional manual feature extraction to deep learning end-to-end learning. Early methods rely heavily on hand-designed feature descriptors (such as HOG, SIFT) and machine learning classifiers (such as SVM), which have limited feature representation ability and weak generalization performance.

[0003] In recent years, the introduction of deep learning technology, especially convolutional neural networks (CNN) and Transformer architecture, has significantly improved the accuracy and robustness of medical image analysis. CNN can automatically learn local texture and shape features of images through local receptive fields and hierarchical feature extraction mechanisms. Transformer achieves global context modeling through self-attention mechanism (Self-Attention), overcoming the limitations of limited receptive fields of CNN.

[0004] However, the current technology still has significant challenges in the face of three-dimensional medical image data. Medical images are essentially continuous data in three-dimensional space (such as axial, sagittal, and coronal slice sequences of CT), and usually contain a time dimension (such as four-dimensional data of dynamic contrast-enhanced MRI). Most existing methods decompose 3D data into 2D slices for processing or directly apply 3D deep learning models with extremely high computational complexity; the inherent defects of 2D image classification methods in 3D medical image analysis; 2D convolutional neural networks (such as ResNet, VGG) perform well in natural image classification, but they have problems such as spatial information collapse, loss of temporal dependence, and annotation ambiguity and low efficiency when processing 3D medical images; and SWIN Transformer and its 3D variants (Video SWIN Transformer) are introduced to solve the long-range dependence problem, but they face several problems in medical image applications, such as computational complexity explosion, local overfitting, and data starvation, especially the global attention mechanism of Transformer that over-smoothes local features and is difficult to capture subtle artifacts in medical images; In particular, the artifact identification task (such as metal artifacts, motion artifacts, radio frequency artifacts, etc.) commonly found in medical images, existing methods may result in loss of spatial context information, low computational efficiency, and missed detection of rare pathological features, thus cannot effectively identify, therefore a class-specific attention-based intra-frame weighted medical image quality control method and system are needed. SUMMARY

[0005] The present application provides a class-specific attention-based intra-frame weighted medical image quality control method and system, which solves the technical problem of weak discrimination ability and low sensitivity to subtle artifacts or body position deviation in existing medical image processing methods when dealing with 3D medical image processing.

[0006] To achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, a neural network-based intra-frame weighted medical image quality control method is provided, comprising:

[0008] Obtaining medical image data, the medical image data including 3D medical images and their corresponding evaluation system; the 3D medical images are CT, MRI, or X-ray scan 3D medical images, such as chest CT images; the evaluation system includes quality control standards, class labels, and annotation labels corresponding to the 3D medical images;

[0009] Data annotation of 3D medical images based on the evaluation system to obtain a plurality of sequence data; dividing the plurality of sequence data into a training set, a validation set, and a test set;

[0010] The medical image quality control model is trained based on the training set, the verification set and the test set; the training process includes data preprocessing, spatial feature extraction, time sequence enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization and class activation map generation.

[0011] Based on the above technical solution, in the intra-frame weighted medical image quality control method and system of the application, in view of the lack of attention to space in the quality control process of CT and MR data in the 3D image algorithm, the application obtains medical image data, which includes 3D medical image and its corresponding evaluation system; the 3D medical image is data-labeled according to the evaluation system to obtain a plurality of sequence data; the plurality of sequence data is divided into a training set, a verification set and a test set; the medical image quality control model is trained according to the training set, the verification set and the test set; the training process includes data preprocessing, spatial feature extraction, time sequence enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization and class activation map generation; the class-guided intra-frame weighted score calculation includes calculating the intra-frame weighted score of each spatial position; the response intensity of each spatial position to a specific quality control category is calculated to generate class-specific weighted features, thereby highlighting the area in each frame that is most discriminative for quality control judgment, and the visualization of 3D medical image quality control is realized through frame perception ability. The key anatomical region is effectively focused, and the sensitivity of the model to subtle artifacts or body position deviation is improved.

[0012] At the same time, through the fusion of the spatial feature extraction of CNN, the time sequence modeling of RNN and the region focusing ability of the class-guided intra-frame weighting mechanism, multi-dimensional, accurate and real-time quality evaluation of medical images is realized, and through frame-level evidence extraction, the frame number and heat map that can be explained are provided while the quality control label is given; not only can the quality control efficiency be significantly improved and the repeated scanning rate be reduced, but also various image types and devices can be adapted.

[0013] In combination with the above first aspect, in a possible implementation manner, the data labeling of the 3D medical image based on the evaluation system to obtain a plurality of sequence data includes:

[0014] The 3D medical image is frame-extracted to obtain a plurality of frame images, and the frame images are integrated according to their corresponding frame order to generate sequence data; the sequence data is data-labeled to obtain labeled sequence data; the data labeling includes class label labeling and labeled label labeling; the labeled sequence data includes a plurality of frame images and class labels of each spatial position; and the frame order is the chronological order of each frame image in the 3D medical image.

[0015] In combination with the above first aspect, in a possible implementation manner, the data preprocessing includes:

[0016] Extract each frame image in the sequence data, convert each frame image into RGB mode, and sort them according to the original frame order to obtain the serialized data.

[0017] In combination with the first aspect, in a possible implementation manner, the spatial feature extraction comprises:

[0018] Extracting the frame image in the serialized data, performing spatial feature extraction on the frame image using a convolutional neural network to obtain a spatial feature map of the frame image, and flattening the spatial feature map according to spatial positions to obtain spatial features corresponding to each spatial position.

[0019] In combination with the first aspect, in a possible implementation manner, the temporal enhancement feature extraction comprises:

[0020] Obtaining the spatial feature corresponding to each spatial position in each frame image, performing temporal modeling on the spatial features of the same spatial position in different frame images using a recurrent neural network to obtain a temporal enhancement feature of the corresponding spatial position, and sequentially obtaining the temporal enhancement features of each spatial position. Specifically, to capture the temporal dependence between video frames, a recurrent neural network RNN such as LSTM, GRU, and Mamba can be introduced to perform temporal modeling on the spatial feature sequence of all frames, output the temporal enhancement feature of each frame, and a calculation formula of the temporal enhancement feature is as follows:

[0021]

[0022] Wherein, is the temporal enhancement feature of the jth spatial position in the tth frame image; is the temporal enhancement feature of the jth spatial position in the (t-1) th frame image; is the spatial feature of the jth spatial position in the tth frame image;

[0023] In combination with the first aspect, in a possible implementation manner, the category-guided intra-frame weighting score calculation comprises the following steps:

[0024] Setting a target category, obtaining the temporal enhancement features of each spatial position of the frame image, calculating the intra-frame weighting score of the target category at each spatial position of the frame image based on each temporal enhancement feature,

[0025] ;

[0026] Wherein, is the intra-frame weighting score of the target category i at the jth spatial position in the tth frame image, that is, the probability of the occurrence of the target category i at the jth spatial position in the tth frame image, and ∈ [0, 1] is used to measure the importance of different spatial positions to class recognition; is the classifier vector corresponding to the target class i, and ∈ B is a set temperature parameter used to control the sharpening and smoothing degree, which can be fixed during training; is the time series enhanced feature corresponding to the jth spatial position of the tth frame image; is the time series enhanced feature corresponding to the kth spatial position of the tth frame image, k ∈ [1, hw], and hw is the total number of spatial positions in the feature map.

[0027] In combination with the first aspect, in a possible implementation manner, the model optimization comprises:

[0028] obtaining the frame-in-weighted scores of the target class at all spatial positions of each frame image and the time series enhanced features of all spatial positions of each frame image; and generating a classification score of the target class based on the frame-in-weighted scores and the time series enhanced features of all spatial positions of each frame image.

[0029] determining a classification result based on the classification score, obtaining a set label, and calculating a loss value based on the classification result and the set label;

[0030] determining whether the loss value is less than a set threshold, and if yes, completing the model training; otherwise, adjusting related parameters and repeating the training.

[0031] In combination with the first aspect, in a possible implementation manner, the generating of the classification score of the target class based on the frame-in-weighted scores and the time series enhanced features of all spatial positions of each frame image comprises:

[0032] obtaining the frame-in-weighted scores corresponding to all spatial positions in the frame image, and performing weighted summation on the time series enhanced features corresponding to each spatial position based on the frame-in-weighted scores corresponding to each spatial position to obtain frame-level class-specific features corresponding to the target class;

[0033] performing time series fusion on the frame-level class-specific features corresponding to all frame images of the target type to obtain global class-specific features of the target type;

[0034] performing global average pooling on the time series enhanced features of the spatial positions in each frame image to obtain background global features;

[0035] fusing the global class-specific features and the background global features through a residual level to obtain final features of the target class;

[0036] performing inner product operation on the final features and the classification vector of the target class to obtain a class score corresponding to the target class.

[0037] With the first aspect above, in a possible implementation, the class activation map is generated, comprising:

[0038] extracting frame-level class-specific features corresponding to each frame image; through the formula:

[0039] ;

[0040] the response intensity of the target class i corresponding to the tth frame image is calculated ; the greater the value of the response intensity, the more relevant the frame image contains to the target class i; wherein, is the frame-level class-specific feature sequence composed of the frame-level class-specific features of the target class i in the tth frame image;

[0041] ;

[0042] select the frame image with the largest response value as the key frame image corresponding to the target class; specifically, through the formula:

[0043] ;

[0044] select the key frame image, wherein t is the number corresponding to the key frame image;

[0045] obtain the spatial feature map corresponding to the key frame image; recalculate the spatial weight map with the class vector; bilinearly interpolate to the original resolution, assign pseudo-color, and superimpose on the original image to obtain the class activation map.

[0046] The second aspect of the present application provides a class-specific attention intra-frame weighted medical image quality control device, comprising: a processor and a storage medium; the storage medium comprises instructions, and the processor is used to run the instructions to realize the method described in the first aspect and any possible implementation manner of the first aspect. The class-specific attention intra-frame weighted medical image quality control device can be an electronic device, or a chip in an electronic device.

[0047] The third aspect of the present application provides a class-specific attention intra-frame weighted medical image quality control system, comprising: a data acquisition module, a data division module and a model training module;

[0048] The data acquisition module is used to acquire a plurality of medical image data;

[0049] The data division module is used to obtain a plurality of sequence data by data labeling on 3D medical images based on an evaluation system; and divide the plurality of sequence data into a training set, a verification set and a test set;

[0050] ​The model training module trains the medical image quality control model based on the training set, the verification set and the test set; the training process includes data preprocessing, spatial feature extraction, time series enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization and class activation map generation.

[0051] In a fourth aspect, the present application provides a computer-readable storage medium having instructions stored therein, which when executed on the class-specific attention intra-frame weighted medical image quality control device, cause the class-specific attention intra-frame weighted medical image quality control device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0052] In a fifth aspect, the present application provides a computer program product comprising instructions, which when executed on the class-specific attention intra-frame weighted medical image quality control device, cause the class-specific attention intra-frame weighted medical image quality control device to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0053] The present application provides a class-specific attention intra-frame weighted medical image quality control method and system, which can obtain medical image data, the medical image data including 3D medical images and their corresponding evaluation system; data labeling is performed on the 3D medical images according to the evaluation system to obtain a plurality of sequence data; the plurality of sequence data are divided into a training set, a verification set and a test set; a medical image quality control model is trained according to the training set, the verification set and the test set; the training process includes data preprocessing, spatial feature extraction, time series enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization and class activation map generation; the class-guided intra-frame weighted score calculation includes calculating the intra-frame weighted score of each spatial position; the response intensity of each spatial position to a specific quality control category is calculated to generate class-specific weighted features, thereby highlighting the area in each frame that is most discriminative for quality control judgment, and the visualization of 3D medical image quality control is realized through frame receptive ability; the key anatomical region is effectively focused, and the sensitivity of the model to subtle artifacts or body position deviation is improved.

[0054] At the same time, through the fusion of the spatial feature extraction of CNN, the time series modeling of RNN and the region focusing ability of the class-guided intra-frame weighting mechanism, multi-dimensional, precise and real-time quality evaluation of medical images is realized, and through frame-level evidence extraction, the frame number and the heat map that are interpretable are provided while the quality control label is given; not only can the quality control efficiency be significantly improved and the repeated scanning rate be reduced, but also various image types and devices can be adapted.

[0055] It should be understood that the description of technical features, technical solutions, advantages or similar language in this application does not imply that all features and advantages can be achieved in any single embodiment. On the contrary, it can be understood that the description of a feature or advantage means that the specific technical feature, technical solution or advantage is included in at least one embodiment. Therefore, the description of technical features, technical solutions or advantages in this specification does not necessarily refer to the same embodiment. Further, the technical features, technical solutions and advantages described in this embodiment can be combined in any appropriate manner. Those skilled in the art will understand that the embodiments can be implemented without one or more specific technical features, technical solutions or advantages of a particular embodiment. In other embodiments, additional technical features and advantages can be identified in specific embodiments that do not embody all embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor.

[0057] Figure 1 The flowchart of the medical image quality control method in the present application is shown.

[0058] Figure 2 The flowchart of the medical image quality control method in the present application is shown.

[0059] Figure 3 The module connection diagram of the medical image quality control system in the present application is shown. DETAILED DESCRIPTION

[0060] The technical solutions of the present application will be described below in conjunction with the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0061] Please refer to Figures 1-2 The first aspect embodiment of the present application provides a neural network-based intra-frame weighted medical image quality control method, comprising:

[0062] Obtaining medical image data, the medical image data including 3D medical image and its corresponding evaluation system;

[0063] 3D medical images are 3D medical images such as chest CT images, X-ray images, etc. of CT, MRI or X-ray scanning; the evaluation system includes quality control standards, category labels and annotation labels corresponding to the 3D medical images used, etc.

[0064] Based on the evaluation system, a plurality of sequence data are obtained by data annotation of the 3D medical images; the plurality of sequence data are divided into a training set, a validation set and a test set;

[0065] The medical image quality control model is trained based on the training set, the validation set and the test set; the training process includes data preprocessing, spatial feature extraction, time series enhanced feature extraction, category guided intra-frame weighted score calculation, model optimization and class activation map generation; the medical image quality control model is used for quality control of 3D medical image data, and the medical image quality control model is a model for quality control of images composed of a convolutional neural network model and a recurrent neural network model in a set order, wherein the recurrent neural network model can be replaced by an LSTM, a GRU and a Mamba network model with the same function.

[0066] It can be understood that quality control is a kind of quality control problem classification based on images themselves, according to image professional knowledge and related quality control standards, to judge which possible quality problems exist in the images; the medical image quality control model is based on the summary of related quality control standards and experience of image experts into expressible knowledge, and sets labels for images according to the knowledge, and then classifies the images according to the labels; the method inputs the sequence images observable by human eyes, which is closer to the actual attention of experts, and is neither an independent image nor an untreated volume data; it is different from traditional 2D image input and 3D voxel data.

[0067] Based on the above technical solution, in the intra-frame weighted medical image quality control method and system with class-specific attention provided in the application, medical image data is obtained, the medical image data includes 3D medical images and their corresponding evaluation system; a plurality of sequence data are obtained by data annotation of the 3D medical images according to the evaluation system; the plurality of sequence data are divided into a training set, a validation set and a test set; the medical image quality control model is trained according to the training set, the validation set and the test set; the training process includes data preprocessing, spatial feature extraction, time series enhanced feature extraction, category guided intra-frame weighted score calculation, model optimization and class activation map generation; the category guided intra-frame weighted score calculation includes calculating the intra-frame weighted score of each spatial position; the response intensity of each spatial position to a specific quality control category is calculated to generate class-specific weighted features, so as to highlight the area in each frame that is most discriminative for quality control judgment, and realize the visualization of 3D medical image quality control through frame perception ability; effectively focus on key anatomical regions, and improve the sensitivity of the model to subtle artifacts or body position deviation.

[0068] At the same time, through the fusion of the spatial feature extraction of the CNN, the time sequence modeling of the RNN and the region focusing ability of the class-guided intra-frame weighting mechanism, multi-dimensional, accurate and real-time quality evaluation of medical images is realized, and through frame-level evidence extraction, the quality control label is given, and the interpretable frame number and heat map are provided; not only can the quality control efficiency be significantly improved and the repeated scanning rate be reduced, but also can be well applied to various image types and devices.

[0069] In a possible implementation, the 3D medical image is data-labeled based on the evaluation system to obtain a plurality of sequence data, including: frame extraction of the 3D medical image to obtain a plurality of frame images, and integration of the frame images according to their corresponding frame order to generate sequence data; data labeling of the sequence data to obtain labeled sequence data; the data labeling includes labeling of a class label and labeling of a labeled label; the labeled sequence data includes a plurality of frame images and a class of each spatial position; the frame order refers to the order followed by each slice image in the original scanning and acquisition after slicing the 3D medical image data along a specific spatial direction, such as an axial direction, a sagittal direction or a coronal direction;

[0070] Specifically, the embodiment refers to the chest CT image quality control guideline, formulates an evaluation system including four categories of quality control standards, and the corresponding class labels include evaluation of scanning range, image centering, scanning position and image artifact; each standard is provided with a specific evaluation subclass, i.e., a labeled label.

[0071] The embodiment sets 15 labeled labels; Table 1 is a table of corresponding quality control standards, class labels and labeled labels in the evaluation system of the embodiment, which details the quality control standards and the corresponding labeled labels.

[0072] The data labeling in the embodiment is composed of a labeling team of more than three radiologists, each tomographic sequence of each image is independently labeled, and the labeling result is finally determined through multiple rounds of consistency; after the labeling is completed, the data set can be divided according to the needs, for example: the data set is randomly divided into 70% training set, 10% validation set and 20% test set.

[0073] Table 1: Table of corresponding quality control standards, class labels and labeled labels

[0074]

[0075] In a possible implementation, the data preprocessing comprises: extracting each frame image in the sequence data, converting each frame image into an RGB mode, and sorting the frame images according to the original frame order to obtain the serialized data. Specifically, the input 3D sequence data is read through DICOM data (medical digital imaging and communication data), which is CT image data in this embodiment, to simulate the visual effect of a doctor directly watching on a PACS (picture archiving and communication system) workstation. The DICOM data is converted into visual RGB three-channel images according to the window width and window level values recorded in the Tag values in the DICOM, and the sequence data is sorted according to the Instance Number to be combined into directional serialized data.

[0076] ;

[0077] wherein T is the number of frames; H and W are spatial resolutions; V is the extracted visual feature; R is that the feature belongs to a real number space; and C is the number of channels, such as three channels in RGB.

[0078] In a possible implementation, the spatial feature extraction comprises: extracting a frame image in the serialized data, and using a convolutional neural network to perform spatial feature extraction on the frame image to obtain a spatial feature map of the frame image; and flattening the spatial feature map according to spatial positions to obtain spatial features corresponding to each spatial position; specifically, the image features are extracted using a CNN, and for each frame image in the 3D image data, a spatial feature of the frame is extracted using a convolutional neural network such as a classic model of ResNet to obtain a feature map ; wherein is the feature map corresponding to the tth frame image; d is a feature dimension, such as 2048 under Resnet50; and h and w are spatial dimensions of the feature map, such as 7x7 under Resnet50.

[0079] The above three-dimensional feature map is flattened according to spatial positions to obtain a feature set of all spatial positions under the frame; the flattened feature is

[0080] ;

[0081] wherein is the spatial feature corresponding to the first spatial position in the tth frame image; and hw is the total number of spatial positions in the feature map. is the feature set under the d feature dimension.

[0082] In a possible implementation, the time sequence enhancement feature extraction includes: obtaining spatial features corresponding to each spatial position in each frame image, using a recurrent neural network to model the spatial features of the same spatial position in different frame images in time sequence, and obtaining time sequence enhancement features of the corresponding spatial position; and sequentially obtaining the time sequence enhancement features of each spatial position. Specifically, to capture the time sequence dependency between video frames, a recurrent neural network RNN such as LSTM, GRU, and Mamba can be introduced to model the spatial feature sequence of all frames in time sequence, and output time sequence enhancement features of each frame. A calculation formula of the time sequence enhancement features is as follows:

[0083]

[0084] wherein, is the time sequence enhancement feature of the jth spatial position in the tth frame image; is the time sequence enhancement feature of the jth spatial position in the (t-1) th frame image; is the spatial feature of the jth spatial position in the tth frame image. It can be understood that the recurrent neural network used to extract the time sequence enhancement features can be a specific variant of the LSTM, GRU, and the like in addition to the basic RNN.

[0085] In this embodiment, the selective state space model Mamba is selected as the time sequence modeler, which can keep the linear complexity constant, and the parameter quantity and the calculation quantity are better than those of other RNN modules with the same hidden dimension.

[0086] In a possible implementation, the class-guided intra-frame weighted score calculation includes the following steps: setting a target class, obtaining time sequence enhancement features of each spatial position of a frame image, calculating intra-frame weighted scores of the target class at each spatial position of the frame image based on each time sequence enhancement feature,

[0087] ;

[0088] wherein, is the intra-frame weighted score of the target class i at the jth spatial position in the tth frame image, that is, the probability of the target class i appearing at the jth spatial position in the tth frame image, and ∈ [0, 1], which is used to measure the importance of different spatial positions to class recognition; is a classifier vector corresponding to the target class i, and ∈ ; B is a set temperature parameter, which is used to control the sharpening and smoothing degree, and can be fixedly set during training; is the time sequence enhancement feature corresponding to the jth spatial position of the tth frame image; is the time sequence enhanced feature corresponding to the kth spatial position of the tth frame image, k∈[1, hw], and hw is the total number of spatial positions in the feature map.

[0089] In a possible implementation, the model optimization comprises: obtaining frame-in weighted scores of a target category at all spatial positions of each frame image and time sequence enhanced features of all spatial positions of each frame image; generating a classification score of the target category based on the frame-in weighted scores and the time sequence enhanced features of all spatial positions of each frame image;

[0090] Based on the classification score, a classification result is determined, a set of labels is obtained, and a loss value is calculated based on the classification result and the set of labels. Specifically, the loss value calculation formula is:

[0091] ;

[0092] wherein L is the loss value, N is the total number of samples in the quality control image set in the training set, is the true label of the ith sample, is the probability that the model predicts the ith sample as a positive sample.

[0093] It is determined whether the loss value is less than a set threshold. If yes, the model training is completed; otherwise, the related parameters are adjusted, and the training is repeated.

[0094] In a possible implementation, the model optimization comprises: obtaining frame-in weighted scores of a target category at all spatial positions of each frame image and time sequence enhanced features of all spatial positions of each frame image; generating a classification score of the target category based on the frame-in weighted scores and the time sequence enhanced features of all spatial positions of each frame image;

[0095] ;

[0096] wherein, is the frame-level specific feature of the target feature i in the tth frame image; is the frame-in weighted score of the target category i at the jth spatial position in the tth frame image; is the time sequence enhanced feature corresponding to the jth spatial position of the tth frame image; and j∈[1, hw];

[0097] The frame-level class-specific features corresponding to all frame images of the target type are time-series fused to obtain global class-specific features of the target type; specifically, the frame-level class-specific features of all frame images in the 3D image data, i.e., the image sequence, for class i are time-series fused to obtain global class-specific features of the entire image sequence for class i; the time-series fusion formula is:

[0098] ;

[0099] wherein, is the global class-specific feature of the target type i in the image sequence;

[0100] The time-series enhanced features of the spatial positions in each frame image are globally averaged and pooled to obtain background global features; specifically, to retain the overall global information of the video, the time-series enhanced features of all frames and all spatial positions are globally averaged and pooled to obtain class background global features; the class background global feature calculation formula is:

[0101] ;

[0102] wherein, g is the class background global feature;

[0103] The global class-specific features and the background global features are fused through a residual level to obtain the final features of the target class; specifically, the class-specific features and the class background global features are fused through a residual connection method, which retains global information and highlights key information related to the class; the residual fusion formula is:

[0104] ;

[0105] wherein, is the final feature of the target class i after fusion; is a weight parameter for adjusting the contribution degree of the class-specific feature in the fusion result, which can be learned or manually set during training;

[0106] The final features and the classification vector of the target class are inner product operated to obtain the class score corresponding to the target class; specifically, the final feature of the target class i after fusion and the classifier vector of the target class i are inner product operated to obtain the classification score of the image sequence belonging to class i:

[0107] ;

[0108] wherein, is the classification score of the image sequence belonging to the target class i, and the higher the score, the greater the probability that the sequence data belongs to the class; is the classifier vector of the target class i at temperature B.

[0109] In a possible implementation, the class activation map generation comprises: extracting frame-level class-specific features corresponding to each frame image; calculating, by using a formula:

[0110]

[0111] calculating a response intensity of the target class i corresponding to the tth frame image ; the greater the value of the response intensity, the more relevant the frame image is to the target class i; wherein, is a frame-level class-specific feature sequence composed of the frame-level class-specific features of the target class i in the tth frame image; and

[0112] selecting a frame image with the maximum response value as a key frame image corresponding to the target class; specifically, by using a formula:

[0113]

[0114] selecting the key frame image, wherein t is the number corresponding to the key frame image;

[0115] obtaining a spatial feature map corresponding to the key frame image; recalculating a spatial weight map by using a class vector; bilinearly interpolating to an original resolution, giving a pseudo color, and superimposing on an original image to obtain a class activation map; specifically, comprising the following steps:

[0116] Step 1: recalculating a spatial weight map by using a class vector; the spatial feature map of the key frame image A is , wherein h x w is the feature map size, c is the channel number, the class vector is , i is the target class, and the formula for calculating the spatial weight map is: , wherein is the activation value of the kth channel at the position , and is the weight of the class i to the kth channel.

[0117] Step 2: bilinearly interpolating to an original resolution; using up-sampling to convert the spatial weight map M into an original image size to obtain a high-resolution heat map . is a single-channel grayscale image, and the value range is usually [0, 1];

[0118] Step 3: giving a pseudo color; using a predefined color lookup table to convert the heat map, i.e., the grayscale image, into a pseudo color image , such as red.

[0119] ​​​​Step four: superimpose the original drawing to get the class activation map; superimpose the pseudo-color image C and the original key frame image, and the superimposed image is the class activation map.

[0120] Finally, output the number value corresponding to the key frame image and the Base64 encoding of the CAM image for the front end browser to display without plugins.

[0121] In a second aspect, the application provides an intra-frame weighted medical image quality control device with class-specific attention, which comprises a processor and a storage medium; the storage medium comprises instructions, and the processor is configured to execute the instructions to implement the method described in the first aspect and any possible implementation manner of the first aspect. The intra-frame weighted medical image quality control device with class-specific attention can be an electronic device or a chip in an electronic device.

[0122] Please refer to Figure 3 In a third aspect, the application provides an intra-frame weighted medical image quality control system with class-specific attention, which comprises a data acquisition module, a data division module, and a model training module.

[0123] The data acquisition module is configured to acquire a plurality of medical image data.

[0124] The data division module is configured to perform data labeling on the 3D medical image based on an evaluation system to obtain a plurality of sequence data; and divide the plurality of sequence data into a training set, a validation set, and a test set.

[0125] The model training module is configured to train a medical image quality control model based on the training set, the validation set, and the test set; and the training process comprises data preprocessing, spatial feature extraction, time series enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization, and class activation map generation.

[0126] In a fourth aspect, the application provides a computer-readable storage medium having instructions stored therein, which, when executed on an intra-frame weighted medical image quality control device with class-specific attention, causes the intra-frame weighted medical image quality control device with class-specific attention to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0127] In a fifth aspect, the application provides a computer program product comprising instructions, which, when executed on an intra-frame weighted medical image quality control device with class-specific attention, causes the intra-frame weighted medical image quality control device with class-specific attention to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0128] Part of the data in the above formula is the numerical value calculated by removing the dimension, and the formula is obtained by software simulation of a large amount of collected data to obtain a formula closest to the real situation; the preset parameters and the preset threshold in the formula are set by the person skilled in the art according to the actual situation or obtained by a large amount of data simulation.

[0129] Working principle of the present application:

[0130] By acquiring medical image data, the medical image data includes 3D medical image and its corresponding evaluation system; according to the evaluation system, the 3D medical image is data labeled to obtain a plurality of sequence data; the plurality of sequence data is divided into a training set, a validation set and a test set; the medical image quality control model is trained according to the training set, the validation set and the test set; the training process includes data preprocessing, spatial feature extraction, time series enhanced feature extraction, class guided intra-frame weighted score calculation, model optimization and class activation map generation; the class guided intra-frame weighted score calculation includes calculating the intra-frame weighted score of each spatial position; by calculating the response intensity of each spatial position to a specific quality control category, a class-specific weighted feature is generated, thereby highlighting the area in each frame that is most discriminative for quality control judgment, and the frame receptive ability is realized. 3D medical image quality control visualization. Effectively focus on key anatomical regions to improve the sensitivity of the model to subtle artifacts or body position deviations; at the same time, through the fusion of CNN spatial feature extraction, RNN time series modeling and the region focusing ability of the class guided intra-frame weighting mechanism, multi-dimensional, precise and real-time quality assessment of medical images is realized, and through frame-level evidence extraction, the frame number and heat map are provided while giving the quality control label; not only can significantly improve the quality control efficiency and reduce the repeated scanning rate, but also can adapt to various image types and devices.

[0131] The above embodiments are only used to illustrate the technical method of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical method of the present application can be modified or replaced equivalently without departing from the spirit and scope of the technical method of the present application.

Claims

1. An intra-frame weighted medical image quality control method of class-specific attention, characterized in that, The application relates to a medical image quality control method and device. The method comprises the following steps: acquiring medical image data, wherein the medical image data comprises 3D medical images and corresponding evaluation systems; performing data labeling on the 3D medical images based on the evaluation systems to obtain a plurality of sequence data; dividing the plurality of sequence data into a training set, a verification set and a test set; training a medical image quality control model based on the training set, the verification set and the test set; the training process comprises data preprocessing, spatial feature extraction, time sequence enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization and class activation map generation; the class-guided intra-frame weighted score calculation comprises calculating intra-frame weighted scores of each spatial position; the class-guided intra-frame weighted score calculation comprises the following steps: setting a target class, acquiring time sequence enhanced features of each spatial position of a frame image, and calculating intra-frame weighted scores of the target class at each spatial position in the frame image based on each time sequence enhanced feature; the data labeling on the 3D medical images based on the evaluation systems to obtain the plurality of sequence data comprises the following steps: extracting frames from the 3D medical images to obtain a plurality of frame images, integrating the frame images according to their corresponding frame orders to generate sequence data, and performing data labeling on the sequence data to obtain labeled sequence data; the data labeling comprises class label labeling and label label labeling. The data preprocessing comprises the following steps: extracting each frame image in the sequence data, converting each frame image into an RGB mode, and sorting the frame images according to the original frame orders to obtain serialized data. The spatial feature extraction comprises the following steps: extracting frame images in the serialized data, using a convolutional neural network to extract spatial features of the frame images to obtain spatial feature maps of the frame images, and flattening the spatial feature maps according to spatial positions to obtain spatial features corresponding to each spatial position. The time sequence enhanced feature extraction comprises the following steps: acquiring spatial features corresponding to each spatial position in each frame image, using a recurrent neural network to model spatial features of the same spatial position in different frame images in time sequence to obtain time sequence enhanced features of the corresponding spatial position, and sequentially acquiring time sequence enhanced features of each spatial position. The model optimization comprises the following steps: acquiring intra-frame weighted scores of the target class at all spatial positions of each frame image and time sequence enhanced features of all spatial positions of each frame image, generating a classification score of the target class based on the intra-frame weighted scores of all spatial positions of each frame image and the time sequence enhanced features, determining a classification result based on the classification score, acquiring a set label, calculating a loss value based on the classification result and the set label, and judging whether the loss value is less than a set threshold value; if yes, the model training is completed; otherwise, relevant parameters are adjusted, and the training is repeated. The classification score of the target class generated based on the intra-frame weighted scores of all spatial positions of each frame image and the time sequence enhanced features comprises the following steps: acquiring intra-frame weighted scores corresponding to all spatial positions in the frame image, weighting and summing time sequence enhanced features corresponding to each spatial position based on the intra-frame weighted scores corresponding to each spatial position to obtain frame-level class-specific features corresponding to the target class. ； wherein, is the intra-frame weighted score of the target class i at the jth spatial location in the tth frame image; is the classifier vector corresponding to the target class i; B is a set temperature parameter; is the temporal enhanced feature corresponding to the jth spatial location in the tth frame image; is the temporal enhanced feature corresponding to the kth spatial location in the tth frame image, k ∈ [1, hw], hw is the total number of spatial locations in the feature map.

2. The intra-frame weighted medical image quality control method of claim 1, wherein, ​ ​ 3. The intra-frame weighted medical image quality control method of claim 1, wherein, ​ ​ 4. The intra-frame weighted medical image quality control method of claim 1, wherein, ​ ​ 5. The intra-frame weighted medical image quality control method of claim 4, wherein, ​ ​ 6. The intra-frame weighted medical image quality control method of claim 1, wherein, ​ ​ ​ ​ 7. The method of claim 1, wherein the method is a class-specific attention intra-frame weighted medical image quality control method. ​ ​ The frame-level class-specific features corresponding to all frame images of the target type are time-series fused to obtain global class-specific features of the target type; The time-series enhanced features of the spatial position in each frame image are globally pooled to obtain background global features; The global class-specific features and the background global features are fused by residual magnitude to obtain final features of the target class; The final features and the classification vector of the target class are subjected to inner product operation to obtain a class score corresponding to the target class.

8. The intra-frame weighted medical image quality control method of claim 1, wherein, The class activation map generation comprises: frame-level class-specific features corresponding to each frame image are extracted by formula: ； a response intensity of a target class i corresponding to the t-th frame image is calculated ; wherein, is a frame-level class-specific feature sequence composed of frame-level class-specific features of the target class i in the t-th frame image; select the frame image with the maximum corresponding intensity value as the key frame image corresponding to the target class; obtain the spatial feature map corresponding to the key frame image; recalculate the spatial weight map with the class vector; bilinearly interpolate to the original resolution, assign pseudo-color, and superimpose on the original image to obtain the class activation map.

9. An intra-frame weighted medical image quality control system with attention-like specificity, applied to the application of the intra-frame weighted medical image quality control method with attention-like specificity according to any one of claims 1 to 8, characterized in that, It comprises: a data acquisition module, a data division module, and a model training module; the data acquisition module is used to obtain a plurality of medical image data; the data division module is used to obtain a plurality of sequence data by data labeling of 3D medical images based on an evaluation system; and the plurality of sequence data are divided into a training set, a validation set, and a test set; the model training module is used to train a medical image quality control model based on the training set, the validation set, and the test set; and the training process comprises data preprocessing, spatial feature extraction, time-series enhanced feature extraction, class-guided intra-frame weighted score calculation, model optimization, and class activation map generation.

Images