Chest DR image quality evaluation method based on feature fusion

Abstract

The invention discloses a chest DR image quality evaluation method based on feature fusion. The method comprises the following steps: firstly, collecting a chest DR image and segmenting the chest DR image to obtain a region of interest; then image omics features and deep learning features are extracted by using an image omics feature extraction network and a deep learning feature extraction network respectively; meanwhile, performing complementary contrast learning to optimize the deep learning feature extraction network in a mode of combining double-feature contrast predictive coding and contrast logarithm ratio upper limit; finally, feature fusion is carried out, and image quality grade classification is carried out through a deep learning classification head. According to the method, on one hand, the interactivity of image omics features and deep learning features is considered, comprehensive interaction between the features is realized by using a feature fusion technology, the representation capability of the fused features is improved, the prediction performance is improved, and the accuracy of quality grade evaluation is ensured; and on the other hand, mutual information between the features is maximized and minimized through a complementary contrast learning strategy, the complementarity between the features is enhanced, and the interpretability of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and image quality assessment, and specifically relates to a chest DR image quality assessment method based on feature fusion. Background Art

[0002] Chest DR image quality assessment plays an important role in ensuring diagnostic accuracy, patient safety, and standardized medical processes. For example, high-quality chest X-ray images help radiologists carefully observe anatomical structures and accurately diagnose diseases, thereby reducing the risk of misdiagnosis and missed diagnosis. Traditional chest DR image quality assessment mainly relies on the radiology quality control team to conduct subjective evaluation of images according to established standards. With the increase in the amount of shooting, this highly repeatable and high-profile evaluation work requires a lot of time and energy and cannot be carried out on a regular basis; and the evaluation process needs to comprehensively consider multiple quality standard points closely related to diagnostic information, such as photographic position, image clarity, and contrast. The evaluation results are easily affected by the experience of the evaluator, resulting in poor evaluation consistency. As science and technology continue to advance towards digitalization and intelligence, the design of objective and efficient automatic evaluation methods for chest DR images has also received increasing attention.

[0003] The current research schemes for chest DR image quality assessment can be divided into two categories according to the different feature extraction methods: one is to design manual features for quality assessment; the other is to use an end-to-end deep learning model to extract deep learning features to assess image quality. The first method often uses a two-step cascade strategy: this method usually first uses a deep learning model to obtain the region of interest in the image, and then uses the existing prior knowledge and digital image processing technology to obtain manual features to quantitatively characterize the quality problems in the chest DR image, and build a machine learning model for image quality assessment; or directly evaluate the corresponding local quality problems in the image after obtaining the manual features. The second method considers using a deep learning model to directly obtain deep learning features in the image, which are used to quantitatively describe the quality problems in the chest DR image: this method usually uses a transfer learning strategy to reduce the risk of overfitting caused by the limited number of images, and uses an end-to-end method to train the deep learning model to automatically obtain image quality assessment results. For example, in the paper "Deep Blind Chest X-Ray Image Quality Assessment with ROI-Guided Attention" by Jin J et al., a dual attention mechanism guided by the region of interest was added to the model to strengthen the consideration of important areas when evaluating image quality.

[0004] However, the above methods also have corresponding defects. On the one hand, since the design of manual features utilizes relevant clinical prior knowledge, the manual features extracted from limited aspects of the image data may not be sufficient to represent the complex visual perception of doctors when performing quality assessment; and there are many quality standard points closely related to the diagnostic quality of chest DR images, including photographic position, image clarity, contrast, artifact influence and other quality standard points closely related to diagnostic information. The first type of method only starts from the image position and layout related to the photographic position, fails to achieve sufficient assessment, and cannot be accurately used for chest DR image quality level assessment. On the other hand, since the features of the deep learning network are not pre-defined and the differences between images of different quality levels are small, obtaining highly expressive deep learning features often requires a large amount of data to train the deep learning network, so there must be enough training samples for training; in addition, compared to natural images, the annotation of medical images often requires professional knowledge, so there are certain difficulties in obtaining large-scale data; at the same time, deep learning features use an end-to-end training method to adaptively learn the features of specific tasks and the complex relationships between them. However, this not only requires a sufficient number of samples for training, but also the interpretability of deep learning features is often weak, which is a significant disadvantage for medical image assessment tasks with high interpretability requirements. Summary of the invention

[0005] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide a chest DR image quality assessment method based on feature fusion, which utilizes feature fusion technology combined with the advantages of deep learning features and imaging genomics features for chest DR image quality assessment tasks, and uses a comparative learning strategy to enhance the complementarity of these two features, thereby improving the performance of the chest DR image quality assessment model.

[0006] In order to achieve the above object, the present invention provides a chest DR image quality assessment method based on feature fusion, comprising the following steps:

[0007] Collect chest DR images, and use an image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include the lung field area, the overlapping area between the scapula and the lung field, the clavicle area, and external metal artifacts;

[0008] The region of interest and chest DR images were input into the radiomics feature extraction network and the deep learning feature extraction network to extract radiomics features and deep learning features respectively;

[0009] The dual-feature contrast prediction coding and contrast log-ratio upper limit are combined to perform complementary contrast learning of radiomics features and deep learning features, and optimize the deep learning feature extraction network.

[0010] The radiomics features and deep learning features are fused, and the image quality levels are classified by the deep learning classification head; the quality levels include excellent, good, fair and poor.

[0011] As a preferred technical solution, the image segmentation network adopts a U-Net network.

[0012] As a preferred technical solution, the radiomics feature extraction network uses the PyRadiomics package to obtain radiomics features from the region of interest, including first-order statistical features, texture features, and shape features.

[0013] As a preferred technical solution, the deep learning feature extraction network includes an input layer, a preliminary extraction layer, a feature extraction layer and an output layer;

[0014] The input layer is used to stack the input region of interest and the chest DR image along the channel dimension, and obtain an input image of a set size through image preprocessing and image enhancement operations;

[0015] The preliminary feature extraction layer is a convolutional layer with a channel size of 3;

[0016] The feature extraction layer uses a ResNet network pre-trained on the ImageNet-1k dataset;

[0017] The output layer uses a fully connected layer to perform dimensionality reduction processing.

[0018] As a preferred technical solution, the extraction of deep learning features is specifically as follows:

[0019] The region of interest and the chest DR image are input to the input layer and stacked along the channel dimension, and an input image of a set size is obtained after image preprocessing and image enhancement operations;

[0020] Use the preliminary extraction layer to perform preliminary feature extraction on the input image of a set size to obtain an initial feature map that is consistent with the input size of the feature extraction layer;

[0021] The initial feature map is input into the feature extraction layer to perform deep feature extraction to obtain a one-dimensional feature vector;

[0022] The output layer is used to reduce the dimensionality of the one-dimensional feature vector to obtain deep learning features with the same dimension as the radiomics features.

[0023] As a preferred technical solution, the dual-feature contrast prediction coding uses an autoregressive model to predict the radiomics features and deep learning features, and uses the InFoNCE loss function to maximize the mutual information between similar features, specifically:

[0024] A perceptron with ReLU nonlinear activation function and two hidden layers is used to predict the radiomics features and deep learning features to obtain the corresponding prediction representation;

[0025] Constructing positive and negative example pairs of dual-feature contrast prediction coding using a supervised approach; when constructing the positive example pairs, the prediction representations of the same batch with the same quality level are considered; when constructing the negative example pairs, the prediction representations of images of the same batch with different labels from the current image are considered;

[0026] The InFoNCE loss function is used to construct the loss function of contrastive predictive coding, and contrastive learning is performed on positive and negative example pairs;

[0027] The loss function of the contrast prediction coding is expressed as:

[0028]

[0029] in, After using radiomics features to predict deep learning features, the InFoNCE loss function between the radiomics features and the predicted deep learning features; After using deep learning features to predict radiomics features, the InFoNCE loss function between the deep learning features and the predicted radiomics features; After using the deep learning feature to predict the deep learning feature, the InFoNCE loss function between the deep learning feature and the predicted deep learning feature; After the radiomics feature is used to predict the radiomics feature, the InFoNCE loss function between the radiomics feature and the predicted radiomics feature is used, and α<1.0 is the coefficient.

[0030] As a preferred technical solution, the upper limit of the contrast logarithmic ratio enhances the perception of feature differences by the deep learning feature extraction network by minimizing unnecessary association information between imaging group features with less mutual information and deep learning features;

[0031] The upper limit of the contrast log ratio optimizes the radiomics feature x h and deep learning features x d The upper limit of the mutual information between them, the objective function is expressed as:

[0032]

[0033] in, is the radiomics feature of the i-th chest DR image, is the deep learning feature of the i-th chest DR image, Based on the conditional distribution The variational approximation of is the radiomics feature of the jth chest DR image, Based on the conditional distribution The variational approximation of is , where N is the number of chest DR images.

[0034] As a preferred technical solution, the tensor fusion method is used to fuse the radiomics features and deep learning features, specifically:

[0035] Learn the linear transformation of deep learning features and radiomics features, and use the Sigmoid activation function to perform nonlinear transformation to obtain the importance score of the corresponding feature; the importance score is represented by an attention weight vector;

[0036] The deep learning features are input into the Sigmoid activation function for feature transformation, and the product of the elements in the deep learning features after feature transformation and their corresponding importance scores is used for gated representation;

[0037] The gated representation of the deep learning feature and the linearly transformed radiomics feature are expanded using the value 1, and the Kronecker product of the two is calculated, which is expressed as:

[0038]

[0039] Among them, x fusion is the Kronecker product, x d,gated is the gated representation of deep learning features, x h is the radiomics feature after linear transformation, represents the outer product;

[0040] After the Kronecker product is flattened, the dimensionality reduction is performed using a fully connected layer to obtain the fusion feature.

[0041] On the other hand, a chest DR image quality assessment system based on feature fusion is provided, which is applied to the above-mentioned chest DR image quality assessment method, including a collection and segmentation module, a feature extraction module, a network optimization module and a feature classification module;

[0042] The collection and segmentation module is used to collect chest DR images, and use an image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include lung field areas, overlapping areas between scapula and lung field, clavicle areas, and external metal artifacts;

[0043] The feature extraction module is used to input the region of interest and the chest DR image into the radiomics feature extraction network and the deep learning feature extraction network respectively to extract radiomics features and deep learning features;

[0044] The network optimization module is used to perform complementary contrast learning of radiomics features and deep learning features by combining dual-feature contrast prediction coding and contrast logarithmic ratio upper limit, and optimize the deep learning feature extraction network;

[0045] The feature classification module is used to fuse the radiomics features and the deep learning features, and classify the image quality levels through the deep learning classification head; the quality levels include excellent, good, fair and poor.

[0046] In another aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned chest DR image quality assessment method.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. Achieve more comprehensive image quality assessment

[0049] Existing chest DR image quality assessment methods either only obtain manual features to assess image quality, or only use deep learning networks to directly obtain deep learning features for end-to-end quality assessment. However, manually designed features cannot accurately describe the complex visual perception of doctors when performing quality assessment. Although deep learning network models can adaptively learn task-related features, training a high-precision model often requires more data samples, and compared with imaging genomics features, deep learning features are less interpretable. The present invention uses imaging genomics features to quantitatively describe the subjective visual perception of human experts, while considering the interactivity of deep learning features and imaging genomics features, and uses feature fusion technology to achieve comprehensive interaction between deep learning features and imaging genomics features, thereby improving the representation ability of fused features. In previous studies, the fused model can often take advantage of the advantages of the two modeling strategies to bring higher predictive performance, but no predecessors have yet combined these two methods for chest DR image quality grade assessment tasks.

[0050] 2. Enhance the complementarity of the two features and the interpretability of the model

[0051] The present invention takes into account the associations and differences between deep learning features and imaging genomics features, and designs a complementary contrast learning strategy that combines dual-feature contrast prediction coding and contrast log ratio upper limit. On the one hand, dual-feature contrast prediction coding is used to maximize the mutual information between the two features, and the model is forced to learn the correlation between the two features by forcing the two features to predict each other; on the other hand, the contrast log ratio upper limit is used to minimize the mutual information of the two features, eliminate unnecessary semantic information between the features, and exclude irrelevant details. The complementary contrast learning strategy of the present application enhances the complementarity between features by maximizing and minimizing the mutual information of the two features. Under this strategy, the learning of the deep learning feature extraction network can refer to the prior knowledge contained in the imaging genomics features, and optimize the deep learning feature extraction network by using loss callback, so that the deep learning features can be more in line with clinical standards, thereby improving the interpretability of the chest DR image quality assessment model. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is an overall flow chart of the chest DR image quality assessment method based on feature fusion in an embodiment of the present invention.

[0054] Figure 2 Schematic diagram of the framework of the deep learning feature extraction network in an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of tensor fusion in an embodiment of the present invention.

[0056] Figure 4 Schematic diagram of the structure of a chest DR image quality assessment system based on feature fusion in an embodiment of the present invention.

[0057] Figure 5 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0060] The present invention mainly focuses on the evaluation of the overall diagnostic quality of chest DR images. Referring to the literature "Jia Xiaoqian, Zhang Xiangli, Liu Zhe, et al. Application value of convolutional neural network in assisting chest digital X-ray image quality control [J]. Journal of Xi'an Jiaotong University: Medical Edition, 2019, 40(5):5.DOI:10.7652 / jdyxb201905023.", the quality level of chest DR images is divided into four levels: excellent, good, general and poor. Figure 1 As shown, the chest DR image quality assessment method based on feature fusion in this embodiment includes the following steps:

[0061] S1. Collect chest DR images and use an image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include the lung field area, the overlapping area between the scapula and the lung field, the clavicle area, and external metal artifacts.

[0062] S2. Input the region of interest and chest DR image into the radiomics feature extraction network and the deep learning feature extraction network respectively to extract radiomics features and deep learning features.

[0063] S3. Complementary comparative learning of imaging genomics features and deep learning features is performed by combining contrast prediction coding and contrast logarithmic ratio upper limit to optimize the deep learning feature extraction network.

[0064] S4. The imaging omics features and deep learning features are fused, and the image quality level is classified through the deep learning classification head.

[0065] Furthermore, the image segmentation network of the present application uses a U-Net network to segment the chest DR image to obtain four regions of interest (i.e., segmentation masks). In this embodiment, the U-Net network is constructed with reference to the paper "Ronneberger O, Fischer P, Brox TU-net: Convolutional networks for biomedical image segmentation [C]. Medical image computing and computer-assisted intervention - MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III18. Springer International Publishing, 2015: 234-241."

[0066] Furthermore, the radiomics features mainly include first-order statistical features, texture features and shape features. In the present invention, the radiomics feature extraction network uses the PyRadiomics package (version v3.0) to automatically obtain radiomics features from the lung field area, the overlapping area of ​​the scapula and lung field, the clavicle area and the in vitro metal artifact area.

[0067] Furthermore, the deep learning feature extraction network includes an input layer, a preliminary extraction layer, a feature extraction layer and an output layer; Figure 2 As shown, the input layer is used to stack the input region of interest and the chest DR image along the channel dimension, and obtain the input image of the set size after image preprocessing and image enhancement operations; the preliminary feature extraction layer is a convolutional layer with a channel size of 3; the feature extraction layer uses the ResNet network pre-trained under the ImageNet-1k dataset; the output layer uses a fully connected layer for dimensionality reduction processing. In the present invention, the ResNet network is used as the encoder network of the deep learning feature extraction network. In the chest DR image quality assessment task, the weight parameters trained under the ImageNet-1k dataset are used to initialize the ResNet network structure, and then the network parameters are fine-tuned and optimized through the chest DR image quality assessment dataset training network, and effective deep learning feature representation is quickly obtained.

[0068] Furthermore, Figure 2 The process of using the transfer learning strategy to obtain deep learning features in the present invention is shown, specifically:

[0069] When doctors evaluate the quality of chest DR images, they pay uneven attention to the entire image, and often pay more attention to areas that may affect the diagnosis of the disease. In view of this, when inputting the model, the present invention explicitly guides the model to pay attention to the key areas through the region of interest; specifically: the region of interest and the chest DR image input input layer are stacked along the channel dimension, and the set size input image is obtained after image preprocessing and image enhancement operations. In this embodiment, the set size is C×H×W=7×224×224, C represents the number of channels, and H and W represent the height and width of the set size input image respectively.

[0070] Next, before inputting the feature extraction layer, the preliminary extraction layer is used to perform preliminary feature extraction on the input image of the set size to obtain an initial feature map that is consistent with the input size of the feature extraction layer.

[0071] Subsequently, the initial feature map is input into the feature extraction layer for deep feature extraction to obtain a one-dimensional feature vector. In this application, the feature extraction layer is a ResNet network, which is a network structure composed of residual blocks containing skip connections. In the ResNet model, the gradient propagation in the training phase is more convenient, and a deep network with more network layers can be formed. In this embodiment, the ResNet network adopts the RseNet34 network. Figure 2 The network structure of ResNet34 is shown in Figure 1. The data first passes through a convolutional layer with 64 channels and a maximum pooling operation, and then is connected to multiple residual blocks. Each residual block has three convolutional layers. The first convolutional layer of the residual block is used for dimensionality increase, and the last convolutional layer is used for dimensionality reduction. After the feature map output by the last residual block is globally averaged and pooled, a one-dimensional feature vector of size 512 can be obtained.

[0072] Finally, in order to promote information balance in the fusion process and ensure the dimensional matching of features, the output layer is used to reduce the dimensionality of the one-dimensional feature vector to obtain deep learning features with the same dimensionality as the imaging omics features.

[0073] Furthermore, after feature extraction, each chest DR image can obtain a feature pair (x d ,x h )∈R D , where x d is the deep learning feature, x hIt is an imaging genomics feature; before the feature fusion is performed, the present invention also enhances the complementarity between the two features through a complementary contrast learning strategy, and the learning of the deep learning features in the strategy can refer to the prior information in the imaging genomics features, thereby enhancing the interpretability of the model. The complementarity between deep learning features and imaging genomics features means that there are associations and differences between the two features, and each has its own emphasis on the expression of information for the same task. When these two features are combined together, a more comprehensive representation can be provided than using only one of the features alone. The complementary contrastive learning strategy (Complementary Contrastive Learning Module, CCLM) proposed in the present invention combines dual-feature contrast predictive coding (Dual-CPC) and contrastive log-ratio upper bound (CLUB), which are used together for the contrastive learning of imaging genomics features and deep learning features.

[0074] Furthermore, contrastive predictive coding (CPC) (refer to the literature "Oord A vd, Li Y, Vinyals O. Representation learning with contrastive predictive coding [J]. arXiv preprint arXiv: 1807.03748, 2018.") uses an autoregressive model to predict the radiomics features and deep learning features, and uses the InFoNCE loss function to maximize the mutual information (MI) between similar features to improve the correlation between the two features and enhance the feature representation of features at different levels. Mutual information estimation aims to measure the dependency between two features, and the purpose of mutual information maximization is to promote the correlation between two features and capture global information across and within features. CPC is a self-supervised contrastive learning strategy, and the constructed positive example pairs often come from the same sample. Inspired by the research of Gao et al. (reference "Gao T, Yao X, Chen D.Simcse: Simple contrastive learning of sentence embeddings[J].arXiv preprint arXiv:2104.08821, 2021."), CPC is improved to obtain the dual-feature contrastive predictive coding (Dual-CPC) strategy of this application, which encourages the deep learning feature extraction network to learn the correlation between the two features by forcing the two features to predict each other, specifically:

[0075] First, a perceptron (MLP) with a ReLU nonlinear activation function and two hidden layers is used to predict the radiomics features and deep learning features to obtain the corresponding prediction representation:

[0076] z d =MLP(x h )∈R D , z h =MLP(x d )∈R D ,

[0077] Among them, z d is the predicted representation of the deep learning features predicted using radiomics features, x h is the radiomics feature, z h is the predicted representation of radiomics features predicted using deep learning features, x h For deep learning features.

[0078] Next, a supervised approach is used to construct the positive and negative pairs of Dual-CPC. That is, when constructing the positive pairs, the prediction representations of the same batch with the same quality level are considered; when constructing the negative pairs, the prediction representations of the same batch with different labels from the current image are considered.

[0079] Finally, the loss function is constructed. In addition to comparing the correlation between the two features, this application and In addition to the two loss functions, we also constructed and The two loss functions are internally aligned; the four loss functions are optimized using the InFoNCE loss function, and finally the loss function of contrastive predictive coding is constructed to perform contrastive learning on positive and negative example pairs. The loss function of dual-feature contrastive predictive coding is expressed as:

[0080]

[0081] in, After using radiomics features to predict deep learning features, the InFoNCE loss function between the radiomics features and the predicted deep learning features; After using deep learning features to predict radiomics features, the InFoNCE loss function between the deep learning features and the predicted radiomics features; After using the deep learning feature to predict the deep learning feature, the InFoNCE loss function between the deep learning feature and the predicted deep learning feature; After the radiomics feature is used to predict the radiomics feature, the InFoNCE loss function between the radiomics feature and the predicted radiomics feature is used, and α<1.0 is the coefficient.

[0082] In this embodiment, the InFoNCE loss function can be expressed as:

[0083]

[0084] in, is the loss function or Z + is with the feature x m The feature set that constitutes the positive example pair, Z - is with the feature x m The set of features that constitute the negative example pair, is with the feature x m The i-th feature in the feature set that constitutes the positive example pair, sim() is the similarity function between the features. In the present invention, the image prediction information of the same batch with the same quality level is considered when constructing the positive example pair. For example, when m=h, n=d in the above formula, and the current image quality level is excellent, then Representation and radiomics features x h The deep learning features of the i-th image with excellent quality in the same training batch of the sample are predicted by its own radiomics features. The similarity sim() between features in the above formula is measured using the vector inner product.

[0085] Furthermore, the contrastive log-ratio upper bound (CLUB) reduces unnecessary correlation information between imaging group features and deep learning features by minimizing mutual information, thereby enhancing the deep learning feature extraction network's perception of feature differences. The purpose of minimizing mutual information is to try to remove semantic information of unnecessary correlation between features and exclude irrelevant details. In the study of Cheng et al. (Cheng P, Hao W, Dai S, et al. Club: A contrastive log-ratio upper bound of mutual information [C]. International conference on machine learning [C]. 2020: 1779-1788.), mutual information estimation was combined with contrastive learning, and the contrastive log-ratio upper bound was used to minimize the mutual information, that is: given two random variables a and b, the variational approximation q based on the conditional distribution p(b|a) θ The upper bound of the contrast log ratio of (b|a) is defined as:

[0086] I vCLUB (a; b): =Ep(a,b) [logq θ (b|a)]-E p(a) E p(b) [logq θ (b|a)],

[0087] Among them, q θ is the true posterior variational approximation of b given a and can be parameterized by the network θ, when q θ The closer (b|a) is to p(b|a), the higher the upper limit of the logarithmic ratio I vCLUB As the upper limit of mutual information, the more accurate it is.

[0088] The present invention refers to the study of Cheng et al. and also uses CLUB to optimize the imaging genomics feature x h and deep learning features x d The upper limit of the mutual information between them, the objective function is expressed as:

[0089]

[0090] in, is the radiomics feature of the i-th chest DR image, is the deep learning feature of the i-th chest DR image, Based on the conditional distribution The variational approximation of is the radiomics feature of the jth chest DR image, Based on the conditional distribution The variational approximation of is used, where N is the number of chest DR images. During training, this objective function reduces the dependency between two features by minimizing mutual information, thus enhancing the perception of feature differences among deep learning feature extraction networks.

[0091] Therefore, the complementary contrast learning strategy of this application enhances the complementarity between features by maximizing and minimizing the mutual information between the two features. Under this strategy, the learning of deep learning features can refer to the prior knowledge contained in the imaging genomics features, and optimize the deep learning feature extraction network by using loss callback, so that the deep learning features can be more in line with clinical standards, improving the interpretability of the chest DR image quality assessment model.

[0092] Furthermore, the present invention is inspired by the feature fusion method in the multimodal fusion task, applies the tensor fusion method to the task of the present invention, and makes slight modifications to the tensor fusion method so as to be suitable for effectively fusing deep learning features and imaging omics features, capturing deeper feature relationships, and enhancing the representation ability of the fused features. The tensor fusion method used in the present invention first controls the expression ability of deep learning features by using a gated attention mechanism, and then realizes the comprehensive interaction between deep learning features and imaging omics features through the Kronecker product, specifically:

[0093] First, learn the deep learning features x d and radiomics features h The linear transformation W dh→d , and use the Sigmoid activation function for nonlinear transformation to obtain the importance score of the corresponding feature. The importance score can be obtained using the attention weight vector w a express.

[0094] Then, the deep learning feature x d The input is sent to the Sigmoid activation function for feature transformation, and the product of the deep learning features after feature transformation and their corresponding importance scores is used for gated representation. Since current imaging omics has used feature selection strategies to filter out unimportant features, the imaging omics features do not use the gated attention mechanism in the fusion strategy.

[0095] Next, in order to preserve the original feature information when features interact, the gated representation of deep learning features and the linearly transformed imaging features are expanded using the value 1; Figure 3 As shown, after expansion, the Kronecker product of the two is calculated, so that the deep learning feature extraction network captures the important interaction between the two feature elements in the feedforward layer, which is expressed as:

[0096]

[0097] Among them, x fusion is the Kronecker product, x d,gated is the gated representation of deep learning features, x h is the radiomics feature after linear transformation represents the outer product; x fusion It is a differential tensor formed in the two-dimensional Cartesian product space, which contains both the original feature information and the information after feature fusion.

[0098] Finally, the Kronecker product is flattened and the dimensionality reduction is performed using a fully connected layer to obtain the fusion feature. After the fusion feature passes through the deep learning classification head, the end-to-end fine-tuning model and automatic prediction and classification of chest DR image quality levels can be achieved.

[0099] In summary, the present invention combines the advantages of radiomics features and deep learning features and applies them to the task of chest DR image quality assessment. By designing and using a complementary contrast learning strategy, radiomics features and deep learning features are contrasted and learned to enhance the complementarity of the two features. At the same time, known task-related predefined features are provided to the deep learning feature extraction network to improve the interpretability of the deep learning features. Subsequently, a tensor fusion method based on a gated attention mechanism and Kronecker product is used to effectively fuse the two features to achieve a more comprehensive image quality assessment.

[0100] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0101] Based on the same idea as the chest DR image quality assessment method based on feature fusion in the above-mentioned embodiment, the present invention also provides a chest DR image quality assessment system based on feature fusion, which can be used to execute the above-mentioned chest DR image quality assessment method based on feature fusion. For ease of explanation, the structural schematic diagram of the embodiment of the chest DR image quality assessment system based on feature fusion only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0102] like Figure 4 As shown, another embodiment of the present invention provides a chest DR image quality assessment system based on feature fusion, including a collection and segmentation module, a feature extraction module, a network optimization module and a feature classification module;

[0103] The collection and segmentation module is used to collect chest DR images and use the image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include the lung field area, the overlapping area between the scapula and the lung field, the clavicle area, and the external metal artifacts;

[0104] The feature extraction module is used to input the region of interest and the chest DR image into the radiomics feature extraction network and the deep learning feature extraction network to extract radiomics features and deep learning features respectively;

[0105] The network optimization module is used to perform complementary contrast learning of radiomics features and deep learning features by combining dual-feature contrast prediction coding and contrast logarithmic ratio upper limit, and optimize the deep learning feature extraction network;

[0106] The feature classification module is used to fuse radiomics features and deep learning features, and classify image quality levels through the deep learning classification head.

[0107] It should be noted that the chest DR image quality assessment system based on feature fusion of the present invention corresponds one to one with the chest DR image quality assessment method based on feature fusion of the present invention. The technical features and beneficial effects described in the embodiment of the above-mentioned chest DR image quality assessment method based on feature fusion are applicable to the embodiment of the chest DR image quality assessment system based on feature fusion. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0108] In addition, in the implementation of the chest DR image quality assessment system based on feature fusion in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the chest DR image quality assessment system based on feature fusion is divided into different program modules to complete all or part of the functions described above.

[0109] like Figure 5 As shown, in one embodiment, a computer-readable storage medium is provided, in which a program is stored in a memory, and when the program is executed by a processor, the method for evaluating chest DR image quality based on feature fusion is implemented, specifically:

[0110] Collect chest DR images, and use image segmentation network to segment chest DR images to obtain regions of interest; the regions of interest include lung field area, overlapping area between scapula and lung field, clavicle area and external metal artifacts;

[0111] The region of interest and chest DR images were input into the radiomics feature extraction network and the deep learning feature extraction network to extract radiomics features and deep learning features respectively;

[0112] The dual-feature contrast prediction coding and contrast log-ratio upper limit are combined to perform complementary contrast learning of radiomics features and deep learning features, and optimize the deep learning feature extraction network.

[0113] The imaging features and deep learning features are fused, and the image quality level is classified through the deep learning classification head.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A chest DR image quality assessment method based on feature fusion, characterized in that: The steps include: Collect chest DR images, and use an image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include the lung field area, the overlapping area between the scapula and the lung field, the clavicle area, and external metal artifacts; The region of interest and chest DR images were input into the radiomics feature extraction network and the deep learning feature extraction network to extract radiomics features and deep learning features respectively; The dual-feature contrast prediction coding and contrast log-ratio upper limit are combined to perform complementary contrast learning of radiomics features and deep learning features, and optimize the deep learning feature extraction network. The radiomics features and deep learning features are fused, and the image quality levels are classified by the deep learning classification head; the quality levels include excellent, good, fair and poor.

2. The chest DR image quality assessment method according to claim 1, characterized in that: The image segmentation network adopts U-Net network.

3. The chest DR image quality assessment method according to claim 1, characterized in that: The radiomics feature extraction network uses the PyRadiomics package to obtain radiomics features from the region of interest, including first-order statistical features, texture features, and shape features.

4. The chest DR image quality assessment method according to claim 1, characterized in that: The deep learning feature extraction network includes an input layer, a preliminary extraction layer, a feature extraction layer and an output layer; The input layer is used to stack the input region of interest and the chest DR image along the channel dimension, and obtain an input image of a set size through image preprocessing and image enhancement operations; The preliminary feature extraction layer is a convolutional layer with a channel size of 3; The feature extraction layer uses a ResNet network pre-trained on the ImageNet-1k dataset; The output layer uses a fully connected layer to perform dimensionality reduction processing.

5. The chest DR image quality assessment method according to claim 4, characterized in that: The extraction of deep learning features is specifically as follows: The region of interest and the chest DR image are input to the input layer and stacked along the channel dimension, and an input image of a set size is obtained after image preprocessing and image enhancement operations; Use the preliminary extraction layer to perform preliminary feature extraction on the input image of a set size to obtain an initial feature map that is consistent with the input size of the feature extraction layer; The initial feature map is input into the feature extraction layer to perform deep feature extraction to obtain a one-dimensional feature vector; The output layer is used to reduce the dimensionality of the one-dimensional feature vector to obtain deep learning features with the same dimension as the radiomics features.

6. The chest DR image quality assessment method according to claim 1, characterized in that: The dual-feature contrast prediction coding uses an autoregressive model to predict the radiomics features and deep learning features, and uses the InFoNCE loss function to maximize the mutual information between similar features, specifically: A perceptron with ReLU nonlinear activation function and two hidden layers is used to predict the radiomics features and deep learning features to obtain the corresponding prediction representation; Constructing positive and negative example pairs of dual-feature contrast prediction coding in a supervised manner; when constructing the positive example pairs, the prediction representations of the same batch with the same quality level are considered; when constructing the negative example pairs, the prediction representations of images of the same batch with different labels from the current image are considered; The InFoNCE loss function is used to construct the loss function of contrastive predictive coding, and contrastive learning is performed on positive and negative example pairs; The loss function of the contrast prediction coding is expressed as: in, After using radiomics features to predict deep learning features, the InFoNCE loss function between the radiomics features and the predicted deep learning features; After using deep learning features to predict radiomics features, the InFoNCE loss function between the deep learning features and the predicted radiomics features; After using the deep learning feature to predict the deep learning feature, the InFoNCE loss function between the deep learning feature and the predicted deep learning feature; After the radiomics feature is used to predict the radiomics feature, the InFoNCE loss function between the radiomics feature and the predicted radiomics feature is used, and α<1.0 is the coefficient.

7. The chest DR image quality assessment method according to claim 1, characterized in that: The contrast logarithm ratio upper limit enhances the perception of feature differences by the deep learning feature extraction network by minimizing the information of unnecessary associations between the imaging group features with less mutual information and the deep learning features; The upper limit of the contrast log ratio optimizes the radiomics feature x h and deep learning features x d The upper limit of the mutual information between them, the objective function is expressed as: in, is the radiomics feature of the i-th chest DR image, is the deep learning feature of the i-th chest DR image, Based on the conditional distribution The variational approximation of is the radiomics feature of the jth chest DR image, Based on the conditional distribution The variational approximation of is , where N is the number of chest DR images.

8. The chest DR image quality assessment method according to claim 1, characterized in that: The tensor fusion method is used to fuse the radiomics features and deep learning features. Specifically: Learn the linear transformation of deep learning features and radiomics features, and use the Sigmoid activation function to perform nonlinear transformation to obtain the importance score of the corresponding feature; the importance score is represented by an attention weight vector; The deep learning features are input into the Sigmoid activation function for feature transformation, and the product of the elements in the deep learning features after feature transformation and their corresponding importance scores are used for gated representation; The gated representation of the deep learning feature and the linearly transformed radiomics feature are expanded using the value 1, and the Kronecker product of the two is calculated, which is expressed as: Among them, x fusion is the Kronecker product, x d,gated is the gated representation of deep learning features, x h is the radiomics feature after linear transformation, represents the outer product; After the Kronecker product is flattened, the dimensionality reduction is performed using a fully connected layer to obtain the fusion feature.

9. A chest DR image quality assessment system based on feature fusion, characterized in that: The chest DR image quality assessment method applied to any one of claims 1-8 comprises a collection and segmentation module, a feature extraction module, a network optimization module and a feature classification module; The collection and segmentation module is used to collect chest DR images, and use an image segmentation network to segment the chest DR images to obtain regions of interest; the regions of interest include lung field areas, overlapping areas between scapula and lung field, clavicle areas, and external metal artifacts; The feature extraction module is used to input the region of interest and the chest DR image into the radiomics feature extraction network and the deep learning feature extraction network respectively to extract radiomics features and deep learning features; The network optimization module is used to perform complementary contrast learning of radiomics features and deep learning features by combining dual-feature contrast prediction coding and contrast logarithmic ratio upper limit, and optimize the deep learning feature extraction network; The feature classification module is used to fuse the radiomics features and the deep learning features, and classify the image quality levels through the deep learning classification head; the quality levels include excellent, good, fair and poor.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the chest DR image quality assessment method according to any one of claims 1 to 8 is implemented.

Images