Classification method of late B-ultrasound images based on multi-view feature weighted contrast learning

Abstract

The present invention discloses a late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning, the method comprising: dividing a liver fibrosis B-ultrasound image data set into multiple subsets, generating multi-view optimized data by rotating subspace technology; inputting the optimized multi-view data into a target model to obtain a representation vector and a probability vector; using the probability vector to calculate the cross entropy loss, using the representation vector to calculate the contrast loss, and calculating the joint loss function; using the joint loss function to train the target model; and using the trained target model to classify liver fibrosis B-ultrasound images. Aiming at the difficulty that the difference between early and late liver fibrosis in B-ultrasound images is not significant, the present invention uses multi-view optimization technology to extract subspace features that have a great influence on the classification results, and enhances the discriminability of the feature through contrast learning, thereby increasing the difference in feature representation of early and late liver fibrosis B-ultrasound images, and realizing high-precision classification of late liver fibrosis B-ultrasound images.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and in particular to a late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning. Background Art

[0002] Advanced liver fibrosis is a common and serious disease, the progression of which can lead to cirrhosis and even liver failure, posing a great threat to patients' health and quality of life. Timely and accurate diagnosis of advanced liver fibrosis is crucial for effective treatment and management. Currently, the diagnosis of advanced liver fibrosis mainly relies on biomarker detection, imaging examinations, and liver biopsy. However, these traditional diagnostic methods have some obvious limitations, such as biomarkers are not specific enough, imaging examinations are difficult to capture subtle changes in early fibrosis, and liver biopsy is invasive and carries certain risks.

[0003] With the development of medical imaging technology, image-based diagnostic methods have made it possible to improve the accuracy of diagnosis of advanced liver fibrosis. Nevertheless, manual analysis of these image data is time-consuming and labor-intensive, and the results are easily affected by the operator's experience and subjective judgment. Therefore, the urgent need to quickly and accurately identify advanced liver fibrosis from a large number of complex medical images has spawned the application of deep learning technology in this field. As an advanced machine learning technology, deep learning has shown its excellent performance in multiple medical image processing fields. By building a neural network model that can automatically extract and learn data features, deep learning can effectively identify pathological features from complex image data, thereby assisting doctors in making more accurate diagnoses. At present, although deep learning has achieved remarkable results in other medical image analysis such as cancer detection and retinopathy diagnosis, it still faces challenges in the automatic classification and diagnosis of advanced liver fibrosis. These challenges include the lack of obvious differences between early and late liver fibrosis in images, how to improve the sensitivity and specificity of the model for different stages of liver fibrosis, and how to effectively integrate these models in actual clinical settings.

[0004] Therefore, in response to the above challenges, developing an efficient deep learning algorithm to classify the presence or absence of advanced liver fibrosis can not only greatly improve the accuracy and efficiency of diagnosis, but also provide patients with more personalized and timely medical interventions, which has great application value. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention discloses a late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning. The method uses gradient inversion technology to extract local features that have a great impact on the classification results, and enhances the discriminability of the features through contrast learning, thereby increasing the difference in feature representation of early-stage and late-stage liver fibrosis B-ultrasound images, and achieving high-precision classification of late-stage liver fibrosis B-ultrasound images.

[0006] The objective of the present invention is to achieve a late B-ultrasound image classification method based on multi-view feature weighted contrast learning through the following technical solution, the method comprising:

[0007] Step 1, dividing the liver fibrosis B-ultrasound image data set into multiple subsets, generating multi-view image optimization data by using a rotation subspace technology, and performing image sampling on the generated multi-view image optimization data;

[0008] Step 2, input the sampled image into the target model to obtain the representation vector and probability vector;

[0009] Step 3, use the probability vector to calculate the cross entropy loss function, use the representation vector to calculate the contrast loss function, and calculate the joint loss function;

[0010] Step 4: Use the joint loss function to train the target model. If the target number of training rounds is reached, go to step 5; otherwise, go to step 1.

[0011] Step 5: Use the trained target model to classify liver fibrosis B-ultrasound images.

[0012] The method of dividing the liver fibrosis B-ultrasound image dataset into multiple subsets and generating multi-view image optimization data by rotating subspace technology comprises the following steps:

[0013] The liver fibrosis B-ultrasound image dataset is randomly divided into Subset , where each subset contains features, n represents the total number of images in the liver fibrosis B-ultrasound image dataset; for each subset, bootstrap sampling is performed to generate a subset ; For each self-sampled subset, apply the principal component analysis method to obtain the principal component coefficient matrix of the subset; convert the generated principal component coefficient matrix into a rotation matrix ; Use the rotation matrix through the formula Generate optimized multiple subspace view data; where D represents the number of features, Represents the generated optimized multiple subspace view data, represents the original liver fibrosis B-ultrasound image dataset, represents the rotation matrix; finally, multiple subspace view data are fused into multi-view image optimization data.

[0014] The step of inputting the sampled image into the target model to obtain the representation vector and the probability vector comprises the following steps:

[0015] Step 201, input the sampled image into the ResNet visual model to obtain a feature map, which is expressed as:

[0016]

[0017] in, represents the feature map of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, represents the ResNet visual model, represents the sampled image, represents the height of the sampled image, represents the width of the sampled image, Indicates the number of channels of the feature map, represents a real number;

[0018] Step 202: feature map of the sampled image Merge dimensions to obtain the feature sequence of the sampled image ;

[0019] Step 203, input the feature sequence of the sampled image into the prediction layer to obtain a probability vector, which is expressed as:

[0020]

[0021] in, represents the probability vector, represents the feature sequence of the sampled image, and is the activation function, , , and is a learnable parameter;

[0022] Step 204, calculate the probability vector Feature map of the sampled image The partial derivative of , the partial derivative The last dimension of is modulo and input into the softmax activation function to get the feature significance weight, which is expressed as:

[0023]

[0024] in, represents the feature significance weight, It means to find the modulus of the last dimension. is the activation function;

[0025] Step 205, calculate the saliency weighted feature according to the feature saliency weight and the feature map of the sampled image, the expression is:

[0026]

[0027] in, represents the significant weighted feature, represents the height of the sampled image, represents the width of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, Feature map representing the sampled image The vector of the i-th row and j-th column of The element in row i and column j represents the feature significance weight;

[0028] Step 206, input the significant weighted features into the mapping layer to obtain the representation vector, which is expressed as:

[0029]

[0030] in, represents the representation vector, represents the significant weighted feature, , , and is a learnable parameter, It means to find the modulus of the last dimension. is the activation function.

[0031] The method of calculating the cross entropy loss function using the probability vector includes the following steps:

[0032] Calculate the cross entropy loss function, the expression is:

[0033]

[0034] in, represents the cross entropy loss function, represents the number of sampled images, represents the probability vector, express The i-th element of the sampled image is labeled , Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. express No. elements;

[0035] The calculation of the contrast loss function using the representation vector comprises the following steps:

[0036] Calculate the contrast loss function, the expression is:

[0037]

[0038]

[0039] in, represents the contrast loss function, represents the number of sampled images, represents the representation vector, yes The i-th vector of yes The jth vector of yes The kth vector of the sampled image is labeled , express No. elements, Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. represents the positive sample set of the i-th sampled image, Contains the indexes of all images in the sampled images that have the same category as the i-th sampled image, Representation calculation and The inner product of Representation calculation and The inner product of .

[0040] The calculation of the joint loss function comprises the following steps:

[0041] Calculate the joint loss function, the expression is:

[0042]

[0043] in, represents the joint loss function, represents the cross entropy loss function, represents the contrast loss function, and Hyperparameters.

[0044] The method of using the trained target model to classify liver fibrosis B-ultrasound images comprises the following steps:

[0045] The liver fibrosis B-ultrasound image to be classified is input into the trained target model to obtain a probability vector and a characterization vector; the characterization vector does not participate in subsequent calculations, and the input image is classified only according to the value of the probability vector; if the value of the probability vector is greater than the set threshold, the input image is judged to be a B-ultrasound image of late liver fibrosis, otherwise, the input image is judged to be a B-ultrasound image of early liver fibrosis.

[0046] Compared with the existing methods, the advantages of the method of the present invention are: this technology provides a late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning. This method innovatively uses the idea of ​​contrast learning to increase the feature differences between early and late-stage liver fibrosis B-ultrasound images, and achieves high-precision classification of late-stage liver fibrosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] In this embodiment, it is assumed that we want to develop an automatic medical image analysis system, and the late stage liver fibrosis recognition algorithm of the system uses the late stage B-ultrasound image classification method based on multi-view feature weighted contrast learning proposed by the present invention.

[0051] Therefore, if Figure 1 As shown, a late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning comprises:

[0052] Step 1, dividing the liver fibrosis B-ultrasound image data set into multiple subsets, generating multi-view image optimization data by using a rotation subspace technology, and performing image sampling on the generated multi-view image optimization data;

[0053] Step 2, input the sampled image into the target model to obtain the representation vector and probability vector;

[0054] Step 3, use the probability vector to calculate the cross entropy loss function, use the representation vector to calculate the contrast loss function, and calculate the joint loss function;

[0055] Step 4: Use the joint loss function to train the target model. If the target number of training rounds is reached, go to step 5; otherwise, go to step 1.

[0056] Step 5: Use the trained target model to classify liver fibrosis B-ultrasound images.

[0057] The method of dividing the liver fibrosis B-ultrasound image dataset into multiple subsets and generating multi-view image optimization data by rotating subspace technology comprises the following steps:

[0058] The liver fibrosis B-ultrasound image dataset is randomly divided into Subset , where each subset contains features, n represents the total number of images in the liver fibrosis B-ultrasound image dataset; for each subset, bootstrap sampling is performed to generate a subset ; For each self-sampled subset, apply the principal component analysis method to obtain the principal component coefficient matrix of the subset; convert the generated principal component coefficient matrix into a rotation matrix ; Use the rotation matrix through the formula Generate optimized multiple subspace view data; where D represents the number of features, Represents the generated optimized multiple subspace view data, represents the original liver fibrosis B-ultrasound image dataset, represents the rotation matrix; finally, multiple subspace view data are fused into multi-view image optimization data.

[0059] The step of inputting the sampled image into the target model to obtain the representation vector and the probability vector comprises the following steps:

[0060] Step 201, input the sampled image into the ResNet visual model to obtain a feature map, which is expressed as:

[0061]

[0062] in, represents the feature map of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, represents the ResNet visual model, represents the sampled image, represents the height of the sampled image, represents the width of the sampled image, Indicates the number of channels of the feature map;

[0063] The core innovation of ResNet is the introduction of the "residual learning" framework. In traditional neural networks, each layer learns features directly from the output of the previous layer. In ResNet, the connection between layers is different. Each layer not only learns new feature transformations, but also adds the input directly to the output. This method is called "skip connection" or "identity mapping".

[0064] The benefit of skip connections is that they allow the network to pass gradients directly to earlier layers during training, avoiding the problem of gradients gradually disappearing in deep networks. In this way, even if the network is very deep, it can still learn effectively and maintain training stability.

[0065] Step 202: feature map of the sampled image Merge dimensions to obtain the feature sequence of the sampled image ;

[0066] Step 203, input the feature sequence of the sampled image into the prediction layer to obtain a probability vector, which is expressed as:

[0067]

[0068] in, represents the probability vector, represents the feature sequence of the sampled image, and is the activation function, , , and is a learnable parameter;

[0069] Step 204, calculate the probability vector Feature map of the sampled image The partial derivative of , the partial derivative The last dimension of is modulo and input into the softmax activation function to get the feature significance weight, which is expressed as:

[0070]

[0071] in, represents the feature significance weight, It means to find the modulus of the last dimension. is the activation function;

[0072] Step 205, calculate the saliency weighted feature according to the feature saliency weight and the feature map of the sampled image, the expression is:

[0073]

[0074] in, represents the significant weighted feature, represents the height of the sampled image, represents the width of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, Feature map representing the sampled image The vector of the i-th row and j-th column of The element in row i and column j represents the feature significance weight;

[0075] Step 206, input the significant weighted features into the mapping layer to obtain the representation vector, which is expressed as:

[0076]

[0077] in, represents the representation vector, represents the significant weighted feature, , , and is a learnable parameter, It means to find the modulus of the last dimension. is the activation function.

[0078] The method of calculating the cross entropy loss function using the probability vector includes the following steps:

[0079] Calculate the cross entropy loss function, the expression is:

[0080]

[0081] in, represents the cross entropy loss function, represents the number of sampled images, represents the probability vector, express The i-th element of the sampled image is labeled , Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. express No. elements;

[0082] The calculation of the contrast loss function using the representation vector comprises the following steps:

[0083] Calculate the contrast loss function, the expression is:

[0084]

[0085]

[0086] in, represents the contrast loss function, represents the number of sampled images, represents the representation vector, yes The i-th vector of the sampled image is labeled , express No. elements, Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. represents the positive sample set of the i-th sampled image, Contains the indexes of all images in the sampled images that have the same category as the i-th sampled image, Representation calculation and The inner product of

[0087] The calculation of the joint loss function comprises the following steps:

[0088] Calculate the joint loss function, the expression is:

[0089]

[0090] in, represents the joint loss function, represents the cross entropy loss function, represents the contrast loss function, and Hyperparameters.

[0091] There are many optimization algorithms that can be used to optimize the target model using the joint loss function, including the Adam optimization algorithm, which is a widely used optimization method for deep learning model training. Adam combines the advantages of two extended gradient descent algorithms: momentum and RMSprop.

[0092] Momentum is mainly used to solve two problems in the gradient descent algorithm: slow convergence and easy to fall into local optimality. It accelerates the learning process by introducing "inertia" in the gradient update rule, that is, considering the update direction of the previous step to adjust the current update direction, which helps to accelerate convergence and reduce oscillation.

[0093] RMSprop uses a different learning rate for each parameter. This adaptive learning rate adjustment method makes it more robust when encountering different types of data distributions. RMSprop dynamically adjusts the learning step size by adjusting the learning rate in the gradient update rule to make it dependent on the size of the most recent gradient.

[0094] The Adam algorithm combines the advantages of both, and adjusts the learning rate of each parameter by calculating the first-order moment estimate (i.e., mean, similar to the concept in momentum) and the second-order moment estimate (i.e., uncentered variance) of the gradient. This makes Adam relatively insensitive to the choice of hyperparameters in practice, especially the learning rate. Due to its high efficiency and adaptive characteristics, Adam is widely used in a variety of deep learning applications, from natural language processing to computer vision tasks.

[0095] The method of using the trained target model to classify liver fibrosis B-ultrasound images comprises the following steps:

[0096] The liver fibrosis B-ultrasound image to be classified is input into the trained target model to obtain a probability vector and a characterization vector; the characterization vector does not participate in subsequent calculations, and the input image is classified only according to the value of the probability vector; if the value of the probability vector is greater than the set threshold, the input image is judged to be a B-ultrasound image of late liver fibrosis, otherwise, the input image is judged to be a B-ultrasound image of early liver fibrosis.

[0097] Those skilled in the art should understand that the above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A late-stage B-ultrasound image classification method based on multi-view feature weighted contrast learning, characterized in that: The method comprises: Step 1, dividing the liver fibrosis B-ultrasound image data set into multiple subsets, generating multi-view image optimization data by using a rotation subspace technology, and performing image sampling on the generated multi-view image optimization data; Step 2, input the sampled image into the target model to obtain the representation vector and probability vector; Step 3, use the probability vector to calculate the cross entropy loss function, use the representation vector to calculate the contrast loss function, and calculate the joint loss function; Step 4: Use the joint loss function to train the target model. If the target number of training rounds is reached, go to step 5; otherwise, go to step 1. Step 5, using the trained target model to classify liver fibrosis B-ultrasound images; The method of dividing the liver fibrosis B-ultrasound image dataset into multiple subsets and generating multi-view image optimization data by rotating subspace technology comprises the following steps: The liver fibrosis B-ultrasound image dataset is randomly divided into Subset , where each subset contains features, n represents the total number of images in the liver fibrosis B-ultrasound image dataset; for each subset, bootstrap sampling is performed to generate a subset ; For each self-sampled subset, apply the principal component analysis method to obtain the principal component coefficient matrix of the subset; convert the generated principal component coefficient matrix into a rotation matrix ; Use the rotation matrix through the formula Generate optimized multiple subspace view data; wherein, D Indicates the number of features, Represents the generated optimized multiple subspace view data, represents the original liver fibrosis B-ultrasound image dataset, represents the rotation matrix; finally, multiple subspace view data are fused into multi-view image optimization data; The step of inputting the sampled image into the target model to obtain the representation vector and the probability vector comprises the following steps: Step 201, input the sampled image into the ResNet visual model to obtain a feature map, which is expressed as: in, represents the feature map of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, represents the ResNet visual model, represents the sampled image, represents the height of the sampled image, represents the width of the sampled image, Indicates the number of channels of the feature map, represents a real number; Step 202: feature map of the sampled image Merge dimensions to obtain the feature sequence of the sampled image ; Step 203, input the feature sequence of the sampled image into the prediction layer to obtain a probability vector, which is expressed as: in, represents the probability vector, represents the feature sequence of the sampled image, and is the activation function, , , and is a learnable parameter; Step 204, calculate the probability vector Feature map of the sampled image The partial derivative of , the partial derivative The last dimension of is modulo and input into the softmax activation function to get the feature significance weight, which is expressed as: in, represents the feature significance weight, It means to find the modulus of the last dimension. is the activation function; Step 205, calculate the saliency weighted feature according to the feature saliency weight and the feature map of the sampled image, the expression is: in, represents the significant weighted feature, represents the height of the sampled image, represents the width of the sampled image, represents the height of the feature map of the sampled image, represents the width of the feature map of the sampled image, Feature map representing the sampled image The vector of the i-th row and j-th column of The element in row i and column j represents the feature significance weight; Step 206, input the significant weighted features into the mapping layer to obtain the representation vector, which is expressed as: in, represents the representation vector, represents the significant weighted feature, , , and is a learnable parameter, It means to find the modulus of the last dimension. is the activation function.

2. The late B-ultrasound image classification method based on multi-view feature weighted contrast learning according to claim 1 is characterized in that: The method of calculating the cross entropy loss function using the probability vector includes the following steps: Calculate the cross entropy loss function, the expression is: in, represents the cross entropy loss function, represents the number of sampled images, represents the probability vector, express The i-th element of the sampled image is labeled , Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. express No. elements.

3. The late B-ultrasound image classification method based on multi-view feature weighted contrast learning according to claim 2 is characterized in that: The method of calculating the contrast loss function using the representation vector includes the following steps: Calculate the contrast loss function, the expression is: in, represents the contrast loss function, represents the number of sampled images, represents the representation vector, yes The i-th vector of yes The jth vector of yes The kth vector of the sampled image is labeled , express No. elements, Indicates Zhang sampling image is an image of late stage liver fibrosis, Indicates Zhang sampling image is an image of early liver fibrosis. represents the positive sample set of the i-th sampled image, Contains the indexes of all images in the sampled images that have the same category as the i-th sampled image, Representation calculation and The inner product of Representation calculation and The inner product of .

4. The late B-ultrasound image classification method based on multi-view feature weighted contrast learning according to claim 3 is characterized in that: The calculation of the joint loss function comprises the following steps: Calculate the joint loss function, the expression is: in, represents the joint loss function, represents the cross entropy loss function, represents the contrast loss function, and Hyperparameters.

5. The late B-ultrasound image classification method based on multi-view feature weighted contrast learning according to claim 4, characterized in that: The method of using the trained target model to classify liver fibrosis B-ultrasound images comprises the following steps: The liver fibrosis B-ultrasound image to be classified is input into the trained target model to obtain a probability vector and a characterization vector; the characterization vector does not participate in subsequent calculations, and the input image is classified only according to the value of the probability vector; if the value of the probability vector is greater than the set threshold, the input image is judged to be a B-ultrasound image of late liver fibrosis, otherwise, the input image is judged to be a B-ultrasound image of early liver fibrosis.

Images