Fracture image classification method, system and equipment based on segmentation guidance and medium

Abstract

The invention provides a fracture image classification method based on segmentation guidance. The method comprises the following steps: acquiring fracture images and constructing a data set; segmenting the fracture image in the data set by using a segmentation model, and generating a binary segmentation image marking a fracture region and a normal region; constructing two feature extraction networks, and respectively performing feature extraction on the segmented fracture image and the fracture image before segmentation; fusing the extracted features by using a cross attention mechanism to obtain a fused feature map; training a classifier by using the feature map and the corresponding category label to obtain a fracture image classification model; and inputting a to-be-classified fracture image into the fracture image classification model to obtain category information of the fracture image. According to the invention, the segmentation result of the fracture image is used as auxiliary information to be introduced into the classification model through the segmentation guide classification method, and the segmentation information and the original image information are fused by using the cross attention mechanism, so that the recognition capability of the classification model on the fracture type is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a fracture image classification method, system, device and medium based on segmentation guidance. Background Art

[0002] In modern medical imaging diagnosis, fracture detection and classification are important links in clinical diagnosis and treatment. Especially for tibial plateau fractures, due to their complexity and diversity, accurate diagnosis is crucial for the formulation of treatment plans and patient rehabilitation. However, traditional manual diagnostic methods rely on the experience and skills of doctors, which is not only time-consuming and labor-intensive, but also difficult to ensure the accuracy and consistency of diagnostic results. In recent years, with the rapid development of medical imaging technology, computer vision and deep learning, image-based fracture detection and classification technology has made significant progress. These technologies use advanced image processing and machine learning algorithms to automatically extract and analyze fracture features from medical images, greatly improving the efficiency and accuracy of diagnosis. In particular, deep learning models have become a research hotspot in the field of fracture detection due to their powerful capabilities in image feature extraction and pattern recognition.

[0003] Among the existing fracture detection methods, models based on convolutional neural networks (CNNs) are widely used, such as YOLO and ResNet. The model is trained with large-scale labeled data, and can automatically extract fracture features from medical images to achieve efficient fracture detection and classification. However, the traditional CNN model still has some shortcomings when processing complex fracture images. First, it is difficult for traditional classification models to fully utilize segmentation information, resulting in low classification accuracy when processing complex fracture images. Second, the model relies on a large amount of high-quality labeled data, which is costly and time-consuming to obtain. Summary of the invention

[0004] The present invention proposes a fracture image classification method, system, device and medium based on segmentation guidance, which solves the problem that traditional classification models in the prior art are difficult to fully utilize segmentation information, resulting in low classification accuracy when processing complex fracture images.

[0005] The technical solution of the present invention is achieved in this way:

[0006] The first aspect of the present invention provides a fracture image classification method based on segmentation guidance, comprising the following steps:

[0007] Acquire fracture images and construct datasets;

[0008] The fracture images in the data set are segmented using the segmentation model to generate a binary segmentation map marking the fracture area and the normal area;

[0009] Construct two feature extraction networks to extract features from the fracture images after segmentation and the fracture images before segmentation respectively;

[0010] The cross-attention mechanism is used to fuse the extracted features to obtain the fused feature map;

[0011] The classifier is trained using the feature map and the corresponding category label to obtain a fracture image classification model;

[0012] The fracture image to be classified is input into the fracture image classification model to obtain the category information of the fracture image.

[0013] Specifically, the method of acquiring fracture images and constructing a data set includes:

[0014] Collect tibial plateau image data, wherein the image data includes fracture images and non-fracture images;

[0015] Preprocessing the fracture images in the imaging data;

[0016] A non-fracture image is selected from the image data as a template image, a fracture image matching the template image is selected from the image data, a fracture area in the fracture image is copied and pasted into the template image, and a pasted fracture image is obtained;

[0017] The pasted image is repaired using the trained diffusion model to obtain a repaired fracture image;

[0018] All repaired fracture images were merged with the fracture images in the imaging data to construct a dataset.

[0019] Furthermore, preprocessing the fracture image in the image data includes: cutting the fracture image into a uniform size and performing histogram equalization processing.

[0020] Specifically, the training method of the diffusion model includes:

[0021] Perform the forward diffusion process to gradually add noise to the preprocessed fracture image to generate a variety of noise images. The noise adding process is as follows:

[0022]

[0023] Among them, x t is the image of the tth step, N(·) represents the normal distribution, I represents the identity matrix; β t represents the amount of noise added in step t;

[0024] Perform the back diffusion process to gradually denoise the noisy image and generate a natural and realistic fracture image. The denoising process is as follows:

[0025] p θ(x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),σ t 2 I)

[0026] Among them, μ θ , σ t are the mean and standard deviation predicted by the neural network parameters θ respectively;

[0027] The loss function of the diffusion model is:

[0028]

[0029] Among them, E represents the expected value of the random variables x0, μ, and t, x0 is the original fracture image, μ is the actual added noise, t represents the time step, and μ θ The noise predicted by the diffusion model.

[0030] Specifically, the method of repairing the pasted image using the trained diffusion model is as follows:

[0031] The pasted image is input into the trained diffusion model, and the noise image x at the previous step t-1 is inferred according to the reverse diffusion process. t-1 , execute the following formula iteratively for multiple times until t = 1, and the repaired fracture image can be obtained;

[0032]

[0033] Among them, α t =1-β t , ε θ (x t , t) represents the noise predicted by the diffusion model.

[0034] Specifically, two independent feature extraction networks are used to respectively extract image features of the fracture image after segmentation and the fracture image before segmentation, and the feature extraction network includes multiple convolutional layers and pooling layers to extract multi-scale image features; the fracture image before segmentation is used to provide overall feature information of the fracture image, and the fracture image after segmentation is used to provide detailed feature information of the fracture area.

[0035] Specifically, the methods of fusing the extracted features using the cross-attention mechanism include:

[0036] The image features before segmentation are used as the query matrix Q orig , the segmented image features are used as the key matrix K seg Sum value matrix V seg;

[0037]

[0038] The segmented image features are used as the query matrix Q seg , the image features before segmentation are used as the key matrix K orig Sum value matrix V orig ;

[0039]

[0040] Among them, W q , W k , W v Represent the weights of the query matrix, key matrix, and value matrix respectively; X orig , X seg Represent the features extracted from the image before and after segmentation respectively;

[0041] Calculate the dot product of the corresponding query matrix and key matrix to obtain the attention weights, and apply the attention weights to the corresponding value matrix to generate interaction features in two directions;

[0042]

[0043] Among them, F interaction1 、F interaction2 They represent the interaction features in two directions, softmax(·) represents the normalization function, and d k represents the dimension of the key matrix;

[0044] Perform weighted average of the interactive features in the two directions to generate a fused feature map;

[0045] F fused =αF interaction1 +βF interaction2

[0046] Among them, F fused represents the fused feature map, α and β represent the weighted average weights of the two interactive features respectively.

[0047] A second aspect of the present invention provides a fracture image classification system based on segmentation guidance, comprising:

[0048] A data acquisition module, used for acquiring tibial plateau image data;

[0049] A preprocessing module, used for preprocessing images;

[0050] The data augmentation module is used to generate new fracture images by copying and pasting, and to repair the generated fracture images using the diffusion model to generate natural and realistic fracture images to expand the image dataset;

[0051] An image segmentation module is used to segment the preprocessed image and generate a binary segmentation map marking the fracture area and the normal area;

[0052] The feature extraction and fusion module is used to extract the feature information of the images before and after segmentation respectively, and fuse the extracted features using the cross-attention mechanism to obtain the fused feature map;

[0053] The classification module is used to classify the fused feature maps.

[0054] A third aspect of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the above-mentioned fracture image classification method when executing the computer program.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned fracture image classification method are implemented.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) The present invention introduces the segmentation results of the fracture image as auxiliary information into the classification model through the segmentation-guided classification method, thereby improving the accuracy of classification. The present invention adopts a cross-attention mechanism to fuse the segmentation information and the original image information. This design enables the network to more effectively utilize the segmentation information and improve the classification model's ability to identify fracture types;

[0058] (2) The present invention generates new fracture images by copying and pasting, and repairs the generated fracture images using a diffusion model, which can generate more training data, especially data on the fracture area, thereby enhancing the training data set of the segmentation model. It ensures that the generated fracture images are more natural visually and more in line with the actual situation. It not only increases the diversity of training data and significantly improves the performance of the segmentation model, but also improves the model's ability to recognize the fracture area, thereby enhancing the classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1The present invention is a flowchart of a fracture image classification method based on segmentation guidance. DETAILED DESCRIPTION

[0061] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Reference Figure 1 A first aspect of an embodiment of the present invention provides a fracture image classification method based on segmentation guidance, comprising the following steps:

[0063] Acquire fracture images and construct datasets;

[0064] The fracture images in the dataset are segmented using a segmentation model (medical segmentation network) to generate a binary segmentation map marking the fracture area and the normal area, with the fracture area being 1 and the normal area being 0.

[0065] Construct two feature extraction networks to extract features from the fracture images after segmentation and the fracture images before segmentation respectively;

[0066] The cross-attention mechanism is used to fuse the extracted features to obtain the fused feature map;

[0067] The classifier is trained using the feature map and the corresponding category label to obtain a fracture image classification model;

[0068] The fracture image to be classified is input into the fracture image classification model (advanced feature extraction network) to obtain the category information of the fracture image.

[0069] Specifically, the method of acquiring fracture images and constructing a data set includes:

[0070] Collect tibial plateau image data, including fracture images and non-fracture images, to ensure that the data set includes fracture images of various fracture types and severity;

[0071] Preprocess the fracture images in the imaging data, including image standardization and data enhancement;

[0072] A non-fracture image is selected from the image data as a template image, a fracture image matching the template image is selected from the image data, a fracture area in the fracture image is copied and pasted into the template image, and a pasted fracture image is obtained;

[0073] Using the trained diffusion model (DDPM or DDIM can be selected as the diffusion model architecture for training) to repair the pasted image, and obtain a repaired fracture image;

[0074] All repaired fracture images were merged with the fracture images in the imaging data to construct a dataset.

[0075] Furthermore, preprocessing the fracture image in the image data includes: cutting the fracture image into a uniform size and performing histogram equalization processing.

[0076] Specifically, the training method of the diffusion model includes:

[0077] Perform the forward diffusion process to gradually add noise to the preprocessed fracture image to generate a variety of noise images. The noise adding process is as follows:

[0078]

[0079] Among them, x t is the image of the tth step, N(·) represents the normal distribution, I represents the identity matrix; β t represents the amount of noise added in step t;

[0080] Perform the back diffusion process to gradually denoise the noisy image and generate a natural and realistic fracture image. The denoising process is as follows:

[0081] p θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),σ t 2 I)

[0082] Among them, μ θ , σ t are the mean and standard deviation predicted by the neural network parameters θ respectively;

[0083] The goal of the diffusion model is to minimize the error between the predicted noise and the actual noise. The loss function of the diffusion model is:

[0084]

[0085] Where E represents the expected value of random variables x0, μ, and t, x0 is the original fracture image, μ is the Gaussian noise actually added, and the Gaussian noise actually added is the random noise sampled from the standard normal distribution N(0,I); t represents the time step, which is usually sampled from a discrete uniform distribution, that is, t is uniformly selected from {1,2,…,T}; μ θis the noise predicted by the diffusion model. By minimizing this loss function, the model can learn how to effectively denoise at different noise levels, thereby generating realistic data samples during the back diffusion process.

[0086] Specifically, the method of repairing the pasted image using the trained diffusion model is as follows:

[0087] The pasted image is input into the trained diffusion model, and the noise image x at the previous step t-1 is inferred according to the reverse diffusion process. t-1 , execute the following formula iteratively for multiple times until t = 1, and the repaired fracture image can be obtained;

[0088]

[0089] Among them, α t =1-β t , ε θ (x t , t) represents the noise predicted by the diffusion model.

[0090] Specifically, when generating fracture images, the key point information in all fracture and non-fracture images is first obtained through the bone key point detection technology to calibrate the anatomical structure of the bone. Then, according to the principle of minimum matching error after affine transformation, the fracture image that best matches the bone image without fracture area to be pasted is found in the bone image containing fracture to ensure the visual effect and anatomical consistency after pasting, and then the fracture area is extracted from the fracture image and pasted to the corresponding area of ​​the non-fracture image. During the pasting process, affine transformation is required to ensure that the pasted fracture area is aligned with the bone structure of the target image to avoid visual unnaturalness and incoherence.

[0091] Specifically, before the fracture image is segmented, the image is preprocessed, cropped to a size of 608×608 pixels, and histogram equalization is performed. The category labels of the fracture image are manually annotated. The category labels are Schatzker types I, II, III, IV, V, and VI. According to the image files of the data set, each type of fracture image is divided into a ratio of training set: test set = 7:1.

[0092] Specifically, two independent feature extraction networks (ResNet-18 network is used in this embodiment) are used to extract image features of the fracture image after segmentation and the fracture image before segmentation, respectively. The feature extraction network includes multiple convolutional layers and pooling layers to extract multi-scale image features; the fracture image before segmentation is used to provide overall feature information of the fracture image, and the fracture image after segmentation is used to provide detailed feature information of the fracture area.

[0093] Specifically, the methods of fusing the extracted features using the cross-attention mechanism include:

[0094] The image features before segmentation are used as the query matrix Q orig , the segmented image features are used as the key matrix K seg Sum value matrix V seg ;

[0095]

[0096] The segmented image features are used as the query matrix Q seg , the image features before segmentation are used as the key matrix K orig Sum value matrix V orig ;

[0097]

[0098] Among them, W q , W k , W v X represents the weights of the query matrix, key matrix, and value matrix, respectively, which are used to linearly transform the input features to generate the query vector, key vector, and value vector; orig , X seg Represent the features extracted from the image before and after segmentation respectively;

[0099] Calculate the dot product of the corresponding query matrix and key matrix to obtain the attention weights, and apply the attention weights to the corresponding value matrix to generate interaction features in two directions;

[0100]

[0101] Among them, F interaction1 、F interaction2 They represent the interaction features in two directions, softmax(·) represents the normalization function, and d k represents the dimension of the key matrix;

[0102] Perform weighted average of the interactive features in the two directions to generate a fused feature map;

[0103] F fused =αF interaction1 +βF interaction2

[0104] Among them, F fusedRepresents the fused feature map, α and β represent the weighted average weights of the two interactive features, which are usually learned during the training process and reflect the importance of each interactive feature in the final fused feature. Their settings can be associated with the attention weight, that is, dynamically adjusted according to the weight distribution calculated in the attention mechanism to better reflect the contribution of different features in a specific task.

[0105] A second aspect of an embodiment of the present invention provides a fracture image classification system based on segmentation guidance, comprising:

[0106] A data acquisition module, used for acquiring tibial plateau image data;

[0107] A preprocessing module, used for preprocessing images;

[0108] The data augmentation module is used to generate new fracture images by copying and pasting, and to repair the generated fracture images using the diffusion model to generate natural and realistic fracture images to expand the image dataset;

[0109] An image segmentation module is used to segment the preprocessed image and generate a binary segmentation map marking the fracture area and the normal area;

[0110] The feature extraction and fusion module is used to extract the feature information of the images before and after segmentation respectively, and fuse the extracted features using the cross-attention mechanism to obtain the fused feature map;

[0111] The classification module is used to classify the fused feature maps.

[0112] A third aspect of an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the steps of the above-mentioned fracture image classification method when executing the computer program.

[0113] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned fracture image classification method are implemented.

[0114] To further illustrate the beneficial effects of the present invention, this embodiment evaluates the impact of using a diffusion model for data augmentation on the performance of a segmentation model and a classification model.

[0115] First, the performance of the segmentation model before and after augmentation is compared, and the results are shown in Table 1 below:

[0116] Before augmentation After augmentation Mdice(%) 0.792 0.824

[0117] As can be seen from Table 1, the mdice metric can measure the overall performance of the model in the semantic segmentation task. The results show that after using the diffusion model to augment the data, the mDice value of the SegNeXt segmentation model increased from 0.792 to 0.824, significantly improving the performance of the segmentation model. Through the additional data generated by the diffusion model, we not only increased the number of training samples, but also improved the diversity of samples, which helped the model to show better robustness on different types of fracture images.

[0118] Next, we compared the performance of the baseline classification model, the pre-augmentation segmentation-guided classification model, and the post-augmentation segmentation-guided classification model. The results are shown in Table 2 below:

[0119]

[0120] As can be seen from Table 2, the Accuracy index can measure the accuracy of the classification model. The accuracy of the segmentation-guided classification method has been significantly improved, which shows that by integrating segmentation information, the accuracy and robustness of our classification model in the fracture classification task have been significantly improved. The guidance of the segmentation model trained with the dataset augmented by the diffusion model has greatly improved the indicators of the classification model.

[0121] The method proposed in this application has important value in clinical applications. By improving the accuracy of fracture classification, doctors can diagnose and treat patients more accurately and reduce the misdiagnosis and mistreatment rates. In addition, the segmentation-guided classification method and diffusion model augmentation strategy can be applied to other medical image analysis tasks, such as tumor detection and organ segmentation, thereby promoting the development of the entire medical image analysis field.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A fracture image classification method based on segmentation guidance, characterized in that: The following steps are involved: Acquire fracture images and construct datasets; The fracture images in the data set are segmented using the segmentation model to generate a binary segmentation map marking the fracture area and the normal area; Construct two feature extraction networks to extract features from the fracture images after segmentation and the fracture images before segmentation respectively; The cross-attention mechanism is used to fuse the extracted features to obtain the fused feature map; The classifier is trained using the feature map and the corresponding category label to obtain a fracture image classification model; The fracture image to be classified is input into the fracture image classification model to obtain the category information of the fracture image.

2. A fracture image classification method based on segmentation guidance as claimed in claim 1, characterized in that: The methods for acquiring fracture images and constructing datasets include: Collect tibial plateau image data, wherein the image data includes fracture images and non-fracture images; Preprocessing the fracture images in the imaging data; A non-fracture image is selected from the image data as a template image, a fracture image matching the template image is selected from the image data, a fracture area in the fracture image is copied and pasted into the template image, and a pasted fracture image is obtained; The pasted image is repaired using the trained diffusion model to obtain a repaired fracture image; All repaired fracture images were merged with the fracture images in the imaging data to construct a dataset.

3. A fracture image classification method based on segmentation guidance as claimed in claim 2, characterized in that: Preprocessing the fracture images in the imaging data includes: cutting the fracture images into a uniform size and performing histogram equalization processing.

4. A fracture image classification method based on segmentation guidance as claimed in claim 2, characterized in that: The training method of the diffusion model includes: Perform the forward diffusion process to gradually add noise to the preprocessed fracture image to generate a variety of noise images. The noise adding process is as follows: Among them, x t is the image of the tth step, N(·) represents the normal distribution, I represents the identity matrix; β t represents the amount of noise added in step t; Perform the back diffusion process to gradually denoise the noisy image and generate a natural and realistic fracture image. The denoising process is as follows: p θ (x t-1 |x t )=N(x k-1 ;m θ (x t ,t),s t 2 I) Among them, μ θ , σ t are the mean and standard deviation predicted by the neural network parameters θ respectively; The loss function of the diffusion model is: Among them, E represents the expected value of the random variables x0, μ, and t, x0 is the original fracture image, μ is the actual added noise, t represents the time step, and μ θ The noise predicted by the diffusion model.

5. A fracture image classification method based on segmentation guidance as claimed in claim 4, characterized in that: The method of using the trained diffusion model to repair the pasted image is: The pasted image is input into the trained diffusion model, and the noise image x at the previous step t-1 is inferred according to the reverse diffusion process. t-1 , execute the following formula iteratively for multiple times until t = 1, and the repaired fracture image can be obtained; Among them, α t =1-β t , ε θ (x t , t) represents the noise predicted by the diffusion model.

6. The fracture image classification method based on segmentation guidance according to claim 1, characterized in that: Two independent feature extraction networks are used to respectively extract image features of the fracture image after segmentation and the fracture image before segmentation, wherein the feature extraction network includes multiple convolutional layers and pooling layers to extract multi-scale image features; the fracture image before segmentation is used to provide overall feature information of the fracture image, and the fracture image after segmentation is used to provide detailed feature information of the fracture area.

7. The fracture image classification method based on segmentation guidance according to claim 1, characterized in that: Methods for fusing extracted features using the cross-attention mechanism include: The image features before segmentation are used as the query matrix Q orig , the segmented image features are used as the key matrix K seg Sum value matrix V seg ; The segmented image features are used as the query matrix Q seg , the image features before segmentation are used as the key matrix K orig Sum value matrix V orig ; Among them, W q , W k , W v Represent the weights of the query matrix, key matrix, and value matrix respectively; X orig , X seg Represent the features extracted from the image before and after segmentation respectively; Calculate the dot product of the corresponding query matrix and key matrix to obtain the attention weights, and apply the attention weights to the corresponding value matrix to generate interaction features in two directions; Among them, F interaction1 、F interaction2 They represent the interaction features in two directions, softmax(·) represents the normalization function, and d k represents the dimension of the key matrix; Perform weighted average of the interactive features in the two directions to generate a fused feature map; F fused =αF interaction1 +BF interaction2 Among them, F fused represents the fused feature map, α and β represent the weighted average weights of the two interactive features respectively.

8. A fracture image classification system based on segmentation guidance, based on the fracture image classification method according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used for acquiring tibial plateau image data; A preprocessing module, used for preprocessing images; The data augmentation module is used to generate new fracture images by copying and pasting, and to repair the generated fracture images using the diffusion model to generate natural and realistic fracture images to expand the image dataset; An image segmentation module is used to segment the preprocessed image and generate a binary segmentation map marking the fracture area and the normal area; The feature extraction and fusion module is used to extract the feature information of the images before and after segmentation respectively, and fuse the extracted features using the cross-attention mechanism to obtain the fused feature map; The classification module is used to classify the fused feature maps.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the fracture image classification method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the fracture image classification method according to any one of claims 1 to 7 are implemented.

Images