LSKA-driven multi-dimensional feature cross-level fusion knee joint cartilage injury classification method

Abstract

The invention discloses an LSKA-driven multi-dimensional feature cross-level fusion knee joint cartilage injury classification method, and the method comprises the steps: obtaining a feature map of a knee joint cartilage injury MRI image and a multi-scale feature map after double-path channel splicing through a conventional YOLOv8 frame, obtaining the middle features after deep convolution and the middle features after expansion deep convolution based on an LSKA mechanism, and carrying out the classification of the knee joint cartilage injury through the multi-dimensional feature cross-level fusion. Further obtaining a self-adaptive weighted feature map; and acquiring the confidence coefficient of the predicted knee joint cartilage injury category by adopting a head network, and performing five classification tasks on the knee joint cartilage injury MRI image. According to the method, the LSKA module is embedded in the traditional YOLOv8 network framework, so that multi-scale feature enhancement can be realized, the problem that the local receptive field of a fixed convolution kernel limits the capture of global context information is solved, and the image feature expression capability is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knee cartilage injury grade classification based on deep learning, and particularly relates to a knee cartilage injury classification method based on multi-dimensional feature cross-level fusion driven by LSKA. BACKGROUND

[0002] Knee cartilage injury is a common joint disease in clinical practice, and its high incidence and irreversibility pose a severe challenge to clinical diagnosis and treatment. Due to the lack of blood vessels and nerve innervation in cartilage tissue, the self-repairing ability after injury is extremely limited, and early accurate diagnosis is crucial to delay the progression of the disease. At present, magnetic resonance imaging (MRI) has become the preferred imaging method for evaluating cartilage injury due to its excellent soft tissue contrast. In recent years, deep learning technology has shown great potential in medical image analysis, especially in the automatic classification of knee cartilage injury, ligament injury and meniscus injury.

[0003] The clinical diagnosis of knee cartilage injury has long relied on the subjective judgment of doctors on MRI images, which has problems such as low efficiency and insufficient consistency. With the development of computer vision technology, machine learning-based auxiliary diagnosis methods have gradually become a research hotspot.

[0004] In recent years, deep learning models have made significant progress in medical image analysis due to their strong feature learning ability. Traditional convolutional neural network architectures such as AlexNet, ResNet and GooleNet have improved feature expression ability by increasing network depth, but the fixed local receptive field of the convolution kernel limits the capture of global context information. Lightweight models such as SqueezeNet and MobileNet reduce computational complexity through channel pruning and depth separable convolution, but perform poorly when dealing with multi-site, multi-stage cartilage injuries. SUMMARY

[0005] The present application discloses a knee cartilage injury classification method based on multi-dimensional feature cross-level fusion driven by LSKA to overcome the above technical problems.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] A knee cartilage injury classification method based on multi-dimensional feature cross-level fusion driven by LSKA, comprising the following steps:

[0008] S1: Obtain a knee cartilage injury MRI image, and obtain a feature map of the knee cartilage injury MRI image based on a convolution module of a YOLOv8 framework;

[0009] S2: According to the feature map of the knee cartilage injury MRI image, the C2f module based on the YOLOv8 framework is used to obtain the feature map of the semantic enhancement path and the feature map of the detail reservation path, and then the multi-scale feature map after the double-path channel splicing is obtained;

[0010] S3: According to the multi-scale feature map after the double-path channel splicing, the intermediate feature after the deep convolution is obtained based on the LSKA mechanism, the intermediate feature after the expansion deep convolution is obtained, and then the adaptive weighted feature map is obtained;

[0011] S4: According to the adaptive weighted feature map, the confidence of the predicted knee cartilage injury category is obtained, and the cross-entropy loss function is used for training based on the LSKA driven YOLOv8 adaptive classification model, so as to complete the five classification task of knee cartilage injury and provide auxiliary basis for clinicians.

[0012] Further, the convolution module based on the YOLOv8 framework includes a convolution layer, a batch normalization layer and a SiLU activation function;

[0013] The method for obtaining the feature value of the knee cartilage injury MRI image is as follows:

[0014] S11: According to the knee cartilage injury MRI image, the feature map output by the convolution layer is obtained based on the convolution layer, so as to obtain the original feature value output by the convolution layer; wherein the value of the pixel point in the feature map output by the convolution layer is the original feature value output by the convolution layer;

[0015] The formula for obtaining the original feature value output by the convolution layer is as follows:

[0016]

[0017] Wherein, represents the feature map output by the convolution layer; i and j are respectively the row index and column index of the pixel point in the knee cartilage injury MRI image; l is the index of the output channel; represents the feature map output by the first layer convolution at the lth output channel and position (i, j); c represents the index of the input channel; C represents the total number of input channels; N is the size of the convolution kernel; m and n are both the spatial dimension index of the convolution kernel in the range of [0, N-1]; I i+m,j+n,c represents the pixel value of the input image I at the input channel c, position (i+m, j+n), represents the convolution kernel of the first layer, represents the bias term of the lth output channel in the convolution kernel of the first layer; I is the input image, i.e. the knee cartilage injury MRI image;

[0018] S12: based on the batch normalization layer, according to the original feature value output by the convolutional layer, the feature value output after batch normalization is obtained; the formula used is as follows:

[0019]

[0020]

[0021]

[0022] wherein, represents the feature value output after batch normalization at the lth output channel and position (i, j) ; represents the original feature value output by the convolutional layer at the lth output channel and position (i, j) ; μ l represents the mean of all spatial position features of the lth output channel; represents the variance of all spatial position features of the lth output channel; ε is a very small positive number; B represents the batch sample capacity; b represents the index of the sample; W out is the width of the output feature map, H out is the height of the output feature map.

[0023]

[0024] wherein, represents the feature value output after batch normalization; γ l represents a learnable scaling factor; β l represents a translation term; BN represents batch normalization;

[0025] S13: based on the SiLU activation function, according to the feature value output after batch normalization, the feature value of the knee cartilage injury MRI image is obtained; the formula used is as follows:

[0026]

[0027] wherein, σ (x) is a Sigmoid function, represents the feature value output after SiLU activation function, that is, the feature value of the knee cartilage injury MRI image; e is the natural base.

[0028] Further, the method for obtaining the multi-scale feature map after the double-path channel splicing is as follows:

[0029] S21: obtain the feature map of the semantic enhancement path and the feature map of the detail preservation path, to obtain the output of the semantic enhancement path and the output of the detail preservation path based on the C2f module of the semantic enhancement path and the detail preservation path;

[0030] The formula used to obtain the feature map of the semantic enhancement path and the feature map of the detail retention path is as follows:

[0031] (X1,X2)=Split(F,C1,C2)

[0032] C1=C2=0.5C out

[0033] Wherein, X1 represents the feature map of the semantic enhancement path; X2 represents the feature map of the detail retention path; Split(·) represents a channel splitting function; F represents the feature map of the knee cartilage injury MRI image; C1 represents the number of channels of X1; C2 represents the number of channels of X2; C out is the number of channels output by the C2f module;

[0034] S22: According to the output of the semantic enhancement path and the output of the detail retention path, a multi-scale feature map after double-path channel splicing is obtained, and the formula used is as follows:

[0035]

[0036] Wherein, is the output of the semantic enhancement path, is the output of the detail retention path, F Concat is the multi-scale feature map after double-path channel splicing; Concat(·) represents a channel splicing operation.

[0037] Further, the method used to obtain the adaptive weighted feature map is as follows:

[0038] S31: According to the multi-scale feature map after double-path channel splicing, a 1x1 convolution layer is used to obtain a multi-scale feature map after double-path channel splicing after channel transformation;

[0039] S32: According to the multi-scale feature map after double-path channel splicing after channel transformation, an intermediate feature after deep convolution is obtained, and the formula used is as follows:

[0040]

[0041] Wherein, represents the intermediate feature after deep convolution; H is the height of the knee cartilage injury MRI image, and W is the width of the knee cartilage injury MRI image; is a one-dimensional deep convolution kernel in the vertical direction; d is the dilation rate; DW represents a deep convolution operation; is a one-dimensional deep convolution kernel in the horizontal direction; F C represents the multi-scale feature map after double-path channel splicing after channel transformation;

[0042] S33: Obtain the intermediate feature after the dilated depth convolution according to the intermediate feature after the depth convolution, and the formula used is as follows:

[0043]

[0044] wherein, Z C represents the intermediate feature after the dilated depth convolution; represents the one-dimensional dilated depth convolution kernel in the vertical direction; represents the one-dimensional dilated depth convolution kernel in the horizontal direction; k is the total number of convolution kernels;

[0045] S34: Obtain the attention map according to the intermediate feature after the dilated depth convolution, and the formula used is as follows:

[0046] A C =M 1×1 * PW Z C

[0047] wherein, A C represents the attention map; M 1×1 represents the 1x1 convolution kernel; * PW represents the point-by-point convolution operation;

[0048] S35: Obtain the multi-scale feature map F C based on the channel transformation after the double-path channel splicing according to the attention map, and obtain the adaptive weighted feature map based on the LSKA module, and the formula used is as follows:

[0049]

[0050] wherein, represents the adaptive weighted feature map; represents element-by-element multiplication; F C represents the multi-scale feature map after the double-path channel splicing after the channel transformation.

[0051] Further, the method for obtaining the confidence of the predicted knee cartilage injury category is as follows:

[0052] S41: Obtain the one-dimensional feature vector based on the global pooling layer according to the adaptive weighted feature map;

[0053] S42: Obtain the response vector of the knee cartilage injury category based on the knee cartilage injury MRI image according to the one-dimensional feature vector and the full connection layer in the head network, so as to obtain the response value of the oth knee cartilage injury category based on the knee cartilage injury MRI image, and the formula used is as follows:

[0054] z=ηx c +b

[0055] wherein, η is a weight matrix; b is a bias vector, z is a response vector of the knee cartilage injury category based on the knee cartilage injury MRI image; x c is a one-dimensional feature vector;

[0056] S43: According to the response value of the oth knee cartilage injury category based on the knee cartilage injury MRI image, the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury category is obtained; the formula used is as follows:

[0057]

[0058] wherein, represents the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury category; z o is the response value of the oth knee cartilage injury category based on the knee cartilage injury MRI image; o represents the index of the knee cartilage injury category, and e is the natural base.

[0059] S44: According to the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury category, a cross-entropy loss function is established as follows:

[0060]

[0061] In the formula: L u CE represents the loss value of the u th knee cartilage injury MRI image sample; o represents the index of the knee cartilage injury category; O represents the total number of knee cartilage injury categories; y o u represents the true label of the u th knee cartilage injury MRI image sample; is the probability that the predicted u th knee cartilage injury MRI image sample belongs to the oth knee cartilage injury category;

[0062]

[0063] In the formula: l represents the overall loss value of the cross-entropy loss function; u represents the index of the knee cartilage injury MRI image sample; U represents the total number of knee cartilage injury MRI image samples.

[0064] Beneficial effects: the knee cartilage injury classification method of the LSKA driven multi-dimensional feature cross-level fusion of the application, the feature value of the knee cartilage injury MRI image is obtained through the traditional YOLOv8 framework, and the multi-scale feature map after the double-path channel splicing is obtained through the C2f module, based on the LSKA mechanism, the intermediate feature after the deep convolution is obtained, in order to obtain the intermediate feature after the expansion deep convolution, and then the adaptive weighted feature map is obtained; and then the head network is used for the five classification task of the knee cartilage injury. The LSKA module is embedded in the traditional YOLOv8 network framework, which can realize multi-scale feature enhancement, solve the problem that the local receptive field of the fixed convolution kernel limits the capture of global context information, effectively improve the image feature expression ability, and perform well in processing multi-site and multi-stage cartilage injury. At the same time, the problems of insufficient calculation efficiency and limited fine feature recognition ability in the knee cartilage injury classification task are effectively solved, and the calculation complexity is low. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0066] Figure 1 The flow chart of the knee cartilage injury classification method of the LSKA driven multi-dimensional feature cross-level fusion of the application;

[0067] Figure 2 The technical flow chart of the knee cartilage injury classification method of the LSKA driven multi-dimensional feature cross-level fusion in the embodiment of the application;

[0068] Figure 3 The convolution module structure schematic diagram in the embodiment of the application;

[0069] Figure 4 The C2f module structure schematic diagram in the embodiment of the application;

[0070] Figure 5 The LSKA module structure schematic diagram in the embodiment of the application;

[0071] Figure 6 The data set example image in the embodiment of the application;

[0072] Figure 7 The experimental result schematic diagram in the embodiment of the application. DETAILED DESCRIPTION

[0073] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments but not all of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0074] The embodiment introduces a knee cartilage injury classification method of multi-dimensional feature cross-level fusion driven by LSKA, which includes the following steps, as shown in the following: Figure 1

[0075] S1: Obtain the knee cartilage injury MRI image, obtain the feature value of the knee cartilage injury MRI image based on the convolution module of the YOLOv8 framework, and then obtain the feature map of the knee cartilage injury MRI image;

[0076] Preferably, the convolution module based on the YOLOv8 framework includes a convolution layer, a batch normalization layer and a SiLU activation function; as shown in the following: Figure 3 wherein x1 represents the distribution dimension of the original feature, x2 represents the distribution dimension of the normalized and scaled feature, and r is a learnable scaling factor.

[0077] The method for obtaining the feature value of the knee cartilage injury MRI image is as follows:

[0078] S11: Based on the convolution layer, the feature map output by the convolution layer is obtained according to the knee cartilage injury MRI image, so as to obtain the original feature value output by the convolution layer; wherein the value of the pixel point in the feature map output by the convolution layer is the original feature value output by the convolution layer;

[0079] The formula used is as follows:

[0080]

[0081] wherein i and j are respectively the row index and column index of the pixel point in the knee cartilage injury MRI image, used for positioning the row and column positions of a single pixel point; l is the index of the output channel, assuming that the total number of convolution kernels is k, then 1≤l≤k; represents the feature map output at the lth output channel and position (i,j) after the first layer of convolution; c represents the index of the input channel; C represents the total number of input channels; N is the size of the convolution kernel; m and n are both the spatial dimension index of the convolution kernel varying in the range of [0,N-1], used for traversing each value in the convolution kernel; i+m,j+n,c represents the pixel value of the input image I at the input channel c and position (i+m,j+n), represents the convolution kernel of the first layer,​ bias term of the lth output channel in the convolution kernel representing the first layer; I is an input image, i.e., a knee cartilage injury MRI image;

[0082] Specifically, the input image of the convolution module, i.e., the knee cartilage injury MRI image, is I ∈

[0083] R H×W×C where H is the height of the knee cartilage injury MRI image, W is the width of the knee cartilage injury MRI image, C is the total number of input channels, R is a real set, and the input image size of the embodiment is 640x640x3. The feature map output at the lth output channel and position (i, j) after the first layer convolution is where the pixel value at position (i, j) is the original feature value at position (i, j) output by the convolution layer.

[0084] S12: Based on the batch normalization layer, the original feature value output by the convolution layer is used to obtain the feature value output after batch normalization; the formula used is as follows:

[0085]

[0086]

[0087]

[0088] wherein, represents the feature value after batch normalization at the lth output channel and position (i, j); represents the original feature value output by the convolution layer at the lth output channel and position (i, j); μ l represents the mean of all spatial position features of the lth output channel; represents the variance of all spatial position features of the lth output channel; ε is a very small positive number; B represents the batch sample capacity; b represents the index of the sample; W out is the width of the output feature map, and H out is the height of the output feature map.

[0089] Specifically, in order to effectively suppress the internal covariate shift phenomenon in the training process of the deep neural network, a batch normalization layer is introduced in the convolution module. The standardization process of the batch normalization layer takes the channel dimension as the operation unit. Assuming that the small batch sample capacity is B, each element in the feature map of the lth channel is normalized.

[0090] ​In order to restore the expression ability lost in the normalization process, make the model more flexible to learn data features, the batch normalization layer introduces a learnable scaling factor γ l ∈R and a translation term β l ∈R are optimized during backpropagation, and the calculation formula is as follows:

[0091]

[0092] Wherein, represents the feature value output after batch normalization; γ l represents the learnable scaling factor; β l represents the translation term; BN represents batch normalization;

[0093] S13: Based on the SiLU activation function, the feature value of the knee cartilage injury MRI image is obtained according to the feature value output after batch normalization;

[0094] The formula used is as follows:

[0095]

[0096] Wherein, σ(x) is the Sigmoid function, represents the feature value output after the SiLU activation function, that is, the feature value of the knee cartilage injury MRI image; e is the natural base.

[0097] Specifically, after SiLU activation, the feature values at all positions jointly constitute the feature map F of the knee cartilage injury MRI image output by the complete convolution module, which is input into the C2f module.

[0098] Specifically, to achieve a dynamic balance between model training stability and feature representation ability, the SiLU activation function is used in this embodiment, which can combine the probability smoothing characteristics of the Sigmoid function and the conditional sparse activation advantage of the ReLU function. In the network architecture proposed in this embodiment, multi-scale learning is achieved by using convolution modules connected in a hierarchical manner, which realizes the extraction of edge features of knee cartilage and texture features of subchondral edema regions from shallow networks, and the extraction of semantic features of cartilage injury categories from deep networks, and then accurately classifies the knee cartilage injury.

[0099] S2: Based on the C2f module of the YOLOv8 framework, the feature map of the semantic enhancement path and the feature map of the detail preservation path are obtained according to the feature value of the knee cartilage injury MRI image, to obtain a multi-scale feature map after the double-path channel splicing;

[0100] Preferably, the method for obtaining the multi-scale feature map after the double-path channel splicing is as follows:

[0101] S21: According to the feature map of the knee cartilage injury MRI image, the feature map of the semantic enhancement path and the feature map of the detail reservation path are obtained based on the C2f module of the YOLOv8 framework, and then the multi-scale feature map after the double-path channel splicing is obtained;

[0102] The formula used to obtain the feature map of the semantic enhancement path and the feature map of the detail reservation path is as follows:

[0103] (X1,X2)=Split(F,C1,C2) (5)

[0104] C1=C2=0.5C out

[0105] Wherein, X1 represents the feature map of the semantic enhancement path; X2 represents the feature map of the detail reservation path; Split(·) represents the channel splitting function; F represents the feature map of the knee cartilage injury MRI image; C1 represents the channel number of X1; C2 represents the channel number of X2; C out is the channel number output by the C2f module.

[0106] Specifically, in order to effectively capture multi-scale damage feature information, the dynamic channel decoupling operation is used to divide the semantic enhancement path X1 and the detail reservation path X2. First, the input feature map is divided into two parts according to the channel number C, wherein HxWxC out is the dimension of the input feature map, C1=C2=0.5C out . Wherein, the X1 path uses the Bottleneck unit to extract the subchondral bone cyst region feature in depth, and the residual design is used to relieve the gradient disappearance and improve the sensitivity to the cartilage fissure. The X2 path retains the original feature details, and the lesion edge blur problem caused by deep down sampling is relieved.

[0107] S22: According to the output of the semantic enhancement path and the output of the detail reservation path, the multi-scale feature map after the double-path channel splicing is obtained, and the formula used is as follows:

[0108]

[0109] Wherein, is the output of the semantic enhancement path, is the output of the detail reservation path, F Concat is the multi-scale feature map after the double-path channel splicing; Concat(·) represents the channel splicing operation;

[0110] Specifically, the double-path output of the semantic enhancement path and the detail reservation path is spliced in the channel to realize multi-scale feature fusion. At this time, the channel number of the feature map becomes 0.5(n+2)C out; while the height and width remain H x W, n = 1 is the number of Bottleneck units.

[0111] Specifically, after obtaining the multi-scale feature after the dual-path channel splicing, the embodiment performs linear transformation on the channel number of the multi-scale feature map F Concat after the dual-path channel splicing through a 1x1 convolution layer to obtain F C , so as to adjust the channel number to be consistent with the channel number C out after the preliminary convolution when inputting, so as to seamlessly connect the subsequent network structure.

[0112] Specifically, the embodiment introduces a C2f module based on anatomical priori for the contradiction between the deep semantic feature information attenuation and the small scale characteristics of the cartilage lesion region in the knee cartilage injury classification task. The module provides strong support for accurate identification of knee cartilage injury through a dual-path feature interaction mechanism (including a semantic enhancement path and a detail reservation path), as shown in Figure 4 .

[0113] S3: Based on the multi-scale feature map after the dual-path channel splicing, an intermediate feature after deep convolution is obtained based on the LSKA mechanism, an intermediate feature after dilated deep convolution is obtained, and then an adaptive weighting feature map is obtained.

[0114] Preferably, the method for obtaining the adaptive weighting feature map is as follows:

[0115] S31: According to the multi-scale feature map after the dual-path channel splicing, a 1x1 convolution layer is used to perform linear transformation on the channel number of the multi-scale feature map F Concat after the dual-path channel splicing, to obtain a multi-scale feature map F C after channel transformation after the dual-path channel splicing, that is, a feature map input to DWConv;

[0116] S32: According to the multi-scale feature map F C after the dual-path channel splicing after channel transformation, an intermediate feature after deep convolution is obtained, and the formula used is as follows:

[0117]

[0118] wherein, represents the intermediate feature after deep convolution; H is the height of the knee cartilage injury MRI image, and W is the width of the knee cartilage injury MRI image; is a one-dimensional depth convolution kernel in the vertical direction; d is the dilation rate; DW represents a depth convolution operation; is a one-dimensional depth convolution kernel in the horizontal direction; F CThe multi-scale feature map after concatenation of the two-path channel after channel transformation;

[0119] Specifically, the LSKA decomposes the k*k convolution into a depthwise convolution (DWConv), a depthwise dilated convolution (DWDConv), and a pointwise convolution (PWConv). To avoid the aggravation of parameter redundancy caused by an excessively large convolution kernel size, the DWConv first extracts spatial features independently for each channel, and by introducing a dilation rate d, decomposes the traditional two-dimensional depthwise convolution kernel k*k into a horizontal convolution kernel (2d-1)*1 and a vertical convolution kernel 1*(2d-1) in cascade.

[0120] S33: Obtain the intermediate feature after the dilated depthwise convolution according to the intermediate feature after the depthwise convolution, and the formula used is as follows:

[0121]

[0122] wherein, Z C represents the intermediate feature after the dilated depthwise convolution; represents a one-dimensional dilated depthwise convolution kernel in the vertical direction; represents a one-dimensional dilated depthwise convolution kernel in the horizontal direction; k is the total number of convolution kernels;

[0123] Specifically, in order to maintain the sensitivity of the long-distance structure of cartilage, ligament and other tissues, the intermediate feature after the depthwise convolution is processed by the dilated depthwise convolution , and the traditional two-dimensional convolution kernel is decomposed into a one-dimensional horizontal convolution kernel and a vertical convolution kernel by a dilation rate d. C .

[0124] S34: Obtain the attention map according to the intermediate feature after the dilated depthwise convolution, and the formula used is as follows:

[0125] A C = M 1×1 * PW Z C (9)

[0126] wherein, A C represents the attention map; M 1×1 represents a 1*1 convolution kernel; and PW represents a pointwise convolution operation.

[0127] Specifically, in order to realize cross-channel information fusion, the LSKA mechanism uses a 1*1 convolution kernel M 1×1The intermediate feature Z after the expansion depth convolution C Point-wise convolution is performed to highlight the feature channels related to knee cartilage damage, and then an attention map A is generated C .

[0128] S35: The multi-scale feature map F after concatenation of the attention map and the channel-transformed dual-path channel C Based on the LSKA module, an adaptive weighted feature map is obtained, and the formula used is as follows:

[0129]

[0130] wherein, represents the adaptive weighted feature map; represents element-wise multiplication; F C represents the multi-scale feature map after concatenation of the channel-transformed dual-path channel, which is the feature map input to the LSKA module;

[0131] Specifically, in order to enhance the knee cartilage damage area feature, the attention map and the input feature F C are multiplied element-wise by Hadamard product to generate an adaptive weighted feature map

[0132] Specifically, in order to significantly enhance the cartilage fissure feature and effectively improve the classification accuracy, the LSKA module is introduced in the network architecture proposed in this embodiment, which improves the sensitivity of long-distance structures of cartilage, ligament and other tissues while maintaining the computing efficiency through efficient kernel decomposition strategy and adaptive feature weighting. The network structure of the LSKA module is shown in Figure 5 .

[0133] S4: According to the adaptive weighted feature map, the confidence of the predicted knee cartilage damage category is obtained, and based on the LSKA-driven YOLOv8 adaptive classification model, this embodiment uses the cross-entropy loss function for training to complete the five-classification task of knee cartilage damage, providing auxiliary basis for clinicians.

[0134] Preferably, the method used to obtain the confidence of the predicted knee cartilage damage category is as follows:

[0135] Specifically, when performing the knee cartilage damage classification task, the head network aims to map the features output by the backbone network to specific knee cartilage damage categories, and its structure is composed of a fully connected layer and a Softmax activation function. Among them, the one-dimensional feature vector x c output after global pooling operation is taken as the input of the fully connected layer.

[0136] S41: Obtain a one-dimensional feature vector x based on the global pooling layer according to the adaptive weighted feature map c ;

[0137] S42: Obtain an output vector of the fully connected layer in the head network, i.e., a response vector of the knee cartilage injury class based on the knee cartilage injury MRI image, according to the fully connected layer in the head network, according to the one-dimensional feature vector x c , so as to obtain a response value of the oth knee cartilage injury class based on the knee cartilage injury MRI image;

[0138] z = ηx c + b (11)

[0139] wherein η is a weight matrix, an element η gq in the weight matrix η represents the contribution degree of the qth input feature to the gth knee cartilage injury output class; b is a bias vector, and z is a response vector of the knee cartilage injury class based on the knee cartilage injury MRI image; x c is a one-dimensional feature vector;

[0140] Specifically, after being processed by the main network embedded with the LSKA module, a one-dimensional feature vector x c is generated, which contains key information such as cartilage texture and semantics in the knee MRI image. In order to convert high-dimensional features into injury classes, the head network uses a fully connected layer for linear transformation to generate response values of each injury class. After obtaining the output vector of the fully connected layer, the response value of the oth knee cartilage injury class based on the knee cartilage injury MRI image can be obtained.

[0141] S43: Obtain the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury class based on the response value of the oth knee cartilage injury class based on the knee cartilage injury MRI image. The formula used is as follows:

[0142]

[0143] wherein represents the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury class; z o is the response value of the oth knee cartilage injury class based on the knee cartilage injury MRI image; o represents the index of the knee cartilage injury class; and e is the natural base.

[0144] Specifically, z is the output vector of the fully connected layer, which contains response values of five knee cartilage injury classes. This z o is the response value corresponding to a certain class in z.

[0145] Specifically, the unnormalized class response value output by the full connection layer cannot be directly used for classification decision-making, and it needs to be converted into a confidence distribution with a probability meaning. To this end, a Softmax activation function is used for normalization processing.

[0146] S44: According to the confidence that the predicted knee cartilage injury MRI image belongs to the oth knee cartilage injury category, the cross-entropy loss function is established as follows:

[0147] Specifically, in the knee cartilage 5 classification task, for the u-th sample (u = 1, 2, …, U) in the training set, U is the total number of training set samples, and the output probability distribution based on the LSKA driven YOLOv8 adaptive classification model is The cross-entropy loss L of a single sample u CE The calculation formula is:

[0148]

[0149]

[0150] In the formula: L u CE represents the loss value of the u-th knee cartilage injury MRI image sample; o represents the index of the knee cartilage injury category; O represents the total number of knee cartilage injury categories, and in this embodiment, there are 5 categories of knee cartilage injury; y o u represents the true label of the u-th knee cartilage injury MRI image sample, and the true label value of the oth knee cartilage injury category is marked; is the probability that the predicted u-th knee cartilage injury MRI image sample belongs to the oth knee cartilage injury category, and in this embodiment, it needs to satisfy to ensure the rationality and effectiveness of the probability distribution.

[0151] Specifically, to avoid interference from individual samples during training, the average loss of all samples in the training set is taken as the optimization target, that is, the calculation formula of the overall loss value L is:

[0152]

[0153] In the formula: L represents the overall loss value of the cross-entropy loss function; u represents the index of the knee cartilage injury MRI image sample; and U represents the total number of knee cartilage injury MRI image samples;

[0154] Specifically, the embodiment adjusts the parameters that increase the loss in the opposite direction of the gradient by back propagation, reduces the influence of error classification, and strengthens the parameters that reduce the loss in the positive direction of the gradient, gradually improves the recognition ability of the model to low contrast and small scale damage features, and realizes accurate classification of five types of damage.

[0155] Specifically, when it is necessary to classify the damage to which the new knee cartilage damage MRI image belongs, the trained LSKA driven YOLOv8 adaptive classification model is used, so that the confidence of the damage class to which the new knee cartilage damage MRI image belongs, that is, the probability of the damage class to which the new knee cartilage damage MRI image belongs, can be obtained.

[0156] A specific embodiment of the present application is as follows:

[0157] A data set from real hospital data provided by the Second Affiliated Hospital of Dalian Medical University is constructed. The data set includes three types of multi-modal MRI image data of sagittal, transverse and coronal, wherein 0-level images are 548, I-level images are 674, II-level images are 582, III-level images are 539, IV-level images are 579, and the image resolution is 640*640. An example image of the data set is shown in Figure 6 .

[0158] The data set is divided into a training set and a test set in a ratio of 8:2 in the embodiment, the training rounds of the model are set to 350 rounds, the initial learning rate is 0.01, and the model with the highest accuracy in the test set is finally saved. And through the SGD optimizer, dynamic adjustment is carried out in the training process to realize efficient optimization. The experimental results of the embodiment are shown in Figure 7 .

[0159] In order to evaluate the performance advantage of the algorithm in the present embodiment in the pathological grading classification of kidney cancer, the present embodiment compares and verifies it with the traditional model and the most advanced SOTA (State-of-the-Art) model, as shown in Table 1. The experimental results show that the method proposed in the present embodiment is superior to other comparison models in various evaluation indexes, and shows obvious performance advantage.

[0160] Table 1 Comparison of experimental results

[0161]

[0162]

[0163] The embodiment introduces an LSKA module (large kernel attention mechanism module) in the Backbone of the YOLOv8 framework, and a feature extraction network is composed of a convolution module, a C2f module and the LSKA module, and a head network is combined to construct an LSKA-driven YOLOv8 adaptive classification model, so as to realize attention focusing on the knee cartilage injury area in an MRI image and multi-scale feature fusion. The network first inputs a multi-dimensional knee cartilage injury MRI image into the feature extraction network, wherein the convolution module adopts a three-layer structure of "convolution layer-batch normalization layer-SiLU activation function" to perform primary feature extraction, and then realizes multi-level feature interaction and attention enhancement through the C2f module and the LSKA module.

[0164] In summary, the embodiment has the following beneficial effects:

[0165] The first multi-dimensional MRI knee cartilage injury hospital real dataset is constructed: in order to ensure the authenticity, clinical relevance and non-invasive diagnosis of the data, the multi-dimensional MRI knee cartilage injury hospital real dataset is constructed to assist the clinician to realize more accurate knee cartilage injury diagnosis.

[0166] The LSKA-driven YOLOv8 adaptive classification model is constructed: the LSKA module is embedded in the YOLOv8 network, and through spatial feature extraction of the depth separable convolution and channel information integration of the pointwise convolution, multi-scale feature enhancement is realized, and the image feature expression ability is effectively improved.

[0167] The knee cartilage injury classification method of multi-dimensional feature adaptive fusion is proposed: the classification model proposed in the embodiment is trained on the knee cartilage injury real dataset constructed, and the classification accuracy is 99.7%.

[0168] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A knee cartilage injury classification method based on LSKA-driven multi-dimensional feature cross-level fusion, characterized in that, The method comprises the following steps: S1: acquiring a knee cartilage injury MRI image, and acquiring a feature map of the knee cartilage injury MRI image based on a convolution module of a YOLOv8 framework; S2: acquiring a feature map of a semantic enhancement path and a feature map of a detail reservation path based on a C2f module of the YOLOv8 framework according to the feature map of the knee cartilage injury MRI image, and then acquiring a multi-scale feature map after a double-path channel is spliced; S3: acquiring an intermediate feature after deep convolution based on a LSKA mechanism according to the multi-scale feature map after the double-path channel is spliced, acquiring an intermediate feature after expansion deep convolution, and then acquiring an adaptive weighting feature map; S4: acquiring a confidence of a predicted knee cartilage injury category based on the adaptive weighting feature map, and training by using a cross-entropy loss function based on a YOLOv8 adaptive classification model driven by the LSKA to complete a five-classification task of the knee cartilage injury and provide an auxiliary basis for a clinician.

2. The LSKA-driven multi-dimensional feature cross-level fusion-based knee cartilage lesion classification method according to claim 1, characterized in that, The convolution module based on the YOLOv8 framework comprises a convolution layer, a batch normalization layer and a SiLU activation function; The method for acquiring the feature value of the knee cartilage injury MRI image comprises the following steps: S11: acquiring a feature map output by the convolution layer based on the convolution layer according to the knee cartilage injury MRI image, so as to acquire an original feature value output by the convolution layer; wherein the value of a pixel point in the feature map output by the convolution layer is the original feature value output by the convolution layer; The formula used for acquiring the original feature value output by the convolution layer is as follows: wherein, represents the feature map output by the convolutional layer; i and j are the row index and column index of the pixel point in the MRI image of the knee cartilage injury, respectively; and l is the index of the output channel; represents the feature map output by the first layer of convolution at the lth output channel and position (i, j); c represents the index of the input channel; C represents the total number of input channels; N is the size of the convolution kernel; m and n are both the spatial dimension index of the convolution kernel varying in the range of [0, N-1]; I i+m,j+n,c represents the pixel value of the input image I at the input channel c, position (i+m, j+n), represents the convolution kernel of the first layer, represents the bias term of the lth output channel in the convolution kernel of the first layer; I is the input image, i.e., the MRI image of the knee cartilage injury; S12: acquiring a feature value output after batch normalization based on the batch normalization layer according to the original feature value output by the convolution layer; the formula used is as follows: wherein, represents the batch-normalized feature value at position (i, j) of the lth output channel; represents the original feature value output by the convolutional layer at position (i, j) of the lth output channel; μ l represents the mean of all spatial position features of the lth output channel; represents the variance of all spatial position features of the lth output channel; ε is a very small positive number; B represents the batch sample capacity; b represents the index of the sample; W out is the width of the output feature map, H out is the height of the output feature map; wherein, represents the output feature value after batch normalization; γ l represents a learnable scaling factor; β l represents a translation term; BN represents batch normalization; S13: acquiring the feature value of the knee cartilage injury MRI image based on the SiLU activation function according to the feature value output after batch normalization; the formula used is as follows: Wherein, σ(x) is a Sigmoid function, The feature value of the knee cartilage injury MRI image after the SiLU activation function is represented as the output feature value.

3. The LSKA-driven multi-dimensional feature cross-level fusion-based knee cartilage lesion classification method according to claim 2, characterized in that, The method for acquiring the multi-scale feature map after the double-path channel is spliced comprises the following steps: S21: acquiring the feature map of the semantic enhancement path and the feature map of the detail reservation path, so as to acquire an output of the semantic enhancement path and an output of the detail reservation path based on the semantic enhancement path and the detail reservation path of the C2f module; The formula used for acquiring the feature map of the semantic enhancement path and the feature map of the detail reservation path is as follows: (X1, X2) = Split (F, C1, C2) C1 = C2 = 0.5C out wherein X1 represents a feature map of a semantic enhancement path; X2 represents a feature map of a detail reservation path; Split(·) represents a channel splitting function; F represents a feature map of the knee cartilage injury MRI image; C1 represents a channel number of X1; C2 represents a channel number of X2; C out is a channel number output by the C2f module. S22: acquiring the multi-scale feature map after the double-path channel is spliced according to the output of the semantic enhancement path and the output of the detail reservation path; the formula used is as follows: wherein, is the output of the semantic enhancement path, is the output of the detail preserving path, Concat is the multi-scale feature map after concatenating the two paths; Concat(·) denotes the channel concatenation operation.

4. The LSKA-driven multi-dimensional feature cross-level fusion-based knee cartilage injury classification method according to claim 3, characterized in that, The method for acquiring the adaptive weighting feature map comprises the following steps: S31: acquiring a multi-scale feature map after a channel transformation of the double-path channel based on the multi-scale feature map after the double-path channel is spliced by using a 1*1 convolution layer; S32: acquiring the intermediate feature after deep convolution based on the multi-scale feature map after the channel transformation of the double-path channel; the formula used is as follows: wherein, represents the intermediate feature after deep convolution; H is the height of the knee cartilage injury MRI image, and W is the width of the knee cartilage injury MRI image; is a one-dimensional deep convolution kernel in the vertical direction; d is the expansion rate; DW represents a deep convolution operation; is a one-dimensional deep convolution kernel in the horizontal direction; F C represents a multi-scale feature map after channel transformation and double-path channel splicing; S33: acquiring the intermediate feature after expansion deep convolution based on the intermediate feature after deep convolution; the formula used is as follows: wherein Z C represents the intermediate feature after dilated depth convolution; represents a one-dimensional dilated depth convolution kernel in the vertical direction; represents a one-dimensional dilated depth convolution kernel in the horizontal direction; k is the total number of convolution kernels; S34: According to the intermediate feature after the expansion depth convolution, an attention map is obtained, and the formula used is as follows: A C = M 1×1 * Z PW Z C wherein A C denotes an attention map; M 1×1 denotes a 1 x 1 convolution kernel; * PW denotes a point-wise convolution operation; S35: The multi-scale feature map F after splicing of the attention map and the double-path channel after channel transformation C Based on the LSKA module, an adaptive weighted feature map is obtained, and the formula used is as follows: wherein, denotes an adaptive weighted feature map; denotes element-wise multiplication; F C denotes the multi-scale feature map after concatenation of the two paths after channel transformation.

5. The LSKA-driven multi-dimensional feature cross-level fusion based knee cartilage lesion classification method according to claim 4, characterized in that, The method used to obtain the confidence of the predicted knee cartilage injury category is as follows: S41: According to the adaptive weighted feature map, a one-dimensional feature vector is obtained based on a global pooling layer; S42: According to the one-dimensional feature vector, a response vector of the knee cartilage injury category based on the knee cartilage injury MRI image is obtained according to a fully connected layer in the head network, so as to obtain a response value of the oth knee cartilage injury category based on the knee cartilage injury MRI image, and the formula used is as follows: z = ηx c + b wherein η is a weight matrix; b is a bias vector, z is a response vector of the knee cartilage injury category based on the knee cartilage injury MRI image; x c is a one-dimensional feature vector; S43: According to the response value of the oth knee cartilage injury category based on the knee cartilage injury MRI image, the confidence of the predicted knee cartilage injury MRI image belonging to the oth knee cartilage injury category is obtained; the formula used is as follows: wherein, represents the confidence that the predicted knee cartilage lesion MRI image belongs to the oth knee cartilage lesion class; z o is the response value based on the oth knee cartilage lesion class of the knee cartilage lesion MRI image; o represents the index of the knee cartilage lesion class, and e is the natural base. S44: According to the confidence of the predicted knee cartilage injury MRI image belonging to the oth knee cartilage injury category, a cross-entropy loss function is established as follows: In the formula: L u CE Loss value of the u-th knee cartilage injury MRI image sample; o represents the index of the knee cartilage injury category; O represents the total number of the knee cartilage injury categories; y o u True label of the u-th knee cartilage injury MRI image sample; Probability that the u-th predicted knee cartilage injury MRI image sample belongs to the o-th knee cartilage injury category; In the formula, L represents the overall loss value of the cross-entropy loss function; u represents the index of the knee cartilage injury MRI image sample; and U represents the total number of knee cartilage injury MRI image samples.

Images