Diabetic foot prediction method, device and equipment based on multi-view feature fusion

Abstract

The present invention is applicable to the fields of medical treatment and computer vision, and provides a method, device and equipment for predicting diabetic foot based on multi-view feature fusion, the method comprising the following steps: obtaining sample information of diabetic foot to be predicted, the sample information including infrared thermal image of foot and color image of foot; constructing a deep learning model: based on the deep learning model generated by transfer learning, applying receptive field optimization algorithm to construct temperature distribution feature enhancement module, applying receptive field optimization algorithm to construct skin texture feature enhancement module, constructing spatial attention enhancement module and multi-view feature fusion module, and obtaining the deep learning model to be trained. The present invention improves the accuracy of prediction of diabetic high-risk foot by fusing multi-view features; in addition, the receptive field optimization algorithm proposed in the present invention can dynamically adjust the size of the receptive field when enhancing the temperature distribution feature for feet of different sizes, and ensure the effectiveness of the temperature distribution feature.

Description

Technical Field

[0001] The present invention belongs to the field of medical treatment and computer vision, and in particular relates to a diabetic foot prediction method, device and equipment based on multi-view feature fusion. Background Art

[0002] Diabetic foot (DF) is a foot ulcer or deep tissue destruction caused by distal neuropathy or vascular lesions in the lower limbs of patients with diabetes (DM). The "Guidelines for the Prevention and Treatment of Diabetic Foot in China (2019 Edition)" proposes that DF prevention is more important than treatment. Therefore, if high-risk diabetic foot patients who have not yet developed foot ulcers can be discovered and intervened in time, the occurrence of foot ulcers and amputations will be effectively reduced. At present, the diagnostic methods recommended by the "Guidelines for the Prevention and Treatment of Diabetic Foot in China (2019 Edition)" have clear limitations in different degrees for the screening of DF, and many tests can only be applied to the detection of large blood vessels and coarse nerve fiber lesions. When these test results are positive, it indicates that the large blood vessels and nerves have already had serious lesions, so it is not conducive to the early screening of DF characterized by damage to small blood vessels and small nerve fibers.

[0003] Infrared Thermography (IRT), as a functional medical imaging technology, can visualize and digitally present the thermal radiation generated by early microneuropathy and microcirculatory disorders on the body surface, which is helpful for the screening of high-risk feet for DM and buys precious time for the early detection and prevention of DF.

[0004] IRT has been widely studied in the field of high-risk diabetic foot screening. Affected by collection specifications, ethical issues, and device configuration, it is generally difficult to obtain tens of thousands of medical infrared thermal images. Less training data will lead to low accuracy in neural network training, such as DFUNet. At present, most of the classification methods for high-risk diabetic feet (or diabetic foot) use deep learning models in the field of deep learning. Through the method of transfer learning, a large and more general image database is first used to train the basic image classification model. When the model can distinguish general categories, its known image classification knowledge is transferred, and this knowledge is used to achieve the purpose of learning to identify high-risk diabetic feet from less data.

[0005] However, this also makes it difficult to modify the model according to the characteristics of diabetic patients' feet when training the high-risk diabetic foot classification model. In addition, existing studies have focused more on the local temperature characteristics in infrared thermal images, lacking other dimensional information and easily losing global feature information; improvement is urgently needed. Summary of the invention

[0006] The purpose of an embodiment of the present invention is to provide a diabetic foot prediction method based on multi-view feature fusion, which can make full use of information of different dimensions without affecting the deep learning model architecture, improve the accuracy of diabetic foot prediction, and solve the problem that the prior art pays more attention to the local temperature features in infrared thermal images, lacks other dimensional information, and is prone to losing global feature information.

[0007] The embodiment of the present invention is implemented as follows: a diabetic foot prediction method based on multi-view feature fusion, the method comprising the following steps:

[0008] Acquire sample information of diabetic foot to be predicted, the sample information including infrared thermal image of foot and color image of foot;

[0009] Build a deep learning model: Based on the deep learning model generated by transfer learning, apply the receptive field optimization algorithm to build a temperature distribution feature enhancement module, apply the receptive field optimization algorithm to build a skin texture feature enhancement module, build a spatial attention enhancement module, and a multi-view feature fusion module to obtain the deep learning model to be trained;

[0010] Fusion of multi-view features: Enhance the temperature distribution features of the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhance the skin texture features in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; perform spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fuse the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map;

[0011] Model training: inputting the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model;

[0012] The diabetic foot prediction model is used to predict the sample information of the diabetic foot to be predicted and output the prediction result.

[0013] Another object of an embodiment of the present invention is to provide a diabetic foot prediction device based on multi-view feature fusion, the device comprising: an information acquisition unit, a module construction unit, a multi-view feature fusion unit, a model training unit and a prediction result output unit;

[0014] The information acquisition unit is used to acquire sample information of diabetic foot to be predicted, and the sample information includes infrared thermal images of the foot and color images of the foot;

[0015] The module construction unit, based on the deep learning model generated by transfer learning, applies the receptive field optimization algorithm to construct a temperature distribution feature enhancement module, applies the receptive field optimization algorithm to construct a skin texture feature enhancement module, constructs a spatial attention enhancement module, and a multi-view feature fusion module to obtain a deep learning model to be trained;

[0016] The multi-view feature fusion unit performs temperature distribution feature enhancement on the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhances the skin texture feature in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; performs spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fuses the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map;

[0017] The model training unit inputs the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updates the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model;

[0018] The prediction result output unit predicts the sample information of diabetic foot to be predicted by using the diabetic foot prediction model and outputs the prediction result.

[0019] Another object of an embodiment of the present invention is to provide a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0020] The embodiment of the present invention provides a diabetic foot prediction method based on multi-view feature fusion. Compared with the prior art, the method has the following beneficial effects:

[0021] On the basis of the existing classic neural network classification algorithm, the characteristics of human feet are fully considered, and the multi-view feature fusion module is applied to improve the classification accuracy of foot infrared thermal images, providing key technical support for the early screening of diabetic foot (or diabetic high-risk foot);

[0022] When performing feature enhancement, the number of multi-layer perceptrons is determined by the receptive field optimization algorithm, aiming to combine the medical definition of foot angiosomes to make the maximum receptive field of the temperature distribution feature enhancement module less than or equal to the size of the foot area to be enhanced, enhance the extracted foot temperature distribution features, reduce the interference of background information on the foot temperature distribution features, and improve the prediction accuracy;

[0023] Multiple enhancement modules in the deep learning model can make full use of information from different dimensions to improve the accuracy of diabetic foot prediction;

[0024] Each enhancement module in the deep learning model is pluggable, and does not require major modifications to the architecture of the deep learning model generated by transfer learning. It has high compatibility and can be directly added to the classification neural network of other foot infrared thermal images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a diabetic foot prediction method based on multi-view feature fusion provided by an embodiment of the present invention;

[0026] Figure 2 is an infrared thermal image of a foot in one embodiment;

[0027] Figure 3 A structural diagram of a temperature distribution feature enhancement module in one embodiment;

[0028] Figure 4 is a color image of a foot in one embodiment;

[0029] Figure 5 A structural diagram of a skin texture feature enhancement module in one embodiment;

[0030] Figure 6 is a structural diagram of a spatial attention enhancement module in one embodiment;

[0031] Figure 7 is a structural diagram of a multi-view feature fusion module in one embodiment;

[0032] Figure 8 is a multi-view feature fusion feature map in one embodiment;

[0033] Fig. 9 A flowchart of a receptive field optimization algorithm provided by an embodiment of the present invention;

[0034] Fig.10 A structural block diagram of a diabetic foot prediction device based on multi-view feature fusion provided by an embodiment of the present invention;

[0035] Fig.11 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] The present invention has noticed in practice that: arteriosclerosis of the microvessels in the foot leads to poor local blood flow in the foot, oxygen and nutrient delivery disorders, and thickening and roughening of the skin; autonomic nerve dysfunction in the foot leads to reduced sweating of the foot and dry skin, which is prone to skin cracking. These features are difficult to distinguish in the infrared thermal image of the foot, but are preserved in the color image of the foot. However, current research focuses more on the local temperature features in the infrared thermal image, and does not pay attention to the information contained in the color image of the foot.

[0038] In the infrared thermal image of the foot, the temperature distribution of the foot of a healthy person tends to gradually decrease from the sole to the surrounding area, often showing a symmetrical butterfly-shaped pattern; due to partial blood vessel blockage, the temperature distribution of the foot of DM patients is uneven, often showing mottled or discrete color changes. Deep learning models generally focus on extracting local temperature features of the image, and do not consider the overall temperature distribution of the foot based on the characteristics of the foot temperature distribution.

[0039] When the receptive field is too small than the foot area to be enhanced, the model will obtain too much local information and lose global information; when the receptive field is too large than the foot area to be enhanced, the model will extract background features and interfere with the foot temperature features. The relationship between the receptive field and the foot area to be enhanced was not considered when designing the deep learning model; the existing technology needs to be improved.

[0040] Figure 1 A flowchart of a diabetic foot prediction method based on multi-view feature fusion provided by an embodiment of the present invention may specifically include steps S101 to S109:

[0041] Step S101, obtaining sample information of diabetic foot to be predicted, the sample information including infrared thermal image of foot and color image of foot; wherein the infrared thermal image of foot and color image of foot can be taken on-site, or taken and stored in advance, and no specific limitation is made here.

[0042] In some examples, after the sample information is obtained and preprocessed, it is not necessarily possible to obtain a highly accurate prediction result; this is because the current classification of diabetic foot is generally based on machine learning or deep learning technology, which has defects: when classifying diabetic foot based on machine learning technology, it is necessary to manually select features and cannot fully utilize the information in the infrared thermal map; when classifying diabetic foot based on deep learning technology, the features in the infrared thermal map of the foot can be fully utilized, but due to the difficulty in obtaining medical infrared thermal images, transfer learning methods need to be used when building the model, resulting in the inability to make major modifications to the deep learning neural network architecture; and existing studies have focused more on local temperature features in infrared thermal maps and lack information in other dimensions. For this reason, it is necessary to make full use of information in different dimensions without affecting the architecture of the deep learning model to improve the accuracy of prediction of high-risk diabetic foot;

[0043] Therefore, in one example, in order to make full use of information of different dimensions, different enhancement modules can be loaded in a pluggable manner in the deep learning model, such as a temperature distribution feature enhancement module, a skin texture feature enhancement module, and so on.

[0044] In one embodiment, the method includes step S103, building a deep learning model: based on the deep learning model generated by using transfer learning, applying a receptive field optimization algorithm to build a temperature distribution feature enhancement module, applying a receptive field optimization algorithm to build a skin texture feature enhancement module, building a spatial attention enhancement module, and a multi-view feature fusion module to obtain a deep learning model to be trained;

[0045] In this step, the temperature distribution feature enhancement module satisfies:

[0046]

[0047] Among them, M temp is the temperature distribution enhancement feature map, F thr Infrared thermal image of the foot. is the nth multi-layer perceptron used to enhance the temperature distribution characteristics. The number of perceptrons is determined by the receptive field optimization algorithm, and σ is the activation function.

[0048] The skin texture feature enhancement module satisfies:

[0049]

[0050] Among them, M skin is the foot skin texture feature map, F col For foot color images, is the nth multi-layer perceptron used to enhance skin texture features. The number of perceptrons is determined by the receptive field optimization algorithm, and σ is the activation function.

[0051] It can be understood that the temperature distribution feature enhancement module and the skin texture feature enhancement module are both constructed by applying the receptive field optimization algorithm; it is easy to know that some parameters in equations (1) and (2) may be the same, such as the activation function σ, which can both use the PRelu function.

[0052] The calculation formula of the spatial attention enhancement module is shown in the following formula (3):

[0053]

[0054] Among them, M spc is the spatial attention enhancement module, is the 2D spatial attention feature map of the foot color image. MaxPool() and AvgPool() are maximum pooling and average pooling respectively. 3×3 is the convolution kernel size, and σ is the PRelu activation function.

[0055] The multi-view feature fusion module satisfies:

[0056] M fuse =σ(F thr +M temp +Con(M skin , M pos )) (4);

[0057] Among them, M fuse is the multi-view feature fusion feature map, M temp is the temperature distribution enhancement feature map, M skin is the foot skin texture feature map, M pos is the feature map enhanced by the foot position information, and σ is the activation function.

[0058] In this step, if Figures 2 to 8 As shown, through the temperature distribution feature enhancement module, the foot infrared thermal map (such as Figure 2 ) to enhance the temperature distribution characteristics and obtain a temperature distribution enhancement feature map; S102: enhance the foot color image (as shown in the figure) by the skin texture feature enhancement module. Figure 4 The skin texture features in the image are used to obtain a foot skin texture feature map; the spatial attention enhancement module is used to perform spatial attention enhancement on the collected foot color image to generate a foot position information enhancement feature map; then, the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map are spliced / fused through the constructed multi-view feature fusion module to generate a multi-view feature fusion feature map (see Figure 8 ).

[0059] In a specific embodiment, a multi-layer perceptron of the temperature distribution feature enhancement module is composed of 4 convolutional layers, and the output feature numbers of each convolutional layer are 64, 8, 12, and 64 respectively, wherein the first layer is a convolutional layer with a convolution kernel size of (1,1), the second layer is a hole convolution layer with a convolution kernel size of (3,3), the fourth layer is a convolutional layer with a convolution kernel size of (3,1). The outputs of all multi-layer perceptrons will be concatenated and integrated using a fully connected layer, and finally the integration result will be tensor added with the result of the foot infrared heat map processed by the convolution layer with a convolution kernel size of (1,1) to obtain the enhanced feature map. Except for the last convolutional layer, each convolutional layer will perform batch normalization (BN) and use the PReLU function for activation, such as Figure 3 shown.

[0060] In a specific embodiment, a multi-layer perceptron of the skin texture feature enhancement module is composed of 4 convolutional layers, and the output feature numbers of each convolutional layer are 64, 16, 24, and 64 respectively, and the convolution kernel sizes of the convolutional layers are (1, 1), (3, 1), (1, 3), and (3, 3) respectively. The outputs of all multi-layer perceptrons will be concatenated and integrated using a fully connected layer. Finally, the integration result and the result of the foot infrared thermal map processed by the convolution layer with a convolution kernel size of (1, 1) are tensor added to obtain the enhanced feature map. Except for the fully connected layer, the output results of each convolutional layer will be batch normalized and activated using the PReLU function, such as Figure 5 shown.

[0061] Step S105, fusing multi-view features: performing temperature distribution feature enhancement on the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhancing the skin texture features in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; performing spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fusing the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map;

[0062] In this step, on the one hand, the features in the infrared thermal map of the foot, the local temperature features in the external thermal map and other dimensional information are fully utilized through enhancement modules such as the temperature distribution feature enhancement module and the skin texture feature enhancement module; on the other hand, the accuracy of prediction of high-risk diabetic feet is improved by fusing multi-view features; on the other hand, the receptive field optimization algorithm used can dynamically adjust the size of the receptive field when enhancing the temperature distribution features for feet of different sizes, ensure the effectiveness of the temperature distribution features, and synergistically improve the accuracy of prediction.

[0063] Step S107, model training: inputting the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model;

[0064] In this step, the multi-view feature fusion feature map is iteratively updated until the deep learning model to be trained is trained. Specifically, steps S101 to S105 are iteratively executed until the ratio of predicted results to actual results does not increase, and the iteration is stopped to obtain a diabetic foot prediction model; wherein, sample information may need to be added or updated, and the parameters of each module that fuses multi-view features can be adjusted as needed.

[0065] In some cases, steps S101 to S105 are iteratively executed, and each round of iteration calculates the number of values ​​that are the same as the actual results predicted by the current classifier according to the loss function, and uses the optimizer to update the parameters of each enhancement module in S101 to S105 and the parameters in the deep learning model. The iteration is stopped until the proportion of predicted results that are the same as the actual results does not increase, and a diabetic foot prediction model is obtained.

[0066] In some cases, the loss function includes but is not limited to a 0-1 loss function, a logarithmic loss function, a square loss function MSE (mean square error), a Hinge loss function, and a cross-entropy loss function.

[0067] In the specific implementation process, the loss function used can be a cross entropy loss function. The calculation formula is as follows:

[0068] loss=-[ylogy 1 +(1-y)log(1-y 1 )](5);

[0069] Among them, loss represents the difference between the true sample label and the predicted probability, y is the true label value, and y 1 The probability value predicted by the classifier.

[0070] Step S109: predicting the sample information of diabetic foot to be predicted by the diabetic foot prediction model and outputting the prediction result.

[0071] In this step, the sample information of diabetic foot to be predicted also includes infrared thermal images of the foot and color images of the foot; in some scenarios, the acquired sample information can be divided to obtain a training data set and a test data set; and the training data set and the test data set can be subjected to feature enhancement and multi-view feature fusion processing; thereafter, data input is performed; in some scenarios, the sample information of diabetic foot to be predicted including infrared thermal images of the foot and color images of the foot can be directly predicted and input into a trained diabetic foot prediction model, which classifies the input sample information and outputs a prediction result; the prediction result represents the symptoms of diabetic foot.

[0072] Specifically, in one embodiment, the prediction accuracy of the diabetic foot prediction model of this embodiment is compared with the existing deep learning model (comparison model 1), and the comparison results in Table 1 are obtained.

[0073] Table 1 is a comparison table of the prediction accuracy of ConvNeXt of the present invention and the prior art

[0074] Deep Learning Models Accuracy (%) Comparison Model 1 (ConvNeXt) 95.52 Diabetic foot prediction model 97.01

[0075] Table 1 above shows the comparison of the classification accuracy before and after using the multi-view feature fusion module when using the ConvNeXt neural network to classify normal feet and diabetic feet. It can be seen that the ConvNeXt with the multi-view feature fusion module (i.e., the diabetic foot prediction model of this method) shows a higher accuracy, indicating that the multi-scale temperature feature fusion module can improve the classification accuracy of the foot infrared thermal image.

[0076] In one embodiment, the deep learning model is a deep learning neural network, including but not limited to a VGG model, a residual convolutional neural network model (ResNet), a Mobilenet model, a DenseNet model, etc.

[0077] like Figure 7 , Figure 8 As shown, in one embodiment, the multi-view feature fusion module will splice the foot skin texture feature map and the foot position information enhancement feature map, and integrate them using a fully connected layer. The integration result is fused with the temperature distribution enhancement feature map by taking the average, and the fusion result is processed with the foot infrared thermal map using a convolution layer with a convolution kernel size of (1,1) to perform tensor addition to obtain a multi-view feature fusion feature map; the multi-view feature fusion feature map can be input into a deep learning model generated by transfer learning for training.

[0078] In one embodiment, during the training of the deep learning model, an optimizer can be used to adaptively adjust the relevant parameters of the deep learning model, the temperature distribution feature enhancement module, the skin texture feature enhancement module, the spatial attention enhancement module, and the relevant parameters of the multi-view feature fusion module, so as to complete the training of the deep learning model faster.

[0079] In one embodiment, the method further comprises:

[0080] In the step of iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained is trained and the diabetic foot prediction model is obtained, the relevant parameters of the deep learning model to be trained are updated using an optimizer;

[0081] The optimizer includes one or more of a stochastic gradient descent optimizer, an adaptive gradient optimizer, and an exponentially weighted moving average optimizer.

[0082] It is understandable that the optimizer includes but is not limited to stochastic gradient descent optimizer (SGD), adaptive gradient optimizer (AdaGrad), and exponentially weighted moving average optimizer (RMSProp).

[0083] In a preferred embodiment, the optimizer used is the Stochastic Gradient Descent Optimizer (SGD).

[0084] In one embodiment, the multi-view feature fusion module satisfies the above formula (4); the multi-layer perceptron includes a hole convolution layer, wherein the hole rate parameter dn of the nth multi-layer perceptron takes the nth prime number;

[0085] Alternatively, the multilayer perceptron includes at least one convolutional layer, multiple batch normalization layers, and at least one activation function layer.

[0086] In an example of this embodiment, the multilayer perceptron includes a dilated convolution layer, wherein the dilated rate parameter dn of the nth multilayer perceptron takes the nth prime number;

[0087] In an example of this embodiment, the multilayer perceptron includes at least one convolutional layer, multiple batch normalization layers, and at least one activation function layer; the multiple batch normalization layers can specifically be 2, 3, 5, etc.

[0088] In some cases, the spatial attention enhancement module such as Figure 6 As shown in the figure, it includes a serial maximum pooling layer and an average pooling layer, as well as a convolution layer with a convolution kernel size of (3,3), and finally activated by the PRelu function.

[0089] In some cases, the structure of the multi-view feature fusion module is as follows Figure 7As shown in the figure, the multi-view feature fusion module will splice the foot skin texture feature map and the foot position information enhancement feature map and integrate them using a fully connected layer. The integration result is fused with the temperature distribution enhancement feature map by taking the average. The fusion result is processed with the foot infrared thermal map using a convolution layer with a convolution kernel size of (1,1) and tensor addition is performed to obtain a multi-view feature fusion feature map.

[0090] like Fig. 9 As shown, in another embodiment, a temperature distribution feature enhancement module is constructed by applying a receptive field optimization algorithm, specifically including the following steps S1011 to S1017:

[0091] S1011, determining the number of foot infrared thermal images input in each batch according to preset batch processing parameters;

[0092] S1013, calculating the number of pixels of each batch of input foot infrared thermal images, and sorting the number of pixels of each foot infrared thermal image;

[0093] S1015, finding the minimum external matrix for the foot infrared thermal image with the least number of pixels after sorting, and finding the number n of multi-layer perceptrons inside the temperature distribution feature enhancement module according to the perceptron number formula;

[0094] The formula for the number of perceptrons satisfies:

[0095]

[0096] Among them, x and y are the length and width of the minimum external matrix of the foot infrared thermal image, respectively;

[0097] S1017. Use n parallel multi-layer perceptrons to perform feature enhancement on the input foot infrared thermal image.

[0098] Specifically, assuming that the number of foot infrared thermal images input in each batch is m=4, and the length and width of the minimum external matrix of the foot infrared thermal image with the least number of pixels are 173 and 75 respectively, then n=2 is calculated, indicating that the temperature distribution enhancement module and the foot skin texture feature enhancement module each have 2 multi-layer perceptrons. And the hollow rate parameter of the first multi-layer perceptron of the temperature distribution enhancement module is r=2, and the hollow rate parameter of the second multi-layer perceptron is r=3.

[0099] In one embodiment, the infrared thermal map of the foot is obtained by an infrared thermal map acquisition device, which can reflect the temperature distribution of the foot; the color image of the foot is obtained by a color image acquisition device, which has the same shooting angle as the infrared thermal map of the foot, wherein the color image of the foot contains skin texture features.

[0100] In some cases, the infrared thermal image acquisition device may be an infrared thermal imager; and the color image acquisition device may be a conventional digital camera.

[0101] like Fig.10 As shown, in another embodiment, a diabetic foot prediction device based on multi-view feature fusion is used in the method as described above, and the device includes: an information acquisition unit 100, a module construction unit 200, a multi-view feature fusion unit 300, a model training unit 400 and a prediction result output unit 500;

[0102] The information acquisition unit 100 is used to acquire sample information of diabetic foot to be predicted, and the sample information includes infrared thermal images of the foot and color images of the foot;

[0103] The module construction unit 200, based on the deep learning model generated by transfer learning, applies a receptive field optimization algorithm to construct a temperature distribution feature enhancement module, applies a receptive field optimization algorithm to construct a skin texture feature enhancement module, constructs a spatial attention enhancement module, and a multi-view feature fusion module to obtain a deep learning model to be trained;

[0104] The multi-view feature fusion unit 300 performs temperature distribution feature enhancement on the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhances the skin texture feature in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; performs spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fuses the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map;

[0105] The model training unit 400 inputs the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updates the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model;

[0106] The prediction result output unit 500 predicts the diabetic foot to be predicted by using the diabetic foot prediction model and outputs the prediction result.

[0107] In this embodiment, the information acquisition unit 100 and the prediction result output unit 500 may be a computer program or a computer executable instruction, characterized as a communication interface that can perform data or information input / output; the prediction result output unit 500 may be a fully connected network layer of a conventional deep learning model, and as the output of the deep learning model, the prediction result of the present invention may be output; the module construction unit 200 may be an application environment of a deep learning model, such as a computer device, a preset program of a server or a cloud server, etc.; the module construction unit 200, the multi-view feature fusion unit 300 and the model training unit 400, etc. are all a computer program or a computer executable instruction, and are not specifically limited here.

[0108] like Fig.11 As shown, in another embodiment, a computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs steps S101 to S109 of the method described above:

[0109] S101, obtaining sample information of diabetic foot to be predicted, where the sample information includes infrared thermal image of foot and color image of foot;

[0110] S103, constructing a deep learning model: based on the deep learning model generated by transfer learning, applying a receptive field optimization algorithm to construct a temperature distribution feature enhancement module, applying a receptive field optimization algorithm to construct a skin texture feature enhancement module, constructing a spatial attention enhancement module, and a multi-view feature fusion module, to obtain a deep learning model to be trained;

[0111] S105, fusing multi-view features: performing temperature distribution feature enhancement on the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhancing the skin texture features in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; performing spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fusing the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map;

[0112] S107, model training: inputting the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained completes training, thereby obtaining a diabetic foot prediction model;

[0113] S109, predicting the sample information of diabetic foot to be predicted by the diabetic foot prediction model and outputting the prediction result.

[0114] In this embodiment, the deep learning model generated by transfer learning is a prior art and will not be described in detail here; the infrared thermal image of the foot and the color image of the foot can be obtained and electronically stored through existing camera equipment; such as infrared thermal imagers, digital cameras, etc.

[0115] In this embodiment, the step of iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained completes the training is not limited to updating the foot infrared thermal map, foot color image, or enhancing the foot infrared thermal map, foot color image; it can also be a combination of the two.

[0116] In addition, when training a deep learning model, the parameters of the deep learning model may be adjusted, and the parameters of each enhancement module in the deep learning model may also be adjusted, but this embodiment is not limited thereto.

[0117] The present embodiment provides a diabetic foot prediction method based on multi-view feature fusion. On the basis of the existing classical neural network classification algorithm, it fully considers the characteristics of the human foot, applies a multi-view feature fusion module, improves the classification accuracy of the infrared thermal image of the foot, and provides key technical support for the early screening of diabetic foot. When performing feature enhancement, the number of multi-layer perceptrons is determined by a receptive field optimization algorithm, aiming to combine the medical definition of the foot vascular corpuscle to make the maximum receptive field of the temperature distribution feature enhancement module less than or equal to the size of the foot area to be enhanced, enhance the extracted foot temperature distribution features, reduce the interference of background information on the foot temperature distribution features, and improve the prediction accuracy. The multiple enhancement modules in the deep learning model can make full use of information of different dimensions to improve the accuracy of diabetic foot prediction. The enhancement modules in the deep learning model are pluggable, and there is no need to make major modifications to the architecture of the deep learning model generated by transfer learning. It has high compatibility and can be directly added to the classification neural network of other foot infrared thermal images.

[0118] Fig.11An internal structure diagram of a computer device in an embodiment is shown. The computer device includes a processor, a memory, a network interface, an input device and a display screen connected via a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement a diabetic foot prediction method based on multi-view feature fusion. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may execute a diabetic foot prediction method based on multi-view feature fusion. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0119] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0121] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A diabetic foot prediction method based on multi-view feature fusion, characterized in that: The method comprises the following steps: Acquire sample information of diabetic foot to be predicted, the sample information including infrared thermal image of foot and color image of foot; Build a deep learning model: Based on the deep learning model generated by transfer learning, apply the receptive field optimization algorithm to build a temperature distribution feature enhancement module, apply the receptive field optimization algorithm to build a skin texture feature enhancement module, build a spatial attention enhancement module, and a multi-view feature fusion module to obtain the deep learning model to be trained; Fusion of multi-view features: Enhance the temperature distribution features of the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhance the skin texture features in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; perform spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fuse the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map; Model training: inputting the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model; The diabetic foot prediction model is used to predict the sample information of the diabetic foot to be predicted and output the prediction result.

2. The method according to claim 1, characterized in that The temperature distribution feature enhancement module satisfies: Among them, M temp is the temperature distribution enhancement feature map, F thr Infrared thermal image of the foot. is the nth multi-layer perceptron used to enhance the temperature distribution characteristics. The number of perceptrons is determined by the receptive field optimization algorithm, and σ is the activation function.

3. The method according to claim 2, characterized in that The skin texture feature enhancement module satisfies: Among them, M skin is the foot skin texture feature map, F col For foot color images, is the nth multi-layer perceptron used to enhance skin texture features. The number of perceptrons is determined by the receptive field optimization algorithm, and σ is the activation function.

4. The method according to claim 3, characterized in that The multi-view feature fusion module satisfies: M fuse =σ(F thr +M temp +Con(M skin ,M pos )); Among them, M fuse is the multi-view feature fusion feature map, M temp is the temperature distribution enhancement feature map, M skin is the foot skin texture feature map, M pos is the feature map enhanced by the foot position information, and σ is the activation function.

5. The method according to claim 2, characterized in that: The multi-layer perceptron includes a dilated convolution layer, wherein the dilated rate parameter dn of the n-th multi-layer perceptron takes the n-th prime number; Alternatively, the multilayer perceptron includes at least one convolutional layer, multiple batch normalization layers, and at least one activation function layer.

6. The method according to claim 1, characterized in that The receptive field optimization algorithm is applied to construct a temperature distribution feature enhancement module for temperature distribution feature enhancement; specifically, the following steps are included: Determine the number of foot infrared thermal images input in each batch according to preset batch processing parameters; Calculate the number of pixels of each batch of input foot infrared thermal images, and sort the number of pixels of each foot infrared thermal image; The minimum external matrix is ​​calculated for the foot infrared thermal image with the least number of pixels after sorting, and the number n of multi-layer perceptrons inside the temperature distribution feature enhancement module is calculated according to the perceptron number formula; The formula for the number of perceptrons satisfies: Among them, x and y are the length and width of the minimum external matrix of the foot infrared thermal image, respectively; Use n parallel multi-layer perceptrons to enhance the features of the input foot infrared thermal image.

7. The method according to claim 1, characterized in that The method further comprises: In the step of iteratively updating the multi-view feature fusion feature map until the deep learning model to be trained is trained and the diabetic foot prediction model is obtained, the relevant parameters of the deep learning model to be trained are updated using an optimizer; The optimizer includes one or more of a stochastic gradient descent optimizer, an adaptive gradient optimizer, and an exponentially weighted moving average optimizer.

8. The method according to claim 1, characterized in that: The foot infrared thermal map is obtained by an infrared thermal map acquisition device, which can reflect the temperature distribution of the foot; the foot color image is obtained by a color image acquisition device, which has the same shooting angle as the foot infrared thermal map, wherein the foot color image contains skin texture features.

9. A diabetic foot prediction device based on multi-view feature fusion, characterized in that: The device comprises: an information acquisition unit, a module construction unit, a multi-view feature fusion unit, a model training unit and a prediction result output unit; The information acquisition unit is used to acquire sample information of diabetic foot to be predicted, and the sample information includes infrared thermal images of the foot and color images of the foot; The module construction unit, based on the deep learning model generated by transfer learning, applies the receptive field optimization algorithm to construct a temperature distribution feature enhancement module, applies the receptive field optimization algorithm to construct a skin texture feature enhancement module, constructs a spatial attention enhancement module, and a multi-view feature fusion module to obtain a deep learning model to be trained; The multi-view feature fusion unit performs temperature distribution feature enhancement on the foot infrared thermal image through the temperature distribution feature enhancement module to obtain a temperature distribution enhancement feature map; enhances the skin texture feature in the foot color image through the skin texture feature enhancement module to obtain a foot skin texture feature map; performs spatial attention enhancement on the foot color image through the spatial attention enhancement module to generate a foot position information enhancement feature map; fuses the foot infrared thermal image, the temperature distribution enhancement feature map, the foot skin texture feature map and the foot position information enhancement feature map through the multi-view feature fusion module to generate a multi-view feature fusion feature map; The model training unit inputs the multi-view feature fusion feature map into the deep learning model to be trained for training; iteratively updates the multi-view feature fusion feature map until the deep learning model to be trained completes the training, thereby obtaining a diabetic foot prediction model; The prediction result output unit predicts the sample information of diabetic foot to be predicted by using the diabetic foot prediction model and outputs the prediction result.

10. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.

Images