Diabetic foot neuropathy typing early warning method and system based on multi-modal data

Abstract

The application discloses a diabetic foot neuropathy typing early warning method and system based on multi-modal data, and belongs to the technical field of intelligent medical treatment and digital health. The method comprises the following steps: collecting multi-modal data such as foot infrared thermal images and plantar pressure; constructing a temperature field and a pressure field and uniformly mapping them to a standardized coordinate system, and segmenting them into multiple anatomical regions; extracting a comprehensive feature vector of each region; obtaining high-dimensional features through a shared coding network; inputting the high-dimensional features into a double-branch neural network, with a typing branch outputting neuropathy type probabilities and a positioning branch outputting abnormal region probability masks; and fusing the two to determine and output typing results and positioning information. The application realizes the collaborative optimization of typing and positioning through a double-branch network architecture, can accurately identify autonomic and motor neuropathies in the early stage, and can perform regional positioning, thereby providing support for early warning and personalized intervention of diabetic foot.

Description

Technical Field

[0001] This application relates to the fields of intelligent medical and digital health technology, and in particular to a method and system for early warning of diabetic foot neuropathy based on multimodal data. Background Technology

[0002] Diabetic foot is a common and serious chronic complication of diabetes. Its development is closely related to abnormalities in the peripheral nerves of the foot, changes in microcirculation, and biomechanical abnormalities. Pathologically, autonomic and motor nerve dysfunction usually precedes obvious sensory nerve damage. Autonomic nerve dysfunction affects the foot's blood flow regulation and temperature control, leading to abnormal local temperature distribution; motor nerve dysfunction can cause foot muscle imbalance, gait changes, and abnormal plantar pressure distribution. The long-term cumulative effect of these changes can easily create high-risk areas, further inducing tissue damage and even ulcers.

[0003] In existing technologies, the assessment of diabetic foot risk often employs a single modality or a few indicators for analysis. For example, it may rely on infrared thermography to identify local temperature abnormalities (e.g., CN111265188A) or plantar pressure to identify stress abnormalities (e.g., US11417425B2). While these methods can provide some indication of risk, they typically offer a rough assessment of the overall foot condition. They struggle to achieve collaborative analysis between different modalities under a unified spatial reference, and also find it difficult to simultaneously identify neuropathy types and locate abnormal sites at a regional scale. This is especially true in the early stages, when sensory neuropathy has not yet fully developed, but autonomic and motor nerve dysfunctions are already manifesting as abnormal temperature control and abnormal mechanical loading, respectively. In these cases, existing technologies struggle to finely differentiate between different pathological mechanisms and provide corresponding regional warning information. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method and system for classifying and warning diabetic foot neuropathy based on multimodal data, thereby resolving the problems of insufficient multimodal fusion, weak regional analysis capabilities, inaccurate early lesion identification, and unclear abnormal area localization in the prior art.

[0005] The first technical solution adopted in this application is: providing a method for early warning of diabetic foot neuropathy based on multimodal data, including the following steps: Multimodal data of the subject's feet were collected, including infrared thermographic images, plantar pressure distribution data, and foot temperature data. The multimodal data are respectively constructed into multiple physical fields including at least temperature field and pressure field, and the multiple physical fields are uniformly mapped to a preset foot standardized coordinate system. Under the foot standardized coordinate system, the foot is divided into multiple anatomical regions. For each anatomical region, corresponding regional features are extracted from the multiple physical fields, and the regional features of the same region are combined to form a comprehensive feature vector of that region. A shared-parameter encoding network is used to encode the comprehensive feature vector of all regions to obtain a high-dimensional feature representation of each region; The high-dimensional feature representations of each region are input into a dual-branch recognition network, which includes a first classification recognition branch and a second localization recognition branch. The first classification recognition branch is used to output the warning probability of the neuropathy classification corresponding to each anatomical region, and the second localization recognition branch is used to output the probability mask of the foot abnormality region. By fusing the classification warning probability and the probability mask, the neuropathy classification warning result and abnormal area location information are determined and output.

[0006] Furthermore, the pressure field includes a static pressure field and a dynamic pressure field. The static pressure field is constructed based on plantar pressure distribution data collected when the subject is standing, and the dynamic pressure field is constructed based on plantar pressure distribution data collected during the subject's walking gait.

[0007] Furthermore, the region features extracted for each anatomical region include: Temperature field characteristics derived from the temperature field, the temperature field characteristics including at least the uniform temperature and temperature variability of the anatomical region; Pressure characteristics from the pressure field, the pressure characteristics including at least the peak pressure and pressure distribution percentage of the anatomical region.

[0008] Furthermore, the regional features extracted for each anatomical region also include dynamic pressure features from the dynamic pressure field, wherein the dynamic pressure features include at least the length of the pressure center trajectory of the anatomical region.

[0009] Furthermore, the first classification identification branch includes two parallel sub-branches, which are used to output the risk probability related to autonomic nervous system dysfunction and the risk probability related to motor nervous system dysfunction, respectively.

[0010] Furthermore, the step of fusing the subtype warning probability and the probability mask includes: determining the neuropathic risk index of the anatomical region based on the risk probability related to the autonomic nervous system dysfunction, the risk probability related to the motor nervous system dysfunction, and the average abnormality probability of the probability mask in the corresponding anatomical region.

[0011] Furthermore, the neuropathy classification warning result is determined based on the comparison results of the risk probability related to the autonomic nervous system dysfunction and the risk probability related to the motor nervous system dysfunction with preset thresholds, respectively. The classification warning result includes autonomic nervous system dysfunction warning, motor nervous system dysfunction warning, or combined dysfunction warning.

[0012] Furthermore, the second localization and recognition branch uses a convolutional neural network and an upsampling structure to generate the probability mask, which is used to indicate the probability of anomalies at each position of the foot corresponding to the foot's standardized coordinate system.

[0013] Furthermore, the dual-branch neural network is trained using a multi-task learning framework, and the total loss function used during training includes a weighted sum of the pattern recognition loss and the localization recognition loss.

[0014] The second technical solution adopted in this application is: providing a diabetic foot neuropathy classification and early warning system based on multimodal data for implementing the aforementioned method, comprising: The data acquisition module is used to collect multimodal data of the subject's feet; The spatial mapping and region partitioning module is used to construct physical fields and perform unified mapping and region partitioning; The feature extraction module is used to extract region features and form a comprehensive feature vector; The feature encoding module is used to obtain high-dimensional feature representations of each region; The dual-branch neural network module is used to input high-dimensional feature representations into the dual-branch neural network to output the fractal warning probability and probability mask; The output module is used to fuse the classification warning probability and the probability mask, and output the neuropathy classification warning result and abnormal area location information.

[0015] Due to the adoption of the above technical solution, this application has at least one of the following beneficial effects compared with the prior art: 1. By mapping infrared thermography, static pressure, and dynamic pressure to a standardized foot coordinate system, multimodal collaborative analysis can be achieved at the regional level, which helps to improve the comparability and fusion effect between different modalities.

[0016] 2. By combining shared coding with dual-branch recognition, a network structure can simultaneously complete classification warning and abnormal area location based on the same feature, which is conducive to improving the accuracy of risk identification and the precision of spatial positioning.

[0017] 3. It can provide early classification and warning of autonomic and motor nerve dysfunction before sensory neuropathy has developed significantly, and output abnormal area indication information, thereby providing support for clinical early screening, follow-up and individualized intervention. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 A schematic diagram of the overall process of a diabetic foot neuropathy classification and early warning method based on multimodal data provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the unified mapping of multimodal data to a standardized foot coordinate system and the division of anatomical regions provided in an embodiment of this application; Figure 3 This is a schematic diagram of a shared coding and dual-branch recognition fusion network structure provided in an embodiment of this application; Figure 4 This is a schematic diagram of a module of a diabetic foot neuropathy classification and early warning system based on multimodal data, provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] Existing technologies often rely on single-modal data or adopt a holistic-to-local analysis strategy, making it difficult to deeply integrate multimodal information such as infrared thermography, static and dynamic pressure within a unified foot spatial reference frame. This results in the inability to accurately distinguish between abnormal mechanisms of the autonomic and motor nerves before sensory neuropathy develops, and also lacks fine-grained synchronous localization of abnormal areas.

[0023] In view of this, this application effectively solves the problems of insufficient multimodal fusion, difficulty in early classification, and inaccurate localization by constructing a unified mapping of multiphysics fields to a standardized coordinate system and utilizing a dual-branch neural network to collaboratively output classification probabilities and region probability masks; Figure 1 As shown, Figure 1 A schematic diagram of the overall process of a diabetic foot neuropathy classification and early warning method based on multimodal data provided in an embodiment of this application includes the following steps: S1: Collect multimodal data of the subject's feet, including infrared thermographic images, plantar pressure distribution data, and foot temperature data; under controlled environmental conditions, collect complete foot data from the subject after 15 minutes of rest; the ambient temperature is controlled at 24℃±1℃, and the humidity is controlled at 40%-60%; the collected data includes: infrared thermographic images, plantar pressure distribution data, and temperature time series data; the acquisition equipment includes: Infrared thermal imager: resolution ≥320×240 pixels, temperature accuracy ±0.1°C, used to acquire infrared thermal images of the soles of the feet, providing raw data for constructing temperature fields.

[0024] Plantar pressure plate: Sensor density ≥ 4 / cm², sampling frequency 100-200Hz, used to collect plantar pressure distribution data and provide raw data for constructing the pressure field.

[0025] Temperature sensor array: sampling frequency 1Hz, attached to key areas of the sole of the foot, used to acquire foot temperature time series data with higher time resolution, as a supplement to infrared thermal imaging data.

[0026] By using a synchronization controller or timestamp alignment method, the temporal consistency of multimodal data is ensured, laying the foundation for the subsequent construction of physical fields that include the time dimension.

[0027] S2: Construct multiple physical fields from the multimodal data, each including at least a temperature field and a pressure field, and map these multiple physical fields uniformly to a predefined foot-normalized coordinate system. Within this system, the foot is segmented into multiple anatomical regions. Specifically: Temperature field Spatial distribution and time series of foot temperature based on infrared thermal images.

[0028] pressure field The pressure spatial distribution constructed based on plantar pressure distribution data can be static or a dynamic field that includes a time dimension.

[0029] Multiple physical fields are uniformly mapped to a pre-defined foot-normalized coordinate system through affine transformation and spatial registration. The coordinate transformation formula is as follows:

[0030] in Let be the affine transformation matrix. It is a translation vector; by identifying key anatomical features of the foot (such as the center of the heel, the second metatarsal head, the fifth metatarsal head, and the center of the toe), the transformation parameters are calculated to achieve spatial alignment of multi-source data.

[0031] like Figure 2 As shown, Figure 2 A schematic diagram illustrating the unified mapping of multimodal data to a standardized foot coordinate system and the division of anatomical regions provided in an embodiment of this application; in the unified coordinate system In this study, the foot is divided into N anatomical regions ( , , … ), where N=12-15, including the hindfoot area, midfoot area, and forefoot area. Specific division methods are as follows: Hind foot area ( - ): Inner side of the heel, middle of the heel, outer side of the heel; Midfoot area ( - ): Medial arch, middle arch, lateral arch; Forefoot area ( - ): The metatarsal head regions and toe regions.

[0032] This classification method is used uniformly for all subjects, ensuring the standardization and comparability of regional characteristics.

[0033] S3: For each anatomical region, extract corresponding regional features from multiple physical fields, and combine the regional features of the same region to form a comprehensive feature vector for that region; for each anatomical region The corresponding regional features are extracted from multiple physical fields, and the regional features of the same region are combined to form a comprehensive feature vector of the region.

[0034] Features extracted from the temperature field include at least regional average temperature and temperature variability; features extracted from the pressure field include at least peak pressure and pressure distribution percentage. As optional extensions, features such as left and right foot temperature difference, temperature-time variation coefficient, pressure anomaly index, pressure center trajectory length, pressure distribution entropy, and gait variation coefficient can also be extracted. These extracted features are then combined into a comprehensive feature vector. .

[0035] S4: A shared-parameter encoding network is used to encode the comprehensive feature vector of all regions to obtain a high-dimensional feature representation of each region; a shared-parameter encoding network is used to encode the comprehensive feature vector of all regions. Encoding is performed to obtain high-dimensional feature representations for each region. All regions use the same network and the same set of weight parameters to ensure that features from different regions are mapped to the same feature space, facilitating subsequent comparisons and reducing model parameters.

[0036] In one specific embodiment, the encoding network employs a two-layer fully connected neural network structure: First layer: Input dimensions and Same dimensions (e.g., 9-dimensional), output 64-dimensional, activation function ReLU; Second layer: 64-dimensional input, 32-dimensional output, activation function ReLU; Output: 32-dimensional high-dimensional feature representation .

[0037] The high-dimensional feature representations of each region are input into a dual-branch neural network, which includes a first classification recognition branch and a second localization recognition branch. The first classification recognition branch is used to output the corresponding neuropathy classification warning probability, and the second localization recognition branch is used to output the probability mask of the foot abnormality region.

[0038] The first classification branch outputs the classification probability of the neuropathic lesion type for each region. In a preferred embodiment, this branch includes two parallel sub-branches, which are used to output the classification probability of autonomic neuropathy and the classification probability of motor neuropathy, respectively.

[0039] Second localization and recognition branch: Outputs a probability mask for abnormal foot areas. The mask corresponds to the original foot image space, and the value of each pixel represents the probability that an anomaly exists at that location.

[0040] By fusing classification probabilities and probability masks, the neuropathic morphology classification results and abnormal region localization information are determined and output; specifically, the probability mask output by the localization branch is... Combined with anatomical region division, calculate each region Average anomaly probability within Then, by integrating the subtype probability and location information, the neuropathic risk index (NRI) for each region is calculated. The final output includes neuropathic classification results, spatial localization of abnormal areas, and risk warning information.

[0041] In steps S1 and S2, as a preferred embodiment, the pressure field includes a static pressure field and a dynamic pressure field. The static pressure field is constructed based on plantar pressure distribution data collected when the subject is standing, and the dynamic pressure field is constructed based on plantar pressure distribution data collected during the subject's walking gait. The static pressure field is described in detail below. and dynamic pressure field The construction principle: The static pressure field is constructed based on plantar pressure distribution data collected from subjects while they are standing. In step S1, in addition to collecting routine plantar pressure distribution data, data is specifically collected on subjects standing on the plantar pressure plate in a relaxed and natural posture for at least 5-10 seconds with stable standing data. In step S2, the collected time-series data are averaged to eliminate the influence of minor swaying, resulting in a stable static pressure distribution map; static pressure field. It is a two-dimensional field that does not include the time dimension, reflecting the anatomical structure and mechanical properties of the foot under static load.

[0042] The dynamic pressure field is constructed based on plantar pressure distribution data collected during the subject's walking gait. In step S1, in addition to collecting routine plantar pressure distribution data, data is specifically collected on the subject walking at a normal pace on a plantar pressure plate, collecting continuous pressure frame data for at least three effective gait cycles. In step S2, these continuous pressure frames are constructed into a four-dimensional dynamic pressure field. Includes two-dimensional spatial coordinates Time dimension The pressure value describes the spatiotemporal changes of pressure during the walking process; important dynamic parameters such as the center of pressure (CoP) trajectory can be extracted from the dynamic pressure field.

[0043] The static pressure field mainly reflects the anatomical structure and static mechanical properties of the foot. Abnormalities in the static pressure field (such as excessively high local peak pressure) may be related to foot deformities. The dynamic pressure field mainly reflects the dynamic properties during gait. Abnormalities in the dynamic pressure field (such as abnormal CoP trajectory and abnormal regional contact time) may be related to weakened motor nerve control function. By distinguishing between static and dynamic pressure fields, this application can more comprehensively assess the functional status of the foot and provide richer underlying features for subsequent classification and identification.

[0044] The regional features extracted for each anatomical region include: Temperature field characteristics derived from the temperature field, including at least the uniform temperature and temperature variability of the anatomical region; regional uniform temperature The average temperature of all pixels within the region. The calculation formula is:

[0045] This feature reflects the overall metabolic and blood perfusion levels in the region; autonomic neuropathy can affect vasomotor function, leading to abnormal local temperature—impaired sympathetic nerve function can cause vasodilation and increased temperature; microcirculatory disturbances can lead to reduced blood flow and decreased temperature.

[0046] Temperature variability The standard deviation of temperature within a region reflects the uniformity of temperature within that region. The calculation formula is:

[0047] Healthy feet have a relatively uniform temperature distribution. Autonomic neuropathy may lead to local microcirculatory disturbances, resulting in discrete cold or hot spots within the area, manifested as increased temperature variability. Pressure characteristics from the pressure field, including at least the peak pressure and pressure distribution percentage of the anatomical region. Peak pressure. The maximum pressure value of all pixels within the region. The calculation formula is:

[0048] This characteristic reflects the maximum pressure load borne by the area and is one of the most direct risk factors for foot ulcer formation; excessively high peak pressure is a typical consequence of motor neuropathy leading to foot deformities and gait abnormalities.

[0049] Pressure distribution ratio The proportion of the sum of pressure within a specific area to the total pressure on the entire foot. The calculation formula is:

[0050] This characteristic reflects the distribution of foot load across anatomical regions. Normally, the foot load is primarily borne by the heel and forefoot, with a lower proportion in the midfoot. Motor neuropathy can lead to abnormal load distribution; for example, arch collapse can significantly increase the proportion of load in the midfoot.

[0051] When the pressure field includes a dynamic pressure field, in step S3, as a preferred embodiment, the regional features extracted for each anatomical region also include dynamic pressure features from the dynamic pressure field, and the dynamic pressure features include at least the length of the pressure center trajectory of the anatomical region.

[0052] Pressure center trajectory length This feature indicates that the center of pressure (CoP) passes through this anatomical region during dynamic gait. The length of the trajectory path at that time. The specific calculation method is as follows: First, the pressure center position CoP(t) for each time frame is extracted from the dynamic pressure field; Then, filter out CoP(t) located in the region All points in time within; Finally, the cumulative path length between the CoP locations corresponding to these time points is calculated.

[0053] This characteristic is an important dynamic parameter, directly reflecting the dynamic force characteristics and gait stability of different areas of the foot during gait. A longer center of pressure trajectory length within an anatomical region may mean: This area bears unstable swaying or torsional loads during the gait support phase; Abnormal inversion and supination movements of the foot occur during propulsion; Weakened motor nerve control leads to a decline in gait control ability.

[0054] For example, in a normal gait, the CoP trajectory transitions quickly and smoothly from the heel to the forefoot, traversing each metatarsal head region in a short time and along a straight path. If motor neuropathy leads to decreased control of the foot muscles, the CoP trajectory may become circuitous and elongated, exhibiting longer trajectory lengths in certain areas.

[0055] In a preferred embodiment, the first subtype recognition branch includes two parallel sub-branches, which are used to output the risk probability related to autonomic nervous system dysfunction and the risk probability related to motor nervous system dysfunction, respectively.

[0056] The two sub-branches have the same network structure, but the network parameters are trained independently, and each learns feature patterns related to specific pathological mechanisms.

[0057] Sub-branches of autonomic neuropathy: Input: High-dimensional feature representations of all regions { , , ..., }, each It is 32-dimensional; Multi-head self-attention layer: 4 attention heads, 32 hidden dimensions, used to capture inter-region correlations; Fully connected layer 1: 32-dimensional → 16-dimensional, activation function ReLU; Fully connected layer 2: 16-dimensional → 2-dimensional, activation function Softmax; Output: Each region probability of autonomic neuropathy .

[0058] Motor neuropathy sub-branch: The structure is exactly the same as the sub-branch of the autonomic neuropathy, but the network parameters are independent.

[0059] Output: Each region probability of motor neuropathy .

[0060] By employing two parallel sub-branches, the complex multi-classification problem (autonomous, motor, complex, normal) is decomposed into two relatively independent binary classification problems. Each sub-branch can focus on learning feature patterns related to its specific pathological mechanism: The autonomic nervous system branches pay more attention to temperature field characteristics, such as abnormal average temperature, increased temperature variability, and temperature difference between the left and right feet; Motor nerve branches are more focused on pressure field characteristics, such as peak pressure, pressure distribution percentage, and CoP trajectory length. This is achieved through separate outputs. and The system can identify complex neuropathies—when and When both are high, it suggests that two pathological mechanisms exist simultaneously in that area.

[0061] The steps of fusing the subtype warning probability and the probability mask include: determining the neuropathic risk index of the anatomical region based on the risk probability associated with autonomic nervous system dysfunction, the risk probability associated with motor nervous system dysfunction, and the average abnormality probability of the probability mask in the corresponding anatomical region.

[0062] In step S6, as a specific fusion method, this application achieves deep fusion of classification probability and probability mask by calculating the neuropathic risk index (NRI) of each anatomical region.

[0063] For each anatomical region NRI is a comprehensive quantitative indicator that takes into account the probability of autonomic neuropathy. The probability of motor neuropathy The average value of the probability mask of the localization branch output within this region. .

[0064] In a preferred embodiment, NRI is calculated using a weighted summation method:

[0065] in: , , The preset weighting coefficients satisfy... + + =1.

[0066] The specific values ​​of the weighting coefficients can be adjusted based on clinical needs and model validation results. In one specific embodiment, the following settings are provided: =0.3, =0.3, =0.4, giving location information a slightly higher weight. This can be adjusted as needed in other application scenarios. Fractal priority mode: =0.4, =0.4, =0.2, with a greater focus on differentiating lesion types; Location-first mode: =0.2, =0.2, =0.6, which focuses more on the spatial location of abnormal areas.

[0067] The neuropathy classification and early warning results are determined by comparing the risk probabilities related to autonomic nervous system dysfunction and motor nervous system dysfunction with preset thresholds, respectively. The classification and early warning results include early warning for autonomic nervous system dysfunction, early warning for motor nervous system dysfunction, or early warning for combined dysfunction.

[0068] The preset threshold can be determined based on clinical experience and model validation results, and is usually set to 0.5. However, it can also be adjusted according to the specific application scenario. For high-specificity scenarios: If it is necessary to reduce the false positive rate, the threshold can be increased to 0.6 or 0.7; For high-sensitivity scenarios: If you need to improve the detection rate, you can lower the threshold to 0.4 or 0.3.

[0069] Given a preset threshold of 0.5, the correspondence between classification probability and lesion type can be determined using the following table:

[0070] The second localization and recognition branch uses a convolutional neural network and an upsampling structure to generate a probability mask. The probability mask is used to indicate the probability of anomalies at each position of the foot corresponding to the foot's standardized coordinate system.

[0071] This branch is responsible for the spatial localization of abnormal regions, and adopts a network structure similar to semantic segmentation:

[0072] Output: Anomaly probability mask The closer the value is to 1, the higher the probability that the position is abnormal; the closer the value is to 0, the higher the probability that the position is normal.

[0073] To ensure the spatial coherence of the localization results, a spatial smoothing regularization term is applied to the localization branch during training: .

[0074] The dual-branch recognition network is trained using a multi-task learning framework. The total loss function used during training includes a weighted sum of the subtype recognition loss and the localization recognition loss. For the autonomic neuropathy sub-branch and the motor neuropathy sub-branch, cross-entropy loss is used respectively.

[0075] in The true label for region Rᵢ (k=0 indicates normal, k=1 indicates lesion).

[0076] The localization branch loss combines binary cross-entropy (BCE) and intersection-over-union (IoU) loss:

[0077] The model employs a multi-task learning framework, jointly optimizing the classification loss and localization loss:

[0078] Cross-entropy loss for classifying autonomic neuropathy, weighting coefficients =1.0; Cross-entropy loss for classifying motor neuropathy = 1.0; For the localization branch, BCE loss + IoU loss, = 0.5; For spatial smoothing regularization, = 0.1.

[0079] This application also provides a diabetic foot neuropathy classification and early warning system based on multimodal data, used to implement the diabetic foot neuropathy classification and early warning method based on multimodal data as described in the above embodiments, such as... Figure 2 As shown, Figure 2 A schematic diagram of the framework of a diabetic foot neuropathy classification and early warning system based on multimodal data provided in an embodiment of this application includes: The data acquisition module is used to collect multimodal data of the subject's feet; The spatial mapping and region partitioning module is used to construct physical fields and perform unified mapping and region partitioning; The feature extraction module is used to extract region features and form a comprehensive feature vector; The feature encoding module is used to obtain high-dimensional feature representations of each region; The dual-branch neural network module is used to input high-dimensional feature representations into the dual-branch neural network to output the fractal warning probability and probability mask; The output module is used to fuse the classification warning probability and the probability mask, and output the neuropathy classification warning result and abnormal area location information.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0082] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0083] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for classifying and early warning of diabetic foot neuropathy based on multimodal data, characterized in that, Includes the following steps: Multimodal data of the subject's feet were collected, including infrared thermographic images, plantar pressure distribution data, and foot temperature data. The multimodal data are respectively constructed into multiple physical fields including at least temperature field and pressure field, and the multiple physical fields are uniformly mapped to a preset foot standardized coordinate system. Under the foot standardized coordinate system, the foot is divided into multiple anatomical regions. For each anatomical region, corresponding regional features are extracted from the multiple physical fields, and the regional features of the same region are combined to form a comprehensive feature vector of that region. A shared-parameter encoding network is used to encode the comprehensive feature vector of all regions to obtain a high-dimensional feature representation of each region; The high-dimensional feature representations of each region are input into a dual-branch recognition network, which includes a first classification recognition branch and a second localization recognition branch. The first classification recognition branch is used to output the warning probability of the neuropathy classification corresponding to each anatomical region, and the second localization recognition branch is used to output the probability mask of the foot abnormality region. By fusing the classification warning probability and the probability mask, the neuropathy classification warning result and abnormal area location information are determined and output.

2. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 1, characterized in that, The pressure field includes a static pressure field and a dynamic pressure field. The static pressure field is constructed based on plantar pressure distribution data collected when the subject is standing, and the dynamic pressure field is constructed based on plantar pressure distribution data collected during the subject's walking gait.

3. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 1, characterized in that, The region features extracted for each anatomical region include: Temperature field characteristics derived from the temperature field, the temperature field characteristics including at least the uniform temperature and temperature variability of the anatomical region; Pressure characteristics from the pressure field, the pressure characteristics including at least the peak pressure and pressure distribution percentage of the anatomical region.

4. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 2, characterized in that, The regional features extracted for each anatomical region also include dynamic pressure features from the dynamic pressure field, wherein the dynamic pressure features include at least the length of the pressure center trajectory of the anatomical region.

5. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 1, characterized in that, The first classification identification branch includes two parallel sub-branches, which are used to output the risk probability related to autonomic nervous system dysfunction and the risk probability related to motor nervous system dysfunction, respectively.

6. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 5, characterized in that, The step of fusing the subtype warning probability and the probability mask includes: determining the neuropathic risk index of the anatomical region based on the risk probability related to the autonomic nervous system dysfunction, the risk probability related to the motor nervous system dysfunction, and the average abnormality probability of the probability mask in the corresponding anatomical region.

7. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 6, characterized in that, The neuropathy classification warning result is determined based on the comparison results of the risk probability related to the autonomic nervous system dysfunction and the risk probability related to the motor nervous system dysfunction with preset thresholds, respectively. The classification warning result includes autonomic nervous system dysfunction warning, motor nervous system dysfunction warning, or combined dysfunction warning.

8. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 1, characterized in that, The second localization and recognition branch uses a convolutional neural network and an upsampling structure to generate the probability mask, which is used to indicate the probability of anomalies at each position of the foot corresponding to the foot's standardized coordinate system.

9. The method for classifying and early warning diabetic foot neuropathy based on multimodal data according to claim 1, characterized in that, The dual-branch neural network is trained using a multi-task learning framework, and the total loss function used during training includes a weighted sum of the pattern recognition loss and the localization recognition loss.

10. A classification and early warning system for diabetic foot neuropathy based on multimodal data, characterized in that, The method for implementing the diabetic foot neuropathy classification and early warning method based on multimodal data as described in any one of claims 1-9 includes: The data acquisition module is used to collect multimodal data of the subject's feet; The spatial mapping and region partitioning module is used to construct physical fields and perform unified mapping and region partitioning; The feature extraction module is used to extract region features and form a comprehensive feature vector; The feature encoding module is used to obtain high-dimensional feature representations of each region; The dual-branch neural network module is used to input high-dimensional feature representations into the dual-branch neural network to output the fractal warning probability and probability mask; The output module is used to fuse the classification warning probability and the probability mask, and output the neuropathy classification warning result and abnormal area location information.

Images