A multi-modal image recognition method and system for dynamic evaluation of diabetic wound healing

Abstract

This application relates to the field of medical image processing technology, and discloses a multimodal image recognition method and system for dynamic assessment of diabetic wound healing. The method includes acquiring infrared thermal imaging images, hyperspectral images, and visible light color images of a diabetic wound collected at the same time of visit; identifying the wound boundary between the diabetic wound and surrounding tissues in the visible light color image, and determining the central reference point of the visible light color image based on the wound boundary; establishing a virtual polar coordinate system based on the central reference point and preset anatomical reference markers; extracting tissue feature information characterizing the pathological state of the wound from the infrared thermal imaging image, hyperspectral image, and visible light color image, and mapping the spatial position of the tissue feature information in its corresponding image to the virtual polar coordinate system; normalizing the mapped tissue feature information, and calculating wound healing assessment indicators based on the normalized tissue feature information in the virtual polar coordinate system.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and more specifically, to a multimodal image recognition method and system for dynamic assessment of diabetic wound healing. Background Technology

[0002] Dynamic assessment of diabetic wound healing is a crucial aspect of clinical management, typically requiring the comprehensive utilization of image information from multiple modalities, including infrared thermography, hyperspectral imaging, and visible light color imaging. However, in actual clinical acquisition, these different modalities are often acquired by different devices at varying fields of view and angles. For example, visible light images taken by handheld cameras by healthcare professionals may be subject to displacement and rotation, while hyperspectral imaging with a fixed support has limited coverage. Furthermore, changes in patient positioning and limb placement during different visits can lead to unpredictable geometric distortions in the images. This inconsistency in acquisition conditions results in a core technical challenge: the inability to establish a stable and unified spatial reference system to align image data from different modalities and time points. Existing technologies often attempt registration based on image content, but this method is prone to failure when the wound appearance changes during the healing process. Ultimately, due to the drift of the spatial reference, the calculated quantitative indicators such as wound area, temperature distribution, and tissue composition lack comparability, making it impossible for physicians to accurately determine whether the wound is truly healed or an apparent change caused by measurement errors, severely impacting the reliability of treatment adjustments and assessments.

[0003] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application provides a multimodal image recognition method and system for dynamic assessment of diabetic wound healing. This method and system can solve the technical problem in existing technologies where inconsistent image acquisition conditions prevent the establishment of a unified spatial reference system, resulting in inaccurate and incomparable dynamic assessment results of wound healing.

[0005] In a first aspect, this application provides a multimodal image recognition method for dynamic assessment of diabetic wound healing, including: Acquire infrared thermal images, hyperspectral images, and visible color images of diabetic wounds collected at the same time of medical visit; Identify the boundary between the diabetic wound and surrounding tissue in a visible light color image, and determine the center reference point of the visible light color image based on the wound boundary; A virtual polar coordinate system is established based on the central reference point and preset anatomical reference markers; Tissue feature information characterizing the pathological state of wounds is extracted from infrared thermal imaging images, hyperspectral images, and visible light color images, and the spatial location of the tissue feature information in its corresponding images is mapped to a virtual polar coordinate system; The mapped tissue feature information is normalized, and wound healing assessment indicators are calculated based on the normalized tissue feature information in a virtual polar coordinate system.

[0006] This technical solution utilizes the geometric center of the wound itself and externally stable anatomical landmarks to construct a virtual polar coordinate system independent of the imaging equipment and patient position. This coordinate system serves as a unified spatial reference, enabling image feature information from different modalities and time points to be mapped to the same reference system for standardized comparison. This fundamentally solves the problems of alignment difficulties and incomparable evaluation indicators caused by changes in acquisition conditions, significantly improving the accuracy and reliability of dynamic evaluation.

[0007] Furthermore, the steps for establishing a virtual polar coordinate system based on the central reference point and preset anatomical reference markers include: Determine the displacement vector pointing from the central reference point to the preset anatomical reference marker; A virtual polar coordinate system is established with the central reference point as the origin and the ray direction where the displacement vector is located as the reference direction.

[0008] This technical solution clarifies the construction method of the virtual polar coordinate system, namely, taking the center of the wound as the origin and the direction pointing to the stable anatomical landmark as the polar axis, ensuring the consistency of the coordinate system direction in different measurements, and providing a reliable angular reference for the accurate mapping of subsequent features.

[0009] Furthermore, the step of mapping the spatial location of tissue feature information in its corresponding image to a virtual polar coordinate system includes: Calculate the pixel distance of tissue feature information relative to the central reference point and the azimuth angle relative to the reference direction in the corresponding image; The pixel distance and azimuth angle are converted into polar radius coordinates and polar angle coordinates in a virtual polar coordinate system, and the polar radius coordinates and polar angle coordinates are associated with and stored with the corresponding tissue feature information.

[0010] By calculating the distance and angle of each feature point relative to the center of the wound and the reference direction, the absolute position, which depends on the image pixel matrix, is transformed into a relative position description with inherent physiological meaning. This achieves coordinate transformation of the data and structured storage of spatial information, which facilitates subsequent quantitative analysis across time and modality.

[0011] Furthermore, the steps for normalizing the mapped tissue feature information include: Obtain the reference value range of each tissue characteristic information within the healthy tissue region; Based on the reference value range, the numerical values ​​of each tissue characteristic information are linearly mapped to a preset unified interval.

[0012] This technical solution solves the problem of inconsistent dimensions and numerical ranges of features across different modalities. By normalizing based on healthy tissue, it eliminates absolute numerical fluctuations caused by factors such as equipment differences and ambient lighting, ensuring that all feature information is represented on a unified and comparable scale, thus providing a standardized data foundation for the calculation of subsequent comprehensive evaluation indicators.

[0013] Furthermore, the steps for calculating wound healing assessment indicators based on normalized tissue characteristic information in a virtual polar coordinate system include: Obtain the polar radius and polar angle coordinates corresponding to the normalized tissue feature information of diabetic wounds at different times of medical visit; Based on the polar radius and polar angle coordinates corresponding to the normalized tissue feature information at different times of medical treatment, the wound area change rate or tissue component evolution ratio is calculated as an indicator for wound healing assessment.

[0014] By comparing the characteristic distribution of different treatment times under the same polar coordinates, key healing indicators such as the shrinkage rate of the wound area and the evolution ratio of specific tissues can be calculated directly and accurately, realizing the true quantification and dynamic tracking of the wound healing process.

[0015] Preferably, the step of extracting tissue feature information characterizing the pathological state of the wound from infrared thermal imaging images, hyperspectral images, and visible light color images includes: Temperature distribution information in infrared thermal imaging images is extracted as tissue feature information; The distribution information of oxyhemoglobin in hyperspectral images is extracted as tissue feature information; Color and texture features are extracted from visible light color images as organizational feature information.

[0016] This technical solution identifies key features directly related to the pathological state of the wound. Temperature distribution reflects inflammation and blood supply, oxyhemoglobin reflects tissue perfusion and hypoxia, while color and texture characterize tissue types such as granulation tissue and necrotic tissue.

[0017] Preferably, when there are multiple preset anatomical reference markers, the step of establishing a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers includes: Search for multiple pre-defined anatomical reference markers within the healthy skin area surrounding the diabetic wound; Calculate the relative distances and orientational relationships between the preset anatomical reference markers; Identify and remove preset anatomical reference markers whose relative distance and orientation relationships deviate, and use the remaining preset anatomical reference markers to deduce the reference direction through preset geometric mapping logic; A virtual polar coordinate system is established based on the central reference point and reference direction.

[0018] By verifying the geometric relationships between multiple markers, abnormal marker points caused by skin folds, temporary marker occlusion, or misidentification can be effectively identified and eliminated. Redundant information is used to improve the accuracy and robustness of the reference direction determination, so that the coordinate system can still be reliably established in complex or interference environments.

[0019] Preferably, the step of identifying the boundary between the diabetic wound and surrounding tissue in a visible light color image includes: The contrast between diabetic wounds and healthy skin was enhanced by color space conversion, and the wound boundaries in the color space-converted visible light color image were extracted using edge detection operators and region growing algorithms. Morphological closing operations are performed on the edges of the wound boundary to connect the edge break points and smooth the contour, resulting in a continuous wound boundary.

[0020] Color space conversion amplifies the difference between the wound and normal skin. Combined with edge detection and region growing, the boundary can be initially located. Morphological closing operation can repair boundary breaks and smooth burrs, ultimately obtaining a complete and continuous wound contour.

[0021] Preferably, the step of determining the center reference point of the visible light color image based on the wound boundary includes: Calculate the geometric centroid of the region enclosed by the wound boundary; The geometric centroid is determined as the central reference point.

[0022] This technical solution clarifies that the central reference point is calculated using the geometric centroid of the wound area. The geometric centroid is a point that is highly representative of the overall shape of the wound contour; its position is stable and not easily affected by small fluctuations in local boundaries. Using the geometric centroid as the origin of the coordinate system ensures the consistency and stability of the reference point's positioning during wound healing and shape changes.

[0023] Secondly, this application also discloses a multimodal image recognition system for dynamic assessment of diabetic wound healing, comprising: The image acquisition module is used to acquire infrared thermal images, hyperspectral images, and visible color images of diabetic wounds collected at the same time of medical visit. The wound boundary and reference point recognition module is used to identify the wound boundary between the diabetic wound and the surrounding tissue in the visible light color image, and to determine the center reference point of the visible light color image based on the wound boundary. The coordinate system establishment module is used to establish a virtual polar coordinate system based on the central reference point and preset anatomical reference markers; The feature mapping module is used to extract tissue feature information that characterizes the pathological state of the wound from infrared thermal imaging images, hyperspectral images and visible light color images, and map the spatial position of the tissue feature information in its corresponding image to a virtual polar coordinate system; The wound healing assessment module is used to normalize the mapped tissue feature information and calculate the wound healing assessment index based on the normalized tissue feature information in a virtual polar coordinate system.

[0024] This application provides a multimodal image recognition method and system for dynamic assessment of diabetic wound healing, aiming to solve the technical problem in existing technologies where the difficulty in aligning multimodal and multi-time-point images due to varying image acquisition conditions leads to inaccurate assessment results and a lack of comparability. By identifying the wound boundary and determining the geometric center as the reference origin, and combining this with stable anatomical reference markers outside the wound to determine the reference direction, a unique virtual polar coordinate system independent of the imaging device's pose and field of view is constructed. This coordinate system acts like a virtual ruler etched onto the patient's body, providing a unified and unchanging spatial benchmark for all information. Therefore, regardless of image offset or rotation in different modalities, or changes in the patient's position during different visits, the extracted key pathological features such as temperature, blood oxygen, and color can be accurately mapped to this stable coordinate system for normalization and comparison. This fundamentally eliminates measurement errors introduced by differences in acquisition geometry, ensuring the objectivity and accuracy of wound healing assessment indicators such as area change rate and tissue composition evolution. It achieves truly quantitative and comparable dynamic tracking of the healing process, providing clinicians with a more reliable basis for decision-making. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating a multimodal image recognition method for dynamic assessment of diabetic wound healing, provided in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram of the structure of a multimodal image recognition system for dynamic assessment of diabetic wound healing, provided in an embodiment of this application.

[0027] Labeling Explanation: 210, Image Acquisition Module; 220, Wound Boundary and Reference Point Recognition Module; 230, Coordinate System Establishment Module; 240, Feature Mapping Module; 250, Wound Healing Assessment Module. Detailed Implementation

[0028] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] In the clinical follow-up management of diabetic foot ulcers, a typical scenario involves patients needing to visit the hospital regularly, such as weekly, for dressing changes and wound condition recording. At each visit, healthcare professionals need to conduct a comprehensive examination of the patient's foot wound. However, due to the complexity of the clinical environment and the non-standardization of procedures, ensuring data consistency presents a significant challenge. For example, at the first visit, a nurse might use a handheld digital camera to take a visible light photograph of the wound, which may include the wound and a small portion of the surrounding skin. Subsequently, a frame-mounted infrared thermal imager is used to measure the temperature, its field of view potentially covering the entire dorsum of the foot. Finally, a hyperspectral imaging device scans the core area of ​​the wound in a darkened environment to analyze tissue composition. At the next visit, a different nurse might be performing the procedure, and the patient's posture and foot placement angle may have changed, as well as the distance and angle of the handheld camera. This results in differences in translation, rotation, and scaling between the two images, taken from different modalities. When doctors attempt to overlay and compare these images to quantify the reduction in wound area or the growth of new granulation tissue, they find that the images cannot be precisely aligned due to the lack of a stable reference point. Directly measured area changes may be mixed with perspective distortion caused by changes in the shooting angle, and the movement of temperature hotspots cannot be determined as being caused by pathological changes or simple positional shifts. This inconsistency and incomparability of data severely hinders the objective, quantitative, and dynamic assessment of the wound healing process.

[0031] Regarding this, firstly, see... Figure 1 This application provides a multimodal image recognition method for dynamic assessment of diabetic wound healing, including: S1. Acquire infrared thermal images, hyperspectral images, and visible light color images of diabetic wounds collected at the same time of medical visit; S2. Identify the boundary between the diabetic wound and the surrounding tissue in the visible light color image, and determine the center reference point of the visible light color image based on the wound boundary. S3. Establish a virtual polar coordinate system based on the central reference point and preset anatomical reference markers; S4. Extract tissue feature information characterizing the pathological state of the wound from infrared thermal imaging images, hyperspectral images, and visible light color images, and map the spatial position of the tissue feature information in its corresponding images to a virtual polar coordinate system; S5. Normalize the mapped tissue feature information and calculate the wound healing assessment index based on the normalized tissue feature information in the virtual polar coordinate system.

[0032] The virtual polar coordinate system is a mathematically defined coordinate system that does not rely on any physical imaging device. Instead, it is directly established on the specific anatomical structures of the patient's body. By selecting a stable center point of the wound itself as the origin and a fixed anatomical feature point outside the wound to define the orientation of the coordinate system, a coordinate system bound to the patient's body is constructed. As long as the selected center point and orientation reference point are stable, the position and orientation of this virtual coordinate system on the patient's body remain constant regardless of changes in the patient's position or the placement of the imaging device. This provides a unified set of data across time and modalities.

[0033] The central reference point is the origin of the virtual polar coordinate system, and theoretically, it should be a point that stably represents the geometric center of the wound. During wound healing, although the shape and size of the wound may change, the position of its geometric center remains relatively stable, serving as a reliable positioning benchmark. An anatomical reference marker is a reference object used to determine the orientation of the virtual polar coordinate system. It is a long-term, fixed, and easily identifiable feature on the patient's body. For example, it could be a prominent bony point on the patient's ankle, such as the medial malleolus, a dark mole, an old small scar, or a specific corner of a toenail. This marker functions similarly to a compass on a map, providing a fixed zero-degree direction for the coordinate system.

[0034] In a specific implementation scenario, when a diabetic foot patient visits the clinic, the system guides medical staff to acquire images using three different devices. First, a high-resolution visible light color image is captured using a standard digital SLR camera under sufficient natural light or standard light source conditions. This image is primarily used for subsequent wound boundary identification and morphological analysis. Next, a medical infrared thermal imager is used in a stable room temperature environment to capture an infrared thermal image of the area including the wound and a large surrounding skin area. Each pixel value in the image represents an absolute temperature reading. Finally, a pushbroom or snapshot hyperspectral imager is used to collect data from the wound area, resulting in a hyperspectral data cube containing reflectance information at hundreds of different wavelengths. After acquisition, these three sets of image data are linked to the same patient's medical record for the same visit and timestamped.

[0035] Next, the system processes the acquired visible light color image to identify wound boundaries and determine a central reference point. Visible light images, due to their high resolution and rich color and texture information, are best suited for accurately outlining wound contours. The system's internal image processing algorithm automatically analyzes the image, searching for color and texture transitions between the typically red, yellow, or black wound tissue and the surrounding normal skin. Through a series of processing algorithms, the system generates a closed curve, which represents the wound boundary. After obtaining this boundary line, the system treats the enclosed area as a two-dimensional geometric figure and calculates its geometric center. This calculated geometric center point is defined as the central reference point for this measurement. The position of this point is calculated based on the entire wound contour, thus exhibiting good stability and being less prone to drastic fluctuations due to minor changes in local boundaries.

[0036] The next crucial step is establishing a virtual polar coordinate system. After determining the central reference point, the system requires a directional reference. When using the system for the first time, healthcare professionals need to manually delineate one or more stable and easily identifiable anatomical reference markers, such as a small mole that has been present for many years, on a visible light image, based on clinical experience within the healthy skin area surrounding the wound. The system records the characteristics and approximate location of this marker. In subsequent measurements, the system automatically searches for and repositions this anatomical reference marker within the estimated area. With the central reference point as the origin and the anatomical reference marker as the reference direction, a complete virtual polar coordinate system can be established. The origin of this coordinate system is at the center of the wound, and its zero-degree direction points to the anatomical reference marker.

[0037] The next step is to unify all image information into this newly established coordinate system. The system processes infrared thermal images and hyperspectral images one by one. For any pixel in the infrared image, there is a position (x, y) in the image's own pixel coordinate system and a corresponding temperature value. The system first roughly aligns the infrared image with the visible light image using image registration techniques, such as feature point-based or mutual information-based methods, to find the pixel positions of the central reference point and anatomical reference markers in the infrared image. Then, the system can calculate the position of the pixel relative to the central reference point and convert it into polar radius and polar angle coordinates (ρ, θ) in a virtual polar coordinate system. In this way, the temperature information of this point is relabeled as (ρ, θ, temperature value). Similarly, for the hyperspectral image, the system first extracts key tissue feature information, such as the oxygenated hemoglobin saturation map calculated through a specific band combination. Each pixel in the image and its corresponding blood oxygen saturation value are also mapped to the virtual polar coordinate system through a similar coordinate transformation and labeled as (ρ, θ, blood oxygen value). The color, texture, and other information of the visible light image itself are also mapped in the same way. Through this step, information from different original images is aggregated onto the same virtual polar coordinate system.

[0038] Before conducting the final assessment, it is necessary to normalize the various mapped tissue characteristics. Because features such as temperature, blood oxygen saturation, and color have different dimensions and numerical ranges, they cannot be directly mathematically calculated or compared. The purpose of normalization is to unify the various tissue characteristics onto the same scale, such as the [0,1] interval. One feasible approach is to sample each feature within the healthy skin area surrounding the wound to obtain a normal value range for that feature in a healthy state. Then, using this range as a benchmark, the feature values ​​within the wound area are linearly or non-linearly stretched and translated to map them to the [0,1] interval. For example, for temperature, if the healthy skin temperature is 32 degrees Celsius, while a severely infected area may reach 36 degrees Celsius, the system can map 32 degrees to 0 and 36 degrees to 1. After this processing, a normalized temperature of 0.5 is comparable to a normalized hypoxemia of 0.5.

[0039] Finally, based on these normalized tissue characteristics, wound healing assessment indicators are calculated in a virtual polar coordinate system. Since all data resides in the same stable coordinate system, dynamic comparisons become very direct. For example, the system can retrieve data from the previous visit at time T1 and the current visit at time T2. By comparing the positions of the wound boundary lines at T1 and T2 in the virtual polar coordinate system, the rate of change in wound area can be accurately calculated. By comparing changes in tissue characteristics, such as the normalized necrotic tissue score, at the same polar coordinate position (ρ, θ), the tissue evolution in a specific area can be tracked. Ultimately, the system can generate a quantitative assessment report, such as a 15% reduction in total wound area compared to last week, with a 30% decrease in necrotic tissue coverage and a 20% increase in new granulation tissue coverage, providing doctors with objective and reliable decision support.

[0040] Using the above method, a virtual polar coordinate system, independent of the imaging equipment and patient position, is constructed by utilizing the geometric center of the wound itself and stable external anatomical landmarks. This coordinate system serves as a unified spatial reference, enabling image feature information from different modalities and time points to be mapped to the same reference system for standardized comparison. This fundamentally solves the alignment difficulties and incomparable evaluation indicators caused by changes in acquisition conditions, significantly improving the accuracy and reliability of dynamic assessment.

[0041] In a preferred embodiment, the step of establishing a virtual polar coordinate system based on a central reference point and preset anatomical reference markers includes: Determine the displacement vector pointing from the central reference point to the preset anatomical reference marker; A virtual polar coordinate system is established with the central reference point as the origin and the ray direction where the displacement vector is located as the reference direction.

[0042] This approach provides a geometrically clear and computationally simple path for establishing a virtual polar coordinate system. Specifically, after processing the visible light image to determine the pixel coordinates P_c(x_c,y_c) of the central reference point and the pixel coordinates P_m(x_m,y_m) of the anatomical reference marker, the system calculates a two-dimensional displacement vector V = (x_m-x_c,y_m-y_c) pointing from the central reference point to the anatomical reference marker. The direction of this vector is fixed because it depends only on the relative positions of the two body feature points. Subsequently, the system uses point P_c as the origin of the virtual polar coordinate system and defines the direction pointed to by vector V as the polar axis of the polar coordinate system, i.e., the zero-degree angle (θ=0) direction. The angles of all other points are calculated relative to this direction. This definition avoids any ambiguity, ensuring that the coordinate system established for each measurement has a completely consistent directional reference, providing a solid foundation for the accurate angle mapping of subsequent features.

[0043] Furthermore, the step of mapping the spatial location of tissue feature information in its corresponding image to a virtual polar coordinate system includes: Calculate the pixel distance of tissue feature information relative to the central reference point and the azimuth angle relative to the reference direction in the corresponding image; The pixel distance and azimuth angle are converted into polar radius coordinates and polar angle coordinates in a virtual polar coordinate system, and the polar radius coordinates and polar angle coordinates are associated with and stored with the corresponding tissue feature information.

[0044] Suppose that in a 1000x800 pixel visible light image, the calculated coordinates of the central reference point P_c are (500, 400). Simultaneously, the coordinates of the anatomical reference marker P_m are (600, 400). Then, according to the aforementioned method, the displacement vector from P_c to P_m is (100, 0), which is horizontal to the right and defined as the zero-degree direction in polar coordinates. Now, we need to map another point P_f in the image, representing a small piece of necrotic tissue, whose coordinates in the image are (530, 440).

[0045] First, calculate the pixel distance of P_f relative to the center reference point P_c. This distance d_pixel = sqrt((530-500)^2 + (440-400)^2) = sqrt(30^2 + 40^2) = sqrt(900 + 1600) = sqrt(2500) = 50 pixels.

[0046] Next, calculate the azimuth angle of P_f relative to the reference direction. The displacement vector from P_c to P_f is (30,40). The angle between this vector and the reference direction (horizontal to the right, vector (1,0)) is θ=atan2(40,30), which is approximately 53.13 degrees.

[0047] Next, unit conversion is performed. Assume a scale bar was placed in the image during shooting, and calibration shows that each pixel in the image represents 0.1 millimeters. Therefore, the polar radius coordinates of point P_f in the virtual polar coordinate system are ρ = 50 pixels * 0.1 millimeters / pixel = 5.0 millimeters.

[0048] Finally, the position of point P_f in the image (530, 440) was successfully converted to coordinates in a virtual polar coordinate system (ρ=5.0mm, θ=53.13°). The system associates these coordinates with the tissue feature information corresponding to point P_f (e.g., color value is black, normalized necrosis score is 0.9) and stores it in the database. By performing this process on every point in the image that needs to be analyzed, the spatial mapping of the feature information of the entire image is completed.

[0049] Furthermore, the steps for normalizing the mapped tissue feature information include: Obtain the reference value range of each tissue characteristic information within the healthy tissue region; Based on the reference value range, the numerical values ​​of each tissue characteristic information are linearly mapped to a preset unified interval.

[0050] By utilizing the patient's own healthy tissue as an internal standard, measurement system errors caused by the environment, equipment, or operation are effectively eliminated. Taking infrared thermography temperature information as an example, assuming the system measures temperatures between 31.5℃ and 32.5℃ within a region of skin around the wound that is considered healthy, this range [31.5, 32.5] is recorded by the system as the healthy temperature reference range for this patient's measurement. Simultaneously, based on a medical knowledge base, the system presets a typical temperature range characterizing pathological changes; for example, temperatures above 35.5℃ typically indicate a significant inflammatory response.

[0051] Based on this information, the system can construct a linear mapping function to transform the measured raw temperature value T_raw to a normalized interval of [0,1]. For example, a mapping rule can be set: values ​​below 32.5℃ (the upper limit of the healthy range) are all mapped to 0, indicating no hotspots; values ​​above or equal to 35.5℃ are all mapped to 1, indicating strong hotspots; for temperatures between 32.5℃ and 35.5℃, linear interpolation is performed. The calculation formula is: T_norm=(T_raw-32.5) / (35.5-32.5).

[0052] Assuming the measured temperature at a point within the wound is 34.0℃, its normalized temperature value is (34.0-32.5) / 3.0≈0.5. If the temperature at another point is 31.8℃, its normalized value is 0. In this way, features of different modalities, such as temperature, blood oxygenation, and water content, can be converted into dimensionless values ​​with similar interpretable meanings, laying the foundation for subsequent multi-feature fusion assessment.

[0053] For the oxyhemoglobin saturation extracted from hyperspectral images, its original numerical range is usually between 0 and 1. However, the baseline blood oxygen levels of different individuals may vary. A reference range for oxyhemoglobin saturation can be determined in healthy skin areas, such as [0.65, 0.75]. Values ​​below the lower limit of this range are then mapped to 0 to indicate severe hypoxia, and values ​​above the upper limit are mapped to 1 to indicate abundant blood supply. Intermediate values ​​are linearly mapped to the [0, 1] interval.

[0054] For color features extracted from visible light images, such as the red and green component values ​​extracted from the α channel of the Lab color space, their normal fluctuation range can be determined within healthy skin areas, for example, [10, 20]. Then, the measured α values ​​within the wound area are linearly mapped to the [0, 1] interval, so that values ​​closer to 1 indicate a higher degree of redness, potentially indicating granulation tissue proliferation. For texture features, such as texture complexity calculated through local binary mode, their normal range can be determined within healthy skin areas, and appropriate normalization processing performed. In this way, features of different modalities, such as temperature, blood oxygen, color, and texture, can be converted into dimensionless values ​​with similar interpretable meanings, laying the foundation for subsequent multi-feature fusion evaluation.

[0055] Furthermore, the steps for calculating wound healing assessment indicators based on normalized tissue characteristic information in a virtual polar coordinate system include: Obtain the polar radius and polar angle coordinates corresponding to the normalized tissue feature information of diabetic wounds at different times of medical visit; Based on the polar radius and polar angle coordinates corresponding to the normalized tissue feature information at different times of medical treatment, the wound area change rate or tissue component evolution ratio is calculated as an indicator for wound healing assessment.

[0056] In one specific embodiment, wound images were acquired and analyzed at time T1 of week one and time T2 of week two. At time T1, a set of points {(ρ_i, θ_i)} representing the wound boundary in a virtual polar coordinate system was obtained. By integrating the closed region formed by these points, the total wound area Area_T1 can be calculated. Assuming Area_T1 = 520 square millimeters. At time T2, the same process yielded a new wound boundary and a new total area Area_T2. Assuming Area_T2 = 480 square millimeters. The calculated wound area change rate was 7.7%, leading to the conclusion that the wound area decreased by approximately 7.7%.

[0057] Assuming that analysis of visible light and hyperspectral images allows for tissue classification at each point, and a normalized score for newly formed granulation tissue (between 0 and 1, where 1 indicates a high degree of certainty in the presence of new granulation tissue) is provided, at time T1, the area of ​​all points with a score greater than 0.8 is calculated, resulting in Area_granulation_T1 = 100 square millimeters. This represents approximately 19.2% of the total area (100 / 520 ≈ 19.2%). At time T2, similarly, Area_granulation_T2 = 150 square millimeters is calculated, representing approximately 31.3% of the total area (150 / 480 ≈ 31.3%). The conclusion is that the coverage rate of new granulation tissue increased from 19.2% to 31.3%, indicating that the wound is healing well.

[0058] For assessing necrotic tissue, the system can make a comprehensive judgment based on normalized temperature, spectral, and color information. For example, areas with normalized temperature values ​​close to 0 (low temperature), areas with normalized oxygenation values ​​close to 0 (low oxygenation), and areas with normalized color values ​​close to 0 (dark brown) typically indicate necrotic tissue. The system can calculate a comprehensive necrotic tissue score for each pixel and statistically analyze the area changes of regions with scores above a threshold. For assessing inflammatory responses, temperature information in infrared images plays a crucial role. The system can track the area changes of regions with normalized temperature values ​​above a certain threshold, such as 0.6, in a virtual polar coordinate system. If the area of ​​this region expands, it may indicate the spread of inflammation; if it shrinks, it indicates that the inflammation is under control. In addition, the system can calculate the average normalized temperature of all pixels within the wound area and compare the trend of average temperature changes between two visits. In this way, the temperature information from infrared images works synergistically with the feature information from other modalities to support a comprehensive assessment of various pathophysiological changes during wound healing.

[0059] Furthermore, the steps for extracting tissue feature information characterizing the pathological state of the wound from infrared thermal imaging images, hyperspectral images, and visible light color images include: Temperature distribution information in infrared thermal imaging images is extracted as tissue feature information; The distribution information of oxyhemoglobin in hyperspectral images is extracted as tissue feature information; Color and texture features are extracted from visible light color images as organizational feature information.

[0060] In practice, an infrared thermal imaging image is itself a two-dimensional temperature matrix. The system directly uses this temperature map as temperature distribution information. During analysis, areas above the baseline temperature of healthy skin are marked as hotspots, which may indicate local inflammation or infection; while cold spots below the baseline temperature may indicate insufficient local blood supply or tissue necrosis.

[0061] For hyperspectral images, the raw data is a data cube containing hundreds of spectral bands. Extracting oxyhemoglobin distribution information requires specialized spectral unmixing calculations. The system utilizes the fact that oxyhemoglobin and deoxyhemoglobin have different absorption peaks in the visible and near-infrared bands, such as 500-650 nm, and calculates the relative concentration or saturation of oxyhemoglobin and deoxyhemoglobin for each pixel by applying a modified Beer-Lambert law or establishing a multiple linear regression model.

[0062] For visible light color images, the system extracts two types of features. Regarding color features, the system first converts the image from the RGB color space to the Lab or HSV color space, which better matches human visual perception. Regarding texture features, the system can use algorithms such as the gray-level co-occurrence matrix to calculate texture descriptors such as contrast, energy, and entropy for each pixel's neighborhood. For example, newly formed granulation tissue typically exhibits a fine, uniform texture, while necrotic tissue may show a rough, irregular texture. By integrating these features, fine classification of wound tissue can be achieved.

[0063] Furthermore, when there are multiple preset anatomical reference markers, the steps for establishing a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers include: Search for multiple pre-defined anatomical reference markers within the healthy skin area surrounding the diabetic wound; Calculate the relative distances and orientational relationships between the preset anatomical reference markers; Identify and remove preset anatomical reference markers whose relative distance and orientation relationships deviate, and use the remaining preset anatomical reference markers to deduce the reference direction through preset geometric mapping logic; A virtual polar coordinate system is established based on the central reference point and reference direction.

[0064] When there are multiple pre-set anatomical reference markers, the redundancy of spatial geometric relationships can be used to improve the robustness of the coordinate system. The core idea is that these anatomical markers form a relatively stable geometric configuration on the patient's body. During subsequent consultations, if most markers can be successfully identified and their relative geometric relationships are basically consistent with the pre-set template, these markers are considered valid. If the position of a marker deviates significantly and inexplicably from other points, the point is considered to have been misidentified or occluded and should be removed.

[0065] During the initial consultation, the system records the N (N≥3) anatomical reference markers selected by the operator and calculates the spatial relationships between each pair of them to form a geometric template. The template can be represented as a set of distance matrices D_ij, where D_ij represents the Euclidean distance between marker points i and j; or it can be a set of vectors consisting of angles and distances.

[0066] During subsequent medical examinations, the system searches for N candidate points in the current image and calculates the current distance matrix D'ij between each pair of points. The system identifies outliers by comparing the differences between the current distance matrix and the template distance matrix. One feasible implementation is to calculate the sum or average deviation of the current distance from the template distance to each candidate point k compared to all other candidate points k. If the sum of deviations for a point k is significantly higher than the sum of deviations for all other points and exceeds a preset threshold, that point is considered an outlier and should be removed. This method is not dependent on a specific triangle configuration and is applicable to any number and layout of reference markers.

[0067] After removing the markers identified as deviating, the system obtains a set of M (M≥2) valid markers. There are several ways to derive a stable and consistent reference direction from these M points. If M=2, meaning there are two remaining valid markers P and Q, the most direct method is to use the vector pointing from P to Q as the reference direction. To ensure the direction is consistent with historical data, the system needs to identify which point is P and which is Q, for example, by matching their relative positions with the markers in the template.

[0068] If M ≥ 3, meaning there are three or more remaining valid marker points, an optimal reference direction can be determined using a fitting method. One approach is to use Protodyakonov analysis or least squares to calculate an optimal rigid transformation, including rotation and translation, that aligns the current set of valid points as closely as possible with the corresponding points in the template. This optimal transformation yields a rotation matrix, from which the reference direction can be extracted. A simpler approach is to select the two points closest to the central reference point from the set of valid points, or the two most spatially representative points (e.g., the two furthest points), and use the line connecting them as the reference direction.

[0069] In this way, regardless of the number of remaining valid markers, the system can derive a reliable reference direction based on a general and adaptive logic, thereby establishing a stable virtual polar coordinate system.

[0070] In one specific embodiment, upon initial use, the operator selects three stable anatomical reference points on the patient's foot: A (the highest point of the medial malleolus), B (a mole near the point where the dorsalis pedis artery pulsation is strongest), and C (a small scar on the head of the first metatarsal). The system records the side lengths (d_AB, d_BC, d_AC) and interior angles (∠ABC, ∠BCA, ∠CAB) of the triangle ABC formed by these three points as a geometric configuration template. During a subsequent follow-up visit, the system automatically searches for and finds three candidate points A', B', and C' in the image. At this point, the system performs a configuration verification: 1. Calculate the side lengths and interior angles of the new triangle A'B'C'.

[0071] 2. Compare the new geometric parameters with the stored template. Suppose that the difference between the side lengths d'_A'B' and d_AB is within the allowable error range, but d'_B'C' is 30% shorter than d_BC, indicating that the positioning of B' or C' is likely incorrect.

[0072] 3. The system can further identify the most suspicious deviation points through cross-validation, such as comparing the relationship between d'_A'C' and d_AC. If it is ultimately determined that point B' is a temporary stain that was incorrectly identified, the system will remove it from the set of reference points.

[0073] 4. At this point, the system uses the remaining, verified reference points A' and C' to derive the reference direction through a pre-defined geometric mapping logic. The simplest logic is to directly use the direction of vector A'C' as the reference direction. A more complex logic could be that if three or more points remain, an optimal rigid transformation can be found through Protodyakonov analysis, resulting in a more robust reference direction determined by all valid points.

[0074] In this way, even if some reference points fail to be identified, the system can still establish a stable and reliable virtual polar coordinate system.

[0075] Furthermore, the steps for identifying the wound boundary between the diabetic lesion and surrounding tissue in a visible light color image include: The contrast between diabetic wounds and healthy skin was enhanced by color space conversion, and the wound boundaries in the color space-converted visible light color image were extracted using edge detection operators and region growing algorithms. Morphological closing operations are performed on the edges of the wound boundary to connect the edge break points and smooth the contour, resulting in a continuous wound boundary.

[0076] Specifically, first, the system converts the input RGB format visible light image to the Lab color space. In this space, the 'a*' channel represents color components ranging from green to red. Granulation tissue in diabetic wounds typically appears bright red, showing a significant difference in color from normal skin in this channel. This greatly enhances the contrast between the wound and the background.

[0077] Next, the system employs a hybrid strategy to extract boundaries. On one hand, the Canny edge detection operator is applied to the 'a*' channel image. This operator can sensitively capture pixels with drastic color changes, thus obtaining a preliminary, potentially discontinuous, edge map. On the other hand, the system automatically identifies a high-probability region in the center of the wound area based on color features, places one or more seed points there, and then executes a region growing algorithm. This algorithm starts from the seed point and continuously merges neighboring pixels with similar colors into the current region until it encounters a strong edge detected by the Canny operator or the color difference exceeds a preset threshold, at which point it stops.

[0078] Finally, the extracted wound boundary undergoes post-processing. Due to factors such as uneven illumination or microvessels, the extracted boundary may contain some small breaks or burrs. The system applies morphological closing operations, first performing a dilation operation, using a small structuring element such as a 3x3 circle to fill the small gaps on the boundary; then performing an erosion operation to restore the overall size of the wound while smoothing out any contour burrs that may have been generated during the dilation process. After this series of processing steps, a complete, continuous, and smooth wound boundary line is finally obtained.

[0079] Furthermore, the step of determining the center reference point of the visible light color image based on the wound boundary includes: Calculate the geometric centroid of the region enclosed by the wound boundary; The geometric centroid is determined as the central reference point.

[0080] The previous step yielded a closed polygonal outline representing the wound boundary, with its vertex coordinate sequence being {(x0,y0),(x1,y1),...,(x n-1 ,y n-1 The system can then use the polygon centroid calculation formula to determine its geometric centroid (Cx, Cy). The advantage of choosing the geometric centroid as the central reference point lies in its stability. During wound healing, even if local irregularities occur at the wound boundary—for example, one side shrinks faster while a new small defect appears on the other—the positional drift of the geometric centroid will be very small as long as the overall shape does not undergo drastic changes. This stability ensures that the origin of the virtual polar coordinate system established at different times of treatment remains highly consistent, which is the basis for achieving accurate dynamic comparison.

[0081] Secondly, see Figure 2 This application also provides a multimodal image recognition dynamic assessment system for diabetic wound healing. This system can be an embedded system integrated into a dedicated handheld device, or a software system deployed on a hospital server or cloud platform, including: Image acquisition module 210 is used to acquire infrared thermal images, hyperspectral images and visible color images of diabetic wounds collected at the same time of medical visit. The wound boundary and reference point recognition module 220 is used to identify the wound boundary between the diabetic wound and the surrounding tissue in a visible light color image, and to determine the center reference point of the visible light color image based on the wound boundary. The coordinate system establishment module 230 is used to establish a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers; The feature mapping module 240 is used to extract tissue feature information that characterizes the pathological state of the wound from infrared thermal imaging images, hyperspectral images and visible light color images, and to map the spatial position of the tissue feature information in its corresponding image to a virtual polar coordinate system. The wound healing assessment module 250 is used to normalize the mapped tissue feature information and calculate the wound healing assessment index based on the normalized tissue feature information in a virtual polar coordinate system.

[0082] By providing a physical system capable of executing the aforementioned evaluation method, the method steps are functionalized into specific modules, clarifying the system's composition and the functional division of each part. This provides clear architectural guidance for the engineering implementation and productization of the evaluation method, enabling its application in clinical equipment or medical information systems.

[0083] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multimodal image recognition method for dynamic assessment of diabetic wound healing, characterized in that, include: Acquire infrared thermal images, hyperspectral images, and visible color images of diabetic wounds collected at the same time of medical visit; Identify the boundary between the diabetic wound and the surrounding tissue in the visible light color image, and determine the center reference point of the visible light color image based on the wound boundary; A virtual polar coordinate system is established based on the central reference point and the preset anatomical reference markers; Tissue feature information characterizing the pathological state of the wound is extracted from the infrared thermal imaging image, the hyperspectral image, and the visible light color image, and the spatial position of the tissue feature information in its corresponding image is mapped to the virtual polar coordinate system; The mapped tissue feature information is normalized, and wound healing assessment index is calculated based on the normalized tissue feature information in the virtual polar coordinate system.

2. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, The step of establishing a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers includes: Determine the displacement vector of the central reference point pointing to the preset anatomical reference marker; The virtual polar coordinate system is established with the central reference point as the origin and the ray direction where the displacement vector is located as the reference direction.

3. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 2, characterized in that, The step of mapping the spatial location of the tissue feature information in its corresponding image to the virtual polar coordinate system includes: Calculate the pixel distance of the tissue feature information relative to the central reference point and the azimuth angle relative to the reference direction in its corresponding image; The pixel distance and the azimuth angle are converted into polar radius coordinates and polar angle coordinates in the virtual polar coordinate system, and the polar radius coordinates and the polar angle coordinates are associated with and stored with the corresponding tissue feature information.

4. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, The step of normalizing the mapped tissue feature information includes: Obtain the reference numerical range of each of the tissue characteristic information within the healthy tissue region; Based on the reference value ranges, the values ​​of each tissue feature information are linearly mapped to a preset unified interval.

5. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 3, characterized in that, The step of calculating the wound healing assessment index based on the normalized tissue feature information in the virtual polar coordinate system includes: Obtain the polar radius coordinates and polar angle coordinates corresponding to the normalized tissue feature information of the diabetic wound at different times of medical visit; Based on the normalized tissue feature information corresponding to the polar radius coordinates and polar angle coordinates at different times of medical visit, the wound area change rate or tissue composition evolution ratio is calculated as the wound healing assessment index.

6. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, The step of extracting tissue feature information characterizing the pathological state of the wound from the infrared thermal imaging image, the hyperspectral image, and the visible light color image includes: The temperature distribution information in the infrared thermal imaging image is extracted as the tissue feature information; The distribution information of oxyhemoglobin in the hyperspectral image is extracted as the tissue feature information; The color and texture features in the visible light color image are extracted as the tissue feature information.

7. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, When there are multiple preset anatomical reference markers, the step of establishing a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers includes: Within the healthy skin area surrounding the diabetic lesion, search for multiple preset anatomical reference markers; Calculate the relative distances and orientational relationships between each of the preset anatomical reference markers; Identify and remove the preset anatomical reference markers whose relative distance and orientation relationships deviate, and use the remaining preset anatomical reference markers to deduce the reference direction through preset geometric mapping logic; The virtual polar coordinate system is established based on the central reference point and the reference direction.

8. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, The step of identifying the boundary between the diabetic lesion and surrounding tissue in the visible light color image includes: The contrast between the diabetic wound and healthy skin is enhanced by color space conversion, and the wound boundary is extracted from the color space converted visible light color image using edge detection operators and region growing algorithms. Morphological closing operations are performed on the edges of the wound boundary to connect the edge break points and smooth the contour, resulting in a continuous wound boundary.

9. The multimodal image recognition method for dynamic assessment of diabetic wound healing according to claim 1, characterized in that, The step of determining the center reference point of the visible light color image based on the wound boundary includes: Calculate the geometric centroid of the region enclosed by the wound boundary; The geometric centroid is determined as the central reference point.

10. A multimodal image recognition system for dynamic assessment of diabetic wound healing, characterized in that, include: The image acquisition module is used to acquire infrared thermal images, hyperspectral images, and visible color images of diabetic wounds collected at the same time of medical visit. The wound boundary and reference point recognition module is used to identify the wound boundary between the diabetic wound and the surrounding tissue in the visible light color image, and to determine the center reference point of the visible light color image based on the wound boundary. The coordinate system establishment module is used to establish a virtual polar coordinate system based on the central reference point and the preset anatomical reference markers; The feature mapping module is used to extract tissue feature information characterizing the pathological state of the wound from the infrared thermal imaging image, the hyperspectral image and the visible light color image, and to map the spatial position of the tissue feature information in its corresponding image to the virtual polar coordinate system; The wound healing assessment module is used to normalize the mapped tissue feature information and calculate the wound healing assessment index based on the normalized tissue feature information in the virtual polar coordinate system.

Images