Skin image feature analysis method based on a reflection confocal microscope

Abstract

The present application belongs to the field of image feature analysis, and specifically refers to a skin image feature analysis method based on a reflective confocal microscope, which comprises distortion correction, image preprocessing and global perception. Through GPU accelerated lookup table encoding and decoding technology, the efficiency of distortion correction is significantly improved, the number of memory access is reduced, real-time and efficient image processing is realized, the quality of the skin medical image is improved in combination with medical imaging technology, which has great significance for clinical application; in combination with feature extraction, matching and global optimization technology, a high-precision global skin image is obtained and updated, effectively solving the geometric distortion problem caused by probe jitter or inter-frame alignment error, and being able to adapt to the instability of artificial handheld operation, generating a global skin image in single-point imaging of a shaking skin lesion position, and ensuring the continuity and consistency of image acquisition.

Description

Technical Field

[0001] This invention relates to the field of image feature analysis, specifically to a method for skin image feature analysis based on a reflection confocal microscope. Background Technology

[0002] Reflection confocal microscopy is a skin imaging diagnostic technique that uses an 830nm semiconductor laser as a point light source to perform dynamic scanning and sampling of the epidermis and superficial dermis layer by layer. It uses computer technology to obtain three-dimensional image information and is also known as skin CT. The skin image feature analysis method based on reflection confocal microscopy refers to a method of processing skin image features through reflection confocal microscopy.

[0003] Among existing approximate solutions, such as CN114399510B, a method and system for skin lesion segmentation and classification that combines images and clinical metadata, this approach addresses technical issues such as blurred skin lesion boundaries, visual similarity between skin lesions of different classes, significant visual changes in skin lesions in dermoscopic images, and artifacts that obscure skin lesions. First, lesion segmentation is performed, using a fusion decoder to decode information at different scales and generate image masks of the lesions. Second, lesion classification is performed by introducing two feature extractors: one based on EfficientNet for extracting features from dermoscopic images and the other based on a shallow convolutional neural network for extracting clinical metadata. The dermoscopic images and clinical data are dynamically fused, and the final output is the category of the skin lesion. This technique improves the performance of skin lesion segmentation and classification. However, existing technologies suffer from technical problems such as skin image deformation during the scanning process using a reflective confocal microscope probe, poor contact in edge areas due to uneven skin surfaces or probe tilt, leading to signal attenuation and deformation, and the potential presence of air bubbles in the gel coupling between the reflective confocal microscope probe and the patch.

[0004] Furthermore, for example, CN114241538B describes an image feature recognition method and system based on Cushing's syndrome. This approach addresses the technical problem of some patients lacking disease awareness and delaying optimal treatment. It employs dynamic electrical stimulation assessment, comparing differences in muscle contraction amplitude through electrical stimulation of the skin to quantify skin elasticity. It enhances texture feature extraction by repeatedly flashing the skin area with a specified color light source. It also uses vascular features to assist diagnosis, outputting a disease risk probability value by analyzing the number and clarity of blood vessels in the target texture extension area. This achieves the technical effect of improving the accuracy of predicting disease risk based on appearance. However, there are still some technical problems, such as significant differences in skin sensitivity among different users, the possibility that a fixed current intensity may cause skin discomfort or damage, the possibility that ambient light may interfere with the imaging effect of the specified color light source and affect texture feature extraction, and the possibility that low vascular visibility in some patients may lead to failure in vascular feature extraction. Existing reflective confocal microscopes use single-point laser scanning with a probe, lacking a comprehensive skin perception pathway. Summary of the Invention

[0005] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a skin image feature analysis method based on a reflection confocal microscope. It addresses the technical problems of existing technologies, such as skin image deformation during the scanning process using the detection probe of a reflection confocal microscope, poor contact in edge areas due to uneven skin surfaces or probe tilting, leading to signal attenuation and deformation, and the potential for air bubbles in the gel coupling between the detection probe and the patch. This solution utilizes GPU-accelerated lookup table encoding and decoding technology to significantly improve distortion correction efficiency, reduce memory accesses, and achieve real-time, efficient image processing. Combined with medical imaging technology, it improves the quality of skin medical images, which is of great significance for clinical applications. Furthermore, it addresses the issue of varying skin sensitivities among users in existing technologies. Significant differences exist between the two systems. Fixed current intensity may cause skin discomfort or damage. Ambient light may interfere with the imaging effect of a specified color light source, affecting texture feature extraction. Low vascular visibility in some patients may lead to failure in vascular feature extraction. Existing reflective confocal microscopes perform single-point laser scanning through the probe, lacking a technical approach for overall skin perception. This solution combines feature extraction, matching, and global optimization techniques to acquire and update high-precision global skin images. It effectively solves the geometric distortion problem caused by probe jitter or inter-frame alignment errors, adapts to the instability of manual handheld operation, and generates global skin images from single-point imaging of shaky lesion locations, ensuring the continuity and consistency of image acquisition. In addition, the frame acquisition frequency is controlled by a feature point number threshold, avoiding unnecessary image acquisition and saving computing resources and storage space.

[0006] The technical solution adopted by this invention is as follows: The skin image feature analysis method based on a reflection confocal microscope provided by this invention includes the following steps:

[0007] Step S1: Distortion correction, build a lookup table, and obtain the true image;

[0008] Step S2: Image preprocessing, using image preprocessing techniques to perform noise reduction and standardization operations on the real image;

[0009] Step S3: Global perception, used to ensure the spatial continuity and consistency of the image. Specifically, it finds the correspondence between overlapping areas between adjacent real images between frames, aligns and stitches real images taken at different times to obtain a global skin image, and dynamically expands the global skin image in real time as the probe moves.

[0010] Further, in step S1, the distortion correction specifically includes the following steps:

[0011] Step S11: Probe distortion modeling. Specifically, the skin is scanned using the detection probe of a reflection confocal microscope to obtain a distorted image. The detection probe of the reflection confocal microscope is referred to as the probe. For edge distortion of the probe, the correspondence between the coordinates of any pixel in the distorted image and the coordinates of the pixel in the distorted corrected image is calculated, and a lookup table is established. The formula used is as follows:

[0012] ;

[0013] ;

[0014] In the formula, Indicates the distortion distance. This represents the pixel coordinates in the real image after distortion correction. Represents the pixel coordinates in the distorted image. Indicates the center point of the distorted image. and Both represent distortion coefficients;

[0015] Step S12: Encode the lookup table to efficiently access the lookup table in the GPU (Graphics Processing Unit) and reduce the number of memory accesses. Unencoded entries need to be read from memory twice, while compressed entries only need to be read once. Specifically, the pixel coordinates in the distortion-corrected real image are encoded into a one-dimensional integer. Each entry in the lookup table only stores the one-dimensional integer. Considering the locality of memory access, the encoding order is determined.

[0016] Step S13: Decoding, used to achieve efficient real-time lens distortion correction. Specifically, the GPU decodes the one-dimensional integer back to the pixel coordinates in the distortion-corrected real image through integer division and modulo operation. A lookup table is used to obtain the mapping relationship, mapping the distorted image acquired by the probe to the real image. The formula used is as follows:

[0017] ;

[0018] In the formula, Round down to the nearest integer. Modulo operation Represents a one-dimensional integer. Represents the width of the actual image. This represents the height of the actual image.

[0019] Further, in step S12, the encoding lookup table specifically includes the following steps:

[0020] Step S121: If the width of the real image is greater than or equal to its height, encode it in row-major order;

[0021] Step S122: If the width of the real image is less than its height, encode it in column-major order to ensure that the one-dimensional integer is unique and reversible;

[0022] Step S123: Calculate a one-dimensional integer, the formula for which the one-dimensional integer is calculated is as follows:

[0023] .

[0024] Furthermore, in step S3, the global perception specifically includes the following steps:

[0025] Step S31: Feature extraction, used to dynamically adjust the frame interval. Specifically, a scale-invariant feature transformation algorithm is used to extract features from the real image. A frame interval and threshold are preset. The number of features in the current frame of the real image is counted. If the number of features is less than the threshold, the next frame of the real image is captured and saved in advance. Otherwise, the next frame of the real image is captured and saved after the frame interval has elapsed. This is to ensure that the real image has sufficient features, avoid expansion failure due to insufficient features, and reduce unnecessary frame captures.

[0026] Step S32: Feature matching, specifically, the nearest neighbor distance ratio method is used to match the features of different real images. Each real image is aligned with the next frame through the homography matrix, and the global skin image is gradually expanded. The homography matrix is ​​used to describe the geometric transformation relationship between real images. Each homography matrix transformation will introduce a small pose error. As the number of frames increases, the error accumulates, causing obvious distortion at the edge of the panoramic image, resulting in geometric distortion.

[0027] Step S33: Probe pose sensing to suppress geometric distortion. Specifically, the probe may shake and shift due to manual hand operation. The probe pose is initialized, including the probe's rotation matrix and translation vector. A gyroscope is used to detect the probe's motion and obtain a 3D motion vector. The gyroscope is fixed to the outer wall of the probe. The probe pose is updated based on the 3D motion vector. The formula for the 3D motion vector is as follows:

[0028] ;

[0029] In the formula, Represents a 3D motion vector. Indicates probe edge Angular acceleration of the axis, Indicates probe edge Angular acceleration of the axis, Indicates probe edge Angular acceleration of the axis;

[0030] Step S34: Global optimization, specifically, distributing the accumulated pose error to the probe's rotation matrix, the probe's translation vector, and the 3D point coordinates of the feature points.

[0031] Furthermore, in step S34, the global optimization specifically includes the following steps:

[0032] Step S341: Based on the feature matching results, obtain all real images containing the same feature, and record the features that are successfully matched in the real images as feature points;

[0033] Step S342: Rotate any feature point A in the world coordinate system to the probe coordinate system, and then project it onto the real image plane to obtain the predicted coordinates;

[0034] Step S343: Calculate the Euclidean distance between the predicted coordinates and the coordinates of feature point A in other real images containing feature point A, i.e., the projection error. Minimize the sum of the projection errors of all feature points, optimize the probe pose, reproject the feature points to all real images, and update the global skin image. The formula for minimizing the sum of the projection errors of all feature points is as follows:

[0035] ;

[0036] In the formula, This represents the rotation matrix of the probe in the i-th frame of the real image. Let represent the translation vector of the probe in the i-th frame of the real image. This represents the 3D coordinates of the j-th feature point. express The 2D pixel coordinates matched by features in the i-th frame of the real image. This represents the intrinsic parameter matrix of the probe. Represents the projection function. This represents the Euclidean distance, i.e., the projection error. This represents the sum of projection errors for all feature points that successfully match the feature.

[0037] This invention provides a method for skin image feature analysis based on a reflection confocal microscope. The beneficial effects achieved by this invention using the above scheme are as follows:

[0038] (1) In view of the technical problems in the existing technology, when the detection probe of the reflection confocal microscope is pressed to scan the skin, the skin image will be deformed. When the skin surface is uneven or the probe is tilted, the edge area will have poor contact, resulting in signal attenuation and deformation. In addition, the detection probe and the patch of the reflection confocal microscope are coupled through gel, and the gel may contain air bubbles. This solution uses GPU-accelerated lookup table encoding and decoding technology to significantly improve the efficiency of distortion correction, reduce the number of memory accesses, realize real-time and efficient image processing, and improve the quality of skin medical images when combined with medical imaging technology, which is of great significance for clinical application.

[0039] (2) In view of the existing technology, there are large differences in skin sensitivity among different users, fixed current intensity may cause skin discomfort or damage, ambient light may interfere with the imaging effect of the specified color light source and affect the extraction of texture features, and the low visibility of blood vessels in some patients may lead to failure of blood vessel feature extraction. The existing reflective confocal microscopes perform single-point laser scanning through the probe and lack a technical approach to overall skin perception. This solution combines feature extraction, matching and global optimization technology to acquire and update high-precision global skin images, effectively solving the geometric distortion problem caused by probe jitter or inter-frame alignment error. It can adapt to the instability of manual handheld operation and generate global skin images in single-point imaging of shaky lesion locations, ensuring the continuity and consistency of image acquisition. In addition, the frame shooting frequency is controlled by the feature point number threshold, avoiding unnecessary image acquisition and saving computing resources and storage space. Attached Figure Description

[0040] Figure 1 A schematic flowchart illustrating the skin image feature analysis method based on a reflection confocal microscope provided by this invention;

[0041] Figure 2 This is a schematic diagram of step S1;

[0042] Figure 3 This is a schematic diagram of step S3.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0045] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0046] Example 1, see Figures 1 to 3 The present invention provides a skin image feature analysis method based on a reflection confocal microscope, the method comprising the following steps:

[0047] Step S1: Distortion correction, build a lookup table, and obtain the true image;

[0048] Step S2: Image preprocessing, using image preprocessing techniques to perform noise reduction and standardization operations on the real image;

[0049] Step S3: Global perception, used to ensure the spatial continuity and consistency of the image. Specifically, it finds the correspondence between overlapping areas between adjacent real images between frames, aligns and stitches real images taken at different times to obtain a global skin image, and dynamically expands the global skin image in real time as the probe moves.

[0050] Example 2, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S1, the distortion correction specifically includes the following steps:

[0051] Step S11: Probe distortion modeling. Specifically, the skin is scanned using the detection probe of a reflection confocal microscope to obtain a distorted image. The detection probe of the reflection confocal microscope is referred to as the probe. For edge distortion of the probe, the correspondence between the coordinates of any pixel in the distorted image and the coordinates of the pixel in the distorted corrected image is calculated, and a lookup table is established. The formula used is as follows:

[0052] ;

[0053] ;

[0054] In the formula, Indicates the distortion distance. This represents the pixel coordinates in the real image after distortion correction. Represents the pixel coordinates in the distorted image. Indicates the center point of the distorted image. and Both represent distortion coefficients;

[0055] Step S12: Encode the lookup table to efficiently access the lookup table in the GPU (Graphics Processing Unit) and reduce the number of memory accesses. Unencoded entries need to be read from memory twice, while compressed entries only need to be read once. Specifically, the pixel coordinates in the distortion-corrected real image are encoded into a one-dimensional integer. Each entry in the lookup table only stores the one-dimensional integer. Considering the locality of memory access, the encoding order is determined.

[0056] Step S13: Decoding, used to achieve efficient real-time lens distortion correction. Specifically, the GPU decodes the one-dimensional integer back to the pixel coordinates in the distortion-corrected real image through integer division and modulo operation. A lookup table is used to obtain the mapping relationship, mapping the distorted image acquired by the probe to the real image. The formula used is as follows:

[0057] ;

[0058] In the formula, Round down to the nearest integer. Modulo operation Represents a one-dimensional integer. Represents the width of the actual image. This represents the height of the actual image.

[0059] Example 3, see Figures 1 to 2 This embodiment is based on the above embodiment. In step S12, the encoding lookup table specifically includes the following steps:

[0060] Step S121: If the width of the real image is greater than or equal to its height, encode it in row-major order;

[0061] Step S122: If the width of the real image is less than its height, encode it in column-major order to ensure that the one-dimensional integer is unique and reversible;

[0062] Step S123: Calculate a one-dimensional integer, the formula for which the one-dimensional integer is calculated is as follows:

[0063] .

[0064] Example 4, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S3, the global perception specifically includes the following steps:

[0065] Step S31: Feature extraction, used to dynamically adjust the frame interval. Specifically, a scale-invariant feature transformation algorithm is used to extract features from the real image. A frame interval and threshold are preset. The number of features in the current frame of the real image is counted. If the number of features is less than the threshold, the next frame of the real image is captured and saved in advance. Otherwise, the next frame of the real image is captured and saved after the frame interval has elapsed. This is to ensure that the real image has sufficient features, avoid expansion failure due to insufficient features, and reduce unnecessary frame captures.

[0066] Step S32: Feature matching, specifically, the nearest neighbor distance ratio method is used to match the features of different real images. Each real image is aligned with the next frame through the homography matrix, and the global skin image is gradually expanded. The homography matrix is ​​used to describe the geometric transformation relationship between real images. Each homography matrix transformation will introduce a small pose error. As the number of frames increases, the error accumulates, causing obvious distortion at the edge of the panoramic image, resulting in geometric distortion.

[0067] Step S33: Probe pose sensing to suppress geometric distortion. Specifically, the probe may shake and shift due to manual hand operation. The probe pose is initialized, including the probe's rotation matrix and translation vector. A gyroscope is used to detect the probe's motion and obtain a 3D motion vector. The gyroscope is fixed to the outer wall of the probe. The probe pose is updated based on the 3D motion vector. The formula for the 3D motion vector is as follows:

[0068] ;

[0069] In the formula, Represents a 3D motion vector. Indicates probe edge Angular acceleration of the axis, Indicates probe edge Angular acceleration of the axis, Indicates probe edge Angular acceleration of the axis;

[0070] Step S34: Global optimization, specifically, distributing the accumulated pose error to the probe's rotation matrix, the probe's translation vector, and the 3D point coordinates of the feature points.

[0071] Example 5, see Figures 1 to 3This embodiment is based on the above embodiment. In step S34, the global optimization specifically includes the following steps:

[0072] Step S341: Based on the feature matching results, obtain all real images containing the same feature, and record the features that are successfully matched in the real images as feature points;

[0073] Step S342: Rotate any feature point A in the world coordinate system to the probe coordinate system, and then project it onto the real image plane to obtain the predicted coordinates;

[0074] Step S343: Calculate the Euclidean distance between the predicted coordinates and the coordinates of feature point A in other real images containing feature point A, i.e., the projection error. Minimize the sum of the projection errors of all feature points, optimize the probe pose, reproject the feature points to all real images, and update the global skin image. The formula for minimizing the sum of the projection errors of all feature points is as follows:

[0075] ;

[0076] In the formula, This represents the rotation matrix of the probe in the i-th frame of the real image. Let represent the translation vector of the probe in the i-th frame of the real image. This represents the 3D coordinates of the j-th feature point. express The 2D pixel coordinates matched by features in the i-th frame of the real image. This represents the intrinsic parameter matrix of the probe. Represents the projection function. This represents the Euclidean distance, i.e., the projection error. This represents the sum of projection errors for all feature points that successfully match the feature.

[0077] Example 6, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S31, the feature extraction specifically involves extracting the nested structure of melanoma in the real image.

[0078] Example 7, see Figures 1 to 3 This embodiment is based on the above embodiment. In step S2, the image preprocessing operation is implemented using the C# language.

[0079] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0081] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for skin image feature analysis based on a reflection confocal microscope, characterized in that: The method includes the following steps: Step S1: Distortion correction, establishing a lookup table to obtain the true image, specifically including the following steps: Step S11: Probe distortion modeling. Specifically, the skin is scanned using the detection probe of a reflective confocal microscope to obtain a distorted image. The detection probe of the reflective confocal microscope is referred to as the probe. For the edge distortion of the probe, the correspondence between any pixel coordinate in the distorted image and the pixel coordinate in the real image after distortion correction is calculated, and a lookup table is established. Step S12: Encoding the lookup table, specifically, encoding the pixel coordinates in the distortion-corrected real image into a one-dimensional integer, with each entry in the lookup table storing only the one-dimensional integer. Considering the locality of memory access, the encoding order is determined, specifically including the following steps: Step S121: If the width of the real image is greater than or equal to its height, encode it in row-major order; Step S122: If the width of the real image is less than its height, encode it in column-major order; Step S123: Calculate a one-dimensional integer; Step S13: Decoding, specifically, the GPU decodes the one-dimensional integer back to the pixel coordinates in the distortion-corrected real image through integer division and modulo operation, obtains the mapping relationship through a lookup table, and maps the distorted image acquired by the probe to the real image; Step S2: Image preprocessing, using image preprocessing techniques to perform noise reduction and standardization operations on the real image; Step S3: Global perception, specifically, finding the correspondence between overlapping areas between adjacent real images in frames, aligning and stitching real images taken at different times to obtain a global skin image, and dynamically expanding the global skin image in real time as the probe moves.

2. The skin image feature analysis method based on a reflection confocal microscope according to claim 1, characterized in that: In step S3, the global perception specifically includes the following steps: Step S31: Feature extraction, used to dynamically adjust the frame interval. Specifically, the scale-invariant feature transformation algorithm is used to extract features from the real image. The frame interval and threshold are preset. The number of features of the current frame real image is counted. If the number of features is less than the threshold, the next frame real image is captured and saved in advance. Otherwise, the next frame real image is captured and saved after the frame interval has elapsed. Step S32: Feature matching, specifically, the nearest neighbor distance ratio method is used to match the features of different real images. Each real image is aligned with the next frame through the homography matrix, and the global skin image is gradually expanded. The homography matrix is ​​used to describe the geometric transformation relationship between real images. Each homography matrix transformation will introduce a small pose error. As the number of frames increases, the error accumulates, causing obvious distortion at the edge of the panoramic image, resulting in geometric distortion. Step S33: Probe pose perception, specifically, initializing the probe pose, which includes the probe's rotation matrix and translation vector, using a gyroscope to detect the probe's motion and obtain a 3D motion vector, the gyroscope being fixed to the outer wall of the probe, and updating the probe pose based on the 3D motion vector; Step S34: Global optimization, specifically, distributing the accumulated pose error to the probe's rotation matrix, the probe's translation vector, and the 3D point coordinates of the feature points.

3. The skin image feature analysis method based on a reflection confocal microscope according to claim 2, characterized in that: In step S34, the global optimization specifically includes the following steps: Step S341: Based on the feature matching results, obtain all real images containing the same feature, and record the features that are successfully matched in the real images as feature points; Step S342: Rotate any feature point A in the world coordinate system to the probe coordinate system, and then project it onto the real image plane to obtain the predicted coordinates; Step S343: Calculate the Euclidean distance between the predicted coordinates and the coordinates of feature point A in other real images containing feature point A, i.e., the projection error. Minimize the sum of the projection errors of all feature points, optimize the probe pose, reproject the feature point to all real images, and update the global skin image.

Images