Image processing method, device and system and computer readable storage medium

Abstract

The invention discloses an image processing method, device and system and a computer readable storage medium. According to the method, multiple frames of to-be-selected nail fold vascular endoscopy NFC images are obtained in real time, definition evaluation is carried out on each frame of to-be-selected NFC image, a high-quality image is selected from evaluation results to serve as a to-be-processed image, manual work is not needed, images which do not meet requirements are automatically eliminated, and wrong processing results caused by low-quality images are reduced. The method comprises the following steps: performing feature extraction on a to-be-processed image through a multi-scale visual converter, capturing features of the image on different scales, and obtaining a multi-scale feature map with rich features; according to the method, a multi-scale feature pyramid is constructed, features on different scales are integrated, and comprehensive and accurate feature representation is formed to adapt to the morphological diversity characteristics of nail fold capillaries. And inputting the multi-scale feature pyramid into a multi-task model, and simultaneously performing multiple tasks of segmentation, classification and key point detection on the multi-scale feature pyramid through a single model, thereby obtaining an image processing result more efficiently and accurately.

Description

[0001] This invention claims priority from Chinese patent application number 202510694901.3 filed with the State Intellectual Property Office on May 27, 2025, entitled “A method, device, system and computer-readable storage medium for image processing,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present application relates to the technical field of nailfold capillaroscopy (NFC), and in particular to an image processing method, device, system and computer-readable storage medium. Background Art

[0003] NFC is a simple and non-invasive in vivo microscopic imaging technology, which is mainly used to observe the microvascular morphology and blood circulation status under the skin folds at the root of the nail.

[0004] In the prior art, it is necessary to manually adjust the focus of a microscope to capture NFC images, and to use manual methods to perform subsequent processing on the captured NFC images. Summary of the Invention

[0005] Based on the above problems, the present application provides an image processing method, device, system and computer-readable storage medium to efficiently and accurately capture and subsequently process NFC images.

[0006] The present application discloses a method for image processing, comprising:

[0007] Acquiring multiple frames of selected nailfold angioscopy NFC images in real time, and performing clarity evaluation on each frame of the selected NFC image to obtain an evaluation result;

[0008] Acquire a candidate NFC image whose clarity reaches a preset value in the evaluation result as an image to be processed;

[0009] Extracting features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map;

[0010] Constructing a multi-scale feature pyramid based on the multi-scale feature map, and inputting the multi-scale feature pyramid into a multi-task model;

[0011] The multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result.

[0012] Optionally, the real-time acquisition of multiple frames of selected nailfold angioscopy NFC images, and the performance of clarity evaluation on each frame of the selected NFC images to obtain an evaluation result, include:

[0013] Controlling the movement of the microscope with a preset step length, and acquiring a frame of the NFC image to be selected after each movement;

[0014] After acquiring a frame of the to-be-selected NFC image, calculating a Laplace variance value of the to-be-selected NFC image of a current frame;

[0015] Based on the Laplace variance value, an evaluation result of the NFC image to be selected in the current frame is obtained.

[0016] Optionally, obtaining, in the evaluation result, a candidate NFC image whose clarity reaches a preset value as the image to be processed includes:

[0017] controlling the movement of the microscope with a first preset step length;

[0018] When the definition reaches the fine search value, controlling the microscope to move with a second preset step length; the second preset step length is smaller than the first preset step length;

[0019] When the clarity reaches the preset value, the to-be-selected NFC image of the current frame is obtained as the image to be processed; and the preset value is greater than the fine search value.

[0020] Optionally, extracting features from the image to be processed by a multi-scale visual transformer to obtain a multi-scale feature map includes:

[0021] Dividing the image to be processed into blocks by the multi-scale visual transformer to obtain multiple patches of preset scales;

[0022] Obtain a query vector, a key vector, and a value vector for each patch respectively through a weight matrix;

[0023] Obtaining an attention score for the patch based on the query vector and the key vector;

[0024] Obtaining a weighted feature of the patch based on the attention score and the value vector;

[0025] The obtained weighted features of each patch are used as the multi-scale feature map.

[0026] Optionally, constructing a multi-scale feature pyramid based on the multi-scale feature map includes:

[0027] Extracting features of each layer of the multi-scale feature map;

[0028] Fusing the features of each layer to obtain fused features;

[0029] The fused features belonging to different patches are connected to obtain the multi-scale feature pyramid.

[0030] Optionally, the multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result, including:

[0031] Integrate the segmentation results, classification results, and key point detection results obtained based on the multi-scale feature pyramid, calculate preset parameters, and obtain calculation results;

[0032] Based on the calculation result, the image processing result is obtained; the image processing result includes a capillary segmentation result, a capillary classification result and key point position coordinates.

[0033] Based on the above-mentioned image processing method, the present application also discloses an image processing device, including: an acquisition unit, a selection unit, an extraction unit, a construction unit and an output unit;

[0034] The acquisition unit is used to acquire multiple frames of NFC images to be selected for nailfold angioscopy in real time, and perform clarity evaluation on each frame of the NFC image to be selected to obtain an evaluation result;

[0035] The selection unit is configured to obtain, from the evaluation result, a candidate NFC image whose clarity reaches a preset value as an image to be processed;

[0036] The extraction unit is configured to extract features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map;

[0037] The construction unit is configured to construct a multi-scale feature pyramid based on the multi-scale feature map, and input the multi-scale feature pyramid into a multi-task model;

[0038] The output unit is used for the multi-task model to perform segmentation, classification and key point detection on the multi-scale feature pyramid to obtain an image processing result.

[0039] Optionally, the acquisition unit includes:

[0040] an acquisition subunit, configured to control the movement of the microscope with a preset step length, and acquire a frame of the NFC image to be selected after each movement;

[0041] a variance calculation subunit, configured to calculate a Laplace variance value of the NFC image to be selected in a current frame after acquiring the NFC image to be selected;

[0042] An evaluation subunit is configured to obtain an evaluation result of the NFC image to be selected in the current frame based on the Laplace variance value.

[0043] Optionally, the selection unit includes:

[0044] a moving subunit, configured to control the movement of the microscope at a first preset step length;

[0045] a fine search subunit, configured to control the movement of the microscope with a second preset step length when the clarity reaches a fine search value; the second preset step length is smaller than the first preset step length;

[0046] The selection subunit is configured to obtain the selected NFC image of the current frame as the image to be processed when the clarity reaches the preset value; the preset value is greater than the fine search value.

[0047] Optionally, the extraction unit includes:

[0048] A blocking subunit, configured to block the image to be processed using the multi-scale visual transformer to obtain a plurality of patches of preset scales;

[0049] a weight matrix subunit, configured to obtain a query vector, a key vector, and a value vector of each patch respectively through a weight matrix;

[0050] a score acquisition subunit, configured to acquire an attention score of the patch based on the query vector and the key vector;

[0051] a weighted feature acquisition subunit, configured to acquire a weighted feature of the patch based on the attention score and the value vector;

[0052] The feature map acquisition subunit is configured to use the acquired weighted features of each patch as the multi-scale feature map.

[0053] Optionally, the construction unit includes:

[0054] An extraction subunit, configured to extract features of each layer of the multi-scale feature map;

[0055] A fusion subunit, used to fuse the features of each layer to obtain fused features;

[0056] The connection subunit is used to connect the fused features belonging to different patches to obtain the multi-scale feature pyramid.

[0057] Optionally, the output unit includes:

[0058] A calculation subunit, configured to integrate the segmentation results, classification results, and key point detection results obtained based on the multi-scale feature pyramid, calculate preset parameters, and obtain calculation results;

[0059] The output subunit is used to obtain the image processing result based on the calculation result; the image processing result includes the capillary segmentation result, the capillary classification result and the key point position coordinates.

[0060] Based on the above image processing method, the present application also discloses an image processing system for implementing the above method, the system comprising: a hardware image acquisition module and a software analysis module;

[0061] The hardware image acquisition module is used to obtain the NFC image to be selected;

[0062] The software analysis module is used to obtain an image processing result based on the NFC image to be selected.

[0063] Optionally, the hardware image acquisition module is an X, Y, and Z three-axis gantry structure; the microscope moves on the Z axis, and the finger fixing platform is located in the plane formed by the X and Y axes.

[0064] Based on the above-mentioned image processing method, the present application further discloses a computer-readable storage medium storing a computer program, which is used to implement the above-mentioned method when executed by a processor.

[0065] The present application discloses an image processing method, device, system and computer-readable storage medium. Multiple frames of NFC images to be selected for nailfold angiography are acquired in real time, and the clarity of each frame of the NFC image to be selected is evaluated to obtain an evaluation result. The NFC images to be selected whose clarity reaches a preset value in the evaluation result are acquired to automatically exclude images that do not meet the requirements without manual work, thereby reducing erroneous processing results caused by low-quality images. Feature extraction is performed on the image to be processed through a multi-scale visual transformer to capture the features of the image at different scales and obtain a multi-scale feature map with richer features. A multi-scale feature pyramid is constructed based on the multi-scale feature map, and features at different scales are integrated to form a more comprehensive and accurate feature representation to adapt to the morphological diversity of nailfold capillaries. The multi-scale feature pyramid is input into a multi-task model, and a single model is used to simultaneously perform multiple tasks of segmentation, classification and key point detection on the multi-scale feature pyramid, thereby obtaining image processing results more efficiently and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0067] Figure 1A flowchart of an image processing method disclosed in an embodiment of the present application;

[0068] Figure 2 A flowchart of another image processing method disclosed in an embodiment of the present application;

[0069] Figure 3 This is a schematic diagram of the structure of an image processing device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0071] Embodiment 1: This application discloses a method for image processing.

[0072] For details, please refer to Figure 1 , an image processing method disclosed in this embodiment includes the following steps:

[0073] Step 101: acquiring multiple frames of candidate nailfold angioscopy NFC images in real time, and performing clarity evaluation on each frame of the candidate NFC image to obtain an evaluation result.

[0074] In the method of this embodiment, a start command can be generated based on the user's finger placement action intention. Specifically, pressure-sensitive array technology can be used to detect the user's finger contact position. A preset algorithm is used to process the pressure distribution data to obtain information about the finger contact surface for subsequent NFC image acquisition. Simultaneously, a start command is generated to control the start of NFC image acquisition. Subsequently, in response to this start command, multiple frames of NFC images are acquired in real time.

[0075] The acquisition process can be specifically to automatically control the movement of the microscope with a preset step size, and acquire a frame of the selected NFC image after each movement. After each frame of the selected NFC image is acquired, the Laplace variance (Laplacian Variance) value of the selected NFC image of the current frame is calculated, that is, the variance value is calculated after the Laplace operator is convolved with the selected NFC image. Among them, the Laplace operator can be used to extract the edge information in the selected NFC image, and the variance value of the selected NFC image after Laplace transform is calculated. Based on the variance value, the evaluation result of the selected NFC image of the current frame can be obtained to indicate the clarity of the NFC image. For example, the larger the variance value, the clearer the edge of the selected NFC image.

[0076] Step 102: Acquire a candidate NFC image whose clarity reaches a preset value in the evaluation result as an image to be processed.

[0077] In the method of this embodiment, the process of acquiring the NFC image is carried out simultaneously with obtaining the evaluation results. As an optional method, the microscope is controlled to move from the starting position with a first preset step size, and the NFC image is continuously acquired during the movement of the microscope. When the clarity reaches the fine search value, the microscope is controlled to move with a second preset step size. The second preset step size is smaller than the first preset step size. For example, the first preset step size can be 0.5mm and the second preset step size can be 0.05mm. In other words, the process of controlling the movement of the microscope with the first preset step size can become a coarse search. When the clarity reaches the fine search value, the fine search is changed to a fine search, that is, the microscope is moved with a smaller step size to find the position of the microscope when the clarity of the NFC image reaches the preset value.

[0078] Accordingly, when the clarity reaches a preset value, the selected NFC image of the current frame is obtained as the image to be processed. That is, the selected NFC image of the current frame is relatively clear. The preset value is greater than the fine search value, for example, the fine search value is 0.5 and the preset value is 1. The image to be processed can be captured by a high-resolution camera (at least 1080p) connected to a microscope.

[0079] That is, in the method of this embodiment, once the user places their finger, the system can automatically locate the nail wrinkle area and obtain a clear image, thus avoiding manual operation and improving the efficiency and accuracy of NFC image acquisition.

[0080] Step 103: extract features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map.

[0081] In the method of this embodiment, the image to be processed can first undergo a round of preprocessing. The resulting image to be processed can be processed using Gaussian filtering to reduce noise. Color correction can also be performed on the image to be processed using white balance and color calibration. Subsequently, the processed image to be processed is packaged and the packaged data is sent to a server for image processing.

[0082] In the method of this embodiment, the server receives the image to be processed (in RGB format), and may first perform two rounds of preprocessing on it. Among them, the image to be processed after one round of preprocessing can be resized, for example, adjusted to a square of 1024×1024 pixels. The image to be processed can be filled, for example, a non-square image can be filled to prevent it from being distorted. In the method of this embodiment, during the model training stage, further data enhancement can be performed on the image to be processed, such as normalizing the pixel value to the range of [0,1]. Data enhancement can also be performed on the image to be processed, such as random horizontal flipping, brightness and contrast adjustment, etc. Finally, the image to be processed after two rounds of preprocessing is obtained.

[0083] In the method of this embodiment, after the two rounds of preprocessing described above, the image to be processed can be divided into blocks and input into a multiscale vision transformer (MViT). The image to be processed can be divided into multiple patches of predetermined scales. Specifically, the image to be processed can be divided into patches of size P×P. Position encoding information can also be added to each patch so that the model can know the location of each patch in the image to be processed.

[0084] In the method of this embodiment, during the model training phase, the query vector, key vector, and value vector of each patch are obtained through the weight matrix. Based on the query vector and key vector, the attention score of the patch is obtained, which can be specifically expressed as follows:

[0085]

[0086] Where A is the attention score, Q is the query vector, K is the key vector, and d is the calculation factor.

[0087] In the method of this embodiment, based on the attention score and the value vector, the weighted features of the patch can also be obtained, which can be specifically expressed by the following formula:

[0088] F=A·V (2)

[0089] Where F is the weighted feature and V is the value vector.

[0090] In the method of this embodiment, attention scores and weighted features are calculated for each patch, that is, the calculations are performed at different scales. The weighted features of each patch are then obtained as a multi-scale feature map with a size of (H / P)×(W / P)×D, where H is the height dimension, W is the width dimension, and D is the feature dimension.

[0091] Step 104: construct a multi-scale feature pyramid based on the multi-scale feature map, and input the multi-scale feature pyramid into a multi-task model.

[0092] In the method of this embodiment, the multi-scale feature map is input into the Feature Pyramid Networks (FPN) to extract the features of each layer of the multi-scale feature map. The extraction method can be to extract from the top layer of the multi-scale feature map downward to the last layer. Specifically, it can be to perform a 1×1 convolution on the top image of the multi-scale feature map to obtain the top-level features, upsample the top-level features, and perform element-level additive fusion with the second-layer features obtained by the 1×1 convolution of the second-layer image. The two-layer features after fusion can also be convolved by 3×3 to reduce the aliasing effect of upsampling. In this way, the features of the top layer and the second layer of the multi-scale feature map can be fused, and the features of all levels of the multi-scale feature map can be fused downward according to this step to finally obtain the fused features.

[0093] In the method of this embodiment, the fused features belonging to different patches are connected to obtain a multi-scale feature pyramid. That is, horizontal connections are established between features of different scales to enable information flow, ultimately obtaining a multi-scale feature pyramid.

[0094] The method of this embodiment applies MViT as the backbone network to NFC image analysis and combines it with FPN. MViT excels at capturing global relationships and multi-scale features, making it suitable for processing capillaries of varying morphologies. FPN, on the other hand, can effectively integrate features of different resolutions, improving the detection capability of small targets. The combination of the two enhances the generalization capability of the method of this embodiment. Furthermore, through the structure of a multi-scale feature pyramid, features at different scales can enhance each other, fully utilizing the information associations between scales.

[0095] Step 105: The multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result.

[0096] In the method of this embodiment, a multi-scale feature pyramid is input into a multi-task learning head, where it is used for segmentation, classification, and keypoint detection. This allows the three previously separate tasks to be integrated into a single network, sharing the underlying feature extraction process and avoiding the issues of fragmented and complex processes. For example, in the subsequent segmentation task, capillary contour information can guide the positioning of keypoint detection tasks, while the relative positions of keypoints can assist in the classification task.

[0097] Specifically, the multi-scale feature pyramid is input into the capillary segmentation head in the multi-task learning head, and the Region Proposal Network (RPN) is used to locate the potential capillary region on the feature map of the multi-scale feature pyramid as the region of interest (ROI) and use the ROIAlign operation to extract fixed-size features. Then, a mask prediction model is constructed. Specifically, four convolutional layers are applied, each layer using a 3×3 convolution kernel, 256 channels, and a ReLUReLu activation function. 1×1 convolution is then applied to reduce the number of channels to 1, and finally a sigmoid function is used to generate a pixel-level probability mask. That is, the output of the capillary segmentation head is a binary segmentation mask for each capillary region.

[0098] In the method of this embodiment, the multi-scale feature pyramid and ROI region are input into the capillary classification head in the multi-task learning head, and fixed-dimensional features are extracted from the ROI region. The classification prediction model is then constructed, specifically by applying two fully connected layers (which may include 1024 and 512 neurons, respectively), each followed by ReLU activation and Dropout operations. The last fully connected layer is made to output the logits of the number of categories, that is, the original prediction value. The softmax activation function is then applied to obtain the category probability distribution. That is, the output result of the capillary classification head is the category prediction of the capillaries, which may include normal, abnormal, and some subcategories.

[0099] In the method of this embodiment, the multi-scale feature pyramid and ROI area are input into the key point detection head in the multi-task learning head, and fixed-dimensional features are extracted from the ROI area. Then the heat map generation model is constructed. Specifically, eight convolution layers can be applied, each layer uses a 3×3 convolution kernel, 256 channels and a ReLU activation function, and then for each key point type (there are 9 types in this embodiment), a separate 1×1 convolution layer is used. Generate a heat map of S×S size, where S is the heat map resolution. Apply softmax to each heat map to obtain the spatial probability distribution, and calculate the weighted average position of the heat map to obtain the key point coordinates. That is, the output result of the key point detection head is the coordinates and visibility flags of 9 key points. Among them, the visibility flag participates in the deep learning model training as a parameter of the key point.

[0100] In the method of this embodiment, during the training and optimization process of the model, the uncertainty weighted loss can be calculated for the output results of the above three tasks. Specifically, the three output results and the true value (including the label, mask, and bounding box) are input to calculate the specific loss of each task. The loss calculation formula can be as follows:

[0101]

[0102] Where, L segm is the segmentation loss, S is the predicted mask in the output result, g is the true segmentation mask, and i is the i-th output result.

[0103]

[0104] Where, L class is the classification loss, y is the true label, is the category probability in the output result, and c is the c-th output result.

[0105]

[0106] Where, L bbox is the bounding box loss, B is the predicted bounding box in the output result of the segmentation task, G is the true bounding box, and B i (j) is the predicted bounding box of the jth nailfold instance in the output of the i-th image segmentation task, G i (j) is the ground-truth bounding box of the j-th nailfold instance in the output of the i-th image segmentation task.

[0107] The key point loss is the cross entropy loss based on the key point heat map.

[0108] Subsequently, a learnable uncertainty parameter is assigned to each task, and the overall weighted loss is finally output for model optimization, which can be expressed as follows:

[0109]

[0110] Where, L total is the overall weighted loss, U and W are coefficients.

[0111] The uncertainty weighted loss calculation in the method of this embodiment can automatically balance the learning difficulty and importance of different tasks based on the adaptive weighted loss function of task uncertainty, ensure the balanced development of the model on each task, and improve the overall performance.

[0112] In the method of this embodiment, the acquired segmentation results, classification results, and keypoint detection results are integrated into a unified representation, and preset parameters are calculated to obtain the calculated results. For example, a nine-point keypoint system is defined for nailfold vessels, including the upper point (U), lower point (D), left-left point (LL), left-left point (LR), right-left point (RL), right-right point (RR), left bottom point, right bottom point, and optional connecting points. This provides a more comprehensive representation of capillary structure, facilitating the accurate calculation of various clinical parameters.

[0113] Specifically, it can be calculated as the Euclidean distance between the top diameter U and D, the Euclidean distance between the arterial branch diameter LL and LR, the Euclidean distance between the venous branch diameter RL and RR, and other clinically relevant parameters.

[0114] Finally, based on the calculation results, the image processing results can be obtained, which may include capillary segmentation results (binary segmentation mask), capillary classification results (normal, abnormal and its subcategories), key point position coordinates, and other clinically relevant parameters (such as diameter, etc.).

[0115] The method described in this embodiment simultaneously performs three tasks on NFC images: capillary segmentation, vessel classification, and key point detection. It can also automatically calculate the various vascular parameters required for clinical diagnosis. The unified architecture eliminates the fragmentation of clinical processes, and maximizes information utilization and effectively integrates resources through cross-task feature sharing, reducing the system deployment threshold and maintenance costs. At the same time, the advanced neural network architecture improves the generalization ability of the model in complex clinical scenarios. During the process, the user only needs to place a finger, and the model can automatically locate the nail fold area on the user's finger to obtain a clear NFC image. The NFC image is then automatically processed to obtain the morphological characteristics and abnormalities of the capillaries in order to generate an intuitive diagnostic report. The above steps are all completed end-to-end automatically. The hardware part is responsible for obtaining high-quality NFC images, and the software part is responsible for in-depth analysis of capillary structure and parameters. The two are seamlessly integrated through an interface. No manual intervention is required, the processing speed is fast, and the efficiency is better than manual operation while also ensuring accuracy. Ultimately, a set of efficient, accurate, and clinically suitable NFC image processing methods is achieved.

[0116] Example 2: This application discloses another method for image processing, please refer to Figure 2 The method described in this embodiment proposes a complete NFC image acquisition and processing method, realizing full process automation.

[0117] Step 201: Initialize the servo motor and microscope on the three-axis gantry structure, and load the pre-trained model on the server side.

[0118] Step 202: In response to the user placing a finger on the finger fixing platform, the servo motor moves on the X and Y axes of the three-axis gantry to the nail wrinkle area.

[0119] Step 203: Control the microscope to move on the Z axis of the three-axis gantry with a step length of 0.5 mm, collect a candidate NFC image after each movement, and perform a clarity evaluation on the candidate NFC image.

[0120] Step 204: When the evaluation result meets the fine search requirement, the microscope moves on the Z axis with a step length of 0.05 mm, collects a candidate NFC image after each movement, and performs a clarity evaluation on the candidate NFC image.

[0121] Step 205: When the evaluation result meets the preset requirements, the movement of the microscope is stopped and the image to be processed is acquired at the current position.

[0122] Step 206: Extract features from the image to be processed through MViT and input the feature map into FPN.

[0123] Step 207: FPN constructs a multi-scale feature pyramid based on the input data and inputs it into the multi-task model.

[0124] Step 208: The multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result.

[0125] Based on the image processing method disclosed in the above embodiment, this embodiment correspondingly discloses an image processing device. Figure 3 , the image processing device includes: an acquisition unit 301, a selection unit 302, an extraction unit 303, a construction unit 304 and an output unit 305;

[0126] The acquisition unit 301 is configured to acquire multiple frames of selected nailfold angioscopy NFC images in real time, and perform clarity evaluation on each frame of the selected NFC image to obtain an evaluation result;

[0127] The selection unit 302 is configured to obtain a candidate NFC image whose clarity reaches a preset value in the evaluation result as an image to be processed;

[0128] The extraction unit 303 is configured to extract features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map;

[0129] The construction unit 304 is configured to construct a multi-scale feature pyramid based on the multi-scale feature map, and input the multi-scale feature pyramid into a multi-task model;

[0130] The output unit 305 is used for the multi-task model to perform segmentation, classification and key point detection on the multi-scale feature pyramid to obtain an image processing result.

[0131] Optionally, the acquisition unit 301 includes:

[0132] an acquisition subunit, configured to control the movement of the microscope with a preset step length, and acquire a frame of the NFC image to be selected after each movement;

[0133] a variance calculation subunit, configured to calculate a Laplace variance value of the NFC image to be selected in a current frame after acquiring the NFC image to be selected;

[0134] An evaluation subunit is configured to obtain an evaluation result of the NFC image to be selected in the current frame based on the Laplace variance value.

[0135] Optionally, the selection unit 302 includes:

[0136] a moving subunit, configured to control the movement of the microscope at a first preset step length;

[0137] a fine search subunit, configured to control the movement of the microscope with a second preset step length when the clarity reaches a fine search value; the second preset step length is smaller than the first preset step length;

[0138] The selection subunit is configured to obtain the selected NFC image of the current frame as the image to be processed when the clarity reaches the preset value; the preset value is greater than the fine search value.

[0139] Optionally, the extraction unit 303 includes:

[0140] A blocking subunit, configured to block the image to be processed using the multi-scale visual transformer to obtain a plurality of patches of preset scales;

[0141] a weight matrix subunit, configured to obtain a query vector, a key vector, and a value vector of each patch respectively through a weight matrix;

[0142] a score acquisition subunit, configured to acquire an attention score of the patch based on the query vector and the key vector;

[0143] a weighted feature acquisition subunit, configured to acquire a weighted feature of the patch based on the attention score and the value vector;

[0144] The feature map acquisition subunit is configured to use the acquired weighted features of each patch as the multi-scale feature map.

[0145] Optionally, the construction unit 304 includes:

[0146] An extraction subunit, configured to extract features of each layer of the multi-scale feature map;

[0147] A fusion subunit, used to fuse the features of each layer to obtain fused features;

[0148] The connection subunit is used to connect the fused features belonging to different patches to obtain the multi-scale feature pyramid.

[0149] Optionally, the output unit 305 includes:

[0150] A calculation subunit, configured to integrate the segmentation results, classification results, and key point detection results obtained based on the multi-scale feature pyramid, calculate preset parameters, and obtain calculation results;

[0151] The output subunit is used to obtain the image processing result based on the calculation result; the image processing result includes the capillary segmentation result, the capillary classification result and the key point position coordinates.

[0152] Based on the above image processing method, the present application also discloses an image processing system for implementing the above method, the system comprising: a hardware image acquisition module and a software analysis module;

[0153] The hardware image acquisition module is used to obtain the NFC image to be selected;

[0154] The software analysis module is used to obtain an image processing result based on the NFC image to be selected.

[0155] In this embodiment, the hardware image acquisition module is a three-axis X, Y, and Z gantry structure. This three-axis gantry utilizes high-precision linear guides and operates high-precision servo motors, ensuring motion accuracy of ±0.01mm. The entire structure can be constructed using an aluminum frame to provide stable support.

[0156] In this embodiment, the microscope (200x magnification) is fixed and moves along the Z axis, while the finger support platform, located in the plane formed by the X and Y axes, is equipped with a comfortable hand support and finger positioning grooves. The finger support platform can use pressure-sensitive array technology to detect the contact position of the user's finger in real time. The pressure distribution data is processed by the algorithm embedded in the ESP32 microcontroller to drive the servo motor to move the microscope. A ring lighting system can also be set up to provide uniform and shadow-free illumination.

[0157] In this embodiment, the system combines a three-axis gantry structure with an ESP32 microcontroller, employing closed-loop feedback control. This allows for dynamic adjustment of the microscope's focus, ensuring automatic location for the clearest vascular imaging even in minimal vibration environments, and obtaining stable, high-resolution, high-contrast images. This enables adaptive focusing and micron-level positioning for microscopic imaging, enabling automated NFC image acquisition without manual adjustments.

[0158] The system in this embodiment requires no manual operation and can fully automatically capture and process capillary images. It simultaneously completes multiple processing tasks while imaging, reducing clinical workflow steps. It also features sub-pixel automatic capillary parameter measurement, with an error of less than 1 pixel in measuring the diameters of the input and output branches of nailfold vessels. It also automatically classifies abnormal capillaries, achieving an accuracy of 0.93 in experiments.

[0159] Based on the above-mentioned image processing method, the present application further discloses a computer-readable storage medium storing a computer program, which is used to implement the above-mentioned method when executed by a processor.

[0160] The embodiments in this specification are described in a progressive manner. As for the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0161] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0162] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0163] The features described in the embodiments of this specification can be replaced with each other or combined to enable professional and technical personnel in this field to implement or use this application.

[0164] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image processing, characterized in that: include: Acquiring multiple frames of selected nailfold angioscopy NFC images in real time, and performing clarity evaluation on each frame of the selected NFC image to obtain an evaluation result; Acquire a candidate NFC image whose clarity reaches a preset value in the evaluation result as an image to be processed; Extracting features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map; Constructing a multi-scale feature pyramid based on the multi-scale feature map, and inputting the multi-scale feature pyramid into a multi-task model; The multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result.

2. The method according to claim 1, characterized in that The real-time acquisition of multiple frames of selected nailfold angioscopy NFC images and the performance of clarity evaluation on each frame of the selected NFC images to obtain an evaluation result include: Controlling the movement of the microscope with a preset step length, and acquiring a frame of the NFC image to be selected after each movement; After acquiring a frame of the to-be-selected NFC image, calculating a Laplace variance value of the to-be-selected NFC image of a current frame; Based on the Laplace variance value, an evaluation result of the NFC image to be selected in the current frame is obtained.

3. The method according to claim 2, characterized in that The step of obtaining a candidate NFC image whose clarity reaches a preset value in the evaluation result as an image to be processed includes: controlling the movement of the microscope with a first preset step length; When the definition reaches the fine search value, controlling the microscope to move with a second preset step length; the second preset step length is smaller than the first preset step length; When the clarity reaches the preset value, the to-be-selected NFC image of the current frame is obtained as the image to be processed; and the preset value is greater than the fine search value.

4. The method according to claim 1, wherein The step of extracting features from the image to be processed by a multi-scale visual transformer to obtain a multi-scale feature map includes: Dividing the image to be processed into blocks by the multi-scale visual transformer to obtain multiple patches of preset scales; Obtain a query vector, a key vector, and a value vector for each patch respectively through a weight matrix; Obtaining an attention score for the patch based on the query vector and the key vector; Obtaining a weighted feature of the patch based on the attention score and the value vector; The obtained weighted features of each patch are used as the multi-scale feature map.

5. The method according to claim 4, characterized in that The constructing a multi-scale feature pyramid based on the multi-scale feature map includes: Extracting features of each layer of the multi-scale feature map; Fusing the features of each layer to obtain fused features; The fused features belonging to different patches are connected to obtain the multi-scale feature pyramid.

6. The method according to claim 1, characterized in that The multi-task model performs segmentation, classification, and key point detection on the multi-scale feature pyramid to obtain an image processing result, including: Integrate the segmentation results, classification results, and key point detection results obtained based on the multi-scale feature pyramid, calculate preset parameters, and obtain calculation results; Based on the calculation result, the image processing result is obtained; the image processing result includes a capillary segmentation result, a capillary classification result and key point position coordinates.

7. An image processing device, characterized in that: include: Acquisition unit, selection unit, extraction unit, construction unit and output unit; The acquisition unit is used to acquire multiple frames of NFC images to be selected for nailfold angioscopy in real time, and perform clarity evaluation on each frame of the NFC image to be selected to obtain an evaluation result; The selection unit is configured to obtain, from the evaluation result, a candidate NFC image whose clarity reaches a preset value as an image to be processed; The extraction unit is configured to extract features from the image to be processed using a multi-scale visual transformer to obtain a multi-scale feature map; The construction unit is configured to construct a multi-scale feature pyramid based on the multi-scale feature map, and input the multi-scale feature pyramid into a multi-task model; The output unit is used for the multi-task model to perform segmentation, classification and key point detection on the multi-scale feature pyramid to obtain an image processing result.

8. An image processing system, characterized in that: For implementing the method according to any one of claims 1 to 6, the system comprises: a hardware image acquisition module and a software analysis module; The hardware image acquisition module is used to obtain the NFC image to be selected; The software analysis module is used to obtain an image processing result based on the NFC image to be selected.

9. The system according to claim 8, characterized in that The hardware image acquisition module is an X, Y, and Z three-axis gantry structure; the microscope moves on the Z axis, and the finger fixing platform is located in the plane formed by the X and Y axes.

10. A computer-readable storage medium, characterized in that A computer program is stored, and when the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 6.

Images