A method and system for automatically measuring nail capillary density

Abstract

The application discloses a kind of nail fold capillary density automatic measurement method and system, the method includes obtaining nail fold image collected by microscope camera;The nail fold image input is identified and detected by pre-trained detection neural network model, to obtain the detection bounding box of candidate blood vessels in the nail fold image;The detection bounding box of candidate blood vessels is screened and handled by blood vessel screening model, to obtain target capillary;According to the quantity of target capillary and the size information of nail fold image, the capillary density of nail fold image is calculated.The present application can automatically measure nail fold capillary density, with high processing efficiency, short processing time, high accuracy of measurement results and other characteristics, meet the actual needs of users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and blood vessel density measurement, and particularly relates to a kind of nail fold capillary density automatic measurement method and system. BACKGROUND

[0002] In the detection of nail fold microcirculation, nail fold capillary density is one of the important physiological parameters. Nail fold capillary density can be defined by the number of distal capillaries per millimeter span in each finger or toe. In the study of some diseases, such as connective tissue disease, chronic kidney disease, pulmonary hypertension of patients with scleroderma or diabetes, nail fold capillary density is a key quantitative parameter.

[0003] Currently, the measurement of nail fold capillary density is generally manually measured by doctors, but clinical measurement of nail fold capillary density is easily affected by subjective factors, causing ambiguity and affecting the doctor's judgment and treatment. For example, direct observation method (DO) or some semi-automatic measurement methods (such as manually marking blood vessels on the computer display nail fold screen and screening distal capillaries) are used to measure capillary density to assist doctors in carrying out work based on the obtained nail fold capillary density. However, the above methods not only consume a lot of time, but also the final results are easily affected by subjective judgment, which is not conducive to assisting doctors in carrying out work.

[0004] Therefore, there is an urgent need for an automatic nail fold capillary density measurement method and system with high measurement efficiency and accuracy to solve the above technical problems. SUMMARY

[0005] The technical problem to be solved by the present application is to provide an automatic nail fold capillary density measurement method and system, which can automatically measure nail fold capillary density, has high processing efficiency, short processing time, high measurement result accuracy and other characteristics, and meets the actual needs of users.

[0006] To solve the above technical problems, the present application provides an automatic nail fold capillary density measurement method, comprising: acquiring a nail fold image collected by a microscope camera; identifying and detecting the input nail fold image by a pre-trained detection neural network model to obtain a detection bounding box of a candidate blood vessel in the nail fold image; screening and processing the detection bounding box of the candidate blood vessel by a blood vessel screening model to obtain a target capillary; and calculating the capillary density of the nail fold image according to the number of target capillaries and the size information of the nail fold image.

[0007] As an improvement of the above scheme, the detection neural network model is built based on an improved YOLOv5 network structure, and the training step of the detection neural network model comprises: performing data enhancement processing on the input nail fold training image by adopting a mosaic-9 data enhancement algorithm to obtain a training set image; performing feature extraction of different levels on the training set image by a backbone network to obtain feature maps of different levels, wherein the feature map input by the bottom convolutional layer is optimized and processed by an SPPCSPC module; fusing the feature maps of different levels by a neck network of a feature pyramid structure to generate a feature map of multi-scale information, wherein the size of the feature map of different levels is enlarged by bilinear interpolation for upsampling; performing prediction processing on the feature map of multi-scale information by a Prediction structure to obtain a prediction box; and calculating a loss value of an objective function by adopting a loss function EIOU, propagating the loss value in reverse, adjusting the position and size of the prediction box, to obtain an optimal prediction box as the detection bounding box.

[0008] As an improvement of the above scheme, the step of screening the detection bounding box of the candidate blood vessel by the blood vessel screening model to obtain the target capillary comprises: performing 90° method screening processing on the detection bounding box of each candidate blood vessel to obtain the target capillary; and counting the number of the target capillaries in the nail fold image.

[0009] As an improvement of the above scheme, the step of performing 90° method screening processing on the detection bounding box of each candidate blood vessel to obtain the target capillary comprises: obtaining the top midpoint coordinates of the detection bounding box of each candidate blood vessel as the detection point of the distal end capillary; performing Pythagorean theorem calculation processing on the detection point of each detection bounding box and the detection points of the left and right two detection bounding boxes adjacent thereto by a Pythagorean theorem method to obtain an angle detection result; when the angle detection result is greater than or equal to 90°, the candidate blood vessel located in the middle is a distal end capillary and is taken as the target capillary, otherwise the candidate blood vessel located in the middle is not a distal end capillary.

[0010] As an improvement of the above scheme, the step of calculating the capillary density of the nail fold image according to the number of the target capillaries and the size information of the nail fold image comprises: determining the arrangement length of the nail fold image according to the arrangement direction of the target capillaries and the size information of the nail fold image; and calculating the capillary density of the nail fold image according to the formula p=S / D; wherein p is the capillary density, S is the number of the target capillaries, and D is the arrangement length of the nail fold image.

[0011] The application further provides an automatic measurement system for nail fold capillary density, comprising: an acquisition module configured to acquire a nail fold image collected by a microscope camera; an image processing module configured to identify and detect the input nail fold image by using a pre-trained detection neural network model to obtain a detection bounding box of a candidate blood vessel in the nail fold image; a screening module configured to screen and process the detection bounding box of the candidate blood vessel by using a blood vessel screening model to obtain a target capillary; and a density calculation module configured to calculate the capillary density of the nail fold image according to the number of the target capillary and the size information of the nail fold image.

[0012] As an improvement of the above scheme, the image processing module comprises: an input processing unit configured to perform data enhancement processing on the input nail fold training image by using a mosaic-9 data enhancement algorithm to obtain a training set image; a feature extraction unit configured to perform feature extraction on the training set image at different levels by using a backbone network to obtain feature maps at different levels, wherein the feature map input by the bottommost convolution layer is subjected to optimized feature processing by an SPPCSPC module; a feature fusion unit configured to fuse the feature maps at different levels by using a neck network of a feature pyramid structure to generate a feature map with multi-scale information, wherein the size of the feature map at different levels is enlarged by using a bilinear interpolation method for up-sampling; a prediction processing unit configured to perform prediction processing on the feature map with multi-scale information by using a Prediction structure to obtain a prediction box; and a regression optimization processing unit configured to calculate a loss value of an objective function by using a loss function EIOU, propagate the loss value in a reverse direction, adjust the position and size of the prediction box, and obtain an optimal prediction box as the detection bounding box.

[0013] As an improvement of the above scheme, the screening module comprises: a screening unit configured to perform 90° screening processing on the detection bounding box of each candidate blood vessel to obtain a target capillary; and a statistical unit configured to count the number of the target capillary in the nail fold image.

[0014] As an improvement of the above scheme, the screening unit comprises: an acquisition subunit configured to acquire a top end midpoint coordinate of the detection bounding box of each candidate blood vessel as a detection point of a far-end capillary; an angle processing subunit configured to perform Pythagorean theorem calculation processing on the detection point of each detection bounding box and the detection points of the left and right two detection bounding boxes adjacent to the detection point by using a Pythagorean theorem method to obtain an angle detection result; and a screening processing subunit configured to determine that the candidate blood vessel located in the middle is a far-end capillary and the candidate blood vessel located in the middle is the target capillary when the angle detection result is greater than or equal to 90°, or determine that the candidate blood vessel located in the middle is not a far-end capillary.

[0015] As an improvement of the above scheme, the density calculation module comprises: an arrangement acquisition unit, configured to determine an arrangement length of the nail fold image according to an arrangement direction of the target capillary and size information of the nail fold image; and a density calculation unit, configured to calculate the capillary density of the nail fold image according to a formula: ρ = S / D, wherein ρ is the capillary density, S is the number of the target capillary, and D is the arrangement length of the nail fold image.

[0016] The present application has the following beneficial effects:

[0017] The nail fold capillary density automatic measurement method and system can quickly and accurately identify all candidate blood vessels in the input nail fold image, automatically calculate the capillary density of the nail fold image through screening processing of all candidate blood vessels, and has the characteristics of high automatic measurement processing efficiency, short processing time, high measurement result accuracy, and the like, thereby meeting the actual needs of users. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart of the nail fold capillary density automatic measurement method of the present application;

[0019] Figure 2 is a training flowchart of the detection neural network model of the present application;

[0020] Figure 3 is a flowchart of the 90° method screening processing of the detection bounding box of each candidate blood vessel to obtain the target capillary;

[0021] Figure 4 is a structural schematic diagram of the nail fold capillary density automatic measurement system of the present application;

[0022] Figure 5 is a structural schematic diagram of the image processing module of the present application;

[0023] Figure 6 is a structural schematic diagram of the screening unit of the present application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings. It is hereby declared that the up, down, left, right, front, back, inner and outer directions appearing or about to appear in the present application are based on the drawings of the present application, and are not specific limitations of the present application.

[0025] As shown in Figure 1 , the embodiment of the present application provides a nail fold capillary density automatic measurement method, comprising:

[0026] S1, acquire the nail fold image collected by the microscope camera;

[0027] It should be noted that the collected nail fold image includes nail fold images in different situations, such as normal nail fold images collected, nail fold images collected in a glare collection environment, nail fold images collected with many small blood vessels, and nail fold images collected in an overexposure collection environment, etc. It is not limited to the above four situations.

[0028] S2, identifying and detecting the input nail fold image through the pre-trained detection neural network model to obtain the detection bounding box of the candidate blood vessels in the nail fold image;

[0029] It should be noted that the detection neural network model is pre-trained and iteratively processed for nail fold images in different situations to obtain the optimal detection neural network model. Through the trained detection neural network model, different situations of nail fold images can be quickly and accurately identified and detected, and all candidate blood vessels of the corresponding nail fold image can be obtained. Detection bounding box, improve the automatic identification and detection capability and efficiency.

[0030] The candidate blood vessels are labeled by the detection bounding box, and the edges of the candidate blood vessels are surrounded by the coordinate box so that the obstacles are completely or mostly in the coordinate box. The detection bounding box is set one by one corresponding to the candidate blood vessels. Among them, the detection bounding box is preferably a rectangular coordinate box, but is not limited thereto.

[0031] S3, screening the detection bounding box of the candidate blood vessels through the blood vessel screening model to obtain the target capillary blood vessel;

[0032] Specifically, the step of screening the detection bounding box of the candidate blood vessels through the blood vessel screening model to obtain the target capillary blood vessel includes:

[0033] Step one, 90° method screening processing is performed on the detection bounding box of each candidate blood vessel to obtain the target capillary blood vessel;

[0034] Step two, the number of target capillary blood vessels in the nail fold image is counted.

[0035] It should be noted that each detection bounding box of the candidate blood vessel is automatically screened by 90° method according to the coordinate information of the detection bounding box, so that the target capillary blood vessel is accurately screened from all candidate blood vessels, and the detection accuracy of the target capillary blood vessel and the accuracy of the screening result are improved. By counting the number of all target capillary blood vessels in the nail fold image, subsequent nail fold capillary density calculation processing is facilitated, and the calculation accuracy of the subsequent nail fold capillary density is ensured.

[0036] S4, calculating the capillary density of the nail fold image according to the number of the target capillary and the size information of the nail fold image.

[0037] Specifically, the step of calculating the capillary density of the nail fold image according to the number of the target capillary and the size information of the nail fold image comprises:

[0038] Step (one), determining the arrangement length of the nail fold image according to the arrangement direction of the target capillary and the size information of the nail fold image;

[0039] It should be noted that the arrangement direction of the target capillary on the nail fold image is determined according to the arrangement direction of the target capillary, and the arrangement length corresponding to the arrangement direction of the nail fold image is determined according to the arrangement direction and the size information of the nail fold image. The size information of the nail fold image is coordinate size information, which can quickly calculate the arrangement length of the nail fold image for subsequent calculation.

[0040] Step (two), calculating the capillary density of the nail fold image according to the formula p = S / D; wherein p is the capillary density, S is the number of the target capillary, and D is the arrangement length of the nail fold image.

[0041] It should be noted that the capillary density of the nail fold can be calculated by the above formula and expressed as the number of capillaries per millimeter in the nail fold. By automatically calculating the capillary density of the nail fold image, it has the characteristics of high automatic measurement processing efficiency, short processing time, high measurement result accuracy, etc., which meets the actual needs of users.

[0042] Preferably, the detection neural network model is built based on an improved YOLOv5 network structure. Since the detection performance of the original YOLOv5 network algorithm cannot meet the required detection recognition accuracy, the accuracy of the final recognition image processing result needs to be improved, which will greatly affect the accuracy of the subsequent calculation processing result. Therefore, the detection neural network model of the present application is built based on an improved YOLOv5 network structure to improve the detection performance and accuracy of the detection neural network model.

[0043] As shown in Figure 2 the training steps of the detection neural network model comprise:

[0044] S10, performing data enhancement processing on the input nail fold training image by using a mosaic-9 data enhancement algorithm to obtain a training set image;

[0045] It should be noted that when inputting the layer, the mosaic-9 data enhancement algorithm is used to randomly crop and scale the input 9 nail fold training images and splice them together to form a training set image and input the backbone network. By using the above method, the diversity and quality of the training set data can be improved and increased, the target sample can be increased, the network training speed can be improved, the generalization ability of the model can be strengthened, and in the normalization process, 9 images are calculated at a time, without the need for a large batch value, thereby reducing the memory requirements of the model.

[0046] S20, extracting features of different levels of the training set image through the backbone network to obtain feature maps of different levels, wherein the feature map input by the bottom convolutional layer is optimized and improved by the SPPCSPC module;

[0047] It should be noted that while the feature extraction processing is performed by the backbone network of YOLOv5, the SPPCSPC module is used to process the feature map input by the bottom convolutional layer. The SPPCSPC module includes a CSP structure and an SPP structure. The SPP module is used to perform max-pooling operations on the input feature map by multiple convolution kernels of different sizes. By using different max-pooling operations, the receptive field can be increased, small and large targets can be better distinguished, and the concatenated operation is performed on the multiple feature maps after pooling to obtain an output of adaptive size, avoiding the distortion problem after the picture resize operation and solving the problem of repeated extraction of related features by the convolutional neural network, greatly improving the speed of generating candidate boxes and saving the calculation cost. Through the CSP structure, on the one hand, the feature map is processed by the regular CBS module, and on the other hand, the auxiliary feature map is processed by the SPP structure. Finally, the processing results of the two parts are combined and output to the neck network. The CSP structure can improve the processing speed and accuracy. Therefore, by using the SPPCSPC module, the object feature extraction ability and algorithm detection performance of the model can be improved, and the image recognition detection accuracy can be ensured.

[0048] S30, fusing the feature maps of different levels through the neck network of the feature pyramid structure to generate a feature map with multi-scale information, wherein the size of the feature map of different levels is enlarged by bilinear interpolation method through upsampling;

[0049] It should be noted that the neck network adopts the structure of the feature pyramid (Feature Pyramid Network), which fuses the feature maps of different levels in the backbone network through upsampling and downsampling operations, realizes the fusion of features of different levels, generates a feature map with multi-scale information, and inputs it into the prediction layer.

[0050] The upsampling method adopts a bilinear interpolation method, which can calculate a weighted average value of attributes (color, transparency, etc.) of four surrounding texture pixels and apply it to the screen pixel. Compared with the existing nearest neighbor interpolation, the gray value of the transformed pixel is equal to the gray value of the input pixel closest to it. Therefore, the bilinear interpolation method used in upsampling can maintain the accuracy of the picture pixels as much as possible, can effectively avoid data distortion problems, and greatly improves the accuracy of image data recognition and detection processing.

[0051] S40, performing prediction processing on the feature maps of the multi-scale information through a Prediction structure to obtain a prediction box;

[0052] S50, calculating a loss value of a target function by using a loss function EIOU, propagating the loss value in reverse, adjusting the position and size of the prediction box, to obtain an optimal prediction box as the detection bounding box.

[0053] It should be noted that in the prediction layer, the feature maps of the multi-scale information are processed through the Prediction structure to obtain a prediction box. The working principle of the Prediction structure is a prior art and will not be described here. By using EIOU (Efficient Intersection Over Union) as a positioning loss function, the difference between the prediction box and the real box, i.e., the loss value, is calculated. The calculation formula of EIOU is:

[0054]

[0055] where Intersection Over Union (IOU) represents the intersection over union of the real box (hand-labeled) and the prediction box, Distance_2 represents the Euclidean distance between the centers of the two boxes, Distance_C represents the diagonal distance of the smallest enclosing rectangle of the two boxes, Distance_2 h and Distance_2 w respectively represent the Euclidean distances of the heights and widths of the two boxes, Distance_C h and Distance_C w respectively represent the heights and widths of the smallest enclosing rectangle of the two boxes.

[0056] EIOU separates the aspect ratio factor on the basis of CIOU, respectively calculates the length and width of the prediction box and the real box, so that the model converges faster and enhances the ability of back propagation, i.e., enhances the ability of back propagation in the convolutional neural network, constantly updates the weights of YOLOv5, adjusts the position and size of the prediction box, to improve the accuracy of the prediction box, so as to obtain an optimal prediction box as the detection bounding box, and complete the training of the detection neural network model.

[0057] Based on the above, the detection neural network model adopts the improved YOLOv5 network structure for training and detection of the nail fold image, which can improve the image detection performance, ensure high accuracy and good stability of image recognition detection, and accurately mark all candidate blood vessels in the nail fold image through the detection bounding box, so as to obtain high-precision capillary density calculation results through further accurate screening processing of all candidate blood vessels.

[0058] As shown in Figure 3 The step of performing 90° method screening processing on the detection bounding box of each candidate blood vessel to obtain the target capillary includes:

[0059] S100, obtaining the top midpoint coordinates of the detection bounding box of each candidate blood vessel as the detection point of the far-end capillary;

[0060] S200, performing Pythagorean theorem calculation processing on the detection point of each detection bounding box and the detection points of the left and right two detection bounding boxes adjacent thereto to obtain an angle detection result;

[0061] It should be noted that the top midpoint coordinates of the detection bounding box can be quickly obtained as the detection point of the far-end capillary according to the coordinate information of the detection bounding box, and the angle detection result is obtained by performing Pythagorean theorem calculation processing on the detection point of any candidate blood vessel and the detection points of the left and right two detection bounding boxes adjacent thereto. For example, three candidate blood vessels A, B and C are sequentially sorted, and their three detection points are defined as a detection point, b detection point and c detection point. The angle detection result of angle b can be obtained by performing Pythagorean theorem calculation on the coordinate information of the a detection point, b detection point and c detection point, so as to facilitate subsequent automatic screening processing through the angle detection result of angle b.

[0062] S300, when the angle detection result is greater than or equal to 90°, the candidate blood vessel located in the middle is a far-end capillary and is taken as the target capillary, otherwise the candidate blood vessel located in the middle is not a far-end capillary.

[0063] It should be noted that when the angle detection result of angle b is greater than or equal to 90°, the candidate blood vessel located in the middle is a far-end capillary and is taken as the target capillary, otherwise the candidate blood vessel located in the middle is not a far-end capillary. The target capillary can be quickly obtained by sequentially performing the above-mentioned automatic screening processing of the 90° method on each candidate blood vessel, and the nail fold capillary density of the input nail fold image can be automatically and quickly calculated by counting the number of target capillaries and combining the arrangement length of the nail fold image.

[0064] Preferably, as shown in Table 1, Table 1 is a test data table of different measurement methods for nail fold images under different conditions:

[0065] Table 1

[0066]

[0067] As can be known from the measurement test data table in the table, the nail fold capillary density under different conditions is calculated by the detection neural network model and the blood vessel screening model built based on the improved YOLOv5 network structure, and the density data is consistent or very close to the actual density data measured by manual measurement. At the same time, compared with directly using the existing YOLOv5 network structure for nail fold image recognition processing, the improved YOLOv5 network structure has higher image recognition detection capability and recognition accuracy, so that the precision of the nail fold capillary obtained by subsequent screening and density calculation processing is also high, and is closer to or equal to the actual measurement density result. Therefore, by using the nail fold capillary density automatic measurement method of the present application, all candidate blood vessels in the input nail fold image can be quickly and accurately recognized and detected, and the nail fold capillary density is automatically calculated by screening all candidate blood vessels. It has the characteristics of high automatic measurement processing efficiency, short processing time, high measurement result accuracy, and meets the actual needs of users.

[0068] As Figure 4 shown, the present application also provides a nail fold capillary density automatic measurement system, comprising an acquisition module 1 for acquiring nail fold images collected by a microscope camera;

[0069] It should be noted that the collected nail fold images include nail fold images under different conditions, such as normal nail fold images, nail fold images collected in a glare collection environment, nail fold images collected with many small blood vessels, and nail fold images collected in an overexposure collection environment, etc., which are not limited to the above four conditions.

[0070] An image processing module 2 is used to recognize and detect the input nail fold image by a pre-trained detection neural network model to obtain the detection bounding box of the candidate blood vessels in the nail fold image.

[0071] It should be noted that the detection neural network model is pre-trained and iteratively processed for nail fold images under different conditions to obtain an optimal detection neural network model. The trained detection neural network model can quickly and accurately recognize and detect nail fold images under different conditions, and obtain the detection bounding box of all candidate blood vessels of the corresponding nail fold image, thereby improving the automatic recognition and detection capability and efficiency.

[0072] The detection bounding box is set in one-to-one correspondence with the candidate blood vessel by marking the candidate blood vessel by detecting the bounding box, and the edge of the candidate blood vessel is surrounded by the coordinate box so that the obstacles are completely or mostly in the coordinate box.

[0073] The screening module 3 is configured to screen the detection bounding box of the candidate blood vessel by using a blood vessel screening model to obtain a target capillary blood vessel.

[0074] Specifically, the screening module 3 includes a screening unit 31 configured to perform 90° method screening processing on the detection bounding box of each candidate blood vessel to obtain a target capillary blood vessel, and a statistical unit 32 configured to count the number of the target capillary blood vessels in the nail fold image.

[0075] It should be noted that the 90° method screening processing is performed on each candidate blood vessel according to the coordinate information of the detection bounding box of the candidate blood vessel to accurately screen the target capillary blood vessel from all candidate blood vessels, thereby improving the detection accuracy of the target capillary blood vessel and the accuracy of the screening result. The number of all target capillary blood vessels in the nail fold image is counted to facilitate subsequent nail fold capillary density calculation processing and to ensure the calculation accuracy of the nail fold capillary density.

[0076] The density calculation module 4 is configured to calculate the capillary density of the nail fold image according to the number of the target capillary blood vessels and the size information of the nail fold image.

[0077] Specifically, the density calculation module 4 includes an arrangement obtaining unit 41 configured to determine the arrangement length of the nail fold image according to the arrangement direction of the target capillary blood vessel and the size information of the nail fold image.

[0078] It should be noted that the arrangement direction of the target capillary blood vessel on the nail fold image is determined according to the arrangement direction of the target capillary blood vessel, and the arrangement length of the nail fold image corresponding to the arrangement direction is determined according to the arrangement direction and the size information of the nail fold image. The size information of the nail fold image is coordinate size information, and the arrangement length of the nail fold image can be quickly calculated by using the coordinate size information for subsequent calculation.

[0079] The density calculation unit 42 is configured to calculate the capillary density of the nail fold image according to the formula ρ=S / D, where ρ is the capillary density, S is the number of the target capillary blood vessels, and D is the arrangement length of the nail fold image.

[0080] It should be noted that the above formula can be used to calculate the capillary density of the nail fold, and is expressed as the number of capillaries per millimeter in the nail fold unit. By automatically calculating the capillary density of the nail fold image, the automatic measurement has the characteristics of high processing efficiency, short processing time, high accuracy of measurement results, and meets the actual needs of users.

[0081] As shown in Figure 5 The image processing module 2 includes:

[0082] The input processing unit 21 is configured to perform data enhancement processing on the input nail fold training image by using the mosaic-9 data enhancement algorithm to obtain a training set image.

[0083] It should be noted that when inputting, the mosaic-9 data enhancement algorithm is used to randomly crop and scale the input 9 nail fold training images and splice them together to form a training set image and input the backbone network. By using the above method, the diversity and quality of the training set data can be improved and increased, the target sample can be increased, the network training speed can be improved, the generalization ability of the model can be strengthened, and in the normalization process, 9 images are calculated at a time, without the need for a large batch value, thereby reducing the memory requirements of the model.

[0084] The feature extraction unit 22 is configured to perform feature extraction of different levels on the training set image by the backbone network to obtain feature maps of different levels, wherein the SPPCSPC module is used to optimize the feature processing of the feature map input by the bottom convolutional layer.

[0085] It should be noted that while the feature extraction processing is performed by the backbone network of YOLOv5, the SPPCSPC module is used to process the feature map input by the bottom convolutional layer. The SPPCSPC module includes a CSP structure and an SPP structure. The SPP module is configured to perform max-pooling operations on the input feature map by using multiple convolution kernels of different sizes. By using different max-pooling operations, the receptive field can be increased, small targets and large targets can be better distinguished, and the concatenated operation is performed on the multiple feature maps after pooling to obtain an output of adaptive size, thereby avoiding the distortion problem after the picture resize operation and solving the problem of repeated extraction of related features by the convolutional neural network, greatly improving the speed of generating candidate boxes and saving the calculation cost. Through the CSP structure, on the one hand, the feature map is processed by the conventional CBS module, and on the other hand, the auxiliary feature map is processed by the SPP structure. Finally, the processing results of the two parts are combined together and output to the neck network. The CSP structure can improve the processing speed and accuracy. Therefore, the use of the SPPCSPC module can improve the object feature extraction capability and algorithm detection performance of the model.

[0086] The feature fusion unit 23 is configured to fuse the feature maps of different levels by a neck network of a feature pyramid structure to generate a feature map of multi-scale information, wherein the upsampling is configured to perform size expansion processing on the feature maps of different levels by bilinear interpolation.

[0087] It should be noted that the neck network adopts a feature pyramid (Feature Pyramid Network) structure, and the feature maps of different levels in the backbone network are fused together by upsampling and downsampling operations to realize the fusion of features of different levels, so as to generate a feature map of multi-scale information and input into the prediction layer.

[0088] The upsampling method is configured to perform size expansion processing on the feature maps of different levels by bilinear interpolation, which can calculate the weighted average value of the properties (color, transparency, etc.) of four surrounding texture pixels and apply it to the screen pixel. Compared with the existing nearest neighbor interpolation, which makes the gray value of the transformed pixel equal to the gray value of the nearest input pixel. Therefore, the bilinear interpolation method used in upsampling can make the feature map size larger while maintaining the accuracy of the picture pixels as much as possible, which can effectively avoid data distortion problems and greatly improve the accuracy of image data recognition and detection processing.

[0089] The prediction processing unit 24 is configured to perform prediction processing on the feature map of multi-scale information by the Prediction structure to obtain a prediction box; and the regression optimization processing unit 25 is configured to calculate the loss value of the objective function by using the loss function EIOU, propagate the loss value in reverse, adjust the position and size of the prediction box, so as to obtain the optimal prediction box as the detection bounding box.

[0090] It should be noted that in the prediction layer, the feature map of multi-scale information is processed by the Prediction structure to obtain a prediction box, and the working principle of the Prediction structure is the prior art which will not be described here. By using EIOU (Efficient Intersection Over Union) as the positioning loss function, the difference between the prediction box and the real box, i.e. the loss value, is calculated. The calculation formula of EIOU is:

[0091]

[0092] Wherein, Intersection Over Union (IOU) represents the intersection over union of the real box (hand-labeled) and the prediction box, Distance_2 represents the Euclidean distance between the centers of the two boxes, Distance_C represents the diagonal distance of the smallest circumscribed rectangle of the two boxes, and Distance_2 h and Distance_2 wEuclidean distance of height and width of two boxes respectively, Distance_C h and Distance_C w Euclidean distance of height and width of minimum bounding rectangle of two boxes respectively.

[0093] EIOU separates the aspect ratio factor on the basis of CIOU, respectively calculates the length and width of the predicted box and the real box, so that the model converges faster, enhances the ability of back propagation, that is, enhances the ability of back propagation in the convolutional neural network, constantly updates the weight of YOLOv5, adjusts the position and size of the predicted box, to improve the accuracy of the predicted box, so as to obtain the optimal predicted box as the detection bounding box, and complete the training of the detection neural network model.

[0094] Based on the above, the detection neural network model uses the improved YOLOv5 network structure to train and detect the nail fold image, which can improve the image detection performance, ensure the high accuracy and good stability of image recognition detection, and accurately mark all candidate blood vessels in the nail fold image through the detection bounding box, so as to further accurately screen all candidate blood vessels for subsequent processing, so as to obtain a high-precision capillary density calculation result.

[0095] As shown in Figure 6 , the screening unit 31 comprises:

[0096] The acquisition subunit 311 is configured to acquire the top midpoint coordinates of the detection bounding box of each candidate blood vessel as the detection point of the distal end capillary; and the angle processing subunit 312 is configured to perform Pythagorean theorem calculation processing on the detection point of each detection bounding box and the detection points of the left and right two detection bounding boxes adjacent thereto to obtain an angle detection result.

[0097] It should be noted that the top midpoint coordinates of the detection bounding box can be quickly obtained as the detection point of the distal end capillary according to the coordinate information of the detection bounding box, and the Pythagorean theorem calculation processing is performed on the detection point of any candidate blood vessel and the detection points of the left and right two detection bounding boxes adjacent thereto to obtain an angle detection result. For example, three candidate blood vessels A, B and C are sequentially sorted, and their three detection points are defined as a detection point, b detection point and c detection point. The coordinate information of the a detection point, b detection point and c detection point can be used to calculate the angle detection result of angle b by the Pythagorean theorem, so as to facilitate subsequent automatic screening processing by the angle detection result of angle b.

[0098] The screening processing subunit 313 determines that the candidate blood vessel located in the middle is the distal end capillary as the target capillary when the angle detection result is greater than or equal to 90°, otherwise the candidate blood vessel located in the middle is not the distal end capillary.

[0099] It should be noted that when the angle detection result of the angle b is greater than or equal to 90°, the candidate blood vessel located in the middle is a far end capillary and is the target capillary, otherwise the candidate blood vessel located in the middle is not a far end capillary. The target capillary can be quickly obtained by sequentially performing the automatic screening process of the 90° method on each candidate blood vessel. The number of target capillaries is counted, and the arrangement length of the nail fold image is combined to automatically and quickly calculate the nail fold capillary density of the input nail fold image.

[0100] In summary, the present application can quickly and accurately identify and detect all candidate blood vessels in the input nail fold image. The nail fold capillary density of the nail fold image is automatically calculated by screening all candidate blood vessels. The present application has the characteristics of high automatic measurement processing efficiency, short processing time, high measurement result accuracy and the like, and meets the actual needs of users.

[0101] The above only discloses preferred embodiments of the present application, and of course cannot limit the scope of the rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope of the present application.

Claims

1. A method of automatically measuring capillary density of a nail fold, characterized by, The method comprises the following steps: obtaining an image of a nail fold collected by a microscope camera; recognizing and detecting the input nail fold image by a pre-trained detection neural network model to obtain a detection bounding box of a candidate blood vessel in the nail fold image; screening the detection bounding box of the candidate blood vessel by a blood vessel screening model to obtain a target capillary blood vessel; calculating a capillary blood vessel density of the nail fold image according to the number of the target capillary blood vessel and size information of the nail fold image; wherein the step of screening the detection bounding box of the candidate blood vessel by the blood vessel screening model to obtain the target capillary blood vessel comprises: performing 90° method screening processing on each detection bounding box of the candidate blood vessel to obtain a target capillary blood vessel; counting the number of the target capillary blood vessel in the nail fold image; the step of performing 90° method screening processing on each detection bounding box of the candidate blood vessel to obtain a target capillary blood vessel comprises: obtaining a top midpoint coordinate of each detection bounding box of the candidate blood vessel as a detection point of a far-end capillary blood vessel; performing Pythagorean theorem calculation processing on the detection point of each detection bounding box and the detection points of the left and right two detection bounding boxes adjacent thereto by a Pythagorean theorem method to obtain an angle detection result; when the angle detection result is greater than or equal to 90°, the candidate blood vessel located in the middle is a far-end capillary blood vessel and is taken as the target capillary blood vessel, otherwise the candidate blood vessel located in the middle is not a far-end capillary blood vessel.

2. The method of claim 1, wherein the method comprises: The detection neural network model is built based on an improved YOLOv5 network structure, and the training steps of the detection neural network model comprise: performing data enhancement processing on the input nail fold training image by a mosaic-9 data enhancement algorithm to obtain a training set image; performing feature extraction of different levels on the training set image by a backbone network to obtain feature maps of different levels, wherein the feature map input by the bottom convolutional layer is optimized and processed by an SPPCSPC module; fusing the feature maps of different levels by a neck network of a feature pyramid structure to generate a feature map with multi-scale information, wherein the size of the feature map of different levels is enlarged by bilinear interpolation for upsampling; performing prediction processing on the feature map with multi-scale information by a Prediction structure to obtain a prediction box; calculating a loss value of an objective function by a loss function EIOU, propagating the loss value in reverse, adjusting the position and size of the prediction box, to obtain an optimal prediction box as the detection bounding box.

3. The method of claim 1, wherein the method comprises: The step of calculating the capillary blood vessel density of the nail fold image according to the number of the target capillary blood vessel and the size information of the nail fold image comprises: determining an arrangement length of the nail fold image according to the arrangement direction of the target capillary blood vessel and the size information of the nail fold image; calculating the capillary blood vessel density of the nail fold image according to the formula ρ=S / D; wherein ρ is the capillary blood vessel density, S is the number of the target capillary blood vessel, and D is the arrangement length of the nail fold image.

4. An automated measurement system for nail capillary density, characterized by, The method comprises the following steps: an obtaining module, configured to obtain an image of a nail fold collected by a microscope camera; An image processing module is configured to recognize and detect the input nail fold image by using a pre-trained detection neural network model to obtain a detection bounding box of a candidate blood vessel in the nail fold image; A screening module is configured to screen the detection bounding box of the candidate blood vessel by using a blood vessel screening model to obtain a target capillary blood vessel; A density calculation module is configured to calculate a capillary blood vessel density of the nail fold image according to a quantity of the target capillary blood vessels and size information of the nail fold image; The screening module includes: A screening unit is configured to perform 90° method screening processing on the detection bounding box of each candidate blood vessel to obtain a target capillary blood vessel; A statistical unit is configured to count a quantity of the target capillary blood vessels in the nail fold image; The screening unit includes: An acquisition subunit is configured to acquire a top end midpoint coordinate of the detection bounding box of each candidate blood vessel as a detection point of a far-end capillary blood vessel; An angle processing subunit is configured to perform Pythagorean theorem calculation processing on the detection point of each detection bounding box and detection points of two detection bounding boxes adjacent to the detection point in a left and right manner by using a Pythagorean theorem to obtain an angle detection result; A screening processing subunit is configured to determine that the candidate blood vessel located in the middle is a far-end capillary blood vessel and is the target capillary blood vessel when the angle detection result is greater than or equal to 90°, and otherwise, determine that the candidate blood vessel located in the middle is not a far-end capillary blood vessel.

5. The nail capillary density automatic measuring system according to claim 4, characterized in that, The image processing module includes: An input processing unit is configured to perform data enhancement processing on the input nail fold training image by using a mosaic-9 data enhancement algorithm to obtain a training set image; A feature extraction unit is configured to perform feature extraction of different levels on the training set image by using a backbone network to obtain feature maps of different levels, wherein an SPPCSPC module is used to perform optimized feature processing on the feature map input by the bottom convolutional layer; A feature fusion unit is configured to fuse the feature maps of different levels by using a neck network of a feature pyramid structure to generate a feature map of multi-scale information, wherein a bilinear interpolation method is used to perform size expansion processing on the feature maps of different levels; A prediction processing unit is configured to perform prediction processing on the feature map of multi-scale information by using a Prediction structure to obtain a prediction box; A regression optimization processing unit is configured to calculate a loss value of an objective function by using a loss function EIOU, propagate the loss value in a reverse direction, adjust a position and a size of the prediction box, and obtain an optimal prediction box as the detection bounding box.

6. The nail capillary density automatic measuring system according to claim 4, wherein The density calculation module includes: An arrangement acquisition unit is configured to determine an arrangement length of the nail fold image according to an arrangement direction of the target capillary blood vessels and size information of the nail fold image; A density calculation unit is configured to calculate a capillary blood vessel density of the nail fold image according to a formula ρ=S / D; wherein ρ is the capillary blood vessel density, S is the quantity of the target capillary blood vessels, and D is the arrangement length of the nail fold image.

Images