Finger capillary refill time measurement method and device based on deep learning

Abstract

The application discloses a finger capillary refill time measurement method and device based on deep learning, through standardization of the measurement process, using a pure vision scheme of deep learning, without other sensors, using a camera to collect finger images and artificial intelligence to automatically collect accurate parameters of specific parts of the finger, through establishment of a deep learning positioning key finger area algorithm, using a method of judging the start and end points of CRT measurement by using color saturation of the finger area, avoiding influences caused by background differences, individual differences and sensor noise, being capable of monitoring peripheral perfusion of human skin in real time in various environments, being capable of avoiding human errors, being capable of measuring more quickly and accurately, and being capable of being widely applied to measurement of finger capillary refill time.

Description

TECHNICAL FIELD

[0001] The present application relates to a neural network-based image processing technology, in particular to a method and device for measuring capillary refill time of a finger image based on a deep neural network model. BACKGROUND

[0002] Currently, the effect of early resuscitation of septic shock is mainly evaluated according to the blood lactate value, and an effective resuscitation strategy is formulated according to the change of the blood lactate value. Exploring peripheral perfusion through skin analysis is a semi-quantitative method for studying tissue perfusion. Capillary refill time (CRT) measurement is an easy-to-use method for evaluating peripheral perfusion. CRT has been proven to be responsive to resuscitation, and its evaluation results can be used to assist in judging and adjusting the treatment plan.

[0003] In the current work, the capillary refill time CRT is mostly measured by manual evaluation method. The manual evaluation method of CRT has the disadvantages of deviation of environmental light conditions, non-standardization of bleaching time and manual pressure, subjectivity of restoring normal color and variability of manual time evaluation. Moreover, the existing capillary refill time CRT measurement technical solutions mostly use traditional algorithms, and manually adjust parameters for parameter collection and processing of the user's fingers, so that the device noise and human error are large, and the measurement efficiency is low. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the present application provides a finger capillary refill time measurement method and device based on deep learning. The present application uses a pure vision scheme (pure vision scheme only needs image acquisition, without other sensors) through the standardization of the measurement process, develops a portable device for measuring CRT, uses a camera to collect finger images and automatically collects accurate parameters of specific parts of the finger by artificial intelligence, establishes a deep learning positioning key finger area algorithm, uses the color saturation of the finger area to determine the starting and ending points of the CRT measurement method, avoids the influence caused by background difference, individual difference and sensor noise, and can monitor the peripheral perfusion of human skin in real time in various environments. Compared with the traditional manual method and the previous technical solutions, the present application avoids human error and is more rapid and accurate. The present application can be widely applied and has high practical value.

[0005] The technical solutions provided by the present application are as follows:

[0006] A capillary refill time measurement method based on finger image and deep learning model, which automatically collects image sequences of specific parts of fingers by artificial intelligence, directly obtains CRT results by analyzing the collected image sequences, and measures the starting point and ending point of measurement by numerical calculation of the hue, saturation and value (HSV) color space of the finger image, so as to measure the capillary refill time. The specific steps are as follows:

[0007] 1) automatically collect image sequences of specific parts of fingers by artificial intelligence;

[0008] 2) image sequence preprocessing: convert the image sequence to the hsv color space to obtain the hsv image of the finger;

[0009] 3) create a deep learning network model based on finger image to generate a key finger area mask mask:

[0010] The hsv image is segmented by deep learning to obtain the key area mask mask, so that the most critical part of the finger area can be accurately located, and the influence of background difference, individual difference and sensor noise can be accurately avoided.

[0011] The patent adopts image mask segmentation neural network (Mask CNN) to generate the key area mask mask. In order to realize efficient forward inference, the image mask segmentation network adopts a simplified EfficientNet as the main part. The input and output of the network are both 64x64 resolution. As shown in Figure 4 The simplified EfficientNet is composed of 3 mobile inverted bottleneck (MBConvk5x5) modules, which can occupy very small calculation amount to realize very high prediction accuracy. The prediction head of the image mask segmentation adopts a standard image segmentation prediction head, which is a 3x3 convolution, to judge whether each input image pixel belongs to the key area.

[0012] The training data set of the key area is collected by professional personnel to collect capillary refill time measurement video data of various personnel, and then each frame of picture of the video is professionally labeled. The collection method is: using a glass piece to press the ventral side of the distal phalanx of the right hand finger, gradually increasing the pressure until the skin turns white, and maintaining for 10 seconds. The time from the end of pressing to the recovery of the skin color is the CRT. A video camera is used to shoot the color change process of the finger from the glass piece end. The professional personnel label the area where the color change of the ventral side of the distal phalanx of the right hand finger is most obvious. The training of the image mask segmentation network adopts Adam optimizer for training, and the number of iterations can be set to 100, that is, 100 epochs of training.

[0013] 4) take the value of s channel (Saturation, indicating the degree of color close to the spectrum) in the image mask area of the hsv image, and calculate the average value s_mean of the s value of the pixels in the image mask area;

[0014] 5) set the smoothing granularity, and smooth s_mean: in the implementation, the average of s_mean is calculated with a granularity of 5 frames, and the smoothing can avoid the influence of abnormal values;

[0015] 6) obtain the measurement starting point: take the minimum value of s in the measurement process as the measurement time starting point st_time;

[0016] 7) obtain the measurement time endpoint:

[0017] Set the granularity standard deviation threshold; then take the value of s when the standard deviation of s in the set granularity range is less than the granularity standard deviation threshold (which can be 10) and multiplied by a calibration coefficient (which can be 0.5-1) as the s threshold, and take the time when s first reaches the s threshold after the measurement starting point st_time as the measurement time endpoint ed_time;

[0018] 8) calculate the CRT: CRT = ed_time-st_time.

[0019] The capillary refill time is ed_time-st_time.

[0020] In the implementation, the capillary refill time measuring device is developed. The capillary refill time measuring device comprises a shell, a screen, a button switch, a camera (or a camera head), a processing circuit, a light source, a pressing device (including a transparent plate, an air pump, an air pipe, an air bag, an exhaust valve, and a buffer material). The screen is used to display the current color and state of the finger; the button is used to operate the start and end of the measurement; the processing circuit comprises a power supply system, an embedded processing terminal, and a control circuit, which provide stable current for the electronic equipment of the whole measuring device, the embedded processing terminal is an arm (Advanced RISC Machine) processor, which is used for data analysis, and the control circuit is used to control the start and stop of the air pump and the electromagnetic air valve. The air pump is connected with the air pipe, the air valve, and the air bag, which are used to provide power for pressing the outer end of the finger, the easy-to-clean buffer material is used to facilitate disinfection and accommodate different sizes of fingers after pressing each person, the transparent plate and the air pump-air bag structure cooperate as the pressing device, the internal light source provides a fixed light source for the camera to collect the image of the finger, and the shell isolates the finger measurement part from the outside world to avoid interference from external light sources.

[0021] The image information collected by the above device is sent to the embedded processing terminal for data analysis, and the analyzed results are finally displayed on the display screen.

[0022] Advantages of the present application:

[0023] The present application provides a capillary refill time measurement method based on a finger image deep learning model. The technical scheme provided by the present application uses a deep learning key finger area positioning algorithm and a method of using finger area color saturation to determine the start and end points of CRT measurement, accurately avoids the influence of background differences, individual differences, and sensor noise on CRT measurement, can efficiently measure CRT, and improves measurement accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A measurement device schematic diagram used in the specific implementation of the present application.

[0025] Figure 2 A measurement device operation flowchart for the specific implementation of the present application.

[0026] Figure 3 A flowchart of the method of the present application.

[0027] Figure 4 A Mask CNN neural network structure diagram of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below by examples in conjunction with the drawings, but the scope of the present application is not limited in any way.

[0029] The present application proposes a capillary refill time measurement method based on deep learning. The image sequence of a specific part of a finger is automatically collected by artificial intelligence, the CRT result is directly obtained by analyzing the collected image sequence, the measurement start point and the measurement end point are calculated by using the numerical value of the finger image hsv color space, and the capillary refill time is measured and obtained.

[0030] The capillary refill time measurement device is as follows: Figure 1The device is shown to include: a housing, a screen, a button switch, a camera (or camera), a processing circuit, a light source, a pressing device (including a transparent plate, an air pump, an air pipe, an air bag, an exhaust valve, a buffer material, etc.). Among them, the screen is used to display the current color and state of the finger; the button is used to operate the start and end of measurement; the processing circuit contains a power supply system, an embedded processing terminal and a control circuit, which provides stable current for the entire measurement device electronic equipment, the embedded processing terminal is an arm (Advanced RISC Machine) processor, which is used for data analysis, and the control circuit is used to control the start and stop of the air pump and the electromagnetic air valve. The air pump is connected with the air pipe, the air valve and the air bag, which is used to provide power for pressing the outer end of the finger, and the easy-to-clean buffer material is used to facilitate disinfection and apply different sizes of fingers after pressing each person. The transparent plate and the air bag structure cooperate as a pressing device, the internal light source provides a fixed light source for the camera to collect finger images, the housing isolates the finger measurement part from the outside world to avoid external light interference.

[0031] The operation flow is shown as Figure 2 The specific operation method is as follows:

[0032] 1. Insert the finger into the capillary refill time measurement device, press the button switch;

[0033] 2. Turn on the light source in the measurement device, the camera starts to shoot the state of the finger, and displays it on the screen. The processing circuit controls the air pump to provide pressing power, the transparent plate presses the distal phalanx of the finger, and gradually increases the pressure until the skin turns white.

[0034] 3. Maintain for 10 seconds, the air pump stops inflating, the air valve opens to end the pressing, and after the skin returns to normal color, the user removes the device from the hand.

[0035] The collected image information is sent to the embedded processing terminal for data analysis, and finally the analyzed results are displayed through the display screen.

[0036] The data analysis process uses a pure visual scheme, which only needs image acquisition and does not need other sensors. The CRT measurement result is directly obtained by analyzing the collected image sequence, and the values in the hsv space of the finger image are used to judge the measurement starting point and the measurement ending point. The algorithm flow is shown as Figure 3 The specific steps of the algorithm are as follows:

[0037] 1. Image sequence preprocessing: convert the image sequence to hsv space

[0038] 2. Generate key finger area mask: segment the key area mask for the hsv image through deep learning semantic segmentation, so that the algorithm can accurately locate the most critical part of the finger area for measurement, and accurately avoid the influence caused by background difference, individual difference, and sensor noise. The training data set of the key area is collected by professionals from various types of personnel capillary refill time measurement video data, and then professional annotation is obtained. Multiple rounds of training are performed using Mask CNN neural network.

[0039] 3. Calculate the average s value size of the finger area: calculate the average s_mean of the s value of the pixels in the finger area mask;

[0040] The finger Hsv image is a three-dimensional data of h, s, and v, and the s value is the s-dimensional data.

[0041] 4. Smoothing processing: calculate the average of s_mean with a granularity of 5 frames, and smoothing processing can avoid the influence of abnormal values

[0042] 5. Measurement starting point judgment: the minimum value of s during the measurement process is taken as the measurement starting point st_time

[0043] 6. Measurement endpoint judgment: use the value when s is stable multiplied by a calibration coefficient (0.5-1) as the threshold, and the time when s first reaches the threshold after the measurement starting point st_time as the measurement endpoint ed_time

[0044] 7. CRT calculation: ed_time-st_time

[0045] The capillary refill time is ed_time-st_time.

[0046] Through the above steps, the capillary refill time CRT based on deep learning is measured, which can quickly and efficiently measure the capillary refill time.

[0047] It should be noted that the purpose of publishing the embodiments is to help further understand the present application, but those skilled in the art can understand that various substitutions and modifications are possible without departing from the scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed content of the embodiments, and the scope of protection claimed by the present application is subject to the scope defined by the claims.

Claims

1. A deep learning-based finger capillary refill time measurement method, which standardizes the measurement process of finger capillary refill time (CRT), designs a portable CRT measurement device, automatically collects images and accurate parameters of a specific part of a finger using artificial intelligence, establishes a deep learning positioning key finger area algorithm, uses the color saturation of the finger area image to determine the start and end points of the CRT measurement, and thus realizes fast and accurate real-time monitoring of CRT; comprising the following steps: 1) Using a capillary refill time measurement device, automatically collecting image sequence information of a specific part of a finger during the measurement process, including accurate parameters of the image; The accurate parameters are the color saturation of the image of the specific part of the distal end of the finger during the measurement process; the specific part is the area where the skin of the pressed finger turns white to normal color during the measurement process; 2) Image sequence preprocessing: convert the image sequence to hsv color space to obtain the hsv image of the finger; 3) Create a deep learning network model of the finger image to generate a key finger area mask; including: 31) Perform semantic segmentation on the finger hsv image obtained in step 2) using deep learning to obtain a key area mask, which allows accurate positioning of the key area of the finger; 32) Use the image mask segmentation neural network Mask CNN to generate a key area mask; Use the prediction head of the image mask segmentation neural network Mask CNN to determine whether each input image pixel belongs to the key area of the finger; Train the key area of the finger; the training data set is obtained by professional annotation of the video for measuring capillary refill time; set the number of iterations and use the Adam optimizer for training; 4) Extract the value of the s channel in the image mask area hsv image and calculate the average value of the s value of the pixels in the image mask area, denoted as s_mean; the s value is the value of the color saturation in the hsv image; 5) Set the smoothing granularity to smooth s_mean: 6) Take the minimum value of s during the measurement process as the measurement start time, denoted as st_time; 7) Obtain the measurement end time: Set the granularity standard deviation threshold; Then multiply the value of s when the standard deviation of s in the set granularity range is less than the granularity standard deviation threshold by the calibration coefficient to obtain the s threshold value; Take the time when s first reaches the s threshold value after the measurement start time st_time as the measurement end time, denoted as ed_time; 8) Calculate the CRT: CRT = ed_time-st_time; The capillary refill time is ed_time-st_time. 2.The method of claim 1, wherein the method further comprises: determining a first time point at which the first image is captured; determining a second time point at which the second image is captured; and determining the capillary refill time based on the first time point and the second time point. In step 32), the image mask segmentation network uses a simplified EfficientNet as the main part; the input and output of the network are both 64x64 resolution; the simplified EfficientNet includes 3 modules of mobile inverted bottleneck MBConv k5x5; the prediction head of the image mask segmentation uses a standard image segmentation prediction head, which is a 3x3 convolution structure. 3.The method of claim 1, wherein the method further comprises: determining a first time point at which the first image is captured; determining a second time point at which the second image is captured; and determining a time interval between the first time point and the second time point. The collection method of the training data set of the key area is specifically as follows: The ventral side of the distal phalanx of the right hand is pressed by a glass sheet, and the pressure is gradually increased until the skin turns white, and the time from the end of the pressing to the recovery of the normal color of the skin is the CRT; a camera is used to shoot the color change process of the finger seen from the end of the glass sheet; The area with the most significant color change of the ventral side of the distal phalanx of the right hand is manually marked. 4.The method of claim 1, wherein the method further comprises: determining a first time point at which the first image is captured; determining a second time point at which the second image is captured; and determining a time interval between the first time point and the second time point. In step 5), s_mean is smoothed, specifically, the average of s_mean is calculated with a granularity of 5 frames to avoid the influence of abnormal values. 5.The method of claim 1, wherein the method further comprises: determining a first time point at which the first image is captured; determining a second time point at which the second image is captured; and determining a time interval between the first time point and the second time point. In step 7), the granularity standard deviation threshold value can be 10, and the calibration coefficient can be 0.5-1.

6. A capillary refill time measuring device, implemented by using the capillary refill time measuring method based on deep learning according to any one of claims 1 to 5, characterized in that, It comprises a shell, a screen, a button switch, a camera or a camera, a processing circuit, a light source, a pressing device; wherein the pressing device comprises a transparent plate, an air pump, an air pipe, an air bag, an exhaust valve, a buffer material; the processing circuit comprises a power supply system, an embedded processing terminal and a control circuit; the embedded processing terminal adopts an arm processor and is used for data analysis; the control circuit is used for controlling the start and stop of the air pump and the electromagnetic air valve; the air pump is connected with the air pipe, the air valve and the air bag and is used for providing power for pressing the outer end of the finger; the buffer material adopts an easy-to-clean buffer material and is used for facilitating disinfection and application to different sizes of fingers after pressing; the transparent plate and the air pump-air bag structure cooperate as the pressing device; the internal light source provides a fixed light source for the camera to collect finger images; the shell isolates the finger measurement part from the outside world to avoid interference from external light sources; the screen is used for displaying the current color and state of the finger; the button is used for starting and ending measurement. Specifically, the finger is inserted into the capillary refill time measuring device, and the button switch is pressed; the light source in the measuring device is turned on, the camera or the camera starts shooting the state of the finger and displays it on the screen, the processing circuit controls the air pump to provide pressing power, the transparent plate presses the ventral side of the distal phalanx of the finger, and the pressure is gradually increased until the skin turns white; maintain for a few seconds, the air pump stops inflating, the air valve opens, and the pressing ends; after the skin returns to normal color, the user removes the device from the hand; the capillary refill time can be measured.

7. The capillary refill time measuring device of claim 6, wherein the capillary refill time measuring device further comprises a capillary refill time measuring device. ​

Images