System for conducting assessment of puncture point for collecting blood

Abstract

Disclosed is a system for conducting assessment of a puncture point for collecting blood. This system involves: acquiring an image of a body site from which blood is to be collected; determining, within the image, a plurality of puncture candidate points on the body site; calculating the degree of similarity between each of the puncture puncture points and respective points in the image; generating a feature map for each of the puncture candidate points and indicating the degree of similarity between each of the puncture candidate points and the points in the image; and determining the degree of puncture suitability for each of the puncture candidate points from the respective feature maps of the puncture candidate points.

Description

System for performing evaluation of puncture points for blood collection - Patents.com Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-127118, filed on August 3, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to evaluation of a puncture location for blood collection.

[0003] Blood tests have become essential in the medical field as they provide basic information for diagnosing various diseases. In recent years, there are expectations that blood tests will also be used to diagnose early signs of disease in order to curb ever-increasing medical costs, and it is expected that demand for blood tests will continue to increase in the future.

[0004] With this increase in demand, there are concerns about the increased burden on medical professionals who collect blood. Against this background, automatic blood collection devices that automatically collect blood without human intervention have been attracting attention in recent years, and there is a particular demand for small, space-saving automatic blood collection devices that can be placed without taking up too much space in medical settings.

[0005] One known example of this type of automatic blood collection device is the blood collection device disclosed in Patent Document 1. In this blood collection device, when a person to be blood-collected places their fingertip at a predetermined position on the device, a needle automatically moves to puncture the fingertip, creating a small wound, and the blood that leaks out is collected in a collection container, thereby collecting the blood. This blood collection device has been made smaller by using the fingertip as the blood collection site.

[0006] Generally, there are some parts of a human finger that are suitable for blood collection and some that are not. However, Patent Document 1 does not take into consideration the suitability of the blood collection position and only discloses that the blood collection device mechanically inserts the puncture needle into a fixed position. As a result, there is a possibility that a sufficient amount of blood may not be collected due to individual differences in the subject's finger, changes in the posture and position of the presented finger, etc.

[0007] On the other hand, one example of research aimed at stable automatic blood sampling is described in Non-Patent Document 1. Non-Patent Document 1 discloses a method of capturing an image of the vein pattern of a finger using a near-infrared camera, analyzing the captured image, and extracting the branching points of the blood vessels as the puncture positions.

[0008] Even when medical professionals manually draw blood, they generally aim to insert a blood collection needle into a vein in the arm, and since branching points of blood vessels are points where multiple blood vessels converge, a relatively large blood flow is expected. Therefore, we aim to achieve stable blood collection by enabling control with a robotic arm so that the blood collection needle can be inserted at any position on the finger, and by programming the blood collection needle to puncture this vascular branching point.

[0009] JP 2019-88391 A

[0010] Li, X., Lin, J., Pang, Y., Huang, L., Zhong, L., Li, Z., “Fingertip Blood Collection Point Localization Research Based on Infrared Finger Vein Image Segmentation”, IEEE Transactions on Instrumentation and Measurement, Vol.71, pp.1-12 (2021)

[0011] However, due to individual differences in the optimal puncture position, it is difficult to obtain a consistent amount of blood. Automating blood collection requires detecting the features of the optimal puncture position, but since there are many variations in the features and the image is affected by fingerprints, it is not easy to establish rules for detecting the features.

[0012] The present invention has been made in consideration of the above points, and proposes a technique for evaluating a blood collection position that allows stable blood collection.

[0013] One aspect of the present invention is a system for performing evaluation of puncture points for blood collection, comprising one or more processors and one or more storage devices that store programs executed by the one or more processors, wherein the one or more processors acquire an image of a biological part from which blood is to be collected, determine a plurality of candidate puncture points for the biological part in the image, calculate a similarity between each of the plurality of candidate puncture points and each point in the image, generate a feature map for each candidate puncture point indicating the similarity between each candidate puncture point and each point, and determine a suitability for puncturing for each of the candidate puncture points from the feature map for each of the candidate puncture points.

[0014] According to one aspect of the present invention, it is possible to more appropriately evaluate a puncturing position suitable for blood sampling.

[0015] 10 shows the configuration of an automatic blood sampling device according to an embodiment of the present specification. FIG. 10 shows an example of an input image captured by a camera. FIG. 10 shows the overall flow of automatic blood sampling processing executed by the automatic blood sampling device. FIG. 10 shows a flowchart of processing to select an optimal puncture point using a learning model in an embodiment of the present specification. FIG. 10 shows an example of a feature map for a selected candidate puncture point. FIG. 10 shows an example of a feature map for a selected candidate puncture point. FIG. 10 shows an example of a feature map for a selected candidate puncture point. FIG. 10 is a diagram for explaining processing to select an optimal puncture point using the learning model shown in FIG. 4. FIG. 10 shows a flowchart of an example of processing to generate a feature map for one candidate puncture point in the step of generating a feature map for each selected candidate puncture point from an input image. FIG. 10 shows a flowchart of an example of learning of a puncture suitability calculation model. FIG. 10 is a schematic diagram for explaining learning of the puncture suitability calculation model described with reference to FIG. 8. FIG. 10 is a flowchart showing the flow of learning of a feature map extraction model. FIG. 10 shows an example of processing of the feature map extraction model during learning. FIG. 10 schematically shows display of a feature map on a display device.

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. It should be noted that this embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are used to designate common components in each drawing.

[0017] The processor realizes a predetermined function by executing a program stored in a primary storage device. The primary storage device stores the program executed by the processor and data necessary for the program execution. The program includes an operating system (OS) (not shown) as well as other programs. The processor may include multiple chips and multiple packages.

[0018] A program is executed by a processor to perform a predetermined process using a storage device and a communication port (communication device). Therefore, a description that uses the program as the subject may also use the processor as the subject. Alternatively, the process executed by a program is a process performed by the computer and computer system on which the program runs.

[0019] A processor operates according to a program, thereby acting as a functional unit (means) that realizes a predetermined function. For example, a processor functions as a unit (means) by operating according to a certain program, and as a unit (means) by operating according to a certain program. The same is true for other programs. Furthermore, a processor also operates as a functional unit (means) that realizes each of the multiple processes executed by each program. A computer and a computer system are devices and systems that include these functional units (means).

[0020] In one embodiment of the present specification, a feature value that distinguishes blood vessels from a background is extracted, and puncture points (puncture positions) suitable for blood collection are evaluated or prioritized based on the feature value. In one embodiment of the present specification, an image of the area to be blood-collected is acquired, and a feature map indicating the degree of similarity between the image and candidate puncture points is generated. The device calculates a puncture suitability, which is the suitability for blood collection at the candidate puncture point, from the feature map. The puncture suitability indicates the expected amount of blood to be collected.

[0021] The device evaluates or prioritizes the puncture points by referring to the puncture suitability, which allows for stable blood collection regardless of individual differences such as the posture of the subject presenting the body part or the condition and shape of the body part.

[0022] The following describes an automatic blood sampling device that selects an optimal puncture point from a body part in consideration of the puncture suitability, and then inserts a puncture needle into the selected puncture point to collect blood. Figure 1 shows the configuration of an automatic blood sampling device 1 according to one embodiment of this specification. The automatic blood sampling device 1 is configured with a calculation unit 2, a light source control unit 3, an image input unit 4, a mechanism control unit 5, a communication unit 6, a camera 7, a near-infrared light source 8, a movement mechanism 9, and a blood sampling tube 10.

[0023] The calculation unit 2 is configured to include a CPU (Central Processing Unit) 21, a memory 22, and an auxiliary storage device 23, which are interconnected via an internal bus 20, and interfaces 24 provided corresponding to the light source control unit 3, the image input unit 4, the mechanism control unit 5, and the communication unit 6. Note that the CPU 21, the memory 22, the auxiliary storage device 23, and each interface 24 may be interconnected by a dedicated bus for the purpose of increasing speed, etc.

[0024] The CPU 21 is a processor that controls the overall operation of the automatic blood collection device 1. The memory 22 is composed of RAM (Random Access Memory) and ROM (Read Only Memory), and is used to permanently store programs required for processing and temporarily store data required for processing. The CPU 21 executes the programs stored in the memory 22, thereby executing various processes of the automatic blood collection device 1 as a whole, as described below.

[0025] The auxiliary storage device 23 is configured, for example, by a flash memory or a hard disk drive, and is used to store data that needs to be permanently saved. Data is input and output to and from the auxiliary storage device 23 via the memory 22 under the control of the CPU 21.

[0026] The interface 24 is a device that has the function of connecting the calculation unit 2 and the corresponding functional block (light source control unit 3, image input unit 4, mechanism control unit 5 or communication unit 6) so that necessary data can be exchanged between the calculation unit 2 and the corresponding functional block.

[0027] The existence of such an interface enables the calculation unit 2 to control the light emission state of the near-infrared light source 8 via the light source control unit 3, to input the output image of the camera 7 via the image input unit 4, and to drive and control the movement mechanism 9 via the mechanism control unit 5. The calculation unit 2 can also communicate with an external device (not shown) connected to the network 11 via the communication unit 6. Such communication enables, for example, part or all of the processing associated with blood collection to be shared with an external device (for example, an external calculation device acting as a host), or to link with various services on the cloud.

[0028] The camera 7 is composed of, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, and captures an image of the subcutaneous blood vessel pattern of the subject's biological part (hereinafter referred to as the finger) 12 from which blood is to be collected, and outputs image data of the captured image to the image input unit 4.

[0029] Near-infrared light is transmitted through the living body and photographed with a camera 7 compatible with the near-infrared wavelength range. Blood vessels appear as dark lines. This is because hemoglobin contained in blood has the property of absorbing near-infrared light. When near-infrared light is transmitted through blood vessels containing a large amount of hemoglobin, the light is weakened, causing the blood vessels to appear darker than their surroundings. In this way, near-infrared images can display the distribution of blood vessels or blood flowing through the blood vessels in the target area. Essentially, the puncture suitability of each point on the blood vessel can be calculated by analyzing this photographed image. Note that light in a wavelength range other than near-infrared may be used as long as similar processing is possible.

[0030] The near-infrared light source 8 is composed of, for example, a near-infrared LED (Light Emitting Diode) or the like, and emits near-infrared light of an intensity corresponding to the amount of driving power based on the driving power provided by the light source control unit 3. The near-infrared light source 8 is fixedly disposed so that its optical axis coincides with the optical axis of the camera 7, at a distance that allows the subject's finger 12 to be inserted, as shown in FIG. 1 . The reason why the subject's finger 12 is sandwiched between the near-infrared light source 8 and the camera 7 in this manner is that a human finger is thick enough to allow near-infrared light to pass through it sufficiently, and a blood vessel pattern with higher contrast can be obtained by transmitting near-infrared light from the side opposite the finger surface facing the camera 7.

[0031] Alternatively, a light source may be placed on the same side as the camera 7, and the vein pattern may be photographed using the light that is reflected back and irradiated onto the surface of the finger 12 to be photographed. If the blood sample is taken from a thick body part such as the arm, a sufficient amount of light may not be able to pass through, making it difficult to obtain contrast in the vein pattern, so the reflected light method is more suitable.

[0032] The moving mechanism 9 is a mechanism that moves the relative position of the puncture needle 13 with respect to the finger 12, which is presented at a predetermined position and in a predetermined state by the subject of blood collection, including the direction in which the puncture needle 13 is inserted into the finger 12 and the direction in which the puncture needle 13 is removed from the finger 12 after having been inserted into the finger 12. The moving mechanism 9 includes a pushing mechanism 16 having the puncture needle 13 replaceably attached to its tip. The pushing mechanism 16 includes an actuator such as a motor (not shown), and the actuator can move the puncture needle 13.

[0033] The light source control unit 3 is a functional block having a function of controlling the brightness of the near-infrared light emitted by the near-infrared light source 8 based on a target value specified by the program executed by the CPU 21 so that the blood vessel pattern becomes the clearest in the image captured by the camera 7. Specifically, the light source control unit 3 controls the amount of driving power applied to the near-infrared light source 8 by PWM (Pulse Width Modulation) so that it becomes the target value specified by the program executed by the CPU 21.

[0034] The image input unit 4 is a functional block that converts the image data provided by the camera 7 into a format that is easy for the calculation unit 2 to handle, and stores the converted image data in the memory 22. The mechanism control unit 5 controls the moving mechanism 9 in accordance with instructions from a program executed by the CPU 21.

[0035] In the automatic blood sampling device 1 having the above configuration, the CPU 21 analyzes the image of the vascular pattern obtained as described above based on the program stored in the memory 22, and calculates the coordinate value of the point on the image that is determined to be most suitable for puncturing (optimal puncture point).

[0036] FIG. 2 shows an example of an input image 30 captured by the camera 7. The input image 30 shows an image 31 of the finger 12. FIG. 2 also shows an example of a trajectory 32 of the puncture needle 13. The image 31 of the finger includes an image 33 of a plurality of blood vessels. In FIG. 2, the image of one of the plurality of blood vessels is indicated by the reference numeral 33 as an example. The puncture needle 13 can only move on the predetermined trajectory 32. By restricting the degree of freedom of movement, the mechanism for moving the puncture needle 13 can be made smaller.

[0037] 2, the puncture needle 13 can move on a trajectory (rotational trajectory) 32 that forms part of a circle. The CPU 21 selects the optimal puncture point from the trajectory 32 of the puncture needle 13. This reduces the processing load for selecting the optimal puncture point. Note that the puncture needle 13 may be movable to any position.

[0038] 3 shows the overall flow of the automatic blood sampling process executed by the automatic blood sampling device 1. When the automatic blood sampling process starts, the CPU 21 executes initialization processing of the programs executed by the CPU 21 and the functional units 3 to 6 (S11). Next, the CPU 21 acquires an image of the finger 12 captured by the camera 7, which is input from the image input unit 4 (S12).

[0039] The CPU 21 analyzes the acquired image and determines whether a finger is included in the image (S13). Since widely known image recognition technology can be used to detect a finger in the image, details of this method will be omitted. Note that a finger in the image may also be detected by a method other than image recognition, such as pressing a mechanical switch.

[0040] If a finger is not detected in the input image (S13: NO), the flow returns to step S12. If a finger is detected in the input image (S13: YES), the CPU 21 refers to the input image and determines whether the light intensity of the near-infrared light source 8 is appropriate (S14). For example, the CPU 21 can determine whether the light intensity is appropriate based on the clarity of the blood vessel pattern in the input image.

[0041] If the light intensity of the near-infrared light source 8 is inappropriate (S14: NO), the CPU 21 adjusts the light intensity of the near-infrared light source 8 using the light source control unit 3 (S15). The CPU 21 instructs the light source control unit 3 on the amount of driving power to be applied to the near-infrared light source 8 that corresponds to the changed light intensity. Thereafter, the flow returns to step S12.

[0042] If the light intensity of the near-infrared light source 8 is appropriate (S14: YES), the CPU 21 analyzes the input image using the learning model and selects the optimal puncture point (S16). Details of step S16 will be described later.

[0043] Next, CPU 21 controls moving mechanism 9 to collect blood at the optimal puncturing point (S17). Specifically, CPU 21 notifies mechanism control unit 5 of the coordinate value of the optimal puncturing point calculated in step S16. When mechanism control unit 5 is notified of the coordinate value of the optimal puncturing point from CPU 21, it controls moving mechanism 9 to cause puncturing needle 13 to puncture finger 12 presented by the subject at that coordinate value.

[0044] When the puncture needle 13 punctures the finger 12 of the person to be sampled, the mechanism control unit 5 operates the moving mechanism 9 to move the puncture needle 13 in a direction to withdraw it from the finger 12 of the person to be sampled, and then controls the moving mechanism 9 to move the blood collection tube 10 to directly below the point on the finger 12 of the person to be sampled that has been punctured by the puncture needle 13. As a result, blood bleeding from the finger 12 of the person to be sampled is received by the blood collection tube 10, and once a specified volume of blood has been sampled, the blood collection is complete. The structure of the moving mechanism 9 and the method of collecting blood into the blood collection tube 10 are arbitrary.

[0045] In one embodiment of the present specification, an optimal puncture point is selected from an input image of a finger using an optimal puncture point selection model, which is a machine learning model. Fig. 4 shows a flowchart of the optimal puncture point selection process S16 using the learning model in one embodiment of the present specification.

[0046] The CPU 21 acquires an image of the fingertip from which blood is to be collected (S21). Two trained learning models are used to select the puncture points. The first learning model extracts a feature map corresponding to all pixels in the image (S22). As will be described in more detail later, the feature map is obtained through self-learning, which does not require labels.

[0047] Next, the CPU 21 applies the second learning model to the feature map at each puncture candidate point (S23). The CPU 21 calculates the puncture suitability from the distribution of the entire image obtained from the feature map and determines whether the point is suitable for puncture (S24). The second learning model has been trained using learning data. The puncture suitability is learned and estimated using the distribution of the feature map of the entire image and training data on the amount of blood collected. While conventional methods focus on local features of the puncture candidate point at the pixel corresponding to the trajectory 32 of the needle 13, this embodiment utilizes features of the entire image to determine the puncture suitability. The input of the second learning model is the feature map of one puncture candidate point, and its output is the puncture point suitability. The trained learning model is stored, for example, in the auxiliary storage device 23. The learning model can be generated using, for example, a deep neural network, and as an example, a convolutional neural network (CNN) or a residual network can be used.

[0048] The puncture suitability indicates the amount of blood drawn at the puncture candidate point, with a higher suitability indicating a larger amount of blood drawn. The training data for the learning model are various combinations of feature maps of the puncture candidate points and values ​​obtained from the blood draw amounts at those points.

[0049] The CPU 21 determines whether the finger in the image is suitable for blood collection based on the puncturing suitability of the multiple puncturing candidate points, and if it determines that the finger is suitable for blood collection, it determines the puncturing point from the multiple puncturing candidate points where blood will actually be collected (S25).

[0050] There are cases where a puncture point determined to be appropriate cannot be obtained from the state of the finger obtained from the image. For example, if the puncture suitability of all puncture candidate points is lower than a predetermined threshold, the finger is determined to be unsuitable for blood collection. As another example, if the average value of the puncture suitability of multiple puncture candidate points is lower than a predetermined threshold empirically determined in advance, the finger is determined to be unsuitable. This makes it possible to more reliably collect appropriate blood.

[0051] After determining that the finger in the image is suitable for blood sampling, the CPU 21 selects the optimal puncturing point from the candidate puncturing points. For example, the CPU 21 can select the candidate puncturing point with the highest puncturing suitability, or a point on the finger that satisfies a preset condition for its position, such as a point closest to the center of the finger, under the condition that the puncturing suitability is higher than a predetermined threshold. In this way, the CPU 21 calculates the puncturing suitability from the feature map, and evaluates and prioritizes the puncturing positions suitable for blood sampling.

[0052] 5A to 5D show examples of feature maps 40A to 40D for selected candidate puncture points. In FIGS. 5A to 5D, a fingertip 41 is indicated by a dashed line. Furthermore, crosses 45A to 45D indicate selected candidate puncture points (pixels), respectively. The feature maps indicate points (regions) that have similar features to the selected candidate puncture point. The feature amount of each point indicates the characteristics of the point, such as its texture, brightness, and spatial structure. These features can be calculated from the brightness value of the pixel of the point and the brightness values ​​of its surrounding pixels.

[0053] 5A to 5D show regions similar to selected puncturing candidate point 45A as black regions. Fig. 5A shows selected puncturing candidate point 45 and a region 461 similar to that point. Fig. 5B shows selected puncturing candidate point 45B and a region 461B similar to that point. Fig. 5C shows selected puncturing candidate point 45C and a region 461C similar to that point. Fig. 5D shows selected puncturing candidate point 45D and regions 461D, 462D similar to that point. In the examples of Figs. 5A to 5D, regions with a high degree of similarity to the puncturing candidate point are automatically visualized.

[0054] The feature map indicates an area having a texture, brightness, and spatial structure similar to that of the selected point. The feature map is generated from a near-infrared image of the fingertip. As described above, a near-infrared image displays blood vessels as dark areas compared to other parts (background). Therefore, when the selected candidate puncture point is in the blood vessel area, the feature map mainly indicates some of the points in the blood vessel area as points similar to the selected candidate puncture point. On the other hand, when the selected candidate puncture point is in the background outside the blood vessel area, the feature map mainly indicates some of the points in the background as points similar to the selected candidate puncture point. In this way, the feature map indicates feature amounts related to the blood vessel and background at the selected candidate puncture point extracted from the input image.

[0055] For example, Figures 5A and 5D show selected puncture candidate points 45A and 45D as examples of points in a blood vessel region, and further show point clouds in the blood vessel region similar to these, while Figures 5B and 5C show selected puncture candidate points 45A and 45D as examples of points in the background, and further show point clouds in the background similar to these.

[0056] Feature maps can be generated using a learning model. It is difficult to describe the feature quantities for the appropriate puncture point (position) for blood collection as rules, and the rules must be adjusted when applied to new data through horizontal deployment. In this way, by generating a feature map using a learning model and calculating the puncture suitability from that feature map, it becomes possible to automatically select a more appropriate puncture point.

[0057] Fig. 6 is a diagram for explaining the process of selecting the optimal puncturing point using the learning model shown in Fig. 4. A near-infrared image of the fingertip of a blood sampling target is input to calculation unit 2 as input image 30 (S21). CPU 21 extracts a feature map of each selected puncturing candidate point from input image 30 (S22).

[0058] Next, CPU 21 applies learning model 211 to each feature map to analyze the spatial pattern and calculate the puncturing suitability of each selected puncturing candidate point. Learning model 211 outputs the puncturing suitability for the input feature map. CPU 21 determines whether each selected puncturing candidate point is reliable as a puncturing point based on the calculated puncturing suitability (S23, S24). In the example shown in Fig. 6, feature map 40B is a feature map of puncturing candidate points with a low probability of being optimal points, and feature map 40D is a feature map of puncturing candidate points with a high probability of being optimal points.

[0059] CPU 21 determines whether the finger in input image 30 is suitable for blood sampling based on the puncturing suitability of the selected puncturing candidate point, and performs blood sampling at one puncturing candidate point if it is suitable (S25). For example, if the highest puncturing suitability exceeds a predetermined threshold, the finger may be determined to be suitable for blood sampling. Furthermore, the puncturing candidate point with the highest puncturing suitability may be selected as the puncturing point where blood sampling is performed.

[0060] In one embodiment of the present specification, a feature map of selected candidate puncture points is generated from an input image using a learning model. The learning model has been trained using training data and is stored, for example, in the auxiliary storage device 23. Fig. 7 shows a flowchart of an example process for generating a feature map of one candidate puncture point in step S22 of generating a feature map for each selected candidate puncture point from an input image.

[0061] The CPU 21 acquires an input image from which a feature map is to be extracted (S31). As described above, an example of the input image is a near-infrared image of a fingertip. Next, the CPU 21 applies a learning model to the acquired image (S32) and extracts a feature map for the puncturing candidate points (S33). The inputs to the learning model are the input image and the coordinates within the image of the selected puncturing candidate points, and the output of the learning model is a feature map extracted from the input image for the puncturing candidate points. The configuration of the learning model that generates the feature map and the learning method thereof will be described in detail below.

[0062] An example of a learning method (training method) for a learning model used to determine the puncturing point where blood will be collected is described below. As described above, the automatic blood collection device 1 in one example of this specification uses two learning models. One is a feature map extraction model that generates a feature map at the candidate puncturing points in the input image. The other is a puncturing suitability calculation model 211 that calculates the puncturing suitability from the feature map.

[0063] First, an example of learning of the puncturing suitability calculation model 211 will be described. Fig. 8 shows a flowchart of an example of learning of the puncturing suitability calculation model 211. First, the CPU 21 acquires a large number of near-infrared images of fingertips to serve as learning samples (S41). The near-infrared images may be newly generated by the automatic blood sampling device 1, or may be stored in advance in one of the databases.

[0064] Next, the CPU 21 extracts a feature map for each point in each near-infrared image (S42). Specifically, the CPU 21 sequentially selects near-infrared images from the learning sample and performs the following process for each selected near-infrared image. The CPU 21 inputs the selected near-infrared image and the coordinates of pixels selected from the near-infrared image into a trained feature map extraction model. The feature map extraction model outputs a feature map according to the input. This results in a feature map for the pixels of the selected near-infrared image. The CPU 21 acquires feature maps for all pixels of the selected near-infrared image or for pixels at preset coordinates.

[0065] Next, the CPU 21 stores the combinations of the feature maps and the blood collection volumes corresponding to the feature maps in learning data (training data) (S43). The blood collection volumes corresponding to the feature maps can be input by the user via the communication unit 6 or obtained from a database (not shown). The learning data is stored in, for example, the auxiliary storage device 23. The learning data includes many combinations of the feature maps and the corresponding blood collection volumes.

[0066] Next, the CPU 21 uses the learning data to train the puncturing suitability calculation model 211, that is, to update the learning parameters (S44). The parameter update can be performed by backpropagating the error between the calculated puncturing suitability and the correct puncturing suitability. The puncturing suitability output by the puncturing suitability calculation model 211 may be the blood collection volume itself or a value calculated from the blood collection volume using a predetermined method. A relationship is satisfied in which the puncturing suitability increases as the blood collection volume increases. As a result, the puncturing suitability can numerically indicate how much blood can be expected to be collected. This learning makes it possible to learn the correlation between the image features around a puncturing point where a large amount of blood is collected and the amount of blood collected.

[0067] When the learning of the puncturing suitability calculation model 211 is completed, the CPU 21 stores the learned puncturing suitability calculation model 211 in the auxiliary storage device 23 (S45). Note that the learning of the puncturing suitability calculation model 211 may be executed by a computer different from the automatic blood sampling device 1, and the learned puncturing suitability calculation model 211 may be stored in the auxiliary storage device 23.

[0068] FIG. 9 is a schematic diagram for explaining the learning of the puncturing suitability calculation model 211 described with reference to FIG. 8 . As described above, many different near-infrared images 30 of fingertips are prepared as learning samples. The training data 35 includes a feature map for each pixel (point) of each near-infrared image 30, and further includes the amount of collected blood corresponding to each feature map. The amount of collected blood corresponding to the feature map is the amount of blood collected when the pixel in the finger of the image from which the feature map was extracted is used as the puncturing point. Note that the puncturing suitability may be calculated using a rule-based model different from the machine learning model.

[0069] As described above, one example of this specification calculates a quantitative numerical index according to the expected blood collection amount, and selects the most suitable point for puncturing based on the relative level of that index. This allows the selection of the next best position for puncturing even when the optimal position is not on the trajectory, preventing a significant decrease in the amount of collected blood, thereby improving the stability of blood collection even with a small blood collection device.

[0070] Next, a method for training the feature map extraction model will be described. In one embodiment of this specification, the training data for the feature map extraction model consists of only input images. In other words, the user does not need to prepare a feature map corresponding to the input image as ground truth data. This allows the feature map extraction model to be trained efficiently.

[0071] 10 is a flowchart showing the flow of learning the feature map extraction model. The CPU 21 acquires many near-infrared images of different fingertips to serve as learning samples for the feature map extraction model (S51), and constructs a learning dataset from these near-infrared images (S52). The near-infrared images used for learning may be newly generated by the automatic blood sampling device 1, or may be stored in advance in an external or internal database.

[0072] Next, the CPU 21 trains the feature map extraction model using the constructed training data set (S53). The training of the feature map extraction model will be described in detail below. When the training of the feature map extraction model is completed, the CPU 21 stores the trained feature map extraction model in the auxiliary storage device 23 (S54). Note that the training of the feature map extraction model may be executed by a computer different from the automatic blood collection device 1, and the trained feature map extraction model may be stored in the auxiliary storage device 23.

[0073] An example of a method for training the feature map extraction model is described below. FIG. 11 shows an example of the processing of the feature map extraction model 215 during training. FIG. 11 shows the processing of the feature map extraction model 215 during training. All of the configuration processing is required during training, but the purpose of the automatic blood sampling process is to extract a feature map 535 using a trained model. Therefore, some of the configuration processing shown in FIG. 11 is not necessary, and some processing not shown in FIG. 11 may be performed.

[0074] The feature map extraction model 215 includes an encoder-decoder network 510. The encoder-decoder network 510 includes multiple blocks 511-517. The blocks 511-516 have a common configuration. Specifically, each block is composed of padding, 3x3 convolution, Relu, padding, 3x3 convolution, and Relu. The block 517 includes a self-attention mechanism 530 in addition to the common configuration with the other blocks. For example, the first half blocks 511-514 are for encoding processing, and the second half blocks 515-517 are for decoding processing. Note that the number of blocks is an example, and the processing of some blocks may be different.

[0075] A near-infrared image 30 is input to the input layer of block 511 of the encoder-decoder network 510. The near-infrared image 30 is composed of H pixels in the height direction and W pixels in the width direction, and is one-channel data. In block 511, the encoder-decoder network 510 performs padding, 3×3 convolution, Relu, padding, 3×3 convolution, and Relu processing. Then, Max pooling is applied to the feature amounts from block 511 to reduce the resolution.

[0076] Similarly, the encoder-decoder network 510 applies block 512, Max pooling, block 513, and Max pooling to the feature in that order. The encoder-decoder network 510 applies block 514 to the feature, and then applies bilinear upsampling to increase the resolution.

[0077] The encoder-decoder network 510 combines the features with increased resolution with the features obtained from block 513 and applies block 515. The encoder-decoder network 510 applies bilinear upsampling to the features obtained from block 515 via skip connections to increase the resolution, and then combines the features with the features obtained from block 512 via skip connections and applies block 516.

[0078] The encoder-decoder network 510 applies bilinear upsampling to the feature obtained from block 516 to increase the resolution, and then applies this to block 517 by combining it with the feature obtained from block 511 via a skip connection. Block 517 performs the same processing as the other blocks. As a result, the resolution of the feature returns to the original resolution of the input image. In other words, the feature has a resolution of height H and width W. Note that the number of channels is different from the 1 of the input image.

[0079] The Encoder-Decoder network 510 executes the processing of the self-attention mechanism 530 on features having the same resolution (H × W) as the input image 30. Finally, a 1 × 1 convolution is applied to the output from the self-attention mechanism 530 to restore the number of channels to the same number of channels (1) as the input image 30, and output as the output image.

[0080] An example of a loss function used to update the parameters of the feature map extraction model 215 is the Structural Similarity Index Measure (SSIM). The SSIM is a function suitable for evaluating the structural similarity between two images. The parameters of the feature map extraction model 215 are updated to reduce the deviation in the spatial patterns between the input image 30 and the output image. Note that a loss function other than SSIM can be used.

[0081] The processing of the self-attention mechanism 530 (self-attention processing) is described below. The self-attention mechanism 530 learns the relationships between pixels and autonomously directs attention to important features. The self-attention mechanism 530 searches for similar regions for each point with the same resolution as the input image. As shown in FIG. 11 , the features input to the self-attention mechanism 530 are three-dimensional tensors 531 having height H, width W, and number of channels C. The height H and width W match the height and width resolution (number of pixels) of the input image.

[0082] The self-attention mechanism 530 converts the three-dimensional tensor 531 into a two-dimensional matrix 532. The two-dimensional matrix 532 has C columns and H×W rows. Each row of the two-dimensional matrix 532 indicates the feature of each pixel. Through the processing described above, the feature of a pixel includes information on the brightness of the pixel and surrounding pixels, and indicates the characteristics of the texture, brightness, and spatial structure. In this way, feature information related to blood vessels and the background at each point is extracted.

[0083] Next, the self-attention mechanism 530 applies Relu and regularization to each row (each pixel) of the matrix 532 to calculate the matrix 533. This extracts and normalizes only important features for each pixel.

[0084] Next, the self-attention mechanism 530 calculates a similarity matrix 535 by taking the inner product of the matrix 533 and its transpose matrix 534. Each element of the similarity matrix 535 indicates the similarity between two pixels. Each row of the similarity matrix 535 indicates the similarity between each pixel and each pixel in the input image. Therefore, each row corresponds to a similarity map for each pixel. The similarity map can separate regions with similar features, i.e., blood vessel regions and the background. As will be described later, a feature map at the selected point is generated from the similarity matrix 535, which is the output of the intermediate layer (hidden layer).

[0085] The self-attention mechanism 530 calculates a matrix 536 from the inner product of the similarity matrix 535 and the matrix 533, and converts the matrix 536 into a three-dimensional tensor 537. As described above, the encoder-decoder network 510 applies a 1×1 convolution to the three-dimensional tensor 537 to restore the number of channels to the same number of channels (1) as the input image 30, and outputs it as an output image from the output layer of the block 517.

[0086] As described above, the self-attention mechanism 530 extracts and normalizes only the significant features for each pixel, then generates a similarity matrix using matrix transformation and dot products. Each row of the similarity matrix corresponds to a similarity map for each point and is used to update the feature map. The feature map is trained to separate regions with similar features. This allows the feature map to be updated so that blood vessels and background can be separated in the input image.

[0087] The similarity matrix 535 indicates the similarity of each combination of pixel feature amounts. Therefore, a feature map of each candidate puncture point can be generated from the similarity matrix 535. That is, the CPU 21 executing the feature map extraction model 215 extracts the similarity between the designated candidate puncture point and each point from the similarity matrix 535, and generates a feature map of the candidate puncture point.

[0088] As described above, the feature map extraction model 215 learns to separate the features of blood vessels and the background, enabling automatic extraction of regions with the same structure as blood vessels. Images of blood vessels captured using near-infrared light are generally unclear, and variations in light transmittance are likely to occur due to differences in not only blood but also tissue composition within the body. When calculating the puncture suitability at a given point, a local feature such as low brightness at that point alone may not accurately distinguish whether there is actually a large amount of blood or whether such a feature is merely a coincidence due to unevenness. As described above, one embodiment of the present specification can calculate the puncture suitability while suppressing the influence of local variations by reflecting not only the local features of a single puncture candidate point but also the blood distribution state over a wider area including the vicinity of the point.

[0089] In addition, the feature map extraction model 215 performs learning (parameter updating) based on the error between the input image and the output image. Conventional methods typically prepare correct labels for the input near-infrared image and its corresponding vascular pattern, and train the network parameters to capture the features of the vascular pattern. However, manual labeling is difficult in terms of time and cost. In the feature map extraction model 215 of this embodiment, the same near-infrared image is treated as an input / output pair, and self-learning is performed because correct labels are not required. Parameters are updated by calculating the error between the network output image and the near-infrared image input image using SSIM. Parameter optimization and attention mechanisms are used to automatically extract features that emphasize blood vessels and background structures. As described above, one embodiment of this specification autonomously learns features related to the fingertip from input data without human intervention, making it possible to select an appropriate puncture location.

[0090] Note that, as long as it is possible to calculate the similarity between each point in the image and other points, the feature map of the candidate puncture points may be generated by a model or process different from the process using the feature map extraction model. For example, a learning model that updates parameters by learning using the feature map as a correct label may be used.

[0091] Next, the image display of the feature map will be described. The automatic blood sampling device 1 according to one embodiment of the present specification displays an image of the feature map in addition to or instead of automatically sampling blood at an appropriate puncturing point. The user can refer to the displayed feature map to decide whether or not to sample blood at the corresponding candidate puncturing point, or can refer to the displayed feature map after blood sampling to confirm whether the appropriate puncturing point was selected.

[0092] Fig. 12 schematically shows the display of a feature map by a display device. As described with reference to Fig. 1 , near-infrared light is irradiated onto finger 12 from near-infrared light source 8, and the transmitted light is detected by camera 7. From the near-infrared image generated by camera 7, a feature map for each of the candidate puncturing points is generated, as described above.

[0093] CPU 21 displays a feature map of the puncturing candidate points on display device 51 in response to a request from the user. Display device 51 may be, for example, a display device of a user terminal connected to calculation unit 2 via network 11 or a display device connected to the calculation unit via interface 24. The feature map to be displayed may be, for example, a feature map of the puncturing point where blood was actually collected or a feature map of the puncturing candidate points selected by the user. The feature map may be displayed alone or together with a captured image of the finger. Furthermore, the corresponding puncturing suitability may be displayed together with the feature map.

[0094] For example, when selecting a candidate puncturing point before blood sampling, CPU 21 displays a captured image of the finger and a feature map on display device 51. The user compares the feature maps of multiple candidate puncturing points and selects a puncturing point that is located in a sufficiently long and thick blood vessel. In response to the user's selection, CPU 21 performs blood sampling at the specified candidate puncturing point.

[0095] The user can visually check the spatial changes in feature values ​​when selecting a puncture point in real time, which allows for improved (optimized) selection of the puncture position and the expected increase in blood collection volume. In addition, by visualizing the feature map of the puncture point, it becomes possible to explain the cause of low blood collection volume in the unlikely event that such volume is low.

[0096] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0097] Furthermore, the above-described configurations, functions, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the program, table, and file that implements each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0098] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A system for performing evaluation of puncture points for blood sampling, comprising: one or more processors; and one or more storage devices for storing programs executed by the one or more processors, wherein the one or more processors obtain an image of a body part from which blood is to be sampled; determine a plurality of candidate puncture points for the body part in the image; calculate a similarity between each of the plurality of candidate puncture points and each point in the image; generate a feature map for each candidate puncture point indicating the similarity between each candidate puncture point and each point; and determine a suitability for puncturing for each of the candidate puncture points from the feature maps of the candidate puncture points.

2. A system as described in claim 1, wherein the one or more storage devices store a trained feature map extraction model that generates the feature map from the image, and a trained puncture suitability calculation model that calculates the puncture suitability from the feature map, wherein the feature map extraction model outputs a feature map indicating the similarity between the puncture candidate points and each point of the image for an input image and puncture candidate points, and the puncture suitability calculation model calculates the puncture suitability from the input feature map.

3. A system as described in claim 2, wherein the feature map extraction model has been trained using a training dataset, the training dataset being composed of a plurality of input images which are input to the feature map extraction model, during training using the training dataset, the feature map extraction model outputs an output image from an output layer and generates a similarity matrix indicating the similarity between points of the input images in an intermediate layer, a feature map of the candidate puncture points is generated from the similarity matrix, and the feature map extraction model is trained according to the error between the output image and the input image.

4. A system as described in claim 2, wherein the feature map extraction model includes an Encoder-Decoder network including a plurality of blocks performing preset image processing, one of the plurality of blocks includes a self-attention mechanism, the self-attention mechanism generates a similarity matrix indicating the similarity between points of an input image from features having the same resolution as the resolution of the image, and the feature map of the candidate puncture points is generated from the similarity matrix.

5. A system as described in claim 4, wherein the feature map extraction model has been trained using a training dataset, the training dataset being composed of a plurality of input images input to the feature map extraction model, during training using the training dataset, the Encoder-Decoder network outputs an output image, and parameters of the Encoder-Decoder network are updated according to an error between the output image and the input image.

6. A system as described in claim 2, wherein the puncture suitability calculation model has been trained using a training dataset, and the training dataset includes a plurality of combinations of feature maps of the puncture points input to the puncture suitability calculation model and correct labels of the puncture suitability indicating the amount of blood drawn at the puncture points.

7. The system of claim 1, wherein the one or more processors display the feature map on a display device.

8. A method in which a system performs evaluation of puncture points for blood sampling, the system acquiring an image of a body part from which blood is sampled, determining a plurality of candidate puncture points of the body part in the image, calculating a similarity between each of the plurality of candidate puncture points and each point in the image, generating a feature map for each candidate puncture point indicating the similarity between each candidate puncture point and each point, and determining a suitability for puncturing for each of the candidate puncture points from the feature maps of the candidate puncture points.

9. A method according to claim 8, wherein the system stores a trained feature map extraction model that generates the feature map from the image, and a trained puncture suitability calculation model that calculates the puncture suitability from the feature map, wherein the feature map extraction model outputs a feature map indicating the similarity between the puncture candidate points and each point of the image for an input image and puncture candidate points, and the puncture suitability calculation model calculates the puncture suitability from the input feature map.

10. A method according to claim 9, wherein the feature map extraction model has been trained using a training dataset, the training dataset being composed of a plurality of input images which are input to the feature map extraction model, during training using the training dataset, the feature map extraction model outputs an output image from an output layer and generates a similarity matrix indicating the similarity between points of the input images in an intermediate layer, a feature map of the candidate puncture points is generated from the similarity matrix, and the feature map extraction model is trained according to the error between the output image and the input image.

Images