System for executing evaluation of puncture point for performing blood sampling

Abstract

To more appropriately evaluate a puncture position suitable for blood sampling.SOLUTION: A system for executing evaluation of a puncture point for performing blood sampling is disclosed. The system obtains an image of a biological portion for performing blood sampling. In the image, a plurality of puncture candidate points of biological portions are determined, similarity between each puncture candidate point of the plurality of puncture candidate points and each point in the image is calculated, a feature map of each puncture candidate point indicating similarity between each puncture candidate point and each point is generated, and the puncture adaptation for each puncture candidate point is determined from each feature map of the puncture candidate points.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to evaluation of a puncture location for blood sampling. [Background technology]

[0002] Blood tests have become indispensable in the medical field as they provide basic information for diagnosing various diseases. In recent years, blood tests are also expected to be used for early diagnosis of disease symptoms in order to curb ever-increasing medical costs, and the demand for blood tests is expected to continue to increase in the future.

[0003] With this increase in demand, there are concerns about the increased burden on medical professionals who collect blood samples. Against this background, automatic blood collection devices that automatically collect blood samples without human intervention have been attracting attention in recent years, and there is a particular demand for small, space-saving automatic blood collection devices that can be placed without taking up space in the limited space available at medical facilities.

[0004] A known example of this type of automatic blood sampling device is the blood sampling device disclosed in Patent Document 1. When a person to be sampled places his / her fingertip at a specified position on the device, a needle automatically moves to puncture the fingertip, creating a small wound, and blood is sampled by collecting the blood in a collection container. This blood sampling device has been made smaller by using the fingertip as the part to be sampled.

[0005] Generally, there are some parts of a human finger that are suitable for blood sampling and some that are not. However, Patent Document 1 does not take into consideration the suitability of the blood sampling position, and only discloses that the blood sampling device mechanically punctures the puncture needle into a fixed position. Therefore, there is a possibility that a sufficient amount of blood cannot be sampled due to individual differences in the subject's finger, changes in the presented posture and position, etc.

[0006] On the other hand, one example of research aimed at stable automatic blood sampling is described in Non-Patent Document 1. Non-Patent Document 1 discloses a method of photographing the vascular pattern of a finger using a near-infrared camera, analyzing the photographed image, and extracting the branching points of the blood vessels as the puncture positions.

[0007] Even when medical professionals manually draw blood, they generally aim the blood collection needle at the vein in the arm, and since the branching point of blood vessels is where multiple blood vessels converge, it is expected that the blood flow rate is relatively high. Therefore, we aim to realize stable blood collection by enabling the blood collection needle to be controlled by a robot arm so that it can puncture any position on the finger, and by programming the blood collection needle to puncture this vascular branching point. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2019-88391 A [Non-patent literature]

[0009] [Non-Patent Document 1] Li, X., Lin, J., Pang, Y., Huang, L., Zhong, L., Li, Z., “Fingertip Blood Collection Point Localization Research Based on Infrared Finger Vein Image Segmentation”, IEEE Transactions on Instrumentation and Measurement, Vol.71, pp.1-12 (2021) Summary of the Invention [Problem to be solved by the invention]

[0010] However, due to individual differences in the optimum puncture position for blood collection, it is difficult to obtain a stable amount of blood. In order to automate blood collection, it is necessary to detect the features of the optimum puncture position for blood collection. However, since there are various variations in the features and the image is affected by fingerprints, it is not easy to establish rules for detecting the features.

[0011] The present invention has been made in consideration of the above points, and proposes a technique for evaluating a blood sampling position that enables stable blood sampling. [Means for solving the problem]

[0012] One aspect of the present invention is a system that performs evaluation of puncture points for blood collection, the system including one or more processors and one or more storage devices that store programs executed by the one or more processors, wherein the one or more processors acquire an image of a body part from which blood is to be collected, determine a plurality of candidate puncture points for the body part in the image, calculate a similarity between each of the plurality of candidate puncture points and each point in the image, generate a feature map for each candidate puncture point indicating the similarity between each candidate puncture point and each point, and determine a puncture suitability for each of the candidate puncture points from the feature maps of the candidate puncture points. Effect of the Invention

[0013] According to one aspect of the present invention, a puncture position suitable for blood sampling can be more appropriately evaluated. [Brief description of the drawings]

[0014] [Figure 1] 1 shows a configuration of an automatic blood sampling device according to an embodiment of the present specification. [Diagram 2] 2 shows an example of an input image captured by a camera. [Diagram 3] 2 shows the overall flow of an automatic blood sampling process executed by an automatic blood sampling device. [Figure 4] 13 shows a flowchart of a process for selecting an optimal puncture point using a learning model in one embodiment of this specification. [Figure 5A]13 shows an example of a feature map for selected candidate puncture points. [Figure 5B] 13 shows an example of a feature map for selected candidate puncture points. [Figure 5C] 13 shows an example of a feature map for selected candidate puncture points. [Figure 5D] 13 shows an example of a feature map for selected candidate puncture points. [Figure 6] FIG. 5 is a diagram for explaining the process of selecting an optimal puncture point using the learning model shown in FIG. [Figure 7] 13 shows a flowchart of an example process of generating a feature map of one candidate puncture point in the step of generating a feature map for each selected candidate puncture point from an input image. [Figure 8] 13 shows a flowchart of a learning example of a puncture suitability calculation model. [Figure 9] FIG. 9 is a schematic diagram for explaining learning of the puncture suitability calculation model described with reference to FIG. 8. [Figure 10] 13 is a flowchart showing a flow of learning of a feature map extraction model. [Figure 11] 13 shows an example of processing of a feature map extraction model during learning. [Figure 12] 3A and 3B are schematic diagrams illustrating display of a feature map by a display device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. It should be noted that this embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are used to refer to the same components in each drawing.

[0016] The processor realizes a predetermined function by executing a program stored in a primary storage device. The primary storage device stores the program executed by the processor and data necessary for executing the program. The program includes an OS (Operating System) (not shown) as well as other programs. The processor may include multiple chips and multiple packages.

[0017] A program is executed by a processor to perform a defined process using a storage device and a communication port (communication device). Therefore, a description that uses a program as the subject may also use a processor as the subject. Alternatively, the process executed by a program is the process performed by the computer and computer system on which the program runs.

[0018] A processor operates as a functional unit (means) that realizes a specified function by operating according to a program. For example, a processor functions as a unit (means) by operating according to a program, and functions as a unit (means) by operating according to a program. The same applies to other programs. Furthermore, a processor also operates as a functional unit (means) that realizes each of the multiple processes executed by each program. Computers and computer systems are devices and systems that include these functional units (means).

[0019] In one embodiment of the present specification, a feature amount that distinguishes blood vessels from a background is extracted, and puncture points (puncture positions) suitable for blood collection are evaluated or prioritized based on the feature amount. In one embodiment of the present specification, an image of the site from which blood is to be collected is acquired, and a feature map is generated that indicates the similarity between the image and a candidate puncture point. The device calculates a puncture suitability, which is the suitability for blood collection at the candidate puncture point, from the feature map. The puncture suitability indicates the expected amount of blood to be collected.

[0020] The device evaluates or prioritizes the puncture points by referring to the puncture suitability, which allows blood to be collected reliably regardless of individual differences such as the posture of the subject when presenting the body part, or the condition and shape of the body part.

[0021] Hereinafter, an automatic blood sampling device will be described which selects an optimal puncture point in a body part taking into consideration the puncture suitability and inserts a puncture needle into the selected puncture point to collect blood. Fig. 1 shows the configuration of an automatic blood sampling device 1 according to one embodiment of this specification. The automatic blood sampling device 1 is configured with a calculation unit 2, a light source control unit 3, an image input unit 4, a mechanism control unit 5, a communication unit 6, a camera 7, a near-infrared light source 8, a movement mechanism 9, and a blood sampling tube 10.

[0022] The calculation unit 2 is configured with a CPU (Central Processing Unit) 21, a memory 22, and an auxiliary storage device 23, which are interconnected via an internal bus 20, and interfaces 24 provided corresponding to the light source control unit 3, the image input unit 4, the mechanism control unit 5, and the communication unit 6. Note that the CPU 21, the memory 22, the auxiliary storage device 23, and each interface 24 may be interconnected by a dedicated bus for the purpose of increasing speed, etc.

[0023] The CPU 21 is a processor that controls the overall operation of the automatic blood sampling device 1. The memory 22 is composed of a RAM (Random Access Memory) and a ROM (Read Only Memory), and is used to permanently store programs required for processing and temporarily store data required for processing. The CPU 21 executes the programs stored in the memory 22, thereby executing various processes of the automatic blood sampling device 1 as a whole, as described below.

[0024] The auxiliary storage device 23 is composed of, for example, a flash memory, a hard disk drive, etc., and is used to store data that needs to be saved permanently. Data is input / output to / from the auxiliary storage device 23 via the memory 22 under the control of the CPU 21.

[0025] The interface 24 is a device having the function of connecting the calculation unit 2 and the corresponding functional blocks (light source control unit 3, image input unit 4, mechanism control unit 5 or communication unit 6) so that necessary data can be exchanged between the calculation unit 2 and the corresponding functional blocks.

[0026] With the presence of such an interface, the calculation unit 2 can control the light emission state of the near-infrared light source 8 via the light source control unit 3, input the output image of the camera 7 via the image input unit 4, and drive and control the moving mechanism 9 via the mechanism control unit 5. The calculation unit 2 can also communicate with an external device (not shown) connected to the network 11 via the communication unit 6. Such communication can, for example, share part or all of the processing associated with blood collection with an external device (for example, an external calculation device serving as a host), or link with various services on the cloud.

[0027] The camera 7 is composed of, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary metal-Oxide Semiconductor) image sensor, and captures the subcutaneous vascular pattern of the subject's biological part (hereinafter referred to as the finger) 12 from which blood will be collected, and outputs image data of the captured image to the image input unit 4.

[0028] Near-infrared light is passed through the living body and photographed by a camera 7 compatible with the near-infrared wavelength range. Blood vessels are shown as dark lines. This is because hemoglobin contained in blood has the property of absorbing near-infrared light, and when near-infrared light is passed through blood vessels containing a lot of hemoglobin, the light is weakened and the blood vessels are shown relatively darker than the surroundings. In this way, the near-infrared image can display the distribution of blood vessels or blood flowing through the blood vessels in the target area. Basically, the puncture suitability of each point on the blood vessel can be calculated by analyzing this photographed image. Note that light of a wavelength range other than near-infrared may be used if the same processing is possible.

[0029] The near-infrared light source 8 is composed of, for example, a near-infrared LED (Light Emitting Diode) and emits near-infrared light of an intensity according to the amount of driving power based on the driving power given from the light source control unit 3. The near-infrared light source 8 is fixedly disposed such that its optical axis coincides with the optical axis of the camera 7 at a distance that allows the blood sample recipient's finger 12 to be inserted, as shown in FIG. 1. The reason why the blood sample recipient's finger 12 is sandwiched between the near-infrared light source 8 and the camera 7 in this manner is that a human finger is thick enough for near-infrared light to pass through, and a blood vessel pattern with higher contrast can be obtained by passing near-infrared light through the side opposite the finger surface facing the camera 7 to capture the image.

[0030] Alternatively, a light source may be placed on the same side as the camera 7, and light may be irradiated onto the surface of the finger 12 to be photographed, and the vascular pattern may be photographed using the light that is reflected back. When blood is to be collected from a thick body part such as the arm, a sufficient amount of light may not be able to pass through and contrast in the vascular pattern may not be obtained, so the reflected light method is more suitable.

[0031] The moving mechanism 9 is a mechanism for moving the relative position of the puncture needle 13 with respect to the finger 12, including the direction in which the puncture needle 13 is inserted into the finger 12 presented in a predetermined position and in a predetermined state by the subject of blood collection, and the direction in which the puncture needle 13 is removed from the finger 12 after being inserted into the finger 12. The moving mechanism 9 includes a pushing mechanism 16 having the puncture needle 13 attached to its tip in a replaceable manner. The pushing mechanism 16 includes an actuator such as a motor (not shown), and the actuator can move the puncture needle 13.

[0032] The light source control unit 3 is a functional block having a function of controlling the brightness of the near-infrared light irradiated by the near-infrared light source 8 so that the blood vessel pattern becomes the clearest in the image captured by the camera 7, based on a target value specified by the program executed by the CPU 21. Specifically, the light source control unit 3 performs PWM (Pulse Width Modulation) control of the amount of driving power applied to the near-infrared light source 8 so that it becomes the target value specified by the program executed by the CPU 21.

[0033] The image input unit 4 is a functional block having a function of converting the above-mentioned image data provided from the camera 7 into a format that is easy for the calculation unit 2 to handle, and storing the converted image data in the memory 22. The mechanism control unit 5 controls the moving mechanism 9 according to instructions of the program executed by the CPU 21.

[0034] In the automatic blood sampling device 1 having the above configuration, the CPU 21 analyzes the image of the vascular pattern obtained as described above based on the program stored in the memory 22, and determines the coordinate value of the point on the image that is determined to be most suitable for puncturing (optimum puncture point).

[0035] FIG. 2 shows an example of an input image 30 captured by the camera 7. The input image 30 shows an image 31 of a finger 12. FIG. 2 further shows an example of a trajectory 32 of the puncture needle 13. The image 31 of the finger includes an image 33 of a plurality of blood vessels. In FIG. 2, an image of one of the plurality of blood vessels is indicated by reference numeral 33 as an example. The puncture needle 13 can only move on a predetermined trajectory 32. By restricting the degree of freedom of movement, the mechanism for moving the puncture needle 13 can be made smaller.

[0036] 2, the puncture needle 13 can move on a trajectory (rotational trajectory) 32 that forms a part of a circle. The CPU 21 selects an optimal puncture point from the trajectory 32 of the puncture needle 13. This reduces the processing load for selecting the optimal puncture point. Note that the puncture needle 13 may be movable to an arbitrary position.

[0037] 3 shows an overall flow of the automatic blood sampling process executed by the automatic blood sampling device 1. When the automatic blood sampling process starts, the CPU 21 executes initialization processing of the programs executed by the CPU 21 and the functional units 3 to 6 (S11). Next, the CPU 21 acquires an image of the finger 12 captured by the camera 7, which is input from the image input unit 4 (S12).

[0038] The CPU 21 analyzes the acquired image and determines whether a finger is included in the image (S13). A widely known image recognition technique can be used to detect a finger in the image, so details will be omitted. Note that a finger in the captured image may be detected by a method other than image recognition, for example, by pressing a mechanical switch.

[0039] If a finger is not detected in the input image (S13: NO), the flow returns to step S12. If a finger is detected in the input image (S13: YES), the CPU 21 refers to the input image and determines whether the amount of light from the near-infrared light source 8 is appropriate (S14). For example, the CPU 21 can determine whether the amount of light is appropriate based on the clarity of the blood vessel pattern in the input image.

[0040] If the light intensity of the near-infrared light source 8 is inappropriate (S14: NO), the CPU 21 uses the light source control unit 3 to adjust the light intensity of the near-infrared light source 8 (S15). The CPU 21 instructs the light source control unit 3 on the amount of driving power to be applied to the near-infrared light source 8 that corresponds to the changed light intensity. After that, the flow returns to step S12.

[0041] If the light amount of the near-infrared light source 8 is appropriate (S14: YES), the CPU 21 analyzes the input image using the learning model and selects the optimal puncture point (S16). The details of step S16 will be described later.

[0042] Next, CPU 21 controls moving mechanism 9 to collect blood at the optimal puncture point (S17). Specifically, CPU 21 notifies mechanism control unit 5 of the coordinate value of the optimal puncture point calculated in step S16. When mechanism control unit 5 is notified of the coordinate value of the optimal puncture point from CPU 21, it controls moving mechanism 9 to cause puncture needle 13 to puncture finger 12 presented by the subject of blood collection at that coordinate value.

[0043] When the puncture needle 13 punctures the subject's finger 12, the mechanism control unit 5 operates the moving mechanism 9 to move the puncture needle 13 in a direction to pull it out of the subject's finger 12, and then controls the moving mechanism 9 to move the blood collection tube 10 to a position directly below the site of the subject's finger 12 punctured by the puncture needle 13. As a result, blood bleeding from the subject's finger 12 is received by the blood collection tube 10, and when a specified volume of blood is collected, blood collection is completed. The structure of the moving mechanism 9 and the method of collecting blood into the blood collection tube 10 are arbitrary.

[0044] In one embodiment of the present specification, an optimal puncture point is selected from an input image of a finger using an optimal puncture point selection model, which is a machine learning model. Fig. 4 shows a flowchart of the optimal puncture point selection process S16 using the learning model in one embodiment of the present specification.

[0045] The CPU 21 acquires an image of the fingertip from which blood is to be collected (S21). Two trained learning models are used to select the puncture point. The first learning model extracts a feature map corresponding to all pixels in the image (S22). As will be described in more detail later, the feature map is obtained by self-learning without the need for labels.

[0046] Next, the CPU 21 applies the second learning model to the feature map at each puncture candidate point (S23). The CPU 21 calculates the puncture suitability from the distribution of the entire image obtained from the feature map, and determines whether the puncture is suitable (S24). The second learning model has been learned (trained) by learning data. The puncture suitability is learned and estimated using the distribution of the feature map of the entire image and training data of the blood collection amount. In the conventional method, attention was paid to the local features of the puncture candidate point of the pixel corresponding to the trajectory 32 of the needle 13, whereas in this embodiment, the puncture suitability is determined by utilizing the features of the entire image. The input of the second learning model is the feature map of one puncture candidate point, and its output is the puncture point suitability. The learned learning model is stored in, for example, the auxiliary storage device 23. The learning model can be generated using, for example, a deep neural network, and as an example, a CNN (Convolutional Neural Network) or a Residual Network can be used.

[0047] The puncture suitability indicates the amount of blood drawn at the puncture candidate point, and the higher the suitability, the larger the amount of blood drawn. The learning data for the learning model are various combinations of feature maps of the puncture candidate points and values ​​obtained from the blood amounts drawn at the puncture candidate points.

[0048] CPU21 determines whether the finger in the image is suitable for blood sampling based on the puncturing suitability of the multiple puncturing candidate points, and if it determines that the finger is suitable for blood sampling, determines the puncturing point where blood will actually be sampled from the multiple puncturing candidate points (S25).

[0049] There are cases where a puncture point determined to be appropriate cannot be obtained from the state of the finger obtained from the image. For example, if the puncture suitability of all puncture candidate points is smaller than a predetermined threshold, the finger is determined to be unsuitable for blood collection. As another example, if the average value of the puncture suitability of multiple puncture candidate points is lower than a predetermined threshold empirically determined in advance, the finger is determined to be unsuitable. This makes it possible to more reliably collect appropriate blood.

[0050] After the finger in the image is determined to be suitable for blood collection, the CPU 21 selects the optimal puncturing point from the puncturing candidate points. For example, the puncturing candidate point with the highest puncturing suitability or the point closest to the center of the finger, which is a preset condition for the position on the finger under the condition that the puncturing suitability is higher than a predetermined threshold, can be selected. In this way, the puncturing suitability is calculated from the feature map, and the puncturing positions suitable for blood collection are evaluated and prioritized.

[0051] 5A to 5D show examples of feature maps 40A to 40D for selected candidate puncture points. In FIGS. 5A to 5D, fingertip 41 is indicated by a dashed line. Also, crosses 45A to 45D respectively indicate selected candidate puncture points (pixels). The feature maps indicate points (regions) that have similar features to the selected candidate puncture point. The feature amount of each point indicates features such as the texture, brightness, and spatial structure of the point. These features can be calculated from the brightness value of the pixel of the point and the brightness values ​​of the surrounding pixels.

[0052] 5A to 5D show regions similar to the selected puncturing candidate point 45A as black regions. FIG. 5A shows the selected puncturing candidate point 45 and a region 461 similar to that point. FIG. 5B shows the selected puncturing candidate point 45B and a region 461B similar to that point. FIG. 5C shows the selected puncturing candidate point 45C and a region 461C similar to that point. FIG. 5D shows the selected puncturing candidate point 45D and regions 461D, 462D similar to that point. In the examples of FIG. 5A to 5D, regions with a high similarity to the puncturing candidate point are automatically visualized.

[0053] The feature map indicates an area having texture, brightness, and spatial structure similar to the selected point. The feature map is generated from a near-infrared image of the fingertip. As described above, a near-infrared image displays blood vessels as dark areas compared to other parts (background). Therefore, when the selected puncture candidate point is in the blood vessel area, the feature map mainly indicates some of the points in the blood vessel area as points similar to the selected puncture candidate point. On the other hand, when the selected puncture candidate point is in the background outside the blood vessel area, the feature map mainly indicates some of the points in the background as points similar to the selected puncture candidate point. In this way, the feature map indicates feature amounts related to the blood vessel and background at the selected puncture candidate point extracted from the input image.

[0054] For example, Figures 5A and 5D show selected puncture candidate points 45A and 45D as examples of points in a blood vessel region, and further show a point group in the blood vessel region similar to these, while Figures 5B and 5C show selected puncture candidate points 45A and 45D as examples of points in the background, and further show a point group in the background similar to these.

[0055] The feature map can be generated using a learning model. It is difficult to describe the feature quantities of the appropriate puncture point (position) for blood sampling as rules, and the rules must be adjusted when applying to new data in a horizontal deployment. In this way, by generating a feature map using a learning model and calculating the puncture suitability from the feature map, it becomes possible to automatically select a more appropriate puncture point.

[0056] Fig. 6 is a diagram for explaining the process of selecting the optimal puncture point using the learning model shown in Fig. 4. A near-infrared image of the fingertip of a blood sampling target is input to calculation unit 2 as input image 30 (S21). CPU 21 extracts a feature map of each of the selected puncture candidate points from input image 30 (S22).

[0057] Next, CPU21 applies learning model 211 to each feature map to analyze the spatial pattern and calculate the puncture suitability of each selected puncture candidate point. Learning model 211 outputs the puncture suitability for the input feature map. CPU21 determines whether each selected puncture candidate point is reliable as a puncture point based on the calculated puncture suitability (S23, S24). In the example shown in Fig. 6, feature map 40B is a feature map of puncture candidate points that have a low probability of being optimal points, and feature map 40D is a feature map of puncture candidate points that have a high probability of being optimal points.

[0058] CPU21 determines whether the finger in input image 30 is suitable for blood sampling based on the puncturing suitability of the selected puncturing candidate point, and performs blood sampling at one puncturing candidate point if it is suitable (S25). For example, if the highest puncturing suitability exceeds a predetermined threshold, the finger may be determined to be suitable for blood sampling. Also, the puncturing candidate point showing, for example, the highest puncturing suitability may be selected as the puncturing point for blood sampling.

[0059] In one embodiment of the present specification, a feature map of selected candidate puncture points is generated from an input image using a learning model. The learning model has been trained using training data and is stored, for example, in the auxiliary storage device 23. Fig. 7 shows a flowchart of a processing example for generating a feature map of one candidate puncture point in step S22 for generating a feature map of each of the selected candidate puncture points from an input image.

[0060] CPU 21 acquires an input image for which a feature map is to be extracted (S31). As described above, an example of the input image is a near-infrared image of a fingertip. Next, CPU 21 applies a learning model to the acquired image (S32) and extracts a feature map for the puncturing candidate point (S33). The input of the learning model is the input image and the coordinates in the image of the selected puncturing candidate point, and the output of the learning model is the feature map extracted from the input image for the puncturing candidate point. The configuration of the learning model that generates the feature map and the learning method thereof will be described in detail later.

[0061] An example of a learning method (training method) of a learning model used to determine a puncturing point for blood sampling will be described below. As described above, the automatic blood sampling device 1 of one example of this specification uses two learning models. One is a feature map extraction model that generates a feature map at a puncturing candidate point of an input image. The other is a puncturing suitability calculation model 211 that calculates the puncturing suitability from the feature map.

[0062] First, an example of learning of puncturing suitability calculation model 211 will be described. Fig. 8 shows a flowchart of an example of learning of puncturing suitability calculation model 211. First, CPU 21 acquires a large number of near-infrared images of fingertips to be learning samples (S41). The near-infrared images may be newly generated by automatic blood sampling device 1, or may be prepared in advance in any of the databases.

[0063] Next, CPU 21 extracts a feature map for each point of each near-infrared image (S42). Specifically, CPU 21 sequentially selects near-infrared images from the learning sample, and executes the following process for each selected near-infrared image. CPU 21 inputs the selected near-infrared image and the coordinates of a pixel selected from the near-infrared image to a trained feature map extraction model. The feature map extraction model outputs a feature map according to the input. This results in a feature map for the pixels of the selected near-infrared image. CPU 21 acquires a feature map for all pixels of the selected near-infrared image, or for pixels at preset coordinates.

[0064] Next, CPU 21 accumulates combinations of the feature maps and the blood collection volumes corresponding to the feature maps in learning data (training data) (S43). The blood collection volumes for the feature maps can be input by a user via communication unit 6 or obtained from a database (not shown). The learning data is stored in, for example, auxiliary storage device 23. The learning data includes many combinations of the feature maps and the blood collection volumes corresponding to the feature maps.

[0065] Next, CPU 21 uses the learning data to train puncturing suitability calculation model 211, that is, to update the learning parameters (S44). The parameters can be updated by backpropagating the error between the calculated puncturing suitability and the correct puncturing suitability. The puncturing suitability output by puncturing suitability calculation model 211 may be the blood collection amount itself or a value calculated from the blood collection amount by a predetermined method. A relationship is satisfied in which the puncturing suitability increases as the blood collection amount increases. As a result, the puncturing suitability can numerically indicate how much blood can be expected to be collected. Through this learning, it is possible to learn the correlation between the image features around a puncturing point where a large amount of blood is collected and the amount of blood collected.

[0066] When the learning of puncturing suitability calculation model 211 is completed, CPU 21 stores the learned puncturing suitability calculation model 211 in auxiliary storage device 23 (S45). Note that the learning of puncturing suitability calculation model 211 may be executed by a computer different from automatic blood sampling device 1, and the learned puncturing suitability calculation model 211 may be stored in auxiliary storage device 23.

[0067] FIG. 9 is a schematic diagram for explaining the learning of the puncturing suitability calculation model 211 described with reference to FIG. 8. As described above, many different near-infrared images 30 of fingertips are prepared as learning samples. Training data 35 includes feature maps for each pixel (point) of each near-infrared image 30, and further includes the collected blood amount corresponding to each feature map. The collected blood amount corresponding to the feature map is the collected blood amount with the pixel in the finger of the image from which the feature map was extracted as the puncturing point. Note that the puncturing suitability may be calculated using a rule-based model different from the machine learning model.

[0068] As described above, one example of the present specification obtains a quantitative numerical index according to the expected blood collection amount, and selects a suitable point for puncturing based on the relative level of the index. This allows the selection of the next best position for puncturing even if the optimal position is not on the trajectory, and a significant decrease in the amount of collected blood can be avoided, thereby improving the stability of blood collection even with a small blood collection device.

[0069] Next, a method for learning the feature map extraction model will be described. In one embodiment of this specification, learning data (training data) for the feature map extraction model is composed of only input images. In other words, the user does not need to prepare a feature map corresponding to the input image as correct answer data. This allows the feature map extraction model to be learned efficiently.

[0070] 10 is a flowchart showing the flow of learning the feature map extraction model. The CPU 21 acquires many near-infrared images of different fingertips that become learning samples for the feature map extraction model (S51), and constructs a learning data set from the near-infrared images (S52). The near-infrared images used for learning may be newly generated by the automatic blood sampling device 1, or may be stored in advance in an external or internal database.

[0071] Next, CPU 21 trains the feature map extraction model using the constructed training data set (S53). The training of the feature map extraction model will be described in detail later. When training of the feature map extraction model is completed, CPU 21 stores the trained feature map extraction model in auxiliary storage device 23 (S54). Note that training of the feature map extraction model may be executed by a computer different from automatic blood sampling device 1, and the trained feature map extraction model may be stored in auxiliary storage device 23.

[0072] An example of a learning method for the feature map extraction model is described below. FIG. 11 shows an example of the processing of the feature map extraction model 215 during learning. FIG. 11 shows the processing of the feature map extraction model 215 during learning. All of the configuration processing is necessary during learning, but the purpose of the automatic blood sampling processing is feature amount map extraction 535 using a trained model. Therefore, some of the configuration processing shown in FIG. 11 is not necessary, and some processing not shown in FIG. 11 may be executed.

[0073] The feature map extraction model 215 includes an Encoder-Decoder network 510. The Encoder-Decoder network 510 includes a plurality of blocks 511-517. The blocks 511-516 have a common configuration. Specifically, each block is composed of padding, 3×3 convolution, Relu, padding, 3×3 convolution, and Relu. The block 517 includes a self-attention mechanism 530 in addition to the configuration common to the other blocks. For example, the first half blocks 511-514 are encoding processes, and the second half blocks 515-517 are decoding processes. Note that the number of blocks is an example, and the processing of some blocks may be different.

[0074] A near-infrared image 30 is input to an input layer of a block 511 of an encoder-decoder network 510. The near-infrared image 30 is composed of H pixels in the height direction and W pixels in the width direction, and is one channel of data. In a block 511, the encoder-decoder network 510 executes the following processes: padding, 3×3 convolution, Relu, padding, 3×3 convolution, and Relu. Then, Max pooling is applied to the features from the block 511 to reduce the resolution.

[0075] Similarly, the Encoder-Decoder network 510 applies block 512, Max pooling, block 513, and Max pooling to the features in that order. The Encoder-Decoder network 510 applies block 514 to the features, and then applies bilinear upsampling to increase the resolution.

[0076] The Encoder-Decoder network 510 combines the feature with the increased resolution with the feature obtained from block 513 and applies block 515. The Encoder-Decoder network 510 applies bilinear upsampling to the feature obtained from block 515 via a skip connection to increase the resolution, and applies block 516 by combining it with the feature obtained from block 512 via a skip connection.

[0077] The Encoder-Decoder network 510 applies bilinear upsampling to the feature obtained from block 516 to increase the resolution, and applies it to block 517 together with the feature obtained from block 511 by the skip connection. Block 517 performs the same processing as the other blocks. As a result, the resolution of the feature is restored to the original resolution of the input image. In other words, the feature has a resolution of height H and width W. Note that the number of channels is different from 1 of the input image.

[0078] The Encoder-Decoder network 510 executes the processing of the self-attention mechanism 530 on features having the same resolution (H×W) as the input image 30. Finally, 1×1 convolution is applied to the output from the self-attention mechanism 530 to restore the number of channels to the same number of channels (1) as the input image 30, and output it as the output image.

[0079] An example of a loss function used to update the parameters of the feature map extraction model 215 is SSIM (Structural Similarity Index Measure). SSIM is a function suitable for evaluating the structural similarity between two images. The parameters of the feature map extraction model 215 are updated so as to reduce the deviation of the spatial patterns between the input image 30 and the output image. Note that a loss function other than SSIM can be used.

[0080] The processing of the self-attention mechanism 530 (self-attention processing) will be described below. The self-attention mechanism 530 learns the relationship between pixels and autonomously directs attention to important features. The self-attention mechanism 530 searches for similar regions for each point with the same resolution as the input image. As shown in FIG. 11, the feature input to the self-attention mechanism 530 is a three-dimensional tensor 531 having height H, width W, and number of channels C. The height H and width W match the height and width resolution (number of pixels) of the input image.

[0081] The self-attention mechanism 530 converts the three-dimensional tensor 531 into a two-dimensional matrix 532. The number of columns of the two-dimensional matrix 532 is C, and the number of rows is H×W. Each row of the two-dimensional matrix 532 indicates the feature amount of each pixel. Through the above processing, the feature amount of a pixel includes information on the brightness of the pixel and the surrounding pixels, and indicates the characteristics of the texture, brightness, and spatial structure. In this way, the feature amount related to the blood vessel and the background at each point is extracted.

[0082] Next, the self-attention mechanism 530 applies Relu and regularization to each row (each pixel) of the matrix 532 to calculate the matrix 533. As a result, only important features are extracted and normalized for each pixel.

[0083] Next, the self-attention mechanism 530 calculates a similarity matrix 535 by the inner product of the matrix 533 and its transpose matrix 534. Each element of the similarity matrix 535 indicates the similarity between two pixels. Each row of the similarity matrix 535 indicates the similarity between each pixel and each pixel in the input image. Therefore, each row corresponds to a similarity map of each pixel. The similarity map can separate areas with similar features, that is, blood vessel areas and the background. As will be described later, a feature map at a selected point is generated from the similarity matrix 535, which is the output of the intermediate layer (hidden layer).

[0084] The self-attention mechanism 530 calculates a matrix 536 from the inner product of the similarity matrix 535 and the matrix 533, and converts the matrix 536 into a three-dimensional tensor 537. As described above, the Encoder-Decoder network 510 applies 1×1 convolution to the three-dimensional tensor 537 to restore the number of channels to the same number of channels (1) as the input image 30, and outputs it as an output image from the output layer of the block 517.

[0085] As described above, the self-attention mechanism 530 extracts and normalizes only the important parts of the features for each pixel, and then generates a similarity matrix using matrix transformation and inner products. Each row of the similarity matrix corresponds to a similarity map for each point and is used to update the feature map. The feature map is trained to separate regions with similar features. This allows the feature map to be updated so that blood vessels and background can be separated in the input image.

[0086] Similarity matrix 535 indicates the similarity of each combination of pixel feature amounts. Therefore, a feature map of each candidate puncture point can be generated from similarity matrix 535. That is, CPU 21 executing feature map extraction model 215 extracts the similarity between the designated candidate puncture point and each point from similarity matrix 535, and generates a feature map of the candidate puncture point.

[0087] As described above, the feature map extraction model 215 learns to separate the features of blood vessels and the background, and can automatically extract areas with the same structure as blood vessels. Images of blood vessels captured using near-infrared light are generally unclear, and unevenness in the ease of light transmission is likely to occur not only due to blood but also due to differences in tissue composition in the living body. When calculating the puncture suitability at an arbitrary point, it may not be possible to accurately distinguish whether there is actually a lot of blood or whether such a feature just occurs by chance due to unevenness, based only on local features such as low brightness at that point. As described above, one embodiment of the present specification can calculate the puncture suitability with reduced influence of local mutations by reflecting not only the local features of a single point to be punctured, but also the distribution state of blood in a wider range including the vicinity of the point.

[0088] In addition, the feature map extraction model 215 performs learning (updating of parameters) based on the error between the input image and the output image. In the conventional method, the input near-infrared image and the corresponding blood vessel pattern are generally prepared as correct labels, and the network parameters are generally trained to capture the features of the blood vessel pattern. However, manual labeling is difficult in terms of time and cost. In the feature map extraction model 215 of this embodiment, the same near-infrared image is treated as an input and output pair, and self-learning is performed because correct labels are not required. The parameters are updated by calculating the error between the output image of the network and the near-infrared image that is the input image using SSIM. The feature values ​​that automatically emphasize the blood vessels and background structures are extracted by parameter optimization and attention mechanism. As described above, one embodiment of this specification makes it possible to autonomously learn the feature values ​​related to the fingertip from the input data without human intervention, and to select an appropriate puncture position.

[0089] Note that, if it is possible to calculate the similarity between each point of an image and other points, a feature map of candidate puncture points may be generated by a model or process different from the process using the feature map extraction model. For example, a learning model that updates parameters by learning using the feature map as a correct answer label may be used.

[0090] Next, the image display of the feature map will be described. The automatic blood sampling device 1 according to one embodiment of the present specification displays an image of the feature map in addition to or instead of automatic blood sampling at an appropriate puncturing point. The user can refer to the displayed feature map to determine whether or not blood should be sampled at the corresponding puncturing candidate point, or can refer to the displayed feature map after blood sampling to confirm whether or not the appropriate puncturing point has been selected.

[0091] Fig. 12 shows a schematic diagram of a feature map displayed by a display device. As described with reference to Fig. 1, near-infrared light is irradiated onto finger 12 from near-infrared light source 8, and the transmitted light is detected by camera 7. From the near-infrared image generated by camera 7, a feature map for each of the candidate puncturing points is generated, as described above.

[0092] CPU 21 displays a feature map of the candidate puncturing points on display device 51 in response to a request from a user. Display device 51 may be, for example, a display device of a user terminal connected to calculation unit 2 via network 11 or a display device connected to the calculation unit via interface 24. The feature map to be displayed may be, for example, a feature map of a puncturing point where blood was actually collected or a feature map of a candidate puncturing point selected by the user. The feature map may be displayed alone or together with a captured image of the finger. Furthermore, the corresponding puncturing suitability may be displayed together with the feature map.

[0093] For example, when selecting a candidate puncturing point before blood sampling, CPU 21 displays a captured image of the finger and a feature map on display device 51. The user compares the feature maps of a plurality of candidate puncturing points and selects a puncturing point that is located in a sufficiently long and thick blood vessel. In response to the user's selection, CPU 21 performs blood sampling at the designated candidate puncturing point.

[0094] The user can visually check the spatial change of the feature amount when selecting the puncture candidate point in real time, which enables the selection of the puncture position to be improved (optimized), and the amount of blood collected can be expected to increase. In addition, by visualizing the feature map of the puncture point, it becomes possible to explain the cause in the unlikely event that the amount of blood collected is low.

[0095] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0096] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0097] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0098] 1 Automatic blood sampling device, 2 Calculation unit, 3 Light source control unit, 4 Image input unit, 5 Mechanism control unit, 6 Communication unit, 7 Camera, 8 Near-infrared light source, 9 Movement mechanism, 10 Blood sampling tube, 13 Insertion needle, 21 CPU, 22 Memory, 23 Auxiliary storage device, 31 Finger image, 32 Needle guide trajectory, 33 Blood vessel image, 211 Puncture suitability calculation model, 215 Feature map extraction model

Claims

1. 1. A system for performing an evaluation of a puncture point for blood collection, comprising: one or more processors; one or more storage devices for storing programs executed by the one or more processors; The one or more processors: Acquire an image of the body part from which blood is to be taken; determining a plurality of candidate puncture points at the body part in the image; calculating a similarity between each of the plurality of puncture candidate points and each point in the image, and generating a feature map for each puncture candidate point indicating the similarity between each puncture candidate point and each point; The system determines a puncture suitability for each of the candidate puncture points from the feature maps of the respective candidate puncture points.

2. 2. The system of claim 1, The one or more storage devices include A trained feature map extraction model that generates the feature map from the image; A trained puncture suitability calculation model that calculates the puncture suitability from the feature map is stored; the feature map extraction model outputs, for an input image and a candidate puncturing point, a feature map indicating a similarity between the candidate puncturing point and each point of the image; The puncture suitability calculation model calculates the puncture suitability from an input feature map.

3. 3. The system of claim 2, The feature map extraction model has been trained using a training data set; the training data set is composed of a plurality of input images to be input to the feature map extraction model; In the learning process using the learning data set, the feature map extraction model outputs an output image from an output layer, and generates a similarity matrix indicating a similarity between points of the input image in an intermediate layer; a feature map of the puncture candidate points is generated from the similarity matrix; The system, wherein the feature map extraction model is trained as a function of an error between the output image and the input image.

4. 3. The system of claim 2, The feature map extraction model includes an Encoder-Decoder network including a plurality of blocks performing preset image processing; One of the blocks includes a self-attention mechanism; The self-attention mechanism generates a similarity matrix indicating a similarity between points of the input image from features having the same resolution as the resolution of the image; The feature map of the candidate puncture points is generated from the similarity matrix.

5. 5. The system of claim 4, The feature map extraction model has been trained using a training data set; the training data set is composed of a plurality of input images to be input to the feature map extraction model; In training with the training data set, the Encoder-Decoder network outputs an output image; The system, wherein parameters of the Encoder-Decoder network are updated according to an error between the output image and the input image.

6. 3. The system of claim 2, The puncture suitability calculation model has been trained using a training data set; The system, wherein the learning data set includes multiple combinations of feature maps of puncture points input to the puncture suitability calculation model and correct labels of the puncture suitability indicating the amount of blood drawn at the puncture point.

7. 2. The system of claim 1, The one or more processors display the feature map on a display device.

8. 1. A method for performing an assessment of a puncture point for blood collection, comprising: The system further comprises: Acquire an image of the body part from which blood is to be taken; determining a plurality of candidate puncture points at the body part in the image; calculating a similarity between each of the plurality of puncture candidate points and each point in the image, and generating a feature map for each puncture candidate point indicating the similarity between each puncture candidate point and each point; determining a puncture suitability for each of the candidate puncture points from the feature map of each of the candidate puncture points.

9. 9. The method of claim 8, The system comprises: A trained feature map extraction model that generates the feature map from the image; A trained puncture suitability calculation model that calculates the puncture suitability from the feature map is stored; the feature map extraction model outputs, for an input image and a candidate puncturing point, a feature map indicating a similarity between the candidate puncturing point and each point of the image; The puncture suitability calculation model calculates the puncture suitability from an input feature map.

10. 10. The method of claim 9, The feature map extraction model has been trained using a training data set; the training data set is composed of a plurality of input images to be input to the feature map extraction model; In the learning process using the learning data set, the feature map extraction model outputs an output image from an output layer, and generates a similarity matrix indicating a similarity between points of the input image in an intermediate layer; a feature map of the puncture candidate points is generated from the similarity matrix; The method of claim 1, wherein the feature map extraction model is trained as a function of an error between the output image and the input image.

Images