Program, information processing method, information processing device, and information processing system

Abstract

Provided are a program and the like capable of assisting in puncture work. In accordance with this program, a computer detects a viewpoint position of a user with respect to a living body and identifies a target puncture position with respect to the living body on the basis of the viewpoint position of the user. The computer also detects the position of the tip of a puncture instrument with respect to the living body, and calculates the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

Description

Program, information processing method, information processing device, and information processing system

[0001] The present disclosure relates to a program, an information processing method, an information processing device, and an information processing system.

[0002] Patent Literature 1 discloses a technology that presents a recommended surgical strategy based on the usage status of medical devices during surgery, the condition of the affected area of ​​the patient, etc. By using the technology disclosed in Patent Literature 1, it is possible to support medical professionals such as doctors who perform medical procedures during surgery.

[0003] Patent No. 7219227

[0004] In medical settings, puncture procedures are performed on subjects for blood sampling, injections, transfusions, placement of intravascular catheters, and the like. In puncture procedures, medical professionals insert a puncture instrument, such as a needle or catheter, from the surface of the subject's body to a target location in a blood vessel. Because the skill level of a puncture procedure depends on the experience and knowledge of the medical professional, there is a need for technology to support the performance of appropriate puncture procedures. The technology disclosed in Patent Document 1 can present recommended surgical strategies, but does not support the puncture procedure.

[0005] In one aspect, an object is to provide a program or the like that can assist in the puncture operation.

[0006] (1) The present disclosure is a program that causes a computer to execute a process of detecting a user's viewpoint position relative to a living body, identifying a target puncture position relative to the living body based on the user's viewpoint position, detecting the position of a tip of a puncture instrument relative to the living body, and calculating the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

[0007] (2) It is preferable that the program of (1) above further causes the computer to execute a process of outputting an alert in accordance with the calculated amount of deviation.

[0008] (3) It is preferable that the program of (1) or (2) above further causes the computer to execute a process of detecting the user's palpation action on the living body and identifying the target puncture position based on the user's viewpoint position during the palpation action.

[0009] (4) It is preferable that the program described in any one of (1) to (3) above further causes the computer to execute a process of detecting the operation of the puncture device and, when it detects that the operation of the puncture device has stopped, detecting the position of the tip of the puncture device relative to the living body.

[0010] (5) It is preferable that the program described in any one of (1) to (4) above further causes the computer to execute a process of acquiring a blood vessel visualization image that visualizes the blood vessels of the living body, calculating a second deviation amount between the blood vessels in the blood vessel visualization image and the target puncture position, and outputting an alert in accordance with the second deviation amount.

[0011] (6) In the program described in any one of (1) to (5) above, it is preferable that the puncture device has a marker provided at a predetermined position, and the program further causes the computer to execute a process of detecting the position of the marker on the puncture device and detecting the position of the tip of the puncture device based on the position of the marker.

[0012] (7) It is preferable that the program described in any one of (1) to (6) above further causes the computer to input the user's biometric information into a learning model that has been trained to output a threshold value used to determine whether to output an alert regarding the amount of deviation between the target puncture position and the position of the tip of the puncture instrument when biometric information about the living body is input, obtain from the learning model the threshold value of the amount of deviation used to determine whether to output an alert, and output an alert if the amount of deviation between the target puncture position and the position of the tip of the puncture instrument is equal to or greater than the threshold value of the amount of deviation.

[0013] (8) It is preferable that the program described in any one of (1) to (7) above further causes the computer to execute a process of detecting the user's palpation position relative to the living body, calculating a third deviation amount between the palpation position and the target puncture position, and outputting an alert if the third deviation amount is equal to or greater than a predetermined value.

[0014] (9) The present disclosure also provides an information processing method in which a computer executes a process of detecting a user's viewpoint position relative to a living body, identifying a target puncture position relative to the living body based on the user's viewpoint position, detecting the position of a tip of a puncture instrument relative to the living body, and calculating the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

[0015] (10) The present disclosure also provides an information processing device that includes a control unit, which detects a user's viewpoint position relative to a living organism, identifies a target puncture position relative to the living organism based on the user's viewpoint position, detects the position of a tip of a puncture instrument relative to the living organism, and calculates the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

[0016] (11) The present disclosure also provides an information processing system including an information processing device having a control unit, a first camera that captures an image of a user's eyes, and a second camera that captures an image of the user's line of sight, wherein the control unit detects the user's viewpoint position relative to a living organism based on an image captured by the first camera and an image captured by the second camera, identifies a target puncture position relative to the living organism based on the user's viewpoint position, detects the position of a tip of a puncture instrument relative to the living organism based on the image captured by the second camera, and calculates the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

[0017] In one aspect, the puncture operation can be assisted.

[0018] FIG. 1 is an explanatory diagram showing an example of the configuration of a puncture support system. FIG. 2 is a block diagram showing example configurations of an information processing device and a wearable device. FIG. 3 is an explanatory diagram showing an example of the configuration of an indwelling needle. FIG. 4 is a flowchart showing an example of a puncture support processing procedure. FIG. 5 is an explanatory diagram of the puncture support processing. FIG. 6 is an explanatory diagram of the puncture support processing. FIG. 7 is an explanatory diagram of the puncture support processing. FIG. 8 is an explanatory diagram of the puncture support processing. FIG. 9 is an explanatory diagram of the puncture support processing. FIG. 10 is an explanatory diagram of the puncture support processing. FIG. 11 is an explanatory diagram of the puncture support processing. FIG. 12 is an explanatory diagram of the puncture support processing.

[0019] The program, information processing method, information processing device, and information processing system of the present disclosure will be described in detail below with reference to drawings illustrating embodiments thereof. In each of the following embodiments, a puncture operation will be described using the forearm of a subject, such as a patient or examinee, as an example of a puncture target site. However, the puncture target site is not limited to the forearm and may be any other suitable site, such as the arm or lower leg. Furthermore, the purpose of the puncture operation may be any purpose, such as blood collection, injection, infusion, or placement of an intravascular catheter.

[0020] (Embodiment 1) A puncture support system that supports a medical professional in performing a puncture operation will be described. FIG. 1 is an explanatory diagram showing an example of the configuration of the puncture support system. The puncture support system 100 of this embodiment includes an information processing device 10, a wearable device 20, and an indwelling needle 30. The puncture support system 100 of this embodiment is installed in a facility such as a medical institution or testing institution where the puncture operation is performed. The indwelling needle 30 is an example of a puncture device. The indwelling needle 30 is, for example, an indwelling needle for a peripheral artery or vein, and a medical professional uses the indwelling needle 30 to puncture the blood vessel of a living body 40 (subject) to be punctured. The indwelling needle 30 may be any medical device that requires puncturing a blood vessel of the living body 40, and may be an indwelling needle for a peripheral artery or vein, a dialysis indwelling needle, a vascular access device such as a PICC (Peripherally Inserted Central venous Catheter), a midline catheter, or a CVC (Central Venous Catheter), or a blood collection and drug administration device composed of a puncture needle and a suction device such as a syringe.

[0021] The information processing device 10 and the wearable device 20 are configured to perform wireless communication. The information processing device 10 is a device capable of various information processing and information transmission and reception, and is configured, for example, as a personal computer, a server computer, or the like. The wearable device 20 is worn by a surgeon (medical professional) performing a puncture procedure. The wearable device 20 is formed, for example, in the shape of glasses and worn on the head (face) of the user (surgeon). The wearable device 20 shown in FIG. 1 has a frame formed by temples 20a (temples) and nose pads 20b (pads) for wearing the device on the user's face, and pupil cameras 21a and 21b and a scenery camera 22 are provided at appropriate locations on the frame. The pupil cameras 21a and 21b are cameras that capture images of the left and right eyes (the eye region including at least the pupil (pupil), iris, and cornea) of the user wearing the wearable device 20, and at least one pupil camera is provided for each eye (eyeball). The pupil cameras 21a and 21b may be configured to capture images of both the left and right eyes (eyeballs) with a single camera. The scenery camera 22 is a camera that captures the scenery viewed by the user (the scenery in the direction of the surgeon's line of sight), and at least one such camera is provided. The wearable device 20 is not limited to the eyeglass-type configuration shown in FIG. 1 , but may be formed into a headset type, or may be configured to be worn by the user by attaching it to the temples of the user's eyeglasses or the brim of a hat. The information processing device 10 and the wearable device 20 are preferably configured to perform wireless communication, but may also be configured to perform wired communication via a cable such as a USB (Universal Serial Bus) cable. The information processing device 10 and the wearable device 20 may also be configured to communicate via a network, which may be the Internet, a public communication line, or a LAN (Local Area Network) established within the facility where the puncture assistance system 100 is installed.

[0022] FIG. 2 is a block diagram showing an example configuration of an information processing device 10 and a wearable device 20. The wearable device 20 includes pupil cameras 21a and 21b (first cameras), a scenery camera 22 (second camera), and a communication unit 23. The pupil cameras 21a and 21b are, for example, near-infrared cameras, and include a near-infrared light-emitting diode (LED) that emits near-infrared light having a wavelength ranging from approximately 750 nm to 2500 nm toward the user's eye, and a light-receiving unit that receives the near-infrared light emitted by the near-infrared LED reflected from the user's eye (cornea). Thus, the pupil cameras 21a and 21b acquire pupil images using near-infrared light. The scenery camera 22 is, for example, a visible light camera, and is an imaging device including a lens and an image sensor. Thus, the scenery camera 22 acquires scenery images using visible light. Note that the scenery camera 22 may be a near-infrared camera, or may be configured to include both a visible light camera and a near-infrared camera. The pupil cameras 21a, 21b and the scenery camera 22 capture, for example, 15 or 30 video images (pupil images, scenery images) per second, and sequentially transmit the captured pupil images and scenery images from the communication unit 23 to the information processing device 10.

[0023] The communication unit 23 is a communication module for transmitting pupil images captured by the pupil cameras 21a and 21b and scenery images captured by the scenery camera 22 to the information processing device 10 via wireless communication. The communication unit 23 transmits pupil images and scenery images via wireless communication compliant with, for example, IEEE 802.15.1, i.e., Bluetooth (registered trademark). The communication unit 23 may be a communication module for wired communication. The communication unit 23 may also be a communication module for connecting to a network, in which case the pupil images and scenery images are transmitted to the information processing device 10 via the network. In addition to the above-described configuration, the wearable device 20 may also be configured to include an input unit that accepts operation inputs from a user. The wearable device 20 may also include a storage unit. In this case, the pupil images captured by the pupil cameras 21a and 21b and the scenery images captured by the scenery camera 22 may be temporarily stored in the storage unit and then transmitted together to the information processing device 10 at a predetermined timing.

[0024] The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a speaker 16, and a reading unit 17, and these units are connected via a bus. The control unit 11 is configured using one or more processors, such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The control unit 11 executes various information processing and control processes to be performed by the information processing device 10 by appropriately executing a program P stored in the storage unit 12. Note that if the control unit 11 includes multiple processors, each process may be executed by a different processor.

[0025] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 pre-stores a program P (program product) executed by the control unit 11 and various data necessary for executing the program P. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the program P. The storage unit 12 may be composed of multiple storage devices, and part of the storage unit 12 may be another storage device connected to the information processing device 10, or another storage device with which the information processing device 10 can communicate.

[0026] The communication unit 13 has a configuration similar to the communication unit 23 of the wearable device 20 and receives pupil images and scenery images captured by the wearable device 20 from the communication unit 23. The communication unit 13 may be a communication module for wired communication or a communication module for connecting to a network. The communication unit 13 may also be configured to have a communication module for communicating with the wearable device 20 and a communication module for connecting to the network separately. The input unit 14 accepts operation input by a user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information in accordance with instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be integrated into a touch panel. The speaker 16 is an audio output unit that outputs a message, a warning sound, or the like in accordance with instructions from the control unit 11.

[0027] The reading unit 17 reads information stored in a portable storage medium 10a, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, or an SD (Secure Digital) card. The program P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 17 and stored in the storage unit 12. The program P and various data may be written to the storage unit 12 during the manufacturing stage of the information processing device 10, or may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

[0028] The information processing device 10 is not limited to a single computer, but may be a multi-computer including multiple computers. Furthermore, the information processing device 10 may be a virtual machine virtually constructed within a single device by software, or may be a cloud server. The following description assumes that the information processing device 10 is a single computer. The program P may be deployed on a single computer or at a single site, or may be distributed across multiple sites and executed on multiple computers interconnected via a network. Furthermore, the input unit 14 and the display unit 15 are not essential for the information processing device 10. The information processing device 10 may be configured to accept operations through a connected computer or to output information to be displayed to an external display device.

[0029] FIG. 3 is an explanatory diagram showing an example of the configuration of an indwelling needle 30. In the following description, the direction along the axis C of the indwelling needle 30 is referred to as the axial direction, and the left side of FIG. 3 is referred to as the "distal side" and the right side as the "proximal side." The indwelling needle 30 includes a catheter 31, a catheter hub 32 connected to the proximal end of the catheter 31, an inner needle 33 (needle) inserted into the catheter 31, an inner needle hub 34 connected to the proximal end of the inner needle 33, and a filter 35 connected to the proximal end of the inner needle hub 34. A syringe (not shown) can be connected to the proximal end of the inner needle hub 34. A hemostatic valve (not shown) may be provided within the catheter hub 32. The indwelling needle 30 does not necessarily include the filter 35. The catheter 31 constitutes an outer needle and extends to the vicinity of the tip of the inner needle 33 inserted into the catheter 31. A blade surface 331 formed at the tip of the inner needle 33 is exposed (protrudes) from the tip of the catheter 31.

[0030] A puncture method using the indwelling needle 30 configured as described above will be described. The surgeon grasps the indwelling needle 30 with the inner needle 33 inserted into the catheter 31, punctures the tip of the inner needle 33 from the surface of the living body 40 (the subject's skin surface) toward the blood vessel, and gradually inserts the inner needle 33 toward the desired site. This causes the tip of the inner needle 33 to advance while cutting through the body tissue. Once the tip of the inner needle 33 is inserted into the blood vessel, the catheter 31 is also inserted into the same blood vessel. After inserting the inner needle 33 into the target blood vessel, the surgeon moves the catheter 31 and catheter hub 32 toward their distal ends to insert the catheter 31 into the blood vessel. After insertion, the surgeon removes the inner needle 33, inner needle hub 34, and filter 35, leaving the catheter 31 and catheter hub 32 behind, thereby leaving the catheter 31 and catheter hub 32 indwelling in the living body.

[0031] In the puncture support system 100 configured as described above, a medical professional (operator) wears the wearable device 20, for example, on their head (face) when performing a puncture operation. The operator palpates the living body 40 to confirm the location of blood vessels within the living body 40, and then performs the puncture operation. During the operator's palpation operation and the puncture operation, the pupil cameras 21 a and 21 b of the wearable device 20 capture images of the operator's eyes (at least the pupil and iris), and the view camera 22 captures images of the scenery in the operator's line of sight. The view in the operator's line of sight includes the state of the operator's palpation operation and the state of the puncture operation relative to the living body 40. The wearable device 20 transmits pupil images captured by the pupil cameras 21 a and 21 b and a view image captured by the view camera 22 to the information processing device 10. The information processing device 10 performs processing to track the operator's line of sight (eye movement) based on the pupil images and view images captured by the wearable device 20. The information processing device 10 also detects whether the surgeon is performing a palpation operation or a puncture operation based on the scenery image. The information processing device 10 identifies the target puncture position based on the surgeon's line of sight (the viewpoint relative to the living body 40) during the palpation operation, and estimates the planned puncture position based on the tip position of the indwelling needle 30 in the scenery image during the puncture operation. The information processing device 10 then determines whether the puncture state (planned puncture position) is appropriate based on the target puncture position and the planned puncture position, and outputs an alert according to the determination result, thereby supporting the medical professional in the puncture operation.

[0032] The following describes a process for supporting the puncture operation by the surgeon using the puncture support system 100 of this embodiment. Fig. 4 is a flowchart showing an example of the puncture operation support process procedure, and Figs. 5A to 7 are explanatory diagrams of the puncture operation support process. The following process is executed by the control unit 11 in accordance with the program P stored in the storage unit 12 of the information processing device 10, but part of the following process may also be realized by a dedicated hardware circuit (for example, an FPGA or ASIC).

[0033] In the puncture support system 100, when the puncture support process is started, the wearable device 20 starts capturing images with the pupil cameras 21a and 21b and the scenery camera 22, and sequentially transmits the pupil images captured by the pupil cameras 21a and 21b and the scenery images captured by the scenery camera 22 to the information processing device 10. For example, the wearable device 20 may start capturing images with the pupil cameras 21a and 21b and the scenery camera 22 after starting communication with the information processing device 10, or, if the wearable device 20 is provided with a power button for instructing the start and end of operation, the wearable device 20 may start capturing images with the pupil cameras 21a and 21b and the scenery camera 22 after the power button is turned on. The wearable device 20 continues capturing images with the pupil cameras 21a and 21b and the scenery camera 22 and transmitting the pupil images and scenery images to the information processing device 10 until the puncture support process by the puncture support system 10 is completed. For example, when communication with the information processing device 10 is terminated or when the power button is turned off, the wearable device 20 terminates photographing using the pupil cameras 21a, 21b and the scenery camera 22 and the transmission of the pupil images and scenery images to the information processing device 10.

[0034] The control unit 11 of the information processing device 10 starts acquiring pupil images and scenery images sequentially transmitted from the wearable device 20 (S11). The control unit 11 first performs a detection process to determine whether the surgeon is performing a palpation operation on the living body 40 based on the scenery image (S12), and determines whether the surgeon has started the palpation operation (S13). FIG. 5A shows an example of a scenery image, in which the surgeon's palpation operation on the living body 40 is captured. The control unit 11 determines whether the living body 40 and the surgeon's fingers, etc. are present in the scenery image, for example, by template matching using a template. If the control unit 11 determines that the living body 40 and the surgeon's fingers, etc. are present, it determines that the surgeon has started the palpation operation. In this case, template images are generated by extracting features from captured images of a typical puncture target site (e.g., a subject's forearm), the surgeon's fingers, and an instrument used for the puncture operation (e.g., a tourniquet), and the template images are stored in the storage unit 12. The control unit 11 then determines whether or not there is an area in the scenery image that matches each template image, thereby determining whether or not there is a living body 40 and the operator's fingers or the like in the scenery image.

[0035] The control unit 11 may also determine whether the living body 40 and the surgeon's fingers, etc. are present in the scenery image by executing an object detection process for detecting the presence or absence of the living body 40 and the surgeon's fingers, etc., in the scenery image. The object detection process here can be performed, for example, using a learning model trained by machine learning to determine whether, when a scenery image is input, a subject in the input scenery image is a pre-learned subject (e.g., the living body 40, the surgeon's fingers, a tourniquet, etc.). In this case, the control unit 11 inputs the scenery image to the learning model and can detect the presence or absence of the living body 40 and the surgeon's fingers, etc., in the scenery image based on output information from the learning model. Furthermore, when scenery images captured in a time series are input, the control unit 11 may determine whether the surgeon is performing a palpation operation using a learning model trained by machine learning to determine whether the input scenery images contain the surgeon's fingers performing a palpation operation on the living body 40. In this case, the control unit 11 inputs the time-series scenery images to the learning model and can detect whether the surgeon's fingers, captured in the scenery image, are performing a palpation operation on the living body 40 based on output information from the learning model. The control unit 11 may also calculate the difference between scenery images captured in time series and determine, based on the difference images, whether the surgeon's fingers have entered the scenery image or whether the surgeon's fingers are moving within the scenery image. For example, the control unit 11 may determine that the surgeon's fingers are present in the scenery image and that the surgeon is performing a palpation operation if the pixel value (color) of each pixel in the difference image matches the color of the surgeon's fingers (e.g., the color of a medical glove) that has been registered in advance. The control unit 11 may also detect a pixel area in the difference image whose pixel value matches the color of the surgeon's fingers (e.g., the color of a medical glove) and determine that the surgeon's fingers are present in the scenery image if the detected area matches the shape of the surgeon's fingers that has been registered in advance. Furthermore, when performing palpation and puncture operations with a predetermined marker attached to the surgeon's fingers or the gloves worn by the surgeon, the control unit 11 may detect the presence or absence of the predetermined marker in the scenery image to determine the presence or absence of the surgeon's fingers in the scenery image.

[0036] If the control unit 11 determines, as a result of the above-described processing, that the surgeon has not started the palpation operation (S13: NO), it continues the processing of step S12. On the other hand, if the control unit 11 determines that the surgeon has started the palpation operation (S13: YES), it executes eye tracking processing to track the surgeon's line of sight (eye movement) based on the pupil image and the scenery image (S14). In this embodiment, the eye tracking processing is started after detecting the surgeon's palpation operation. However, the palpation operation detection processing is not necessarily performed. After processing step S11, the processing from step S14 onward may be performed without performing steps S12 to S13. While performing the palpation operation, the surgeon visually searches for the site to be punctured (planned puncture location), and the control unit 11 tracks the surgeon's line of sight (point of gaze) through the eye tracking processing. In the eye tracking processing, the control unit 11 identifies the surgeon's line of sight using, for example, pupil center corneal reflection (PCCR). The corneal reflex method is a process for determining the gaze direction by utilizing the change in the positional relationship between the center of the pupil and the Purkinje image as the gaze direction (eye direction) changes. In addition to the corneal reflex method, the eye tracking process may also use a bright pupil method, in which the optical axes of a near-infrared LED and a light receiving unit are brought closer together to detect the pupil as brighter than the iris, or a dark pupil method, in which the optical axes of a near-infrared LED and a light receiving unit are separated to detect the pupil as darker than the iris. In addition to the non-contact eye tracking process described above, contact eye tracking processes using a search coil method or electrooculography may also be performed. The search coil method involves attaching a contact lens incorporating a coil to the surgeon's eye to measure eye movement (rotation). The electrooculography method involves attaching electrodes to the medial and lateral canthi of each eye to measure the potential difference generated between the cornea and retina during eye movement, thereby determining the gaze direction of the surgeon. Through the above-described processing, the control unit 11 identifies the gaze direction of the surgeon's left and right eyes based on the images of the surgeon's left and right pupils, and identifies the surgeon's viewpoint position in the scenery image based on the gaze direction of the left and right eyes.By setting the distance between the living body 40 and the eyes of the surgeon in advance, the control unit 11 can identify the viewpoint position of the surgeon relative to the living body 40 in the scenery image based on the line of sight of the eyes of the surgeon and the distance from the eyes of the surgeon to the living body 40. The control unit 11 may identify the viewpoint position of the surgeon in the scenery image by searching for a line of sight intersection in three dimensions.

[0037] The control unit 11 tracks the surgeon's viewpoint relative to the living organism 40 in the scenery image by sequentially performing the above-described process of identifying the surgeon's viewpoint relative to the living organism 40 in the scenery image based on pupil images and scenery images captured in chronological order. The control unit 11 extracts points in the scenery image at which the surgeon looks (gazing) at the living organism 40 for a predetermined period of time (e.g., 0.5 seconds or 1 second) or more as gaze points, plots each gaze point on the scenery image, and generates a viewpoint map connecting the gaze points to show the trajectory of the surgeon's viewpoint position. Figure 5B shows an example of the viewpoint map, in which each gaze point is represented by a circle corresponding to the gaze order, and movement trajectories (saccades) between gaze points are represented by straight lines. In the example of Figure 5B, the circles for each gaze point are the same size, but the longer the gaze time at each gaze point, the larger the circle may be.

[0038] While performing the eye tracking process, the control unit 11 determines whether the conditions for setting the target puncture position (target setting conditions) are met (S15). For example, the target setting condition may be when the surgeon gazes at a certain gaze point for a predetermined period of time (e.g., 2 or 3 seconds) or more, and the gaze point at that time may be set as the target puncture position. Under this target setting condition, the control unit 11 determines whether the surgeon gazes at a certain gaze point for a predetermined period of time or more during the eye tracking process. If it is determined that the surgeon gazed at the gaze point, it determines that the target setting condition is met (S15: YES) and sets the gaze point at that time as the target puncture position (S16). In this case, the surgeon can set the desired target puncture position as the target puncture position by gazing at the desired location for a predetermined period of time or more. Note that the target setting condition may also be, for example, when the surgeon performs a predetermined operation, and the gaze position at the time the surgeon performs the predetermined operation may be set as the target puncture position. Under this target setting condition, the control unit 11 determines whether the surgeon has performed a predetermined operation while performing eye tracking processing. If it determines that the surgeon has performed the predetermined operation, it determines that the target setting condition is met (S15: YES) and sets the current gaze point as the target puncture position (S16). In this case, the surgeon can set the desired target puncture position as the target puncture position by performing a predetermined operation while gazing at the desired target puncture position. Note that the predetermined operation may be, for example, an operation input based on the surgeon's blink (e.g., blinking twice) that can be detected by eye tracking processing. Furthermore, the predetermined operation may be an operation input using an operation button provided on the wearable device 20 or the information processing device 10, or a voice input of a predetermined message (e.g., a message such as "Set this as the target position") via a microphone provided on the wearable device 20 or the information processing device 10. Furthermore, the predetermined operation may be a gesture input by the surgeon if the wearable device 20 or the information processing device 10 is equipped with a sensor that can detect gestures made by the surgeon's head, mouth, etc. The target setting conditions may be configured so that they can be changed for each operator, or may be configured so that they can be changed for each site to be punctured.

[0039] If the control unit 11 determines that the target setting conditions are not satisfied (S15: NO), it continues the processing of step S14. On the other hand, if the control unit 11 determines that the target setting conditions are satisfied (S15: YES), it sets, for example, the current viewpoint position (gazing point) as the target puncture position (S16). FIG. 5C shows an example of setting the target puncture position. For example, if the target setting conditions are satisfied when the surgeon is gazing at the fourth gaze point in the viewpoint map of FIG. 5B, the position indicated by P1 in FIG. 5C is set as the target puncture position. Note that, after setting the target puncture position, the control unit 11 may notify the surgeon that setting of the target puncture position is complete by audio output via the speaker 16 or by displaying a message on the display unit 15. At this time, the control unit 11 may display on the display unit 15 an image in which the target puncture position P1 is plotted on a landscape image such as that shown in FIG. 5C. When notified that setting of the target puncture position is complete, the surgeon ends the palpation operation and proceeds to the puncture procedure.

[0040] Next, the control unit 11 executes a detection process to detect the presence or absence of the indwelling needle 30 (puncture device) in the scenery image based on the scenery image (S17) and determines whether the indwelling needle 30 has been detected (S18). FIG. 6A shows an example of a scenery image captured of the surgeon's puncture operation on the living body 40. In the scenery image of FIG. 6A, the target puncture position is indicated by a mark P1 to facilitate comparison between the target puncture position and the planned puncture position. However, the actually captured scenery image does not include a mark indicating the target puncture position. Note that, when a scenery image captured by the scenery camera 22 is configured to be displayed on the display unit 15, the scenery image may be displayed with a mark P1 indicating the target puncture position added, as shown in FIG. 6A. In this case, the surgeon can perform the puncture operation while confirming the target puncture position using the scenery image with the mark P1 added. In the indwelling needle 30 detection process, the control unit 11 determines whether the indwelling needle 30 is present in the scenery image, for example, by template matching using a template. In this case, template images are generated by extracting the characteristics of a typical indwelling needle 30 and the tip portion of the indwelling needle 30 (the tip portion of the inner needle 33 or catheter 31) from photographed images, and the template images are stored in the storage unit 12. The control unit 11 then determines whether or not there is an area in the scenery image that matches each template image, thereby determining whether or not there is an indwelling needle 30 in the scenery image.

[0041] Here, too, the control unit 11 may determine whether the indwelling needle 30 is present in a scenery image by object detection processing using a learning model. For example, the learning model may be a learning model that, when a scenery image is input, determines whether the subject in the input scenery image is a previously learned subject (the indwelling needle 30 or the tip of the indwelling needle 30). Alternatively, when scenery images captured in time series are input, the control unit 11 may determine whether the surgeon is performing the puncture operation using a learning model trained by machine learning to determine whether the input scenery images contain the surgeon's fingers performing the puncture operation on the living body 40. In this case, the control unit 11 inputs the scenery images in time series to the learning model, and based on the output information from the learning model, can detect whether the surgeon's fingers captured in the scenery images are performing the puncture operation on the living body 40. Alternatively, the control unit 11 may calculate the difference between the scenery images captured in time series and, based on the difference image, determine whether the indwelling needle 30 has entered the scenery image or whether the indwelling needle 30 is moving within the scenery image. For example, if the pixel value (color) of each pixel in the difference image matches the pre-registered color of the indwelling needle 30 (e.g., the color of the inner needle 33 or the catheter 31), the control unit 11 may determine that the indwelling needle 30 is present in the scenery image and determine that the surgeon is performing the puncture procedure. The control unit 11 may also detect a region of pixels in the difference image whose pixel value matches the color of the indwelling needle 30 (e.g., the color of the inner needle 33 or the catheter 31), and determine that the indwelling needle 30 is present in the scenery image if the detected region matches the pre-registered shape of the indwelling needle 30. Furthermore, if a marker is provided at a predetermined location on the indwelling needle 30, the control unit 11 may detect the presence or absence of the marker in the scenery image by detecting the presence or absence of the marker in the scenery image. A configuration using an indwelling needle 30 with a marker will be described in embodiment 2 below.

[0042] If the control unit 11 determines that the indwelling needle 30 cannot be detected in the scenery image as a result of the above processing (S18: NO), it continues processing in step S17. If the control unit 11 determines that the indwelling needle 30 has been detected (S18: YES), it determines whether the surgeon will start the puncture operation (whether the surgeon is about to start the puncture operation) based on the scenery image (S19). For example, the control unit 11 executes the indwelling needle 30 detection process for scenery images captured in time series, identifies the position of the tip of the indwelling needle 30 relative to the living body 40 in each scenery image, and determines that the surgeon has started (is about to start) the puncture operation if the position of the tip of the indwelling needle 30 relative to the living body 40 has not moved for a predetermined time (e.g., 2 or 3 seconds) or more (S19: YES). If the control unit 11 determines that the surgeon will start the puncture operation (S19: YES), it identifies the position of the tip of the indwelling needle 30 at this time (the time when it was determined that the puncture operation should be started) as the planned puncture position (S20). In this case, the surgeon can set the intended puncture location at the tip of the indwelling needle 30 by aligning the tip with the intended puncture location and not moving it for more than a predetermined time (e.g., 2 or 3 seconds).

[0043] In addition to the above-described process, whether or not the surgeon starts the puncture operation may be determined, for example, based on whether or not the surgeon has performed a predetermined operation. In this case, the position of the tip of the indwelling needle 30 at the time the surgeon performs the predetermined operation can be identified as the planned puncture position. In this case, the surgeon can set the desired location as the planned puncture position by performing the predetermined operation while aligning the tip of the indwelling needle 30 with the desired location. The predetermined operation here may also be an operation input by the surgeon's blink, an operation input using an operation button provided on the wearable device 20 or the information processing device 10, a voice input using a predetermined message (e.g., a message such as "Set this as the puncture position"), a gesture input by the surgeon, or the like.

[0044] If the operator determines not to start the puncture procedure (S19: NO), the control unit 11 returns to the process of step S17 and continues the processes of steps S17 to S19. If the operator determines to start the puncture procedure (S19: YES), the control unit 11 specifies the position of the tip of the indwelling needle 30 at this time (tip position) as the planned puncture position (S20). For example, the control unit 11 specifies the position of the tip of the indwelling needle 30 detected in the scenery image in step S18, immediately before the operator determined in step S19 to start the puncture procedure, as the planned puncture position. Figure 6B shows the scenery image shown in Figure 6A with a mark P2 indicating the planned puncture position added.

[0045] The controller 11 calculates the amount of deviation between the target puncture position set in step S16 and the planned puncture position identified in step S20 (S21). Fig. 6C shows an enlarged view of a region including the target puncture position P1 and the planned puncture position P2 on the living body 40 in the scenery image shown in Fig. 6B . The controller 11 may calculate the linear distance (number of pixels) of a line segment connecting the target puncture position P1 and the planned puncture position P2 in the two-dimensional scenery image as shown in Fig. 6C , and use this as the amount of deviation. The controller 11 may also calculate the amount of deviation between the target puncture position P1 and the planned puncture position P2 in the vertical and horizontal directions of the scenery image. Fig. 7 shows another example of the amount of deviation. As shown in Fig. 7 , the controller 11 may also calculate the amount of deviation in the longitudinal direction (extension direction) of the living body 40 (here, the forearm) and in the orthogonal direction perpendicular to the longitudinal direction. The longitudinal direction of the living body 40 can be identified by, for example, approximating the outer edge 40a of the living body 40 to a straight line in the scenery image.

[0046] The control unit 11 outputs a message according to the calculated amount of deviation (S22). For example, if the amount of deviation is equal to or greater than a predetermined value, the control unit 11 outputs a message such as "There is a large deviation from the target position" or "Please try again" to warn the surgeon (output an alert). If the amount of deviation is less than a predetermined value, the control unit 11 outputs a message such as "Close to the target position" or "Let's start puncturing" to notify the surgeon that the deviation between the target puncture position and the planned puncture position is small. Furthermore, the number of messages is not limited to two levels (two types), and three or more levels of messages may be provided. For example, the control unit 11 may be configured to output a message saying "Let's start puncturing" when the amount of deviation is less than a first threshold (e.g., 1 mm), to output a message saying "There is a slight deviation from the target position" when the amount of deviation is equal to or greater than the first threshold but less than a second threshold (e.g., 2 mm), and to output a message saying "Let's try again" when the amount of deviation is equal to or greater than the second threshold. Furthermore, when the amount of deviation is equal to or greater than a first threshold and less than a second threshold, the control unit 11 may identify an operation to be performed to move the planned puncture position closer to the target puncture position, and may output a message such as, for example, "Please move a little more to the right." The message may be output as audio via the speaker 16, displayed on the display unit 15, or by lighting or flashing a lamp provided on the wearable device 20 or the information processing device 10. The amount of deviation used to determine whether to output an alert may be the linear distance, the amount of deviation in the longitudinal direction and / or the orthogonal direction of the living body 40 (e.g., the forearm), or a combination of these. For example, the device may be configured to output an alert when the linear distance is equal to or greater than a first threshold and the amount of deviation in the orthogonal direction is equal to or greater than a second threshold. Alternatively, the device may be configured not to output a message when the amount of deviation is less than the first threshold.

[0047] When an alert (warning message) is output, the surgeon may start over from the palpation operation or from the puncture operation. If the surgeon starts over from the palpation operation, the control unit 11 proceeds to step S12 and repeats steps S12 to S22. In this case, the control unit 11 resets the target puncture position and observes the amount of deviation of the planned puncture position from the newly set target puncture position. If the surgeon starts over from the puncture operation, the control unit 11 proceeds to step S17 and repeats steps S17 to S22. In this case, the control unit 11 observes the amount of deviation of the newly identified planned puncture position from the already set target puncture position.

[0048] Through the above-described processing, in the puncture support system 100 of this embodiment, when the surgeon punctures a blood vessel of a subject, the target puncture position is set based on the surgeon's line of sight (the surgeon's viewpoint relative to the living body 40) during palpation, and the planned puncture position is set based on the position of the tip of the indwelling needle 30 during the puncture operation. Then, by outputting an alert if the planned puncture position and the target puncture position are separated by a predetermined distance or more, it is possible to notify the surgeon that there is a high possibility that the puncture operation will fail due to the planned puncture position being misaligned from the target puncture position. Therefore, the surgeon can prevent the puncture operation from failing by, for example, redoing the puncture operation. In particular, when a blood vessel that cannot be visually confirmed is to be punctured, it is necessary to rely on the blood vessel position confirmed by palpation. However, by outputting an alert if the deviation from the target puncture position is large, the success rate of the puncture operation can be improved. In this embodiment, the target puncture position is set based on the surgeon's viewpoint position relative to the living body 40, and the planned puncture position is set based on the tip position of the indwelling needle 30 relative to the living body 40, so it is not affected by ambient light, the skin color (race) of the living body 40, the condition of the skin surface of the living body 40 (amount of body hair), etc.

[0049] In this embodiment, at least one of the process of detecting whether the surgeon is performing a palpation motion based on a scenery image, the eye-tracking process based on the pupil image and scenery image, and the process of detecting the presence or absence of an indwelling needle 30 (puncture instrument) in the scenery image based on the scenery image is not limited to being performed locally by the information processing device 10. For example, a server may be provided to perform the process of detecting whether the surgeon is performing a palpation motion based on the scenery image. In this case, the information processing device 10 may be configured to transmit scenery images sequentially acquired from the wearable device 20 to the server and receive the results determined by the server (information indicating whether the surgeon has started a palpation motion). A server may also be provided to perform the eye-tracking process. In this case, the information processing device 10 may be configured to transmit pupil images and scenery images acquired from the wearable device 20 to the server and receive a viewpoint map generated by the server, showing the trajectory of the surgeon's gaze point. Furthermore, a server may be provided to perform the process of detecting the indwelling needle 30 in the scenery image based on the scenery image. In this case, the information processing device 10 can be configured to transmit a scenery image acquired from the wearable device 20 to a server and receive the result determined by the server (information indicating the presence or absence of the indwelling needle 30 in the scenery image). Furthermore, in this embodiment, the wearable device 20 may be provided with a control unit so that at least one of the above-described processes is executed locally by the wearable device 20.

[0050] (Embodiment 2) A puncture support system that supports a puncture operation using an indwelling needle 30 provided with a marker will be described. The puncture support system of this embodiment can be realized by devices similar to the puncture support system 100 of embodiment 1 shown in Fig. 1 except for the indwelling needle 30, and therefore a description of the configuration of each device will be omitted.

[0051] FIG. 8 is an explanatory diagram showing an example of the configuration of an indwelling needle 30 of embodiment 2. The indwelling needle 30 of this embodiment has a marker provided at an appropriate location on the distal end side of the indwelling needle 30 of embodiment 1 shown in FIG. 3. The marker is a sign that functions as a landmark for detecting the distal position of the indwelling needle 30 relative to the living body 40 during, for example, a puncture operation. FIG. 8 shows marker M1 provided on the outer peripheral surface of the distal end of the catheter 31 and marker M2 provided adjacent to the blade surface 331 on the outer peripheral surface of the distal end of the inner needle 33; however, it is sufficient if either marker M1 or M2 is provided. When using an indwelling needle 30 provided with only marker M1, the puncture operation is performed with marker M1 positioned on the same side as the blade surface 331. This allows for accurate imaging of the marker M1 when the indwelling needle 30, which is used with the blade surface 331 facing upward during puncture, is photographed from above.

[0052] The shapes of the markers M1 and M2 are not particularly limited and may be any suitable shape, such as a circle, triangle, square, star, or cross. Complex shapes are preferred for the markers M1 and M2, as they reduce the possibility of erroneous detection. Star-shaped markers M1 and M2 are shown in FIG. 8 . The markers M1 and M2 may be formed over a predetermined length in the circumferential direction of the indwelling needle 30 (catheter 31, inner needle 33), or may be formed over the entire circumference. When the markers M1 and M2 are formed in the circumferential direction, the shape of the markers M1 and M2 may be linear, such as a single line, two lines, or a dashed line. The shape of the markers M1 and M2 may be set depending on the type of indwelling needle 30.

[0053] In the puncture support system 100 of this embodiment, the control unit 11 of the information processing device 10 can execute the same process as that shown in Fig. 4 and can assist the surgeon in the puncture operation based on pupil images captured by the pupil cameras 21a and 21b of the wearable device 20 and scenery images captured by the scenery camera 22. In this embodiment, in steps S17 and S18 in Fig. 4, the control unit 11 executes a process to detect the presence or absence of markers M1 and M2 provided on the indwelling needle 30 in the scenery image, thereby detecting the presence or absence of the indwelling needle 30 in the scenery image. In addition, in step S19 in Fig. 4, the control unit 11 determines that the surgeon should start the puncture operation if the positions of the markers M1 and M2 in the scenery images captured in time series have not moved for a predetermined time (e.g., 2 or 3 seconds) or more.

[0054] In step S20 in FIG. 4 , the control unit 11 estimates the tip position of the indwelling needle 30 based on the positions of markers M1 and M2 in the scenery image and identifies the estimated position as the planned puncture position. FIG. 9 is an explanatory diagram of the process of estimating the tip position of the indwelling needle 30. FIG. 9 shows a scenery image of a puncture operation using an indwelling needle 30 equipped with only marker M1. As shown in FIG. 9 , the position of marker M1 does not coincide with the tip position of the indwelling needle 30, so the control unit 11 estimates a predetermined position based on marker M1 as the tip position of the indwelling needle 30. As can be seen from FIG. 9 , the scenery image during the puncture operation is captured from the proximal end side of the indwelling needle 30. Therefore, the control unit 11 may estimate a position in the scenery image that is a predetermined distance (predetermined pixels) above the position of marker M1 (position P2 in FIG. 9 ) as the tip position of the indwelling needle 30. If the indwelling needle 30 in the scenery image is photographed in a state where it is tilted relative to the vertical direction, the control unit 11 may estimate the tip position of the indwelling needle 30 to be a position a predetermined distance above the position of the marker M1 in the direction of tilt (the axial direction of the indwelling needle 30), or may estimate the tip position of the indwelling needle 30 to be a position a predetermined distance above the longitudinal direction of the living body 40 (for example, the forearm) as shown in Figure 7. When an indwelling needle 30 provided with markers M1 and M2 is used, the tip position of the indwelling needle 30 may be estimated from the relative positions of the markers M1 and M2 in the scenery image.

[0055] In this embodiment, the information processing device 10 can execute the same processes as in embodiment 1, and can achieve the same effects. Furthermore, in this embodiment, the indwelling needle 30 is provided with markers M1 and M2, so the presence or absence of the indwelling needle 30 (markers M1 and M2) in the scenery image can be detected with high accuracy. Furthermore, the tip position of the indwelling needle 30 (planned puncture position) is estimated based on the positions of the markers M1 and M2 in the scenery image, so the planned puncture position can be set with high accuracy. Therefore, the amount of deviation between the target puncture position and the planned puncture position can be measured more accurately, making it possible to accurately output a necessary alert.

[0056] In this embodiment, the markers M1 and M2 are preferably provided on the tip side of the indwelling needle 30, but they do not have to be provided on the tip side. By registering the distance between the tip position of the indwelling needle 30 and the markers M1 and M2 in advance, the control unit 11 can estimate the tip position of the indwelling needle 30 from the positions of the markers M1 and M2 in the scenery image.

[0057] In this embodiment, the scenery camera 22 may be a near-infrared camera having a near-infrared LED and a near-infrared light receiving unit, and in this case, the markers M1 and M2 may be markers containing a near-infrared fluorescent dye that emits near-infrared fluorescence of a specific wavelength when irradiated with excitation light. In such a configuration, the scenery camera 22 emits near-infrared light in the direction of the surgeon's line of sight using the near-infrared LED, and receives the near-infrared light emitted by the near-infrared LED reflected from the living body 40, the indwelling needle 30, etc. using the light receiving unit, thereby acquiring a scenery image using near-infrared light. When the markers M1 and M2 contain a near-infrared fluorescent dye, the shape, range (area), etc. of the markers M1 and M2 in the scenery image can be detected with high accuracy.

[0058] The markers M1 and M2 may contain a near-infrared fluorescent dye that emits near-infrared fluorescence in a wavelength range of approximately 750 nm to 2500 nm, for example, or may contain a fluorescent dye that emits fluorescence in a wavelength range corresponding to green, blue, or the like (e.g., wavelengths of 400 to 600 nm) that is not present in the subject's living body 40. By differentiating the response wavelength of the markers M1 and M2 from the response wavelength of the blood vessels of the living body 40, the markers M1 and M2 can be clearly distinguished and detected from the blood vessels. The markers M1 and M2 may be formed, for example, by incorporating the fluorescent dye into a base material by kneading or coating, and the formed markers M1 and M2 may be attached to a predetermined location on the indwelling needle 30. The markers M1 and M2 may also be formed by directly incorporating the fluorescent dye into a predetermined location on the indwelling needle 30 by kneading or coating, for example. The markers M1 and M2 may be attached to the indwelling needle 30 during manufacturing, or may be attached by a surgeon or the like before use of the indwelling needle 30. The markers M1 and M2 may be made of a removable material, such as a seal member.

[0059] Note that the configuration and detection method of the markers M1, M2 and the scenery camera 22 are not limited as long as the scenery camera 22 can detect the markers M1, M2. The markers M1, M2 may include, for example, a light-emitting body that emits light of a specific wavelength, an upconversion (UC) phosphor that emits light of a specific wavelength, or a reflector that reflects irradiated light of a specific wavelength. The markers M1, M2 may have a reflective structure that reflects light by applying surface treatment such as embossing or unevenness. The markers M1, M2 may include a material that absorbs visible light, such as colored ink, and be detected by a visible light camera.

[0060] (Embodiment 3) A puncture support system having a function of generating a blood vessel visualization image that visualizes blood vessels in a living body 40 will be described. The puncture support system of this embodiment can be realized by devices similar to the puncture support system 100 of embodiment 1 shown in Fig. 1 , and therefore a description of the configuration of each device will be omitted. In this embodiment, the scenery camera 22 of the wearable device 20 is a near-infrared camera having a near-infrared LED and a near-infrared light receiving unit.

[0061] In this embodiment, the near-infrared LED of the scenery camera 22 can be an LED that emits near-infrared light with a wavelength of approximately 600 nm to 2500 nm, taking into account absorption by hemoglobin in blood. An LED that emits near-infrared light with a wavelength of approximately 700 nm to 1400 nm, and more preferably approximately 780 nm to 940 nm, may be used. The near-infrared LED of the scenery camera 22 emits near-infrared light of the above-mentioned wavelength in the direction of the surgeon's line of sight, thereby irradiating the living body 40, the indwelling needle 30, and the surgeon's fingers with near-infrared light. The light-receiving unit of the scenery camera 22 receives near-infrared light reflected from the living body 40, the indwelling needle 30, and the surgeon's fingers, and acquires a scenery image using near-infrared light. The scenery camera 22 captures images of the palpation and puncture operations performed by the surgeon during the palpation and puncture operations. When a living body 40 is present within the photographing range of the scenery camera 22 , the scenery image includes a visualized image of blood vessels representing the blood vessels within the living body 40 .

[0062] The spectral sensitivity wavelength range of the light receiving unit of the scenery camera 22 is not particularly limited as long as it corresponds to the emission wavelength of the near-infrared LED, and can be set appropriately taking into account the response wavelength of blood vessels. For example, since the response wavelength of blood vessels in the living body 40, i.e., the wavelength of light absorbed by blood vessels, is approximately 800 nm to 900 nm, the spectral sensitivity wavelength range of the light receiving unit of the scenery camera 22 can be set to a range including 800 nm to 900 nm. If the indwelling needle 30 is provided with markers M1 and M2 (see FIG. 8 ) containing near-infrared fluorescent dyes that emit near-infrared fluorescence, the spectral sensitivity wavelength range of the light receiving unit of the scenery camera 22 can be set to a range including the response wavelength of the markers M1 and M2, i.e., the wavelength of light reflected by the markers M1 and M2, and the response wavelength of the blood vessels. The light receiving unit of the scenery camera 22 may include a blood vessel detection light receiving unit having a spectral sensitivity wavelength corresponding to the blood vessel response wavelength, and a marker detection light receiving unit having a spectral sensitivity wavelength corresponding to the blood vessel response wavelength of the markers M1 and M2. In this case, the light receiving unit for blood vessel detection acquires a scenery image including a visualized blood vessel image, and the light receiving unit for marker detection acquires a scenery image including a marker image. The spectral sensitivity wavelengths of the light receiving unit for blood vessel detection and the light receiving unit for marker detection may be the same or partially overlap, or may be different from each other.

[0063] Furthermore, the scenery camera 22 of this embodiment may include a near-infrared camera for acquiring a visualized blood vessel image, and a visible light camera for acquiring photographed images of the living body 40 and the indwelling needle 30. When using such a scenery camera 22, if the markers M1 and M2 contain a near-infrared fluorescent dye, the marker images may be acquired by the near-infrared camera, and if the markers M1 and M2 do not contain a near-infrared fluorescent dye, the scenery image including the markers M1 and M2 may be acquired by the visible light camera.

[0064] Fig. 10 is a flowchart showing an example of the procedure for the puncture operation support process according to the third embodiment, and Fig. 11 is an explanatory diagram of the puncture operation support process. The process shown in Fig. 10 is the process shown in Fig. 4 with steps S31 to S35 added between steps S16 and S17. Explanation of the same steps as in Fig. 4 will be omitted. Note that steps S18 to S22 in Fig. 4 are not shown in Fig. 10.

[0065] In the puncture support system 100 of this embodiment, the scenery camera 22 of the wearable device 20 acquires a scenery image captured using near-infrared light. Therefore, in step S12, the control unit 11 of the information processing device 10 performs a detection process to determine whether the surgeon is performing a palpation operation on the living body 40, based on the scenery image captured using near-infrared light. In addition, in step S14, the control unit 11 performs an eye-tracking process to track the surgeon's line of sight, based on the pupil image and the scenery image captured using near-infrared light.

[0066] In the information processing device 10 of this embodiment, after processing step S16, the control unit 11 generates a blood vessel visualization image based on the scenery image (S31). For example, the control unit 11 extracts blood vessels from the scenery image to generate a blood vessel visualization image showing an area corresponding to the blood vessels. The method for extracting blood vessels from the scenery image is not limited, but edge detection based on the brightness values ​​of the image may be performed to identify the position (range) of the blood vessels based on the known straightness, length, thickness, extension direction, etc. of the blood vessels, or a method such as pattern matching may be used. The control unit 11 may also identify blood vessels in the scenery image based on the known response wavelength of the blood vessels.

[0067] The control unit 11 calculates the amount of deviation (second deviation) between the blood vessel in the blood vessel visualization image generated in step S31 and the target puncture position set in step S16 (S32). Fig. 11 shows a scenery image including the blood vessel visualization image, and shows an enlarged view of a region including the target puncture position P1 on the living body 40 and a blood vessel 40b in the blood vessel visualization image. As shown in Fig. 11, the control unit 11 identifies the blood vessel closest to the target puncture position P1 in the two-dimensional scenery image, and calculates the distance (number of pixels) between the identified blood vessel and the target puncture position P1, thereby determining the amount of deviation.

[0068] The control unit 11 determines whether the calculated amount of deviation is equal to or greater than a predetermined value (S33). If it determines that the amount of deviation is equal to or greater than the predetermined value (S33: YES), it outputs a message such as "The target position is deviated from the blood vessel" or "Please start again from the palpation operation" to warn the surgeon (output an alert) (S35). If it determines that the amount of deviation is less than the predetermined value (S33: NO), it determines whether the target puncture position is appropriate based on the positional relationship between the blood vessel in the blood vessel visualization image and the target puncture position (S34). For example, the control unit 11 determines whether the target puncture position is near a point where the blood vessel branches, as in the area circled by a dashed line in Figure 11. If it determines that the target puncture position is near a point where the blood vessel branches, it determines that the target puncture position is inappropriate. If it determines that the target puncture position is not near the point, it determines that the target puncture position is appropriate. Specifically, the control unit 11 measures the distance between the target puncture position and the point where the blood vessel branches, and if the measured distance is less than a predetermined value, it determines that the target puncture position is inappropriate.

[0069] If the control unit 11 determines that the target puncture position is inappropriate (S34: NO), it outputs a message such as "The target position is close to a branching point of the blood vessel" or "Select a straight point on the blood vessel" to warn the surgeon (outputs an alert) (S35). If the control unit 11 determines that the target puncture position is appropriate (S34: YES), it proceeds to the processing of step S17. If an alert is output, the surgeon starts over from the palpation operation. Therefore, after processing step S35, the control unit 11 proceeds to the processing of step S12 and repeats the processing from step S12 onwards.

[0070] By the above-described processing, in this embodiment, a blood vessel visualization image can be generated from a scenery image captured by the scenery camera 22, which is a near-infrared camera, and it is possible to determine whether the target puncture position is appropriate for the blood vessels in the blood vessel visualization image and output an alert. The configuration of this embodiment can be applied to the puncture support system 100 of the first and second embodiments described above, and even when applied to the puncture support system 100 of the first and second embodiments, similar processing can be performed and similar effects can be obtained. Furthermore, the modified examples described as appropriate in the first and second embodiments can also be applied to this embodiment.

[0071] In the above-described processing, the predetermined value (threshold value for the distance between the blood vessel and the target puncture position P1) used to determine whether to output an alert in step S33, and the predetermined value (threshold value for the distance between the branching point of the blood vessel and the target puncture position P1) used to determine whether to output an alert in step S34 may be configurable, and may be switchable depending on attribute information such as the subject's age and gender.

[0072] (Embodiment 4) The puncture support systems 100 of the above-described embodiments 1 to 3 are configured to output a message according to the amount of deviation between the target puncture position and the planned puncture position, thereby outputting an alert to the surgeon. This embodiment describes a puncture support system configured to use a learning model to determine a threshold value used to determine whether to output an alert based on the amount of deviation between the target puncture position and the planned puncture position. The puncture support system of this embodiment can be realized by devices similar to the puncture support system 100 of embodiment 1 shown in FIG. 1 , so a description of the configuration of each device will be omitted. Note that the information processing device 10 of this embodiment, in addition to the configuration shown in FIG. 2 , stores in the storage unit 12 a learning model that has learned training data by, for example, machine learning.

[0073] FIG. 12A is an explanatory diagram showing an example of the configuration of a first learning model, and FIG. 12B is an explanatory diagram showing an example of the configuration of a second learning model. FIG. 12A shows a first learning model Ma that estimates, from the subject's biological information, a threshold used to determine whether to issue an alert for the amount of deviation between the target puncture position and the planned puncture position. FIG. 12B shows a second learning model Mb, a variation of the first learning model Ma, that estimates the success rate of puncture for each of predetermined deviation amounts (first to third deviation amounts, etc.). In the second learning model Mb of FIG. 12B, the first deviation amount is less than 1 mm, the second deviation amount is 1 mm or more but less than 2 mm, and the third deviation amount is 2 mm or more, but these values ​​are not limited. The learning models Ma and Mb are expected to be used as program modules constituting artificial intelligence software. The learning models Ma and Mb perform a predetermined calculation on the input values ​​and output the calculation results, and the memory unit 12 stores data such as coefficients and thresholds of the functions that define this calculation as the learning models Ma and Mb.

[0074] The first learning model Ma is trained to input biometric information of a subject, perform a calculation based on the input data to estimate a threshold used to determine whether to issue an alert for the amount of deviation between the target puncture position and the planned puncture position during a puncture procedure on the subject, and output the calculated result. The first learning model Ma can be configured using algorithms such as a support vector machine (SVM), a decision tree, a random forest, a convolutional neural network (CNN), or a transformer, or may be configured by combining multiple algorithms. The biometric information, which is input data, may be attribute information including the subject's age and gender, and may further include personal information such as the subject's diagnosis, medical history, edema status, height, and weight. The input data of the first learning model Ma may also include environmental information such as the temperature and humidity of the location where the puncture procedure is performed. The threshold, which is output data, may be an arbitrary numerical value or a range of values ​​that can be used as a threshold.

[0075] The first learning model Ma includes an input layer to which biometric information is input, an output layer that outputs a threshold value for the deviation between the target puncture position and the planned puncture position, and an intermediate layer that calculates an output value based on the input data. The input layer of the first learning model Ma has multiple input nodes, and each piece of information contained in the biometric information is input via each input node. The input layer of the first learning model Ma may be configured to input only the subject's age and gender. The intermediate layer of the first learning model Ma extracts features of the input data using various functions and thresholds, and outputs the extracted feature values ​​to the output layer. The output layer of the first learning model Ma has multiple output nodes, each associated with a possible threshold value, and each output node outputs a confidence level for each associated value. The output value of each output node is, for example, a value between 0 and 1, and the sum of the output values ​​from each output node is 1 (100%). The information processing device 10 estimates the numerical value associated with the output node that outputs the largest output value (certainty) in the first learning model Ma as the threshold value for the deviation amount between the target puncture position and the planned puncture position. Note that the first learning model Ma may be configured to have a single output node that outputs the numerical value with the highest confidence, instead of having multiple output nodes that output confidence levels for each numerical value.

[0076] The first learning model Ma is generated by machine learning using training data that associates training biometric information with a correct threshold value for the deviation between the target puncture position and the planned puncture position. The correct threshold value can be a threshold value determined by medical professionals such as doctors and nurses based on the results (success or failure) of a puncture procedure performed on a subject of the training biometric information. When the first learning model Ma receives input biometric information from the training data, the model learns so that the output value from the output node corresponding to the correct threshold value approaches 1 and the output values ​​from the other output nodes approach 0. In the learning process, the first learning model Ma performs calculations based on the input biometric information to calculate output values ​​from each output node. The first learning model Ma then compares the calculated output values ​​of each output node with a value corresponding to the correct threshold value (1 for the output node corresponding to the correct threshold value and 0 for the other output nodes) and optimizes parameters used in the calculation process so that the two values ​​approximate each other. The parameters are parameters such as weights (coupling coefficients) between nodes in the first learning model Ma, and the parameter optimization method can be the backpropagation method, the steepest descent method, etc. In this way, when the biometric information of a subject is input, a first learning model Ma can be obtained that outputs a threshold value used to determine whether to output an alert based on the amount of deviation between the target puncture position and the planned puncture position in a puncture operation on the subject.

[0077] 12B is trained to input the subject's biometric information, similar to the first learning model Ma, and to perform a calculation based on the input data to estimate the success rate of a puncture operation on the subject when the puncture operation is performed with a predetermined deviation between the target puncture position and the planned puncture position, and to output the calculation result. The second learning model Mb can also be configured using an algorithm such as SVM, decision tree, random forest, CNN, or Transformer, or may be configured by combining multiple algorithms.

[0078] The second learning model Mb includes an input layer to which biometric information is input, an output layer that outputs a puncture success rate corresponding to a predetermined deviation between the target puncture position and the planned puncture position, and an intermediate layer that calculates an output value based on the input data. The output layer of the second learning model Mb has multiple (four in this example) output nodes to which selectable values ​​(e.g., 80% or more, 60% or more, 40% or more, less than 40%) of the puncture success rate are associated for each of the predetermined deviation amounts. Each output node outputs a confidence level for each associated puncture success rate. For each deviation amount, each output node outputs an output value, for example, between 0 and 1, and the sum of the output values ​​from each output node is 1 (100%). With this configuration, the second learning model Mb can estimate the puncture success rate associated with the output node that outputs the largest output value (confidence level) for each deviation amount as the success rate when a puncture procedure is performed at each deviation amount. By using the second learning model Mb configured in this way, the information processing device 10 can identify the puncture success rate for each deviation amount based on each output value output from the second learning model Mb when the subject's biometric information is input. Note that the second learning model Mb may be configured to have a single output node that outputs the puncture success rate with the highest confidence level, instead of having multiple output nodes that output the confidence level for each puncture success rate for each deviation amount.

[0079] The second learning model Mb is generated by machine learning using training data that associates training biometric information with the correct puncture success rate for each deviation between the target puncture position and the planned puncture position. The correct puncture success rate can be determined based on the results (success or failure) of a medical professional performing a puncture procedure on a subject with the training biometric information at each deviation. When the second learning model Mb receives input biometric information from the training data, it learns so that, for each deviation, the output value from the output node corresponding to the correct puncture success rate approaches 1 and the output values ​​from the other output nodes approach 0. During the learning process, the second learning model Mb performs calculations based on the input biometric information to calculate output values ​​from each output node. The second learning model Mb then compares the calculated output value of each output node for each deviation amount with a value corresponding to the correct puncture success rate (1 for the output node corresponding to the correct puncture success rate, and 0 for the other output nodes), and optimizes the parameters used in the calculation process so that the two values ​​approximate each other. Again, parameters such as the weights (coupling coefficients) between nodes in the second learning model Mb are optimized using backpropagation, steepest descent, or the like. This allows for the acquisition of a second learning model Mb that, when a subject's biometric information is input, outputs the puncture success rate when a puncture operation is performed on the subject at each deviation amount.

[0080] The learning models Ma and Mb may be learned by another learning device. The learned learning models Ma and Mb generated by learning on the other learning device are downloaded from the learning device to the information processing device 10 via a network or via the portable storage medium 10a and stored in the storage unit 12.

[0081] Fig. 13 is a flowchart showing an example of the procedure for supporting the puncture operation according to embodiment 4. The process shown in Fig. 13 is the same as the process shown in Fig. 4 except that steps S41 to S42 are added before step S11 and steps S43 to S44 are added instead of step S22. Explanation of the same steps as in Fig. 4 will be omitted. Note that steps S11 to S20 in Fig. 4 are not shown in Fig. 13.

[0082] In the information processing device 10 of this embodiment, the control unit 11 first acquires biometric information of the person to be punctured (S41). The biometric information may be manually input by the surgeon via the input unit 14, or may be acquired by the control unit 11 via the communication unit 13 from an electronic medical record server or the like via a network. The biometric information may include, for example, the age and gender of the person, as well as information such as the person's diagnosis, medical history, edema status, height, and weight. If the input data of the first learning model Ma includes environmental information, the control unit 11 may acquire the temperature and humidity of the location where the puncture is to be performed, for example, from a thermometer and hygrometer connected to the input unit 14.

[0083] Based on the acquired biometric information, the control unit 11 sets a threshold for the amount of deviation between the target puncture position and the planned puncture position during the puncture operation on the subject (S42). Here, the control unit 11 inputs the biometric information into the first learning model Ma and acquires the threshold for the amount of deviation as an output value from the first learning model Ma. Specifically, the control unit 11 identifies the output node that outputs the largest output value (certainty) among the output values ​​from each output node of the first learning model Ma, and estimates and sets the numerical value associated with the identified output node as the threshold for the amount of deviation. The control unit 11 may also estimate the threshold for the amount of deviation using the second learning model Mb. In this case, the control unit 11 inputs the biometric information into the second learning model Mb, identifies the output node that outputs the largest output value (certainty) among the output nodes corresponding to each amount of deviation, and estimates the puncture success rate associated with the identified output node as the puncture success rate for each amount of deviation. The control unit 11 then sets the deviation amount at which the puncture success rate is estimated to be 80% or higher as the threshold for the deviation amount during the puncture operation. For example, if the puncture success rate for the first deviation amount and the second deviation amount is estimated to be 80% and the puncture success rate for the third deviation amount is estimated to be less than 40%, the control unit 11 sets the second deviation amount as the threshold. The process of estimating the threshold for the deviation amount from the biometric information is not limited to the process using the learning models Ma and Mb, but may also be a rule-based process. For example, by previously associating the biometric information with the threshold for the deviation amount, it is possible to estimate the threshold for the deviation amount corresponding to the biometric information of the subject.

[0084] After processing step S42, the control unit 11 proceeds to processing step S11 and executes the processing of steps S11 to S21. The control unit 11 then determines whether the amount of deviation calculated in step S21 is equal to or greater than the threshold value set in step S42 (S43). If the control unit 11 determines that the amount of deviation is equal to or greater than the threshold value (S43: YES), it outputs a message such as "There is a large deviation from the target position" or "Please try again" to warn the surgeon (outputs an alert) (S44). If the control unit 11 determines that the amount of deviation is less than the threshold value (S43: NO), it skips processing step S44 and terminates the series of processes. Note that if an alert is output, the surgeon starts over from the palpation or puncture operation, and the control unit 11 repeats the processing of steps S12 to S22. If the control unit 11 determines that the amount of deviation is less than the threshold value (S43: NO), it may output a message such as "Close to the target position" or "Let's start puncture."

[0085] Through the above-described processing, in this embodiment, the threshold value for the amount of deviation between the target puncture position and the planned puncture position, which is used to determine whether to output an alert when performing a puncture operation, can be switched according to the biometric information of the person to be punctured. For example, as blood vessels become less elastic with age, it is necessary to perform the puncture operation while capturing the center of the blood vessel, and therefore a stricter threshold value for the amount of deviation between the target puncture position and the planned puncture position must be set. In this embodiment, the threshold value for the amount of deviation can be switched according to the subject's biometric information, enabling a more appropriate alert to be output depending on the subject. The configuration of this embodiment is applicable to the puncture support systems 100 of the first to third embodiments described above. Even when applied to the puncture support systems 100 of the first to third embodiments, similar processing can be performed and similar effects can be obtained. Furthermore, the modified examples described in the first to third embodiments described above can also be applied to this embodiment.

[0086] In this embodiment, the learning models Ma and Mb are used to determine the threshold for the deviation between the target puncture position and the planned puncture position. Additionally, the learning model may be used to determine the operator's gaze time (e.g., 2 or 3 seconds) at the gaze point, which is used as a condition for setting the target puncture position (target setting condition). For example, the state of the operator's palpation movement may be detected based on a landscape image, and the gaze time used as the target setting condition may be determined based on the state of the palpation movement. Specifically, a learning model may be used that has been trained to output the gaze time used as the target setting condition when the operator inputs information such as the time required for the palpation movement (elapsed time) and the agility of the palpation movement. In this case, the target setting condition can be switched depending on the state of the operator's palpation movement, enabling support processing tailored to the operator.

[0087] Fifth Embodiment A puncture support system will be described that outputs an alert when a deviation of a predetermined distance or more exists between the palpation position palpated by an operator during a palpation operation on a living body 40 and a set target puncture position. The puncture support system of this embodiment can be realized by devices similar to the puncture support system 100 of the first embodiment shown in Fig. 1 , and therefore a description of the configuration of each device will be omitted.

[0088] Fig. 14 is a flowchart showing an example of the puncture operation support processing procedure of embodiment 5, and Fig. 15 is an explanatory diagram of the puncture operation support processing. The processing shown in Fig. 14 is the processing shown in Fig. 4 with step S51 added between YES in step S13 and step S14, and steps S52 to S54 added between steps S16 and S17. Explanations of the same steps as in Fig. 4 will be omitted. Note that steps S18 to S22 in Fig. 4 are not shown in Fig. 14.

[0089] The control unit 11 of the information processing device 10 of this embodiment executes a detection process to determine whether the surgeon is performing a palpation operation on the living body 40 based on the scenery image (S12). If it is determined that the surgeon has started the palpation operation (S13: YES), the control unit 11 identifies the surgeon's palpation position on the living body 40 (S51). FIG. 15 shows an example of a scenery image captured during a palpation operation. The control unit 11 identifies the surgeon's fingers in the scenery image, and if the surgeon's fingers do not move relative to the living body 40 for a predetermined time (e.g., 2 or 3 seconds) or more, the control unit 11 identifies the identified finger position as the palpation position. In the example of FIG. 15, the surgeon is performing a palpation operation with his thumb, and the center of the first joint of the thumb is identified as palpation position P3. The surgeon's fingers in the scenery image can be identified using pattern matching, a learning model trained by machine learning to extract the surgeon's fingers in the scenery image, or the like.

[0090] Thereafter, the control unit 11 executes an eye tracking process (S14) and sets a target puncture position (S16). After the process of step S16, the control unit 11 calculates the amount of deviation (third deviation) between the palpation position identified in step S51 and the target puncture position identified in step S16 (S52). The example of Fig. 15 shows a state in which the fourth gaze point detected by the eye tracking process is set to the target puncture position P1, and the control unit 11 calculates the linear distance (number of pixels) of the line segment connecting the palpation position P3 and the target puncture position P1 as the amount of deviation.

[0091] The control unit 11 determines whether the calculated amount of deviation is equal to or greater than a predetermined value (S53). If it determines that the amount of deviation is equal to or greater than the predetermined value (S53: YES), it outputs a message such as "Look at the location where you are about to puncture" to warn the surgeon (outputs an alert) (S54). If the control unit 11 determines that the amount of deviation is less than the predetermined value (S53: NO), it proceeds to the processing of step S17. If an alert is output, the surgeon starts over from the palpation operation. Therefore, after processing step S54, the control unit 11 proceeds to the processing of step S12 and repeats the processing from step S12 onwards.

[0092] According to the above-described processing, in this embodiment, if the palpation position where the surgeon palpates the living body 40 deviates by a predetermined distance or more from the target puncture position set based on the surgeon's viewpoint position relative to the living body 40, an alert is output, thereby notifying the surgeon that there is a high possibility that the puncture operation will fail. The surgeon searches for the blood vessel to be punctured by palpating the living body 40, and the palpation position is likely to be on the blood vessel. Therefore, by moving the target puncture position closer to the palpation position, the target puncture position can be set on the blood vessel, thereby improving the accuracy of setting the target puncture position.

[0093] The configuration of this embodiment can be applied to the puncture support systems 100 of the above-mentioned embodiments 1 to 4, and similar processes can be executed and similar effects can be obtained even when applied to the puncture support systems 100 of the above-mentioned embodiments 1 to 4. Furthermore, the modified examples described as appropriate in the above-mentioned embodiments 1 to 4 can also be applied to this embodiment.

[0094] In the above-described process, the predetermined value (threshold value for the distance between the palpation position P3 and the target puncture position P1) used to determine whether to output an alert in step S53 may be, for example, 5 mm, 1 cm, etc., but may be set to any value. Furthermore, the threshold value may be switchable depending on attribute information such as the age and gender of the subject.

[0095] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0096] The features described in each of the above-mentioned embodiments can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations, regardless of the reference format. Furthermore, while the claims use a format in which a claim references two or more other claims (multiple claim format), this is not limited to this format. A multiple claim (multi-multi claim) that references at least one other multiple claim may also be used.

[0097] REFERENCE SIGNS LIST 10 Information processing device 11 Control unit 12 Memory unit 13 Communication unit 16 Speaker 20 Wearable device 21a, 21b Pupil camera 22 Scenery camera 23 Communication unit 30 Indwelling needle M1, M2 Marker

Claims

1. A program that causes a computer to execute the following processes: detect the user's viewpoint position relative to a living body; identify a target puncture position relative to the living body based on the user's viewpoint position; detect the position of the tip of a puncture instrument relative to the living body; and calculate the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

2. The program according to claim 1, which causes the computer to execute a process of outputting an alert according to the calculated amount of deviation.

3. The program according to claim 1 or 2, which causes the computer to execute a process of detecting the user's palpation action on the living body, and identifying the target puncture position based on the user's viewpoint position during the palpation action.

4. The program according to claim 1 or 2, which causes the computer to execute the process of detecting the operation of the puncture device, and, when it is detected that the operation of the puncture device has stopped, detecting the position of the tip of the puncture device relative to the living body.

5. The program according to claim 1 or 2, which causes the computer to execute the following processes: acquiring a blood vessel visualization image that visualizes the blood vessels of the living body; calculating a second amount of deviation between the blood vessels in the blood vessel visualization image and the target puncture position; and outputting an alert according to the second amount of deviation.

6. The program according to claim 1 or 2, which causes the computer to execute the following process: the puncture device has a marker provided at a predetermined position; the position of the marker on the puncture device is detected; and the position of the tip of the puncture device is detected based on the position of the marker.

7. The program according to claim 2, which causes the computer to execute the following process: inputting the user's biometric information into a learning model that has been trained to output a threshold value used to determine whether to output an alert regarding the amount of deviation between the target puncture position and the position of the tip of the puncture instrument when biometric information about the living body is input; obtaining from the learning model the threshold value of the amount of deviation used to determine whether to output an alert; and outputting an alert if the amount of deviation between the target puncture position and the position of the tip of the puncture instrument is equal to or greater than the threshold value of the amount of deviation.

8. The program according to claim 1 or 2, which causes the computer to execute the following process: detect the user's palpation position relative to the living body; calculate a third deviation amount between the palpation position and the target puncture position; and output an alert if the third deviation amount is equal to or greater than a predetermined value.

9. An information processing method in which a computer executes the processes of: detecting a user's viewpoint position relative to a living body; identifying a target puncture position relative to the living body based on the user's viewpoint position; detecting the position of a tip of a puncture device relative to the living body; and calculating the amount of deviation between the target puncture position and the position of the tip of the puncture device.

10. An information processing device having a control unit, wherein the control unit detects a user's viewpoint position relative to a living organism, identifies a target puncture position relative to the living organism based on the user's viewpoint position, detects the position of a tip of a puncture instrument relative to the living organism, and calculates the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

11. An information processing system comprising an information processing device having a control unit, a first camera that captures an image of a user's eyes, and a second camera that captures an image of the user's line of sight, wherein the control unit: detects the user's viewpoint position relative to a living organism based on an image captured by the first camera and an image captured by the second camera; identifies a target puncture position relative to the living organism based on the user's viewpoint position; detects the position of the tip of a puncture instrument relative to the living organism based on the image captured by the second camera; and calculates the amount of deviation between the target puncture position and the position of the tip of the puncture instrument.

Images