Photoacoustic imaging apparatus and photoacoustic imaging method

Abstract

We propose a photoacoustic imaging device and method capable of generating highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in deep regions of a target area, with significantly higher accuracy and while ensuring the safety of the subject, for target tissues or organs within the subject. [Solution] The control unit sets the scan path of the light source probe unit to encompass the target area specified by the vision unit, and guides and controls the tip of the robot tube according to the scan path.

Description

Technical Field

[0001] The present invention relates to a photoacoustic imaging technique for detecting and imaging photoacoustic waves generated from a photoabsorber in a subject by irradiating the subject with light.

Background Art

[0002] In recent years, as a photoacoustic imaging technique and a photoacoustic tomography technique, a photoacoustic wave imaging device including an LED light source unit that irradiates a subject with pulsed light and an ultrasonic detector that detects acoustic waves as ultrasonic waves generated by an object in the subject has been proposed (see Citation Document 1).

[0003] This photoacoustic wave imaging device irradiates an object such as a blood vessel with light and measures the shape of the object non-contact at the resolution of the image level by the ultrasonic waves emitted by the excited object.

[0004] Conventionally, minimally invasive diagnostic and surgical tools such as endoscopes and laparoscopes in the medical field provide surgical access to the target site while minimizing trauma to the patient. For example, an endoscope is inserted into a body cavity such as the stomach or intestine and is used for observing the tissue surface in the body cavity, diagnosing by collecting diseased pieces with forceps, etc., and treatment.

[0005] In recent years, as an endoscope and a laparoscope in the medical field, a robotic system by remote operation has been proposed to enable surgical access through a natural opening or a minimally invasive route (see Citation Document 2).

[0006] This robotic system has a tool capable of directing at least one degree of freedom to the distal end via an elongated main body extending between the proximal end and the distal end using a control device that mechanically transmits user input to the distal operation segment, and is a direct drive endoscope method system in which the user can simultaneously control multiple degrees of freedom in combination with a frame that is movably connected to the tool and directs at least one degree of freedom.

Prior Art Documents

[0007] [Patent Document 1] Japanese Patent Publication No. 2015-29550 [Patent Document 2] Patent No. 6085530 [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Incidentally, it is conceivable to combine a remotely controlled robotic system, as described in Reference 2, with photoacoustic imaging technology. Specifically, a light source integrated probe of a photoacoustic wave imaging device is mounted on the end effector, which is the distal end of the robotic system, and the light source integrated probe is scanned over the target area of ​​the subject in response to remote control by the examiner.

[0009] However, actually mounting a light source-integrated probe on the end effector of a robotic system and scanning the target area of ​​the subject while ensuring that the distance from the surface is uniform requires a high level of technical skill, even if the examiner is dexterous.

[0010] In particular, imaging deep internal organs such as the liver, stomach, and intestines, which have traditionally been considered impossible to image beyond the skin surface area (a depth of about 15 mm from the skin), has been extremely difficult to achieve with conventional robotic systems. This requires the ability to scan the abdomen percutaneously with a light source-integrated probe while freely controlling it without damaging or pressing it against the organs.

[0011] This invention has been made in consideration of the above points, and aims to propose a photoacoustic imaging apparatus and a photoacoustic imaging method that can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep regions of a target area with significantly higher accuracy and while ensuring the safety of the subject, for target tissues or organs within the subject. [Means for solving the problem]

[0012] To solve these problems, the present invention provides a photoacoustic imaging device that detects photoacoustic waves generated from a light absorber in a subject and images the light absorber based on said photoacoustic waves, comprising: a robot tube configured to be bendable and extendable in the direction of travel from its base to its tip, and capable of being driven to guide its tip in a desired direction and position in response to external operation or autonomously; a light source probe unit provided at the tip of the robot tube, which integrally includes a light irradiation unit that irradiates pulsed light of a wavelength absorbed in the subject, and a photoacoustic wave detection unit that detects photoacoustic waves generated from the light absorber; a vision unit that, based on the image content of the adjacent area at the tip of the robot tube captured by an imaging sensor attached to the light source probe unit, specifies a target area of ​​the tissue or organ to be examined in the subject, either by external input or automatically; and a control unit connected to the base of the robot tube for controlling the robot tube, the light source probe unit, and the vision unit, respectively, wherein the control unit sets the scan path of the light source probe unit to include the target area specified by the vision unit, and guides and controls the tip of the robot tube according to the scan path.

[0013] As a result, in the photoacoustic imaging device, when a target area of ​​tissue or organ to be examined within a subject is specified, the scan path of the light source probe unit is set to include the target area, and the tip of the robot tube is guided and controlled according to that scan path. This allows the device to scan the target tissue or organ within the subject with a precision that is even higher than manual scanning, and generates highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target area.

[0014] Furthermore, in the present invention, the vision unit includes a bio-element recognition unit that sequentially recognizes tissues or organs corresponding to a target region based on the image content of the target region captured by the imaging sensor, while referring to a bio-element estimation model constructed by deep learning using surface feature patterns classified for each human tissue or organ as training data.

[0015] As a result, when specifying a target area of ​​tissue or organ to be examined within a subject, the photoacoustic imaging device can significantly improve the recognition accuracy and speed of the tissue or organ corresponding to that target area.

[0016] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject, and the control unit guides and controls the tip of the robot tube so that the distance to the surface of the biological part is maintained within a certain range based on the three-dimensional shape data detected by the RGB-D sensor.

[0017] As a result, the photoacoustic imaging device can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in a nearly homogeneous tracing state when guiding and controlling the tip of the robot tube according to the scanning path of the light source probe unit, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0018] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject; the vision unit has an internal motion detection unit that detects the pulsation of tissue or peristalsis of organs corresponding to a target region sequentially recognized by the biological element recognition unit based on the time change of the three-dimensional shape data detected by the RGB-D sensor; and the control unit is configured to guide and control the tip of the robot tube while making minute fluctuations in the opposing distance to the surface of the biological part in synchronization with the pulsation of tissue or peristalsis of organs detected by the internal motion detection unit.

[0019] As a result, the photoacoustic imaging device can generate highly reproducible, real-time, three-dimensional photoacoustic wave images of peripheral blood vessels in a more uniform tracing state when guiding and controlling the tip of the robot tube according to the scanning path of the light source probe unit, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0020] Furthermore, in the present invention, the vision unit includes a photoacoustic wave image generation unit that generates a photoacoustic wave image based on photoacoustic waves detected by a photoacoustic wave detection unit in a light source probe unit, and an abnormality state detection unit that detects abnormalities in tissues or organs corresponding to target regions sequentially recognized by a biological element recognition unit based on the photoacoustic wave image generated by the photoacoustic wave image generation unit.

[0021] In this way, photoacoustic imaging devices can detect abnormal conditions (such as heterogeneity or the presence or absence of specific tissues) in the target area of ​​tissues or organs based on photoacoustic wave images. This allows imaging to extend to areas deeper than the target surface area (approximately 15 mm) as is possible with endoscopes and laparoscopes, enabling more advanced imaging diagnostics.

[0022] Furthermore, in this invention, the vision unit includes a disease symptom estimation unit that estimates the disease and symptoms of a tissue or organ based on the abnormal state of the tissue or organ detected by the abnormal state detection unit, while referring to a disease content estimation model constructed by deep learning using abnormal state feature patterns classified for each human tissue or organ as training data.

[0023] In this way, photoacoustic imaging devices can estimate diseases and symptoms of tissues or organs based on the abnormal state of those tissues or organs, and by analyzing that abnormal state, it becomes possible to diagnose diseases and symptoms of tissues and organs that could not be detected by imaging of the target surface area alone, such as with endoscopy or laparoscopy, with relatively high accuracy.

[0024] Furthermore, in the present invention, the imaging sensor has an RGB-D sensor for detecting three-dimensional shape data including the proximity distance from the detection outlet of the acoustic wave detection unit to the surface of the biological site in the subject, and the vision unit is based on the temporal change of the three-dimensional shape data detected by the RGB-D sensor. While detecting the pulsation of the arterial blood vessels of the subject, from the fluctuation width between the peak and the bottom of the fluctuation component reflecting the pulsation of the arterial blood vessels, a blood perfusion determination unit for determining the ease of change of the blood perfusion of the arterial blood vessels is further provided. The disease symptom estimation unit reflects the ease of change of the blood perfusion of the arterial blood vessels determined by the blood perfusion determination unit in the analysis of the abnormal state of the tissue or organ.

[0025] Thus, in the photoacoustic imaging device, it is also possible to detect abnormal states of tissues and organs caused by the ease of change of blood perfusion in the arterial blood vessels of the subject.

[0026] Furthermore, in the present invention, the vision unit includes a photoacoustic wave image generation unit that generates a photoacoustic wave image based on the photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit, and a photoacoustic wave image evaluation unit that evaluates the sharpness of the photoacoustic wave image generated by the photoacoustic wave image generation unit for each scan path. When the sharpness of the photoacoustic wave image evaluated by the photoacoustic wave image evaluation unit is below a predetermined level, the control unit causes the light source probe unit to scan while feedback-inductively controlling the tip of the robot tube according to the corresponding scan path.

[0027] Thus, in the photoacoustic imaging device, when the photoacoustic wave image of the tissue or organ corresponding to the target area becomes unclear as a result of micro-vibrations such as tremors and spasms occurring in the subject itself during the sliding movement of the light source probe unit, by automatically scanning again, a three-dimensional photoacoustic wave image of the peripheral blood vessels can be stably generated even for a three-dimensional shape such as an organ.

[0028] Furthermore, in the present invention, in a photoacoustic imaging method that detects photoacoustic waves generated from a light absorber in a subject and images the light absorber based on said photoacoustic waves, a light source probe unit is provided at the tip of a robot tube that is bendable from its base to its tip and expandable and contractible in the direction of travel, and can be driven to guide its tip in a desired direction and position in response to external operation or autonomously. The probe unit integrally includes a light irradiation unit that irradiates pulsed light of a wavelength absorbed in the subject and a photoacoustic wave detection unit that detects photoacoustic waves generated from a light absorber. The method comprises a first step of specifying, by external input or automatically, a target area of ​​the tissue or organ to be examined in the subject, based on the image content of the adjacent area at the tip of the robot tube, which is imaged by an imaging sensor attached to the light source probe unit, and a second step of setting a scan path of the light source probe unit to include the target area specified in the first step, and guiding and controlling the tip of the robot tube according to the scan path.

[0029] As a result, in the photoacoustic imaging method, when a target area of ​​tissue or organ to be examined within a subject is specified, the scan path of the light source probe unit is set to include that target area, and the tip of the robot tube is guided and controlled according to that scan path. This allows the robot tube to scan the target tissue or organ within the subject with a higher precision than manual scanning, and generates highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target area. [Effects of the Invention]

[0030] According to the present invention, it is possible to realize a photoacoustic imaging apparatus and a photoacoustic imaging method that can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of a target region of tissue or organ with significantly higher accuracy than conventional methods while ensuring safety for the subject. [Brief explanation of the drawing]

[0031] [Figure 1] This is an external perspective view showing the configuration of the photoacoustic imaging device according to this embodiment. [Figure 2] Figure 1 is a schematic diagram showing the configuration of the robot tube. [Figure 3] Figure 1 is a conceptual diagram showing the configuration of the light source probe unit. [Figure 4] This is a conceptual diagram representing the model structure for deep learning. [Figure 5] This is a block diagram showing the internal configuration of the control system in a photoacoustic imaging device. [Figure 6] This is a conceptual diagram showing the functional configuration of the vision unit in another embodiment. [Modes for carrying out the invention]

[0032] An embodiment of the present invention will be described in detail below with reference to the drawings.

[0033] (1) Configuration of the photoacoustic imaging device according to this embodiment Figure 1 is a schematic external view of the photoacoustic imaging apparatus 1 according to this embodiment. This photoacoustic imaging apparatus 1 consists of a robot tube 2, a light source probe unit 3 provided at the tip of the robot tube 2, a vision unit 4 attached to the light source probe unit 3, and a control unit 5 for controlling the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0034] The robot tube 2 is configured to be flexible and extendable in the direction of travel from its base to its tip, and is configured to be driveable to guide its tip to a desired direction and position in response to external operation or autonomously.

[0035] The light source probe unit 3 is located at the tip of the robot tube 2 and integrally includes a light irradiation unit 10 that irradiates pulsed light of a wavelength absorbed within the subject, and a photoacoustic wave detection unit 11 that detects photoacoustic waves generated from a light absorber.

[0036] The vision unit 4, based on the imaging data of the adjacent area at the tip of the robot tube 2, which is captured by the imaging sensor 12 attached to the light source probe unit 3, specifies the target area of ​​the tissue or organ to be examined in the subject, either via external input or automatically.

[0037] The vision unit 4 has only the imaging sensor 12 attached to the tip of the robot unit 2, alongside the light source probe unit 3; all other functional components are housed within the same housing as the control unit 5. The imaging sensor 12 is electrically connected to the main body of the vision unit 4 via the robot unit 2 and the universal code 15.

[0038] The vision unit 4 is connected to the base end of the robot tube 2 via a universal cord 15 and has a photoacoustic wave image generation unit 20 that processes a light absorber with photoacoustic waves detected by the light source probe unit 3.

[0039] Furthermore, the vision unit 4 is connected to an image display unit 21, which consists of a liquid crystal monitor or the like, capable of simultaneously displaying the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 and the image captured by the imaging sensor 12.

[0040] The control unit 5 is housed in the same enclosure as the vision unit 4 and is connected to the base end of the robot tube 2, and is configured to control the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0041] (2) Robot tube configuration As shown in Figure 2(A), one end (base end) of the retractable mechanism 30 of the robot tube 2 is connected to the bending mechanism 31 via a universal cord 15 drawn out from an input / output connector connected to the control unit 5, and the other end of the retractable mechanism 30 is connected to the bending mechanism 31.

[0042] The telescopic mechanism 30 is mounted so that the shaft 33 can slide coaxially and rotate coaxially within the tubular holder 32. Specifically, the telescopic mechanism 30 is configured such that the shaft 33 can slide linearly within the tube relative to the tubular holder 32 in response to the drive of a sliding mechanism (for example, a rack and pinion mechanism or a worm gear mechanism), and the shaft 33 can rotate coaxially with respect to the tubular holder 32 in response to the drive of a rotation mechanism (for example, a planetary gear mechanism) (Figure 2(B)).

[0043] The bending mechanism 31 is a structure that partially applies the motion support device described in Patent No. 6734927 of the present applicant. Specifically, the bending mechanism 31 has a multi-joint structure 35 in which series links are connected so as to be rotatable relative to each other, and all links together can be deformed to bend freely. The rear link is slid in the direction of connection between each link via a linear member inserted inside each link, and the sliding direction, sliding speed and sliding position are driven to reach a desired state, thereby causing the multi-joint structure 35 to bend (extend or flex) (Figure 2(C)).

[0044] In the bending mechanism 31, the base side that holds the articulated structure 35 is connected to the telescopic mechanism 30, and a light source probe unit 3 is provided as an end effector at the leading link (tip) of the articulated structure 35.

[0045] In this way, the robot tube 2 can guide the light source probe unit 3, which acts as an end effector, in a desired direction and position while driving it to bend freely and extend and retract freely in the direction of travel.

[0046] (3) Configuration of the light source probe unit As shown in Figure 3, the light source probe unit 3 is provided as an integrated unit comprising a light irradiation unit 10 that irradiates pulsed light of a wavelength absorbed inside the subject, and a photoacoustic wave detection unit 11 that detects photoacoustic waves generated from a light absorber.

[0047] In this embodiment, the light source probe unit 3 is configured such that a pair of light irradiation units 10 sandwich the photoacoustic wave detection unit 11, allowing inspection to be performed in either the forward or reverse scanning direction.

[0048] Furthermore, the light source probe unit 3 has its irradiation port of the light irradiation section 10 and its detection port of the photoacoustic wave detection section 11 attached to the tip of the bending mechanism section 31 of the robot tube 2, along its longitudinal direction (the direction in which the links are connected).

[0049] The light irradiation unit 10 consists of multiple light-emitting diode elements (not shown) connected in series and arranged along the longitudinal direction (axial direction). The light irradiation unit 10 irradiates the subject with each pulse of light, which has an infrared wavelength (for example, a wavelength of approximately 850 nm) from each of the multiple light-emitting diode elements in response to the current supplied from the light source drive unit 40 (Figure 5).

[0050] Furthermore, by pre-setting the wavelength of pulsed light according to the type of object being examined (light absorber such as hemoglobin, blood vessels, nerve tissue, lymph, tumors, etc.), it is possible to image only the desired object.

[0051] Then, the pulsed light irradiated onto the subject from the light irradiation section 10 of the light source probe unit 3 is absorbed by an object (light absorber) within the subject. The object expands and contracts (returns from its expanded size to its original size) in accordance with the irradiation intensity (amount absorbed) of the pulsed light, thereby generating photoacoustic waves from the object.

[0052] In this way, when the light irradiation unit 10 irradiates the target object (blood vessels, nerve tissue, tumor, etc.) of the subject in the light source probe unit 3, specific molecules in the body become excited due to the absorption of light by the object, and heat is generated when they return from the excited state to the steady state. The photoacoustic wave detection unit 11 detects the photoacoustic waves (photosonic waves) of molecules generated by the temperature difference (thermal expansion) in the surrounding area.

[0053] (4) Configuration of the Vision Unit As shown in Figure 3 above, the vision unit 4 has an imaging sensor 12 attached to the light source probe unit 3. The imaging sensor 12 captures images of the nearby area at the tip of the robot tube 2, and based on the captured images, it specifies the target area of ​​the tissue or organ (liver, stomach, intestines, etc.) to be examined in the subject, either via external input or automatically.

[0054] The imaging sensor 12 has an RGB-D sensor 41 for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit 11 to the surface of a biological part within the subject. The RGB-D sensor 41 is a sensor that can acquire depth information from the sensor to each point in the image, in addition to texture information similar to that of a general imaging camera, and can obtain three-dimensional information of the imaging environment.

[0055] Furthermore, the vision unit 4 includes a photoacoustic wave image generation unit 20 (Figure 5) that generates a photoacoustic wave image based on the photoacoustic waves detected by the photoacoustic wave detection unit 11 in the light source probe unit 2.

[0056] This vision unit 4 includes a bio-element recognition unit 50 (Figure 5) as a means for performing functional processing on the image content based on the imaging sensor 12.

[0057] The biological element recognition unit 50 uses surface feature patterns classified by human tissue or organ as training data and, while referring to a biological element estimation model constructed by deep learning, sequentially recognizes the tissue or organ corresponding to the target region based on the image content of the target region captured by the imaging sensor.

[0058] Specifically, the bio-element recognition unit 50 applies a bio-element estimation model consisting of three modules—a convolutional neural network (CNN) layer, a batch normalization layer, and an activation function layer (tanh function)—and a fully connected layer, as shown in Figure 4, in order to recognize the tissue or organ of the subject.

[0059] In a biocomponent estimation model, partial surface images of tissues or organs from the current frame and the two frames prior can be input to calculate the likelihood for multiple types of biocomponents related to those tissues or organs.

[0060] Furthermore, the biological element recognition unit 50, as a recognition result, post-processes the data by identifying the most frequently observed biological element among the 15 frames (the current frame and the previous 14 frames) as the current tissue or organ. In this way, the biological element recognition unit 50 learned a biological element estimation model using supervised learning with its own dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0061] As a result, the photoacoustic imaging device 1 can significantly improve the recognition accuracy and speed of tissues or organs corresponding to the target area when specifying a target area of ​​tissue or organ to be examined within a subject.

[0062] (5) Internal configuration of the control system in the photoacoustic imaging device Figure 5 shows the internal configuration of the control system in the photoacoustic imaging device 1. The control unit is responsible for the overall control of the photoacoustic imaging device 1 and is mainly composed of a microcomputer consisting of an MCM (Multi-Chip Module) equipped with a CPU (Central Processing Unit) and memory.

[0063] The control unit 5 controls the robot tube 2, the light source probe unit 3, and the vision unit 4, respectively.

[0064] The control unit 5 sets the scan path of the light source probe unit 3 so as to encompass the target area of ​​the subject's tissue or organ designated by the vision unit 4. In other words, the control unit 5 sets a scan path for each target area so that the target area of ​​the subject's tissue or organ to be examined, which is externally input or automatically designated by the vision unit 4, is evenly covered, and so that the light source probe unit 3 moves along the surface of the target area in a nearly non-contact manner.

[0065] This scan path is set as a path in a three-dimensional coordinate system, centered on the target area of ​​the subject's tissue or organ, based on the three-dimensional information of the imaging environment obtained by the imaging sensor 12 (RGB-D sensor 41) in the vision unit 4. The paths are mutually orthogonal to each other, with the direction of travel and the perpendicular direction of the light source probe unit 3 as mutually orthogonal axes.

[0066] Furthermore, the control unit 5 sends a trigger signal to the light source drive unit 40 in the light source probe unit 3 to drive and control the light irradiation. The light source drive unit 40 generates a DC current from power supplied from an external power source (not shown) and, based on the pulsed trigger signal from the control unit 5, switches a switch consisting of, for example, an FET (Field Effect Transistor) on or off, thereby emitting pulsed light of a predetermined width and frequency from multiple light-emitting diode elements.

[0067] Furthermore, the control unit 5 sends a sampling trigger signal to the photoacoustic wave image generation unit 20 in the vision unit 4, causing it to generate an image with a resolution corresponding to the target object (a tomographic image based on acoustic waves) in real time, and display the image on the image display unit 21.

[0068] The control unit 5 controls the telescopic mechanism 30 of the robot tube 2 to slide and rotate the bending mechanism 31, and while controlling the bending mechanism 31 to bend the light source probe unit 3 provided at its tip, it guides it to cover the target area of ​​the subject's tissue or organ according to the set scan path.

[0069] When the control unit 5 guides and controls the light source probe unit 3, it rotates the extension / retraction mechanism 30 of the robot tube 2 to orient the irradiation port of the light irradiation section 10 and the detection port of the photoacoustic wave detection section 11 of the light source probe unit 3 in the desired direction.

[0070] In other words, when the light source probe unit 3 is slid, the control unit 5 controls the orientation of the light source probe unit 3 so that the incident angle of the light irradiation section 10 in the light source probe unit 3 remains perpendicular to the sliding direction.

[0071] As a result, the photoacoustic imaging device 1 can stably generate three-dimensional photoacoustic wave images of peripheral blood vessels, even if the surface has a three-dimensional shape, by always irradiating the target area of ​​the subject's tissue or organ at a perpendicular angle.

[0072] (6) Operation and effects of the photoacoustic imaging device In the above configuration, when the target area of ​​the tissue or organ to be examined in the subject is specified by an external input or automatically, the control unit 5 sets the scan path of the light source probe unit 3 to include the specified target area and guides and controls the tip of the robot tube 2 according to the scan path.

[0073] As a result, the photoacoustic imaging device 1 can scan target tissues or organs within a subject with significantly higher precision than a human hand, and can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in the deep parts of the target region.

[0074] Furthermore, in the photoacoustic imaging device 1, the bio-element recognition unit 50 in the vision unit 4 uses surface feature patterns classified by human tissue or organ as training data and, while referring to a bio-element estimation model constructed by deep learning, sequentially recognizes the tissue or organ corresponding to the target region based on the image content of the target region captured by the imaging sensor 12.

[0075] As a result, the photoacoustic imaging device 1 can significantly improve the recognition accuracy and speed of tissues or organs corresponding to the target area when specifying a target area of ​​tissue or organ to be examined within a subject.

[0076] Furthermore, in the photoacoustic imaging device 1, the RGB-D sensor 41 of the imaging sensor 12 in the vision unit 4 detects three-dimensional shape data, including the proximity distance from the detection port of the photoacoustic wave detection unit 11 to the surface of the biological tissue within the subject. Based on this detected three-dimensional shape data, the control unit 5 guides and controls the tip of the robot tube 2 so that the distance to the surface of the biological tissue is maintained within a certain range.

[0077] As a result, the photoacoustic imaging device 1 can generate highly reproducible three-dimensional photoacoustic wave images of peripheral blood vessels in a nearly homogeneous tracing state when guiding and controlling the tip of the robot tube 2 according to the scanning path of the light source probe unit 3, which is set to encompass the target area of ​​the tissue or organ to be examined within the subject.

[0078] (7) Other embodiments In the above-described embodiment, the control unit 5 of the photoacoustic imaging apparatus 1 was configured to set the scan path of the light source probe unit 3 so as to include the target area of ​​the subject's tissue or organ designated by the vision unit 4. However, in addition to this, the present invention may also provide the vision unit 4 with a photoacoustic wave image evaluation unit 60 for evaluating the clarity of the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 for each scan path.

[0079] Further details will be described later, but Figure 6 shows the functional configuration of the additional functions of the vision unit 4 of the photoacoustic imaging device 1 in this embodiment. This vision unit 4 has an internal motion detection unit 61 and a vascular latent flow determination unit 62 as means for performing functional processing on the image content based on the imaging sensor 12, and a photoacoustic image evaluation unit 60, an abnormality state detection unit 63 and a disease symptom estimation unit 64 as means for performing functional processing on the photoacoustic wave image based on the photoacoustic wave image generation unit 20.

[0080] Specifically, the photoacoustic wave image evaluation unit 60 analyzes the photoacoustic wave image generated by the photoacoustic wave image generation unit 20 for each scan path to detect peripheral blood vessels within the target area of ​​the tissue or organ of the subject shown in the photoacoustic wave image.

[0081] The photoacoustic wave image evaluation unit 60 can use, for example, so-called deep learning methods, so-called template matching methods, SVM (Support Vector Machine), and machine learning methods such as AdaBoost to detect peripheral blood vessels inside a target region of the subject's tissue or organ.

[0082] Next, the photoacoustic wave image evaluation unit 60 calculates the sharpness of the edges in the images of peripheral blood vessels inside the skin and body tissue at the extremities of the detected subject, and evaluates whether the calculated sharpness is below a predetermined level.

[0083] If the clarity of the photoacoustic wave image evaluated by the photoacoustic wave image evaluation unit 60 is below a predetermined level, the control unit 5 causes the light source probe unit 3 to scan the tip of the robot tube 2 while providing feedback guidance control according to the corresponding scan path.

[0084] The predetermined level of clarity of the photoacoustic wave image at this time is set based on the examiner's visual confirmation, using a level that allows for practically sufficient visual recognition of the three-dimensional photoacoustic image of superficial peripheral blood vessels in the subject's tissue or organ.

[0085] As a result, in the photoacoustic imaging device 1, if the photoacoustic wave image of the tissue or organ corresponding to the target area becomes unclear due to the occurrence of micro-vibrations such as tremors or convulsions in the subject itself when the light source probe unit 3 slides, the device automatically rescans, thereby enabling the stable generation of a three-dimensional photoacoustic wave image of peripheral blood vessels, even if they have a three-dimensional shape such as an organ.

[0086] In addition, the present invention describes a case in which, in the photoacoustic imaging apparatus 1, the control unit 5 guides and controls the tip of the robot tube 2 so that the distance between the detection port of the photoacoustic wave detection unit 11 and the surface of a biological part within the subject is maintained within a certain range, based on three-dimensional shape data detected from the vision unit 4 (RGB-D sensor 41 of the imaging sensor 12). However, the present invention may also provide more delicate guidance control that takes into account the biological activity in the target area of ​​the subject's tissue or organ.

[0087] In other words, in the vision unit 4 of the photoacoustic imaging device 1, an internal motion detection unit 61 is provided in addition to the RGB-D sensor 41 of the imaging sensor 12. This internal motion detection unit 61 detects the pulsation of tissue or peristalsis of organs corresponding to the target region sequentially recognized by the biological element recognition unit 50, based on the time change of the 3D shape data detected by the RGB-D sensor 41 of the imaging sensor 12.

[0088] Specifically, the internal motion detection unit 61 calculates the time required for one heartbeat or peristalsis from the period of the feature points of the waveform detected from the 3D shape data detected by the RGB-D sensor 41, and calculates the number of heartbeats or peristalsis by removing 60 at that time. Compared to a method that calculates the number of heartbeats or peristalsis from frequency analysis, which requires continuous measurement data for a certain period of time, the method that calculates the number of heartbeats or peristalsis from the period of the measured feature points can remove motion artifacts by removing the period of feature points that were falsely detected due to noise caused by body movement. In addition, the minimum measurement time required is shorter than that of frequency analysis, and the number of heartbeats or peristalsis can be calculated from just a few heartbeats or peristalsis, making it possible to calculate the number of heartbeats or peristalsis at shorter intervals.

[0089] The control unit 5 controls the tip of the robot tube 2 by subtly varying the distance to the surface of the living body in synchronization with the pulsation of tissue or peristalsis of organs detected by the internal motion detection unit 61, thereby enabling the generation of highly reproducible 3D photoacoustic wave images of peripheral blood vessels in real time with even more uniform tracing.

[0090] Furthermore, in this embodiment, the photoacoustic imaging device 1 may be equipped with a function to detect abnormalities in the tissues or organs of the subject. Specifically, in the vision unit 4 of the photoacoustic imaging device 1, an abnormality detection unit 63 is provided in addition to the photoacoustic wave image generation unit 20.

[0091] The abnormality detection unit 63 detects abnormalities in tissues or organs corresponding to target regions sequentially recognized by the biological element recognition unit 50, based on the photoacoustic wave image generated by the photoacoustic wave image generation unit 20.

[0092] Specifically, the abnormality detection unit 63 analyzes and interprets the photoacoustic wave image corresponding to the target area of ​​tissue or organ to determine the abnormality occurring in the tissue or organ corresponding to the target area (such as partial heterogeneity of the tissue or organ, the presence or absence of specific tissue, or the presence of tumors occurring in the tissue or organ), and improves the detection accuracy through the learning effect of deep learning.

[0093] Thus, the photoacoustic imaging device 1 detects abnormal conditions in tissues and organs corresponding to the target area detected based on photoacoustic wave images, enabling imaging to areas deeper than the target surface area (approximately 15 mm) as with endoscopes and laparoscopes, thus enabling more advanced imaging diagnostics.

[0094] Furthermore, in addition to the configuration of the abnormal state detection unit 63 described above, the photoacoustic imaging device 1 may also be provided with a disease symptom estimation unit 64 that reflects the detection results from the abnormal state detection unit 63. That is, in the vision unit 4 of the photoacoustic imaging device 1, the disease symptom estimation unit 64 uses abnormal state feature patterns classified for each human tissue or organ as training data, and while referring to a disease content estimation model constructed by deep learning, estimates the disease and symptoms of the tissue or organ based on the abnormal state of the tissue or organ detected by the abnormal state detection unit 63 by analyzing the abnormal state.

[0095] Specifically, the disease symptom estimation unit 64 applies a disease content estimation model consisting of three modules—a convolutional neural network (CNN) layer, a batch normalization layer, and an activation function layer (tanh function)—and a fully connected layer, as shown in Figure 4 above, in order to estimate the disease and symptoms of the subject's tissues or organs.

[0096] The disease content estimation model takes information about the abnormal state of tissues or organs detected by the abnormal state detection unit 63 in the current frame and the frame two frames prior as input, and makes it possible to calculate the likelihood for multiple types of diseases and symptoms in tissues or organs.

[0097] Furthermore, the disease symptom estimation unit 64, as an estimation result, post-processes the most frequently observed abnormal state of tissue or organ among the 15 frames (the current frame and the previous 14 frames) as the disease / symptom related to the current tissue or organ. In this way, the disease symptom estimation unit 64 learned a disease content estimation model using supervised learning with its own dataset. For optimization, a cross-entropy loss function and Adam (adaptive movement estimation) with a learning rate of 0.001 were used.

[0098] As a result, the photoacoustic imaging device 1 can diagnose diseases and symptoms of tissues and organs that could not be detected by imaging of the target surface area alone, such as with endoscopy or laparoscopy, with relatively high accuracy by estimating the disease or symptoms of the tissue or organ based on the abnormal state of the tissue or organ and analyzing the abnormal state of the tissue or organ.

[0099] Furthermore, in this embodiment, the photoacoustic imaging device 1 may be provided with a function to determine how easily the blood perfusion of the subject's arterial vessels can change. Specifically, in the vision unit 4 of the photoacoustic imaging device 1, the vascular latent flow determination unit 62 detects the pulsation of the subject's arterial vessels based on the time change of the three-dimensional shape data detected by the RGB-D sensor 41 of the imaging sensor 12, and determines how easily the blood perfusion of the arterial vessels can change from the range of variation in the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial vessels.

[0100] Specifically, the vascular latent flow determination unit 62, similar to the internal motion detection unit 61 described above, calculates the time required for one arterial pulsation from the period of the waveform feature points detected from the 3D shape data detected by the RGB-D sensor 41, and calculates the number of pulsations by removing 60 at that time.

[0101] Thus, in the photoacoustic imaging device 1, the disease symptom estimation unit 64 in the vision unit 4 reflects the ease with which arterial blood flow perfusion is affected, as determined by the vascular latent flow determination unit 62, in the analysis of abnormal tissue or organ conditions, thereby enabling the analysis of abnormal tissue or organ conditions caused by the ease with which arterial blood perfusion is affected in the subject.

[0102] Furthermore, in the above-described embodiment, the photoacoustic imaging device 1 was configured to image a region (approximately 15 mm) deeper than the target surface region of the tissue or organ detected based on the photoacoustic wave image, primarily to detect the vascular condition. However, the present invention is not limited to this, and by setting the wavelength of the pulsed light emitted from the light irradiation section of the light source probe unit, it may also be possible to image tumors, metabolic processes, and other things that are not visible on the surface of the tissue or organ, which are located in a region deeper than the target surface region of the tissue or organ of the subject.

[0103] Furthermore, in the above-described embodiment, regarding the vision unit 4 of the photoacoustic imaging apparatus 1, the case described was one in which the imaging sensor 12 is positioned on the tip side (end effector side) of the light source probe unit 3. However, the present invention is not limited to this, and the light source probe unit 3 may be positioned on the tip side of the imaging sensor 12, or the light source probe unit 3 and the imaging sensor 12 may be arranged in parallel (in a positional relationship perpendicular to the longitudinal direction). [Explanation of symbols]

[0104] 1... Photoacoustic imaging device, 2... Robot tube, 3... Light source probe unit, 4... Vision unit, 5... Control unit, 10... Light irradiation unit, 11... Photoacoustic wave detection unit, 12... Imaging sensor, 15... Universal code, 20... Photoacoustic wave image generation unit, 21... Image display unit, 30... Telescopic mechanism unit, 31... Bending mechanism unit, 32... Tubular holder, 33... Shaft, 35... Multi-joint structure, 40... Light source drive unit, 41... RGB-D sensor, 50... Biological element recognition unit, 60... Photoacoustic wave image evaluation unit, 61... Intracellular motion detection unit, 62... Blood vessel latent flow determination unit, 63... Abnormal state detection unit, 64... Disease symptom estimation unit.

Claims

1. In a photoacoustic imaging device that detects photoacoustic waves generated from a light absorber within a subject and images the light absorber based on said photoacoustic waves, A robot tube configured to be flexible and extendable in the direction of travel from its base to its tip, and capable of being driven to guide its tip to a desired direction and position in response to external operation or autonomously; A light source probe unit comprising a light irradiation unit provided at the tip of the robot tube and irradiating pulsed light of a wavelength absorbed within the subject, and a photoacoustic wave detection unit for detecting photoacoustic waves generated from the light absorber, is provided as an integral part of the light source probe unit. A vision unit that, based on the image of the area adjacent to the tip of the robot tube captured by an imaging sensor attached to the light source probe unit, specifies the target area of ​​the tissue or organ to be examined in the subject, either via external input or automatically. A control unit connected to the base end of the robot tube, for controlling the robot tube, the light source probe unit, and the vision unit, respectively. Equipped with, The control unit sets the scan path of the light source probe unit to include the target area specified by the vision unit, and guides and controls the tip of the robot tube according to the scan path. A photoacoustic imaging apparatus characterized by the following features.

2. The vision unit includes a bio-element recognition unit that sequentially recognizes tissues or organs corresponding to a target region based on the image content of the target region captured by the imaging sensor, while referring to a bio-element estimation model constructed by deep learning using surface feature patterns classified by human tissue or organ as training data. The photoacoustic imaging apparatus according to feature 1.

3. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The control unit guides and controls the tip of the robot tube so that the distance to the surface of the biological tissue is maintained within a certain range, based on the three-dimensional shape data detected by the RGB-D sensor. The photoacoustic imaging apparatus according to claim 1 or 2.

4. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The vision unit includes an internal motion detection unit that detects the pulsation of tissue or peristalsis of organs corresponding to the target region sequentially recognized by the biological element recognition unit based on the time change of the three-dimensional shape data detected by the RGB-D sensor. The control unit controls the tip of the robot tube by making minute fluctuations in the distance from the surface of the biological part, in synchronization with the pulsation of the tissue or peristalsis of the organ detected by the internal motion detection unit. The photoacoustic imaging apparatus according to feature 2.

5. The aforementioned vision unit is A photoacoustic wave image generation unit generates a photoacoustic wave image based on the photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit, An abnormality detection unit detects abnormalities in tissues or organs corresponding to the target region, which are sequentially recognized by the biological element recognition unit, based on the photoacoustic wave image generated by the photoacoustic wave image generation unit. The photoacoustic imaging apparatus according to claim 2, characterized by comprising:

6. The vision unit includes a disease symptom estimation unit that estimates the disease and symptoms of the tissue or organ by analyzing the abnormal state, based on the abnormal state of the tissue or organ detected by the abnormal state detection unit, while referring to a disease content estimation model constructed by deep learning using abnormal state feature patterns classified for each human tissue or organ as training data. The photoacoustic imaging apparatus according to feature 5.

7. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The vision unit further includes a vascular latent flow determination unit that, based on the time change of the three-dimensional shape data detected by the RGB-D sensor, detects the pulsation of the arterial blood vessels of the subject and determines how easily the blood perfusion of the arterial blood vessels changes from the range of variation in the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial blood vessels. The disease symptom estimation unit incorporates the ease with which the blood flow perfusion of the arterial vessels changes, as determined by the vascular latent flow determination unit, into the analysis of the abnormal state of the tissue or organ. The photoacoustic imaging apparatus according to feature 6.

8. The aforementioned vision unit is A photoacoustic wave image generation unit generates a photoacoustic wave image based on the photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit, The system includes an optical acoustic wave image evaluation unit that evaluates the clarity of the optical acoustic wave image generated by the optical acoustic wave image generation unit for each scan path, If the clarity of the photoacoustic wave image evaluated by the photoacoustic wave image evaluation unit is below a predetermined level, the control unit causes the light source probe unit to scan the tip of the robot tube while providing feedback guidance control according to the corresponding scan path. The photoacoustic imaging apparatus according to feature 1.

9. In a photoacoustic imaging method that detects photoacoustic waves generated from a light absorber within a subject and images the light absorber based on said photoacoustic waves, A light source probe unit is provided at the tip of a robot tube that is bendable and expandable in the direction of travel from its base to its tip, and can be driven to guide the tip in a desired direction and position in response to external operation or autonomously. The probe unit integrally includes a light irradiation unit that irradiates pulsed light of a wavelength absorbed within the subject, and a photoacoustic wave detection unit that detects photoacoustic waves generated from the light absorber. A first step involves specifying, via external input or automatically, the target area of ​​the tissue or organ to be examined in the subject, based on the image content of the adjacent area at the tip of the robot tube, which is captured by an imaging sensor attached to the light source probe unit. The second step involves setting the scan path of the light source probe unit to include the target area specified in the first step, and guiding and controlling the tip of the robot tube according to the scan path. A photoacoustic imaging method characterized by comprising:

10. In the first step described above, using surface feature patterns classified by human tissue or organ as training data, and referring to a bio-element estimation model constructed by deep learning, the system sequentially recognizes the tissue or organ corresponding to the target region based on the image content of the target region captured by the imaging sensor. The photoacoustic imaging method according to feature 9.

11. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. In the second step, the tip of the robot tube is guided and controlled based on the three-dimensional shape data detected by the RGB-D sensor so that the distance to the surface of the biological tissue is maintained within a certain range. The photoacoustic imaging method according to claim 9 or 10, characterized by the features described above.

12. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The system includes a third step of detecting the pulsation of tissue or peristalsis of an organ corresponding to the target region sequentially recognized in the first step, based on the time change of the three-dimensional shape data detected by the RGB-D sensor, In the second step, the tip of the robot tube is guided and controlled while making minute changes in the distance to the surface of the biological part so as to be synchronized with the pulsation of the tissue or peristalsis of the organ detected in the third step. The photoacoustic imaging method according to feature 10.

13. A fifth step is to generate a photoacoustic wave image based on the photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit, A sixth step in which, based on the photoacoustic wave image generated in the fifth step, abnormalities in tissues or organs corresponding to the target region sequentially recognized in the first step are detected. The photoacoustic imaging method according to claim 10, characterized by comprising the above.

14. The system includes a seventh step in which, using abnormality state characteristic patterns classified for each human tissue or organ as training data, a disease content estimation model constructed by deep learning is referenced, and the abnormality state of the tissue or organ detected in the sixth step is used to estimate the disease or symptoms of the tissue or organ by analyzing the abnormality state. The photoacoustic imaging method according to feature 13.

15. The imaging sensor has an RGB-D sensor for detecting three-dimensional shape data, including the proximity distance from the detection port of the acoustic wave detection unit to the surface of a biological part within the subject. The eighth step further comprises detecting the pulsation of the arterial vessels of the subject based on the time change of the three-dimensional shape data detected by the RGB-D sensor, and determining how easily the blood perfusion of the arterial vessels changes from the range of variation in the peak and bottom interval of the fluctuating component that reflects the pulsation of the arterial vessels, In the seventh step, the susceptibility of the blood flow perfusion of the arterial vessels to changes, as determined in the eighth step, is reflected in the analysis of the abnormal state of the tissue or organ. The photoacoustic imaging method according to feature 14.

16. The ninth step includes evaluating the clarity of the photoacoustic wave image based on the photoacoustic wave detected by the photoacoustic wave detection unit in the light source probe unit for each scan path. In the second step, if the clarity of the photoacoustic wave image evaluated in the ninth step is below a predetermined level, the tip of the robot tube is guided and controlled in a feedback manner according to the corresponding scan path, causing the light source probe unit to scan. The photoacoustic imaging method according to feature 9.

Images