TOMOGRAPH IMAGE PROCESSING APPARATUS, TOMOGRAPH IMAGE PROCESSING METHOD, PROGRAM, AND INFORMATION RECORDING MEDIUM

Abstract

This tomographic image processing device (101) generates pseudo-tomographic images from a moving image of cross-sections captured via a light-receiving surface of a puncture member that is inserted in a living body. Here, an acquisition unit (111) acquires a moving image of the cross-sections captured via the light-receiving surface of the puncture member being inserted in the living body. A correction unit (112) performs image correction on each frame of the moving image on the basis of the shape of the light-receiving surface. A tracking unit (113) tracks candidates, of a picture of a tissue in the living body, satisfying a prescribed tracking condition. An estimation unit (114) estimates a representative vector on the basis of movement vectors of the candidates. An extraction unit (115) extracts a candidate that has a movement vector similar to the representative vector. On the basis of portray positions in frames in which a picture of the extracted candidate is portrayed in the moving image and the sequence of said frames, a calculation unit (116) calculates an arrangement position where the picture of the candidate is to be disposed. A generation unit (117) generates pseudo-tomographic images of the living body by portraying the picture of the extracted candidate at the calculated arrangement position.

Description

Tomographic image processing device, tomographic image processing method, program, information recording medium, and insertion body

[0001] The present invention relates to a tomographic image processing device, a tomographic image processing method, a program, an information recording medium, and a puncture body for generating a pseudo tomographic image of a living body from a video of a cross-section of the living body captured through a light receiving surface of the puncture body inserted into the living body.

[0002] BACKGROUND ART Conventionally, techniques have been proposed for observing the inside of a living body by inserting a piercing body into the living body.

[0003] For example, Patent Document 1 discloses a prior art technique in which, in order to examine a fixed organ of a subject, a needle is introduced into a predetermined area of ​​the solid organ, an optical probe is inserted through the lumen of the needle, and the predetermined area is imaged using the optical probe.

[0004] In the prior art, the optical probe is configured to be placed through a needle into a solid organ and includes: a fiber optic bundle; and a spatula for protecting the distal end of the fiber optic bundle.

[0005] In this prior art, the spatula includes a shaft, a head, and a case that covers the optical fiber bundle and the shaft.

[0006] In the prior art, the head of the spatula has a length configured to allow the optical probe to image solid organs while the case is held inside the needle.

[0007] On the other hand, Non-Patent Document 1 discloses a technology in which the end of a single optical fiber with a diameter of 125 μm is etched into a needle shape, and the intensity of excitation light introduced from the end is measured, thereby enabling measurement of the fluorescence spectrum.

[0008] Special Publication No. 2013-505043

[0009] Atsushi Seki, Kazuhiro Watanabe, Fluorescence Measurement Using a Needle-Shaped Optical Fiber End, IEICE 2019 General Conference, Optical Fiber Application Technology (B-13), Presentation No. B-13-23, https: / / www.ieice.org / publications / conferences / summary.php?id=CONF0000122214, https: / / www.ieice-taikai.jp / 2019general / jpn / webpro / _html / cs.html, Published March 5, 2019

[0010] In the above prior art, the field of view range for imaging is limited to the thickness of the optical fiber bundle.

[0011] For this reason, the above-mentioned prior art cannot capture images of the shape and other aspects of the tissue of a living body that extend beyond the field of view, or the shape and other aspects of the tissue of a living body in the depth direction.

[0012] Therefore, there is a strong demand for a technology that can obtain tomographic images of living tissue with cellular-level resolution.

[0013] The present invention solves the above-mentioned problems and relates to a tomographic image processing device, a tomographic image processing method, a program, an information recording medium, and a puncture body for generating a pseudo tomographic image of a living body from a video of a cross-section of the living body captured through a light receiving surface of the puncture body that is inserted into the living body.

[0014] The tomographic image processing device of the present invention acquires a video of a cross section of a living body through a light receiving surface of a penetrating body inserted into the living body, performs image correction on each frame of the acquired video based on the shape of the light receiving surface, tracks candidate images of tissues of the living body that satisfy desired tracking conditions in the image-corrected video, estimates a representative vector in the image-corrected video based on the movement vector of the tracked candidate, extracts candidates whose movement vector is similar to the estimated representative vector, calculates a placement position where the image of the extracted candidate should be placed based on the drawing position in a frame in which the image of the extracted candidate is drawn in the video and the order of the frames, and generates a pseudo tomographic image of the living body by drawing the image of the extracted candidate at the calculated placement position.

[0015] The piercing body according to the present invention is a thin wire made of bundled optical fibers, which is pierced into a living body in the longitudinal direction of the thin wire, and one end of the thin wire serves as a light receiving surface for photographing a cross section of the living body.

[0016] According to the present invention, it is possible to provide a tomographic image processing device, a tomographic image processing method, a program, an information recording medium, and an insertion body for generating a pseudo tomographic image of a living body from a video of a cross-section of the living body photographed through a light receiving surface of the insertion body inserted into the living body.

[0017] FIG. 1 is an explanatory diagram showing a schematic configuration of a tomographic image processing apparatus according to an embodiment of the present invention. FIG. 1 is an explanatory diagram explaining the appearance of a puncture body. FIG. 1 is an explanatory diagram explaining the appearance of a puncture body. FIG. 2 is a representative photograph of a drawing in which the tip of a bamboo spear-shaped puncture body is photographed in grayscale. FIG. 3 is an explanatory diagram showing a representative photograph of a drawing in which the tip of a bamboo spear-shaped puncture body is photographed in grayscale. FIG. 4 is an explanatory diagram showing a representative photograph of a drawing in which the tip of a round pencil-shaped puncture body is photographed in monochrome binarized. FIG. 5 is a drawing-substitute photograph showing an example of an image photographed with a microscopic endoscope in grayscale. FIG. 6 is an explanatory diagram showing a drawing-substitute photograph showing an example of an image photographed with a microscopic endoscope, binarized in monochrome. FIG. 7 is an explanatory diagram schematically showing an image photographed with a bamboo spear-shaped puncture body. FIG. 8 is an explanatory diagram showing the result of image correction of an image photographed with a bamboo spear-shaped puncture body. FIG. 9 is an explanatory diagram showing the result of image correction of an image photographed with a round pencil-shaped puncture body. FIG. 10 is a flowchart showing control of tomographic image processing according to this embodiment. FIG. 1 is a representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art in grayscale; FIG. 2 is an explanatory diagram of a monochrome binarized representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art; FIG. 3 is a representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art in grayscale; FIG. 4 is an explanatory diagram of a monochrome binarized representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art; FIG. 5 is a representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art in grayscale; FIG. 6 is an explanatory diagram of a monochrome binarized representative photograph of a drawing comparing the pseudo tomographic image of this embodiment with an excision specimen image of the prior art.

[0018] The following describes embodiments of the present invention. Note that these embodiments are for illustrative purposes only and do not limit the scope of the present invention. Therefore, those skilled in the art can adopt embodiments in which each or all of the elements of the present embodiments are replaced with equivalents. Furthermore, elements described in each example can be omitted as appropriate depending on the application. In this way, all embodiments constructed in accordance with the principles of the present invention are included in the scope of the present invention.

[0019] 1 is an explanatory diagram showing the general configuration of a tomographic image processing apparatus according to an embodiment of the present invention, which will be described below with reference to this diagram.

[0020] As shown in the figure, the tomographic image processing apparatus 101 according to this embodiment includes an acquisition unit 111, a correction unit 112, a tracking unit 113, an estimation unit 114, an extraction unit 115, a calculation unit 116, and a generation unit 117.

[0021] Typically, a computer executes a program to implement this tomographic image processing device 101. The computer is connected to various output devices and input devices, and transmits and receives information to and from these devices.

[0022] A program executed by a computer can be distributed or sold by a server to which the computer is connected for communication, or it can be recorded on a non-transitory information recording medium such as a CD-ROM (Compact Disk Read Only Memory), flash memory, or EEPROM (Electrically Erasable Programmable ROM), and then the information recording medium can be distributed, sold, etc.

[0023] The program is installed on a non-transitory information recording medium such as a hard disk, solid-state drive, flash memory, EEPROM, etc., possessed by the computer. The information processing device of this embodiment is then realized by the computer. Generally, the computer's central processing unit (CPU) reads the program from the information recording medium into random access memory (RAM) under the control of the computer's operating system (OS), and then interprets and executes the code contained in the program. However, in an architecture in which the information recording medium can be mapped within a memory space accessible by the CPU, explicit loading of the program into RAM may not be necessary. Various pieces of information required during program execution can be temporarily stored in RAM.

[0024] Furthermore, as mentioned above, it is desirable for the computer to be equipped with a GPU (Graphics Processing Unit) for performing various image processing calculations at high speed. By using a GPU and libraries such as TensorFlow, it becomes possible to utilize the learning and classification functions in various artificial intelligence processes under the control of the CPU.

[0025] It should be noted that the information processing device of this embodiment may be configured using a dedicated electronic circuit rather than a general-purpose computer. In this embodiment, the program may be used as a resource for generating wiring diagrams, timing charts, and the like for the electronic circuit. In this embodiment, an electronic circuit that meets the specifications defined in the program is configured using an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), and the electronic circuit functions as a dedicated device that performs the functions defined in the program, thereby realizing the information processing device of this embodiment.

[0026] For ease of understanding, the following description will be given assuming that the tomographic image processing apparatus 101 is realized by a computer executing a program.

[0027] The acquiring unit 111 acquires a moving image of a cross section of a living body via a light receiving surface of a piercing body that is inserted into the living body.

[0028] Here, the piercing body in this embodiment is a thin wire made of bundled optical fibers, which is pierced into a living body in the longitudinal direction of the thin wire, with one end of the thin wire serving as a light receiving surface for photographing a cross section of the living body. When the piercing body is pierced into a living body, the longitudinal direction of the thin wire is the depth direction of the living body, that is, the direction from the outside of the living body to the inside of the living body. When pierced approximately vertically through the skin of the living body, the piercing direction is the so-called Z-axis direction, but it is also possible to pierce the skin of the living body at an angle, or to pierce the inside of the living body approximately parallel to the skin after piercing.

[0029] 2 and 3 are explanatory diagrams illustrating the appearance of the insertion body. The following description will be made with reference to these figures.

[0030] These figures show two types of piercing bodies 121. In either case, the light receiving surface 122 of the piercing body 121 is provided at the stump of the thin wire 123.

[0031] The thin wire 123 can be, for example, an imaging fiber with about 10,000 pixels, which is made by bundling about 10,000 optical fibers to a diameter of about 0.35 mm (several hundred μm). For example, existing commercially available products include HDIG from Sumita Optical Glass Co., Ltd. and FIGH from Fujikura Corporation. The specifications of these imaging fibers can be changed as appropriate depending on the application.

[0032] The light receiving surface 122 side of the piercing body 121 is formed into a needle shape by polishing the end of an extremely thin wire 123, so the piercing body 121 can be easily inserted into a living body, is minimally invasive, and causes almost no bleeding. Therefore, it is possible to pierce the tissue repeatedly at intervals to observe changes over time.

[0033] Figure 4 is a representative grayscale photograph of the tip of a bamboo spear-shaped piercing body. Figure 5 is an explanatory diagram of a representative monochrome binarized photograph of the tip of a bamboo spear-shaped piercing body. In the piercing body 121 shown in these figures, the light-receiving surface 122 is configured as a plane that forms a predetermined angle with the longitudinal direction of the thin wire 123. In the example shown in this figure, the angle θ between the light-receiving surface 122 and the longitudinal direction of the thin wire 123 is 60 degrees, and the tip of the thin wire 123 on the light-receiving surface 122 side is acute. If the thin wire 123 is likened to a bamboo, the light-receiving surface 122, which is one of its stumps, is a cross-section of the bamboo with the tip cut off at an angle, and the vicinity of the stump of the piercing body 121 has a bamboo spear-like shape. If the cross-section of the thin wire 123 perpendicular to the longitudinal direction is approximately circular, i.e., if the thin wire 123 is a round wire, the shape of the light-receiving surface 122 is approximately elliptical.

[0034] FIG. 6 is a representative grayscale photograph of the tip of a round-pencil-shaped puncture body. FIG. 7 is an explanatory diagram of a representative monochrome binarized photograph of the tip of a round-pencil-shaped puncture body. In the puncture body 121 shown in these figures, the light-receiving surface 122 is configured as a cone-shaped surface with the longitudinal direction of the thin wire 123 as the central axis. In the example shown in this figure, the angle θ between the cone-shaped surface of the light-receiving surface 122 and the longitudinal direction of the thin wire 123 is 45 degrees or 60 degrees, and the tip of the light-receiving surface of the thin wire 123 is needle-shaped at a right angle. If the thin wire 123 is likened to a round pencil, the light-receiving surface 122, which is one of its stumps, corresponds to the exposed cone-shaped surface when the tip of the round pencil is sharpened to expose the lead, and the vicinity of the stump of the puncture body 121 will have the shape of a round pencil. If the thin wire 123 is a round wire, the light-receiving surface 122, when unfolded in a plane, has an approximately fan-shaped shape.

[0035] In addition, various other shapes can be adopted for the light receiving surface 122 of the piercing body 121, such as a polygonal pyramidal surface shape or a shape with two faces like the tip of a precision flathead screwdriver. The angle formed between the light receiving surface 122 and the longitudinal direction of the thin wire 123 can also be changed as appropriate.

[0036] The end of the thin wire 123 of the piercing body 121 opposite to the light receiving surface 122 side is placed on the objective lens side of the confocal microscope endoscope unit, and the end on the light receiving surface 122 side functions as an endoscope probe.

[0037] In a confocal endoscope, when laser light is irradiated onto the thin wire 123 through an objective lens using a multicolor laser system (wavelength 475 nm / 561 nm), the laser light passes through the thin wire 123 and is irradiated as excitation light into the inserted biological tissue from the light receiving surface 122.

[0038] For example, if a Fucci (Fluorescent Ubiquitination-based Cell Cycle Indicator) system, which changes fluorescence wavelength depending on the state of the cell cycle, is introduced into cancer cells, red fluorescence will be emitted from the cell nuclei in the G1 phase and green fluorescence will be emitted from the cell nuclei in the S / G2 / M phases based on the excitation light.

[0039] The emitted fluorescence reaches the objective lens of the microscope endoscope via the light receiving surface 122 and the thin wire 123, where it is converted into an electrical signal by a photomultiplier tube, and the cross-section of the living body can be captured in the form of an image file. If continuous images are taken while the piercing body 121 is inserted at a substantially constant speed, the cross-section of the living body can be captured in video format. In this video, the tissues of the living body (cells, cell nuclei, organelles, etc.) are captured in a manner that moves in the opposite direction to the direction in which the piercing body 121 is moving.

[0040] As described above, the shape of the light receiving surface 122 is an ellipse or a cone (fan shape), but the opposite end of the thin wire 123 is circular. This is to be observed with an endoscope microscope.

[0041] Figure 8 is a photograph showing an example of an image captured by a microscopic endoscope in grayscale. Figure 9 is an explanatory diagram in which a photograph showing an example of an image captured by a microscopic endoscope has been converted into monochrome binarization. As shown in these figures, the light received by the light-receiving surface 122, i.e., the image obtained by the microscopic endoscope, is projected within a circle.

[0042] 10 is an explanatory diagram that schematically illustrates an image captured using a bamboo spear-shaped piercing body. This diagram is a representation of an image equivalent to one frame of a video captured by a microscope endoscope using a bamboo spear-shaped piercing body 121. The cross section 201 of the living body in the captured frame 200 is circular, and the direction of movement 203 in which the living body tissue 202, shown shaded in this diagram, moves in the video is opposite to the direction of movement of the piercing body 121.

[0043] In this figure, a bamboo spear-shaped piercing body 121 is assumed, so unless each tissue 202 gets caught on the piercing body 121, deforms, tears, or adheres to the light receiving surface 122, each tissue 202 is expected to move parallel in approximately the same direction within the field of view.

[0044] As described above, the shape of the light receiving surface 122 is elliptical for the bamboo spear-shaped piercing body 121 and fan-shaped for the round pencil-shaped piercing body 121, but this is transformed into a circle within the video frame. Therefore, the correction unit 112 performs image correction so that the directions in which the biological tissue 202 moves within the video become parallel.

[0045] In the bamboo spear-shaped insertion body 121, the directions in which the biological tissue 202 is expected to move within the video are parallel to each other, so the correction unit 112 can correct the image by calculating the coordinate values ​​of each pixel in each frame of the acquired video and projecting the calculated coordinate values ​​as each value of the Cartesian coordinate system.

[0046] 11 is an explanatory diagram showing the result of image correction of an image taken with a bamboo spear-shaped puncture object. As shown in this figure, image correction of a bamboo spear-shaped puncture object 121 corresponds to adjusting the size and position to the desired ones.

[0047] Below, we consider the case of a round pencil-shaped puncture body 121. Figure 12 is an explanatory diagram that schematically shows an image captured with a round pencil-shaped puncture body. This diagram is a representation of an image equivalent to one frame of a video captured by a microscopic endoscope using the round pencil-shaped puncture body 121. As with the bamboo spear-shaped puncture body 121, the cross section 201 of the living body in the captured frame 200 is circular, and the direction of movement 203 in which the living body tissue 202, shown shaded in this diagram, moves in the video corresponds to the direction opposite to the direction of movement of the puncture body 121.

[0048] In this figure, a round pencil-shaped piercing body 121 is assumed, so each tissue 202 is expected to diffuse and move in a direction spreading from the center of the circular cross section 201 (corresponding to the pole of the cone shape) within the field of view.

[0049] The correction unit 112 performs image correction on each frame of the acquired moving image based on the shape of the light receiving surface 122 .

[0050] Furthermore, in the case of the round pencil-shaped insertion body 121, the direction in which the biological tissue 202 is expected to move in the video is the direction of diffusion from the center (pole), so the correction unit 112 can perform image correction by identifying a pole common to each frame of the acquired video, determining the polar coordinate values ​​of each pixel in each frame of the acquired video using a polar coordinate system based on the identified pole, and projecting the determined polar coordinate values ​​as each value of the Cartesian coordinates.

[0051] The simplest method for identifying the pole is to identify the approximately circular shape of the cross section 201 in each frame and use the center of the circle as the pole. Other methods will be described later.

[0052] The coordinate values ​​of the polar coordinate system are expressed by the distance from the pole and the angle from a specific direction extending from the pole. Figure 13 is an explanatory diagram showing the result of image correction of an image taken with a round pencil-shaped insertion device. As shown in this figure, the distance and angle of each point are projected as values ​​of the Cartesian coordinate system.

[0053] Since the biological tissue 202 is expected to move radially from the pole, when the projection described above is performed, the coordinate value of the Cartesian coordinate on the side where the distance is projected gradually increases, but the coordinate value of the Cartesian coordinate on the side where the angle is projected remains constant.

[0054] Therefore, in each frame of the projected moving image, the biological tissues 202 are expected to move in parallel directions 203 .

[0055] Note that since the image distortion is large near the poles, the vicinity of the poles may be removed from the projection target.

[0056] It is desirable that the correction unit 112 remove noise using a Gaussian filter or the like.

[0057] Then, the tracking unit 113 tracks candidates for the image of the biological tissue 202 that satisfy the desired tracking conditions in the image-corrected moving image.

[0058] As described above, in this embodiment, the tissue to be observed ("tissue" is a general term that encompasses cells, organelles, extracellular matrices, biomolecular structures, etc.) is made to emit fluorescence by administering a fluorescent probe to the tissue to be observed, or by causing the tissue to produce a fluorescent protein such as Fucci.

[0059] Therefore, the tracking conditions that can be adopted are: the color of the image is the same as the color of the fluorescence emitted by the tissue (such as the wavelength of the fluorescence) and has a certain brightness value; the size of the image consisting of the area of ​​that color matches the range of expected tissue sizes; the shape of the image consisting of the area of ​​that color matches the expected shape of the tissue (for example, approximately circular); and combinations of these.

[0060] It is also possible to directly apply an image processing technique that satisfies these tracking conditions. For example, a technique such as SIFT (Scale-Invariant Feature Transform) may be applied to extract feature points corresponding to the desired biological tissue 202 and track them.

[0061] Only images that satisfy the tracking conditions are considered as candidates, and images that do not, such as areas of a color different from the expected fluorescence, or areas of the same color as the expected fluorescence but that do not match in size or shape, are not considered in the following processing.

[0062] That is, the tracking unit 113 does not track images of tissues that do not satisfy desired tracking conditions.

[0063] This allows tissue 202 or dust that has come off the living body and adhered to the light receiving surface 122 to be treated as an error.

[0064] Now, when a candidate for the image of the desired biological tissue 202 is tracked in each frame of the video, the estimation unit 114 estimates a representative vector in the image-corrected video based on the movement vector of the tracked candidate.

[0065] This representative vector is a vector in the opposite direction to the insertion vector along which the inserter 121 moves after being inserted into the living body.

[0066] While the insertion body 121 is being inserted into the living body at a constant speed, the movement vector of the candidate is expected to be constant in magnitude and direction, but the magnitude of the movement vector may change due to friction, deformation, contraction, etc. However, the movement vector is considered to be generally in the same direction.

[0067] Therefore, the most frequent value or average value of the direction of the movement vectors of the candidates can be used as the direction of the representative vector in the moving image.

[0068] Next, the extraction unit 115 extracts candidates similar to the estimated representative vector. Here, "candidates similar to the representative vector" means candidates whose direction of the representative vector and the direction of the movement vector of the candidate fall within a predetermined tolerance range.

[0069] Candidates that do not fall within the allowable range are not subject to extraction, so that, for example, tissue 202 or dust that has detached from the living body and is moving while sliding relative to the inserter 121 can be treated as errors.

[0070] Then, the calculation unit 116 calculates the placement position where the extracted candidate image should be placed, based on the drawing position within the frame in which the extracted candidate image is drawn in the video and the order of the frames.

[0071] Finally, the generating unit 117 generates a pseudo tomographic image of the living body by drawing the images of the extracted candidates at the calculated arrangement positions.

[0072] The order of the frames is a numerical value corresponding to the depth to which the piercing body 121 is inserted into the living body, and the later an image appears in the frame, the deeper the tissue corresponding to that image is located in the living body. Therefore, the depth direction of the placement position can be determined based on the order in which the image appears in the frame.

[0073] In addition, in a moving image, the image of the biological tissue 202 is expected to move in the same direction as the direction of the representative vector. Therefore, in each frame of the moving image, a coordinate axis perpendicular to the direction of the representative vector can be considered, and the coordinate value of the image on the coordinate axis can be adopted in a direction perpendicular to the depth of the placement position.

[0074] In reality, the image of the biological tissue 202 may move and deviate from the direction of the representative vector within a predetermined tolerance range. Therefore, based on the position of the image at a specific time, such as when tracking of the image of the biological tissue 202 begins, the coordinate value of the image on a coordinate axis perpendicular to the direction of the representative vector can be calculated, and this can be used as the position in a direction perpendicular to the depth direction of the placement position.

[0075] In addition, when calculating the placement position of the image, one of the following may be selected: the position of the image in the frame where tracking starts, the position of the image in the frame during tracking, or the position of the image in the frame where tracking ends, or an averaging process may be performed as appropriate.

[0076] The calculation unit 116 may correct the calculated placement position according to the direction of the representative vector. For example, distortion occurs due to the angular difference in the longitudinal direction between the light receiving surface 122 and the thin wire 123, so correction is applied when calculating the placement position. For example, if the angular difference is 60 degrees, the distortion is 1:2, so the placement position can be determined at twice the distance in the longitudinal direction within the video. This correction can be performed using any correction value.

[0077] In addition, the distribution of the placement positions calculated this time can be scaled along the direction of the representative vector so that it matches the distribution obtained when a tissue section of an excised specimen from another individual, created using existing methods, is observed under a microscope. If the two distributions match, the placement conditions are met.

[0078] Furthermore, the direction and magnitude of the representative vector (insertion vector) may change while the insertion object 121 is being inserted. In this case, the placement position is corrected taking into account the change. In this case, it is also possible to apply correction techniques, etc., used when taking panoramic photos by rotating a handheld camera such as a smartphone.

[0079] The candidate image that the generation unit 117 draws at the calculated placement position can, in the simplest case, be an image at a specific time, such as when tracking begins, or an average image of the images drawn during tracking can be drawn at the calculated placement position.

[0080] In the case of the round pencil-shaped insertion body 121, the following method can be adopted to identify the position of the pole.

[0081] First, the pole positions are provisionally determined, and processing is performed by the correction unit 112, tracking unit 113, estimation unit 114, and extraction unit 115.

[0082] Then, the direction of the movement vector of the extracted candidate image should all be along one axis of the Cartesian coordinate system within the tolerance range.

[0083] Therefore, the pole position is corrected so that the error in the direction becomes smaller, that is, so that the direction of the movement vector of the candidate image becomes closer to parallel.

[0084] By repeating this process, it becomes possible to more accurately identify the pole even when the center of the cone shape at the tip of the round pencil-shaped insertion body 121 is misaligned.

[0085] The following describes in detail the control of the tomographic image processing executed by the tomographic image processing apparatus of this embodiment. In this processing, one example of the various aspects described above is taken as a representative example, but various aspects can be realized by changing the details of each processing according to the above explanation.

[0086] 14 is a flowchart showing the control of tomographic image processing according to this embodiment. The following description will be made with reference to this figure.

[0087] First, the tomographic image processing apparatus 101 clears the image buffer for tomographic images (step S400).

[0088] Next, the tomographic image processing apparatus 101 acquires a moving image obtained by capturing a cross section of the living body via the light receiving surface 122 of the piercing body 121 that is inserted into the living body (step S401).

[0089] Then, the tomographic image processing apparatus 101 repeats the following process for each frame of the moving image (step S402).

[0090] That is, the tomographic image processing apparatus 101 resizes each frame to a predetermined processing size while maintaining the aspect ratio, and projects it onto orthogonal coordinates based on the shape of the light receiving surface (step S403).

[0091] Then, the tomographic image processing device 101 applies a Gaussian filter to the projected image to remove noise (step S404), and then applies a coarse Gaussian filter and a fine Gaussian filter to take the difference, thereby removing the background and enhancing the signal (step S405).

[0092] Furthermore, the tomographic image processing apparatus 101 applies a LoG (Laplacian of Gaussian) filter or the like to enhance signals corresponding to the shape and size of the desired tissue (step S406).

[0093] Then, the tomographic image processing apparatus 101 again applies the coarse Gaussian filter and the fine Gaussian filter to obtain the difference, thereby removing the background and enhancing the signal (step S407).

[0094] Thereafter, the tomographic image processing apparatus 101 performs binarization processing to extract the signal region (step S408), thereby specifying the position of the image of the desired tissue.

[0095] After completing the repetition for each frame (step S409), the tomographic image processing apparatus 101 repeats the following process for a pair of two adjacent frames (a pair of an immediately preceding frame and a current frame) (step S410).

[0096] That is, the tomographic image processing device 101 compares the previous frame with the current frame by template matching (step S411), selects each image of tissue newly detected in the current frame (for which no template image is set) (step S412), and repeats the following process for each image (step S413).

[0097] That is, if the position of the image in the current frame is within a predetermined area that is not the periphery of the field of view (step S414; Yes), the tomographic image processing device 101 considers that a new image to be tracked has appeared, sets the position of the image as the tracking position (step S415), sets a template image based on the image (step S416), and saves an ROI (Region of Interest) image (color channel image) to be pasted onto the pseudo tomographic image based on the image in the current frame and the frame of the original video (step S417), and repeats this process (step S418).

[0098] On the other hand, if no new image to be tracked has appeared (step S414; No), the process proceeds to step S418.

[0099] Next, the tomographic image processing device 101 selects each image of tissue that has moved within an estimated movement amount between the previous frame and the current frame as a result of template matching comparison (step S420), and repeats the following process for each image (step S421).

[0100] That is, the tomographic image processing apparatus 101 obtains the correlation coefficient and position between the immediately preceding frame and the current frame of the image (step S422).

[0101] If the calculated correlation coefficient is equal to or greater than a threshold and the position is within a predetermined area that is not on the periphery of the field of view, the image has been tracked (step S423; Yes), and the tomographic image processing device 101 updates the tracking position based on the image in the current frame (step S424), updates the template image (step S425), and saves an ROI image (color channel image) to be pasted onto the pseudo tomographic image based on the image in the current frame and the frame of the original video (step S426).

[0102] Furthermore, the tomographic image processing apparatus 101 calculates a movement vector based on the position of the image in the immediately preceding frame and the position of the image in the current frame (step S427), and proceeds to step S431.

[0103] On the other hand, if tracking is not possible and the image is lost (step S423; No), the tomogram processing apparatus 101 cancels the setting of the template image of the image (step S428) and ends tracking.

[0104] Then, the tomographic image processing device 101 calculates the placement position based on the order of the previous frame, the position of the image in the previous frame, the movement vector of the image, the representative vector (insertion vector), etc. (step S429), and draws an average image based on the ROI image saved for the tissue in the image buffer for tomographic images at the calculated placement position after correcting the placement position and image distortion resulting from the light receiving surface shape (step S430).

[0105] After the repetition for the moving image between the previous frame and the current frame is completed (step S431), and after the repetition for all pairs of frames is completed (step S432), the tomographic image processing device 101 outputs the image drawn in the image buffer as a pseudo tomographic image of the living body (step S433), and ends this processing.

[0106] In addition, the average of the calculated movement vectors (corresponding to the representative vector) is calculated, and for the image of a movement vector that deviates significantly from the average, tracking is terminated, and the placement position is not calculated, and drawing to the image buffer is not performed (not shown).

[0107] For example, when there is a lot of dust adhering to or remaining on the light receiving surface 122, the influence of the image not linked to the movement of the insertion object 121 can be eliminated by adopting the above-described method.

[0108] (Experimental results) A cancer-bearing model was prepared by subcutaneously transplanting cultured cancer cells (human fibrosarcoma cells: HT1080) into immunodeficient mice (Balb / cAjcl nu / nu), and a cancer-bearing model animal was prepared by subcutaneously transplanting cultured cancer cells (mouse colon cancer cells: Colon26) into mice (Balb / cAjcl).

[0109] The cancer cells were equipped with a Fucci system, which changes the fluorescence wavelength depending on the state of the cell cycle, and red fluorescence was emitted from the G1 phase cell nuclei, while green fluorescence was emitted from the S / G2 / M phase cell nuclei.

[0110] For insertion imaging, the mice were anesthetized by inhalation, the tumor was held in place with a restraining device, and a hole was made in the skin on the surface of the tumor with an injection needle to create an insertion site for the endoscopic probe.

[0111] The endoscopic probe was controlled by an automatic manipulator and inserted at a constant speed of 4 mm / min from the insertion side (skin side) to the surface of the tumor on the opposite side, while continuous images were taken at a shooting speed of 18 fps (18 images per second), and clear fluorescence originating from cell nuclei was detected within the tumor tissue.

[0112] In this case, the imaging was minimally invasive, the insertion scar was very small, and no bleeding was observed, making repeated imaging possible over a long period of time.

[0113] Fluorescent images obtained by microscopic endoscopic imaging are output in video format through continuous shooting, and cell nuclei (red or green) moving in the opposite direction to the direction of travel of the endoscopic probe are captured.

[0114] The pseudo-tomographic image reconstruction program was created primarily based on the Tracking program of OpenCV, an open-source computer vision library, and enabled tracking of each object (each cell) by recognizing objects (cell nuclei in this experimental example) in the microscopic endoscopic image for each channel (the conditions for recognition are determined by various parameters such as size).

[0115] In this case, the system was designed so that each recognized cell nucleus was assigned a vector value based on its movement, and the average value was calculated for each frame.

[0116] By determining the vector value (penetration vector) of the endoscopic probe based on this vector average value (representative vector), it is possible to estimate how the endoscopic probe progressed (direction and speed) relative to the penetrated tissue during tomographic image reconstruction, and the placement position is corrected accordingly.

[0117] Nonspecific fluorescent signals and cells attached to the surface of the optical fiber were excluded by detecting values ​​of the object's vector value close to 0 (static state) or deviation from the vector value of the endoscopic probe.

[0118] Within this framework, a pseudo-tomographic image was reconstructed by projecting the recognized cell nuclei (average image during tracking) to the initial recognition point.

[0119] FIG. 15 is a representative photograph comparing the pseudo tomographic image of this embodiment with an excision specimen image of a conventional technique in grayscale. FIG. 16 is an explanatory diagram of a representative photograph, binarized in monochrome, comparing the pseudo tomographic image of this embodiment with an excision specimen image of a conventional technique. Comparing the pseudo tomographic image in the upper row of these figures with the excision specimen image in the lower row, it can be seen that the pseudo tomographic image obtained is similar to a normal fluorescence microscope image of a tissue slice of an excision specimen. Note that in this figure, the pseudo tomographic image is curved and band-shaped because the placement position is corrected to take into account changes in the representative vector (insertion vector).

[0120] Fig. 17 is a representative photograph comparing the pseudo tomographic image of this embodiment with an excised specimen image of the prior art in grayscale. Fig. 18 is an explanatory diagram of a representative photograph, binarized in monochrome, comparing the pseudo tomographic image of this embodiment with an excised specimen image of the prior art. In these figures, the representative vector (insertion vector) is assumed to be constant, so the pseudo tomographic image is a straight band.

[0121] It should be noted that this embodiment can be used regardless of the type of setup of the optical fiber and the microscope endoscope.

[0122] For example, the multicolor laser system of the confocal microscope unit can be expanded, and simultaneous imaging in three or more colors can be achieved by adding lasers such as 430 nm and 630 nm. Figure 19 is a representative photograph comparing the pseudo-tomographic image of this embodiment with an excised specimen image obtained using conventional technology in grayscale. Figure 20 is an explanatory diagram of a representative photograph, binarized in monochrome, comparing the pseudo-tomographic image of this embodiment with an excised specimen image obtained using conventional technology. Similarly, in these figures, pseudo-tomographic images similar to those obtained using a conventional fluorescence microscope are obtained by observing tissue slices of excised specimens.

[0123] In addition, the diameter of the optical fiber can be changed from 0.2 mm to 0.5 mm, and can be changed to suit the target tissue and application in terms of FOV (Field of View) and invasiveness.

[0124] In addition, the tip shape of the endoscopic probe can be changed by adjusting the polishing method of the optical fiber. In the above experimental example, the tip was polished into a bamboo spear shape, but it can also be polished into a pencil shape or a polygonal pyramid shape at any angle. Even if it is polished into a dome shape, it can be handled within the scope of this embodiment by performing projection according to the shape.

[0125] (Summary) The tomographic image processing device of this embodiment is configured to include: an acquisition unit that acquires a video of a cross section of the living body through a light-receiving surface of a penetrating body that is inserted into the living body; a correction unit that performs image correction on each frame of the acquired video based on the shape of the light-receiving surface; a tracking unit that tracks candidates for images of tissues of the living body that satisfy desired tracking conditions in the image-corrected video; an estimation unit that estimates a representative vector in the image-corrected video based on the movement vector of the tracked candidate; an extraction unit that extracts candidates whose movement vector is similar to the estimated representative vector; a calculation unit that calculates a placement position where the image of the extracted candidate should be placed based on the drawing position in a frame in which the image of the extracted candidate is drawn in the video and the order of the frames; and a generation unit that generates a pseudo tomographic image of the living body by drawing the image of the extracted candidate at the calculated placement position.

[0126] Furthermore, in the tomographic image processing device according to this embodiment, the piercing body can be configured to be a thin wire made of bundled optical fibers, the light receiving surface being one end of the thin wire, and the direction in which the piercing body is inserted into the living body being the longitudinal direction of the thin wire.

[0127] In the tomographic image processing apparatus according to this embodiment, the light receiving surface can be configured to be a plane that forms a predetermined angle with the longitudinal direction of the thin wire.

[0128] Furthermore, in the tomographic image processing device according to this embodiment, the correction unit can be configured to perform the image correction by calculating the coordinate values ​​of each pixel in each frame of the acquired moving image and projecting the calculated coordinate values ​​as each value of an orthogonal coordinate system.

[0129] In the tomographic image processing apparatus according to this embodiment, the light receiving surface can be configured to have a conical surface shape with the longitudinal direction of the thin wire as the central axis.

[0130] Furthermore, in the tomographic image processing device according to this embodiment, the correction unit can be configured to perform the image correction by identifying a pole common to each frame of the acquired moving image, calculating the polar coordinate values ​​of each pixel in each frame of the acquired moving image using a polar coordinate system based on the identified pole, and projecting the calculated polar coordinate values ​​as each value of a Cartesian coordinate system.

[0131] Furthermore, in the tomographic image processing device according to this embodiment, the correction unit can be configured to identify the position of the pole so that the direction of the movement vector of the image of the extracted candidate and one of the axes of the Cartesian coordinates become closer to being parallel.

[0132] In addition, in the tomographic image processing device according to this embodiment, the calculation unit can be configured to perform position correction according to the direction of the representative vector so that the distribution of the calculated placement positions satisfies predetermined placement conditions.

[0133] In the tomographic image processing apparatus according to this embodiment, the calculation unit can be configured to perform the position correction by scaling along the direction of the representative vector.

[0134] In addition, in the tomography processing apparatus according to this embodiment, the generation unit can be configured to draw an average image while the image of the extracted candidate is being tracked at the calculated placement position.

[0135] Furthermore, the tomographic image processing apparatus according to this embodiment can be configured to set the desired tracking conditions based on the color of the fluorescence emitted by the tissue.

[0136] Furthermore, the tomographic image processing apparatus according to this embodiment can be configured so that a fluorescent probe that fuses with the tissue is administered to the living body, causing the tissue to emit fluorescence.

[0137] Furthermore, the tomographic image processing apparatus according to this embodiment can be configured so that the tissue emits fluorescence by producing a fluorescent protein in the tissue.

[0138] Furthermore, in the tomographic image processing device according to this embodiment, the tissue is a cell or molecule of the living body, and the desired tracking condition can be set based on whether or not at least one of the size, brightness value, and shape of the area in which the desired color is drawn is within a predetermined range.

[0139] In the tomographic image processing apparatus according to this embodiment, the tracking section can be configured not to target images of tissues that do not satisfy the desired tracking conditions for tracking.

[0140] The tomographic image processing method according to this embodiment includes: an acquisition step of acquiring a video of a cross section of the living body via a light receiving surface of a penetrating body inserted into the living body; a correction step of performing image correction on each frame of the acquired video based on the shape of the light receiving surface; a tracking step of tracking, in the image-corrected video, a candidate image of the living body's tissue that satisfies a desired tracking condition; an estimation step of estimating a representative vector in the image-corrected video based on the movement vector of the tracked candidate; an extraction step of extracting a candidate whose movement vector is similar to the estimated representative vector; a calculation step of calculating a placement position where the image of the extracted candidate should be placed based on the drawing position in a frame in which the image of the extracted candidate is drawn in the video and the order of the frames; and a generation step of generating a pseudo tomographic image of the living body by drawing the image of the extracted candidate at the calculated placement position.

[0141] The program according to this embodiment configures a computer to function as: an acquisition unit that acquires a video of a cross section of a living body through a light receiving surface of a penetrating body that is inserted into the living body; a correction unit that performs image correction for each frame of the acquired video based on the shape of the light receiving surface; a tracking unit that tracks candidates for images of tissues of the living body that satisfy desired tracking conditions in the image-corrected video; an estimation unit that estimates a representative vector in the image-corrected video based on the movement vector of the tracked candidate; an extraction unit that extracts candidates whose movement vector is similar to the estimated representative vector; a calculation unit that calculates a placement position where the image of the extracted candidate should be placed based on the drawing position in a frame in which the image of the extracted candidate is drawn in the video and the order of the frames; and a generation unit that generates a pseudo-tomographic image of the living body by drawing the image of the extracted candidate at the calculated placement position.

[0142] The program can be distributed or sold by recording it on a non-transitory computer-readable information recording medium, or via a transitory transmission medium such as a computer communication network.

[0143] The piercing body of this embodiment is a thin wire made of bundled optical fibers, and is configured to be inserted into a living body in the longitudinal direction of the thin wire, with one end of the thin wire serving as a light receiving surface for photographing a cross section of the living body.

[0144] In addition, in the insertion body according to this embodiment, the light receiving surface can be configured to be a plane that forms a predetermined angle with the longitudinal direction of the thin wire.

[0145] In addition, in the insertion body according to this embodiment, the light receiving surface can be configured to have a conical surface shape with the longitudinal direction of the thin wire as the central axis.

[0146] The present invention allows for various embodiments and modifications without departing from the broad spirit and scope of the present invention. Furthermore, the above-described embodiments are intended to illustrate the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and their equivalents are considered to be within the scope of the present invention. This application claims priority based on patent application No. 2021-060196, filed in Japan on Wednesday, March 31, 2021, and the contents of that basic application are incorporated herein to the extent permitted by the laws and regulations of the designated countries.

[0147] According to the present invention, it is possible to provide a tomographic image processing device, a tomographic image processing method, a program, and an information recording medium for generating a pseudo tomographic image of a living organism from a video of a cross-section of the living organism captured through a light receiving surface of an inserting body inserted into the living organism.

[0148] 101 Tomographic image processing device 111 Acquisition unit 112 Correction unit 113 Tracking unit 114 Estimation unit 115 Extraction unit 116 Calculation unit 117 Generation unit 121 Insertion body 122 Light receiving surface 123 Thin wire 200 Frame 201 Cross section 202 Tissue 203 Movement direction

Claims

1. an acquisition unit that acquires a video of a cross section of the living body via a light receiving surface of a piercing body that is inserted into the living body; a correction unit that performs image correction for each frame of the acquired moving image based on the shape of the light receiving surface; a tracking unit that tracks candidates of images of tissues of the living body that satisfy a desired tracking condition in the image-corrected moving image; an estimation unit that estimates a representative vector in the image-corrected moving image based on the movement vectors of the tracked candidates; an extraction unit that extracts candidates whose motion vectors are similar to the estimated representative vector; a calculation unit that calculates a placement position where the image of the extracted candidate should be placed based on a drawing position in a frame where the image of the extracted candidate is drawn in the video and an order of the frames; a generating unit for generating a pseudo tomographic image of the living body by drawing an image of the extracted candidate at the calculated arrangement position. A tomographic image processing apparatus comprising:

2. The piercing body is a thin wire made of bundled optical fibers, The light receiving surface is one end of the thin wire, The direction in which the piercing body is inserted into the living body is the longitudinal direction of the thin wire.

2. The tomographic image processing apparatus according to claim 1,

3. The light receiving surface is formed in a planar shape that forms a predetermined angle with the longitudinal direction of the thin wire.

3. The tomographic image processing apparatus according to claim 2.

4. The correction unit obtains coordinate values ​​of each pixel in each frame of the acquired moving image, and performs image correction by projecting the obtained coordinate values ​​as each value of an orthogonal coordinate system.

4. The tomographic image processing apparatus according to claim 3.

5. The light receiving surface is formed in a conical shape with the longitudinal direction of the thin wire as the central axis.

3. The tomographic image processing apparatus according to claim 2.

6. The correction unit is Identifying a pole common to each frame of the captured video; Using a polar coordinate system based on the specified pole, the polar coordinate values ​​of each pixel in each frame of the acquired moving image are calculated, and the calculated polar coordinate values ​​are projected as each value of an orthogonal coordinate system. By doing so, the image correction is performed.

6. The tomographic image processing apparatus according to claim 5,

7. The correction unit is a direction of a motion vector of the extracted candidate image; One axis of the orthogonal coordinate system; Position the poles so that they are close to parallel.

7. The tomographic image processing apparatus according to claim 6.

8. The calculation unit corrects positions according to the direction of the representative vector so that the distribution of the calculated placement positions satisfies a predetermined placement condition.

8. The tomographic image processing apparatus according to claim 4, wherein the tomographic image processing apparatus is a tomographic image processing apparatus.

9. The calculation unit performs the position correction by performing scaling along the direction of the representative vector. The tomography apparatus according to claim 8 .

10. The generating unit draws an average image while the image of the extracted candidate is being tracked at the calculated arrangement position.

2. The tomographic image processing apparatus according to claim 1,

11. The desired tracking conditions are set based on the color of the fluorescence emitted by the tissue.

2. The tomographic image processing apparatus according to claim 1,

12. A fluorescent probe that fuses with the tissue is administered to the living body, causing the tissue to emit fluorescence.

12. The tomographic image processing apparatus according to claim 11.

13. By producing a fluorescent protein in the tissue, the tissue emits fluorescence.

12. The tomographic image processing apparatus according to claim 11.

14. the tissue is a cell or molecule of the organism; The desired tracking condition is set based on whether or not at least one of the size, brightness value, and shape of the area in which the desired color is drawn is within a predetermined range.

10. The tomographic image processing apparatus according to claim 9.

15. The tracking unit does not target images of tissue that do not satisfy the desired tracking condition.

15. The tomographic image processing apparatus according to claim 14.

16. an acquiring step of acquiring a moving image of a cross section of the living body through a light receiving surface of the piercing body to be inserted into the living body; a correction step of correcting an image of each frame of the acquired moving image based on the shape of the light receiving surface; a tracking step of tracking candidate depictions of said biological tissue in said image-corrected video that satisfy a desired tracking condition; an estimation step of estimating a representative vector in the image-corrected moving image based on the movement vectors of the tracked candidates; an extraction step of extracting candidates whose motion vectors are similar to the estimated representative vector; a calculation step of calculating a placement position where the image of the extracted candidate should be placed based on a drawing position in a frame where the image of the extracted candidate is drawn in the video and an order of the frames; A generating step of generating a pseudo tomographic image of the living body by drawing an image of the extracted candidate at the calculated arrangement position. A tomographic image processing method comprising:

17. Computer, an acquisition unit that acquires a video of a cross section of the living body via a light receiving surface of a piercing body that is inserted into the living body; a correction unit that performs image correction for each frame of the acquired moving image based on the shape of the light receiving surface; a tracking unit that tracks candidates of images of tissues of the living body that satisfy a desired tracking condition in the image-corrected moving image; an estimation unit that estimates a representative vector in the image-corrected moving image based on the movement vectors of the tracked candidates; an extraction unit that extracts candidates whose motion vectors are similar to the estimated representative vector; a calculation unit that calculates a placement position where the image of the extracted candidate should be placed based on a drawing position in a frame where the image of the extracted candidate is drawn in the video and an order of the frames; a generating unit for generating a pseudo tomographic image of the living body by drawing an image of the extracted candidate at the calculated arrangement position. A program characterized by causing the program to function as a

18. A non-transitory computer-readable information recording medium having the program according to claim 17 recorded thereon.