Information processing device and information processing program

Abstract

To identify a range of an endoscopic image out of a screen displaying the endoscopic image.SOLUTION: An information processing device 3 acquires a designation region 20 out of an endoscopic screen 8, identifies the range of an endoscopic image 4 based on the acquired designation region 20, extracts from the endoscopic screen 8 the endoscopic image 4 whose range is identified, generates a virtual endoscopic image 6 obtained in a case of viewing a bronchus model 5 from a viewpoint identical to a viewpoint from which the endoscopic image 4 is captured, and displays the extracted endoscopic image 4 with the virtual endoscopic image 6 on a navigation screen 7.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] Patent Document 1 discloses an endoscopic examination support system that stores volume data relating to a tubular object having a lesion and endoscopic image data relating to the lesion, generates multiple virtual endoscopic images based on the volume data, and displays on a display unit a combination of images that maximizes the similarity between the endoscopic image and the virtual endoscopic image.

[0003] Patent Document 2 discloses an image processing device that acquires an endoscopic image and a three-dimensional image, acquires viewpoint information from the acquired endoscopic image and three-dimensional image, generates a virtual endoscopic image based on the three-dimensional image and the viewpoint information, and displays the endoscopic image and the virtual endoscopic image on a screen.

[0004] Patent Document 3 discloses an information processing device that displays an endoscopic image and an image showing a virtual endoscopic viewpoint based on hollow organ volume data.

[0005] Patent Document 4 discloses an image processing device that displays an endoscopic image and a three-dimensional image generated based on the insertion direction of the endoscope and the positional relationship of the subject. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-173 [Patent Document 2] International Publication No. 2019 / 130868 [Patent Document 3] Japanese Patent Application Publication No. 2023-176613 [Patent Document 4] International Publication No. 2017 / 158896 Summary of the Invention [Problem to be solved by the invention]

[0007] In endoscopic examinations and the like, an information processing device is sometimes used that estimates the viewpoint of an endoscope in a living body from an endoscopic image and notifies a medical professional of the direction in which the endoscope should travel so that the endoscope reaches the target location.

[0008] To estimate the viewpoint of the endoscope, it is necessary to acquire an endoscopic image. However, depending on the model of endoscope, it may not be possible to acquire the endoscopic image as image data, and the endoscopic image may be displayed on a dedicated endoscopic screen.

[0009] In addition to the endoscopic image, the endoscopic screen also displays additional information related to the endoscopic examination, and the display position, shape, and size of the endoscopic image on the endoscopic screen may differ depending on the model of endoscope.

[0010] The present disclosure has been made in consideration of the above circumstances, and aims to provide an information processing device and an information processing program that can identify the range of an endoscopic image from a screen that displays the endoscopic image. [Means for solving the problem]

[0011] An information processing device of a first aspect according to the technology of the present disclosure includes a processor, which acquires a designated area designated by a user from an endoscopic screen including an endoscopic image of a living organism, the designated area being an area composed of at least one pixel included in the endoscopic image, identifies the range of the endoscopic image based on the acquired designated area, extracts the endoscopic image whose range has been identified from the endoscopic screen, generates a virtual endoscopic image obtained when a three-dimensional model of the living organism previously photographed from the same viewpoint as the viewpoint at which the endoscopic image was photographed, and displays the endoscopic image extracted from the endoscopic screen together with the virtual endoscopic image.

[0012] An information processing device of a second aspect according to the technique of the present disclosure is the information processing device according to the first aspect, wherein the processor acquires a closed region designated by a user as a designated region.

[0013] An information processing device of a third aspect according to the technology of the present disclosure is an information processing device according to the second aspect, in which the processor identifies the range of the endoscopic image based on a change area formed by pixels that include at least one pixel included in a closed area and whose pixel values ​​change over time together with neighboring pixels.

[0014] In a fourth aspect of the information processing device according to the technology of the present disclosure, in the information processing device according to the second aspect, the closed area is an area drawn using closed area information selected by a user from at least one closed area information pre-registered in a storage device.

[0015] In a fifth aspect of the information processing device according to the technology of the present disclosure, in the information processing device according to the first aspect, when multiple ranges of the endoscopic image are detected, the processor identifies the range of the endoscopic image that occupies the largest area in the specified region as the range of the endoscopic image.

[0016] An information processing device of a sixth aspect relating to the technology of the present disclosure is an information processing device of the first aspect, in which, when multiple ranges of an endoscopic image are detected, the processor identifies, among the multiple ranges of the endoscopic image, the range of the endoscopic image that has the highest degree of correspondence with a specified area as the range of the endoscopic image.

[0017] An information processing device of a seventh aspect according to the technique of the present disclosure is the information processing device according to the first aspect, wherein the processor acquires a first point on an endoscopic image designated by a user as a designated area.

[0018] An information processing device of an eighth aspect according to the technology of the present disclosure is an information processing device according to the seventh aspect, in which the processor identifies the range of the endoscopic image based on a change area composed of pixels that include a pixel designated as a first point and whose pixel values ​​change over time together with neighboring pixels.

[0019] An information processing device of a ninth aspect according to the technology of the present disclosure is an information processing device of the eighth aspect, in which the processor further acquires a second point that is specified by the user and is not within the range of the endoscopic image, and configures the change area so that it does not include the pixel specified as the second point.

[0020] An information processing device of a 10th aspect of the technology disclosed herein is an information processing device of any one of the first to ninth aspects, in which, after the processor has determined the range of an endoscopic image, if an endoscopic image of another model is acquired that was captured by an endoscope of a different model from the endoscope that captured the endoscopic image used to determine the range of the endoscopic image, the processor re-determines the range of the endoscopic image based on the newly acquired specified area.

[0021] An information processing device of an eleventh aspect according to the technology of the present disclosure is an information processing device according to any one of the first to ninth aspects, wherein the processor stops displaying the endoscopic image and the virtual endoscopic image when an image other than the endoscopic image is acquired.

[0022] An information processing device of a twelfth aspect of the technology disclosed herein is an information processing device according to any one of the third, eighth, and ninth aspects, in which the processor refers to the pixel value of each pixel for each of a predetermined number of endoscopic screen images, each of which includes one endoscopic image taken in chronological order, and identifies a change area based on the amount of change in pixel value for each pixel at the same position in chronological order.

[0023] An information processing device of a thirteenth aspect relating to the technology of the present disclosure is an information processing device of the twelfth aspect, in which the processor defines a collection of pixels, including at least one pixel included in the specified area, whose pixel value change amount is greater than or equal to a threshold, as a changed area.

[0024] An information processing device of a 14th aspect of the technology disclosed herein is an information processing device of any one of the first to ninth aspects, in which the processor displays an operation item on the endoscopic screen for re-identifying the range of the endoscopic image.

[0025] In a 15th aspect of the information processing device according to the technology of the present disclosure, in the information processing device according to the first aspect, the processor further acquires model information of the endoscope capturing the endoscopic image, and if range information representing the range of the endoscopic image for the model information is pre-registered in the storage device, the processor identifies the range of the endoscopic image using the range information corresponding to the model information of the endoscope.

[0026] An information processing device of a sixteenth aspect according to the technique of the present disclosure is the information processing device of the fifteenth aspect, wherein the processor outputs to the outside the correspondence between the endoscope model information and the range information stored in the storage device.

[0027] An information processing device of a seventeenth aspect according to the technique of the present disclosure is the information processing device according to the second aspect, wherein the processor identifies a closed region as the range of the endoscopic image.

[0028] An information processing program of an 18th aspect according to the technology of the present disclosure is a program for causing a computer to execute a process of acquiring a designated area designated by a user from an endoscopic screen including an endoscopic image of a living organism, the designated area being an area composed of at least one pixel included in the endoscopic image, identifying the range of the endoscopic image based on the acquired designated area, extracting the endoscopic image whose range has been identified from the endoscopic screen, generating a virtual endoscopic image obtained when a three-dimensional model of the living organism previously photographed from the same viewpoint as the viewpoint at which the endoscopic image was photographed, and displaying the endoscopic image extracted from the endoscopic screen together with the virtual endoscopic image. [Effects of the Invention]

[0029] According to the present disclosure, it is possible to identify the range of an endoscopic image from a screen that displays the endoscopic image. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 10 is a diagram showing an example of an endoscope screen. [Figure 3] FIG. 2 is a diagram illustrating an example of a bronchial model. [Figure 4] FIG. 10 is a diagram illustrating an example of a navigation screen. [Figure 5] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device configured by a computer. [Figure 6] 10 is a flowchart showing an example of the flow of a notification process of navigation information of an endoscope. [Figure 7] 10A and 10B are diagrams showing examples of designating a designated region that surrounds the entire endoscopic image. [Figure 8] 10A and 10B are diagrams showing examples of designating a designated area that surrounds a part of an endoscopic image. [Figure 9] 10 is a flowchart showing an example of the flow of a specification process for specifying a range of an endoscopic image. [Figure 10] 10A and 10B are diagrams illustrating an example of specifying the range of an endoscopic image by selecting a model of endoscope. [Figure 11] 10A and 10B are diagrams illustrating an example of designating a designated area along the contour of an endoscopic image. [Figure 12] 10A and 10B are diagrams illustrating an example in which ranges of a plurality of endoscopic images are detected for a specified region. [Figure 13] FIG. 10 is a diagram showing an example of designating a first location and a second location. DETAILED DESCRIPTION OF THE INVENTION

[0031] Hereinafter, the present embodiment will be described with reference to the drawings. The same components and processes are denoted by the same reference numerals throughout the drawings, and duplicated explanations will be omitted. The dimensional proportions in the drawings are exaggerated for the sake of explanation, and may differ from the actual proportions.

[0032] Fig. 1 is a diagram showing an example of the configuration of an information processing system 1 according to the first embodiment. As shown in Fig. 1, the information processing system 1 includes an endoscope 2 and an information processing device 3. The tip of the endoscope 2 is provided with an imaging device such as a camera.

[0033] There are no restrictions on the location of the living body into which the endoscope 2 is inserted, and it may be, for example, an endoscope 2 for the digestive organs or a ureteroscope, but as an example, the endoscope 2 of the present disclosure will be described as a bronchoscope inserted into the bronchi of the subject.

[0034] When inserting the endoscope 2 into the bronchi and moving it to the desired position for examination, the bronchi have many branching locations and similar shapes. Therefore, a navigation device may be used that notifies the medical staff operating the endoscope 2 of support information such as the position and direction of the endoscope 2 in the bronchi and the direction in which the endoscope 2 should be advanced to reach the desired position for examination (hereinafter referred to as the "target position"). The information processing device 3 is an example of a navigation device that notifies the medical staff of support information (hereinafter referred to as "navigation information") related to the movement of the endoscope 2, such as the direction of travel of the endoscope 2.

[0035] Therefore, the information processing system 1 transmits images captured by the endoscope 2, i.e., endoscopic images 4, to the information processing device 3. The endoscopic images 4 may be moving or still images, and the endoscope 2 transmits the endoscopic images 4 to the information processing device 3 at predetermined intervals. As an example, the information processing device 3 receives 30 endoscopic images 4 per second from the endoscope 2, but there is no restriction on the number of endoscopic images 4 per second that the information processing device 3 receives from the endoscope 2, i.e., the frame rate.

[0036] Depending on the model of the endoscope 2, for example, the manufacturer of the endoscope 2 may not disclose the image data format or acquisition method of the endoscopic image 4, and therefore the endoscopic image 4 may only be displayed on the endoscopic screen 8 displayed by a display program dedicated to that model provided by the manufacturer of the endoscope 2. In such cases, the information processing device 3 acquires the endoscopic image 4 from the endoscopic screen 8, estimates the position and posture (hereinafter referred to as the "viewpoint") of the endoscope 2 that captured the endoscopic image 4, and notifies the traveling direction of the endoscope 2.

[0037] Hereinafter, an information processing device 3 that notifies a medical worker of the traveling direction of an endoscope 2, for an endoscope 2 that can only acquire an endoscopic image 4 from an endoscopic screen 8, will be described.

[0038] As shown in FIG. 1, the information processing device 3 includes the following functional units: an acquisition unit 3A, a specification unit 3B, a model acquisition unit 3C, a storage unit 3D, a generation unit 3E, an extraction unit 3F, and a display unit 3G.

[0039] The endoscopic image 4 received by the information processing device 3 from the endoscope 2 is displayed on an endoscopic screen 8 that is displayed on a display unit 14 (see FIG. 5) by a display program pre-installed in the information processing device 3. On the endoscopic screen 8, the endoscopic image 4 is updated according to the frame rate of the imaging device built into the endoscope 2. The display program is provided by the manufacturer of the endoscope 2, for example, for each model of the endoscope 2.

[0040] FIG. 2 is a diagram showing an example of an endoscopic screen 8. In addition to the endoscopic image 4, the endoscopic screen 8 also displays information related to the endoscope 2 (hereinafter referred to as "accompanying information"). A thumbnail image 9, which is an example of the accompanying information, displays a snapshot of the endoscopic image 4. Furthermore, an area 8A of the endoscopic screen 8 displays accompanying information such as a management number assigned to the endoscope 2 for managing the endoscope 2 and the settings of the endoscope 2. Thus, in addition to the endoscopic image 4, the endoscopic screen 8 displays accompanying information represented by text and images, and operation objects such as buttons. In other words, the endoscopic screen 8 provides a user interface (I / F) for the endoscope 2. The display position of the accompanying information on the endoscopic screen 8, the content of the accompanying information, and the display content of the endoscopic screen 8, such as the display position, display size, and display shape of the endoscopic image 4, and the display position of the display content, differ for each endoscopic screen 8 associated with the model of the endoscope 2.

[0041] The acquisition unit 3A acquires model information indicating the model of the endoscope 2, which is transmitted from the endoscope 2 along with the endoscopic image 4. The model information is information that can be read by the information processing device 3, so the information processing device 3 can identify the model of the endoscope 2 that is capturing the endoscopic image 4.

[0042] As described above, where and at what size the endoscopic image 4 is displayed on the endoscopic screen 8 may differ depending on the model of endoscope 2 used, so the medical professional specifies a designated area 20 that indicates the approximate display position of the endoscopic image 4 on the endoscopic screen 8 where the endoscopic image 4 is displayed. The operation by the medical professional to specify the designated area 20 is performed via the operation unit 13.

[0043] There are no restrictions on the operating method used by the user to specify the designated area 20, and the user may specify the designated area 20 by operating a mouse, or by operating a touch panel superimposed on the display unit 14. In Figure 2, the area indicated by the dotted line indicates the designated area 20 specified by the medical staff.

[0044] The acquisition unit 3A acquires position information of the designated area 20 designated by the user from the endoscopic screen 8. In the present disclosure, acquiring the position information of the designated area 20 is simply referred to as "acquiring the designated area 20." The acquisition unit 3A notifies the identification unit 3B of the acquired designated area 20.

[0045] Meanwhile, the acquisition unit 3A acquires the pixel values ​​of each pixel that constitutes the endoscopic screen 8 at intervals corresponding to the frame rate of the imaging device built into the endoscope 2. Since the endoscopic screen 8 can be reproduced from the pixel values ​​of each acquired pixel, acquiring the pixel values ​​of each pixel that constitutes the endoscopic screen 8 is referred to as "acquiring a snapshot of the endoscopic screen 8." Each snapshot of the endoscopic screen 8 includes one endoscopic image 4 captured in chronological order.

[0046] For ease of explanation, each snapshot of the endoscopic screen 8 is referred to as an "endoscopic screen image 23," and each of the endoscopic screen images 23 arranged in chronological order at intervals corresponding to the frame rate of the imaging device built into the endoscope 2 may be referred to as a "frame." The acquisition unit 3A stores the acquired endoscopic screen images 23 in chronological order in the storage unit 3D.

[0047] The identification unit 3B shown in FIG. 1, which has received the designated area 20 from the acquisition unit 3A, identifies the range of the endoscopic image 4 on the endoscopic screen 8 based on the designated area 20. As already explained, the designated area 20 indicates the approximate display position of the endoscopic image 4 on the endoscopic screen 8, and therefore the designated area 20 does not necessarily correspond directly to the range of the endoscopic image 4. As will be explained in detail later, there are multiple methods for specifying the designated area 20. Specifically, the medical professional may specify the designated area 20 as a closed area drawn, for example, with a circle, ellipse, polygon, or freehand line, or may specify it as a single point on the endoscopic image 4.

[0048] Therefore, the identification unit 3B identifies the range of the endoscopic image 4 using the specified designated area 20 in accordance with the method of designating the designated area 20. In this way, identifying the range of the endoscopic image 4 using the specified designated area 20 is referred to as "identifying the range of the endoscopic image 4 based on the designated area 20."

[0049] To identify the range of the endoscopic image 4, for example, the endoscopic screen image 23 is used. Since the endoscopic screen image 23 is stored in the storage unit 3D, the identification unit 3B acquires a plurality of endoscopic screen images 23 arranged in chronological order from the storage unit 3D. A method for identifying the range of the endoscopic image 4 based on the specified region 20 will be described later.

[0050] In this disclosure, a closed region refers not only to an area surrounded by a contour line, but also to an open section where no contour line exists, as long as the distance between the endpoints of the open section is within a predetermined range, and the contour lines intersect when they are extended from each endpoint of the open section in the direction of the contour line.

[0051] The specifying unit 3B notifies the extracting unit 3F of the range of the endoscopic image 4 specified from the endoscopic screen 8.

[0052] The extraction unit 3F, which receives the range of the endoscopic image 4 from the identification unit 3B, extracts the endoscopic image 4 from the endoscopic screen 8 according to the received range of the endoscopic image 4, and notifies the display unit 3G and the generation unit 3E of the extracted endoscopic image 4.

[0053] Meanwhile, a bronchial model 5 representing the shape of the subject's bronchi is used to inform medical personnel of the traveling direction of the endoscope 2. The bronchial model 5 is an example of a three-dimensional model that is generated in advance, for example, by CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) before an endoscopic examination is performed. FIG. 3 shows an example of the bronchial model 5. The bronchial model 5 is generated in advance for each subject and pre-stored in a 3D storage unit. The internal structure of the bronchi tends to be more clearly depicted using CT than MRI. Therefore, it is preferable to generate the bronchial model 5 using CT.

[0054] The model acquisition unit 3C shown in FIG. 1 acquires a bronchial model 5 of a subject to be examined from the storage unit 3D, and notifies the generation unit 3E of the acquired bronchial model 5.

[0055] The generation unit 3E uses the bronchial model 5 received from the model acquisition unit 3C to estimate a viewpoint within the bronchial model 5 such that when the bronchial model 5 is viewed from inside the bronchial model 5, an image similar to the endoscopic image 4 received from the extraction unit 3F is obtained, and generates a virtual endoscopic image 6 which is an image of the bronchial model 5 viewed from the estimated viewpoint.

[0056] That is, every time the generation unit 3E receives an endoscopic image 4, it generates a virtual endoscopic image 6 (hereinafter referred to as a "virtual endoscopic image 6 corresponding to the viewpoint of the endoscope 2") that is obtained when the bronchial model 5 is viewed from the same viewpoint as the viewpoint at which the endoscopic image 4 was captured. The generation unit 3E notifies the display unit 3G of the generated virtual endoscopic image 6 corresponding to the viewpoint of the endoscope 2.

[0057] The viewpoint of the endoscope 2 can be estimated, for example, by comparing the similarities between the endoscopic image 4 and virtual endoscopic images 6 generated from various viewpoints, and determining the viewpoint from which the virtual endoscopic image 6 with the highest similarity is obtained as the viewpoint of the endoscope 2. When the generation unit 3E receives the endoscopic image 4 and a virtual endoscopic image 6 generated from a predetermined viewpoint, the generation unit 3E may estimate the viewpoint of the endoscope 2 using an estimation model that estimates the viewpoint difference between the input endoscopic image 4 and the virtual endoscopic image 6. In other words, there are no restrictions on the method for estimating the viewpoint of the endoscope 2 in the generation unit 3E, and any estimation method may be used as long as it is a method that can estimate the viewpoint of the endoscope 2.

[0058] The display unit 3G displays the endoscopic image 4 extracted from the endoscopic screen 8 by the extraction unit 3F on the navigation screen 7 together with the virtual endoscopic image 6 generated by the generation unit 3E, which corresponds to the viewpoint of the endoscope 2 estimated from the endoscopic image 4.

[0059] The navigation screen 7 is a screen that notifies medical personnel of navigation information. Fig. 4 is a diagram showing an example of the navigation screen 7. As shown in Fig. 4, the navigation screen 7 displays an endoscopic image 4 and a virtual endoscopic image 6 that corresponds to the viewpoint of the endoscope 2. Note that, since the endoscopic image 4 is already displayed on the endoscopic screen 8, the display unit 3G may display only the virtual endoscopic image 6 that corresponds to the viewpoint of the endoscope 2 on the navigation screen 7.

[0060] If the endoscopic image 4 changes as the endoscope 2 moves, the virtual endoscopic image 6 is also updated, so that the virtual endoscopic image 6 that follows the movement of the endoscope 2 is displayed on the navigation screen 7, and the position of the tip of the endoscope 2 is reflected in the bronchial model 5, making it easier to grasp the position of the tip of the endoscope 2. Furthermore, by searching for a path from the position of the tip of the endoscope 2 to the target location, the traveling direction for the endoscope 2 to reach the target location can be notified to the medical staff. The traveling direction of the endoscope 2 is notified by voice and by display on the navigation screen 7, for example.

[0061] 1 stores a bronchial model 5 for each subject, an endoscopic screen image 23, a display program for displaying an endoscopic screen 8, various parameters referenced in the process of each functional unit performing processing, and various programs executed by the information processing device 3. Note that the information processing device 3 does not necessarily have to include the storage unit 3D, and an external device other than the information processing device 3, such as a data server, may be used as the storage unit 3D.

[0062] Furthermore, the information processing device 3 may include functional units other than the functional units illustrated in Fig. 1. For example, the information processing device 3 may include an output unit that outputs information obtained by the information processing device 3 and information stored in the storage unit 3D to the outside.

[0063] 1 having these functional units is configured by, for example, a computer. Fig. 5 is a diagram showing an example of the configuration of the information processing device 3 configured by a computer.

[0064] 5, the information processing device 3 includes a control unit 10, an interface (I / F) unit 12, an operation unit 13, a display unit 14, and a storage unit 15. The control unit 10, the I / F unit 12, the operation unit 13, the display unit 14, and the storage unit 15 are connected via a bus 16 so as to be able to exchange various information with one another.

[0065] The control unit 10 controls the operation of the information processing device 3 based on instructions from a medical professional. The control unit 10 is an example of a processor, and includes a CPU (Central Processing Unit) 10A, a ROM (Read Only Memory) 10B, and a RAM (Random Access Memory) 10C, which are responsible for processing the various functional units of the information processing device 3 shown in Fig. 1. The ROM 10B pre-stores a control program 11 that the CPU 10A reads to notify navigation information of the endoscope 2, various programs including display programs for each model of the endoscope 2 provided by the manufacturer of the endoscope 2, and various parameters that the CPU 10A references when controlling the operation of the information processing device 3. The RAM 10C is used as a temporary work area for the CPU 10A.

[0066] The I / F unit 12 exchanges various types of information with the endoscope 2 using wireless communication or wired communication. The information processing device 3 acquires the endoscopic image 4 and model information of the endoscope 2 from the endoscope 2 through the I / F unit 12. In addition, the information processing device 3 outputs various types of information to the outside of the information processing device 3 through the I / F unit 12.

[0067] The operation unit 13 is used by a medical professional to input, for example, instructions and various information to the information processing device 3. There are no restrictions on the operation form of the operation unit 13, and it is possible to accept operations using, for example, a switch, a touch panel, an electronic pen, a keyboard, a mouse, etc.

[0068] The display unit 14 displays, for example, information processed by the control unit 10 and various screens such as the endoscope screen 8 and the navigation screen 7.

[0069] The storage unit 15 stores, for example, a bronchial model 5 for each subject, an endoscope screen image 23, a display program for displaying the endoscope screen 8, various parameters referenced by the information processing device 3, and various programs executed by the information processing device 3. The storage unit 15 is an example of a storage device that maintains stored information even if the power supplied to the storage unit 15 is cut off, and for example, a semiconductor memory such as an SSD (Solid State Drive) is used, but a hard disk may also be used.

[0070] Next, the processing of the information processing device 3 that notifies the navigation information of the endoscope 2 will be described in detail.

[0071] 6 is a flowchart showing an example of the flow of a notification process of navigation information executed by the information processing device 3 when an instruction to start an endoscopic examination is received by a medical professional operating the operation unit 13. The CPU 10A of the information processing device 3 reads a control program 11 from the ROM 10B and executes the notification process. The control program 11 is an example of an information processing program disclosed herein.

[0072] It is assumed that the storage unit 15 stores at least N (N is an integer equal to or greater than 3) endoscopic screen images 23 in time series corresponding to the model of the endoscope 2. The variable N is a value set in advance by a medical professional and is stored in the storage unit 15. Hereinafter, the variable N will be referred to as the "prescribed number of frames." It is also assumed that the display unit 14 displays an endoscopic screen 8 corresponding to the model of the endoscope 2 being used. Prior to executing the notification process, the information processing device 3 acquires model information from the endoscope 2.

[0073] First, a medical professional operates the operation unit 13 to designate a designated area 20 on the endoscope screen 8 so that at least one pixel constituting the endoscope image 4 is included.

[0074] After the medical staff has finished specifying the specified area 20, in step S10, the CPU 10A acquires the specified area 20 specified on the endoscope screen 8 by the medical staff.

[0075] 7 and 8 are diagrams showing examples of designation of the designated region 20 by a closed region. In the example shown in Fig. 7, the designated region 20 is designated so as to surround the entire endoscopic image 4. In the example shown in Fig. 8, the designated region 20 is designated so as to include a part of the endoscopic image 4. In the examples shown in Fig. 7 and 8, examples are shown in which a circle and a rectangle are used as the closed region representing the designated region 20, but as already explained, the designated region may also be another polygon such as a triangle or a pentagon, or a closed region drawn with a freehand line, etc.

[0076] The CPU 10A may display on the display unit 14 a menu showing at least one graphic shape pre-registered in the storage unit 15, allowing the medical professional to select one, and then display the selected graphic on the endoscope screen 8. For example, if a circle is selected from the displayed menu, the CPU 10A draws the circle on the endoscope screen 8. The medical professional adjusts the position and size of the circle drawn on the endoscope screen 8 to specify the designated area 20. Therefore, the medical professional does not need to operate a mouse or the like to draw a graphic shape indicating the designated area 20, such as a circle or rectangle, for each endoscopic examination.

[0077] The shape of the figure displayed on the menu by the CPU 10A is a figure that forms a closed area, such as a circle or a rectangle. Note that the shape of the figure displayed on the menu is information that represents the characteristics of the closed area to be drawn on the endoscope screen 8, and is therefore an example of closed area information.

[0078] 7 and 8, there are no restrictions on the shape of the designated region 20 or the range of the designated region 20, as long as at least one pixel constituting the endoscopic image 4 is included in the designated region 20. In other words, medical personnel can designate the designated region 20 without worrying about ensuring that the designated region 20 does not extend beyond the endoscopic image 4 or that the designated region 20 is not too small relative to the endoscopic image 4. Therefore, the operability of designating the designated region 20 is improved compared to when the designated region 20 must be designated along the contour of the endoscopic image 4, for example.

[0079] In step S20 of FIG. 6, CPU 10A executes a specification process for specifying the range of endoscopic image 4 based on designated region 20 acquired by the process of step S10.

[0080] As the endoscope 2 moves through the bronchi, the illumination conditions for the bronchi due to the built-in lighting in the endoscope 2 also change, and so the endoscopic image 4 changes during the endoscopic examination. That is, among the pixels that make up the endoscopic screen 8, the pixel values ​​of the individual pixels that make up the endoscopic image 4 change over time.

[0081] On the other hand, the designated area 20 includes at least one pixel that constitutes the endoscopic image 4. Therefore, by sequentially performing a process of finding a pixel whose pixel value changes from among the pixels that constitute the endoscopic screen 8 included in the designated area 20, and then further finding pixels whose pixel values ​​change from all pixels adjacent to that pixel, for each pixel whose pixel value changes, a collection of pixels whose pixel values ​​change can be obtained. The collection of pixels whose pixel values ​​change thus obtained represents the range of the endoscopic image 4. In other words, the CPU 10A identifies, as the range of the endoscopic image 4, a collection of pixels (hereinafter referred to as a "changing area") that includes at least one pixel included in the designated area 20 and is composed of pixels whose pixel values ​​change over time together with neighboring pixels.

[0082] FIG. 9 is a flowchart showing an example of the process of specifying in step S20 of FIG.

[0083] In step S100, the CPU 10A acquires N endoscope screen images 23 in chronological order from the storage unit 15. In this case, it is preferable that the CPU 10A acquires N endoscope screen images 23 including the most recently acquired endoscope screen image 23.

[0084] In step S110, CPU 10A refers to the pixel value of each pixel for each endoscopic screen image 23 acquired by the processing of step S100, and acquires N pixel values ​​in chronological order for each pixel located at the same position in each endoscopic screen image 23.

[0085] For example, the pixel value of a pixel at each position in each endoscope screen image 23 is V i It is represented as (x, y). Index i is a variable that identifies N endoscopic screen images 23 in chronological order, and index i takes an integer value from 1 to N. Of the N endoscopic screen images 23, for example, an endoscopic screen image 23 represented by an index i of "1" represents the most recent endoscopic screen image 23, and an endoscopic screen image 23 represented by an index i of "N" represents the oldest endoscopic screen image 23. In other words, the more recently acquired an endoscopic screen image 23 is, the smaller the index i value that is associated with it.

[0086] On the other hand, the variables x and y respectively represent the x-coordinate value and the y-coordinate value that represent the position of a pixel in the endoscope screen image 23.

[0087] Therefore, the pixel values ​​acquired by the CPU 10A and N pixel values ​​along the time series of the pixel at the coordinates (x, y) of the endoscope screen image 23 are V1(x, y), V2(x, y), . . . , V N It is expressed as (x,y).

[0088] In step S120, CPU 10A calculates the amount of change in pixel value along the time series for each pixel of endoscope screen image 23 using N pixel values ​​along the time series for each pixel of endoscope screen image 23 acquired by the processing of step S110.

[0089] As long as the amount of change in pixel value over time can be shown, there are no restrictions on the method for calculating the amount of change in pixel value, and any calculation formula may be used to calculate the amount of change in pixel value. Formulas (1) to (4) show examples of calculation formulas for calculating the amount of change in pixel value.

[0090] (Number 1) Max(V1(x,y), ,V N (x,y))-Min(V1(x,y), ,V N (x,y) (1) UpperM%(V1(x,y), ,V N (x,y))-BottomM%(V1(x,y),···,V N (x,y) (2) Max(|V1(x,y)-V2(x,y)|,··,|V N-1 (x,y)-V N (x,y)|) (3) Ave(|V1(x,y)-V2(x,y)|,··,|V N-1 (x,y)-V N (x,y)|) (4)

[0091] In formulas (1) to (4), the Max operator extracts the maximum value from the values ​​in the parentheses, and the Min operator extracts the minimum value from the values ​​in the parentheses. The UpperM% operator extracts the value closest to the top 10% of the values ​​in the parentheses, and the BottomM% operator extracts the value closest to the bottom 10% of the values ​​in the parentheses. The Ave operator calculates the average value of the values ​​in the parentheses.

[0092] In step S130, CPU 10A identifies a changed area from endoscope screen 8 based on the amount of change in pixel value calculated by the process in step S120.

[0093] For example, the CPU 10A identifies as a changed region a group of pixels in which the amount of change in pixel value is equal to or greater than a predetermined threshold and which includes at least one pixel included in the specified region 20. The threshold value to be compared with the pixel value is stored in advance in the storage unit 15, for example, and can be adjusted by a medical professional. Since the range of the changed region changes even for the same endoscopic screen 8 by adjusting the threshold value, it is preferable that the medical professional adjust the threshold value so that the changed region matches the range of the endoscopic image 4 as closely as possible. For example, the CPU 10A may store threshold values ​​set by the medical professional for each model of endoscope 2 in the storage unit 15, and identify the changed region from the endoscopic screen 8 using the threshold value corresponding to the model of endoscope 2. The identified changed region represents the range of the endoscopic image 4 on the endoscopic screen 8. This completes the identification process shown in FIG. 9.

[0094] Step S30 is executed after the identification process is completed by the process of step S20 in Fig. 6. In step S30, CPU 10A extracts an image of the range identified by the process of step S20 from endoscope screen 8 as endoscope image 4.

[0095] In step S40, CPU 10A estimates the viewpoint of the endoscope 2 using the endoscopic image 4 extracted by the processing of step S30 and the virtual endoscopic image 6 generated from the bronchial model 5. As an example, CPU 10A estimates the viewpoint difference between the endoscopic image 4 and the virtual endoscopic image 6 using an estimation model that estimates the viewpoint difference between the endoscopic image 4 and the virtual endoscopic image 6, and estimates the viewpoint of the endoscope 2 using the known viewpoint from which the virtual endoscopic image 6 was generated and the estimated viewpoint difference. As already explained, there are no restrictions on the method for estimating the viewpoint of the endoscope 2, and any estimation method may be used as long as it is a method that allows the viewpoint of the endoscope 2 to be estimated.

[0096] In step S50, CPU 10A generates a virtual endoscopic image 6 that is obtained when the bronchial model 5 is viewed from the same viewpoint as the viewpoint of the endoscope 2 estimated by the processing in step S40. The generated virtual endoscopic image 6 corresponds to the viewpoint of the endoscope 2. CPU 10A also searches for a path from the position corresponding to the estimated viewpoint of the endoscope 2 to the target location using a known search method, for example, a binary search method.

[0097] In step S60, the CPU 10A displays a navigation screen 7 including the endoscopic image 4 and a virtual endoscopic image 6 corresponding to the viewpoint of the endoscope 2 generated by the processing of step S50. The CPU 10A also notifies the medical staff through a speaker of a voice indicating the traveling direction of the endoscope 2, such as "rightward" or "downward," along the route to the target location searched for by the processing of step S50. The CPU 10A may display the traveling direction of the endoscope 2 on the navigation screen 7.

[0098] There may be cases where the endoscopic image 4 is not correctly extracted from the endoscopic screen 8, and, for example, a part of the endoscopic image 4 or an image other than the endoscopic image 4 is displayed in the display position of the navigation screen 7 where the endoscopic image 4 is supposed to be displayed. For this reason, the CPU 10A may display, on the navigation screen 7, a re-identification button (not shown), which is an example of an operation item for redoing the identification of the range of the endoscopic image 4.

[0099] If the re-identification button is pressed, the process proceeds to step S10, where CPU 10A acquires designated region 20 again and identifies the range of endoscopic image 4 on endoscopic screen 8 again.

[0100] In step S70, CPU 10A determines whether an instruction to end the notification process has been received from the medical staff. If an instruction to end the notification process has not been received, CPU 10A proceeds to step S30, and continues to extract an image of the range identified by the process of step S20 from endoscope screen 8 as endoscope image 4.

[0101] That is, until receiving an end instruction from the medical staff, the information processing device 3 extracts, from the endoscopic screen 8, endoscopic images 4 that are updated at intervals corresponding to the frame rate of the imaging device built into the endoscope 2. Then, the CPU 10A estimates the viewpoint of the endoscope 2 for each extracted endoscopic image 4, and displays the extracted endoscopic image 4 and a virtual endoscopic image 6 corresponding to the viewpoint of the endoscope 2 on the navigation screen 7, and notifies the medical staff of the traveling direction of the endoscope 2 to reach the target location.

[0102] On the other hand, if an end instruction is received, the notification process shown in FIG. 6 ends.

[0103] As already explained, there is a dedicated endoscopic screen 8 for each model of endoscope 2. Therefore, if the endoscope 2 used in the endoscopic examination is a model that the information processing device 3 has used in the past, the range of the endoscopic image 4 will have been identified at least once from the endoscopic screen 8 corresponding to the model of the endoscope 2.

[0104] Therefore, in the process of step S20, if the acquired model information of the endoscope 2 is not any model information that has been acquired before, the CPU 10A may store range information that indicates the range of the identified endoscopic image 4 in the storage unit 15 in association with the model information of the endoscope 2. The association between the model information of the endoscope 2 and the range information is managed, for example, as a correspondence table.

[0105] In this case, before acquiring the designated region 20 by the process of step S10 in FIG. 6, the CPU 10A determines whether or not the model information of the endoscope 2 acquired in advance has been registered in the correspondence table.

[0106] If the model information of the acquired endoscope 2 is registered, the range information associated with the model information in the correspondence table can be set as the range of the endoscopic image 4. Therefore, the processes of steps S10 and S20 are omitted and the process proceeds to step S30.

[0107] On the other hand, if the acquired model information of the endoscope 2 is not registered, the notification process is executed from the process in step S10, and the range of the endoscopic image 4 is identified based on the specified region 20.

[0108] In this way, by registering the range of the endoscopic image 4 on the endoscopic screen 8 for each model of endoscope 2, if the endoscope 2 has been used for an endoscopic examination in the past, the range of the endoscopic image 4 can be identified without having the medical professional specify the designated area 20.

[0109] The CPU 10A may allow the medical staff to select the model of the endoscope 2, and then specify the range of the endoscopic image 4 using the correspondence table.

[0110] 10 is a diagram showing an example of specifying the range of the endoscopic image 4 when the medical professional is prompted to select the model of the endoscope 2. In the operation of specifying the designated area 20 by the medical professional executed prior to the processing of step S10 in FIG. 6, the CPU 10A displays on the display unit 14 a selection dialog 22 that displays a list of model names of the endoscopes 2 that correspond to the model information registered in the correspondence table.

[0111] The CPU 10A acquires from the correspondence table range information associated with model information represented by the model name selected by the medical professional from the selection dialog 22, and displays a designated area 20 having a shape according to the range information on the endoscope screen 8. In the example shown in Fig. 10, when model A is selected from the selection dialog 22, a designated area 20A is displayed on the endoscope screen 8, when model B is selected, a designated area 20B is displayed on the endoscope screen 8, and when model C is selected, a designated area 20C is displayed on the endoscope screen 8.

[0112] The ranges of the designated areas 20A, 20B, and 20C displayed on the endoscope screen 8 can be adjusted by a medical professional using a mouse or the like. When the ranges of the designated areas 20A, 20B, and 20C are adjusted, the CPU 10A updates the range information in the correspondence table associated with the model information represented by the selected model name, using the adjusted ranges of the designated areas 20A, 20B, and 20C as the new ranges of the endoscopic image 4 on the endoscope screen 8. Therefore, as in the selection dialog 22 shown in Fig. 10, the CPU 10A may display the last update date and time, which indicates the most recent update date and time of the range information, for each model name.

[0113] By having the medical staff select the model name of the endoscope 2 from the selection dialog 22, the medical staff can visually confirm the range of the endoscopic image 4 indicated by the range information registered in the correspondence table. Therefore, if the range information registered in the correspondence table deviates from the range of the endoscopic image 4 actually displayed on the endoscopic screen 8, the medical staff can adjust the range information.

[0114] In addition, the CPU 10A may, in accordance with instructions from a medical professional, output information associating the model information of the endoscope 2 registered in the correspondence table with the range information of the endoscopic image 4 to the outside of the information processing device 3, for example, via the I / F unit 12. By outputting the information of the correspondence table to the outside, the same correspondence table can be shared with, for example, another information processing device 3.

[0115] Although examples of designating the designated region 20 have been described above using FIGS. 7 and 8, the user may designate the designated region 20 along the contour of the endoscopic image 4.

[0116] 11 is a diagram showing an example of designating the designated area 20 along the contour of the endoscopic image 4. In this case, for example, before designating the designated area 20, the medical professional issues a designation instruction to the information processing device 3 to designate the designated area 20 along the contour of the endoscopic image 4. Upon receiving this designation instruction, the CPU 10A does not execute the processing of step S20 in FIG. 6, but simply sets the range of the designated area 20 acquired by the processing of step S10 as the range of the endoscopic image 4. Therefore, compared to when the rough display position of the endoscopic image 4 is designated by the designated area 20, the load required to identify the range of the endoscopic image 4 is reduced.

[0117] Incidentally, since medical personnel may specify the approximate display position of the endoscopic image 4 as the designated area 20, they may specify the designated area 20 in a manner that includes, for example, the endoscopic image 4 as well as an area other than the endoscopic image 4 where pixel values ​​change. Meanwhile, in the processing of step S20 shown in Fig. 6, the range of the endoscopic image 4 is identified based on the amount of change in pixel values.

[0118] Therefore, depending on how the designated region 20 is specified, multiple change regions may be identified in the processing of step S20, and multiple ranges of the endoscopic image 4 may be detected. In such a case, CPU 10A identifies the range candidate of the endoscopic image 4 that occupies the largest area in the designated region 20 from among the multiple detected ranges of the endoscopic image 4 (hereinafter, sometimes referred to as "range candidate") as the correct range of the endoscopic image 4. This is because, when specifying the designated region 20, medical professionals tend to specify the designated region 20 so that the area occupied by the endoscopic image 4 is largest relative to the designated region 20, so that it is clear that the designated region 20 indicates the position of the endoscopic image 4.

[0119] Fig. 12 is a diagram showing an example in which a plurality of range candidates for the endoscopic image 4 are detected for the specified region 20. In Fig. 12, it is assumed that the pixel values ​​of the pixels constituting the region 8B, region 8C of the endoscopic screen 8 indicated by diagonal lines, and the endoscopic image 4 have changed, and therefore the region 8B, region 8C, and the endoscopic image 4 are detected as range candidates for the endoscopic image 4.

[0120] In this case, CPU 10A specifies, as the range of endoscopic image 4, the endoscopic image 4 that occupies the largest area in specified region 20 among region 8B, region 8C, and endoscopic image 4.

[0121] Furthermore, instead of comparing the area of ​​the range of the endoscopic image 4 that occupies the designated region 20, the CPU 10A may compare the degree of match between the candidate range of the endoscopic image 4 and the designated region 20. The degree of match between the designated region 20 in the endoscopic image 4 and the candidate range is an index that represents the similarity between the shape and size of the designated region 20.

[0122] The CPU 10A identifies, among the multiple range candidates for the endoscopic image 4, the range candidate for the endoscopic image 4 that has the highest degree of match with the designated area 20 as the correct range of the endoscopic image 4. This is because, when a medical professional specifies the designated area 20, they specify the designated area 20 while being conscious of the shape and size of the endoscopic image 4, and therefore the shape and size of the designated area 20 tend to be close to the shape and size of the endoscopic image 4.

[0123] 12 are areas where pixel values ​​change due to changes in characters or noise, the pixel values ​​often do not change as frequently as in endoscopic image 4. Therefore, CPU 10A may perform a comparison after weighting each pixel constituting each range candidate detected as a range candidate in endoscopic image 4 according to the amount of change in pixel value. For example, CPU 10A performs the comparison so that the smaller the amount of change in pixel value, the smaller the influence that pixel has on the comparison result.

[0124] In this way, the information processing device 3 of the present disclosure can identify the correct range of the endoscopic image 4 from the multiple range candidates of the endoscopic image 4, even if multiple range candidates of the endoscopic image 4 are detected from the endoscopic screen 8.

[0125] Depending on the situation, medical personnel may change the model of the endoscope 2 during the endoscopic examination, or connect an ultrasound probe that captures ultrasound images of the target location using ultrasound to the information processing device 3.

[0126] Therefore, if the model of endoscope 2 is changed after the range of endoscopic image 4 has been identified by processing in step S20 of Figure 6, and an endoscopic image 4 is acquired that has been captured by an endoscope 2 of a model different from the endoscope 2 that captured the endoscopic image 4 used to identify the range of endoscopic image 4, the process proceeds to step S10, and CPU 10A re-identifies the range of endoscopic image 4 based on the newly acquired specified area 20.

[0127] Whether or not the model of the endoscope 2 has been changed can be determined from the acquired model information of the endoscope 2. By re-specifying the range of the endoscopic image 4, it is possible to continue to notify the navigation information even if the model of the endoscope 2 has been changed. Note that an endoscopic image 4 captured by an endoscope 2 of a different model is an example of an endoscopic image of a different model.

[0128] On the other hand, when an ultrasound probe is connected to the information processing device 3 instead of the endoscope 2, an ultrasound image is acquired instead of the endoscopic image 4. In this case, there is no need to notify the traveling direction of the endoscope 2, so the CPU 10A terminates the notification process shown in FIG. 6 and stops displaying the endoscopic image 4 and the virtual endoscopic image 6 corresponding to the viewpoint of the endoscope 2. For example, if a medical professional stops using the endoscope 2, the endoscopic image 4 on the endoscopic screen 8 will no longer change. Therefore, the CPU 10A can terminate the notification process shown in FIG. 6 and close the navigation screen when the amount of change in pixel value of each pixel in the already specified range of the endoscopic image 4 on the endoscopic screen 8 becomes less than the threshold. By terminating the notification process shown in FIG. 6, it is possible to stop notification of navigation information in conjunction with the input of an image other than the endoscopic image 4.

[0129] <Modification of the method for specifying the specified area 20> In the acquisition of the designated area 20 in step S10 of Fig. 6, an example has been described in which the designated area 20 is designated as a closed area, but the method by which the medical professional designates the designated area 20 is not limited to designating it as a closed area. For example, the medical professional may designate one point on the endoscopic image 4 as the designated area 20. The one point on the endoscopic image 4 designated by the medical professional is referred to as a "first point."

[0130] In this case, CPU 10A acquires the designated first location as designated area 20 in step S10.

[0131] Then, in step S20, CPU 10A calculates the amount of change in pixel value of each pixel adjacent to the designated area 20, i.e., the pixel constituting the first point, and identifies the change area including the pixel of the first point as the range of the endoscopic image 4, starting from the first point.

[0132] Therefore, medical personnel can more easily specify the designated area 20 than when specifying the designated area 20 as a closed area.

[0133] There are many variations in the method of specifying a point on the endoscopic image 4 and identifying the range of the endoscopic image 4.

[0134] For example, in addition to the first point, the medical professional may further specify one point on the endoscopic screen 8 that is not within the range of the endoscopic image 4. The point on the endoscopic screen 8 that is not within the range of the endoscopic image 4 specified by the medical professional is referred to as the "second point."

[0135] Fig. 13 is a diagram showing an example of designating a first location and a second location, in which a location 21A represents the first location and a location 21B represents the second location.

[0136] If a designation order is defined such that a first location is designated and then a second location is designated, CPU 10A recognizes the first designated location as the first location.

[0137] In the process of identifying the changed area starting from the first point, the CPU 10A configures the changed area so as not to include the pixel of the endoscopic screen 8 that is designated as the later designated point, i.e., the second point. As a result, the endoscopic image 4 indicated by the diagonal lines in FIG. 13 is identified as the changed area.

[0138] When identifying a change area starting from the first point, the condition that pixels at the second point are not included is added, which may improve the accuracy of identifying the range of the endoscopic image 4 compared to when only the first point is specified.

[0139] In addition, in a situation where the position of the endoscopic image 4 is specified by a method of specifying one point on the endoscopic image 4, if, for example, the endoscopic image 4 on the navigation screen 7 displays only a portion of the original endoscopic image 4 and the medical professional presses the re-identification button, the CPU 10A may impose a restriction so that the method of specifying the specified area 20 by the medical professional is a method of specifying other than a method of specifying one point on the endoscopic image 4. For example, when re-identifying the range of the endoscopic image 4, the CPU 10A causes the method of specifying the specified area 20 by a closed area as shown in FIGS. 7 and 8 to be performed.

[0140] If a designation method that designates one point on the endoscopic image 4 is used again when re-identifying the range of the endoscopic image 4, there is a high probability that the endoscopic image 4 will not be displayed correctly on the navigation screen 7, just as it was the previous time. Therefore, by having the medical professional use a different designation method, the accuracy of identifying the range of the endoscopic image 4 may be improved compared to when the same designation method is repeatedly used. Naturally, by designating multiple points on the endoscopic screen 8 that are not within the range of the endoscopic image 4 and multiple points on the endoscopic screen 8 that are not within the range of the endoscopic image 4, the change area may be configured to include multiple points within the range of the endoscopic image 4, but not include multiple points on the endoscopic screen 8 that are not within the range of the endoscopic image 4.

[0141] As described above, the information processing device 3 of the present disclosure acquires the specified region 20 from the endoscopic screen 8, identifies the range of the endoscopic image 4 based on the acquired specified region 20, extracts the endoscopic image 4 with the identified range from the endoscopic screen 8, generates a virtual endoscopic image 6 obtained when the bronchial model 5 is viewed from the same viewpoint as the viewpoint at which the extracted endoscopic image 4 was captured, and displays the extracted endoscopic image 4 together with the virtual endoscopic image 6 on the navigation screen 7. Therefore, even if the endoscopic image 4 can only be acquired from the endoscopic screen 8, it is possible to identify the range of the endoscopic image 4 on the endoscopic screen 8 and notify navigation information for the endoscope 2.

[0142] In the above embodiment, the notification process is implemented by software processing. However, the process equivalent to the flowchart of the notification process may be implemented by hardware. In this case, the notification process can be executed faster than when the notification process is implemented by software processing.

[0143] In the above embodiment, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU 10A) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0144] Furthermore, the operations of the processors in the above embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processors is not limited to the order described in the above embodiments, and may be changed as appropriate.

[0145] In the above embodiment, an example has been described in which the control program 11 is stored in the ROM 10B. However, the storage destination of the control program 11 is not limited to the ROM 10B. The control program 11 can also be provided in a form recorded on a computer-readable storage medium.

[0146] For example, the control program 11 may be provided in a form recorded on an optical disk such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a Blu-ray disc. The control program 11 may also be provided in a form recorded on a portable semiconductor memory such as a USB (Universal Serial Bus) memory or a memory card. ROM 10B, a CD-ROM, a DVD-ROM, a Blu-ray disc, a USB, and a memory card are examples of non-transitory storage media.

[0147] Furthermore, the control unit 10 may download the control program 11 from an external device connected to a communication line via the I / F unit 12, and store the downloaded control program 11 in the ROM 10B of the control unit 10.

[0148] The following additional notes are further disclosed regarding the above embodiment.

[0149] (Appendix 1) a processor; The processor: acquiring a designated area designated by a user from an endoscope screen including an endoscope image of a living body, the designated area being an area constituted by at least one pixel included in the endoscope image; Identifying a range of the endoscopic image based on the acquired designated region; extracting the endoscopic image with the specified range from the endoscopic screen; generating a virtual endoscopic image obtained when viewing a three-dimensional model of the living body photographed in advance from the same viewpoint as the viewpoint at which the endoscopic image was photographed; The endoscopic image extracted from the endoscopic screen is displayed together with the virtual endoscopic image. Information processing device.

[0150] (Appendix 2) The processor acquires a closed region designated by a user as the designated region. 10. The information processing device according to claim 1.

[0151] (Appendix 3) The processor identifies the range of the endoscopic image based on a change region formed by pixels that include at least one pixel included in the closed region and whose pixel values ​​change over time together with neighboring pixels. 3. The information processing device according to claim 2.

[0152] (Appendix 4) The closed region is an area drawn using closed region information selected by a user from at least one closed region information registered in advance in a storage device. 4. The information processing device according to claim 2 or 3.

[0153] (Appendix 5) When a plurality of ranges of the endoscopic image are detected, the processor identifies the range of the endoscopic image that occupies the largest area in the specified region as the range of the endoscopic image. 5. The information processing device according to any one of Supplementary notes 1 to 4.

[0154] (Appendix 6) When a plurality of ranges of the endoscopic image are detected, the processor identifies, among the ranges of the plurality of endoscopic images, the range of the endoscopic image that has the highest degree of coincidence with the specified region as the range of the endoscopic image. 5. The information processing device according to any one of Supplementary notes 1 to 4.

[0155] (Appendix 7) The processor acquires a first point on the endoscopic image designated by a user as the designated region. 10. The information processing device according to claim 1.

[0156] (Appendix 8) The processor identifies a range of the endoscopic image based on a change region formed by pixels that include the pixel designated as the first point and whose pixel values ​​change over time together with neighboring pixels. 8. The information processing device according to claim 7.

[0157] (Appendix 9) The processor further acquires a second point that is specified by a user and is not within the range of the endoscopic image, and configures the changed region so as not to include the pixel specified as the second point. 9. The information processing device according to claim 8.

[0158] (Appendix 10) After specifying the range of the endoscopic image, if an endoscopic image of a different model is acquired that is captured by an endoscope of a different model from the endoscope that captured the endoscopic image used to specify the range of the endoscopic image, the processor re-specifies the range of the endoscopic image based on the newly acquired specified region. 10. The information processing device according to any one of Supplementary notes 1 to 9.

[0159] (Appendix 11) The processor stops displaying the endoscopic image and the virtual endoscopic image when an image different from the endoscopic image is acquired. An information processing device according to any one of Supplementary notes 1 to 10.

[0160] (Appendix 12) The processor refers to the pixel value of each pixel for each of a predetermined number of endoscopic screen images, each of which includes one of the endoscopic images captured in time series, and identifies the change area based on the amount of change in pixel value for each pixel at the same position in time series. An information processing device according to any one of Supplementary Note 3, Supplementary Note 8, and Supplementary Note 9.

[0161] (Appendix 13) The processor determines, as the changed region, a group of pixels including at least one pixel included in the specified region and having a change in pixel value equal to or greater than a threshold. 13. The information processing device according to claim 12.

[0162] (Appendix 14) The processor displays an operation item on the endoscope screen for allowing the user to redo the specification of the range of the endoscope image. An information processing device according to any one of Supplementary notes 1 to 13.

[0163] (Appendix 15) The processor further acquires model information of the endoscope capturing the endoscopic image, and if range information representing the range of the endoscopic image for the model information is pre-registered in a storage device, identifies the range of the endoscopic image using the range information corresponding to the model information of the endoscope. An information processing device according to any one of Supplementary notes 1 to 14.

[0164] (Appendix 16) The processor outputs to an external device the correspondence between the model information of the endoscope and the range information stored in the storage device. 16. The information processing device according to claim 15.

[0165] (Appendix 17) The processor identifies the closed region as a range of the endoscopic image. 3. The information processing device according to claim 2.

[0166] (Appendix 18) acquiring a designated area designated by a user from an endoscope screen including an endoscope image of a living body, the designated area being an area constituted by at least one pixel included in the endoscope image; Identifying a range of the endoscopic image based on the acquired designated region; extracting the endoscopic image with the specified range from the endoscopic screen; generating a virtual endoscopic image obtained when viewing a three-dimensional model of the living body photographed in advance from the same viewpoint as the viewpoint at which the endoscopic image was photographed; and causing a computer to execute a process of displaying the endoscopic image extracted from the endoscopic screen together with the virtual endoscopic image. Information processing program. [Explanation of symbols]

[0167] 1. Information Processing Systems 2 Endoscopy 3. Information processing equipment 3A Acquisition Department 3B Specific part 3C Model Acquisition Department 3D storage 3E generation part 3F Extraction section 3G display 4 Endoscopic images 5 Bronchial Model 6 Virtual endoscopy images 7 Navigation screen 8 Endoscope Screen 8A, 8B, 8C area 9 Thumbnail Images 10. Control Unit 10A CPU 10B ROM 10C RAM 11 Control Program 12 I / F units 13 Operation unit 14 Display unit 15 Storage Unit 16 Bus 20, 20A, 20B, 20C designated area 22 Selection Dialog 23 Endoscope screen image

Claims

1. a processor; The processor: acquiring a designated area, which is an area configured by at least one pixel included in an endoscopic image of a living body, from an endoscopic screen including the endoscopic image of the living body and which is designated by a user; Identifying a range of the endoscopic image based on the acquired designated region; extracting the endoscopic image with the specified range from the endoscopic screen; generating a virtual endoscopic image obtained when a three-dimensional model of the living body photographed in advance is viewed from the same viewpoint as the viewpoint at which the endoscopic image was photographed; The endoscopic image extracted from the endoscopic screen is displayed together with the virtual endoscopic image. Information processing device.

2. The processor acquires a closed region designated by a user as the designated region. The information processing device according to claim 1 .

3. The processor identifies the range of the endoscopic image based on a change region formed by pixels that include at least one pixel included in the closed region and whose pixel values ​​change over time together with neighboring pixels. The information processing device according to claim 2 .

4. The closed region is an area drawn using closed region information selected by a user from at least one closed region information registered in advance in a storage device. The information processing device according to claim 2 .

5. When a plurality of ranges of the endoscopic image are detected, the processor identifies the range of the endoscopic image that occupies the largest area in the specified region as the range of the endoscopic image. The information processing device according to claim 1 .

6. When a plurality of ranges of the endoscopic image are detected, the processor identifies, among the ranges of the plurality of endoscopic images, the range of the endoscopic image that has the highest degree of coincidence with the specified region as the range of the endoscopic image. The information processing device according to claim 1 .

7. The processor acquires a first point on the endoscopic image designated by a user as the designated region. The information processing device according to claim 1 .

8. The processor identifies a range of the endoscopic image based on a change region formed by pixels that include the pixel designated as the first point and whose pixel values ​​change over time together with neighboring pixels. The information processing device according to claim 7 .

9. The processor further acquires a second point that is specified by a user and is not within the range of the endoscopic image, and configures the changed region so as not to include the pixel specified as the second point. The information processing device according to claim 8 .

10. After specifying the range of the endoscopic image, if an endoscopic image of a different model is acquired that is captured by an endoscope of a different model from the endoscope that captured the endoscopic image used to specify the range of the endoscopic image, the processor re-specifies the range of the endoscopic image based on the newly acquired specified region.

10. The information processing device according to claim 1.

11. The processor stops displaying the endoscopic image and the virtual endoscopic image when an image different from the endoscopic image is acquired.

10. The information processing device according to claim 1.

12. The processor refers to the pixel value of each pixel for each of a predetermined number of endoscopic screen images, each of which includes one of the endoscopic images captured in time series, and identifies the change area based on the amount of change in pixel value for each pixel at the same position in time series.

10. The information processing device according to claim 3, claim 8, or claim 9.

13. The processor determines a group of pixels including at least one pixel included in the specified region and having a change in pixel value equal to or greater than a threshold as the changed region. The information processing device according to claim 12.

14. The processor displays an operation item on the endoscope screen for allowing the user to redo the specification of the range of the endoscope image.

10. The information processing device according to claim 1.

15. The processor further acquires model information of the endoscope capturing the endoscopic image, and if range information representing the range of the endoscopic image for the model information is pre-registered in a storage device, identifies the range of the endoscopic image using the range information corresponding to the model information of the endoscope. The information processing device according to claim 1 .

16. The processor outputs to an external device the correspondence between the model information of the endoscope and the range information stored in the storage device. The information processing device according to claim 15.

17. The processor identifies the closed region as a range of the endoscopic image. The information processing device according to claim 2 .

18. acquiring a designated area, which is an area configured by at least one pixel included in an endoscopic image of a living body, from an endoscopic screen including the endoscopic image of the living body and which is designated by a user; Identifying a range of the endoscopic image based on the acquired designated region; extracting the endoscopic image with the specified range from the endoscopic screen; generating a virtual endoscopic image obtained when a three-dimensional model of the living body photographed in advance is viewed from the same viewpoint as the viewpoint at which the endoscopic image was photographed; and causing a computer to execute a process of displaying the endoscopic image extracted from the endoscopic screen together with the virtual endoscopic image. Information processing program.

Images