Diagnosis assistance device, diagnosis assistance system, display information generation method, display information generation program, and non-transitory recording medium for recording display information generation program

Abstract

This diagnosis assistance device includes a processor and the processor acquires endoscopic image data and identifies, from the endoscopic image data, a user-recognized region recognized in the endoscopic image when the endoscopic image data is displayed on a display.

Description

Diagnostic support device, diagnostic support system, display information generation method, display information generation program, and non-transitory recording medium for recording the display information generation program

[0001] The present invention relates to a diagnostic support device that displays an observation range, a diagnostic support system, a display information generating method, a display information generating program, and a non-transitory recording medium for recording the display information generating program.

[0002] In recent years, endoscopic devices equipped with endoscopes that are inserted into a subject to observe the interior of the subject and perform treatment using treatment tools have become widespread. Computer-aided diagnosis (CAD) and computer-aided detection (CADe) have also been developed, which use AI (artificial intelligence) to provide support information for identifying lesions and other identification results based on images obtained by the endoscopic device. Endoscopic observation using such endoscopes requires thorough observation of the target area.

[0003] Unlike computed tomography (CT) or magnetic resonance imaging (MRI), endoscopes cannot capture images of a wide area, so it is necessary to observe the subject comprehensively. However, because it is difficult to determine which part of the organ is being viewed from the endoscopic image, doctors and other medical professionals often fail to notice areas that have been missed. Therefore, a technology is known that displays a 2D or 3D organ model during endoscopic observation and visualizes observed and unobserved areas on the organ model.

[0004] For example, Japanese Patent Application Laid-Open Publication No. 2018-50890 (hereinafter referred to as Patent Document 1) discloses a technology that provides a landmark image detection unit that detects anatomical landmark images corresponding to the organ being imaged, and indicates observed and unobserved areas by using the landmark image as a base point and allocating multiple endoscopic images to corresponding parts of a virtual model.

[0005] Japanese Patent Application Publication No. 2018-50890

[0006] However, in Patent Document 1, the captured area, i.e., the area displayed on the display, is processed as having been observed, but the user does not necessarily recognize every corner of the image displayed on the display. The resolution and visual acuity of the human eye are affected by various factors. For example, when the image moves relatively fast, the resolution and visual acuity of the eye decrease, and the user's recognition ability also decreases. As a result, there is a possibility that the observation area recognized by the user and the observed area displayed on the display in Patent Document 1 will differ. The present invention aims to provide a diagnostic support device, a diagnostic support system, a display information generation method, a display information generation program, and a non-transitory recording medium for recording the display information generation program, which are capable of presenting an observed area that sufficiently reduces the discrepancy with the user's recognition by enabling the user to set a recognized area that takes into account the user's recognition ability.

[0007] A diagnostic support device according to one aspect of the present invention has a processor, which acquires endoscopic image data and identifies, from the endoscopic image data, a user-recognized area in an endoscopic image when the endoscopic image data is displayed on a display.

[0008] A diagnostic support device according to another aspect of the present invention includes a first data acquisition unit that acquires endoscopic image data, and a field of view identification unit that identifies, from the endoscopic image data, a recognized area in an endoscopic image when the endoscopic image data is displayed on a display.

[0009] A diagnostic support system according to one aspect of the present invention includes an endoscope that transmits endoscopic image data, a diagnostic support device that includes a processor that directly or indirectly acquires the endoscopic image data and identifies, from the endoscopic image data, a user-recognized area in the endoscopic image when the endoscopic image data is displayed on a display, and the display that receives a signal output from the diagnostic support device and displays the organ model.

[0010] A display information generation method according to one aspect of the present invention is a display information generation method for displaying an observed area on an observation object model, in which a first data acquisition unit acquires image data, and an effective field of view identification unit identifies a recognized area in an image when the image data is displayed on a display from the endoscopic image data.

[0011] A display information generation program according to one aspect of the present invention is a display information generation program for displaying an observed area on an observation object model, and causes a computer to execute a procedure of causing a first data acquisition unit to acquire image data and an effective field of view identification unit to identify, from the endoscopic image data, a recognized area in the image when the image data is displayed on a display.

[0012] A non-transitory recording medium for recording a display information generation program according to one aspect of the present invention is a non-transitory recording medium for recording a display information generation program for displaying an observed area on an observation object model, and records the display information generation program for causing a computer to acquire image data and execute a procedure for identifying, from the endoscopic image data, a recognized area in the image when the image data is displayed on a display.

[0013] According to the present invention, by making it possible to set a recognized area taking into account the user's recognition ability, it is possible to present an observed area that is sufficiently close to the user's recognition.

[0014] FIG. 1 is a block diagram showing a diagnosis support device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram for explaining an example of a recognized region. FIG. 3 is an explanatory diagram for explaining landmarks when the hollow organ is a bladder. FIG. 4 is an explanatory diagram for explaining landmarks when the hollow organ is a bladder. FIG. 5 is an explanatory diagram for explaining a display example. FIG. 6 is an explanatory diagram for explaining a display example. FIG. 7 is an explanatory diagram for explaining a display example. FIG. 8 is an explanatory diagram for explaining a display example. FIG. 9 is a flowchart for explaining the operation of the embodiment. FIG. 10 is an explanatory diagram for explaining the operation of the embodiment. FIG. 11 is an explanatory diagram for explaining the operation of the embodiment. FIG. 12 is a block diagram showing a modified example.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] 1 is a block diagram showing a diagnosis support device according to one embodiment of the present invention. This embodiment determines an area of ​​an image obtained by an endoscope that has been observed (an observed area), and makes it possible to present to a user areas that have been observed by the endoscope and areas that have not yet been observed by the endoscope (an unobserved area). In this case, this embodiment employs an observation area that the user recognizes as having observed (hereinafter referred to as a recognized area) as the observed area presented to the user. For example, the moving speed of the tip of the endoscope is detected, and the size of the recognized area that has been observed is changed in accordance with the detected moving speed.

[0017] It is generally known that dynamic visual acuity when tracking a moving subject is poorer than static visual acuity when viewing a still image, and that visual acuity declines as the speed of movement increases. It is also known that visual acuity is poorer in the periphery of the visual field than in the center.

[0018] Therefore, in this embodiment, the recognized area is set according to the imaging conditions of the endoscopic image data, such as the moving speed of the tip of the endoscope. For example, for a still image or an image moving at a relatively slow speed, it is assumed that a person has the visual acuity to observe a wide range of the image, and all or most of the imaging range is set as the recognized area. Then, as the moving speed of the image increases, only the central range of the imaging range is set as the recognized area.

[0019] In the following description, the "photographed area" refers to the lens-based effective field of view, which is the entire range of the image obtained by the endoscopic imaging device. The "recognized area" described above is the area of ​​the photographed area where a person is considered capable of recognizing and processing image information, and is also considered to be the biologically effective field of view range. In this embodiment, the photographed area, i.e., the entire area of ​​the endoscopic image displayed on the display, other than the recognized area, i.e., the area that is likely not recognized by the user, is not presented to the user as having been observed.

[0020] FIG. 2 is an explanatory diagram for explaining an example of a recognized region.

[0021] The left column of Figure 2 shows an example in which a recognized area RA1 is defined as a rectangular area within a predetermined range from the center of the endoscopic image, relative to the entire captured area PA1 of the image (endoscopic image) captured by the endoscope. The center column of Figure 2 also shows recognized areas RA2 and RA3, relative to the captured area PA1, when the moving speed of the endoscope tip (moving speed of the image sensor) is equal to or greater than a predetermined value. The recognized area RA3 represents a recognized area when the moving speed of the endoscope tip is faster than the recognized area RA2. In this example, the processor 30 (described later) can set a first predetermined range when the moving speed is a first moving speed and a second predetermined range when the moving speed is a second moving speed faster than the first moving speed, and sets a recognized area in which the second predetermined range is smaller than the first predetermined range. In other words, the center column of Figure 2 shows that the faster the moving speed of the imaging device, the narrower the recognized area becomes.

[0022] 2 shows an example in which the recognized area RA4 is set to a circular shape instead of a square, and the recognized area RA4 is set to a predetermined range from the position (coordinates) of the lesion LA detected by the CADe, rather than the center of the image. For example, the recognized area RA4 is set around a target area such as a frame LF indicating the lesion.

[0023] If multiple lesions are detected by CADe, a predetermined range from the center points of the detected lesions may be set as the recognized region. If the number of lesions detected by CADe is equal to or greater than a predetermined number, the entire endoscopic image may be set as the recognized region. The detection targets by CADe or the like may include not only lesions but also blood vessels, nerves, bleeding points, etc.

[0024] By displaying such recognized regions, for example, within an organ model, it is possible to present to the user regions that have already been observed and regions that have not yet been observed. For example, regions other than the recognized regions within the photographed region may be considered unobserved regions or photographed regions. In the former case, if regions that have not been imaged by the endoscope are also considered unobserved regions, the organ model is presented as being divided into recognized regions that the user has observed and recognized, and unobserved regions that are not yet observed (recognized). In the latter case, the organ model is divided into three regions: recognized regions that the user has observed and recognized, unobserved regions that have not yet been photographed by the endoscope, and photographed regions that have been photographed but not yet recognized.

[0025] The diagnosis support device 1 shown in FIG. 1 supports the diagnosis of a subject observed by an endoscope 10, which is an observation device. In this embodiment, an example of observing (or inspecting) the inside of a specific hollow organ (e.g., bladder, stomach, intestines, etc.) inside a patient as the inside of the subject will be described. The endoscope 10 may have the function of performing observation in two observation modes: normal light observation and special light observation. The diagnosis support device 1 receives endoscopic images (image data) directly or indirectly from the endoscope 10. A surgeon as a user of the diagnosis support device 1 performs an endoscopic examination of the inside of a specific hollow organ (also simply referred to as a hollow organ or organ), such as a bladder, of a patient lying in a supine position on a bed, for example. Note that the following description will be given of an example of observing a bladder with an endoscope; however, various hollow organs of the human body, not limited to the bladder, can be observed with an endoscope.

[0026] The endoscope 10 has an operation section (not shown) and a flexible insertion section. The endoscope 10 is, for example, an endoscope used for bladder examinations. Illumination light is supplied to the endoscope 10 from a light source device 20. This illumination light is emitted from an illumination window 14 at the tip of the insertion section. This illumination light illuminates the interior of a predetermined tubular organ within the subject into which the tip of the insertion section is inserted.

[0027] The distal end of the insertion section is provided with an objective optical system 12 and an imaging element 11. The imaging element 11 has an imaging surface disposed at the imaging position of the objective optical system 12, and photoelectrically converts the optical image of the subject formed on the imaging surface and outputs it as an imaging signal. In this way, the imaging element 11 images the inner wall of the bladder or the like illuminated by the illumination light from the light source device 20. The objective optical system 12 and the imaging element 11 constitute an imaging section (or imaging device) that images the inside of a tubular organ and outputs the captured image. The imaging signal obtained by the imaging element 11 is converted into a digital signal by an A / D converter 13 and then supplied to the processor 30.

[0028] Each component within processor 30 may be configured by a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), NPU (Neural Processing Unit), etc., and may operate according to a program stored in memory (not shown) to control each part, or may realize some or all of its functions using hardware electronic circuits.

[0029] The processor 30 acquires endoscopic image data from the endoscope 10 and identifies the user's recognized area in the endoscopic image when the endoscopic image data is displayed on the display from the received endoscopic image data.

[0030] The processor 30 is provided with an image processing unit 31. The image processing unit 31, which serves as a first data acquisition unit, receives an imaging signal input from the endoscope 10 and performs predetermined signal processing on the received imaging signal to generate an endoscopic image (endoscopic image data). The image processing unit 31 outputs the generated endoscopic image to the image synthesis unit 40. The image synthesis unit 40, which serves as a display information generation unit, synthesizes a virtual model image (described later) with an endoscopic image and outputs a signal of the synthesized image via an output terminal 41. The synthesized image output from the output terminal 41 is supplied to, for example, a display device. In this way, the synthesized image of the endoscopic image and the virtual model image is displayed on the display screen of the display device.

[0031] The endoscopic image from the image processing unit 31 is supplied to the lesion detection AI 32, the landmark detection AI 33, the movement speed and direction calculation unit 34, the image cropping unit 35, and the virtual model pasting unit 36. The lesion detection AI 32 is an AI having an inference model for detecting lesions. For example, such an inference model can be constructed by annotating endoscopic images acquired by an endoscope to indicate lesions as training data, and then providing a large amount of training data to the inference model for deep learning. The lesion detection AI 32 infers the presence or absence of a lesion in the input endoscopic image and, if present, the location of the lesion. As an inference result, it outputs, for example, information indicating the type of lesion, and displays a frame image indicating the location (coordinates) of the lesion on the endoscopic image.

[0032] The organ information acquisition unit 38, which serves as a second data acquisition unit, stores organ model information for various organs of the human body as observation target model information. The user selection unit 37 is configured by an input device such as a keyboard (not shown) or an interface device that accepts external input, and acquires information indicating which observation target region the input image represents based on a user operation, and outputs the information to the organ information acquisition unit 38. The organ information acquisition unit 38 outputs model information (observation target model information) for the observation target region designated by the user to the virtual model pasting unit 36. The virtual model pasting unit 36 ​​is configured to generate a virtual model image (observation target model) based on the model information corresponding to the organ information designated by the user.

[0033] While FIG. 1 illustrates an example in which the observation target region is specified by the user selection unit 37, the observation target region may be automatically determined and model information selected by utilizing the region determination AI. Such region determination AI can be constructed by deep learning using a large number of endoscopic images annotated with region names. The region determination AI infers the region name from the input image. By providing this inference result to the organ information acquisition unit 38, model information corresponding to the input image can be obtained.

[0034] The processor 30 is provided with a landmark detection AI 33 to obtain a criterion for determining which position of a hollow organ an endoscopic image indicates. The landmark detection AI 33 is an AI having an inference model that detects landmarks in an endoscopic image. Landmarks are anatomically important positions or features that a physician uses when inserting and advancing an endoscope during an examination. For example, in the case of the bladder, the left and right ureteral orifices can be used as landmarks. Accurately detecting landmarks can detect which position within a hollow organ an image captures. Such an inference model can be constructed by annotating endoscopic images acquired by an endoscope with landmarks to use as training data, and then providing a large amount of training data to the inference model for deep learning. The landmark detection AI 33 infers the presence or absence of landmarks in the input endoscopic image and, if present, the location of the landmark, and can output information indicating, for example, the landmark location as an inference result. The landmark detection AI 33 outputs the detection results to the virtual model pasting unit 36.

[0035] Figures 3 and 4 are explanatory diagrams for explaining landmarks when the hollow organ is a bladder. Figure 3 shows the anatomical structure of the bladder, and Figure 4 shows a virtual model image of the bladder. Figure 3 shows the inside of the bladder by cutting the bladder along a plane including the apex and the urethral opening. Figure 4 shows an example in which a circular image, which is commonly used in bladder schemas, is used.

[0036] 3, bladder B is a sac-shaped organ with folds 51 lined along the inner wall, and these folds 51 are surrounded by detrusor muscle 52. The bottom of bladder B has a funnel-shaped trigone 55 formed by a pair of ureteral orifices 53 and a urethral sphincter 54. Urine flows from the kidneys (not shown) through the ureteral orifices 53 into bladder B and is stored there. The urine stored in bladder B is excreted through the urethra (urethral sphincter 54).

[0037] 4, the virtual model image M represents the anterior wall Wf and posterior wall Wb of the bladder B as circles. That is, in the virtual model image M, the ventral range from the apex T to the urethral opening A is represented by a circle at the bottom of the page as the anterior wall Wf, and the remaining dorsal range is represented by a circle at the top of the page as the posterior wall Wb. Circles indicating the left ureteral opening L and the right ureteral opening R are shown on the posterior wall Wb. Note that a doctor may make a diagnosis by determining that the area outside a vertical line passing through the left ureteral opening L and the right ureteral opening R is the left wall or the right wall.

[0038] When detecting landmarks, the surgeon inserts the insertion portion of the endoscope 10 into the bladder B and observes an area including a pair of ureteral orifices 53. An endoscopic image (landmark image) obtained through this observation is provided to the landmark detection AI 33, which detects the pair of ureteral orifices 53 as landmarks. The detection results of the landmark detection AI 33 are provided to the virtual model pasting unit 36. Note that the landmark detection AI 33 may detect not only the left ureteral orifice L and the right ureteral orifice R, but also the urethral orifice A and the apex T as landmarks, or may detect other parts as landmarks. Furthermore, the landmark detection AI 33 is capable of detecting landmarks not only for the bladder B but also for various hollow organs of the human body.

[0039] The virtual model pasting unit 36, which serves as the first and second position specifying units, determines the observation range of the endoscope 10 based on the detection results of the landmark detection AI 33. The virtual model pasting unit 36 ​​compares the landmark (ureteral orifice 53) in the landmark image in the bladder B with the left ureteral orifice L and the right ureteral orifice R in the virtual model image M to determine the area (photographed area) on the virtual model image M of the endoscopic image (landmark image) obtained by capturing an image of the landmark.

[0040] The movement speed and direction calculation unit 34 calculates the movement speed and direction of the image sensor 11. For example, the movement speed and direction calculation unit 34 receives the endoscopic image from the image processing unit 31, detects the movement of an object between frames using optical flow, and calculates the velocity vector of the image sensor 11. The movement speed and direction calculation unit 34 provides the calculated velocity vector of the image sensor 11 to the image cropping unit 35 and the virtual model pasting unit 36. The virtual model pasting unit 36 ​​moves the region of the endoscopic image on the virtual model image M, i.e., the photographed region, based on the velocity vector. That is, the virtual model pasting unit 36 ​​subsequently identifies the imaging position of the endoscopic image on the subject based on the velocity vector and using the landmark image as a reference, and links this imaging position to a region of the organ model (virtual model image).

[0041] The image cropping unit 35, which serves as a visual field identification unit, crops the input image based on the velocity vector of the image sensor 11 and sets a partial region of the input image as a recognized region. The image cropping unit 35 outputs information about the recognized region to the virtual model pasting unit 36. For example, when the moving speed of the image sensor 11 is lower than a predetermined threshold, the image cropping unit 35 may output the entire range of the input image as the recognized region, and when the moving speed of the image sensor 11 is higher than the predetermined threshold, the image cropping unit 35 may be configured to increase the amount of cropping of the input image as the moving speed increases, thereby outputting a recognized region with a smaller area. The image cropping unit 35 may also determine an image region of a predetermined shape (e.g., rectangular or circular) centered on the center of the input image as the recognized region, or an image region centered on the position of a lesion or the like detected by the lesion detection AI 32 as the recognized region.

[0042] Furthermore, when the movement speed of the image sensor 11 is greater than a predetermined threshold, the image trimming unit 35 may divide the speed into multiple stages and change the size of the recognized area for each stage. Note that, when the movement speed of the image sensor 11 is faster than a predetermined threshold, the image trimming unit 35 may determine that no recognized area exists. For example, when the movement speed is divided into two stages, a first movement speed and a second movement speed, as in the example in the center column of FIG. 2 , and the recognized area is set by dividing the movement speed into two stages, a first movement speed and a second movement speed, the image trimming unit 35 may set the second predetermined range to zero, i.e., determine that no recognized area exists, when the second movement speed is equal to or greater than a predetermined upper recognition speed limit.

[0043] Furthermore, for example, when the image trimming unit 35 acquires information (upper limit detectable speed information) on the moving speed (detectable speed) of the imaging element 11 at which lesion detection by the lesion detection AI 32 is possible, it may set the upper limit recognition speed to the detectable speed, and if the moving speed of the imaging element 11 is equal to or greater than the detectable speed, it may set the second predetermined range to zero and deem that no recognized area exists.

[0044] The virtual model pasting unit 36 ​​composites an image indicating the photographed area onto the virtual model image. Furthermore, the virtual model pasting unit 36 ​​composites an image indicating the recognized area onto the virtual model image. The virtual model pasting unit 36 ​​also composites images indicating the unobserved area and the photographed area onto the virtual model image, and updates the images of the unobserved area, the photographed area, and the recognized area on the virtual model image each time a recognized area is output from the image trimming unit 35. Hereinafter, the images indicating the unobserved area, the photographed area, and the recognized area will be referred to as area images, and the process of composite the area image onto the virtual model image will be referred to as pasting or mapping. In this way, the virtual model pasting unit 36 ​​outputs a virtual model image in which the unobserved area, the photographed area, and the recognized area are mapped in a distinguishable manner to the image composition unit 40. That is, the processor 30 generates and outputs a signal that distinguishably displays the recognized area from the non-recognized area on the organ model (virtual model image).

[0045] It has been explained that the virtual model pasting unit 36 ​​pastes three types of area images, namely, unobserved areas, photographed areas, and recognized areas, onto the virtual model image, but it may also be configured to paste two types of area images, namely, unobserved areas and recognized areas, onto the virtual model image.

[0046] (Light source conditions) In addition, this embodiment adopts the observation mode as the imaging condition for endoscopic image data, and may be configured to change the settings of the recognized area when, for example, the normal light observation mode and the special light observation mode are different.

[0047] The light source information output unit 39 is configured to obtain light source information about the illumination light emitted by the light source device 20 and output it to the image cropping unit 35. The image cropping unit 35 may determine the size of the recognized region based on the light source information from the light source information output unit 39. For example, the light source device 20 may be capable of emitting white light for normal light observation and narrow band illumination (NBI) light for special light observation, which has lower brightness than white light. The human eye achieves higher resolution with brighter light. Therefore, the image cropping unit 35 may set the size of the recognized region relatively wide when the light source information indicates that white light is being emitted, and may set the size of the recognized region relatively narrow when the light source information indicates that NBI light is being emitted.

[0048] In some cases, the light source device 20 can emit both white light for normal light observation and TXI (structural color enhancement) light for structural color enhancement, which has a higher contrast than white light. The human eye can obtain higher resolution as the contrast increases. Therefore, the image cropping unit 35 may set the size of the recognized region to be relatively narrow when the light source information indicates that white light is being emitted, and may set the size of the recognized region to be relatively wide when the light source information indicates that TXI light is being emitted.

[0049] That is, in this embodiment, the processor 30 can change the size of the predetermined range in FIG. 2 according to the observation light of the endoscope, and sets a recognized area in which the predetermined range is large for observation light that can provide higher resolution, and the predetermined range is relatively small for observation light that can provide only relatively low resolution.

[0050] (Transparency condition) In addition, this embodiment may adopt the transparency condition of the fluid that fills the field of view of the endoscope as the imaging condition for endoscopic image data, and may change the setting of the recognized area if the transparency is different.

[0051] The image trimming unit 35 may be configured to perform image analysis processing. The image trimming unit 35 determines the transparency of the fluid filling the field of view through image analysis processing of the input endoscopic image. For example, the recognizable field of view is narrowed when there is residual urine in the bladder or when smoke (due to surgery) is present. Therefore, the image trimming unit 35 divides the transparency of the field of view into two levels, setting a fifth predetermined range for the first transparency level and a sixth predetermined range narrower than the fifth predetermined range for the second transparency level, which is lower than the first transparency level, to determine the recognized field. In other words, the lower the transparency of the field of view, the narrower the recognized field is made by the image trimming unit 35.

[0052] (Condition of Accumulated Time) In the present embodiment, the setting of the recognized area may be changed in consideration of the accumulated time taken to capture an image of the same area as an imaging condition for the endoscopic image data.

[0053] The image trimming unit 35 may have a timer function for measuring time. When the cumulative imaging time for the same region reaches or exceeds a predetermined time, the image trimming unit 35 may determine that the user's recognition process has covered the entire image and output an image in which the entire imaged region is recognized without trimming. Endoscopic images (image data) are sequentially input to the image trimming unit 35. Even if a region of a given image data has not been determined to be a recognized region, if the same region has been imaged for a predetermined time or longer in sequentially input image data, it can be determined that the region is highly likely to have undergone recognition processing, even if it is at the edge of the image. Therefore, a signal is output from the processor 30 so that the portion of the image whose cumulative imaging time exceeds the predetermined time is identified as a recognized region.

[0054] (Display Examples) Figures 5 to 8 are explanatory diagrams for explaining display examples. During endoscopic examination, the display 50 shown in Figure 6 is connected to the output terminal 41 of the processor 30. Figure 5 shows how the insertion section 61 of the endoscope 10 is inserted into the body and a luminal organ at a target site is observed. An imaging element 11 is disposed at the tip of the insertion section 61, and an endoscopic image is acquired by the imaging element 11 to observe a wall surface 62 of the luminal organ. The dashed line and square frame in Figure 5 indicate the field of view 63 of the imaging element 11 at the tip of the insertion section 61. Figure 6 shows an example of a screen display in this case. On the display screen 50a of the display 50, an endoscopic image P63 corresponding to the field of view 63 is displayed on the left side, and a virtual model image M1 is displayed on the right side.

[0055] In this embodiment, the image trimming unit 35 sets a recognized area RA63 in the center of the endoscopic image P63. In FIG. 6, the recognized area RA63 is indicated by a dashed rectangular frame, but this rectangular frame may or may not be displayed on the display screen 50a. The area on the virtual model image M1 corresponding to the recognized area RA63 is displayed as a recognized area MRA63 in the virtual model image M1. The dashed arrow in FIG. 6 indicates the correspondence between the recognized area RA63 and the recognized area MRA63, but is not actually displayed on the display screen 50a. The hatched area outside the recognized area MRA63 on the virtual model image M1 indicates a photographed area MP63 corresponding to the endoscopic image P63. This photographed area MP63 does not have to be displayed.

[0056] That is, in the past, only the photographed region MP63 corresponding to the entire endoscopic image P63 was displayed on the virtual model image M1. However, in this embodiment, for example, a central region of the endoscopic image, which is narrower than the photographed region MP63 depending on the moving speed of the image sensor 11, is displayed as the recognized region MRA63. In this manner, a region of the captured endoscopic image that the physician believes can be reliably recognized is displayed as the recognized region on the virtual model image M1. Thereafter, similar operations are repeated, and the recognized region is determined using endoscopic images (image data) acquired later in time series. As a result, even if a region was not determined as a recognized region in the previous image capture, if it is determined as a recognized region in a subsequent endoscopic image, a signal indicating the recognized region on the virtual model image M1 is output from the processor 30. In this way, the region reliably recognized by the physician can be displayed on the virtual model image M1, enabling comprehensive observation of the entire range of the target tubular organ.

[0057] 7 and 8 are explanatory diagrams showing the relationship between the moving speed of the image sensor 11 and the setting of the recognized area.

[0058] 7 and 8 show an example in which a circular endoscopic image P11 is displayed in the endoscopic image display area 50aa on the display screen 50a. These figures show that the faster the movement speed of the image sensor 11, the smaller the recognized area set by the image cropping unit 35. The example in FIG. 7 shows a rectangular recognized area RA11a that is considered to have been observed (recognized) during slow movement, and a rectangular recognized area RA11b that is considered to have been observed (recognized) during fast movement. The example in FIG. 8 shows a circular recognized area RA11c that is considered to have been observed (recognized) during slow movement, and a circular recognized area RA11d that is considered to have been observed (recognized) during fast movement.

[0059] In Figures 7 and 8, an example has been described in which the moving speed is divided into two stages and a recognized area corresponding to each of the low speed and high speed stages is set, but as mentioned above, the moving speed may be divided into three or more stages and the recognized area may have three or more stages of size.

[0060] Next, the operation of the embodiment configured as above will be described with reference to Figures 9 to 12. Figure 9 is a flowchart for explaining the operation of the embodiment. Figures 10 to 12 are explanatory diagrams for explaining the operation of the embodiment.

[0061] The insertion section of the endoscope 10 is inserted into the bladder to observe the bladder. In this case, the user selection section 37 supplies an operation signal indicating that the observation target area is the bladder to the organ information acquisition section 38. The organ information acquisition section 38 outputs model information related to the bladder to the virtual model pasting section 36. An imaging signal of the inside of the bladder is acquired by the imaging element 11 provided at the tip of the insertion section of the endoscope 10, converted into a digital signal by the A / D converter 13, and then supplied to the processor 30. The image processing section 31 of the processor 30 generates an endoscopic image by predetermined signal processing and outputs it to the image synthesis section 40. In addition, the virtual model pasting section 36 generates a virtual model image of the bladder and outputs it to the image synthesis section 40. The image synthesis section 40 synthesizes the endoscopic image from the image processing section 31 and the virtual model image from the virtual model pasting section 36, and outputs the synthesized image to the output terminal 41.

[0062] A display 50 shown in Fig. 10 is connected to the output terminal 41 of the processor 30. An endoscopic image display area 50aa is provided on the left side of a display screen 50a of the display 50. An endoscopic image P21 obtained by inserting an endoscope into the bladder is displayed in the endoscopic image display area 50aa. In addition, a virtual model image M1 is displayed on the right side of the display screen 50a.

[0063] The endoscopic image from the image processing unit 31 is also supplied to the landmark detection AI 33 and the virtual model pasting unit 36. The landmark detection AI 33 determines whether a landmark is detected in S1 of FIG. 9 . The landmark detection AI 33 repeats the detection process until a landmark is detected. When the landmark detection AI 33 detects a landmark (YES in S1), it outputs the detection result to the virtual model pasting unit 36. The virtual model pasting unit 36 ​​compares the ureteral opening 53 detected as a landmark in the landmark image with the left ureteral opening L and right ureteral opening R in the virtual model image M1 to determine the area (photographed area) on the virtual model image M1 of the endoscopic image (landmark image) obtained by capturing the landmark. The virtual model pasting unit 36 ​​pastes an area image indicating the area of ​​the landmark image onto the virtual model image M1 and outputs the image to the image synthesis unit 40. In this way, the area image of the landmark image is mapped onto the virtual model image M1 on the display 50 (S2).

[0064] The endoscopic image from the image processing unit 31 is also supplied to a movement speed and direction calculation unit 34, which calculates a velocity vector indicating the movement speed and direction of the image sensor 11 and outputs the velocity vector to an image trimming unit 35 and a virtual model pasting unit 36 ​​(S3). Based on the velocity vector, the virtual model pasting unit 36 ​​detects the relative position of the previous area image, i.e., the area image mapped based on the landmark position (S4). Based on the velocity vector, the image trimming unit 35 trims the endoscopic image (S5) to obtain an area image of the recognized area within the photographed area. The virtual model pasting unit 36 ​​pastes the trimmed area image at the relative position calculated from the velocity vector (S6). When pasting a new area image at a position where an area image has already been mapped, the virtual model pasting unit 36 ​​prioritizes pasting in the order of the recognized area, the photographed area, and the unobserved area. As a result, when the entire area of ​​the observation target site is observed while being recognized, the entire area of ​​the virtual model image M1 becomes a recognized area, and it can be confirmed that the entire area of ​​the observation target organ has been observed.

[0065] The virtual model pasting unit 36 ​​outputs the virtual model image M1 onto which the area image is mapped to the image composition unit 40. The image composition unit 40 combines the endoscopic image from the image processing unit 31 with the virtual model image M1 onto which the area image from the virtual model pasting unit 36 ​​is mapped, and outputs the combined image. In this way, a composite image of the endoscopic image and the virtual model image M1 is displayed on the display screen 50a of the display 50. Thereafter, the operations of S3 to S6 are repeated, and the number of area images mapped onto the virtual model image M1 increases.

[0066] 10 shows an example of mapping area images divided into two types, recognized areas RA21a to RA21c and an unobserved area UB, onto a virtual model image M1. Note that the recognized areas RA21a to RA21c are indicated by rectangular frames to show the size of the area image, and the plain areas other than the recognized areas RA21a to RA21c represent the unobserved area UB.

[0067] 11 shows an enlarged view of the recognized areas RA21a to RA21c in FIG. 10. As shown in FIG. 11, the largest recognized area RA21a was obtained when the image sensor 11 moved slowly, indicating that, for example, the image cropping unit 35 did not perform any trimming and a recognized area of ​​the same size as the captured area was set. The medium-sized recognized area RA21b was obtained when the image sensor 11 moved at a medium speed and indicated that the image cropping unit 35 trimmed the recognized area to a size smaller than the captured area. The smallest recognized area RA21c was obtained when the image sensor 11 moved at a high speed and indicated that the image cropping unit 35 trimmed the recognized area to a size smaller than the recognized area RA21b.

[0068] 10 and 11, the dotted patterns of the recognized areas RA21a and RA21b indicate that these areas were obtained by the second observation, and the diagonal lines of the recognized area RA21a indicate that these areas were obtained by the third observation. In this way, when the same area has been observed multiple times, the virtual model pasting unit 36 ​​may be configured to paste, onto the virtual model image, an area image that allows the number of observations to be distinguished.

[0069] The user can easily understand which areas of the bladder have been observed with the endoscope by checking the recognized areas plotted on the virtual model image M1 in Fig. 10. The recognized areas plotted on the virtual model image M1 are not simply areas that have been photographed, but areas that are likely to have been recognized by a doctor, and by checking the display of the recognized areas, the doctor can reliably confirm the entire area of ​​the tubular organ being observed.

[0070] 12 shows an example in which, under the same observation conditions as in FIG. 11, area images divided into three types—recognized areas RA21a-RA21c, photographed areas PA21b and PA21c, and an unobserved area UB—are mapped onto a virtual model image M1. Note that the sizes of the recognized areas RA21a-RA21c are indicated by solid rectangular frames. The filled-in area surrounding the recognized area RA21b is the photographed area PA21b before trimming of the recognized area RA21b, and the filled-in area surrounding the recognized area RA21c is the photographed area PA21c before trimming of the recognized area RA21c. Furthermore, the solid areas not surrounded by rectangular frames other than the recognized areas RA21a-RA21c and the photographed areas PA21b and PA21c indicate the unobserved area UB.

[0071] In this manner, in this embodiment, a portion of the central area of ​​the endoscopic image is set as a recognized area according to the movement speed of the image sensor at the tip of the endoscope, and this recognized area is presented to the user as the observed area. This allows the user to observe the recognized area in a recognizable manner even when the movement speed is fast, and makes it possible to reliably observe the entire area of ​​the object to be observed even if the discrepancy between the range presented to the user as the observed area and the range actually recognized is small.

[0072] (Modification) Fig. 13 is a block diagram showing a modification. In Fig. 13, the same components as those in Fig. 1 are given the same reference numerals and the description thereof will be omitted.

[0073] 1 differs from FIG. 1 in that an endoscope 10A equipped with a motion sensor 15 is employed. The motion sensor 15 detects the motion of the image sensor 11 at the tip of the insertion portion and outputs the detection result to a movement speed and direction calculation unit 34. The movement speed and direction calculation unit 34 calculates the speed and direction of movement of the image sensor 11 based on the detection result of the motion sensor 15.

[0074] Other configurations and effects are the same as those in FIG.

[0075] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0076] Furthermore, among the technologies described herein, many of the controls and functions, mainly those illustrated in flowcharts, can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored, in whole or in part, as a computer program product on a portable medium such as a flexible disk, CD-ROM, or nonvolatile memory, or on a storage medium such as a hard disk or volatile memory, and can be distributed or provided at the time of product shipment, via a portable medium, or via a communication line. A user can easily realize the diagnostic support device, diagnostic support system, display information generation method, display information generation program, and non-transitory recording medium for recording the display information generation program of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. A diagnostic support device having a processor, wherein the processor acquires endoscopic image data and identifies, from the endoscopic image data, a user-recognized area in an endoscopic image when the endoscopic image data is displayed on a display.

2. The diagnostic support device of claim 1, wherein the processor acquires organ model information, identifies the imaging position of the endoscopic image on the subject, links the imaging position to a part of the organ model, and generates and outputs a signal that displays the recognized area and areas other than the recognized area on the organ model in a distinguishable manner.

3. The diagnostic support device according to claim 1, wherein the processor defines the recognized area as a predetermined range from the center point of the image.

4. The diagnostic support device according to claim 1, wherein information on the coordinates of the detection target in the image is acquired, and the recognized area is set to a predetermined range from the coordinates.

5. The diagnostic support device of claim 1, wherein the processor acquires information on the coordinates of the detection target in the image, sets the recognized area based on the imaging conditions of the endoscopic image data, and sets the recognized area to a predetermined range from the center point of the image or the coordinates.

6. The diagnostic support device of claim 5, wherein the imaging condition is the moving speed of the tip of the endoscope, and the processor is capable of setting, as the predetermined range, a first predetermined range when the moving speed is a first moving speed, and a second predetermined range when the moving speed is a second moving speed that is faster than the first moving speed, and sets the second predetermined range to be smaller than the first predetermined range.

7. The diagnostic support device according to claim 6, wherein the processor sets the second predetermined range to zero when the second moving speed is equal to or greater than a predetermined speed.

8. The diagnosis support device according to claim 7, wherein information on an upper limit of a detectable speed by CADe for the endoscopic image is acquired, and the predetermined speed is set as the detectable speed.

9. The diagnostic support device of claim 5, wherein the imaging condition is endoscopic observation light, and the processor is capable of setting, as the predetermined range, a third predetermined range when the endoscopic observation light is a first light, and a fourth predetermined range when the endoscopic observation light is a second light, and sets the fourth predetermined range to be larger than the third predetermined range.

10. The diagnostic support device of claim 5, wherein the imaging condition is the transparency of the fluid that fills the field of view of the endoscope, and the processor is capable of setting as the predetermined range a fifth predetermined range when the field of view has a first transparency and a sixth predetermined range when the field of view has a second transparency that is lower than the first transparency, and sets the sixth predetermined range to be smaller than the fifth predetermined range.

11. The diagnostic support device of claim 2, wherein the processor distinguishes between areas other than the recognized area and an unobserved area, defines the area outside the recognized area in the endoscopic image as the photographed area, and defines the area in which the endoscopic image has not been acquired as the unobserved area.

12. The diagnostic support device of claim 2, wherein the image data includes first image data and second image data acquired chronologically later than the first image data, and the processor outputs the signal so that even if a region is not determined to be the recognized region in the first image data, if it is determined to be the recognized region in the second image data, the region is identified as the recognized region on the organ model.

13. The diagnostic support device of claim 12, wherein the image data includes first image data and at least one second image data acquired chronologically later than the first image data, and the processor outputs the signal so that even if a region in the first image data is not determined to be the recognized region, a region that has been imaged as the second image data for a predetermined period of time is identified as the recognized region on the organ model.

14. A diagnostic support device comprising: a first data acquisition unit that acquires endoscopic image data; and a field of view identification unit that identifies, from the endoscopic image data, a recognized area in an endoscopic image when the endoscopic image data is displayed on a display.

15. A diagnostic support device as described in claim 14, comprising: a first position identification unit that identifies the imaging position of the endoscopic image in the subject; a second position identification unit that links the imaging position to a part of an organ model; and a display information generation unit that generates and outputs a signal that displays the recognized area and areas other than the recognized area on the organ model in a distinguishable manner.

16. A diagnostic support system comprising: an endoscope that transmits endoscopic image data; a diagnostic support device that includes a processor that directly or indirectly acquires the endoscopic image data and identifies, from the endoscopic image data, a user-recognized area in the endoscopic image when the endoscopic image data is displayed on a display; and a display that receives a signal output from the diagnostic support device and displays an organ model.

17. A display information generation method for displaying an observed area on an observation object model, comprising: a first data acquisition unit acquires image data; and an effective field of view identification unit identifies, from the endoscopic image data, a recognized area in the image when the image data is displayed on a display.

18. A display information generation method as described in claim 17, wherein a second data acquisition unit acquires observed object model information, a first position identification unit identifies the imaging position of the image on the observed object, a second position identification unit links the imaging position to a part of the observed object model, and a display information generation unit generates and outputs a signal that displays the recognized area and areas other than the recognized area on the observed object model in a distinguishable manner.

19. A display information generation program for displaying an observed area on an observation object model, the display information generation program causing a computer to execute the following steps: cause a first data acquisition unit to acquire image data; and cause an effective field of view identification unit to identify, from the endoscopic image data, a recognized area in the image when the image data is displayed on a display.

20. The display information generation program according to claim 19, further causing the computer to execute the steps of: causing a second data acquisition unit to acquire observed object model information; causing a first position identification unit to identify the imaging position of the image on the observed object; causing a second position identification unit to link the imaging position to a part of the observed object model; and causing a display information generation unit to generate and output a signal that displays the recognized area and areas other than the recognized area on the observed object model in a distinguishable manner.

21. A non-transitory recording medium having recorded thereon a display information generation program for displaying an observed area on an observation object model, the display information generation program causing a computer to acquire image data and execute a procedure for identifying, from the endoscopic image data, a recognized area in the image when the image data is displayed on a display.

22. A non-transitory recording medium as described in claim 21, which records a display information generation program that causes the computer to execute the steps of: acquiring observed object model information; identifying the imaging position of the image on the observed object; linking the imaging position to a part of the observed object model; and generating and outputting a signal that displays the recognized area and areas other than the recognized area on the observed object model in a distinguishable manner.

Images