DEVICE, METHOD AND NON-VOLUNTARY, COMPUTER-READABLE MEDIUM FOR DETECTING A BODY PART IMAGED BY AN ENDOSCOPE

Abstract

Body part identification device (22) for an endoscope which inserts an insertion section (24) into a body cavity of a patient and takes an image of the body cavity, wherein a distal part of the insertion section (24) includes an imaging unit (30) which is configured to identify a body part in a running image just taken by the imaging unit (30), wherein the body part identification device comprises: an insertion depth detector (91, 96) configured to detect an insertion depth of the insertion section (24) inserted into the body cavity; a moving image capture unit (90), configured to capture the moving image; a legacy image acquisition unit (92) configured to acquire at least one legacy image from a legacy image storage unit (52) based on the insertion depth detected by the insertion depth detector, wherein the legacy image storage unit (52) stores the legacy images acquired during at least one previous endoscopic examination; the old images contain additional information about the body part depicted, and a body part identifier (98) configured to identify the depicted body part in the moving image by comparing the old image captured by the old image acquisition unit (92) with the moving image, and determining that the old image is similar to the moving image, wherein the body part identifier (98) is configured to read the imaged body part information of the old image which has the greatest similarity to the running image and to determine the imaged body part based on the read imaged body part information.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present invention relates to a body part identification device and a non-volatile, computer-readable medium for identifying a body part that has been recorded by an imaging unit at a distal end of an insertion section of an endoscope. 2. Description of the relevant state of the art

[0002] Endoscopes for taking images of a patient's body cavity are widely known. An insertion section of the endoscope, which contains an imaging unit at its distal end, is inserted into the patient's body cavity, and the imaging unit captures the image. The captured image is displayed in real time on a monitor. A physician views the imaged body part on the monitor.

[0003] As is known, the insertion section of the endoscope is equipped with a distal part, a flexible part, and a flexible tube, arranged in that order starting from the distal end. The distal part contains the imaging unit. The flexible part bends to change the direction of the distal part. The flexible tube is flexible. An actuating unit is connected to the proximal end of the insertion section. The actuating unit is equipped with an actuating element (e.g., an angled knob), a locking button, and the like. The actuating element is used to bend the flexible part. The locking button allows the imaging unit to acquire a still image.

[0004] During endoscopic examination, it is essential that the physician is aware of the body part being visualized, for example, whether it is the esophagus or stomach, or whether it is the cardia, pylorus, or gastric notch. However, it is difficult to distinguish the body part (i.e., the gastrointestinal tract) being examined endoscopically by simply looking at the image on the monitor, so the body part may not be correctly identified. To address this problem, a device for identifying the body part being visualized has been developed.For example, JP 4 262 741 B2 and US 2015 / 0 208 947 A1 (corresponding to JP 2014 - 076 174 A) disclose a device for identifying a depicted body part based on the insertion depth of the insertion section, the bending dimension of the insertion section (the flexible part), and the rotation dimension of the insertion section with respect to its axis. In an oral examination where the endoscope is inserted through the patient's mouth, such as an examination of the upper gastrointestinal tract, the insertion depth and rotation dimension are determined based on an offset of the insertion section with respect to, for example, a mouthpiece attached to the patient's mouth. The bending dimension is demonstrated, for example, by the actuation stroke of the actuating element to deflect the flexible part.

[0005] JP 2012-165 838 A discloses a method for identifying the imaged body part. According to this method, three-dimensional image data of the patient's body cavity are generated based on an image acquired by a CT (computed tomography) or MRI (magnetic resonance imaging) scan, as well as the insertion depth, flexion dimension, and rotation dimension of the insertion segment. The three-dimensional image data are compared with a captured image (the moving image) to identify the imaged body part.

[0006] However, the method for identifying the depicted body part based on insertion depth, bending dimension, and rotation dimension according to Japanese patent JP 4 262 741 B2 and US 2015 / 0 208 947 A1 exhibits low accuracy in identifying the depicted body part. Even if the insertion depth is the same, the depicted body part can vary from patient to patient depending on individual characteristics (body shape, age, sex, etc.). The depicted body part identified based on insertion depth may differ from the actual depicted body part.

[0007] The rotation is detected based on the offset of the insertion section relative to the mouthpiece attached to the patient's mouth. In other words, the rotation is determined relative to the direction of the patient's mouth. If the direction of the patient's mouth changes (e.g., if the patient changes the direction of their face) during the examination, the rotation of the insertion section may be detected erroneously, even if the insertion section itself has not rotated. The bending is detected based on the actuation stroke of the actuator. The bending direction of the flexible part varies according to a change in the rotation, which is the reference quantity for determining the bending direction.For example, there may be a shift of up to 90° between the current imaging direction and the imaging direction determined based on the proven bending dimension and the proven rotational magnitude.

[0008] The procedure described in JP 2012-165 838 A generates three-dimensional image data of the patient's body cavity, and the depicted body part is identified by comparing the three-dimensional image data with the live image. This procedure exhibits higher accuracy in identifying the depicted body part than the methods mentioned above. However, the procedure described in JP 2012-165 838 A requires time and effort to generate the three-dimensional image data of the patient's body cavity from the examination data obtained through CT or MRI scans, which are performed in addition to the endoscopic examination.

[0009] From JP 2013-94 562 A, a body part identification device for an endoscope is known, in which the insertion depth of an insertion section inserted into a body cavity is used to determine a previous image, which is then compared with the continuously acquired image. The body part information from the previous image with the highest similarity to the current image is read, and the depicted body part is identified based on the read body part information. REVELATION OF THE INVENTION

[0010] An objective of the present invention is to provide a device, a method and a non-volatile, computer-readable medium for identifying a body part imaged by an endoscope, which are able to easily improve the accuracy of identifying the imaged body part.

[0011] To achieve the above objective and further objectives, a body part identification device according to the present invention includes the features of claim 1.

[0012] Preferred training courses are listed in the dependent requirements.

[0013] One aspect of the present invention creates a non-volatile, computer-readable medium containing instructions stored therein according to claim 9. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and further objectives and advantages of the present invention will become clearer from the following detailed description of the preferred embodiments when read in conjunction with the accompanying drawings, where the same reference numerals denote the same or corresponding parts in all views. The drawings show: Fig. 1. An explanatory view illustrating an endoscope system; Fig. 2 a schematic explanatory diagram illustrating a procedure for identifying a depicted body part; Fig. 3. An explanatory view that illustrates stored old images, each with additional information attached; Fig. 4. An explanatory view illustrating the stored past images classified according to the investigation; Fig. 5 an explanatory view that illustrates an example of a search for old images; Fig. 6. An explanatory view that schematically illustrates the functions of a processor device; Fig. 7 an explanatory view illustrating an endoscopic examination support screen; Fig. 8 a flowchart to illustrate a procedure for identifying the depicted body part; Fig. 9 an explanatory view showing an example in which the old images to be compared with a running image are selected depending on the presence or absence of old images of a patient of interest; Fig. 10 an explanatory view illustrating an example where a correspondence table is used to reduce the number of old images to be compared with the current image; Fig. 11 is an explanatory view that presents an example where an insertion depth, a rotation dimension, and a bending dimension are used to reduce the number of old images to be compared with the moving image; and Fig. 12 is a clear illustration that demonstrates an example where the insertion depth, rotation dimension and bending dimension serve to reduce the number of old images to be compared with the moving image. DESCRIPTION OF PREFERRED VERSIONS (First Version)

[0015] As in Fig. As shown in Figure 1, an endoscope system 10 comprises an endoscope device 12 and an image server 14, which are interconnected via a communication network 16, such as the Internet, or a LAN (Local Area Network). The endoscope device 10 includes an endoscope 18, a mouthpiece 20, a processor unit 22, a display 72, and a keypad 74. The processor unit 22 is an example of a body part identification device.

[0016] The endoscope 18 includes an insertion section 24, which is to be inserted into a body cavity of a patient, and an operating unit 26, which is located at a proximal end of the insertion section 24. The endoscope 18 is connected to the processor unit 22 via a cable 28. The insertion section 24 includes a distal part 32, a flexible part 34, and a flexible tube 36. The distal part 32 incorporates an imaging unit 30. The flexible part 34 bends to change the direction of the distal part 32 (i.e., the imaging direction of the imaging unit 30). The flexible tube 36 is flexible. The imaging unit 30 contains an image sensor, such as a CCD (charge-coupled device), a lens, and a light window. The lens gathers light onto the image sensor. The light window directs illumination light onto a body part to be imaged.The imaging unit 30 acquires an image of the body part located axially along the insertion section 24 and anterior to the distal part 32. The imaging unit 30 transmits image signals generated by the image acquisition to the processor unit 22. Scale markings 40 are located on an outer surface of the insertion section 24. These scale markings 40 are, for example, points arranged longitudinally along the section 24 at a predetermined center-to-center distance (e.g., 1 cm). The scale markings 40 serve to indicate the insertion depth (hereinafter referred to as the insertion depth) of the insertion section 24. The insertion depth indicates the length of the insertion section 24 inserted into the patient's body cavity. The insertion depth is, for example, a measurement determined relative to the position of the mouthpiece 20.In other words, the insertion depth is the depth or position of the distal part 32 within the body cavity, relative to the position of the mouthpiece 20. One unit of insertion depth is, for example, centimeters (cm). The method for measuring the insertion depth is explained below.

[0017] Angled wires, a light guide, and a signal cable run through the insertion section 24. The angled wires are used to bend the flexible part 34. The light guide consists of an optical fiber. The optical fiber transmits the illumination light from a light source to the light window. The signal cable transmits the image signals output by the imaging unit 30 to the processor unit 22 and transmits various signals (e.g., an imaging command) from the processor unit 22 to the imaging unit 30. The control unit 26 contains a release button and an angle knob 42. The release button is pressed to initiate a still image acquisition. The angle knob 42 is turned (operated) to wind or retract the angled wires for bending the flexible part 34. In particular, the angle wire is wound up or pulled depending on the extent of the actuation (one rotation stroke) of the angle knob 42 in order to bend the flexible part 34.In other words: In the endoscope 18, the bending dimension of the flexible part 34 is controlled according to the actuation stroke of the angle knob 42.

[0018] The mouthpiece 20 has a tubular shape and is attached to the patient's mouth. The mouthpiece 20 holds the patient's mouth open. The mouthpiece 20 facilitates the smooth insertion of the insertion section 24 and prevents damage to the insertion section 24 (for example, by the patient biting down). The insertion section 24 is inserted into the patient's body cavity through an opening in the mouthpiece 20.

[0019] The mouthpiece 20 is equipped with small cameras 46 and 48, which serve to measure the insertion depth of the insertion section 24. Each of the cameras 46 and 48 contains an image sensor (for example, a CCD). The cameras 46 and 48 capture images of the outer surface of the insertion section 24. This results in one or more scale markings 40 being depicted in the images captured by the cameras 46 and 48.

[0020] Cameras 46 and 48 are located opposite each other with respect to the opening of the mouthpiece 20, such that the scale marking 40 is located in at least one of the imaging fields of cameras 46 and 48, regardless of the degree (rotation dimension) of the rotation of the insertion section 26, which is inserted through the mouthpiece 20 and rotates about its axis. In this embodiment, camera 46 is located on the side of the patient's lower jaw. Camera 48 is located on the nasal side of the patient. Thus, cameras 46 and 48 are opposite each other.

[0021] Cameras 46 and 48 are connected to and controlled by the processor 22. Cameras 46 and 48 begin imaging in response to the processor 22 being switched on. While the processor 22 is switched on, cameras 46 and 48 record a video at, for example, 30 frames per second. Each recorded frame is output as a camera image to the processor 22. Based on the camera images, the processor 22 counts the scale markings 40 passing through the mouthpiece 20. The scale markings 40 are arranged with a predetermined center-to-center distance, so that the insertion depth is determined by counting the scale markings 40 passing through the mouthpiece 20.

[0022] The processor unit 22 has an imaging control function for inputting the imaging command to the imaging unit 30 of the endoscope 18, enabling the imaging unit 30 to perform the imaging. The processor unit 22 has a display control function for generating a display image based on the image signals output by the imaging unit 30, allowing the display 72 to show the image. The imaging unit 30 can capture moving images and still images. Moving image capture occurs continuously, for example, while the processor unit 22 is switched on. Still image capture occurs in response to the imaging command, which is entered by the trigger button of the control unit 26.

[0023] The processor device 22 has a function (body part identification function) for identifying a body part (hereinafter referred to as the depicted body part) that is in a running image 49 (see Fig. 2) is shown, which is an image being captured by the imaging unit 30 of the endoscope 18. The processor device 22 is an example of a body part identification device.

[0024] As schematically in Fig. As shown in Figure 2, the current image is compared with older images 50 stored in the older image server 14. This comparison is made using pattern matching to identify the older image 50 that is similar to the current image 49. This allows the body part depicted to be identified. The older images 50 are images acquired during at least one previous endoscopic examination. Each older image 50 stores body part information as additional information. This body part information is read from the older image 50 that is similar to the current image 49 and is used to identify the body part depicted in the current image 49. It is assumed that the images of the same body part exhibit a high degree of similarity to each other.Consequently, the body part depicted in the moving image 49 is identified by identifying the old image 50 which bears a high resemblance to the moving image 49.

[0025] Fig. Figure 2 shows that the distal part 32 of the insertion section 24 has reached the stomach S. In this case, the body part depicted in the moving image 49 is the "stomach," and the body part depicted in the old image 50, which bears a strong resemblance to this moving image 49, is also the "stomach." By comparing the moving image 49 with each of the old images 50, the old image 50 that bears a high resemblance to the moving image 49 is determined. The body part depicted in the moving image 49 is identified as the "stomach" based on the body part depicted "stomach" in the old image 50, which was determined to be a highly similar image to the moving image 49.

[0026] The insertion depth is determined to reduce the number of old images 50 that are compared to the current image 49. These multiple old images 50 are stored on the old image server 14. Comparing the current image 49 with each of the old images 50 on the old image server 40 is time-consuming, as it involves searching for the old image 50 that is similar to the current image 49. To shorten the comparison time, the old images 50 are extracted based on the insertion depth. This reduces the number of old images 50 that need to be compared with the current image 49.

[0027] The moving image 49 can be a single frame within the video or a still image captured during a still image recording. The purpose of identifying the body part depicted in the moving image 49 is to inform a physician (conducting the examination) of the depicted body part. Therefore, in the case of the video, it is not necessary to identify the body part in each individual frame. In this case, a process for identifying the body part is performed at time intervals of, for example, one second.

[0028] The old image server 14 contains an old image storage unit 52 in which the multiple old images 50 are stored. The old image server 14 is connected to the processor device 22 via the communication network 16. Responding to a search query entered by the processor device 22 to search for the old image(s) 50, the old image server 14 retrieves the requested old image(s) 50 from the old image storage unit 52 and sends the retrieved old image(s) 50 to the processor device 22.

[0029] As in Fig. As shown in Figure 3, each old image 50 stored in the old image storage unit 52 stores the additional information attached to the image data. This additional information contains various types of information, such as patient identification information (e.g., a patient name, patient ID, or the like) to identify a patient of interest, imaging data and time information about the date and time of image acquisition, insertion depth information representing the insertion depth, and body part information about the body part depicted. The patient of interest here is either the subject of the examination or depicted in the current image 49.

[0030] The old image 50 is an image taken during a previous examination using an endoscope device that has a function for determining insertion depth and a function for identifying a depicted body part. These functions are similar to or identical with those of the endoscope device 12 described for this embodiment. The endoscope device 12 records the measured insertion depth and the identified depicted body part as additional information on the running image 49, which serves to identify the depicted body part. In other words, the insertion depth of the insertion section 24 at the time the running image 49 is taken and the body part identified based on the insertion depth are recorded as additional information on the running image 49 in real time. The patient identification information is also recorded as additional information on the running image 49.

[0031] As described above, the moving image 49 is a single frame of the video or a still image. All or part of the moving images 49 are recorded and stored as old images 50 in the old image storage unit 52. Consequently, the old image storage unit 52 stores the old images 50, each containing the insertion depth information, the depicted body part information, and the patient identification information as additional information.

[0032] The depicted body part information is information that represents a depicted body part. For example, if the subject of investigation is the upper gastrointestinal tract, then the depicted body part information is information about an organ such as the esophagus, stomach, or duodenum. The depicted body part information can contain information about a depicted body part within a fine category, into which a coarse category (in this case, an organ) is subdivided. For example, the depicted body part within the fine category could be a part of the stomach, such as the cardia, the gastric notch, or the pylorus, if the depicted body part within the coarse category is the stomach. The more subdivided or segmented the body part information is, the more precise the identification of the depicted body part becomes.

[0033] As in Fig. As shown in Figure 4, the old image storage unit 51 stores the two or more old images 50 for each examination E (e.g., for each examination E1, one examination E2, etc.). Additional information, such as the body part depicted, is recorded with each of the old images 50. In this example, examinations E1 and E2 are performed on different patients A and B, respectively. Alternatively, examination E can be performed two or three times on the same patient. In this case, the old images 50 acquired in the examinations E performed on the same patient are stored in the old image storage unit 52. In this example, examinations E1 and E2 are examinations of the upper gastrointestinal tract. The old images 50 acquired in each of the examinations E1 and E2 contain the old images 50 of the depicted body part, e.g.,the oral cavity (corresponding to 2 “oral2” in the drawing), the esophagus, or the stomach. The old images 50 used for comparison with the current image 49 are retrieved from the old image storage unit 52.

[0034] As in Fig. As shown in Figure 5, the legacy image storage unit 52 is a database in which the legacy image 50 is searched for based on a search condition, such as patient identification information, information about the insertion depth, and the body part depicted, as explained above. For example, the legacy image 50 is searched for by specifying the search condition (e.g., the patient identification information and the information about the insertion depth). The legacy image storage unit 52 then extracts the legacy image(s) 50 that meet the search condition.

[0035] In a Fig. In example 5, a patient A is designated as the patient identification information. A range of 36 to 46 cm is defined as the insertion depth L. This search extracts old images 50a to 50c. Old image 50a is an old image of the depicted body part "Cardia," taken at a location corresponding to an insertion depth L of 37 cm. Old image 50b is an old image of the depicted body part "Gastric Angle," taken at a location corresponding to an insertion depth L of 42 cm. Old image 50c is an old image of the depicted body part "Pylorus," taken at a location corresponding to an insertion depth of 43 cm.

[0036] As described above, the old image server 14 can search for the old image 50 of the depicted body part according to the specific insertion depth L. Thus, the old image server 14 extracts the old image(s) 50 that were taken at an insertion depth L similar to that at which the current image 49 is being taken.

[0037] The patient identification information serves as the search criterion. This extracts the previous image(s) (50) of the same patient as the current image (49). The previous image (50) is then compared to the current image (49) to identify the body part depicted in the current image (49). If the previous images (50) are from the same patient of interest as the current image (49), the extracted previous image (50) will have a higher similarity to the current image (49) than a previous image (50) extracted from the previous images (50) of a different patient (unlike the patient of interest depicted in the current image (49)). This improves the accuracy in identifying the depicted body part.

[0038] As in Fig. As shown in Figure 6, the processor device 22 includes an image processing unit 88, a control unit 89, a continuous image acquisition unit 90, an insertion depth detector 91, a past image acquisition unit 92, a body part identifier 98, a screen control unit 100 and a storage device 87.

[0039] The image processing unit 88 performs image processing on the image signals output by the imaging unit 30 to generate a viewing image. The image processing unit 88 outputs the viewing image as the moving image 49. The moving image 49 is input into the screen control unit 100 and displayed on the display 72. The screen control unit 100 executes a process (display image generation process) to generate the display image and a process (display control process) to display the display image, and then outputs the display image (the moving image 49) to the display 72. This displays the moving image 49 on the display 72 in real time for viewing by the physician.

[0040] During video recording, the image signals of the individual frames that make up the video are input from the imaging unit 30 to the image processing unit 88 at predetermined time intervals. Based on the image signals of the respective individual frames, the image processing unit 88 generates the moving images 49 and outputs them sequentially to the display control 100. If the trigger button is pressed, the image signals for the still image are input from the imaging unit 30 to the image processing unit 88. The image processing unit 88 generates the moving image 49 (the still image) and outputs it to the display control 100. The moving images 49 (the video and the still image) are stored in the storage device 87, for example, a memory module or a hard disk drive.

[0041] The controller 89 centrally controls each section of the processor device 22. The controller 89 issues an execution start command for a body part identification process to control the continuous image acquisition unit 90, the old image acquisition unit 92, and the body part identifier 98. During video recording, the execution start command is issued at one-second intervals, as described above. In the case of still image recording, the execution start command is issued at the time the still image is captured. Upon receiving the execution start command at each section, the body part identification process, which is a process for identifying the depicted body part, is initiated. The controller 89 sends the patient identification information, which is used as a search condition for finding the old image(s) 50, to the old image acquisition unit 92.For example, the patient identification information is contained in an examination order, which is entered into the processor unit 22 before the examination begins. The controller 89 reads the patient identification information from the examination order and enters the patient identification information into the image acquisition unit 92.

[0042] Upon receiving the execution start command, the continuous image acquisition unit 90 captures the moving image 49, which is output by the image processing unit 88, at the moment the execution start command is entered. Upon receiving the execution start command from the controller 89, the continuous image acquisition unit 90 transmits the captured moving image 49 to the body part identifier 98.

[0043] The camera images captured by cameras 46 and 48 are transmitted in real time to the insertion depth detector 91. This detector sequentially analyzes the input camera images. Based on the result of this analysis, the insertion depth detector 91 determines the insertion depth of the insertion section 24. Specifically, the outer surface of the insertion section 24, with its scale markings 40, is mapped in the camera image. As the insertion section 24 is inserted into the patient's body cavity, it moves relative to the mouthpiece 20. With the movement of the insertion section 24, the scale markings 40 (i.e., in this example, the dots) sequentially pass through the area between cameras 46 and 48.

[0044] The insertion depth detector 91 counts the number of points on the scale markings 40 that are depicted in each of the sequentially input camera images. In this way, the insertion depth detector 91 detects the insertion depth. For example, the insertion depth represents a measurement (in centimeters) of the insertion of the distal end of the insertion section 24 relative to the mouthpiece 20. Alternatively, the scale markings 40 can be numbers, each representing an insertion depth. The insertion depth can be detected by reading (detecting) the number from the camera image.

[0045] Upon receiving the execution start command, the continuous image acquisition unit 90 and the legacy image acquisition unit 92 record the insertion depth output by the insertion depth detector 91 at the time the execution start command is entered. Upon receiving the execution start command, the legacy image acquisition unit 92 retrieves the legacy image(s) 50, which are used to identify the imaged body part, from the legacy image server 14. Specifically, the legacy image acquisition unit 92 sends the search query to the legacy image server 14. The search query specifies the insertion depth recorded by the insertion depth detector 91 at the time the execution start command is entered, and the patient identification information determined by the controller 89, as search conditions.

[0046] The insertion depth defined as the search condition is set to correspond to a predetermined range that includes the insertion depth input by the insertion depth detector 91. In this embodiment, for example, the insertion depth is set as a 10 cm range around the input insertion depth (the mean value). The range containing the insertion depth is defined as the search condition to account for the detection or measurement error of the insertion depth. For example, the insertion depths for examining the stomach are not always exactly the same between previous examinations and the current examination, even if the same body part (in this example, the stomach) is examined in two or more examinations on the same patient. This is because the organ (e.g., the stomach) expands or contracts. For this reason, the predetermined range containing the measured insertion depth is defined as the search condition.

[0047] Upon receiving the search query, the legacy image server 14 retrieves the legacy image(s) 50 that match the search condition specified by the query and sends the identified legacy image(s) 50 to the legacy image acquisition unit 92. The legacy image acquisition unit 92 then extracts the legacy image(s) 50 from the legacy image server 14. In this way, the legacy image(s) 50 are extracted according to the specified penetration depth.

[0048] The insertion depth defined as the search condition is not the insertion depth L itself, which is detected by the insertion depth detector 91, but rather a predetermined range around the insertion depth L (this is the mean value). For example, according to Fig. 5 In one case, the current insertion depth L, which is detected by the insertion depth detector 91, is 41 cm, and a search condition is set for a range of 10 cm around the insertion depth L (= 41 cm) (the mean value), i.e., a range of 36 to 46 cm. In this way, the old images 50 to 50c with insertion depths within this range are determined (see Fig. 5) The old image capture unit 92 gives the old images 50 (the old images 50a to 50c in this example) that were captured in this way to the body part identifier 98.

[0049] Responding to the input of the execution start command from the controller 89, the body part identifier 98 compares the current image 49 input from the moving image acquisition unit 90 with each of the old images 50 input from the old image acquisition unit 92. The body part identifier 98 performs, for example, a pattern matching process known per se, in which a contour, a color, or the like is compared to calculate the similarity between the current image 49 and each of the old images 50. From the old images 50, the body part identifier 98 determines the old image 50 that has the greatest similarity to the current image 49.The body part identifier 98 reads the depicted body part information from the old image 50, which has the greatest similarity to the moving image 49, and determines the depicted body part represented by the depicted body part information of the old image 50 as the depicted body part in the moving image 49.

[0050] In particular, a procedure for calculating similarity by comparing the moving image 49 with the old image 50 is performed as follows: First, a feature size, such as a shape or color, is extracted from each of the moving image 49 and the old image 50, representing a blood vessel structure in the mucosa or a ductal structure in the mucosal surface. Next, the similarity is calculated by comparing the feature size of the moving image 49 with the feature size of the old image 50. These processes are performed on each of the old images 50 to calculate the similarity between the moving image 49 and each old image 50. The old image 50 with the greatest similarity to the moving image 49 is identified among the old images 50.

[0051] After the feature size is extracted from each of the old images 50 and the current image 49, and before the similarity between each of the old images 50 input by the old image acquisition unit 92 and the current image 49 is calculated, a filter can be applied to reduce the number of old images 50 used to calculate the similarity. This is because the mucosa (for example, a waviness or similar feature of the mucosa) appears differently in color and shape from organ to organ in the display (e.g., the esophagus, stomach, or similar). The difference in the depicted mucosa is caused by a difference in feature size. There is a high probability that the images with different feature sizes represent different organs.For this reason, after the feature size of each of the old images 50 and the moving image 49 has been extracted, the old image 50 with the feature size that is significantly different from that of the moving image 49 can be eliminated from the old images 50 compared with the moving image 49.

[0052] The body part identifier 98 provides information about the identified depicted body part to the screen control 100. Together with the information about the identified depicted body part, the body part identifier 98 provides the current image 49 used for the identification of the depicted body part and the old image 50, which was determined to be the image with the greatest similarity to the current image 49, to the screen control 100.

[0053] Based on the information of the identified depicted body part, the moving image 49 used to identify the depicted body part, and the old image 50 determined to be the image most similar to the moving image 49, the screen controller 100 generates an endoscopic examination support screen (hereinafter referred to simply as the examination support screen) 102 (see Fig. 7) and controls a display of the examination and support screen 102 on the display 72. In this way, the screen control 100 acts as an image generator to create the examination support screen 102, and it also acts as a display control to control the display of the examination support screen 102.

[0054] As in Fig. As shown in Figure 7, the examination support screen 102 contains a moving image display area 102a, a body part display area 102b, and an old image display area 102c. The moving image display area 102a shows the moving image 49, the patient identification information of the patient depicted in the moving image 49, and the imaging data and time information for the moving image 49. The body part display area 102b shows the identified body part depicted in the moving image 49, as identified by the body part identifier 98. The body part display area 102b also displays information about the insertion depth recorded at the time of imaging the moving image 49.

[0055] The moving image 49 is successively updated on the examination support screen 102 during video recording. The body part information shown and the previous image 50, which is used to identify the body part shown, are updated at equal intervals as the body part identification process progresses. If a still image is captured—in other words, if the moving image 49 is a still image—the still image is displayed, interrupting the video display.

[0056] In this embodiment, the identified depicted body part is displayed as text information. Furthermore, a circular marker is provided on schematic image 106 of a human body. This also displays the identified depicted body part as image information.

[0057] Of the 50 old images compared to the current image 49 for identifying the depicted body part, the old image 50 with the greatest similarity to the current image 49 is displayed in the old image display area 102c. This area also displays patient identification information and the imaging date and time of the displayed old image 50. This ensures that the image 50 used for identifying the depicted body part is displayed in such a way that verification of the old image 50 is possible.

[0058] The following will be based on the in Fig. The flowchart shown in Figure 8 illustrates the operation of the setup described above. The endoscope device 12 begins recording the video at the start of the examination. In the processor unit 22, the image processing unit 88 generates the moving image 49 based on the image signals from the imaging unit 30. The screen control 100 displays the moving image 49 on the display 72. The still image is captured by pressing the trigger button of the control unit 26. The body part shown in the moving image 49 changes with the insertion depth of the insertion section 24. Thus, the moving image 49 is viewed, showing the body part depicted according to the insertion depth of the insertion section 24.

[0059] Based on the camera images recorded by cameras 46 and 48, the insertion depth detector 91 determines the insertion depth of the insertion section 24 of the endoscope 18.

[0060] The controller 89 issues an execution start command for the body part identification process at predetermined time intervals during video recording. In the case of still image recording, the controller 89 issues the execution start command in response to the recording of the still image. The execution start command is issued to the continuous image acquisition unit 90, the still image acquisition unit 92, and the body part identifier 98.

[0061] Upon receiving the execution start command, the continuous image acquisition unit 90 captures the current image 49 output by the image processing unit 88. Upon receiving the execution start command, the legacy image acquisition unit 92 captures the insertion depth measured by the insertion depth detector 91. The legacy image acquisition unit 92 sends the search request to the legacy image server 14. The search request specifies the patient identification information and the predetermined area containing the measured insertion depth as the search condition. The legacy image server 14 retrieves the legacy image(s) 50 that meet the search condition and sends the legacy image(s) 50 to the legacy image acquisition unit 92.Since the old image(s) 50 are determined based on the insertion depth, the body part depicted in the extracted old image(s) 50 is located in an area close to the insertion depth that corresponds to the body part depicted in the current image 49.

[0062] The extracted old images 50 and the moving image 49 are input into the body part identifier 98. The body part identifier 98 compares the moving image 49 with each of the extracted old images 50 and calculates the similarity between the moving image 49 and each of the old images 50. The body part identifier 98 determines the old image 50 with the greatest similarity to the moving image 49 and reads the depicted body part information of the old image 50 with the highest similarity. The depicted body part information represents the depicted body part in the old image 50 with the highest similarity. The body part identifier 98 determines the depicted body part in the old image 50 to be the depicted body part in the moving image 49.

[0063] The screen control 100 generates the examination support screen 102 (see Fig. 7) Based on the identified body part shown, the previous image 50, which is used to identify the body part shown, the current image 49, and the insertion depth corresponding to the current image 49. The examination support screen 102 is displayed on the screen 72. This informs the physician performing the examination about the body part shown in the current image 49.

[0064] During video recording, the moving image 49 is successively updated on the examination support screen 102. The examination start command is issued at one-second intervals. In response, the body part identification process is executed at the same intervals as the execution start command. The depicted body part information is updated at intervals of approximately one second. In the case of the video, the depicted body part in the moving image 49 changes successively with a change in the insertion depth of the insertion section 24. Since the depicted body part information is updated at intervals of approximately one second, the update of the depicted body part information keeps pace with the change in the displayed moving image 49. The body part identification process can be executed at intervals of more or less than one second.

[0065] The body part information shown and the previous image 50, used to identify the body part shown, are updated at the same time intervals as the body part identification process. If a still image is captured, it is displayed as a moving image 49, interrupting the displayed video. The body part information shown and the previous image 50 corresponding to the still image (i.e., the moving image 49) are displayed while the still image is shown. Alternatively, both the video and the still image can be displayed simultaneously on the examination support screen 102. This allows the video and the still image to be viewed at the same time.

[0066] The body part identifier 98 records both the insertion depth information, which represents the insertion depth, and the depicted body part information, which represents the identified depicted body part, as additional information to the running image 49 on which the body part identification process was performed. Thus, when the running image 49 is saved as an old image 50, the old image 50 stores the insertion depth information and the displayed body part information.

[0067] In the case of the video recording described for this embodiment, the body part identification process is performed at time intervals of, for example, a few seconds, so that the insertion depth information and the body part information depicted are recorded for each individual frame captured within these intervals. If the moving image 49 is the still image, the insertion depth information and the body part information depicted are recorded for each still image. The video and the still image, which are recorded as moving images 49, are stored in whole or in part and serve as old images 50.

[0068] As described above, in the embodiments of the present invention, the old images 50 are acquired based on the insertion depth. The moving image 49 is compared with each of the old images to identify the old image 50 that is similar to the moving image 49. This identifies the body part depicted in the moving image 49. The accuracy in identifying the depicted body part is higher in the embodiments of the present invention than in those of Japanese Patent No. 4262741 and US Patent 2015 / 0208947, which describe devices for identifying the depicted body part based on the insertion depth of the insertion section, the bending dimension of the insertion section (the flexible section), the rotation dimension of the insertion section about its axis, and the like. It is taken into account that the insertion depth, the bending dimension, and the rotation dimension contain a high degree of measurement error.High identification accuracy is ensured by comparison using image analysis, such as pattern matching. In the embodiments of the present invention, the insertion depth serves only to reduce the number of old images 50 that need to be compared with the current image 49. Therefore, the insertion depth has no detrimental effect on identification accuracy, even if it is faulty. Reducing the number of old images 50 by adjusting the insertion depth also shortens the processing time.

[0069] In embodiments of the present invention, the current image 49 is compared with the previous image 50 to improve the accuracy in identifying the depicted body part. Consequently, the accuracy in identifying the depicted body part is improved in a simple manner without the use of extensive apparatus, compared to a prior art in which the accuracy in identifying the depicted body part is improved by using three-dimensional image data obtained by CT or MRI scans, as described, for example, in Japanese patent application publication 2012-165838.

[0070] In this embodiment, the current image 49 and the previous image 50 of the same patient are compared. The similarity between the current image 49 and the previous image 50 of the same patient is higher than the similarity between the current image 49 and the previous image 50 of other patients, even if the body part depicted is the same. Thus, the comparison between the current image 49 and the previous image 50 of the same patient allows for the extraction of the previous image 50 with a high degree of similarity to the current image 49. As a result, the accuracy of body part identification is further improved.

[0071] In this embodiment, according to Fig. 7. Marking 108 is provided on the schematic image, which indicates the identified depicted body part. This provides the identified depicted body part as text information and image information (see Fig. 7) identified. This makes it easy for a user to recognize the depicted body part.

[0072] In this embodiment, according to Fig. 7. The current image 49 and the old image 50, which was identified as showing the same body part as in the current image 49, are displayed in a comparable manner. The current image 49 is visually compared with the old image 50 to verify whether the identified body part is correct. (Second embodiment)

[0073] In the above embodiment, the current image 49 and the old image 50 of the same patient (the same patient of interest) are used for comparison. Alternatively, to identify the depicted body part, the old image 50 of a patient (a different patient) who is different from the patient of interest can be used.

[0074] The comparison between the current image 49 and the old image 50 of different patients results in a similarity that is lower than that between the current image 49 and the old image 50 of the same patient (the same patient of interest), even if the body part depicted is the same. However, if the old image server 14 does not contain the old image 50 of the same patient as the current image 49, the current image 49 cannot be compared with the old image 50 of the same patient. In this case, it is more effective to use the old image 50 of a different patient for comparison with the current image 49 of the patient of interest.

[0075] As in Fig. As shown in Figure 9, it is preferred that the image acquisition unit 92 acquires the image(s) 50 of the patient of interest if the image storage unit 52 contains the image(s) 50 of the patient of interest. It is preferred that the image acquisition unit 92 acquires the image(s) 50 of a different patient if the image storage unit 52 does not contain the image(s) 50 of the patient of interest. (Third embodiment)

[0076] An example of a third embodiment according to Fig. 10 uses a correspondence table 111, which stores the correspondence between the insertion depths and the candidates for the depicted body part. In the first embodiment, the insertion depth itself is used as a search condition (search term) to locate the old image 50. In the third embodiment, the old image acquisition unit 92 first refers to the correspondence table 111 to determine the candidates for the depicted body part based on the proven insertion depth. The old image acquisition unit 92 identifies the candidates for the depicted body part as a search condition in order to retrieve the old images 50 from the old image storage unit 52, corresponding to the candidates for the depicted body part.

[0077] There is a certain correspondence between the insertion depth and the depicted body part. Correspondence table 111 stores the correspondence between the insertion depth range and the candidate for the depicted body part according to the insertion depth range. In a Fig. In the example shown, the insertion depth is denoted by L (cm). A range of "0 < L ≤ 10" of insertion depth corresponds to the depicted body parts "oral (oral cavity)" and "esophagus". A range of "10 < L ≤ 30" of insertion depth corresponds to the depicted body parts "esophagus" and "cardia". A range of "30 < L ≤ 40" corresponds to the depicted body parts "esophagus", "cardia", and "gastric notch". A range of "40 < L ≤ 42" corresponds to the depicted body parts "cardia", "gastric notch", and "pylorus". A range of "42 < L ≤ 45" corresponds to the depicted body parts "gastric notch", "pylorus", and "duodenum".

[0078] For example, if the proven insertion depth L is 36 cm, the old image acquisition unit 92 refers to the correspondence table 111 and identifies "esophagus," "cardia," and "gastric notch" as candidates for the depicted body part. The old image acquisition unit 92 acquires the old image(s) 50 with the additional information that the information about the depicted body part corresponds to "esophagus," "cardia," and "gastric notch." In this example, the depicted body part information serves as the search condition, so the additional information of the old image 50 can only contain the depicted body part information. The insertion depth information is not required.

[0079] Correspondence table 111 allows the use of the old image 50 with the additional information in the form of only the depicted body part information. The additional information of the old image 50 can be entered automatically by the endoscope device 12, as described for the first embodiment, or alternatively, it can be entered manually. When entering the additional information manually, it is difficult to indicate the insertion depth information. With regard to the depicted body part information, for example, the physician can identify the depicted body part when viewing the old image 50, and can enter the depicted body part information of the identified depicted body part as the additional information of the old image 50. The method using correspondence table 111 is effective when the insertion depth information cannot be recorded in the old image 50.

[0080] Note that the correspondence between insertion depths and the candidate for the depicted body part in correspondence table 111 is based, for example, on standard values ​​for adults. Such correspondence can also vary among adults depending on sex, body shape, or similar factors. For this reason, two or more reference tables may be provided and used selectively, according to the sex, body shape, or similar characteristics of the patients. (Fourth embodiment)

[0081] In an example of a fourth embodiment according to the Fig. 11 and Fig. In the fourth embodiment, the old image acquisition unit 92 captures the old image(s) 50 based on the proven insertion depth, the bending dimension, the flexible section 34, and the rotation dimension of the insertion section 24 about its axis. In other words, in the fourth embodiment, the number of old images 50 to be compared with the current image 49 is reduced based on the insertion depth, the bending dimension, and the rotation dimension.

[0082] The imaging direction of the distal section 32 is changed by bending the flexible section 34. The flexible section 34 is bent vertically and horizontally by actuating the angle knob 42. The insertion section 24 can rotate about its axis.

[0083] During endoscopic examination, the patient, as the subject of the examination, is usually in a lateral recumbent position, lying on their side on a table. The insertion direction of the insertion section 24 of the endoscope 18 relative to the patient is often limited or predetermined. For example, if the patient is in a lateral recumbent position with their left side down and their right side up, the insertion section 24 is inserted such that the upper part of the displayed image corresponds to the patient's right side and the lower part of the image corresponds to the patient's left side.Assuming that the essentially straight upper gastrointestinal tract extends along the patient's body axis and that the relative positional relationship between the patient's position and the insertion direction of the insertion section 24 is essentially constant, the direction of the distal section 32 inside the patient's body is determined by measuring the bending dimension and the rotation dimension. The direction of the distal section 32 is the imaging direction.

[0084] Inside the processor device 22 of this embodiment are provided the insertion depth detector 51, a bending dimension detector 112, and a rotation dimension detector 113. The bending dimension detector 112 detects, for example, the actuation stroke of the angle knob 52 from the endoscope 18 in order to determine the bending dimension of the flexible section 34. The angle knob 42 is equipped with an angle knob for aiming the flexible section 34 in the vertical direction and an angle knob for aiming the flexible section 34 in the horizontal direction. The actuation stroke (the rotation dimension) of the angle knob 42 is detected, for example, by a rotary encoder. The determined actuation stroke is output to the bending dimension detector 112. Based on the actuation stroke of the angle knob 42, the bending dimension detector 112 determines the bending dimension of the flexible section 34 in vertical and horizontal directions and transmits the determined bending dimension to the old image acquisition unit 92.The rotation detector 113 analyzes the camera images recorded by cameras 46 and 48 to determine the movement (displacement) of the scale markings 40 around the axis. In this way, the rotation detector 113 detects the rotation dimension.

[0085] The image acquisition unit 92 acquires the insertion depth, bending dimension, and rotation dimension in response to the input of the execution start command for the body part identification process. The bending direction relative to a reference position is determined by measuring the bending dimension in the vertical and horizontal directions. The reference position is a position where the bending dimension is zero in both the vertical and horizontal directions, and the axis of the distal section 32 coincides with the insertion axis of the insertion section 24. Assuming that the patient's orientation and the insertion direction of the insertion section 24 are constant, and that the upper gastrointestinal tract runs substantially along the body axis, the approximate direction (the imaging direction) of the distal section 32 is determined based on the bending direction and the rotation dimension.

[0086] As in the Fig. 11 and Fig. As shown in Figure 12, there are, for example, five imaging directions. The five imaging directions are the axial direction, in which the axis of a tract, such as the gastrointestinal tract, coincides with the axis of the insertion section 24, and the four directions described below. The four directions are: a frontal direction (abdominal side or abdominal direction), a posterior direction (back), a right direction, and a left direction with respect to the patient's body, which is lying on its left side. Based on the bending dimension and the rotation dimension, one of the five directions is determined as the imaging direction for the running image 49. In this example, one of the five imaging directions is recorded as imaged body part information on each of the previous images 50.

[0087] As in Fig.As shown in Figure 12, based on the rotation and bending dimensions, the old image acquisition unit 92 performs an image direction determination process to define the image direction. If the image direction of the current image 49 is defined as the frontal direction (the abdominal side), the reverse direction (back), represented by hatching and located opposite the frontal direction (from the abdominal side), is eliminated from the five directions. The remaining four directions (the tract axis direction, the frontal direction (the abdominal side), the direction to the right, and the direction to the left) are defined as the image directions, which are designated as the search condition.

[0088] In this case, the search condition can only be set to the frontal direction by the image direction determination process. However, this example takes into account that setting the image direction has a significant error. For this reason, only the image direction with the lowest probability is eliminated. This extracts the old image(s) of the current image direction without omitting any, even if the specified image direction differs from the current image direction (for example, in the case where the specified image direction is the frontal direction and the current image direction is to the right).

[0089] The purpose of determining the image direction is to reduce the number of old images 50 that need to be compared with the current image 49. By eliminating only one or a subset of the image directions with low probability, the comparison time can be reduced. If the image direction determination has a high degree of accuracy, the direction (the frontal direction in this example) can be defined as the search condition for the image direction determination process.

[0090] The old image acquisition unit 92 transmits the search query, which contains the specified imaging direction and insertion depth as the search condition, to the old image server 14 in order to acquire the old image(s) 50 that meet the search condition.

[0091] The body part identifier 98 compares the old image(s) 50 with the current image 49 and identifies the old image 50 that is highly similar to the current image 49, in a manner similar to the embodiments described above. The body part identifier 38 reads the depicted body part and the image direction as "Depicted Body Part Information" from the additional information of the identified old image 50. This allows the body part identifier 98 to identify the depicted body part in the current image 49 and the image direction of the depicted body part. The depicted body part and the image direction are displayed as "Depicted Body Part Information" on the examination support screen 102, similar to the embodiments described above.

[0092] In this example, the image direction is displayed as body part information. This informs the physician about the image direction of the current image 49.

[0093] In the embodiments described above, the processor device 22 functions as a body part identification device for identifying the body part depicted in the moving image 49. Alternatively, a computer, such as a server, can be separate from the processor device 22. The server can function as the body part identification device.

[0094] In the present case, the information (e.g., the moving image 49 captured by the imaging unit 30, the image captured by the camera 46 or 48 at the mouthpiece 20, or the actuation stroke of the angle knob 42) necessary to identify the imaged body part is transmitted from the processor device 22 to the server. A CPU of the server functions as a moving image acquisition unit 90, as a past image acquisition unit 92, as an insertion depth detector 91, as a body part identifier 98, and the like. The past image acquisition unit 92 identifies the imaged body part in a manner similar to the embodiments described above, and it generates the examination support screen 102 and sends it to the processor device 22. This configuration also achieves effects similar to the embodiments described above.

[0095] The above embodiments are examples in which the cameras 46 and 48 serve to measure the insertion depth and the rotation of the insertion section 24. Alternatively, the insertion depth and the rotation can be detected, for example, by a detector device such as a trackball in a mouse that serves as an input device for a computer.

[0096] Various changes and modifications to the invention are possible and fall within its scope of protection.

Claims

[1] Body part identification device (22) for an endoscope which inserts an insertion section (24) into a body cavity of a patient and takes an image of the body cavity, wherein a distal part of the insertion section (24) includes an imaging unit (30) which is configured to identify a body part in a moving image just taken by the imaging unit (30), the body part identification device comprising: an insertion depth detector (91, 96) configured to detect an insertion depth of the insertion section (24) inserted into the body cavity; a moving image capture unit (90), configured to capture the moving image; a legacy image acquisition unit (92) configured to acquire at least one legacy image from a legacy image storage unit (52) based on the insertion depth detected by the insertion depth detector, wherein the legacy image storage unit (52) stores the legacy images acquired during at least one previous endoscopic examination; the old images contain additional information about the body part depicted, and a body part identifier (98) configured to identify the depicted body part in the moving image by comparing the old image captured by the old image acquisition unit (92) with the moving image, and determining that the old image is similar to the moving image, wherein the body part identifier (98) is configured to read the imaged body part information of the old image which has the greatest similarity to the running image and to determine the imaged body part based on the read imaged body part information. [2] Body part identification device according to claim 1, further comprising a correspondence table (111) representing a correspondence between the insertion depth and a candidate for the depicted body part, wherein the old image acquisition unit (92) refers to the correspondence table (111) and determines the candidate for the depicted body part based on the insertion depth detected by the penetration depth detector (91), and acquires the old image corresponding to the identified candidate from the old image storage unit (52). [3] Body part identification device according to claim 1 or 2, further comprising: a bending detector (112) configured to detect a bending dimension of the insertion section (24); and a rotation detector (113) configured to detect a rotation dimension of the insertion section (24) with respect to an axis of the insertion section (24), wherein the old image acquisition unit (92) acquires the old image based on the insertion depth, the bending dimension and the rotation dimension. [4] Body part identification device according to one of claims 1 to 3, wherein the old image acquisition unit (92) acquires the old image of a patient of interest from the old image storage unit (52), wherein the patient of interest is depicted in the current image. [5] Body part identification device according to one of claims 1 to 3, wherein the old image acquisition unit (92) acquires the old image of another patient from the old image storage unit (52), wherein the other patient is different from a patient of interest who is depicted in the current image. [6] Body part identification device according to one of claims 1 to 3, wherein the old image acquisition unit (92) acquires the old image of a patient of interest if the old image storage unit (52) contains the old image of the patient of interest, and the old image acquisition unit (92) acquires the old image of another patient different from the patient of interest if the old image storage unit (52) does not contain the old image of the patient of interest, wherein the patient of interest is depicted in the current image. [7] Body part identification device according to any one of claims 1 to 6, further comprising: a screen generator (100) configured to generate an endoscopic examination support screen (102) which displays the running image and the body part depicted in the running image, wherein the body part depicted is identified by the body part identifier (98); and a display controller configured to control a display of the endoscopic examination support screen (102). [8] Body part identification device according to claim 7, wherein the endoscopic examination support screen (102) displays the current image and the old image in a comparable manner, the old image containing the same body part depicted as in the current image, and the body part depicted in the current image being identified by the body part identifier (98). [9] Non-volatile, computer-readable medium containing instructions which, when executed by a computer, cause the computer to perform operations to identify a depicted body part in a moving image being captured by an imaging unit, wherein the imaging unit is contained in a distal part of an insertion section (24) of an endoscope which inserts the insertion section (24) into a body cavity of a patient and captures an image of the body cavity, wherein the operations comprise: Demonstrate the insertion depth of the insertion section inserted into the body cavity; Capturing the moving image; Capturing at least one old image from an old image storage unit (52), based on the insertion depth, wherein the old image storage unit (52) stores the old images that were acquired during at least one previous endoscopic examination, the old images contain additional information about the body part depicted, and Identifying the depicted body part in the moving image by comparing the captured old image with the moving image, and by determining the old image similar to the moving image. Whereby the depicted body part information of the old image that has the greatest similarity to the moving image is read, and the depicted body part is determined based on the read depicted body part information.

Images