Endoscope device, image processing device, in-hospital system, image processing method, and image processing program

Abstract

This endoscope device comprises: an endoscope including an imaging device at an insertion section distal end; and a processor. The processor: detects a specific site on the basis of an image feature of an endoscopic image acquired by the imaging device during endoscope insertion in endoscopic examination; controls the imaging device so as to achieve a focused state of the specific site; determines, on the basis of the endoscopic image, a directional relationship between an up / down / left / right direction of the imaging device and an anatomical position of a human body into which the endoscope is inserted; and on the basis of the directional relationship, continuously rotates the endoscopic images sequentially acquired during the insertion with regard to the specific site, thereby performing directional unification processing to unify the orientation of the sequentially acquired endoscopic images with reference to the anatomical position.

Description

Endoscope device, image processing device, in-hospital system, image processing method, and image processing program

[0001] The present invention relates to an endoscope apparatus, an image processing apparatus, an in-hospital system, an image processing method, and an image processing program that utilize images obtained when an endoscope is inserted for observation or treatment.

[0002] An endoscope is a device that is inserted into the body and enables observation of diseased areas that cannot be seen from the outside. An endoscope has a long, thin, flexible insertion section that is inserted into, for example, a patient's body cavity. The endoscope captures an image of the area to be observed using an imaging element provided at the tip of the insertion section. The image captured by the endoscope (endoscopic image) is supplied to a video processor. The endoscopic image processed by the video processor is displayed on a monitor display screen.

[0003] Inserting such endoscopes into human body lumens is relatively difficult. For this reason, when performing observations or procedures using an endoscope, for example, in the case of examining the large intestine, the insertion portion is first brought to the site to be observed, and then the insertion portion is removed. When inserting an endoscope into the upper or lower digestive tract, the endoscope must be inserted according to the shape of the digestive tract, which can be narrow depending on the location.

[0004] At this time, the doctor advances and retracts the insertion section and twists it to insert it into the area to be observed. Therefore, particularly when inserting the endoscope, not only does the doctor have little time to observe, but the images obtained are often not suitable for observation.

[0005] In this way, endoscopic images are captured by changing the orientation of the tip, which is located away from the doctor's hands. Therefore, Japanese Patent Application Laid-Open Publication No. 2010-220794 (hereinafter referred to as Patent Document 1) discloses a technology that automatically aligns the orientation of the subject area shown in multiple endoscopic images taken at different times of the same subject area.

[0006] JP 2010-220794 A

[0007] In particular, images obtained when a doctor is concentrating on inserting an endoscope into a narrow and winding lumen are not suitable for observation because they are not images obtained for observation purposes (images of the endoscope tip moving or images during position change).Furthermore, even if the technology of Patent Document 1 is applied, it is difficult to use images obtained during endoscope insertion for observation or treatment.

[0008] The present invention aims to provide an endoscopic device, an image processing device, an in-hospital system, an image processing method, and an image processing program that can make it possible to easily use images obtained when a physician is concentrating on the insertion operation, such as during endoscope insertion (i.e., intermediate progress images that were not originally intended to be used for observation or treatment), for the observation and treatment of various parts. Of course, the present invention also aims to make it possible to easily use images obtained in preparation for not only insertion into a narrow lumen but also removal from a lumen or other operational situations in which the distal end of the endoscope is changed, for the observation and treatment of various parts.

[0009] An endoscopic device according to one aspect of the present invention comprises an endoscope including an imaging device at the tip of an insertion portion, and a processor, wherein the processor detects a specific site based on image features of an endoscopic image acquired by the imaging device when the endoscope is inserted during an endoscopic examination, controls the imaging device to bring the specific site into focus, determines a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image, and performs a direction unification process to unify the orientation of the endoscopic images acquired sequentially during the insertion of the specific site based on the anatomical position by continuously rotating the endoscopic images acquired sequentially during the insertion.

[0010] An image processing device according to one aspect of the present invention includes a processor, which detects a specific site based on image features of an endoscopic image acquired by an imaging device provided at the tip of the endoscope insertion portion when the endoscope is inserted during an endoscopic examination, determines a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image, and performs direction unification processing to unify the orientations of the endoscopic images acquired sequentially during the insertion of the specific site based on the anatomical position.

[0011] An image processing method according to one aspect of the present invention receives data of temporally consecutive endoscopic images from an endoscope that acquires images using an imaging device provided at the tip of an insertion portion, detects a specific area based on image features within the endoscopic images, rotates and corrects each of the temporally consecutive endoscopic images based on the image of the specific area within the endoscopic images, and associates the rotated and corrected images with text regarding symptoms of the specific area as an examination result.

[0012] Another aspect of the in-hospital system of the present invention includes a receiving unit that receives data of temporally consecutive endoscopic images from an endoscope that acquires images using an imaging device provided at the tip of an insertion portion, and a processor, wherein the processor detects a specific area based on image features within the endoscopic images, rotates and corrects each of the temporally consecutive endoscopic images based on the image of the specific area within the endoscopic images, and associates the rotated and corrected images with text regarding symptoms of the specific area as the examination result.

[0013] An image processing method according to one aspect of the present invention detects a specific region based on image features of an endoscopic image acquired by an imaging device provided at the tip of an endoscope insertion portion when an endoscope is inserted during an endoscopic examination, determines a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image, and performs direction unification processing to unify the orientations of the endoscopic images acquired sequentially during the insertion of the specific region based on the anatomical position.

[0014] An image processing program according to one aspect of the present invention causes a computer to execute a procedure for detecting a specific site based on image features of an endoscopic image acquired by an imaging device provided at the tip of an endoscope insertion portion when the endoscope is inserted during an endoscopic examination, determining a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image, and performing direction unification processing to unify the orientations of the endoscopic images acquired sequentially during the insertion of the specific site based on the anatomical position.

[0015] According to the present invention, it is possible to easily use images of the progress of position changes (images of the endoscope tip moving process) obtained during processes such as preparation for observation or treatment by changing the position of the endoscope tip for observation or treatment of each part.

[0016] 1 is a configuration diagram showing an endoscopic device according to a first embodiment of the present invention. FIG. 1 is an explanatory diagram showing the oral cavity, nasal cavity, larynx, and pharynx of a human body. FIG. 2 is an explanatory diagram illustrating the anatomical orthotopic position. FIG. 3 is a flowchart illustrating the operation of the first embodiment. FIG. 3 is an explanatory diagram illustrating the insertion of an endoscope. FIG. 4 is an explanatory diagram illustrating an example of the relationship between an endoscopic image and the anatomical orthotopic position. FIG. 4 is an explanatory diagram illustrating a method for determining the relationship between a specific region and the anatomical orthotopic position. FIG. 5 is an explanatory diagram illustrating an endoscopic image to be acquired. FIG. 6 is a diagram illustrating a display example. FIG. 7 is an explanatory diagram illustrating an example of a specific region. FIG. 8 is an explanatory diagram illustrating an example of a specific region. FIG. 9 is a flowchart illustrating the operation flow adopted in the second embodiment. FIG. 10 is an explanatory diagram illustrating image synthesis in the second embodiment. FIG. 11 is an explanatory diagram illustrating image synthesis in the second embodiment. Flowchart illustrating a modified example of the second embodiment. Block diagram showing a third embodiment. Flowchart illustrating the operation of the third embodiment. Flowchart illustrating the operation of the third embodiment. Flowchart illustrating the operation of the third embodiment.

[0017] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0018] First Embodiment FIG. 1 is a configuration diagram showing an endoscopic device according to a first embodiment of the present invention. This embodiment uses endoscopic images, such as endoscope tip movement process images or position change progress images, acquired by an endoscope during processes such as changing the tip position of the endoscope (insertion section) to prepare for observation or treatment. The relationship between the display orientation of a specific body part included in the endoscopic image on the screen and the anatomical orthogonal position of the specific body part is calculated, and the up / down / left / right directions of the endoscopic image displayed on the screen are aligned with the anatomical orthogonal position. Furthermore, this embodiment performs processing to improve the image quality of the specific body part in the endoscopic image, thereby making it easier to use the endoscopic image acquired during the endoscope position change process for observation, treatment, etc. Note that endoscopic images include not only endoscope tip movement process images and position change progress images, but also observation images for observing and examining an affected area, examination images, and treatment images during a treatment. Differences between these images appear as differences such as multiple consecutive images of the same object (observation images, examination images, images during treatment), sequential images of different objects in a time series, or images changing in the direction of the lumen hole (images of the endoscope tip moving, images during position change).In other words, it is possible to determine the difference by examining the characteristics of consecutive images in a time series (object pattern, changes in shading).

[0019] In FIG. 1 , an endoscopic image from an endoscope 20 is input to an image processing device 10. The endoscope 20 includes an image sensor 22. The image sensor 22 is provided, for example, at the tip of an insertion section 21. The endoscope 20 has an optical system (not shown) that guides an optical image of a subject to the imaging surface of the image sensor 22. This optical system and the image sensor 22 constitute an imaging device. The image sensor 22 is composed of a CCD, CMOS sensor, or the like, and photoelectrically converts the optical image of the subject from the optical system to obtain a captured image (image signal) of the subject. The optical system may also include lenses and apertures (not shown) for zooming and focusing, and may also include a zoom (magnification change) mechanism, focus and aperture mechanisms (not shown) that drive these lenses. Image information of the captured image (endoscopic image) captured by the image sensor 22 is supplied to the image processing device 10.

[0020] The image processing device 10 includes a control unit 11, an imaging control unit 12, an image acquisition unit 13, an image processing unit 14, a specific part determination unit 15, an anatomical orthogonal comparison unit 16, a display control unit 17, and a recording control unit 18. The control unit 11 and each component of the image processing device 10 may be configured by a processor using a CPU (Central Processing Unit) or FPGA (Field Programmable Gate Array), may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of the functions by a hardware electronic circuit.

[0021] The control unit 11 performs overall control of the entire image processing device 10. The imaging control unit 12 generates an imaging control signal for controlling imaging by the endoscope 20 and provides it to the endoscope 20. The imaging control unit 12 controls the drive of the imaging element 22 and also controls the zoom and focusing of an optical system (not shown). In other words, the imaging control unit 12 can perform autofocus control of the imaging element 22.

[0022] The image acquisition unit 13 acquires an endoscopic image, which is image information from the endoscope 20, during an endoscopic examination. The image processing unit 14 performs predetermined image signal processing on the endoscopic image acquired by the image acquisition unit 13. For example, the image acquisition unit 13 performs predetermined signal processing, such as color adjustment processing, matrix conversion processing, noise removal processing, and various other signal processing, on the endoscopic image acquired by the imaging element 22. The display control unit 17 provides the image (endoscopic image) obtained by the image processing of the image processing unit 14 to the monitor 30 for display. The monitor 30 is, for example, a display device having a display screen such as an LCD (liquid crystal display device).

[0023] As described above, in an examination or treatment using the endoscope 20, a physician operates the endoscope 20 to insert the insertion section 21 of the endoscope 20 into the human body. A bending section (not shown) is provided at the tip of the insertion section 21, and the physician operates a bending knob (not shown) or the like provided on the endoscope 20 to bend the bending section or move the insertion section 21 back and forth, thereby inserting the tip of the insertion section 21 to the site to be observed. During an endoscopic examination, it is relatively difficult to insert the insertion section 21 in accordance with the state of a lumen, which may be narrow or curved, and the images obtained during the process of changing the position of the tip section from time to time are distorted up, down, left, and right, making it difficult to observe such a passing site at the same position in the image while performing focusing or other image quality improvement controls.

[0024] For this reason, images taken during endoscope insertion until a specific examination site is reached (or during movement of the tip of the endoscope) may not be used for diagnosis, etc. However, during endoscope insertion, images of various sites other than the observation site are also acquired as the endoscope passes through the site on its way to the observation site. For example, in an endoscopic examination of the stomach, images of the larynx and pharynx are also acquired during insertion. In other words, if endoscopic images acquired during insertion could be used for observation or treatment, it would be possible to obtain valuable information that would be useful, for example, during the next examination.

[0025] However, when the cylindrical endoscope tip (insertion section) is moved along a lumen, the image tends to rotate clockwise or counterclockwise without maintaining its upright position. Therefore, the present application attempts to align the upright position of each image based on the anatomical characteristics of the observation site, thereby obtaining images similar to those published in academic papers, even when the endoscope is moved. Images with such upright positioning make it easier to detect characteristic findings and symptoms. Furthermore, clear images can be captured through focusing, exposure control, and image processing to obtain such images. By capturing, processing, and recording images similar to reference images in academic papers, the images are, in a sense, normalized, making it possible to compare the size of tumors, lesions, and follicles, thereby enabling appropriate assessment, examination, and diagnosis.

[0026] Therefore, this embodiment is provided with a specific part determination unit 15 and an anatomical orthogonal comparison unit 16. The specific part determination unit 15 determines a specific part of the human body from an endoscopic image acquired by the endoscope 20. A specific part is a part from which specific anatomical features can be identified, and includes not only cases in which the anatomical features of a specific part can be directly identified from an image of the specific part, but also cases in which the anatomical features of the specific part can be inferred from information on other specific parts from which anatomical features have been identified.

[0027] As described below, a specific region can be determined by referring to a database stored in the recording unit based on the region's characteristic color or shape, or the pattern of blood vessels or other visible blood vessels on the surface. Another method involves using an external receiver to receive the transmissions of a magnetic or other transmitter at the endoscope's tip and determine the location of the endoscope's tip on the body. Representative images of the specific region (e.g., images with adjusted image quality, as in papers) and data indicating the image characteristics of each region may be recorded in the recording unit 40, and the specific region determination unit 15 may determine the specific region by referring to the information recorded in the recording unit 40. The information recorded in the recording unit 40 may also be used to determine the image representation (e.g., focus, exposure, color, angle of view, and top / bottom orientation) when capturing each region. Essentially, the determination, capture, and recording of images that are similar to the representative image may be controlled.

[0028] FIG. 2 is an explanatory diagram showing the oral cavity, nasal cavity, larynx, and pharynx of the human body.

[0029] Upper gastrointestinal endoscopy includes oral endoscopy, in which the endoscope is inserted through the mouth as indicated by arrow A, and transnasal endoscopy, in which the endoscope is inserted through the nose as indicated by arrow B. In these examinations, the insertion section 21 of the endoscope is inserted through the mouth or nose and reaches the upper gastrointestinal tract (esophagus, stomach, and duodenum) for examination. That is, in both examinations, the insertion section 21 passes through various regions before reaching the esophagus. The unit of region may be an organ with a specific function, or a portion thereof, such as its entrance or its side, may also be used as the unit. Large organs such as the large intestine and stomach have several regions to be examined, but here, a region refers to a divided portion of the organ.

[0030] For example, in oral endoscopy, when the insertion section 21 passes through the oral cavity, it passes through the lips, teeth, and the hard palate, which blocks the passage to the nostrils when eating and drinking to prevent food from entering the nose, then the soft palate, which is the soft part at the back of the hard palate, and passes through the epiglottis at the entrance to the trachea on its way from the oropharynx to the esophagus. The epiglottis has the function of acting as a lid to prevent food from going into the trachea during swallowing and guiding it into the esophagus.

[0031] Furthermore, the insertion section 21 passes through the oropharynx and hypopharynx and reaches the esophagus. The esophagus and trachea branch off at the oropharynx, and the larynx connects the oropharynx and trachea. The larynx and hypopharynx are adjacent to each other. The larynx is an organ commonly known as the "Adam's apple," and separates the trachea from the pharynx. Air taken in through the nose and mouth is directed into the trachea, and food and drink are directed into the esophagus.

[0032] The larynx is not only a passageway for air, but is also an important part of the trachea that vibrates the vocal cords to produce sound, and since this area is adjacent to the esophagus when the endoscope is inserted, it is possible to capture images of part of it. Even with a transnasal endoscope, the insertion section 21 passes through the nasal cavity, nasopharynx, oropharynx, and hypopharynx, so images of these areas can be captured when the upper endoscope is inserted and removed.

[0033] While pharyngeal observations have been used to examine the presence or absence of swelling and follicles, images of these areas can also be used to diagnose infections and other conditions. Furthermore, in patients with chronic COPD, even one lung abnormality can cause intrathoracic pressure to compress the esophagus, reducing its circularity. The shape of the esophagus can be determined from images of the esophageal tract taken before and after the endoscope is inserted into the esophagus. Furthermore, as shown in Figure 11 , respiratory organs such as the trachea and lungs are located near the route along which the upper endoscope passes. If there is an abnormality in the respiratory organs, such as the lungs, the respiratory organs and areas before and after them may be affected by pressure due to intrathoracic pressure, which can cause deformation of the shape of the esophageal lumen or the bronchial entrance. This means that information about other organs can be obtained during the gastrointestinal tract examination. In addition to detecting lesions such as tumors and polyps, respiratory abnormalities can also be diagnosed using parameters such as abnormal color changes and abnormal esophageal shape. If any abnormality is detected, it is possible to recommend a bronchial side examination, either during or after this examination. The irregularity determination may be made by comparing the detected lumen shape with images (healthy and unhealthy, etc.) stored in a database in the recording unit 40 and determining the degree of similarity with unhealthy images, or by calculating the circularity from the lumen outline and making a numerical determination. The irregularity is determined to be an abnormality if the difference in shape exceeds a preset threshold value by comparing with normal conditions (images and numerical values) estimated from general anatomical data and previous medical examination data.

[0034] The specific site determination unit 15 determines the specific site by image analysis and inference processing of the endoscopic image. For example, the specific site determination unit 15 can identify a specific site such as the nasal cavity or stomach by determining the shape of the lumen from image features. Furthermore, for example, the specific site determination unit 15 can determine a specific site such as the large intestine by determining changes in image features (image history) that indicate that the insertion unit 21 is progressing while bending, and can also determine whether it is the descending colon or the transverse colon.

[0035] When the specific region determination unit 15 determines that the endoscopic image is an image having characteristics specific to a specific region (an image of a specific region classified by anatomical characteristics), it outputs the determination result to the imaging control unit 12, the image processing unit 14, and the anatomical orthogonal comparison unit 16. The imaging control unit 12 outputs an imaging control signal to the endoscope 20 to improve the image quality of the image of the specific region. For example, the imaging control unit 12 performs tracking AF (autofocus) to perform autofocus while tracking the specific region. The imaging control unit 12 may also perform exposure adjustment, illumination light control, etc. The image processing unit 14 also performs image processing to improve the image quality of the image of the specific region and improve visibility.

[0036] Furthermore, even for endoscopic images of the same body part, the orientation of the image (the up / down / left / right orientation of the image displayed on the display screen) can vary depending on the orientation of the lumen and the positioning during insertion. Therefore, anatomical positioning is used to clarify the positional relationship between each internal body part and the endoscopic image. Medical image data can also be organized based on anatomical features. That is, medical image data can be created with orientation uniformity processed taking into account the anatomical position, or medical image data of internal body parts organized based on anatomical features. This improves the visibility of medical image data and makes it easier to classify, organize, and compare. Internal body parts organized based on anatomical features refer to each organ and its component parts, such as those named using "anatomical terms" compiled by academic societies. It is assumed that the names of such internal body parts can be determined based on the anatomical position and the images obtained by capturing the internal body parts using an image table or inference model for referencing the internal body parts names. Information about the type of examination performed can also be used as a reference when determining the internal body parts.

[0037] FIG. 3 is an explanatory diagram for explaining the anatomical position.

[0038] Anatomical position (anatomical orthotopic position) refers to a body position in which both feet face forward, both arms are externally rotated, palms face forward, and thumbs point outward. In anatomical orthotopic position, anterior is ventral and posterior is dorsal. Furthermore, upper is head side and lower is foot side. Furthermore, the center of the body is medial or inner, and the direction away from the center is lateral or outer. Furthermore, the base of the arms and legs is proximal, and the fingertips side is distal.

[0039] For example, two orthogonal directions on a plane perpendicular to the direction of travel of the insertion portion 21 are defined as the up-down direction and the left-right direction. The image sensor 22 is fixed to the tip of the insertion portion 21, and for example, the up-down, left-right directions of the imaging surface of the image sensor 22 (hereinafter also referred to as the up-down, left-right directions of the imaging device) and the up-down, left-right directions of the insertion portion 21 are assumed to coincide. When an endoscopic image acquired by the endoscope 20 is displayed on the display screen of the monitor 30 without image correction, an image whose up-down and left-right directions coincide with the up-down and left-right directions of the insertion portion 21 is displayed as is in the up-down, left-right directions on the display screen of the monitor 30. Note that the vertical scanning direction of the monitor 30 is defined as the up-down direction of the display screen, and the horizontal scanning direction is defined as the left-right direction of the display screen.

[0040] For simplicity, the following description will refer to the orientation of an image displayed on a display screen. That is, the orientation of an endoscopic image corresponds to the up, down, left, and right directions of the imaging device.

[0041] However, the up / down and left / right directions of the insertion portion 21 are directions unrelated to the directions based on the anatomical position, and moreover, as the insertion portion 21 is twisted during insertion, the relationship between the up / down and left / right directions of the insertion portion 21 and the directions based on the anatomical position changes. As a result, the orientation of the endoscopic image displayed on the display screen of the monitor 30 is unrelated to the anatomical position, and the relationship between them changes over time.

[0042] In this embodiment, the anatomical orthogonal position comparison unit 16 determines the up / down / left / right orientation of the image of the specific part based on the anatomical orthogonal position. That is, the anatomical orthogonal position comparison unit 16 determines the directional relationship between the orientation of the endoscopic image (up / down / left / right directions of the imaging device) and the anatomical orthogonal position. The anatomical orthogonal position comparison unit 16 detects the positional relationship between the specific part in the image and the anatomical orthogonal position. The detection result of the anatomical orthogonal position comparison unit 16 can also be considered to be torsion information of the insertion unit 21 based on the anatomical orthogonal position.

[0043] The anatomical orthogonal position comparison unit 16 determines the anatomical orthogonal position of an image portion of a specific region by image analysis of the endoscopic image. For example, for a specific region such as the nasal cavity or stomach, the anatomical orthogonal position comparison unit 16 can determine the relationship between each image portion of the specific region and the anatomical orthogonal position based on the shape of the region. Furthermore, for a specific region such as the esophagus or trachea, the anatomical orthogonal position comparison unit 16 can determine the relationship between each image portion of the specific region and the anatomical orthogonal position based on the shape of each portion in the endoscopic image. Furthermore, for a lumen such as the large intestine, the anatomical orthogonal position comparison unit 16 can determine the relationship between each image portion of the specific region and the anatomical orthogonal position based on changes in image features.

[0044] Note that there may be specific parts whose relationship with the anatomical orthogonal position cannot be determined from the image of the specific part. In this case, the anatomical orthogonal position comparison unit 16 may use information on the specific part whose relationship with the anatomical orthogonal position has been determined to infer the relationship with the anatomical orthogonal position for the specific part whose relationship with the anatomical orthogonal position could not be determined, thereby determining the relationship between each image portion of the specific part and the anatomical orthogonal position.

[0045] The time-series image change determination unit 19 determines changes in images obtained sequentially as the endoscope is inserted, and determines the direction of movement of the tip of the endoscope (insertion, removal, or scanning observation of a specific area, etc.). The time-series image change determination unit 19 is a block composed of circuits, software, an inference model, etc., and determines time-series image changes, and can also determine the state in which the imaging device is approaching a specific affected area.

[0046] In this way, in intraluminal examinations to observe specific internal body regions, similar to the principle of determining one's current location by observing specific scenery along the road while driving down a road, it is possible to determine the position of the endoscope inside the body and the relationship between the vertical and horizontal directions of the image and the specific direction of the luminal cross-section at a specific position (although this is not possible with a car moving with its wheels in the direction of gravity and the road). Once an image of a specific region is obtained, information about the insertion position can be obtained based on that image information, and the vertical relationship between different images can be aligned based on whether the specific region is detected in the vertical, horizontal, or horizontal directions of the image. This innovation allows for the scanning of specific internal regions and the follow-up of specific internal regions during separate examinations. It also makes it possible to observe and record specific regions in the same way, regardless of the physician's endoscope usage habits. Although the term "lumen" is used, the stomach, for example, is more like a bag than a tube, but the same concept applies. Although there are individual differences, the general shape of the bag is fixed, making it possible to determine which part of the stomach is being observed. In the case of the large intestine, the shape of the tube can make it difficult to tell where inside the colon you are looking, but this can be used as a reference if the insertion depth of the scope can be detected. Taking the example of a car, this is similar to how a driver can lose track of their destination if they are driving through a monotonous landscape. However, on highways and other roads, there are signs called "kilometer posts" that allow drivers to confirm their location. The scale on the endoscope tube functions in a similar way, so it is possible to visually check it to input information about the insertion depth, or to take a photograph with a camera to make a judgment, but it is also possible to use this information to determine where the observation is being made.

[0047] The display control unit 17 can provide the endoscopic image processed by the image processing unit 14 to the monitor 30 for display. In this embodiment, an example in which tracking AF is performed is used to facilitate explanation of the improved image quality of a specific region, a feature of the present invention. However, the specific region may also be subjected to processing to improve image quality, visibility, and observability (as previously stated, making it possible to facilitate use of each region for observation and treatment). Images with excellent image quality and suitable for observation and treatment can be obtained on the display screen of the monitor 30 by performing at least one of focusing, exposure adjustment, illumination control, and image processing for visibility. Note that tracking AF is not necessarily required; an image of a precisely focused region may be used as an image of the specific region for observation and treatment. Furthermore, focus control is not required, and pan focus may be used.

[0048] Furthermore, the display control unit 17 can also rotate and correct the image of the specific region and output it to the monitor 30 so that the vertical and horizontal directions of the image of the specific region are aligned with the anatomical position on the display screen. That is, the display control unit 17 continuously rotates the endoscopic images of the specific region acquired sequentially during insertion, thereby displaying endoscopic images that have been processed to unify the orientation of the sequentially acquired endoscopic images based on the anatomical orthogonal position (hereinafter referred to as orientation unification processing). This ensures that the specific region is always displayed in the same orientation on the screen, making it even more suitable for observation and treatment. This is also one of the processes for improving the image quality, visibility, and observability of the specific region, as mentioned above, and is an example of what was previously expressed as making it easier to use for observation and treatment of each region. Here, not only does this process make a single image easier to see, but it also emphasizes the continuity of multiple temporally consecutive images, making it easier to understand the relationship between them. This process can be considered more advanced, as sequential image processing for specific region observation.

[0049] The recording control unit 18 can provide the endoscopic image processed by the image processing unit 14 to the recording unit 40 for recording. The recording unit 40 is a recording device that records on a predetermined recording medium such as a hard disk or memory medium. The recording control unit 18 can also provide the image displayed by the display control unit 17 to the recording unit 40 for recording. Therefore, the image recorded in the recording unit 40 makes it easier to observe and treat a specific area. Furthermore, a knowledge database may be provided in the recording unit 40 so that reference information for judging the image can be recorded.

[0050] For specific sites, endoscopic images acquired during insertion are subjected to AF control and image quality improvement processing, making the images suitable for observation and treatment. Therefore, the control unit 11 may detect a lesion or the like by image analysis of the endoscopic image acquired during insertion and record the endoscopic image with a marking indicating the detection in the recording unit 40. Note that even if the control unit 11 cannot recognize a lesion, it may detect a state in which a change can be recognized compared to the normal state, such as redness or phlegm, and apply a marking.

[0051] By marking the endoscopic image during insertion, i.e., the image of the non-observation site, it becomes easier to focus on the marked site and observe it when removing the insertion section 21, increasing the likelihood of identifying lesions and the like in sites other than the observation site. For example, if the observation site is the stomach, a doctor will not normally focus on the pharynx. However, in this embodiment, if a lesion is present in the pharynx, marking is applied to the endoscopic image of the pharynx acquired during insertion, so that the doctor is more likely to focus on the pharynx, which is not a site that is normally observed, and observe it when removing the insertion section 21. Note that if a mark is applied to a specific site during insertion, the control unit 11 may display a specific recommendation to encourage the doctor to focus on that site when removing the insertion section 21.

[0052] Furthermore, the insertion of the endoscope may cause redness at a specific site, such as the vocal cords. In this embodiment, the image quality of the endoscopic image acquired during insertion is also good, so it is possible to perform image comparison with the endoscopic image acquired during removal with relatively high accuracy. Therefore, by comparing the endoscopic images acquired during endoscope insertion and removal, it is also possible to determine whether or not redness has occurred at the vocal cords or the like due to the insertion of the insertion section 21. Such images of the endoscope tip movement process can be effectively used to obtain various evidence. This is an important technology that can be used in medical procedures, such as for reporting and cause investigation.

[0053] (Operation) Next, the operation of the embodiment configured as described above will be described with reference to Figs. 4 to 8. Fig. 4 is a flowchart for explaining the operation of the first embodiment. Fig. 5 is an explanatory diagram for explaining the insertion of an endoscope, and Fig. 6 is an explanatory diagram showing an example of the relationship between an endoscopic image and an anatomical position. Fig. 7 is an explanatory diagram for explaining a method for determining the relationship between a specific site and an anatomical position. Fig. 8 is an explanatory diagram showing an acquired endoscopic image, and Fig. 9 is an explanatory diagram showing a display example.

[0054] As shown in Fig. 5, in an upper gastrointestinal endoscopy, for example, the insertion section 21 of the endoscope 20 is inserted through the mouth or the like. The physician advances the insertion section 21 from the esophagus toward the stomach by advancing and retracting the insertion section 21 and operating the control section 20a that constitutes the endoscope 20 to bend the bending section at the tip of the insertion section 21 or twist the insertion section 21. During this insertion of the insertion section 21, for example, four endoscopic images P1 to P4 shown in Fig. 6 are obtained. The arrows in the endoscopic images P1 to P4 indicate the same direction based on the anatomical orthogonal position.

[0055] The long side of the rectangular frame in Fig. 6 indicates the left-right direction of the imaging device, and the short side indicates the up-down direction of the imaging device, and the inclination of the rectangular frame indicates the orientation of the endoscopic images P1 to P4. The arrows in Fig. 6 indicate an example of changes in the relationship between the orientation of the endoscopic image and the direction of the anatomical orthogonal position. The anatomical orthogonal position comparison unit 16 determines this relationship between the orientation of the endoscopic image and the direction of the anatomical orthogonal position.

[0056] In FIG. 7 , endoscopic image P5 shows an example in which the lumen is the specific site, and endoscopic image P6 shows an example in which the pharynx and larynx are the specific site. Endoscopic image P5 shows a substantially linear lumen, making it difficult to determine the relationship between the specific site and the anatomical orthogonal position from endoscopic image P5 alone. In contrast, endoscopic image P6 shows the pharynx and larynx, and image features indicate that the pharynx is posterior and the larynx is anterior, making the relationship between endoscopic image P6 and the anatomical orthogonal position clear. The anatomical orthogonal position comparison unit 16 determines the relationship between each specific site and the anatomical orthogonal position from these image features. Furthermore, the anatomical orthogonal position comparison unit 16 can also infer the relationship between the specific site and the anatomical orthogonal position for endoscopic image P5 from the relationship with the anatomical orthogonal position determined for endoscopic image P6 and the characteristics of changes in the endoscopic image over time. Furthermore, in cases where the lumen is curved, the anatomical orthogonal position comparison unit 16 can determine the relationship between the specific site and the anatomical orthogonal position from the characteristics of changes in the endoscopic image due to the bending.

[0057] Figure 4 shows the flow of a gastrointestinal endoscopy. Image acquisition begins in S1 of Figure 4, and the acquired endoscopic image is displayed on the display screen of the monitor 30 (first display). At this timing, the acquired image may be recorded in the recording unit 40. In this embodiment, such recorded images can be used for processing in subsequent steps, and can also be used generally as evidence or reports for medical procedures. The endoscopic image acquired by the imaging element 22 is imported into the image processing device 10 by the image acquisition unit 13, and then subjected to predetermined signal processing by the image processing unit 14. The display control unit 17 provides the signal-processed endoscopic image to the monitor 30 for display.

[0058] The control unit 11 determines whether the insertion unit 21 has been inserted through the mouth, nose, or the like through image analysis (S2). This determination can be made by detecting changes in the image that have characteristics of progress images during position change of the endoscope tip. In other words, taking the example of a lumen, an image of the interior of the lumen (deep portion (deeper in the direction of the lumen length)) where illumination light from the endoscope tip does not reach would have a black central portion (see FIG. 14 ) that conforms to the cross-sectional shape of the lumen (which is often approximately circular), and as the image progresses toward that portion, the periphery of the black circular portion gradually becomes brighter, resulting in an image pattern that flows radially from there to the periphery of the screen, and this image change can be determined. This determination is made by the time-series image change determination unit 19.

[0059] When the insertion of the insertion section 21 begins (YES in S2), the specific region determination section 15 determines the image features and detects the specific region (S3). The specific region can be determined by referring to a database stored in the recording section, which stores the region's characteristic color or shape, or patterns of blood vessels or other visible blood vessels on the surface. Another method for determining the specific region is to use an external receiver to receive the results of transmissions from a magnetic or other transmitter located at the tip of the endoscope and determine where the tip of the endoscope is located on the body. In addition to referring to a knowledge database, the specific region may also be determined using an inference model trained using images of each region as training data.

[0060] In S4, the imaging control unit 12 controls the endoscope 20 to perform tracking AF on the specific region. The image processing unit 14 also performs predetermined image signal processing to improve the image quality of the specific region (S4). In this way, an image of the specific region that is highly visible and suitable for observation or treatment can be obtained.

[0061] The anatomical orthogonal comparison unit 16 also determines the positional relationship (orientation relationship) between the specific part in the image and the anatomical orthogonal position (S5). For example, the anatomical orthogonal comparison unit 16 obtains information about the twist of the insertion unit 21 during insertion.

[0062] The display control unit 17 performs rotation correction on the endoscopic image based on the detection result (torsion information) of the anatomical orthogonal position comparison unit 16, and then outputs the image to the monitor 30 (S6). As a result, the endoscopic image of the specific region is displayed on the display screen of the monitor 30 in a manner that maintains a constant directional relationship with the anatomical orthogonal position (second display).

[0063] The system includes a processor that receives continuous image data (endoscopic images) from an endoscope that acquires images using an imaging device at the tip of the insertion portion, and performs this rotation correction image processing and displays the results. The receiver can be configured using the image acquisition unit 13. Since a specific region is detected based on image features within the endoscopic image data, each of the continuous images is rotationally corrected based on the detected region.

[0064] FIG. 8 shows a series of endoscopic images P7 to P14 obtained during insertion, and shows, for example, frames of a moving image obtained by imaging. The circles in endoscopic images P10 to P12 in FIG. 8 indicate examples of the same specific region detected from the images. The specific region determination unit 15 detects the circled specific region in the captured endoscopic image P10. Based on the detection result of the specific region determination unit 15, the imaging control unit 12 performs tracking AF, and as a result, the same specific region is captured in focus in the endoscopic images P11 and P12. This specific region is captured with appropriate exposure and is subjected to image quality improvement processing in the image processing unit 14.

[0065] However, when the endoscopic images P10-P12 were acquired, the insertion section 21 was twisted, and as indicated by the inclination of the endoscopic images P10R-P12R, the endoscopic images P10-P12 were rotated at different angles relative to the anatomical orthogonal position. The anatomical orthogonal position comparison unit 16 determines the positional relationship (torsion information) between the specific region in each of the endoscopic images P10-P12 and the anatomical orthogonal position. The display control unit 17 displays endoscopic images P10P-P12P, which are obtained by rotating the endoscopic images P10R-P12R, on the monitor 30 based on the torsion information. Note that the endoscopic images P10P-P12P are obtained by cropping and enlarging the portions corresponding to the specific regions. Note that, although an example of displaying the cropped images has been shown, only the cropping process may be performed.

[0066] FIG. 9 shows a display example in this case. The left side of FIG. 9 shows a display example of the monitor 30 when an endoscopic image P10 is acquired, and the right side shows a display example of the monitor 30 when an endoscopic image P12 is acquired. The left side of the display screen of the monitor 30 displays endoscopic images DP10 and DP12 (first display) with improved image quality, and the right side of the display screen displays endoscopic images DP10P and DP12P (second display) that have been rotated and enlarged based on the anatomical orthogonal position. The first display can be used, for example, to confirm the insertion direction when inserting the insertion section 21. Furthermore, the second display not only has good image quality but is also displayed in an orientation based on the anatomical orthogonal position, making it easy to use for observation, treatment, and the like. The first and second displays may be displayed side by side, not only horizontally but also vertically.

[0067] The recording control unit 18 provides the endoscopic image to the recording unit 40 for recording (S7). In this case, an endoscopic image in which the visual recognition of the specific site has been improved is recorded. The recording control unit 18 also records specific site information related to the specific site. The recording control unit 18 also records information about any problems discovered during insertion.

[0068] The control unit 11 determines whether or not it is no longer possible to detect the features of the specific part in S8 (feature end?). If feature detection has not ended (NO in S8), the process returns to S5. If feature detection has ended (YES in S8), the process returns to S2. This specific part is determined by the specific part determination unit 15.

[0069] In the above description, an example has been explained in which all endoscopic images acquired during insertion of the endoscope 20 are used for both the first display and the second display, but different frames may be set for the first display, the second display, or the recording. Furthermore, while it has been described that tracking AF is performed on images after specific portions have been detected, it is also possible to set frames for which tracking AF is performed and frames for which it is not performed. This prevents the effects of tracking AF from deteriorating the image portion related to insertion of the image to be confirmed for insertion.

[0070] If the control unit 11 determines in S2 that the mode is not the insertion mode (NO in S2), it determines in S11 whether the mode is the removal mode.

[0071] This can be determined by the time-series image change determination unit 19 detecting changes in images that have characteristics of progress images during position change of the endoscope tip. In other words, taking the example of a lumen, an image of the back of the lumen (deep in the direction of the lumen length) where illumination light from the endoscope tip does not reach will have a black central portion (see FIG. 14 ) that follows the cross-sectional shape of the lumen (which is often approximately circular), and as light exits from that portion, the periphery of the black circular portion will gradually become darker and an image pattern will converge radially from the periphery of the screen, and this image change can be determined.

[0072] When the control unit 11 determines in S11 that the mode is not the removal mode (NO in S11), it determines in S21 whether the mode is the observation confirmation mode.

[0073] This can be determined by the time-series image change determination unit 19 detecting changes in images that have characteristics of progress images during changes in the position of the endoscope tip. That is, in the example of a lesion on the side of a lumen, during insertion or removal, the image of the back of the lumen (deeper in the direction of the lumen length), where the illumination light from the endoscope tip does not reach, will have a black center (see FIG. 14 ) that conforms to the cross-sectional shape of the lumen (which is often approximately circular) (detected in the image data), and as the endoscope tip is bent from this state toward the wall of the target, it can be determined that the hole pattern that was in the center moves to the periphery of the screen, and then a specific patterned area is observed for a long period of time by moving closer or further away or by changing the viewing direction, and similar patterns such as the unevenness of blood vessels or lesions are captured in the successive images, and this can be determined.

[0074] In the observation confirmation mode (YES in S21), the control unit 11 determines and differentiates the lesion by known image processing, AI (artificial intelligence) processing, etc., and displays the results on the monitor 30 (S22).

[0075] In the removal mode (YES in S11), the control unit 11 performs a missed detection determination (S12). In the missed detection determination, it is determined whether or not the removal was too fast or whether or not a part that may be a lesion was missed based on image features.

[0076] Next, the specific region determination unit 15 determines the image features and detects the specific region (S13). This image feature determination analyzes patterns contained in images obtained from the imaging unit at the tip of the endoscope insertion unit to determine which region in the human body the imaging unit (endoscope tip) is located at, detecting structures, colors, irregularities, vascular patterns, and the like that are characteristic of that region. The specific region determination unit 15 uses information such as the obtained patterns to determine the region by referencing a database that associates images with regions. The specific region determination unit 15 can be configured with electronic circuits, a processor, memory, and the like. The specific region determination unit 15 can also use an inference model, such as a model trained using training data in which images of specific regions are annotated with information representing the region. Even if the region does not have characteristics, it is possible to determine which region the image corresponds to based on the patterns of the surrounding regions. For example, regions before and after a pattern unique to the vocal cords can be determined to be the entrance to the trachea or esophagus. Furthermore, it is not necessary to rely on endoscopic images; the location can be determined by detecting a signal from a transmitter installed at the tip of the endoscope using a sensor installed outside.If the examination proceeds according to a specific time schedule, the location can be determined using information on the elapsed time since the start of the examination.

[0077] The control unit 11 uses the specific site information recorded during insertion to recommend reconfirmation at the time of removal, if necessary (S14). Next, in S15 to S17, the same processing as in S5 to S7 during insertion is performed. That is, even during removal, torsion information is acquired, the endoscopic images are rotated based on the torsion information, and the orientation of each endoscopic image is aligned based on the anatomical orthogonal position to produce a second display. Images containing the specific site information obtained at the time of removal are also recorded. Thus, temporally consecutive image data (endoscopic images) are received from the endoscope that acquires images using an imaging device provided at the tip of the insertion portion, the specific site is detected based on image features within the images of the endoscopic image data, and each of the temporally consecutive endoscopic images is rotationally corrected based on the specific site image and recorded.

[0078] Furthermore, if the control unit 11 determines in S21 that the mode is not the observation confirmation mode (NO in S21), it determines in S25 whether the mode is the specific part confirmation mode. The specific part confirmation mode is for confirming the recorded specific part information. In the specific part confirmation mode, the control unit 11 displays the recorded results for each specific part (S26). If a different examination is required for a part, the control unit 11 recommends the different examination (S27).

[0079] In the example of FIG. 4, the specific region confirmation mode is performed during the endoscopic examination, but the specific region confirmation mode may be performed after the examination.

[0080] In this embodiment, the image quality of a specific region of the human body in an endoscopic image acquired by the endoscope during insertion is improved by processing the image. The relationship between the on-screen orientation of the specific region and the anatomical orthogonal position is determined, and the up, down, left, and right directions of the endoscopic image displayed on the screen are aligned with the anatomical orthogonal position. By aligning the orientation of the image of the specific region with the anatomical orthogonal position (orientation unification processing), for example, the image portion of the vocal cords in the pharynx is aligned with the upper position on the endoscopic image, and the image portion of the esophagus is aligned with the lower position on the endoscopic image. Because orientation unification processing is performed, the endoscopic image obtained during insertion of the endoscope has excellent visibility, making it easier to use for observation, treatment, and the like.

[0081] In addition, the above explanation shows an example in which direction unification processing is performed on endoscopic images acquired when the endoscope is inserted, but the timing for performing direction unification processing may be when the endoscope is inserted, when it is removed, or after the examination is completed.

[0082] Furthermore, in the above explanation, an example was given in which direction unification processing was performed on endoscopic images acquired when the endoscope was inserted, but direction unification processing may also be performed on endoscopic images acquired when the endoscope was removed. In the above embodiment, the features of the present application were clearly summarized using an example in which processing was performed separately for insertion, removal, and observation and confirmation, but the present invention does not strictly separate insertion, removal, and observation and confirmation, and actively acquires valuable information if it can be obtained in any state, and attempts to effectively utilize the acquired information even if it is related to another clinical department.

[0083] (Examples of specific portions) FIGS. 10 to 12 are explanatory diagrams showing examples of specific portions.

[0084] Figures 10 and 11 show examples of specific regions that can be detected during upper endoscopy. The example in Figure 10 shows that the soft palate, hard palate, tongue, hyoid bone, epiglottis, thyroid cartilage, nasopharynx, oropharynx, hypopharynx, cricoid cartilage, etc. can be detected as specific regions. The example in Figure 11 shows that the esophagus, cricoid ligament, tracheal muscle, tracheal cartilage, airway mucosa, airway epithelium, tracheal glands, and lamina propria can be detected as specific regions.

[0085] 12 shows examples of specific regions that can be detected during insertion in a lower gastrointestinal endoscopy and captured images of each specific region. The example in FIG. 12 shows that the cecum, ileum, ascending colon, hepatic flexure, transverse colon, descending colon, sigmoid colon, SD junction, and rectum can be detected as specific regions. In a lower gastrointestinal endoscopy, the anus, dentate line, and the like can also be detected as specific regions.

[0086] It should be noted that the present invention is not limited to the above example, and specific parts can be detected in various parts of the human body.

[0087] In this embodiment, when the endoscope is inserted, specific regions other than the region to be observed can be detected and images suitable for observing and treating those specific regions can be obtained, making it possible to check for various diseases in each specific region other than the region to be observed. For example, according to this embodiment, during upper gastrointestinal endoscopy, it is possible to check for diseases such as nasopharyngeal cancer, oropharyngeal cancer, hypopharyngeal cancer, acute epiglottitis, eosinophilic sinusitis, acute sinusitis, chronic sinusitis, and allergic rhinitis, and during lower gastrointestinal endoscopy, it is possible to check for various diseases such as anal fissures, hemorrhoids, internal hemorrhoids, and anal fistulas.

[0088] (Second embodiment) Fig. 13 is a flowchart showing the operation flow employed in a second embodiment. In Fig. 13, the same steps as in Fig. 4 are assigned the same reference numerals, and the description thereof will be omitted. The hardware configuration in this embodiment is the same as in Fig. 1. In this embodiment, endoscopic images acquired during insertion are synthesized.

[0089] FIG. 13 differs from FIG. 4 in that S6 is omitted and S31 and S32 are added. After S5, in S31, the control unit 11 performs image synthesis and displays the generated synthetic image. At this timing, the acquired image may be recorded in the recording unit 40. In this embodiment, such recorded images can be used for image synthesis, and can also be used generally as evidence or reports for medical procedures. Next, in S8, the control unit 11 determines whether feature detection has been completed. If feature detection has been completed (YES in S8), the recording control unit 18 provides the generated synthetic image to the recording unit 40 for recording.

[0090] 14 and 15 are explanatory diagrams for explaining image synthesis in a second embodiment. This embodiment not only rotates and corrects images, but also synthesizes the corrected images to obtain a new examination result image. This system relates to a system having a receiving unit that receives temporally consecutive image data (endoscopic images) from an endoscope that acquires images using an imaging device installed at the tip of an insertion section. This system may be part of an in-hospital system for creating reports, or part of an endoscopic system that acquires examination images. A specific region is detected based on image features within the endoscopic image data, and each of the temporally consecutive endoscopic images is rotationally corrected based on the image of the specific region within the endoscopic images. The rotationally corrected images are associated with text regarding the symptoms of the specific region as the examination result (generated as a report). However, before the rotationally corrected images are used as the examination result, they are panoramic-combined as shown in FIG. 15. Of course, if the region of interest can be contained within a single frame without synthesis, the panoramic-combining step is not necessary.

[0091] 14 shows endoscopic images P21 to P25 obtained by sequentially capturing images along the time axis. These endoscopic images P21 to P25 include an image of the hole in the lumen, shown in solid, and images Lp1 to Lp5 of the lesion area along the lumen. Images Lp1 to LP5 are portions of a single continuous lesion area LP, and were captured by dividing the image into frames as the insertion section 21 was advanced through the lumen.

[0092] These images are not intended to capture images of the lesion, but are intended to be used as guide information during insertion, assuming that the target area to be examined is located deeper than the lumen. In other words, these images should be called images of the endoscope tip moving process or progress images during position change. Note that if the lesion is clearly visible as in this figure, the doctor will notice it and begin observing it, but in actual images, the lesion may not be so clearly visible. This example assumes a situation in which the doctor simply passes by like this and does not perform the examination unless the area is the doctor's current target of examination.

[0093] In this embodiment, as in the first embodiment, torsion information is acquired based on the anatomical orthogonal position, and the endoscopic images P21 to P25 are subjected to orientation unification processing based on this torsion information so that the up, down, left, and right directions are consistent based on the anatomical orthogonal position. Therefore, as shown in Figure 15, by superimposing the overlapping portions of adjacent frames of these endoscopic images P21 to P25 and combining them into a panoramic image, a composite image PL that includes the entire lesion LP can be obtained.

[0094] In this embodiment, the orientation of each successively acquired endoscopic image is aligned based on the anatomical orthogonal position, making it possible to display the entirety of a relatively large lesion through image synthesis. For example, it is possible to display the entirety of a lesion that is long in the longitudinal direction of the lumen (lumen direction), making it easier to recognize the entire lesion. When the entire image is displayed in this way, lesions that would not have been noticeable in the individual images of each frame during the movement of the endoscope tip or the individual images of each frame during the position change process can be more easily noticed because the shape and boundaries of the lesion are clearly indicated. For example, it can also be easier to identify lesions through inference based on image assessment. Furthermore, when passing through such a specific internal region, performing focus control, exposure control, and other image processing control appropriate for that region may make it easier to more clearly identify such lesions or diseased areas. In this embodiment, an image processing method is obtained in which a specific region is detected based on image features of an endoscopic image acquired by an imaging device attached to the tip of the endoscope insertion portion during insertion of the endoscope during endoscopic examination, and based on the endoscopic image, the directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted is determined. Based on this directional relationship, the endoscopic images sequentially acquired during insertion are continuously rotated, thereby performing orientation unification processing to unify the orientations of the sequentially acquired endoscopic images based on the anatomical position. Since the directional relationship with the anatomical position of the body is determined, it is easy to organize the images by linking them to anatomical terms for the region.

[0095] (Modification) Fig. 16 is a flowchart showing a modification of the second embodiment. In Fig. 16, the same steps as in Fig. 13 are assigned the same reference numerals, and description thereof will be omitted. The hardware configuration in this modification is the same as in Fig. 1. In the example of Fig. 13, endoscopic images are synthesized when the endoscope is inserted, but in this modification, endoscopic images are synthesized in a specific site confirmation mode after the endoscope is inserted.

[0096] 16, when the endoscope is inserted, the processes of S31 and S32 in FIG. 13 are omitted, and in the specific region confirmation mode (YES in S25), S35 corresponding to S31 is executed. Furthermore, S27 is followed by S36 corresponding to S32. That is, in this modification, the generation and recording of the composite image are performed in the specific region confirmation mode.

[0097] The other configurations, actions, and effects are the same as those of the second embodiment.

[0098] 16 shows an example in which endoscopic images are synthesized and recorded during the endoscopic examination, but they may also be synthesized and recorded after the examination. Focusing and lighting control, which involve position control of the optical system, are preferably controlled during the examination, but adjustments such as gain adjustment of screen brightness, color, contrast, and gradation may also be performed on image data recorded after the endoscopic examination.

[0099] (Third Embodiment) Fig. 17 is a block diagram showing a third embodiment. In Fig. 17, the same components as those in Fig. 1 are given the same reference numerals, and their description will be omitted. This embodiment shows an example in which endoscopic images acquired when an endoscope is inserted are used after the examination.

[0100] The first examination device 60 in FIG. 17 corresponds to the endoscope 20, image processing device 10, and recording unit 40 in FIG. 1. The recording unit 40 stores a first examination result 40A and a second examination result 40B. The first examination result 40A is an examination result including an endoscopic image of the observation target region. That is, the first examination result 40A includes an examination result obtained with the physician's intention to observe. The second examination result 40B is an examination result including an endoscopic image of a region other than the observation target region (hereinafter referred to as a non-observation target region) at the time of endoscope insertion, with the image orientation unified based on the anatomical orthogonal position. That is, the second examination result 40B includes an examination result obtained without the physician's intention to observe. These examination results are transmitted to the in-hospital system 50. Of course, the second examination result may supplement the first examination result and may be obtained by recording an item that could not be recorded as the first examination result, regardless of the physician's intention.

[0101] The arithmetic and control unit 51 of the in-hospital system 50 comprehensively controls each unit of the in-hospital system 50. The arithmetic and control unit 51 and each unit in the in-hospital system 50 may be configured by a processor using a CPU or the like, may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of its functions by a hardware electronic circuit. The in-hospital system 50 processes not only medical procedures such as tests and treatments, but also various hospital operations such as reception, prescriptions, and accounting, but FIG. 17 mainly shows a configuration related to diagnosis using test results.

[0102] The display output control unit 52 of the in-hospital system 50 controls various image displays, printing, and data outputs. The communication unit 53 is capable of communicating with external lines such as the Internet 110, and enables information searches by acquiring information from the Internet 110, etc. As will be described later, the learning assistance unit 54 uses information acquired from the Internet 110, etc. to create training data, etc. for constructing an AI inference model.

[0103] The data input / output unit 56 of the in-hospital system 50 receives the first test result 40A and the second test result 40B from the first testing device 60 and provides them to the data storage unit 59. The data storage unit 59 is configured with a predetermined recording medium and stores various data. The data input / output unit 56 and the image group input unit 55 form a receiving unit.

[0104] As described above, the orientation unification process can be performed after the endoscopic examination. The image group input unit 55 in the in-hospital system 50 receives the first examination result 40A and the examination result including the endoscopic image acquired by the first examination device 60 during endoscope insertion and not subjected to orientation unification process (hereinafter referred to as the uncorrected second examination result 40B), and outputs it to the image processing device 95. The image processing device 95 as the second processor has the same functions as the image processing device 10 as the first processor in the first examination device 60, and performs orientation unification process on the uncorrected second examination result 40B acquired by the image group input unit 55 to obtain data similar to the second examination result 40B. If the first examination device 60 cannot obtain the second examination result 40B, the image processing device 95 provides the obtained second examination result 40B to the data storage unit 59 for storage.

[0105] Generally, doctors do not pay attention to images related to medical fields other than their own specialty. For example, if the endoscope 20 of the first examination device 60 is a gastrointestinal endoscope, a gastroenterologist who focuses on the first examination result 40A will not pay attention to the second examination result 40B, which includes image portions of the bronchi, etc. In this embodiment, diagnosis is performed using endoscopic images obtained when an endoscope that is not commonly used is inserted, thereby obtaining information useful for diagnosis and treatment.

[0106] The report unit 80 in the in-hospital system 50 includes a first test result report unit 81, a second test result report unit 82, and an additional test / treatment information generation unit 83. The report unit 80 generates a report based on the first test results 40A and the second test results 40B. A doctor controls the report unit 80 by operating the input operation unit 57. The first test result report unit 81 of the report unit 80 reads the first test results 40A from the data storage unit 59 and displays the test results based on the first test results 40A on a monitor (not shown). The doctor diagnoses the observation target area while referring to the monitor display and creates a medical record including information such as the presence or absence of a lesion, its condition, treatment method, and prescription. The first test result report unit 81 records the medical record created by the doctor as a first test result report on a recording medium (not shown).

[0107] On the other hand, a doctor who requests an examination using the first examination equipment 60 is often a specialist in the area to be observed, and may not be an expert in areas not to be observed, and therefore does not usually use the results of the second examination.

[0108] In this embodiment, the second test result report unit 82 of the report unit 80 retrieves the second test result 40B from the data storage unit 59 and automatically performs a diagnosis using the second test result. The second test result report unit 82 performs automatic text generation to automatically generate diagnostic content, such as the presence of a lesion, as text as a result of the diagnosis.

[0109] The second examination result report unit 82 may retrieve the corresponding text by image search using a database in which text corresponding to each image of a specific case is prepared, or if there are image features (differences in shape, location, size or color of the lesion) that differ from the images in the prepared database, it may be configured to retrieve text prepared for each feature.In addition to database search, the second examination result report unit 82 may input the image into an inference model that has been trained by annotating the image and information that can be read from it and that can be converted into text, and then output the text.

[0110] The second examination result report unit 82 also performs flagging, generating various flags corresponding to images of the detected lesions. The flags indicate the presence or absence of abnormalities, the location information of the lesions, etc. Possible flags include an abnormality flag indicating some kind of abnormality, a lesion flag indicating the lesion, a reexamination flag recommending a reexamination, etc. The second examination result report unit 82 records the text and flags as a second examination result report on a recording medium (not shown).

[0111] For example, suppose the target observation site is the digestive tract and the specific non-target observation site is the throat. For example, if there is a "red area" in the throat, the image of the specific site may include a red image portion, and some kind of flag corresponding to the red image portion may be embedded in the second examination result 40B. However, a gastroenterologist diagnosing the target observation site may not be able to identify a lesion by looking at the endoscopic image of the throat. In response to this, the second examination result report unit 82 performs lesion identification for the red image portion, and if it determines that the red image portion is a lesion, sets an abnormality flag, lesion flag, reexamination flag, etc., corresponding to the red image portion and generates text indicating that the image portion is a lesion. The second examination result report unit may also generate text, etc., recommending a reexamination by a specialist physician, if necessary. If infectious symptoms are identified from the throat image, etc., instructions for immediate diagnosis and treatment may be issued.

[0112] The second test result report unit 82 may use, for example, a knowledge DB (database) unit 90 and an inference unit 100 to perform flagging and automatic text conversion.

[0113] The additional examination and treatment information generating unit 83 not only leaves a report but also generates prescriptions, appointments, and other information encouraging patients to visit the hospital. This information may be generated by input operations by a doctor, or may be generated by the knowledge DB unit 90 or the inference unit 100.

[0114] The knowledge DB unit 90 stores test data including case images for various cases, and knowledge information N1, N2, ... (hereinafter, when there is no need to distinguish between these pieces of information, they will be referred to as knowledge information N) which is medical knowledge such as the cause of the case and treatment methods for the case. The text conversion unit 91 of the knowledge DB unit 90 receives the second test results 40B from the report unit 80, and determines whether the second test results 40B contain abnormalities such as lesions by comparing the images included in the second test results 40B with the knowledge information N1, N2, ... for each case, and generates a text explanation based on the knowledge information N about the presence or absence of the abnormality and its content.

[0115] The inference unit 100 also stores inference information AI1, AI2, ... (hereinafter, referred to as inference information AI when there is no need to distinguish between these pieces of information) which are inference models for the disease name, cause of occurrence, treatment method, etc. of each case, corresponding to test data including case images for each of various cases. The text generation unit 101 of the inference unit 100 is provided with the second test results 40B from the report unit 80, performs inference using each piece of inference information on the test data including the images included in the second test results 40B, determines whether the second test results 40B contain any abnormalities such as lesions, and generates a text explanation based on the inference information AI regarding the presence or absence of the abnormality and its content.

[0116] The learning assistance unit 54 supplements the knowledge information N in the knowledge DB unit 90 and the inference information AI in the inference unit 100 by utilizing information from the Internet 110 and the like.

[0117] Like the first inspection device 60, the second inspection device 70 includes an endoscope and an image processing device (not shown). The image processing device of the second inspection device 70 has the same functions as the image processing device 10 of the first inspection device 60. The endoscope of the second inspection device 70 is an endoscope different from the endoscope 20 of the first inspection device 60. For example, if the endoscope 20 of the first inspection device 60 is a digestive endoscope, the endoscope of the second inspection device 70 is a bronchial endoscope or the like. In this case, the second inspection device 70 obtains a second inspection result 70A similar to the second inspection result 40B, and also obtains a third inspection result 70B different from the second inspection result 70A when the endoscope is inserted.

[0118] The in-hospital system 50 can also be linked to a user terminal 120 and has a log recording unit 58 that records a daily log 121 acquired by the user terminal 120. The user terminal 120 is equipped with various sensors as necessary, and is configured to create a daily log 121 containing health-related information, such as exercise time, pulse rate, blood pressure, body temperature, heart rate, sleep time, toilet time, electrocardiogram, etc., for the user who carries the user terminal 120.

[0119] The daily log 121 may or may not include what is called biometric information or vital information. A dedicated sensor may be required to acquire vital information, and even if such a sensor is a separate device from the user terminal 120, the acquired information is considered to be linked. The in-hospital system is also capable of acquiring data obtained from such devices from the user terminal 120. This data may be temporarily recorded in a recording unit within the user terminal 120.

[0120] The user terminal 120 receives the daily log 121 via the communication unit 53 and provides it to the log recording unit 58. The log recording unit 58 records the daily log 121 for each user. Meanwhile, the in-hospital system 50 is equipped with a patient database (not shown) that records information on the history of symptoms, etc., for users who have acquired the daily log 121 via the user terminal 120. The learning assistance unit 54 is able to improve accuracy by supplementing the knowledge information N and inference information AI in the knowledge DB unit 90 through learning using the daily log 121 for a specific user recorded in the log recording unit 58, the user's history of symptoms, etc., and information from the Internet 110.

[0121] In other words, the in-hospital system 50 and the Internet contain logs and symptom histories of multiple users (various patients), and detailed analysis is possible by selecting and using information on patients with similar profiles and daily logs to that of the user in question. When there is information on patients with similar symptoms, the treatment records of patients with similar profiles and lifestyle patterns are more useful than the treatment records of other patients with different ages, genders, and lifestyle patterns. Accuracy can also be improved by taking genetic information into consideration, grouping (categorizing) similar groups of users, comparing information, and using AI to make inferences.

[0122] In the above description, the first inspection device 60 unconditionally performs orientation unification processing to create the second inspection result 40B in which the orientation of the image of the specific region is unified based on the anatomical upright position. However, the enable control unit 40C, which is configured by the control unit 11 in the image processing device 10, may be configured to control whether or not to perform orientation unification processing on the endoscopic image acquired during endoscope insertion. In this case, the knowledge DB unit 90 or the inference unit 100 may be configured to control the enable control unit 40C based on the contents of the daily log 121 read from the log recording unit 58. The enable control unit 40C enables observation using an image representation required for each situation, or an image representation that suits the preference of a medical professional such as a doctor, or an image representation that is easy for a patient to understand when viewing.

[0123] In addition, the additional examination and treatment information generation unit 83 may take into account not only the report created by the second examination result report unit 82 but also the contents of the daily log 121 recorded in the log recording unit 58 to determine the need for re-examination, other examinations, treatment, etc.

[0124] The user terminal 120 is assumed to be a device that the user uses on a daily basis, and by acquiring information from it as a log, it becomes possible to monitor the user's condition on a daily basis. The acquired data is judged to be normal or not, and when the judgment approaches abnormality over time, the treatment information generation unit 83 can recommend the next medical procedure or use it as a memorandum.

[0125] Even with a simple pedometer (registered trademark) function, if a user who used to walk 10,000 steps a day suddenly starts walking only 1,000 steps one day, it is likely that some change in their lifestyle has occurred. By determining such a situation, events that occurred prior to that day can be obtained from the daily log 121, medical record information, etc., and used as medical information. For example, if there is a record of a medical institution advising the patient to rest, it is possible to compare the relationship between that advice and changes in lifestyle. If the result is a result of receiving advice, the medical institution can commend the patient for adhering to the advice, motivating them to stay healthy.

[0126] In addition to the Pedometer (registered trademark), it is also possible to detect the user's breathing sounds from the voice picked up by the microphone of a smartphone, compare the waveform with that of normal breathing sounds, and suspect respiratory diseases.

[0127] Abnormal breath sounds include intermittent adventitious sounds, continuous adventitious sounds, and pleural rubs. Depending on the user's condition, characteristic sound waveforms, such as specific frequency components or repetitive patterns, can be obtained. It can also determine the normal respiratory rate, which is considered 12 to 18 breaths per minute. While active integration with the hospital's system is not necessary if the patient is within the normal range, this information, such as the absence of respiratory symptoms, may be important for diagnosis and treatment in the presence of other illnesses. When respiratory abnormalities are detected, the following decisions can be made to provide a more tailored, yet less burdensome, treatment approach for the user. Decision 1: Examine the throat during the next endoscopic examination during the next health check. Decision 2: Recommend immediate medical consultation. Decision 3: Consult the doctor during the next medical checkup. Similar applications can be applied to internal sounds, such as heart sounds, arterial sounds, and bowel sounds, in addition to lung sounds.

[0128] (Operation) Next, the operation of the embodiment configured as above will be described with reference to Figures 18 to 20. Figures 18 to 20 are flow charts for explaining the operation of the third embodiment.

[0129] 18 to 20 show an example of cooperation between the in-hospital system 50, the first inspection device 60, and the user terminal 120. Fig. 18 shows the operation flow of the first inspection device 60, which is an endoscope device, Fig. 19 shows the operation flow of the user terminal 120, and Fig. 20 shows the operation flow of the in-hospital system 50.

[0130] 18, information such as patient information about a patient undergoing an endoscopic examination is input by a user operating an input operation unit (not shown) provided in the first examination equipment 60. The patient information may already be recorded in a patient database in the in-hospital system 50, and the control unit 11 may read the patient information from the patient database. The input information is recorded in the recording unit 40. In addition, the records in the patient database are updated with information about the examination to be undergone.

[0131] The control unit 11 determines whether the examination has started in S42. When the examination has started (YES in S42), image acquisition begins (S43). The endoscopic image acquired by the image sensor 22 is input into the image processing device 10 by the image acquisition unit 13, and predetermined signal processing is performed by the image processing unit 14. The display control unit 17 provides the signal-processed endoscopic image to the monitor 30 for display. In addition, the recording control unit 18 provides the signal-processed endoscopic image to the recording unit 40 for recording.

[0132] Next, the control unit 11 determines by image analysis whether the insertion unit 21 has been inserted through the mouth, nose, etc. (S45). When insertion of the insertion unit 21 begins (YES in S45), the control unit 11 executes insertion processing. When the control unit 11 determines in S45 that the mode is not the insertion mode (NO in S45), it determines in S46 whether the mode is the removal mode. If the mode is the removal mode (YES in S46), the control unit 11 executes removal processing (S47).

[0133] If the controller 11 determines in S46 that the mode is not the removal mode (NO in S46), the controller 11 executes the observation confirmation mode in S50. In the observation confirmation mode, for example, still image capture is performed. Next, the controller 11 acquires the first examination result 40A.

[0134] After the insertion process of S45 and the removal process of S47, if there is enable control, the control unit 11 acquires the second test result 40B. In the next step S49, the control unit 11 outputs information as necessary.

[0135] The in-hospital system 50 is provided with an image processing device 95, and the second test result 40B can also be acquired by the image processing device 95. For example, if the second test result 40B is not generated by the first testing device 60, the control unit 11 may sequentially transmit the first test result 40A and the uncorrected second test result 40B to the in-hospital system 50 during the endoscopic examination (S49). Alternatively, the control unit 11 may transmit the first test result 40A and the second test result 40B, or the first test result 40A and the uncorrected second test result 40B, to the in-hospital system 50 all at once after the examination. In this case, the control unit 11 outputs information by determining NO in S42 (S44).

[0136] In this way, the first test result 40A and the second test result 40B may be created in the first testing equipment 60, and the first testing equipment 60 may output the first test result 40A and the uncorrected second test result 40B, and the second test result 40B may be obtained in the in-hospital system 50.

[0137] In S61 of FIG. 19 , the user terminal 120 acquires information using various sensors and sequentially records the acquired information in a memory within the user terminal 120 (not shown). This creates a daily log 121. The user terminal 120 determines an operation in S62. If an operation has occurred (YES in S62), the user terminal 120 executes a function corresponding to the operation (S63). For example, if the user terminal 120 has a telephone sending and receiving function, a telephone sending and receiving operation may be performed. If no operation has been performed (NO in S62), information recording continues. Note that if there is an operation from the in-hospital system 50 requesting transmission of the daily log 121, the accumulated daily log 121 is transmitted to the in-hospital system 50 in response to this request (S63).

[0138] 20, various information is input by user operation of an input operation unit (not shown). The calculation control unit 51 determines whether the input information is information related to reception, reservations, etc. (S72). If the input information is information related to reception, reservations, etc. (YES in S72), the calculation control unit 51 records individual information such as schedules in the patient database in S80, and the process returns to S71.

[0139] If the determination in S72 is NO, the calculation and control unit 51 determines whether the input information is information regarding examination and treatment results (S73). If the determination in S73 is NO, the calculation and control unit 51 determines whether the input information is information regarding test results (S74). If the input information is information regarding test results (YES in S74), the calculation and control unit 51 determines whether the input information is information regarding the first test result 40A (S75). If the input information is information regarding the first test result 40A (YES in S75), the calculation and control unit 51 records the first test result 40A in the data storage unit 59 (S76). Furthermore, in S77, if a second test result 40B or a third test result 70B (nth test result) is available, the calculation and control unit 51 records these test results in the data storage unit 59. The second test result 40B and the second test result 70B are flagged and converted to text as necessary.

[0140] If there is a daily log 121 (terminal information) from the user terminal 120, the calculation control unit 51 records this information in the log recording unit 58. After S77 ends or if the determinations of S74 and S75 are NO, the calculation control unit 51 controls to enable the direction unification process based on the daily log 121 (S78). In S79, the calculation control unit 51 executes prescription, accounting, etc., and returns the process to S71.

[0141] If the input information is information about the examination and treatment results (YES in S73), the calculation and control unit 51 determines in S81 whether or not there are test results or other information. In S81, it is determined whether, for example, endoscopic images or X-ray images are present. If present, the calculation and control unit 51 recommends that the doctor check the test results as necessary (S82), and if not present, the process proceeds to S83.

[0142] A study that collects new cases in the future to verify hypotheses about what will happen to patients over a specific period of time is called a "prospective study," while a study that collects past cases to verify hypotheses is called a "retrospective study." To conduct such a verification, it is necessary to track changes in the health status of specific patients. Research can also be conducted by tracking changes in the health status of a group of patients with a specific profile and specific medical condition. In this embodiment, daily logs and in-hospital test results are linked, creating an environment that facilitates such research.

[0143] For example, when multiple patient information sets are collected with similar profiles and histories (determined by accumulating daily logs), the collected patient information can be tagged and organized into groups. For example, if initially patient data on patients in their fifties is collected and many patients with similar symptoms are examined, new groupings can be made, such as categorizing men and women into separate categories. The creation of a new group or an increase in the number of people in the group is considered an "update" of the group, and in this case, more detailed insights can be obtained as knowledge.

[0144] When determining a treatment plan by predicting what will happen to a patient's condition, it is possible to recommend papers appropriate to the patient's profile, disease name, and symptoms (the paper's author may also be considered as a researcher or institution such as a university, hospital, or institution that should be accessed). In particular, medical image data that has been processed to be oriented in a unified manner taking into account the anatomical orthogonal orientation, a feature of this application, or medical image data organized based on anatomical features, can be easily linked to other research using the names of body parts included in anatomical terms. When such body parts are used as keywords for classification, it becomes possible to link information other than images.

[0145] Of course, when starting a prospective study, images organized with unified site information and orientation processing make it easier to compare data with other patients and other medical institutions. Furthermore, once such classifications are organized, it becomes possible to determine which cases are rare, for example, by determining whether they cannot be included in a predefined classification, or whether there is no data in a particular row or column when creating a table of cases and classifications. In other words, the rarity of test results can be determined. Because rare cases are difficult to collect even if one is aware of the need, the system and method described herein make it possible to effectively utilize data from patients who happen to visit a clinic for examination in research.

[0146] In S83, the calculation and control unit 51 records the results of the doctor's examination and treatment in the patient database. The calculation and control unit 51 then plans the next examination and treatment, presents it to the doctor, and records it in the patient database (S84). The doctor also refers to the examination results and papers to determine causal relationships, which serve as knowledge information for the case. The in-hospital system also searches the diagnosis results held by the in-hospital system and papers published outside the hospital, extracts information that can be determined to indicate a causal relationship, and displays it to the doctor via the display / output control unit. If no information that can be determined to indicate a causal relationship is extracted, the display / output control unit notifies the doctor that the case may be rare, thereby promoting research. The calculation and control unit 51 temporarily records the causal relationship information determined by the doctor in a memory (not shown) of the learning support unit 54 to accumulate knowledge information N. The learning support unit 54 can update the knowledge information N based on the temporarily recorded causal relationship information.

[0147] When multiple patient information with similar profiles is collected, the collected patient information can be organized by tagging and grouping. When multiple patients with similar profiles receive medical examinations, they can be classified into separate categories by gender or age group. The creation of new groups or the addition of new members is considered an "update" of a group, and in this case, more detailed knowledge is gained. This type of innovation can provide the basis for inferring that patients with certain symptoms have a certain disease from information on multiple patient groups with similar histories, or recommend retrospective studies that suggest that people with certain symptoms or lifestyles are more likely to develop a certain disease. If the daily log is lacking, additional information such as medical interview results can be added. Medical professionals can add this information, or patients and their associates can take the lead in completing a checklist on a smartphone screen in the form of a questionnaire, such as "drinks alcohol" or "parents are...". Of course, AI can also automatically create organized data sets from natural language texts of medical interviews and conversations, diagnostic images, and logs acquired during daily life. By linking with PCs and smartphones that potential patients use on a daily basis, the process of medical interviews is simplified and further information can be obtained in a "monitoring" manner.

[0148] As described above, in this embodiment, endoscopic images acquired during endoscope insertion can be used after the examination. Endoscopic images acquired during endoscope insertion, which have not received much attention because they represent areas not targeted for observation, can be automatically converted into text and a report can be generated from the endoscopic images to identify symptoms and other details. This allows lesions and other conditions to be easily identified, even when a doctor who is not a specialist in areas not targeted for observation diagnoses them, and also makes it easier to determine whether treatment or reexamination is necessary.

[0149] As explained above, when an endoscope with a cylindrical tip is inserted along a lumen, the image tends to rotate clockwise or counterclockwise without maintaining its vertical alignment. However, by aligning the image vertically based on the anatomical characteristics of the observation area, it is possible to obtain images similar to those published in academic papers. Images with such vertical alignment facilitate the detection of characteristic findings and symptoms. Furthermore, clear images can be captured through focusing, exposure control, and image processing. This facilitates diagnosis and increases the accuracy of diagnoses and assessments. Presenting the images to patients and their families has the advantage of providing a visually understandable explanation. This facilitates understanding and facilitates smooth examinations. Images with vertical alignment can be easily compared with previous images, which is useful for monitoring the progress of regular patients. Of course, the results of medical interviews can also be used as a reference. In particular, by linking with smartphones, the medical interview process can be simplified and further information can be obtained in a "supervisory" manner. The examples in this application are examples in which insertion, removal, and observation are handled separately to clearly outline the features of this application, but the intention is that in any situation, if valuable information is obtained, it can be effectively utilized, even if it is related to other medical departments.

[0150] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

Claims

1. An endoscopic device comprising: an endoscope including an imaging device at the tip of an insertion portion; and a processor, wherein the processor detects a specific site based on image features of an endoscopic image acquired by the imaging device when the endoscope is inserted during an endoscopic examination; controls the imaging device so as to bring the specific site into focus; determines a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image; and performs direction unification processing to unify the orientations of the endoscopic images acquired sequentially during the insertion of the specific site based on the anatomical position by continuously rotating the endoscopic images acquired sequentially during the insertion based on the directional relationship.

2. The endoscopic device according to claim 1, characterized in that the processor determines that the endoscopic images acquired by the imaging device to detect a specific area based on the image features are images captured during position change, rather than for observation, as the imaging device passes over the specific area.

3. The endoscopic device according to claim 1, characterized in that the processor displays the endoscopic image before the direction unification processing and the endoscopic image before the direction unification processing side by side on a display screen.

4. The endoscope device according to claim 1, characterized in that the processor detects as the specific part a part in which anatomical features of the human body can be detected based on image features of the endoscopic image.

5. The endoscope device according to claim 1, characterized in that the processor detects as the specific part a part from which an anatomical feature can be inferred from a part from which an anatomical feature of the human body can be detected based on the image characteristics of the endoscopic image.

6. The endoscope device according to claim 1, characterized in that the processor determines the directional relationship based on the shape of the lumen obtained from the image features of the endoscopic image.

7. The endoscope device according to claim 1, characterized in that the processor determines the directional relationship based on a change in image characteristics of the endoscope image.

8. The endoscope device according to claim 1, characterized in that the processor performs a trimming process on a portion corresponding to a specific site in the endoscope image that has been subjected to the direction unification process.

9. The endoscope device according to claim 1, characterized in that the processor performs at least one of image processing for focusing, exposure adjustment, illumination light control, and visibility measures on the portion corresponding to the specific portion.

10. The endoscope device according to claim 1, characterized in that the processor unifies the image portion of the vocal cords in the pharynx to the upper side on the endoscopic image and the image portion of the esophagus to the lower side on the endoscopic image by a direction unification process.

11. An image processing device comprising: a processor which detects a specific site based on image features of an endoscopic image acquired by an imaging device provided at the tip of the endoscope insertion portion when the endoscope is inserted during an endoscopic examination; determines a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image; and performs direction unification processing to unify the orientations of the endoscopic images obtained sequentially during the insertion with respect to the specific site based on the anatomical position by continuously rotating the endoscopic images obtained sequentially during the insertion based on the directional relationship.

12. The image processing device according to claim 11, characterized in that the processor detects, as the specific part, a part in which an anatomical feature of the human body can be detected based on the image features of the endoscopic image.

13. The image processing device according to claim 11, characterized in that the processor detects as the specific part a part from which an anatomical feature can be inferred from a part from which an anatomical feature of the human body can be detected based on the image features of the endoscopic image.

14. The image processing device according to claim 11, wherein the processor synthesizes a series of endoscopic images after the direction unification process has been performed.

15. An image processing method comprising the steps of: receiving data of temporally consecutive endoscopic images from an endoscope which acquires images using an imaging device provided at the tip of an insertion portion; detecting a specific area based on image features within the endoscopic images; rotating and correcting each of the temporally consecutive endoscopic images based on the image of the specific area within the endoscopic images; and associating the rotationally corrected images with text relating to symptoms of the specific area as the examination result.

16. An in-hospital system comprising: a receiving unit that receives data of endoscopic images that are consecutive in time from an endoscope that acquires images using an imaging device provided at the tip of an insertion portion; and a processor, wherein the processor detects a specific area based on image features within the endoscopic images, rotates and corrects each of the consecutive endoscopic images based on the image of the specific area within the endoscopic images, and associates the rotated and corrected images with text relating to symptoms of the specific area as examination results.

17. The in-hospital system according to claim 16, wherein the processor receives a daily log from a user terminal that acquires and stores the daily log, which is information related to the user's health, and enables the rotation correction based on the daily log.

18. An image processing method comprising the steps of: detecting a specific site based on image features of an endoscopic image acquired by an imaging device provided at the tip of an endoscope insertion portion when an endoscope is inserted during an endoscopic examination; determining a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image; and performing direction unification processing to unify the orientations of the endoscopic images obtained sequentially during the insertion of the specific site based on the anatomical position by continuously rotating the endoscopic images obtained sequentially during the insertion based on the directional relationship.

19. The image processing method according to claim 18, further comprising the step of: synthesizing the series of endoscopic images that have been subjected to the direction unification processing into a panoramic image.

20. An image processing program for causing a computer to execute the steps of: detecting a specific site based on image features of an endoscopic image acquired by an imaging device provided at the tip of an endoscope insertion portion when an endoscope is inserted in an endoscopic examination; determining a directional relationship between the up, down, left, and right directions of the imaging device and the anatomical position of the human body into which the endoscope is inserted based on the endoscopic image; and performing direction unification processing to unify the orientations of the endoscopic images acquired sequentially during the insertion of the specific site based on the anatomical position by continuously rotating the endoscopic images acquired sequentially during the insertion based on the directional relationship.

Images