A second endoscope system, a first endoscope system, a method for operating the second endoscope system, and a method for operating an endoscope device.

Abstract

The present invention provides a second endoscopic system, a first endoscopic system, and an inspection method by an endoscope, that enable easy access to a target portion such as an affected part. Provided is an endoscopic inspection method using a second endoscopic system on a subject whose organ has been inspected using a first endoscopic system to observe an organ to be observed of the subject, the method including: acquiring time-series operation content information in the first endoscopic system as operation unit information (S31); estimating an operation process of a case where the subject is inspected using the second endoscopic system; comparing the estimated operation process and the operation unit information; and outputting operation guide information for operating the second endoscopic system in order to observe a characteristic portion in the organ to be observed by the second endoscopic system (S37). The operation unit information is image change information estimated using the asymmetric property of the organ to be observed.

Description

Technical Field

[0001] The present invention relates to acquiring chronological operation content information from a first endoscope system, and based on this operation content information, a second endoscope system capable of performing operations for examination, the first endoscope system for this purpose, Operation method of the second endoscopy system and an endoscope Device operation and a method.

Background Art

[0002] Conventionally, it has been proposed to display a treatment procedure as a guide when performing treatment. For example, in Patent Document 1, in performing dental treatment, an imaging unit that captures an image of the oral cavity of a treatment subject, a storage unit that stores a plurality of product information and treatment procedure information for each use of each product, a treatment subject in the captured image is detected, an imaging image analysis unit that specifies a treatment step, a treatment procedure control unit that selects a treatment procedure corresponding to the treatment step, and a display control unit that displays the treatment procedure and the captured image side by side are provided. A dental treatment support device is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As described above, Patent Document 1 describes displaying a treatment procedure during treatment, and a dentist or the like can perform treatment according to the displayed treatment procedure. After performing dental treatment, when treating the affected part again, the target part of the treatment can be easily found. However, when reexamining the target part such as the affected part found during the first endoscopy or the like, it is not easy to reach the target part again.

[0005] For example, even if endoscopic examinations and treatments are performed by an endoscopy specialist (expert) at a specialized hospital (for example, using the first endoscopic system described later), it is inconvenient if the patient can only receive follow-up examinations or treatments at the same hospital. Therefore, it would be convenient, considering the burden on the patient, if follow-up examinations or treatments could be requested at another medical institution such as a clinic. However, in that case, it is necessary to locate the area again at the other medical institution. On the other hand, small hospitals such as clinics may not always be able to correctly observe the area requiring follow-up. Therefore, a system is desired that can support follow-up examinations and treatments at other medical institutions that have different equipment (for example, the second endoscopic system described later) and resources such as doctors.

[0006] This invention has been made in view of these circumstances, and provides a second endoscopic system, a first endoscopic system, that allows easy access to a target area such as an affected area. Operation method of the second endoscopy system and endoscopes Device operation The purpose is to provide a method. [Means for solving the problem]

[0007] To achieve the above objective, the second endoscope system according to the first invention is an endoscope system for observing the target organs of a subject who has undergone organ examination using the first endoscope system. The second endoscope system includes: an acquisition unit that acquires time-series operation content information of the first endoscope system as operation unit information; an insertion operation determination unit that estimates the operation process when the subject undergoes examination using the second endoscope system; and an operation guide unit that compares the operation process estimated by the insertion operation determination unit with the operation unit information and outputs operation guide information for operating the second endoscope system in order to observe characteristic parts of the target organs with the second endoscope system. The operation unit information is image change information estimated using the asymmetry of the target organs.

[0008] The second endoscopic system according to the second invention is characterized in that, in the first invention, the asymmetry information of the organ to be observed is determined based on the anatomical positional relationships of multiple parts within a specific organ. The second endoscope system according to the third invention is the first invention, wherein the operation unit information is information relating to an operation that continues over a predetermined period of time. The second endoscope system according to the fourth invention is the third invention described above, wherein the operation unit information is an operation start image and information regarding the operation from the start to the end of the operation.

[0009] The second endoscope system according to the fifth invention is characterized in that, in the first invention, the operation guide information output by the operation guide unit is guide information for observing the characteristic parts of the target organ under the same observation conditions as the first endoscope system. The second endoscope system according to the sixth invention is the first invention described above, wherein the operation unit information is image change information indicating a sequence of identical operations.

[0010] The second endoscopic system according to the seventh invention determines a first direction when detecting asymmetry of the organ to be observed, in the first invention described above. The second endoscopic system according to the eighth invention, in the first invention described above, refers to the direction in which fluid accumulates, determined by the direction of gravity, or the direction determined by the positional relationship of internal structures already detected, when detecting the asymmetry of the organ to be observed.

[0011] In the second endoscope system according to the ninth invention, the operation unit information is determined by reflecting the angle at which a lever or knob for rotating the tip of the endoscope system is turned, in the first invention described above. The second endoscope system according to the tenth invention is the first invention, wherein the operation unit information is information that defines the process until the observation direction of the tip of the endoscope system changes as an operation unit. The second endoscope system according to the 11th invention is the same as the 10th invention, wherein the observation direction of the tip of the endoscope system is changed by twisting the endoscope system, or by angling the endoscope system, or by pushing the endoscope system into the body.

[0012] The second endoscopic system according to the twelfth invention is, in the first invention, the operation unit information is information in which the process until the shape of the observed organ changes is used as the operation unit. The second endoscope system according to the 13th invention is an endoscope system in which, in the 12th invention, the operation unit information is information in which the operation unit is the process until the shape of the estimated organ changes by insufflating air and / or water and / or suction using the endoscope system, or by pushing the endoscope system in. The second endoscope system according to the 14th invention is, in the 12th invention, the operation unit information is information in which the operation unit is the process until the state of the mucous membrane of the estimated organ changes by spraying a dye and / or staining agent using the first endoscope system or by supplying water using the first endoscope system.

[0013] The second endoscope system according to the 15th invention, in the first invention, provides operational guide information for operating the second endoscope system in order to observe characteristic parts of the target organ, when comparing the operation process estimated by the insertion operation determination unit with the operation unit information, by comparing multiple operation unit information and weighing them during observation. multiple copies If the position does not require follow-up observation, the operation unit information is corrected to exclude the operation of the overlapping area and compared. The second endoscope system according to the 16th invention, in the first invention, observes characteristic parts of the target organ under the same observation conditions as the first endoscope system, based on the operation unit information described above.

[0014] The method for operating the second endoscope system according to the 17th invention is a method for operating the second endoscope system to examine the target organ of a subject who has undergone organ examination using the first endoscope system, wherein the second endoscope system comprises an acquisition unit, an insertion operation determination unit, and an operation guide. Department The second endoscope system is equipped with the following features: the acquisition unit acquires time-series operation information of the first endoscope system as operation unit information; the insertion operation determination unit estimates the operation process for the subject to undergo examination using the second endoscope system; the operation guide unit compares the estimated operation process with the operation unit information and outputs operation guide information for operating the second endoscope system to observe characteristic parts of the target organ; the operation unit information is estimated using the asymmetry of the target organ and is image change information.

[0015] The first endoscopic system according to the 18th invention includes an input unit that inputs images of a subject's organs in chronological order, an operation unit determination unit that divides the images of the organs acquired in chronological order into operation units and determines the operation performed for each operation unit, a recording unit that records information regarding the image and endoscopic operation in each operation unit determined by the operation unit determination unit as operation unit information, and an output unit that outputs the operation unit information recorded in the recording unit.

[0016] The first endoscope system according to the 19th invention is, in the above-mentioned 18th invention, the operation unit determination unit is , take I got a good deal. the above Based on the image, the operation is divided into the above-mentioned units based on whether or not at least one of the insertion direction, rotation direction, or bending direction of the tip of the first endoscope has changed. The first endoscope system according to the 20th invention is, in the above-mentioned 18th invention, the operation unit determination unit is , take I got a good deal. the above Based on the image, the direction of manipulation is determined based on the asymmetry of the anatomical structure. In the first endoscope system according to the 21st invention, in the 18th invention described above, the recording unit records start images and end images among consecutive images belonging to the operation unit, and also records operation information indicating the operation state in the operation unit. In the first endoscope system according to the 22nd invention, in the 18th invention described above, the recording unit records operation information after discovering an object serving as a landmark near the target.

[0017] In the operation method of an endoscope device according to the 23rd invention, the endoscope device includes an input unit, operation a unit determination unit, a recording unit, and an output unit. The operation method of the endoscope device is such that the input unit acquires images of the subject's organ in a time series, the operation unit determination unit divides the images of the organ acquired in the time series into operation units, determines the operations performed by the first endoscope for each operation unit, the recording unit records, for each determined operation unit, the image in the operation unit and information regarding the endoscope operation as operation unit information, and the output unit outputs the operation unit information recorded by the recording unit. The second endoscope system according to the 24th invention is a second endoscope system for observing the organ of a subject who has undergone an examination of the organ using the first endoscope system. For the subject who has undergone an examination using the first endoscope system, it has an input unit for inputting the recorded operation unit information, an imaging unit for acquiring images of the subject's organ in a time series, a unit that divides the images acquired in the time series into operation units, estimates the operation state of the second endoscope system for each operation unit, compares the estimated operation state with the operation unit information, and outputs guide information for observing under the same observation conditions as the first endoscope system, namely an operation guide unit.

[0018] The program according to the 25th invention causes a computer, which observes an organ to be observed of a subject using a second endoscope system, for a subject who has undergone an examination of an organ using a first endoscope system, to acquire chronological operation content information in the first endoscope system as operation unit information, estimate an operation process when the subject undergoes an examination using the second endoscope system, compare the estimated operation process with the operation unit information, and output guide information for observing a feature site in the organ to be observed under the same observation conditions as those of the first endoscope system. The operation unit information is image change information indicating a continuity of the same operation estimated using the asymmetry of the organ to be observed. The program according to the 26th invention causes a computer to acquire images of a subject's organ chronologically, divide the chronologically acquired images of the organ into operation units, determine an operation performed by a first endoscope for each operation unit, record, for each determined operation unit, information on the image and the endoscope operation in the operation unit in a recording unit as operation unit information, and output the operation unit information recorded in the recording unit.

Advantages of the Invention

[0019] According to the present invention, a second endoscope system, a first endoscope system, Operation method of the second endoscopy system and an endoscope Device operation method are provided that enable easy access to a target site such as an affected part.

Brief Description of the Drawings

[0020] [Figure 1A] FIG. is a block diagram mainly showing the electrical configuration of an endoscope system according to an embodiment of the present invention. [Figure 1B] FIG. is a block diagram mainly showing the electrical configuration of an endoscope system according to an embodiment of the present invention. [Figure 2] FIG. is a diagram showing an example of a path until reaching an object serving as a mark such as an affected part in an endoscope system according to an embodiment of the present invention. [Figure 3] This is a flowchart showing the operation of the first endoscope system in an endoscope system according to one embodiment of the present invention. [Figure 4] This is a flowchart showing the operation of a second endoscope system in an endoscope system according to one embodiment of the present invention. [Figure 5] This figure shows the insertion process of an endoscope in an endoscope system according to one embodiment of the present invention. [Figure 6] This figure shows the insertion process of an endoscope in an endoscope system according to one embodiment of the present invention. [Figure 7] This figure shows an example of an image captured during endoscope insertion in an endoscope system according to one embodiment of the present invention. [Figure 8] This figure shows an example of an image captured during endoscope insertion in an endoscope system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0021] The following describes an example in which the present invention is applied to an endoscope system as one embodiment of the present invention. This endoscope system is designed to replicate the examinations, diagnoses, and treatments performed by endoscopists (in this specification, examinations, diagnoses, and treatments may be collectively referred to as examinations or examinations, etc.), and is capable of recording and transmitting information that serves as a clue to reproduce the examinations, etc. Furthermore, in order to reproduce the examinations, etc. performed by endoscopists, images during the examination are recorded, and image changes and the image features captured are utilized.

[0022] Specifically, the operator's status is determined based on the following image changes: (1) If the pattern of changes in the examination image is constant (for example, like driving a car through a tunnel), it is determined that the endoscope is moving in a straight line; and (2) if the pattern of changes is unusual, it is determined that rotation or twisting has been applied to the endoscope. Furthermore, since it is not easy to determine up, down, left, and right in endoscopic images, these are defined using the structure of the organ being observed with the endoscope (for example, from the perspective of the endoscope, the vocal cord side is down in the pharynx, and the gastric angle side is up in the stomach). In this specification, the position (direction) in the endoscopic image is indicated using anatomical orientation (anatomical orientation will be explained later using Figure 5). Using this information as a guide, even non-endoscopic specialists can reproduce ideal examinations.

[0023] Figure 2 illustrates how a non-specialist can reach the target site, such as the affected area, based on records of examinations performed by an endoscopist. In Figure 2, Tg is the target site, such as the affected area, discovered by the specialist, and the non-specialist manipulates the endoscope to reach this target site Tg. Ob is a landmark object along the route to the target site Tg. Once this landmark Ob is found, it is used as a clue to locate the target site Tg. Although only one target site is depicted in Figure 2, there may be multiple target sites.

[0024] To specifically explain the procedure for finding the target Tg site, as shown in Figure 2, the specialist operates the first endoscopic system to advance it straight along route R1, and at position L1, the endoscope is bent and rotated to change the direction of advancement. After this, the endoscope is further advanced along route R2, and at position L2, the endoscope is bent and rotated to change the direction of advancement to route R3. As the endoscope continues in this state, an image of landmark Ob is captured at position L3, and at position L4, the route is changed to R4, and finally the target Tg site such as the affected area is found.

[0025] At this time, the images acquired by the first endoscope system and the operating state at that time are recorded in units of operation. That is, the images up to the point where the image changes due to the insertion direction of the tip of the first endoscope system, as well as rotation, bending, etc., are recorded as units of operation. In the example in Figure 2, the section from the start of the operation along route R1 to position L1 is one unit of operation, the section from position L1 along route R2 to position L2 is one unit of operation, the section from position L2 along route R3 to marker Ob at position L3 is one unit of operation, the section from position L3 along route R4 to position L4 is one unit of operation, and the section from position L4 along route R5 to target site Tg is one unit of operation. In Figure 2, for simplicity, the explanation is given using up, down, left, and right movement in two dimensions. However, in reality, movement occurs within a three-dimensional structure, so operations involving screen rotation and operations that change the observation position up, down, left, and right should also be considered.

[0026] As mentioned above, if a specialist records images and information such as the operating status until they reach the target Tg site (transient gland) of the affected area in the recording unit, an operating guide (operational advice) can be generated based on this information. A non-specialist can easily reach the target Tg site by operating the second endoscopic system while receiving the operating guide. In other words, when a non-specialist uses the second endoscopic system to perform an examination on the same patient (subject) who was examined by a specialist, they can easily reach the target Tg site shown in Figure 2 by receiving the operating guide.

[0027] For example, when a non-specialist inserts the tip of the second endoscope system into the patient's (subject's) body and reaches position L1, operation guide information such as rotation and bending will be displayed. Subsequently, operation guide information will be displayed at positions L2, L3 (position of landmark Ob), L4, etc. Reference guide information will also be displayed at positions other than these. When displaying guides indicating the direction of operation, such as upward, downward, rightward, and leftward, at each position, this specification uses the anatomical orientation to indicate the direction within the image.

[0028] Furthermore, when checking only the areas to be monitored, it may be inefficient to reflect the entire process of the examination performed during the previous consultation / examination as a unit of operation. For example, if the observation was performed in the order A→B, B→C, C→B, B→D during the previous consultation / examination, that is, if the observation was performed in the order of A, B, C, then returning to B, and then D, and it is determined that C does not need to be monitored in subsequent consultations / examinations, then the guidance should be performed in the order A→B, B→D. In other words, the guidance can be performed while omitting "B→C, C→B". For example, when viewing the lesser curvature after gastric insertion, there are cases where the greater curvature is viewed from the cardia before going to the lesser curvature. If the area to be ultimately monitored is the lesser curvature, the part where the specialist (expert) viewed the greater curvature should be omitted, and only the movement from the cardia to the lesser curvature should be recorded as a unit of operation (i.e., the movement from cardia insertion to the greater curvature and back to the vicinity of cardia insertion should be omitted).

[0029] Thus, to address situations where, as a result of observation, a target area for follow-up becomes unnecessary, the possibility of omitting the redundant portion of the operation can be determined. If omission is possible, the area that is no longer a target area for observation should not be guided. This omission of redundant operations is possible because it is broken down into operational units. If there is an identification signal for each point indicating whether it is a target for follow-up or not, it is possible to narrow down the operational units for guidance according to this identification signal and simplify the operational guide.

[0030] The possibility of omitting the aforementioned redundant operations and the "omission of operations" where redundant operations are omitted when possible will be explained using Figure 2. For example, suppose in the previous inspection, the process involved going from position L1, past position L2, to a further position (e.g., L2a), observing at that position, then returning to position L2, and subsequently observing in the order of L3 (marker Ob), L4, and Tg (target object). If, in this previous inspection, it was determined that follow-up observation at position L2a was unnecessary, then in subsequent follow-up observations, the operation guide from position L2 to position L2a may be omitted. In this case, in subsequent follow-up observations, the operation guide should be changed in the order of position L1 → position L2 → position L3.

[0031] If the operation unit information remaining in the history allows for the resetting of specific guide starting points, such as "move forward and then back" or "look right and then left," then multiple operation unit information included in the history may be corrected to generate operation unit information for the guide with the guide starting points reset, and this operation unit information may be made available for reference for the guide. In other words, to create operation guide information for endoscopy operations during re-examination or follow-up to observe characteristic parts of the target organ, the operation process estimated by the insertion operation determination unit is compared with the operation unit information. At this time, multiple temporally adjacent operation unit information from the set of operation unit information obtained during the previous examination is compared, and if there is no need for follow-up observation of overlapping parts during observation, the operation unit information may be corrected to exclude the operation of those overlapping parts and then compared. New data equivalent to the operation unit information for comparison may be created by strictly correcting the operation unit information for the guide, or the guide may be generated in anticipation without having to create new data.

[0032] The method of indicating direction within an image based on anatomical orientation will be explained using Figure 5. The tip of the endoscope is cylindrical, and the imaging unit is located inside it. Furthermore, images taken of the inside of the digestive tract, which has a complex shape, make it difficult to determine which direction is up or down, or right or left. In other words, while it is possible to understand which direction is up or down, and forward or backward, from the image in general landscape or portrait photographs, it is difficult to determine direction from images of the inside of the digestive tract, and some definition is needed to represent direction.

[0033] Therefore, anatomical position is used to indicate direction. Anatomical position is the posture of standing straight with palms facing forward (towards the direction the face is pointing), and directions are expressed based on this anatomical position. This premise is particularly useful when describing parts that change direction easily, such as the limbs. However, even with anatomical position as a premise, expressing the direction of limbs and the brain can still be confusing, so clearer expressions, such as those described later, are preferred.

[0034] In anatomical orientation, superiority is defined as the direction of the head, and inferiority as the direction of the feet. Left and right are defined from the perspective of the observer. That is, if a doctor is facing a patient, the left half of the patient's body is to the doctor's right. If a doctor is observing the patient's back, the right half of the patient's body is to the doctor's right. Anterior is defined as the direction the face is facing, and posterior is the direction the back is facing.

[0035] Figure 5 shows the direction according to the anatomically correct position. Note that when performing a gastric endoscopy, the entire body is actually turned to the side (left lateral decubitus position), but in Figure 5, for the sake of drawing convenience, the head is depicted to the side (left-facing), and the body from the neck down is depicted facing forward. In the example shown in Figure 5, the insertion route of the endoscope Ro is shown by a dashed line. The tip of the endoscope is inserted through the oral cavity (OC) (depending on the endoscope model, it may also be inserted through the nasal cavity (NC)), passes through the esophagus (ES), and proceeds to the stomach (St). Before passing through the esophagus (ES), there is a branching point with the vocal cords (Vc) and trachea (Tr), and if the endoscope mistakenly proceeds towards the vocal cords (Vc), it will not be able to reach the target site.

[0036] Image P5A in Figure 5 shows the tip of the endoscope just before it enters the esophagus (ES) after insertion through the oral cavity (OC). In image P5A, the lower side of the screen, which is anterior in the anatomical position, is the vocal cords (VC) and trachea (Tr), while the upper side of the screen, which is posterior in the anatomical position, is the esophagus (ES). To insert the tip of the endoscope toward the stomach, in image P5A of Figure 5, it is necessary to advance towards the esophagus (ES) in the direction of the upper part of the screen, which is posterior in the anatomical position. In fact, in the image just before the endoscope enters the esophagus, it is difficult to tell which is up and which is down, so the direction is indicated according to the anatomical position. In this case, there is asymmetry in the organs, and this asymmetry can be used to determine the direction of continuous movement. This point will be explained using Figures 6 and 7.

[0037] Furthermore, in Figure 5, when the tip of the endoscope is advancing towards the duodenum, it will proceed towards the pylorus (Py). When advancing from the stomach (St) to the pylorus (Py), the tip of the endoscope should be advanced along the wall of the stomach (St), and the direction of the endoscope tip should be changed so that the pylorus (Py) is visible. Image P5B in Figure 5 is a side view of the pylorus (Py). Once this image P5B is visible, the tip of the endoscope should be bent to change its direction.

[0038] Thus, when a non-specialist operates the second endoscopic system to reach the target site Tg of the affected area, it is not easy because the tip of the endoscope must be directed in the appropriate direction at branching points, etc. In this embodiment, the information obtained when a specialist operates the first endoscopic system 10A to reach the target site Tg of the affected area is recorded, and when a non-specialist performs an examination on the same patient, operation guide information is displayed based on the recorded information. By operating according to the operation guide information, the non-specialist can easily find the target site of the affected area discovered by the specialist.

[0039] Next, using Figures 6(a)(b) and 7, we will explain the operating procedure when the tip of the endoscope is inserted into the body cavity and the sequence of images obtained at that time. Figure 6 shows the insertion of the endoscope EDS through the patient's oral cavity (OC), passing through the patient's stomach (St), and examining the pylorus (Py). Note that, as with Figure 5, for the sake of drawing convenience, Figure 6(a) depicts the head as being sideways (facing left) and the body from the neck down as facing forward.

[0040] Figure 6(a) shows the insertion of the endoscope EDS into the oral cavity (OC), and Figure 6(b) shows the insertion of the endoscope EDS into the esophagus (ES) and stomach (St). Figure 7 shows images P6a to P6f acquired when the endoscope EDS is inserted into the digestive tract, and these images change moment by moment. Images P6a to P6c show the insertion of the endoscope EDS into the esophagus (ES) at times T1 to T3, during which the tip of the endoscope EDS is advancing straight without rotation or bending. As a result, the elliptical shape (hole-like) of the esophagus (ES) gradually becomes larger.

[0041] Images P6a to P6c are from the first operating unit. At time T4, the endoscope EDS is rotated, and the image acquired at this time shows the rotation of the elliptical (hole-shaped) projection. Images P6c to P6d are from the second operating unit. From time T3 to time T4, the tip of the endoscope EDS rotates in order to search for the pylorus Py within the gastric portion St. That is, when the tip of the endoscope EDS moves a predetermined distance downward along the wall of the gastric portion St, it reaches the vicinity of the pylorus Py, so at this timing the tip of the endoscope EDS is rotated to find the pylorus Py. Once the pylorus Py is found, it can be advanced into the duodenum.

[0042] At time T5, the endoscope EDS is bent, and the image acquired at this time shows the central part of the elliptical (hole-like) shape moving. Images P6e to P6f are images from the third operation unit.

[0043] Thus, organs possess asymmetry, and this asymmetry can be used to determine the continuity of movement. A series of images showing the sequence of movements until the continuity of movement is interrupted is used as the unit of operation. A tubular endoscope involves movements such as insertion, withdrawal, bending (in four directions) of the tip, and twisting. Moreover, these movements involve several components, making it difficult to analyze them as units of operation without utilizing asymmetry information. For example, the inside of a pipe is a symmetrical space and not asymmetry, so even if it twists during insertion, it cannot be broken down into units of operation. However, the inside of internal organs is an asymmetrical space. In other words, unit of operation information is image change information that indicates the continuity of the same movement, estimated using the asymmetry of the observed organ.

[0044] Specifically, in Figure 7, as shown at times T1 to T3, when the shape of the organ is similar and its size gradually changes, it represents an image change when the tip of the endoscope is inserted. Also, as shown at times T3 to T4, when the shape is the same (same size) but the position of protrusions, etc., rotates, it represents an image when the tip of the endoscope is rotated. Furthermore, as shown at times T5 to T6, when the shape is the same (same size) but its center position moves, it represents an image when the tip of the endoscope is bent. By utilizing the asymmetry of the internal organs, it is possible to determine whether the same action is continuous or not. In other words, the operation unit information is image change information that indicates a continuation of the same action estimated using the asymmetry of the observed organ.

[0045] In Figure 7, the operations of straight-line movement, rotation, and bending are shown individually. However, in reality, multiple operations may be performed in combination, and in this case as well, by utilizing the asymmetry of the internal organs, the operations can be broken down into their individual components and the operation information obtained. In other words, by using the approach explained in Figure 7, it is possible to make a decomposed determination (including direction and amount) of each operation mode, such as insertion direction, withdrawal direction, twisting direction, and tip bending direction, and to separate and determine what kind of combined operation modes are being performed at that time. In the future, it is possible that endoscopes capable of bending in directions other than the tip, or endoscopes with specifications that allow bending in directions other than up, down, left, and right, as well as tip control that has a function similar to a zoom lens and can move the tip closer or further away, may be developed, and it goes without saying that the same approach can be applied in such cases as well.

[0046] Furthermore, information on operational units does not need to be limited to operations related to changes in the observation position of the endoscope tip. In other words, it is desirable to pass on information, transfer techniques, and communicate procedures that improve observation conditions, visibility, and detectability by changing the state of the target area or any obstructions during endoscopic observation. Failure to perform these procedures may result in situations where it becomes impossible to compare images with previous examinations during follow-up. The following procedures are commonly performed when observing target areas using an endoscopic system and should be considered as operational units. For example, dyes or stains may be sprayed to make the shape of the target area, such as irregularities, and the difference between lesions and normal areas more visible. Also, water may be used with the endoscopic system (for the purpose of washing away mucus, etc.) to improve visibility. In other words, active interventions other than positional changes can alter the state of the mucosa of the suspected organ, and it is important to consider the process leading up to the change in the state of the organ's mucosa as an operational unit.

[0047] As mentioned above, the unit of operation information is image information that shows a sequence of identical actions. Here, identical actions include actions such as simply inserting a certain amount, twisting and rotating a certain amount, or turning a knob a certain amount to bend the tip (in Figure 7, "Insertion direction, rotation, tip bending"). These are operations that can be divided into time segments so that different actions do not get mixed up, and are assumed to be simplified operations performed within a time range that does not include other operations. However, classifying "identical operations" into too short time segments may result in overly subdivided and confusing operation instructions. Also, even a single unit of operation such as insertion, if it continues for several minutes, can become an unsettling guide during the operation. Therefore, it is preferable that identical actions be divided into time segments that make it easy for the operator of the second endoscope system to refer to the guide and perform the operation (for example, a few seconds to several tens of seconds). In reality, experienced specialists can use techniques such as twisting while inserting, so it may be good to provide clear guidance for that as well. In other words, it may be possible to divide the operation into two components, insertion and twisting, and provide time-division guidance for each. In situations like this, where one thing happens while doing another, the combination of operations can be separated and determined using the approach explained in Figure 7.

[0048] Figure 6, mentioned above, shows the route when inserting an endoscope (EDS) from the oral cavity (OC) towards the pylorus (Py). Figure 8 shows examples of images acquired by the endoscope (EDS) during this insertion. Image P11 is a top view of the vocal cords (VC), and image P12 is a top view of the esophagus (ES). Image P13 is an image taken when the endoscope enters the stomach (St), image P14 is a top view of the pylorus (Py), and image P15 is a side view of the pylorus (Py). As can be seen from these figures, internal organs are asymmetrical, not symmetrical. By utilizing this asymmetry, it is possible to estimate the transition of operations during continuous motion and determine the unit of operation based on the breaks in the operation.

[0049] In this embodiment, as will be described later, when a specialist operates the endoscope, the continuous images are divided into operation units, and operation unit information is recorded for each operation unit (see, for example, operation unit information 35b in Figure 1A and S11 in Figure 3). This operation unit information can be said to be image change information that indicates a sequence of identical movements estimated using the asymmetry of the organ being observed (see, for example, Figure 7). Furthermore, the asymmetry information of the organ being observed is determined based on the anatomical positional relationship of multiple parts within a specific organ. It may also be aligned with anatomical directional representation. In addition, the operation unit information is information about an operation that continues over a predetermined period of time. Furthermore, the operation unit information may include the operation start image and information about the operation from the start to the end. The operation unit information may also include the end image, and / or information that serves as a marker for finding the target site, and / or information about the target site, and / or operation information before discovery (see, for example, S41 and S48 in Figure 4).

[0050] Furthermore, when a specialist performs an examination using an endoscope, this is done by operating a lever or knob or other control part for rotating the tip of the endoscope system, and when this operation is performed, the operation unit often changes from the first unit to the second unit. Therefore, the operation unit information may be determined by reflecting the angle at which the lever or knob for rotating the tip of the endoscope system is turned. Alternatively, the operation unit information may be information that defines the process until the observation direction of the tip of the endoscope system changes as the unit of operation. The observation direction of the tip of the endoscope system may be changed by twisting the endoscope system, by adjusting the angle of the endoscope system, or by pushing the endoscope system into the body.

[0051] Furthermore, the unit of operation information may also be information that defines the process until the shape of the observed organ changes as the unit of operation. The unit of operation information may also be information that defines the process until the shape of the estimated organ changes as the unit of operation by insufflating air, water, or suction using the endoscopic system, or by pushing the endoscopic system in. The unit of operation information is information that defines the process until the state of the mucosa of the estimated organ changes as the unit of operation by spraying dyes or stains using the first endoscopic system, or by insufflating water (for the purpose of mucosal washing) using the first endoscopic system. In other words, the unit of operation information is not limited to operations related to changes in the observation position of the tip of the endoscope, but may also include operations that change the state of the target area or anything obstructing it during endoscopic observation to improve the observation state, visibility, and detectability.

[0052] Furthermore, as explained with reference to Figure 5, in this embodiment, the anatomical orientation is used to represent the direction. Therefore, the first direction may be determined when detecting the asymmetry of the organ under observation. Alternatively, when detecting the asymmetry of the organ under observation, the direction in which fluid accumulates, determined by the direction of gravity, or the direction determined by the positional relationship of internal structures already detected, may be referenced.

[0053] In this embodiment, when a specialist operates the endoscope, the operation unit information is recorded, and when a non-specialist operates the endoscope, an operation guide is displayed based on the operation unit information. In other words, the non-specialist observes the target organ by manually operating the endoscope based on the operation guide. However, the system is not limited to this, and characteristic parts of the target organ may be observed automatically under the same observation conditions as the first endoscope system based on the operation unit information.

[0054] Furthermore, in this embodiment, when a non-specialist performs an examination using the second endoscope system, guide information is output so that characteristic parts of the organ to be observed can be observed under the same observation conditions as when a specialist performs the examination (see, for example, S37 and S47 in Figure 4). Here, similar observation conditions include the size of the object captured in the image and the field of view, and are conditions to ensure that the positional relationship between the imaging unit and the object being observed is the same when observing the object. In addition, conditions may be set to allow for the determination of changes in color by controlling the illumination and exposure for the object being observed in the same way, and optical conditions such as focus and field of view (including up, down, left, right within the image, and the positions of the person who observed last time and the person observing this time) may also be set to be the same.

[0055] Next, the configuration of one embodiment of an endoscopic system to which the present invention is applied will be described using Figures 1A and 1B. This endoscopic system consists of a first endoscopic system and a second endoscopic system for observing the target organs of a subject (including a patient) who has undergone organ examination (including diagnosis and treatment) using the first endoscopic system. Specifically, the endoscopic system according to this embodiment consists of an endoscopic system 10A, an auxiliary device 30 provided in the hospital system server, etc., and a second endoscopic system 10B. Here, for the sake of ease of explanation, the endoscopic system 10A and the second endoscopic system 10B are, for example, endoscopes inserted from the oral cavity through the esophagus to examine the stomach or duodenum, and will be described using the case of performing a gastric or duodenal endoscopic examination on a subject as an example. The endoscopic system 10A is described as an endoscope used for the subject's first examination, and the second endoscopic system 10B is described as an endoscope used for the subject's second and subsequent examinations. The endoscopic system 10A and the second endoscopic system 10B may be the same model of endoscope, but will be described as different models here.

[0056] Furthermore, if the second endoscope system 10B is the same model as the endoscope system 10A, it may be the same device or a different model / device. In other words, when examinations are performed at different times, the examinations may not be exactly the same due to changes in circumstances (including the patient's physical condition such as the affected area, changes in physical and mental constraints and capacity such as changes in doctors, fatigue or familiarity, or surrounding circumstances such as assistants, peripheral equipment, and the environment). Therefore, in this embodiment, it is sufficient if information can be transferred when multiple examinations are performed at different times (in most cases, the examination dates will be different, but it may also be possible to consider the possibility of immediate re-examination).

[0057] The endoscopic system 10A is used by physicians to observe the pharynx, esophagus, stomach, and duodenum for examinations, procedures, surgeries, etc. This endoscopic system 10A includes a control unit 11A, an imaging unit 12A, a light source unit 13A, a display unit 14A, an ID management unit 15A, a recording unit 16A, an operation unit 17A, an inference engine 18A, a clock unit 20A, and a communication unit 21A. Note that the above-mentioned parts may be housed in a single device, or they may be distributed across multiple devices.

[0058] The control unit 11A consists of one or more processors, each having a processing unit such as a CPU (Central Processing Unit), a memory for storing a program (the program may be stored in the recording unit 16A), and so on. It executes the program and controls each part of the endoscope system 10A. The CPU of the control unit 11A works in cooperation with the CPU of the control unit 31 of the auxiliary device 30 to execute the program and realize the flow operation shown in Figure 3. The control unit 11A performs various controls when the endoscope system 10A performs an endoscopic examination on a subject (patient), and also controls the transmission of image data P1 acquired during the examination to the auxiliary device 30 located in the hospital system or on a server.

[0059] The imaging unit 12A is located at the tip of the insertion section of the endoscope system 10A and includes an optical lens, image sensor, imaging circuit, image processing circuit, etc. The imaging unit 12A consists of a small image sensor and an imaging optical system that forms an image of the object on this image sensor, and it is assumed that the specifications such as the focus position and the focal length of the optical lens are predetermined. The imaging unit 12A may also be equipped with autofocus and depth of field expansion (EDOF function), in which case it is possible to determine the distance and size of the object. If the imaging unit 12A has a shooting angle of view of about 140 to 170 degrees, it can capture a wide area. The imaging optical system may have a zoom lens. The imaging unit 12A acquires video image data at predetermined time intervals determined by the frame rate, processes this image data, and then records it to the recording unit 16A. In addition, when the release button in the operation unit 17A is operated, the imaging unit 12A acquires still image data, and this still image data is recorded to the recording unit 16A. The imaging unit 12A functions as an imaging unit that acquires images of the subject's organs in a time series (see, for example, S1 in Figure 3).

[0060] Image P1 is an image acquired by the imaging unit 12A and is transmitted to the input unit 32 of the auxiliary device 30 via the communication unit 21A. Image P1 is a time-series image; image P11 is an image acquired immediately after the tip of the endoscope system 10A is inserted into the oral cavity, and image P20 is an image acquired immediately before the endoscope system 10A is withdrawn from the oral cavity. Images P11 to P16 are a series of images belonging to an operation unit. Similarly, images P15 to P19 are also images belonging to a different operation unit. An operation unit is a series of images until the image pattern changes due to operations such as changing the insertion direction, rotation, and bending the tip, until the specialist reaches the target site such as the affected area.

[0061] Note that while Figure 1A only shows two operational units, there may be three or more operational units depending on the examination content. Also, in Figure 1A, images P11-P16 represent the first operational unit, and images P15-P19 represent the second operational unit. In this example, images P15 and P16 overlap between the first and second operational units. However, images do not need to overlap between the two operational units, and images between the two operational units do not need to belong to an operational unit (the latter case corresponds to no operation being performed).

[0062] The light source unit 13A includes a light source and a light source control unit. The light source unit 13A illuminates the target object with appropriate brightness. The light source is positioned at the tip of the endoscope system 10A to illuminate the inside of the body, such as a diseased area, and the light source control unit controls the illumination by the light source. By adjusting the brightness and measuring the luminance of the reflected light, it is possible to measure the distance to the target object. It is also possible to irradiate the object with light consisting of a predetermined pattern (light consisting of bright and dark parts) and measure the distance and the surface irregularities of the object from the difference between the irradiated pattern and the imaged pattern. In this case, it is desirable to use light in a wavelength range other than visible light, such as infrared light.

[0063] The display unit 14A displays images of the inside of the body based on image data acquired by the imaging unit 12A. The display unit 14A can also display operation guides superimposed on the examination images. For example, it can indicate the vicinity of a specific area (affected area). These operation guides may be displayed based on the inference results of the inference engine 18A. Furthermore, menu screens for operation and display of the endoscopy system 10A can also be displayed.

[0064] The ID management unit 15A manages IDs to identify the subject (patient) when a specialist uses the endoscopy system 10A for examinations. For example, the specialist may input the subject's (patient's) ID through the operation unit 17A of the endoscopy system 10A. Alternatively, the ID management unit 15A may associate the ID with the image data acquired by the imaging unit 12A.

[0065] The recording unit 16A has electrically rewritable non-volatile memory and records adjustment values ​​for operating the endoscope system 10A, programs used in the control unit 11A, etc. It also records image data acquired by the imaging unit 12A. The operation unit 17A has various operation units, such as an operation unit (also called an interface) for bending the tip of the endoscope system 10A in any direction, an operation unit for the light source, an operation unit for image acquisition, and an operation unit for treatment instruments, etc. The subject (patient) ID may be entered through the operation unit 17A.

[0066] The inference model is located within the inference engine 18A. This inference model may consist of various inference models, such as an inference model that infers potential affected areas like tumors and polyps in images acquired by the imaging unit 12A, and an operation guide for operating the endoscope system 10A. The inference engine 18A may be composed of hardware, software (programs), or a combination of hardware and software.

[0067] The clock unit 20A has a calendar function and a timing function. When image data is acquired by the imaging unit 12A, it may output the date and time of acquisition, or the elapsed time from the start of the inspection. When recording image data in the recording unit 16A, this time information may also be recorded. Furthermore, when outputting image data from the communication unit 21A to the auxiliary device 30, this time information may be output in association with the image data. In addition, when acquiring images in operation units, the time information output from the clock unit 20A may be associated with the image data.

[0068] The communication unit 21A has a communication circuit (including a transmitting circuit and a receiving circuit) and exchanges information with the auxiliary device 30. That is, it transmits image data acquired by the imaging unit 12A to the auxiliary device 30. The communication unit 21A may also communicate information with the second endoscope 30 in addition to the auxiliary device 30. Furthermore, the communication unit 21A may communicate with other servers or hospital systems, in which case it can collect and provide information from other servers or hospital systems. It may also receive inference models generated by external learning devices.

[0069] The auxiliary device 30 is installed in the hospital system or server. The hospital system is connected by wired or wireless communication to equipment such as endoscopes, personal computers (PCs), and mobile devices such as smartphones within one or more hospitals. The server is connected to equipment such as endoscopes and the hospital system via a communication network such as the internet or an intranet. The endoscope system 10A may be connected to the auxiliary device 30 in the hospital system, or it may be connected directly to the auxiliary device 30 in the server, or it may be connected to the auxiliary device 30 through the hospital system.

[0070] The auxiliary device 30 includes a control unit 31, an input unit 32, an ID management unit 33, a communication unit 34, a recording unit 35, an inference engine 37 with a pre-configured inference model, and an operation unit determination unit 37. These units may be housed within a single device, or they may be distributed across multiple devices. Furthermore, these units may be connected via a communication network such as the Internet or an intranet.

[0071] The control unit 31 consists of one or more processors, each having a processing unit such as a CPU (Central Processing Unit), a memory that stores a program (the program may be stored in the recording unit 35), and so on. The control unit 31 executes the program and controls each part of the auxiliary device 30. The control unit 31 performs overall control of the auxiliary device 30 so that when a non-specialist performs the same or similar examination on the same patient using a second endoscopy system 10B (or endoscopy system 10A) after a specialist has performed an examination on the patient using the endoscopy system 10A, the control unit 31 can output an operation guide to locate the affected area of ​​the patient. The CPU of the control unit 31 of the auxiliary device 30 works in cooperation with the CPU of the control unit 11A to execute the program and realize the flow operation shown in Figure 3. In this embodiment, the CPU in the processor and the program stored in the memory realize functions such as the operation unit determination unit.

[0072] The input unit 32 has an input circuit (communication circuit) and receives the input image P1 acquired by the imaging unit 12A. Based on the input image P1 received by the input unit 32, the operation unit determination unit 37 determines the image group of the operation unit. This image group is output to the inference engine 37, which uses an inference model to infer operation information for reaching the location of a target area such as the affected area, and outputs operation information Iop. Operation information Iop includes operation information for operating the operation unit and the endoscopic image at that time. In this embodiment, operation information Iop is output by inference using an inference model, but operation information Iop may also be output based on image similarity determination. The input unit 32 functions as an input unit that inputs images of the subject's organs in chronological order (see, for example, S1 in Figure 3).

[0073] The ID management unit 33 manages the IDs of the subjects (patients). As mentioned above, when a specialist performs an examination using the endoscopy system 10A, the subject's (patient's) ID is entered, and an image P1 associated with this ID is transmitted from the endoscopy system 10A. The ID management unit 33 associates the ID associated with this image P1 with the subject's (patient's) ID information recorded in the recording unit 35, etc. Furthermore, when a non-specialist performs an examination using the second endoscopy system 10B, the necessary operation information Iop is output based on the ID information.

[0074] The communication unit 34 has a communication circuit and exchanges information with the endoscope system 10A and the second endoscope system 10B. The communication unit 34 may also communicate with other servers and hospital systems, in which case it can collect and provide information from other servers and hospital systems. Operation information Iop generated in the inference unit 36 ​​is transmitted to the second endoscope system 10B via the communication unit 34. In this case, operation information Iop corresponding to the ID of the patient being examined using the second endoscope system 10B is transmitted to the communication unit 21B of the second endoscope system 10B via the communication unit 34. The communication unit 34 functions as an output unit that outputs operation unit information recorded in the recording unit (see, for example, S23 in Figure 3).

[0075] The recording unit 35 has an electrically rewritable non-volatile memory and can record image data input from the imaging unit 12A to the input unit 32, information such as the patient's profile, examination history, and examination results, and programs used in the control unit 31. Furthermore, when the patient is examined using the endoscopy system 10A (which may include the second endoscopy system 10B), the recording unit 35 may record image data based on the image P1 at that time, and may also record operation information Iop inferred and output by the inference engine 37.

[0076] The recording unit 35 records the examination image 35a and the operation unit information 35b. As mentioned above, when a subject (patient) is examined using the endoscopic system 10A, the recording unit 35 records image data based on the image P1 at that time. This image data is recorded as the examination image 35a.

[0077] Operation unit information 35b is recorded for each ID of a subject (patient) undergoing an examination (including consultation and treatment) using the endoscopy system 10A. In this case, since one subject may undergo multiple examinations, it is advisable to distinguish them by examination date and time, etc. Furthermore, as explained using Figure 7, since there are multiple operation units in a single examination, operation unit information 35b is recorded for each operation unit, including the start image 35ba, end image 35bb, operation information 35bc, and time information 35bd.

[0078] The operation unit information 35b records the start image 35ba, end image 35bb, operation information 35bc, and time information 35bd. The start image 35ba is the first image belonging to the operation unit as determined by the operation unit determination unit 37. For example, in image group P1, image P12 is the start image belonging to the first operation unit, and image P15 is the start image belonging to the next operation unit. The end image 35bb is the last image belonging to the operation unit as determined by the operation unit determination unit 37. For example, in image group P1, image P16 is the end image belonging to the last operation unit, and image P19 is the end image belonging to the next operation unit. Note that image P11 is the image taken when the endoscope was inserted, and image P20 is the image taken when the endoscope was withdrawn.

[0079] The operation information 35bc is information relating to the operating status of the endoscope system 10A, and operation information is recorded for each image data and / or operation unit. Operation information may also be acquired based on changes in the image acquired by the imaging unit 12A. As mentioned above, by utilizing the asymmetry of the internal organs, it is possible to determine whether or not the same continuous operation is occurring, and the same continuous operation can be determined as a single operation unit. For example, when a specialist performs a straight-line operation, a rotation operation, or a bending operation on the tip of the endoscope system 10A, the image changes according to the operation. The image also changes when operations such as water injection or suction are performed. In response to these image changes, the control unit 31, etc., acquires operation information and records it as operation information 35bc. In addition to acquiring operation information based on images, if, for example, operation information performed by the operation unit 17A in the endoscope system 10A is transmitted to the auxiliary device 30 in association with image data, this associated operation information may also be acquired.

[0080] The time information 35bd is the time information for each individual image of the operation unit. For example, the time information may indicate the year, month, day, hour, minute, and second at which the image was acquired. Alternatively, the start of the operation may be used as the reference time, and the elapsed time from this reference time may be used as the time information.

[0081] Furthermore, as part of the operation unit information 35b, an object is designated near the target area, such as the affected area, to serve as a landmark for the target area (see landmark Ob in Figure 2), and an image of this landmark Ob (which may also include positional information) is recorded (see S17 in Figure 3). In addition, an image of the target area Tg (which may also include positional information) is recorded as part of the operation unit information 35b. Furthermore, information on the operations performed by the specialist from landmark discovery to targeting is also recorded in the recording unit 35 as part of the operation unit information 35b (see S19 in Figure 3).

[0082] The recording unit 35 functions as a recording unit that records information regarding images and endoscopic operations in each operation unit determined by the operation unit determination unit as operation unit information (see, for example, S11 in Figure 3). The recording unit records the start and end images in the sequence of images belonging to the operation unit, as well as operation information indicating the operation state in the operation unit (see, for example, S11 in Figure 3). The recording unit records operation information after the discovery of a landmark near the target (see, for example, S17 and S19 in Figure 3).

[0083] The operation unit determination unit 36 ​​determines whether to divide the images into operation units based on the images input chronologically by the input unit 32 (see, for example, S7 and S11 in Figure 3). In other words, it determines, based on the images, whether they represent images of the same operation or action being repeated. For example, suppose a specialist advances the tip of an endoscope in a straight line, then bends the tip while advancing, and then advances it in a straight line again after a while. In this case, the images taken during the forward movement up to the bending operation become one operation unit, and then the images taken from the bending operation until the straight forward movement becomes another operation unit. Note that a specialist may not perform only one operation, but may perform a combination of operations. For example, they may perform bending and rotation operations while moving forward. In such cases, it may be better to distinguish between simple and complex operations, or to distinguish between them separately, so the determination should be made according to the operation state, including the timing of the start and end of the operation.

[0084] Furthermore, the operation unit determination unit 36 ​​determines the direction of operation based on the asymmetry of the anatomical structure in the image acquired by the imaging unit (see, for example, S13, S15 in Figure 3). As mentioned above, it is not easy to express the direction in which the tip of the endoscope is facing, such as forward, backward, right, or left, within a body cavity (see, for example, Figure 5). Therefore, in this embodiment, the direction of operation is determined based on the asymmetry of the anatomical structure.

[0085] Furthermore, the determination of the operation unit may be made not only based on the image, but also based on information such as operation information attached to the image data, or based on both the image and operation information. In addition, sensors may be provided at the tip and / or insertion and / or operation parts of the endoscope, and operation information may be acquired based on the output from these sensors. For example, if a sensor is provided at the tubular section, the shape of the scope can be recognized, and as a result, situations such as pressing on the greater curvature of the stomach can be grasped more accurately. Also, in order to detect button operations (since in the case of air insufflation only involves closing a hole) and angle operations with greater accuracy, a sensor may be mounted at the operation part. In addition, a transmission source may be provided at the tip of the endoscope, and a sensor may be provided to detect the signal from the transmission source outside the body, and operation information may be acquired based on the output from this sensor. The operation unit information determined by this operation unit determination unit 36 ​​is output to the inference engine 37.

[0086] The operation unit determination unit 36 ​​may include hardware circuits for performing the above determination, or it may be implemented using software. Alternatively, the control unit 31 may also perform this function. In other words, the determination may be performed by the hardware circuits of the control unit 31 and / or by software on the CPU. Furthermore, the operation unit determination unit 36 ​​may include an inference model and determine the operation unit through inference.

[0087] The operation unit determination unit 36 ​​divides the images of organs acquired in chronological order into operation units and functions as an operation unit determination unit that determines the operation performed for each operation unit (see, for example, S7 and S11 in Figure 3). The operation unit determination unit also divides the images acquired by the imaging unit into operation units based on whether or not at least one of the insertion direction, rotation direction, or bending direction of the tip of the first endoscope has changed (see, for example, S7 and Figure 7 in Figure 3). The operation unit determination unit determines the direction of the operation based on the asymmetry of the anatomical structure in the images acquired by the imaging unit (see, for example, S13 in Figure 3, and P5A and P5B in Figure 5).

[0088] The inference engine 37 may be composed of hardware, software (program), or a combination of hardware and software. An inference model is set in this inference engine 37. In this embodiment, the inference engine 37 is located within the auxiliary device 30, but it may also be located in equipment such as an endoscope, and inference may be performed within the equipment.

[0089] The inference engine 37, equipped with an inference model, performs inference when it receives image data of image P1 as input to its input layer, and outputs operational information Iop related to endoscopic operation from its output layer. This operational information Iop is information that displays operational guidance (operational advice) to enable a non-specialist to perform operations equivalent to those performed by a specialist when inserting the second endoscopic system 10B into a body cavity, and to reach the target area such as the affected area. In other words, it includes images of the operation units acquired when the specialist performed the operation, and information indicating the operational status of the operating part at that time. Note that it is not necessary to include all images and information indicating the operational status; it is sufficient to include images and information that represent the key points of the operation.

[0090] The inference engine 37 may generate an inference model for displaying an operational guide to reach a target area such as the affected area, using time-series images (including operational information in Figure 1A) obtained from an examination performed by a specialist using the endoscopic system 10A. To generate this inference model, training data based on numerous time-series images is input to the input layer of the inference engine 37. Figure 1A shows example image groups P2 and P3, but many other image groups are also input.

[0091] In image group P2, similar to image group P1, for example, image P22 is the starting image belonging to the first operation unit, image P25 is the last image belonging to this operation unit, image P26 is the first image belonging to the next operation unit, and image P29 is the last image belonging to this operation unit. Also, image P21 is the image at the time of insertion in the series of time-series images, and image P30 is the image at the time of extraction. In image group P3, similar to image group P1, image P32 is the starting image belonging to the first operation unit, image P35 is the last image belonging to this operation unit, image P36 is the first image belonging to the next operation unit, and image P39 is the last image belonging to this operation unit. Also, image P31 is the image at the time of insertion in the series of time-series images, and image P40 is the image at the time of extraction.

[0092] Furthermore, in image group P2, operation information is assigned in the same manner as determined by the operation unit determination unit 36, with information Isa indicating that it is the same image and information Idi indicating that it is a different image. Similarly, in image group P3, operation information is assigned in the same manner as determined by the operation unit determination unit 36, with information Isa indicating that it is the same image and information Idi indicating that it is a different image.

[0093] By using numerous images, such as image sets P2 and P3, as training data and performing machine learning such as deep learning using this training data, an inference model for operation guidance can be generated. Here, we will explain deep learning. "Deep learning" is a multi-layered structure of the "machine learning" process using neural networks. A typical example is the "feedback neural network," which sends information from front to back to make a judgment. In its simplest form, a feedback neural network only requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons given by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be classified. Each neuron in the input layer and the hidden layer, and the hidden layer and the output layer, are connected by connection weights, and a bias value is added between the hidden layer and the output layer, making it easy to form a logic gate.

[0094] While a three-layer neural network is sufficient for simple classification, adding numerous hidden layers allows for learning how to combine multiple features during the machine learning process. In recent years, networks with 9 to 152 layers have become practical in terms of training time, classification accuracy, and energy consumption. Alternatively, one can use a "convolutional neural network," which compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, for handling more complex information and analyzing information where meaning changes depending on the order, one can use a "recurrent neural network" (fully connected recurrent neural network), where information flows bidirectionally.

[0095] To implement these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays) may be used. However, since much of the processing in neural networks involves matrix multiplication, processors specialized for matrix calculations, such as GPUs (Graphics Processing Units) and Tensor Processing Units (TPUs), may also be used. In recent years, such artificial intelligence (AI) dedicated hardware, called "Neural Network Processing Units (NPUs)," have been designed to be integrated and embedded together with CPUs and other circuits, and are sometimes included as part of the processing circuit.

[0096] Other machine learning methods include, for example, support vector machines and support vector regression. Learning in these methods involves calculating classifier weights, filter coefficients, and offsets, and there are also methods that utilize logistic regression. When a machine is to make a judgment, a human needs to teach it how to make that judgment. In this embodiment, a method of deriving image judgment through machine learning is employed, but other rule-based methods that apply rules acquired by humans through empirical rules and heuristics may also be used.

[0097] The second endoscopic system 10B shown in Figure 1B is, as mentioned above, an endoscope used when a patient undergoes an examination or other procedure by a non-specialist after a specialist has performed an examination or other procedure using the endoscopic system 10A. The second endoscopic system 10B may be the same model as the endoscopic system 10A, or even the exact same equipment, but in this embodiment, it is shown as a different model of endoscope. The second endoscopic system 10B functions as a second endoscopic system for observing the target organs of a patient who has undergone an organ examination (including diagnosis and treatment) using the first endoscopic system.

[0098] The auxiliary device 30 outputs a set of operation assistance images P4 to the second endoscopy system 10B for providing operational guidance during re-examinations performed by non-specialists. The set of operation assistance images P4 for re-examinations consists of time-series images from insertion into the body cavity to image P43, which corresponds to the location of the target area such as the affected area, when a re-examination is performed using the second endoscopy system 10B. The set of operation assistance images P4 for re-examinations may be created based on images P11 to P20, etc., from the images P1 acquired during the first examination. Image P43 in the set of operation assistance images P4 for re-examinations contains operational information Iop, which is the result of inference by the inference engine 36, and may display guidance such as "Perform this operation."

[0099] Although Figure 1B only shows image P43, which corresponds to the target area, if know-how regarding angle, distance, up / down / left / right, lighting, etc. is useful when accessing the target area, an image of a landmark Ob (image at position L3 in Figure 2) may be displayed in front of it. In this case, the system may stop briefly before the landmark and provide more detailed guidance on how to access the target area from that position. The example shown in Figure 2 is a case where an easily identifiable location (position L3) is used as a landmark (e.g., the pylorus), and the view is taken by bending the camera from there (the target), with image P43 corresponding to the target area Tg. If the target area Tg can be easily observed without displaying the image of landmark Ob, then the image of landmark Ob is unnecessary, and only image P43 needs to be displayed. Therefore, the operation assistance image group P4 may display both the image of landmark Ob and the image P43 corresponding to the goal Tg, or it may display only the image P43 of the target area.

[0100] The second endoscope system 10B includes a control unit 11B, an imaging unit 12B, a light source unit 13B, a display unit 14B, an ID management unit 15B, a recording unit 16B, and an operation unit 17B. These are the same as the control unit 11A, imaging unit 12A, light source unit 13A, display unit 14A, ID management unit 15A, recording unit 16A, and operation unit 17A of the endoscope system 10A. Therefore, the additional configurations and functions of the second endoscope system 10B are described supplementarily, and a detailed explanation is omitted.

[0101] The control unit 11B consists of one or more processors, each having a processing unit such as a CPU (Central Processing Unit), a memory that stores a program (the program may be stored in the recording unit 16B), and so on. The control unit 11B executes the program and controls each part of the second endoscope system 10B. The control unit 11B performs various controls when the endoscope system 10B performs a re-examination of the subject (patient). The CPU of the control unit 11B executes the program stored in the recording unit 16B, etc., and realizes the operation of the flow shown in Figure 5. In this embodiment, the CPU and the program stored in the memory within the processor realize functions such as the acquisition unit, operation determination unit, and operation guide unit.

[0102] Furthermore, the control unit 11B uses the images acquired by the imaging unit 12B during the re-examination and the set of operation assistance images P4 output from the auxiliary device 30 to cause the guide unit 19B to execute an operation guide to reach the target area such as the affected area. To create the operation guide and determine whether the tip of the second endoscope 19B is near an object for searching the target area such as the affected area, the guide unit 19B may perform inference using an inference model, or a similar image determination unit 23B, described later, may perform similar image determination. The operation guide created by the control unit 11B is displayed on the display unit 14B, and the display unit 14B may also indicate that the tip of the endoscope is near an object or target area.

[0103] As mentioned above, the imaging unit 12B is the same as the imaging unit 12A, so a detailed explanation will be omitted. However, this imaging unit 12B functions as an imaging unit that acquires images of the subject's organs in a time series (see, for example, S33 in Figure 4).

[0104] The communication unit 21B has a communication circuit (including a transmitting circuit and a receiving circuit) and exchanges information with the auxiliary device 30. For example, it receives operation information Iop output from the auxiliary device 30. Operation information Iop includes a start image, an end image, operation information, and time information for each operation unit (these are recorded in the recording unit 35 as operation unit information 35b). Operation information Iop may also include a target area image (P43), and may also include a landmark image. If a specialist performs a re-examination of the organ targeted by the first endoscope 10A on the same subject (patient), the ID of this subject (patient) is transmitted to the auxiliary device, and the operation unit information 35b of this subject (patient) is obtained. Operation information Iop may, of course, consist only of the necessary information from the operation unit information 35b. Image data acquired by the imaging unit 12B may also be transmitted to the auxiliary device 30.

[0105] Furthermore, the communication unit 21B may communicate information with the endoscopy system 10A in addition to the auxiliary device 30. The communication unit 21B may also communicate with other servers or hospital systems, in which case it can collect and provide information from these other servers or hospital systems. It may also receive inference models generated by external learning devices.

[0106] The communication unit 21B functions as an acquisition unit that acquires time-series operational content information in the first endoscope system as operational unit information (see, for example, S31 in Figure 4). The aforementioned operational unit information is image change information estimated using the asymmetry of the organ being observed (see, for example, Figure 7). The aforementioned operational unit information is image change information that shows a sequence of identical actions (see, for example, image P1 in Figure 1A and images P6a to P6f in Figure 7). Furthermore, the asymmetry information of the organ being observed is determined based on the anatomical positional relationship of multiple parts within a specific organ (see, for example, Figure 7). The communication unit 21B also functions as an input unit that inputs recorded operational unit information for a subject who has undergone examination using the first endoscope system (see, for example, S31 in Figure 4).

[0107] The signal output unit 22B outputs a signal indicating that the tip of the second endoscope system 10B has reached the vicinity of a target area such as an object or a diseased area. For example, the position may be indicated to a doctor or other medical professional by irradiating with a light source from the light source unit 13B, so that the irradiated light is visible from outside the digestive tract wall.

[0108] The similar image determination unit 23B compares the image data acquired by the imaging unit 12B with the group of operation assistance images P4 used during re-examination and determines the degree of similarity. The group of operation assistance images P4 used during re-examination includes a start image, an end image, etc., for each operation unit. These images are compared with the current endoscopic image acquired by the imaging unit 12B to determine whether or not they are similar images. There are various methods for determining the similarity of images, such as pattern matching, so a method suitable for this embodiment should be used as appropriate from among these methods.

[0109] When the physician inserts the second endoscope system 10B into the patient's body cavity, the similar image determination unit 23B determines whether each image in the operation assistance image group P4 is similar to the image acquired by the imaging unit 12B. In performing this determination, since the operation assistance image group P4 is divided into operation units, the similar image determination unit 23B determines which operation unit the currently acquired image group is similar to. If the image acquired by the imaging unit 12B is similar to the end of an operation unit, the guide unit 19B displays the operation information on the display unit 14B based on the operation information Iop.

[0110] Furthermore, as explained with reference to Figures 5 to 8, the similar image determination unit 23B determines the operation process of the second endoscope system 10B by detecting changes in the endoscopic image pattern. Based on operation unit information Iop, etc., the imaging unit 12B divides the continuous images acquired into operation units and determines the operation currently being performed for each operation unit, such as insertion, rotation, bending, etc. (see Figure 7). The similar image determination unit 23B functions as an insertion operation determination unit that estimates the operation process when a patient undergoes an examination (including diagnosis and treatment) using the second endoscope system (see S37 in Figure 4).

[0111] When the similar image determination unit 23B finds an image P43 that is highly similar to the target object (see marker Ob in Figure 2), the display unit 14B indicates that the target area, such as the affected area (target area Tg in Figure 2), is located near this position. The physician can then search for the target area by carefully examining this area according to the operating information. If it cannot be found immediately, air may be insufflated, or the endoscope may be slightly withdrawn to create space for observation. Once the target area is found, the physician can monitor its progress. Depending on the condition of the target area, surgery or other procedures may be necessary.

[0112] Furthermore, similar image determination may be performed not only by the similar image determination unit 23B based on the similarity of the images, but also by inference. That is, an inference engine may be provided within the similar image determination unit 23B, and an inference model for similar image determination may be set in this inference engine, and similarity may be determined by inference. In this case, the inference engine functions as a similar image estimation unit having a similarity estimation model that estimates the similarity of images based on the images of the endoscopic examination. Even when a non-specialist operates the endoscope, the determination result from this similar image determination unit 23B can be used to guide the endoscope to the vicinity of the target area, such as the affected area.

[0113] The guide unit 24B provides operational guidance (which may also be called operational advice) to a non-specialist using the second endoscopy system 10B, based on the judgment result of the similar image determination unit 23B. Specifically, the guide unit 24B may use the judgment result of the similar image determination unit 23B to divide the time-series images acquired by the imaging unit 12B into operational units, compare the operational information contained in the operational unit information with the currently acquired sequential images, and display whether the operation was performed correctly or incorrectly based on the comparison result. In other words, it guides the user to perform operations equivalent to those performed by a specialist, enabling them to observe the affected area or other organs under the same observation conditions as a specialist.

[0114] Furthermore, the guide unit 24B may perform event detection and display corresponding information. For example, guide operations such as rotation, bending, water injection, or air supply may be performed within a certain operation unit. The guide unit 24 may display an operation guide for proceeding to the next operation unit at the timing of switching between operation units. This guide display may be superimposed on the endoscopic image displayed on the display unit 14B, or the display unit 14B may display it via voice guidance or the like. In other words, the display unit 14B may not only display the guide so that it can be seen, but may also convey the guide information to non-specialists via voice or the like.

[0115] Furthermore, in this embodiment, the guide unit 24B outputs guide information so that when a non-specialist performs an examination using the second endoscope system 10B, the characteristic parts of the organ to be observed can be observed under the same observation conditions as when a specialist performs the examination (see, for example, S37 and S47 in Figure 4). Here, similar observation conditions include the size of the object captured in the image and the field of view, and are conditions to ensure that the positional relationship between the imaging unit and the object being observed is the same when observing the object. In addition, conditions may be set to allow for the determination of changes in color by controlling the illumination and exposure for the object being observed in the same way, and optical conditions such as focus and field of view (including up, down, left, right within the image, and the positions of the person who observed last time and the person observing this time) may also be set to be the same.

[0116] Furthermore, in this embodiment, the explanation of how to enable observation under the same conditions as the first endoscope system in the second endoscope system mainly focuses on aligning the positional relationship. However, it may also be a concept that encompasses other conditions, and it is sufficient if the first and second endoscope systems can share this information in the same way. Also, exposure and positional relationship (the positional relationship between the object being observed and the imaging unit) are related, and when the object being observed and the imaging unit are close together, the reflection of the illumination light increases, and the gloss of the object being observed may cause specular reflection or flare, resulting in incorrect exposure. In this sense, the positional relationship is an element that should be considered when making the observation conditions the same. In addition, it may be better to make the conditions the same as those for procedures such as water injection and suction, and the influence of the instruments used, and it is possible to apply this information to be carried over. By considering the above elements, exposure, visibility, and positional relationship may change. Furthermore, even when making the observation conditions the same, it is not necessary to strictly match them; they can be said to be the same as long as they fall within an acceptable range considering appropriate thresholds, etc.

[0117] The guide unit 24B compares the operation process and operation unit information estimated by the insertion operation determination unit and functions as an operation guide unit that outputs operation guide information for operating the second endoscope system in order to observe characteristic parts of the target organ with the second endoscope system (see, for example, S37 and S39 in Figure 4). The operation guide information output by the operation guide unit is guide information for observing characteristic parts of the target organ under the same observation conditions as the first endoscope system (see, for example, S37 and S39 in Figure 4). When comparing the operation process and operation unit information estimated by the insertion operation determination unit, the operation guide information for operating the second endoscope system in order to observe characteristic parts of the target organ is compared by comparing multiple temporally adjacent operation unit information, and if there are overlapping parts during observation that do not require further observation, the operation of these overlapping parts is removed from the operation unit information and compared.

[0118] Furthermore, the similar image determination unit 23B and the guide unit 24B divide the images acquired in a time series into operation units, estimate the operating state of the second endoscope system for each operation unit, compare this estimated operating state with the operation unit information, and function as an operation guide unit that outputs guide information for observation under the same observation conditions as the first endoscope system (see, for example, S37 and S39 in Figure 4).

[0119] For example, in Figure 2, when a non-specialist inserts the second endoscopic system 10B into a body cavity and moves it straight along route R1, upon reaching position L1, the system guides the non-specialist to bend or rotate the tip of the second endoscopic system 10B to proceed along route R2. Also, when moving along route R3 and approaching marker Ob, the system indicates that the target area Tg, such as the affected area, is approaching. In this way, the system memorizes the operations performed by the specialist when they reached the target area Tg using the endoscopic system 10A, and provides guidance for operations to reach this target area Tg. Therefore, even a non-specialist can easily operate the second endoscopic system 10B to reach the target area Tg and perform observation and treatment.

[0120] As described above, in this embodiment, when a non-specialist performs an examination, diagnosis, treatment, etc. using the first endoscopy system 10A, and then performs an examination, etc. using the second endoscopy system 10B, the second endoscopy system 10B is guided so that the target area can be observed under the same observation conditions as when the endoscopy specialist used the first endoscopy system 10A. The following are examples of target areas that require such follow-up observation. • Benign lesions that may become malignant in the future • Sites where lesions may appear in the future (for example, common sites for lesions when infected with H. pylori, common sites for lesions when not infected with H. pylori, etc.) • After treatment (may recur)

[0121] Furthermore, when performing an examination using the second endoscopic system 10B, the following auxiliary information can be obtained as clues for the examination, and the examination can be performed by using this auxiliary information. • Visual information of the area being monitored, such as the size, shape, texture, and color of the lesion. The following information regarding the imaging environment is required: whether dyes such as indigo carmine or stains such as methylene blue are used; whether observation light such as WLI (White Light Imaging) or NBI (Narrow Band Imaging) is used; whether image processing settings such as structural enhancement are used; air volume; patient position; equipment information such as video processor and scope type; condition of the mucosa surrounding the lesion (whether there are any ambiguous features); distance between the lesion and the scope and viewing angle; and imaging of the scope insertion amount, twist amount, angle, and curvature. • Past findings information, for example, information on the degree of the pharyngeal reflex. • Information regarding the time period, such as the time since the start of the test and the date the test was conducted.

[0122] To detect areas that require monitoring as described above, it is advisable to use AI (Artificial Intelligence) as described below. • Detection AI to identify areas requiring follow-up, such as AI for area recognition, image-based CADe (Computer-Aided Detection), CADex (Computer-Aided Diagnosis), and AI for detecting treated areas. Information recorded in electronic medical records can also be used as a substitute. AI for recognizing the characteristics of areas requiring follow-up, such as AI for detecting the size, shape, response, and color of lesions, and AI for detecting the condition of the surrounding mucous membrane. • AI for recognizing the observation environment, for example, AI that can detect the presence or absence of staining agents such as indigo from an image. AI for estimating air volume, such as AI that estimates based on air pressure sensor output, AI that estimates based on cumulative air supply time, and AI that estimates based on images. • An AI for estimating the distance to the lesion, for example, an AI that estimates the distance and viewing angle between the lesion and the endoscope tip by using the insertion amount of the endoscope tip, the twist, angle, and curvature of the endoscope tip, etc.

[0123] Next, using the flowchart shown in Figure 3, we will explain the operation of recording the process of how a specialist uses the endoscopic system 10A to reach a target area such as the affected area. This operation of the endoscope 1 is realized through the cooperation of the control unit 11A in the endoscopic system 10A and the control unit 31 in the auxiliary device 30. Specifically, this is achieved by the CPUs provided in each control unit controlling each part of the endoscopic system 10A and the auxiliary device 30 according to the program stored in memory.

[0124] When the flow of Endoscope 1 begins, first, imaging starts (S1). Here, the image sensor in the imaging unit 12A acquires time-series image data at time intervals determined by the frame rate. When imaging starts, image data of the body cavity is acquired, and this image data is processed by the image processing circuit in the imaging unit 12A. The display unit 14A displays images of the body cavity using the processed image data. The specialist operates the endoscope system 10A while viewing this image, moving the tip toward the target site such as the affected area. The processed image data is also transmitted to the input unit 32 in the auxiliary device 30 via the communication unit 21A. In this step, images of the patient's organs are acquired time-series by the imaging unit.

[0125] Next, it is determined whether or not the image of the inner wall surface has changed (S3). Here, the operation unit determination unit 36 ​​in the auxiliary device 30 determines whether or not the image of the inner wall surface inside the body cavity acquired by the endoscope system 10A has changed. As explained using Figures 5 to 8, when an endoscope is inserted into a body cavity, the organ being observed changes, and therefore the image of the inner wall surface also gradually changes. Furthermore, even for the same organ, the image may change due to changes in the direction of the tip where the image sensor is located. In this step, the operation unit determination unit 36 ​​makes a determination based on the change in the image. Note that this determination is not limited to the operation unit determination unit 36, but may also be performed in other blocks such as the control unit 31. Alternatively, an inference model for determining changes in the inner wall surface may be set in the inference engine 37, and the inference engine 37 may be used to determine the image change of the inner wall surface. If, as a result of this determination, there is no change in the image of the inner wall surface, the system enters a standby state until a change occurs.

[0126] If the image of the inner wall surface changes as a result of the judgment in step S3, the image is temporarily recorded (S5). Here, the image data input in the input unit 32 is temporarily recorded as the inspection image 35a in the recording unit 35. Note that any memory capable of temporarily recording image data is acceptable, not just the recording unit 35.

[0127] After provisionally recording the image, it is then determined whether the image change pattern has changed due to the insertion direction, rotation, tip bending, etc. (S7). This determination is made if, as a result of the determination in step S3, the image of the inner wall surface has changed, and it is determined whether the cause of this change is endoscopic manipulation by a specialist, such as changing the insertion direction of the endoscope tip, rotating the tip, or bending the tip. In other words, it is determined whether the change in the image change pattern is due to manipulation by a specialist, rather than simply because the part of the organ being observed has changed.

[0128] The change in this image change pattern is determined by the operation unit determination unit 36 ​​based on the image input in the input unit 32. For example, in Figure 2, if the device was moving straight along route R1, and then at position L1 the direction of the tip curves and the device moves in the direction of route R2, the image change pattern usually changes at this position L1. The change in the image change pattern may simply be a change in the image pattern itself (for example, the image pattern may change from a circle to a square), or it may be a change in the way the image pattern changes. In any case, it is sufficient to determine whether or not the image has changed due to an operation by a specialist or the like.

[0129] In addition to images, other information such as operation information associated with image data or sensor output from a sensor provided at the tip of the endoscope may also be used. For example, the determination may be made based on information such as operation information attached to the image data, or it may be made based on information such as images and operation information. Furthermore, a sensor may be provided at the tip of the endoscope, and operation information may be acquired based on the output from this sensor. Alternatively, a transmission source may be provided at the tip of the endoscope, and a sensor may be provided to detect the signal from the transmission source outside the body, and operation information may be acquired based on the output from this sensor. Moreover, this determination may be made not only by the operation determination unit 36, but also by an inference model set in the inference engine 37 (or inference engine 18A).

[0130] If the result of the determination in step S7 indicates that the image pattern has not changed due to the insertion direction, rotation, or tip bending, then another event is performed (S9). Other events include various events performed by a specialist, such as air insufflation, water injection, suction, and still image capture. When a specialist performs another event, the location and type of the event are recorded. These recorded events are displayed when a non-specialist performs an examination (see S39 in Figure 4).

[0131] Other events include operations and processes other than insertion direction, rotation (up / down relationship), and tip bending. For example, these include the use of treatment instruments, changes in shooting parameters such as exposure and focus, image processing including HDR (High Dynamic Range) and depth stacking, switching light sources for special light observation, image processing to emphasize specific structures, dye spraying and staining, and other operations and processes that add an extra step to discover the target object. Information that such an extra step has been taken serves as a useful guide. In addition to the observation guide, there may also be a guide that instructs the removal of polyps if they are found. To realize this guide, it may be possible to appropriately select how much of the record of the first endoscopic system has been used in the guide. For example, it may be possible to limit it to "observation-related" information, or to include "treatment-related" information, or to allow the first side, second side, or a third party to choose. If treatment-related information is included, displaying "a polyp was previously removed at this location" would be helpful during follow-up observations.

[0132] On the other hand, if the result of the determination in step S7 indicates that the image change pattern has changed due to the insertion direction, rotation, tip bending, etc., the continuous portion of the image change pattern is treated as an "operation unit," and operation content information is recorded in chronological order, recording the start and end images of the operation (S11). Here, based on the result of the determination in step S7, a series of images from the image in which the image change pattern has been determined to have changed until the next image in which the image pattern has been determined to have changed is treated as an "operation unit," and image data is recorded in chronological order in the recording unit 35. The image that marks the start of the operation unit in this series of images is recorded as the start image 35ba, and the last image in the series of images is recorded as the end image 35bb.

[0133] Furthermore, in step S11, the operation unit determination unit 36 ​​performs image analysis on a series of images to obtain operation information and extracts operation information attached to the images, recording these as operation information 35bc. The operation unit determination unit 36 ​​also extracts time information from the time information of the first and last images included in the operation unit and records it as time information 35bd. Note that this information does not have to be recorded all at once; information may be extracted and recorded as appropriate while repeatedly executing steps S3 → S21 → S3.

[0134] Step S11 can be described as a determination step in which images of organs acquired in chronological order are divided into operation units, and the operation performed by the first endoscope is determined for each operation unit. Step S11 can also be described as a recording step in which, for each determined operation unit, information regarding the image and endoscopic operation in that operation unit is recorded as operation unit information in the recording unit.

[0135] In step S11, after recording various information about the operation unit, it is then determined whether or not there is an image change based on the asymmetry of the anatomical structure (S13). As explained using Figures 5 to 8, when an object is simply in a pipe shape, the left-right, up-down, and down directions are not immediately apparent. Therefore, in this embodiment, the asymmetry of the anatomical structure of the organ is used to determine the position (direction) of the tip, etc. In this step, the operation unit determination unit 36 ​​analyzes the image input by the input unit 32 and determines whether or not there is an image change based on the asymmetry of the anatomical structure. For example, in Figure 7, from time T3 to time T4, the projection of the cavity changes from the upper right 1 o'clock direction to the 12 o'clock direction. In this way, image changes can be detected by utilizing the existence of asymmetry in the organ.

[0136] If the result of the judgment in step S13 indicates a change in the image, it is determined that there has been a change in the direction of the tip, and the change in the direction of operation is recorded (S15). Since there has been a change in the direction of operation, this is recorded in the recording unit 35. As mentioned above, the direction of the tip of the endoscope can be determined by utilizing the asymmetry of the anatomical structure, and since there was a change in the image in step S13, it can be said that the direction of the tip has changed, so in step S15, the changed direction of operation is recorded in the recording unit 35. If the direction of operation changes, the following series of images may be recorded as an "operation unit".

[0137] If recording occurs in step S15, or if there is no image change as a result of the determination in step S13, or if other events are performed in step S9, then it is determined whether or not a marker has been found (S17). A marker is, for example, a marker Ob in Figure 2, an object located near a target area Tg such as an affected area, which serves as a landmark when searching for this target area Tg. The image information and / or positional information of this marker Ob is included in the operation information output from the auxiliary device 30, so the guide unit 24B determines whether or not a marker has been found based on this information and the image acquired by the imaging unit 12B.

[0138] If a landmark is found as a result of the assessment in step S17, the landmark image is recorded, and the operations performed before the discovery are also recorded (S19). First, the landmark image is recorded. Also, since the target area, such as the affected area, is near the landmark, the operations performed to find the target area in this vicinity and to find the target area image are recorded. In other words, the specialist records what operations were performed from the landmark to the target area. With this record of operations, even a non-specialist can easily reach the target area by operating the endoscope while referring to the operation record.

[0139] The flow shown in Figure 3 assumes a case where accessing the target area starting from a landmark is useful, and shows an example where the process is carried out in the order of landmark discovery followed by target area discovery. However, as mentioned above, there are cases where it is easy to guide to the target area even without a landmark, and in this case, the image of the landmark may not be recorded in the operation unit information. In that case, steps S17 and S19 may be omitted, and the target area may be found directly. Whether or not to record a landmark may be decided by a specialist, or it may be recorded automatically using AI or the like. There are also cases where the endoscope is withdrawn or removed. Although not specifically explained, there are also difficult withdrawals, and in such cases, the image of the end of the difficult section may be recorded in the operation unit information, and when the image of the end of the difficult section is detected, the target may be reset.

[0140] Next, it is determined whether or not the target site has been found (S20). In this step, a specialist determines whether or not the target site has been found. Whether or not the target site has been found may be determined by the specialist recording this fact, or based on specific operations such as taking a still image, or by the AI ​​automatically determining based on the image or operations. If the target site has been found, an image of the target site may be recorded. If, as a result of this determination, the system does not determine the target site for a via, it returns to step S19.

[0141] In step S20, if the target site is found, or if the object was not found in step S17, the next step is to determine whether the procedure is complete or not (S21). If the specialist finds the target site and completes the necessary recording, the procedure may be determined to be complete at this point. If there are multiple target sites, the procedure may be determined to be complete when the last target site is found. Alternatively, the procedure may be determined to be complete when all operations are completed, such as withdrawing the endoscope from the body cavity. If the determination in this step indicates that the procedure is not complete, the procedure returns to step S3 and the aforementioned actions are performed.

[0142] If the determination in step S21 is successful and the procedure is completed, relevant data regarding landmarks, target sites, etc., is transmitted (S23). Here, since the operational information up to the point where the specialist reaches the target site is recorded, this information is transmitted to terminals, servers, etc. that require the information. For example, if the second endoscopy system 10B requests the transmission of relevant data regarding landmarks, target sites, etc., the corresponding data may be transmitted based on the specified patient ID, etc. Alternatively, the operation unit information 35b recorded in the recording unit 35 may be transmitted to an external device all at once. Step S23 functions as an output step that outputs the operation unit information recorded in the recording unit. When data transmission is performed in step S23, this flow is terminated.

[0143] In this flow, when the image pattern changes due to changes in insertion direction, rotation, tip bending, etc., the sequence of images until the next change in the image pattern is recorded as an operation unit (see S7 and S11). At this time, the starting image, ending image, and operation details are recorded as operation unit information 35b. In addition, it is determined whether or not there has been an image change based on anatomical asymmetry, and if there has been a change, the change in tip direction is determined and the change in operation direction is recorded (see S13 and S15). When an object is found, an image of the object is recorded (see S17 and S19). The object is located near the target site, such as the affected area, and if the object is found, the target site can be easily found. Furthermore, if the operation before discovery is recorded, the target site can be reached even more easily based on this operation record. In other words, if the information on how a specialist reaches the target site, such as the affected area, is recorded, a non-specialist can easily reach the target site, such as the affected area, using this information (see flowchart in Figure 4).

[0144] In this flow, in addition to continuous images, operation content information, operation start image, and operation end image are recorded as operation unit information (see S11). The operation unit information is not limited to the information recorded in this flow, as it is sufficient to provide information that a non-specialist using the second endoscopy system 10B can use to reach the target area such as the affected area. For example, it is not limited to the start image (initial image), but may also be an image from the early stages of the operation unit, an image near the end, or even an image in the middle. Furthermore, the determination of the change in the image pattern in step S7 may be based on the asymmetry of anatomical structures. Usually, if there is a change in the image pattern in step S7, there is often an image change based on the asymmetry of anatomical structures. In addition to image information, time information such as the elapsed time from the start of the examination may also be used as operation unit information. For example, instead of the end image, time information related to the timing of completion may be used as operation unit information.

[0145] Furthermore, in this flow, objects (which serve as landmarks) located near the target area, such as the affected area, are detected. Recording these landmark objects has the advantage of encouraging non-specialists to observe them carefully when they are found. However, the recording of landmark detection may be omitted, and only the detection of the target area, such as the affected area, may be recorded. In addition, in step 7 and S11, recording was performed as an operation unit based on changes in the image change pattern, and in steps S13 and S15, the direction of the tip was determined and recorded based on the asymmetry of the anatomical structure. These processes may be performed in a single step rather than in separate steps.

[0146] Furthermore, the explanation of the flow of endoscope 1 shown in Figure 3 assumed that the endoscope system 10A and the auxiliary device 30 would work together to process the data. However, the explanation is not limited to this; the endoscope system 10A may independently record the operational information until the specialist reaches the target site. In this case, the image change determination in step S3 is performed by the image processing circuit in the control unit 11A and / or the imaging unit 12A, and if there is a change in the image, it is temporarily recorded in the memory of the recording unit 16A or the like in the endoscope system 10A in step S5. Similarly, the determinations in steps S3, S7, S13, and S17 are also performed by the image processing circuit in the control unit 11A and / or the imaging unit 12A, and if there is a change, it is recorded in the memory of the recording unit 16A or the like in the endoscope system 10A. Then, in step S21, if it is determined that the procedure is complete, all the information recorded inside the endoscope system 10A up to that point is sent together to the auxiliary device 30 (see S23).

[0147] Next, using the flowchart shown in Figure 4, we will explain the operation of the endoscope 2 when a non-specialist uses the second endoscope system 10B to reach the target area such as the affected area. The purpose of this operation is for a non-specialist to perform an examination on the same person who was examined by a specialist, using the second endoscope system 10B, and targeting the same affected area. When performing this examination, the operation information from the examination performed by the specialist (based on the operation unit information 35b in Figure 1A, etc.) is provided to the second endoscope system 10B, and the non-specialist can easily reach the target area such as the affected area by operating the second endoscope system 10B according to the operation guide based on this operation information.

[0148] The operation of this endoscope 2 is realized by the control unit, such as the CPU of the control unit 11B within the second endoscope system 10B, controlling each part within the second endoscope system 10B according to a program stored in memory. This flow of endoscope 2 enables an endoscopic examination method in which the target organs of a patient who has undergone organ examination using the first endoscope system are observed using the second endoscope system.

[0149] When the flow of endoscope 2 begins, first, relevant data such as landmarks and target areas are acquired (S31). Here, the control unit 11B acquires relevant data such as landmarks and target areas from the auxiliary device 30 via the communication unit 21B. As mentioned above, the recording unit 35 of the auxiliary device 30 stores operation unit information 35b (see, for example, S23 in Figure 3). The control unit 11B then transmits the ID of the patient being examined (including diagnosis and treatment) using the second endoscope system 10B (recorded in the ID management unit 15B) to the auxiliary device 30, and acquires data related to the landmarks and target areas of the examination from the operation unit information 35b corresponding to the patient ID. At this time, it is advisable to acquire the starting image (starting image 35ba), ending image (ending image 35bb), operation content information (operation information 35bc), and operation time information 35bd for each operation unit. This step can be said to be a step in which time-series operation content information in the endoscope system 10A is acquired as operation unit information. Furthermore, step S1 can be described as a step in which time-series operational information of the first endoscope system is acquired as operational unit information.

[0150] After acquiring the relevant data, imaging is then started (S33). Here, the image sensor in the imaging unit 12B acquires time-series image data at time intervals determined by the frame rate. When imaging starts, image data of the body cavity is acquired, and this image data is processed by the image processing circuit in the imaging unit 12B. The display unit 14B displays the image of the body cavity using the image processed image data. A non-specialist operates the endoscope system 10B while viewing this image, moving the tip toward the target site such as the affected area.

[0151] Once imaging begins, the next step is to determine whether or not a starting image has been detected (S35). Since a starting image (starting image 35ba) for each operation unit was acquired in step S31, in this step, the similar image determination unit 23B compares the image acquired by the imaging unit 12B with the starting image to determine whether or not a starting image has been detected. There are generally multiple operation units in a single examination, consultation, or treatment. Therefore, the starting images are sequentially read from the input operation unit information, and the read starting images are compared with the acquired images to determine whether or not the starting images and acquired images match or are similar. If, as a result of this determination, a starting image is not detected, the process proceeds to the determination in step S53 to determine whether or not the endpoint has been reached.

[0152] On the other hand, if a starting image is detected as a result of the determination in step S35, the operation content information (insertion, rotation direction, amount, time) is then referenced in chronological order and reference information is displayed (S37). As mentioned above, the details of the operation performed for each operation unit are recorded in the information unit information 35b. Therefore, in this step, the similar image determination unit 23B and the guide unit 24B display the operation content (operation guide) for the operation unit corresponding to the starting image detected in step S35 on the display unit 14B.

[0153] In step S37, when displaying the operation guide, the similar image determination unit 23B first determines the operating state of the second endoscope system 10B, such as insertion in a straight line, rotation, or bending, based on the image acquired by the imaging unit 12B. In other words, in this step, the operation process when the patient undergoes an examination using the second endoscope system is estimated. The similar image determination unit 23B may also determine the operating state based on sensor information provided at the tip of the second endoscope system 10B, or based on information regarding the operating state of the operation unit 17B, or by combining these pieces of information.

[0154] After determining the operating state, the guide unit 24B then compares the operating state included in the operating unit information input from the auxiliary device 30 with the operating state determined by the similar image determination unit 23B, creates an operation guide based on the comparison result, and displays this operation guide on the display unit 14B. In other words, the operation information recorded in the operating unit information corresponding to the current operating unit and the current operating information of the second endoscope system 10B determined by the similar image determination unit 23B are displayed as reference information. Non-specialists can learn how to operate the second endoscope 10B by referring to this reference information. If the two are the same (even if not exactly the same, they should be approximately the same), it can be said that the operation is being performed towards the target area such as the affected area, as it is essentially the same operation as an examination performed by a specialist. In this step, the estimated operating process is compared with the operating unit information, and guide information is output for observing characteristic areas in the target organ under the same observation conditions as the first endoscope system.

[0155] Next, the system displays whether the operation was performed correctly and determines other events, and then displays the appropriate response (S39). Here, the control unit 11B compares the operation information of the specialist, which is recorded as operation unit information, with the operation status actually performed by the non-specialist, determines whether the operation is good or insufficient, and displays whether it is good or bad on the display unit 14B based on the determination result. For example, if the operation unit status is recorded as a forward bending operation, but the determination in step S37 shows that a right rotation operation was performed, the operation status is different, and the system advises correcting the operation.

[0156] Furthermore, based on the information recorded in the operation unit information, etc., it is determined whether events such as air supply operations, water injection operations, and suction operations are necessary, and the operations to be performed based on the determination result are displayed. Other events include, as mentioned above, operations and processes other than insertion direction, rotation (up and down relationship), and tip bending. For example, these include the use of treatment instruments, changes in shooting parameters such as exposure and focus, image processing including HDR (High Dynamic Range) and depth stacking, switching of light sources such as special light observation, image processing to emphasize specific structures, dye spraying and staining, and other operations and processes that add an extra step to discover the target object. If a specialist performed other events when examining a subject, those events are recorded (see S9 in Figure 3), and in this step, the corresponding display is made based on the recorded event information.

[0157] Next, it is determined whether or not a landmark image has been detected (S41). As mentioned above, a landmark object is designated near the target, such as the affected area, to be used when searching for the target (see landmark Ob in Figure 2), and this landmark image (which may include position information) is recorded in the operation unit information 35b within the auxiliary device 30. In this step, the similar image determination unit 23B compares the landmark image of the landmark object with the current image acquired by the imaging unit 12B, and determines whether or not a landmark image has been detected based on this comparison.

[0158] If the determination in step S41 does not detect a landmark image, it is determined whether or not the endpoint image of the operation unit has been reached (S49). As mentioned above, the recording unit 35 in the auxiliary device 30 records an endpoint image 35bb for each operation unit. In this step, the similar image determination unit 23B compares the endpoint image (endpoint image 35bb) with the current image acquired by the imaging unit 12B, and determines whether or not the endpoint image has been detected based on this comparison.

[0159] If the result of the determination in step S49 is not the endpoint image of the operation unit, it is determined whether to repeat the operation (S51). A non-specialist may operate the second endoscope 10B aiming for a landmark or target, but may have difficulty reaching the landmark or target and may have to repeat the operation. In this step, the control unit 11B determines whether the non-specialist is repeating the operation based on the image acquired by the imaging unit 12B, the operation unit 17B, and the output of sensors provided in the device. If the result of this determination is that it is not a repeat, the process returns to step S37 and the above operations are repeated. On the other hand, if the result of the determination is that it is a repeat, the process returns to step S35 and the above operations are repeated.

[0160] Returning to step S41, if a landmark image is detected, a discovery display is shown (S43). Here, the control unit 11B or the guide unit 24B displays on the display unit 14B that a landmark has been found to reach the target area. This display allows non-specialists to understand that the target area, such as the affected area, is located near the landmark, and they should carefully observe the area around the landmark to find the target area.

[0161] Next, the landmark is recorded (S45). Here, since the landmark has been found, an image of the landmark is recorded in the recording unit 16B. Subsequently, the pre-discovery operations are displayed (S45). Information about the operations performed by the specialist from the discovery of the landmark to the target site is recorded in the operation unit information 35b of the recording unit 35 (see S19 in Figure 3). Therefore, in this step, the control unit 11B or the guide unit 24B displays the operations for guidance based on the recorded pre-discovery operation information.

[0162] Next, it is determined whether or not the target area has been found (S48). Since the image of the target area is recorded in the operation unit information 35b, the similar image determination unit 23B compares the image of the target area with the current image acquired by the imaging unit 12B and determines whether or not the target area has been detected based on this comparison. If the result of this determination is that the target area has not been found, the process returns to step S47 and the pre-discovery operation display is performed. On the other hand, if the target area has been found, the discovery of the target area is displayed and the target area is recorded, etc.

[0163] If the target area is found in step S48 and the discovery is indicated, or if the endpoint image of the operation unit is detected as a result of the determination in step S49, or if the starting image was not detected in step S35, then it is determined whether the procedure is complete or not (S53). Here, a non-specialist determines whether the predetermined examination has been completed using the second endoscopy system 10B. If a target such as an affected area is found and the examination or imaging is performed, it may be determined as complete. If there are multiple target areas, it may also be determined as complete when the last target area is found. It may also be determined as complete when the non-specialist decides to end the examination. If the result of this determination is not complete, the procedure returns to step S35 and the aforementioned operations are performed. On the other hand, if the result of the determination is complete, the termination operation of this flow is performed.

[0164] Thus, in the flow of Endoscope 2, relevant data regarding the target and object during examination and testing is acquired in Endoscope System 10A (see S31). When the image acquired by the imaging unit 12B in the second Endoscope System 10B is identical or similar to the starting image of the operation unit (S35 Yes), an operation guide is displayed on the display unit 14B based on the operation content information (see S37). Then, when a landmark of the target area is detected (see S41), the operation to find the target area is displayed (S47). In other words, since a guide display is provided for each operation unit based on the operation performed by the specialist, even non-specialists can easily reach the target.

[0165] In other words, in the Endoscope 2 flow, the operation information recorded chronologically in the first endoscope system is acquired as "operation unit information" for operations that continue over a predetermined period of time (see S31), the operation process in the second endoscope system is estimated for the target organ, which is the same organ of the same patient as in the first endoscope system, the estimated operation process is compared with the "operation unit information," and guide information is output for observing characteristic parts of the target organ under the same observation conditions as the first endoscope system (see S35-S39). As a result, even non-specialists can perform observations of target areas such as the affected area at the same level as specialists.

[0166] Furthermore, the aforementioned "operation unit information" is image change information that indicates a sequence of identical actions. As explained using Figure 5, by dividing and recording the sequence of images taken when performing an examination (including diagnosis and treatment) using the endoscope system 10A into operation units, it can be used as an operation guide when performing an examination using the second endoscope system 10B. In addition, when dividing the sequence of images into operation units, as explained using Figure 5, the division can be easily done by utilizing the asymmetry of the organ being observed.

[0167] Furthermore, information on the asymmetry of the observed organ is determined based on the anatomical positional relationships of multiple parts within the specific organ (rather than the direction of gravity at the time of examination). Within internal organs, the direction of gravity is unknown, making it difficult to determine positional relationships such as up, down, left, and right. However, the asymmetry of the observed organ can be determined based on the positional relationships of multiple parts within the specific organ.

[0168] Furthermore, in this flowchart, an object (which serves as a landmark) located near the target area, such as the affected area, is detected (see S41). When a non-specialist detects an object that serves as a landmark, they can quickly locate the target area by carefully observing it. However, the detection of the landmark can be omitted, and only the detection and determination of the target area, such as the affected area, may be performed.

[0169] Furthermore, in this flow, all processing is performed only within the blocks of the second endoscope system 10B. However, the second endoscope system 10B may also be implemented in cooperation with external devices such as the auxiliary device 30. In this case, the second endoscope system acquires an endoscopic image, transmits this acquired endoscopic image to an external device (including a server, etc.) such as the auxiliary device 30, and the external device performs the processing in steps S35 to S51, etc. The second endoscope system 10B then displays the results of the processing in the external device. In this case, the second endoscope system is composed of the external device and the second endoscope system 10B.

[0170] As described above, one embodiment of the present invention provides a second endoscope system for observing the target organs of a subject who has undergone organ examination using a first endoscope system. This second endoscope system includes an acquisition unit (for example, the communication unit 21B shown in Figure 1B, see S31 in Figure 4) that acquires time-series operation content information in the first endoscope system as operation unit information, an insertion operation determination unit (for example, the similar image determination unit 23B in Figure 1B, see S37 in Figure 4) that estimates the operation process when the subject undergoes an examination (including diagnosis and treatment) using the second endoscope system, and an operation guide unit (for example, the guide unit 24B in Figure 1B, see S37 in Figure 4) that compares the operation process estimated by the insertion operation determination unit with the operation unit information and outputs guide information for observing characteristic parts of the target organs under the same observation conditions as the first endoscope system. The above-mentioned operation unit information is image change information that shows a sequence of identical actions estimated using the asymmetry of the target organs. With this configuration, the second endoscopic system can easily access target areas such as the affected area. Furthermore, the second endoscopic system allows non-specialists to perform observations of the affected area or other target areas at the same level as a specialist, even when the observations are being conducted by a non-specialist or when different equipment is being used.

[0171] Furthermore, the first endoscopic system according to one embodiment of the present invention includes an input unit that inputs images of the subject's organs in chronological order (for example, input unit 32A in Figure 1A, see S1 in Figure 3), an operation unit determination unit that divides the images of the organs acquired in chronological order into operation units and determines the operation performed for each operation unit (for example, operation unit determination unit 36 ​​in Figure 1A, see S11 in Figure 3), a recording unit that records information regarding the image and endoscopic operation in each operation unit determined by the operation unit determination unit as operation unit information (for example, recording unit 35 in Figure 1A, see S11 in Figure 3), and an output unit that outputs the operation unit information recorded in the recording unit (for example, communication unit 34 in Figure 1A, see S23 in Figure 3). With this configuration, the first endoscopic system can acquire information that allows even non-specialists to perform observations of target areas such as affected areas at the same level as specialists.

[0172] In one embodiment of the present invention, an endoscope inserted through the oral cavity and esophagus to examine (including diagnosis and treatment) the stomach or duodenum was used as an example. However, the present invention is not limited to gastric endoscopes or duodenal endoscopes, but can be applied to various other endoscopes such as laryngoscopes, bronchoscopes, cystoscopes, cholangioscopies, intravascular endoscopes, upper gastrointestinal endoscopes, duodenal endoscopes, small bowel endoscopes, capsule endoscopes, colonoscopes, lower gastrointestinal endoscopes, thoracoscopies, laparoscopes, arthroscopes, spinal endoscopes, and epidural endoscopes.

[0173] Furthermore, while one embodiment of the present invention described an example of using images acquired using an image sensor, the invention is not limited to this example, and for example, images using ultrasound may also be used. For example, in the case of the upper gastrointestinal tract, ultrasound images may be used for examination, diagnosis, and treatment of lesions that cannot be observed with optical images from an endoscope, such as the pancreas, pancreatic duct, gallbladder, bile duct, and liver. In the case of the lower gastrointestinal tract, if there is a need to observe anal fistulas, prostate cancer, lesions that cannot be seen with optical images from a colonoscope (e.g., adhesions), or the small intestine, ultrasound images may be used for examination, diagnosis, and treatment of lesions that cannot be seen with optical images. In ultrasound diagnosis, it may be necessary to visualize lesions for follow-up, etc., and in this case, in order to reach the affected area again, the operation is the same as when using a normal endoscope, by adjusting it in the same way as the previous time.

[0174] Furthermore, the determination of the operation unit was based on the image (see S7 and S11 in Figure 3). However, the determination may also be based on the output of a sensor installed on the endoscope, or on the operation information from the control unit of the endoscope. In addition, in the second endoscope system 10B, the operation unit may be determined based on other information, similar to the first endoscope 10A, in addition to the image.

[0175] Furthermore, while the explanation focuses on medical applications, particularly within the body, especially the luminal cavity of the digestive tract, there are many other devices that observe the inside of the body using similar procedures, and the invention described in this application can also be applied to laparoscopes and industrial endoscopes. This invention is not limited to flexible endoscopes; so-called rigid endoscopes also involve insertion and rotation operations, and the invention can be used as a guide during these operations as well. In the case of rigid endoscopes, the element of insertion angle is added when inserting into a body cavity, but even here, the invention described in this application can be applied if the operation guide is based on, for example, insertion at approximately vertical angle. This is because it is possible to determine whether the insertion angle has changed based on the image obtained during insertion.

[0176] Furthermore, although logic-based determination and inference-based determination have been described in one embodiment of the present invention, in this embodiment, either logic-based determination or inference-based determination may be appropriately selected and used. In addition, a hybrid determination method may be used in the determination process, partially utilizing the advantages of each.

[0177] Furthermore, in one embodiment of the present invention, the control units 11A, 11B, and 31 were described as devices composed of a CPU, memory, etc. However, in addition to being configured in software using a CPU and a program, some or all of each part may be configured as hardware circuits, and may also be hardware configurations such as gate circuits generated based on a programming language written in Verilog or VHDL (Verilog Hardware Description Language), or may be hardware configurations that utilize software such as a DSP (Digital Signal Processor). Of course, these may be combined as appropriate.

[0178] Furthermore, the control units 11A, 11B, and 31 are not limited to CPUs; any element that performs the function of a controller may suffice, and the processing of each of the above-described parts may be performed by one or more processors configured as hardware. For example, each part may be a processor configured as an electronic circuit, or it may be each circuit part of a processor configured as an integrated circuit such as an FPGA (Field Programmable Gate Array). Alternatively, a processor composed of one or more CPUs may perform the functions of each part by reading and executing a computer program recorded on a recording medium.

[0179] Furthermore, in one embodiment of the present invention, the auxiliary device 30 was described as having a control unit 31, an input unit 32, an ID management unit 33, a communication unit 34, a recording unit 35, an operation unit determination unit 36, and an inference engine 37. However, these do not need to be located within a single device; for example, they may be distributed as long as they are connected by a communication network such as the Internet.

[0180] Similarly, the endoscope system 10A and the second endoscope system 10B were described as having control units 11A, 11B, imaging units 12A, 12B, light source units 13A, 13B, display units 14A, 14B, ID management units 15A, 15B, recording units 16A, 16B, operation units 17A, 17B, inference engine 18A, clock unit 20A, communication units 21A, 21B, signal output unit 22B, similar image determination unit 23B, and guide unit 24B. However, these do not need to be located within a single device, and each unit may be dispersed.

[0181] Furthermore, in recent years, artificial intelligence capable of making decisions based on various criteria has become increasingly common, and improvements that allow for the simultaneous execution of each branch in the flowchart shown here also fall within the scope of this invention. If the user can input their opinion on whether such control is good or bad, the embodiments shown in this application can learn the user's preferences and be customized to suit that user.

[0182] Furthermore, components from different embodiments may be combined as appropriate. In particular, operations utilizing biological responses, including voice recognition, require appropriate sensors, interfaces, and judgment circuits, but these are not specifically described to avoid complexity. However, it should be noted that the present invention can also be achieved by various improved and alternative technologies that can substitute for these manual user operations.

[0183] Furthermore, among the technologies described in this specification, the controls mainly explained in flowcharts can often be configured by program, and may be stored on recording media or recording units. The method of recording on these recording media or recording units may be to record at the time of product shipment, to use distributed recording media, or to download via the internet.

[0184] Furthermore, although the operation in this embodiment was explained using a flowchart in one embodiment, the order of the processing steps may be changed, any step may be omitted, steps may be added, and the specific processing content within each step may be changed.

[0185] Furthermore, even if the operation flow in the claims, specification, and drawings is described using terms that indicate order, such as "first" and "next," for convenience, this does not mean that it is essential to perform the actions in that order in places where it is not specifically explained.

[0186] The present invention is not limited to the embodiments described above, and in practice, the components can be modified and implemented without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0187] 10A... Endoscope system, 10B... Second endoscope, 11A... Control unit, 11B... Control unit, 12A... Imaging unit, 12B... Imaging unit, 13A... Light source unit, 13B... Light source unit, 14A... Display unit, 14B... Display unit, 15A... ID management unit, 15B... ID management unit, 16A... Recording unit, 16B... Recording unit, 17A... Operation unit, 17B... Operation unit, 18A... Inference engine, 20A... Clock unit, 21A... Communication Signal unit, 21B...Communication unit, 22B...Signal output unit, 23B...Similar image determination unit, 24B...Guide unit, 30...Auxiliary device, 31...Control unit, 32...Input unit, 33...ID management unit, 34...Communication unit, 35...Recording unit, 35a...Inspection image, 35b...Operation unit information, 35ba...Start image, 35bb...End image, 35bc...Operation information, 35be...Time information, 36...Operation unit information, 37...Inference engine

Claims

1. In a second endoscopic system used to observe the target organs of a subject who has undergone organ examination using the first endoscopic system, An acquisition unit that acquires time-series operational information of the first endoscope system as operational unit information, For the above-mentioned subject, an insertion operation determination unit is provided to estimate the operational process when undergoing an examination using the second endoscope system, An operation guide unit compares the operation process estimated by the insertion operation determination unit with the operation unit information, and outputs operation guide information for operating the second endoscope system in order to observe characteristic parts of the target organ with the second endoscope system. It has, The second endoscopic system is characterized in that the above-mentioned unit of operation information is image change information estimated using the asymmetry of the above-mentioned organ under observation.

2. The second endoscopic system according to claim 1, characterized in that the asymmetry information of the observed organ is determined based on the anatomical positional relationships of multiple parts within a specific organ.

3. The second endoscopic system according to claim 1, characterized in that the above operation unit information is information relating to an operation that continues over a predetermined period of time.

4. The second endoscopic system according to claim 3, characterized in that the above operation unit information is an operation start image and information regarding the operation from the start to the end of the operation.

5. The second endoscope system according to claim 1, characterized in that the operation guide information output by the operation guide unit is guide information for observing the characteristic parts of the target organ under the same observation conditions as the first endoscope system.

6. The second endoscopic system according to claim 1, characterized in that the above operation unit information is image change information indicating a sequence of identical operations.

7. The second endoscopic system according to claim 1, characterized in that a first direction is determined when detecting asymmetry of the above-mentioned organ under observation.

8. The second endoscopic system according to claim 1, characterized in that, in detecting the asymmetry of the above-mentioned organ under observation, the direction in which fluid accumulates, determined by the direction of gravity, or the direction determined by the positional relationship of internal structures already detected, is referenced.

9. The second endoscope system according to claim 1, characterized in that the above operating unit information is determined by reflecting the angle at which a lever or knob for rotating the tip of the endoscope system is turned.

10. The second endoscope system according to claim 1, characterized in that the above operation unit information is information that defines the process until the observation direction of the tip of the endoscope system changes as the operation unit.

11. The second endoscope system according to claim 10, characterized in that the observation direction of the tip of the endoscope system is changed by twisting the endoscope system, by angling the endoscope system, or by pushing the endoscope system into the body.

12. The second endoscopic system according to claim 1, characterized in that the above-mentioned unit of operation information is information that uses the process until the shape of the observed organ changes as the unit of operation.

13. The second endoscopic system according to claim 12, characterized in that the above operation unit information is information that defines the process until the shape of the estimated organ changes by insufflating air and / or water and / or aspirating using the endoscopic system, or by pushing the endoscopic system in.

14. The second endoscope system according to claim 12, characterized in that the above operation unit information is information that defines the process until the state of the mucous membrane of an estimated organ changes by spraying a dye and / or staining agent using the first endoscope system, or by supplying water using the first endoscope system.

15. The second endoscope system according to claim 1 is characterized in that, when comparing the operation process estimated by the insertion operation determination unit with the operation unit information, the operation unit information is compared with a plurality of operation unit information, and if there is no need to continue observation of overlapping areas during observation, the operation unit information is corrected to exclude the operation of the overlapping area and compared.

16. The second endoscope system according to claim 1, characterized in that, based on the above-mentioned unit of operation information, characteristic parts of the above-mentioned organ to be observed are observed by automatic operation under the same observation conditions as the first endoscope system.

17. A method for operating a second endoscope system to examine the target organs of a subject who has undergone organ examination using a first endoscope system, wherein the second endoscope system comprises an acquisition unit, an insertion operation determination unit, and an operation guide unit, and the method for operating the second endoscope system is as follows: The acquisition unit acquires the time-series operation content information of the first endoscope system as operation unit information. The insertion operation determination unit estimates the procedure for the subject to undergo examination using the second endoscope system. The above operation guide unit compares the estimated operation process with the operation unit information and outputs operation guide information for operating the second endoscope system to observe characteristic parts of the target organ with the second endoscope system. The above unit of operation information is estimated using the asymmetry of the observed organ and represents image change information. A method for operating a second endoscope system, characterized by the following:

18. An input unit for inputting images of the subject's organs in chronological order, The image of the above organs acquired in chronological order is divided into operation units, and the operation unit determination unit determines the operation performed for each operation unit. For each operation unit determined by the above-mentioned operation unit determination unit, a recording unit records the image and information related to the endoscopic operation in that operation unit as operation unit information. An output unit that outputs the operation unit information recorded in the recording unit, A first endoscopic system characterized by having [a certain feature].

19. The first endoscope system according to claim 18, characterized in that the operation unit determination unit divides the acquired image into operation units based on whether or not at least one of the insertion direction, rotation direction, and bending direction of the tip of the first endoscope has changed.

20. The first endoscope system according to claim 18, characterized in that the operation unit determination unit determines the direction of operation based on the asymmetry of the anatomical structure of the acquired image.

21. The first endoscope system according to claim 18, characterized in that the recording unit records a start image and an end image in a sequence of images belonging to the above operation unit, and also records operation information indicating the operation state in the above operation unit.

22. The first endoscope system according to claim 18, characterized in that the recording unit records operational information after the discovery of a landmark object near the target.

23. A method for operating an endoscope device, wherein the endoscope device comprises an input unit, an operation unit determination unit, a recording unit, and an output unit, and the method for operating the endoscope device is: The above input unit acquires images of the subject's organs in chronological order. The above operation unit determination unit divides the images of the organ acquired in chronological order into operation units, and determines the operation performed by the first endoscope for each operation unit. The recording unit records, for each determined operation unit, information regarding the image and endoscopic operation in that operation unit as operation unit information. The output unit outputs the operation unit information recorded in the recording unit. A method for operating an endoscope device, characterized by the following features.

24. In a second endoscopic system used to observe the organs of a subject who has undergone organ examination using the first endoscopic system, For a patient who underwent examination using the first endoscopic system, there is an input unit for inputting recorded operation unit information, The imaging unit acquires images of the subject's organs in a time series, The above images, acquired in a time series, are divided into operation units, the operating state of the second endoscope system is estimated for each operation unit, and the estimated operating state is compared with the operation unit information to output guide information for observation under the same observation conditions as the first endoscope system. A second endoscopic system characterized by having the following.

25. For a patient who has undergone organ examination using the first endoscopic system, a computer using the second endoscopic system observes the target organ of the patient. The time-series information of the operations performed in the first endoscope system described above is acquired as operation unit information. For the above-mentioned subject, the procedure for undergoing the examination using the second endoscopic system is estimated. The estimated operation process is compared with the operation unit information, and guide information is output for observing characteristic parts of the target organ under the same observation conditions as the first endoscope system. A program that causes the above computer to perform the following action: The program is characterized in that the above-mentioned unit of operation information is image change information that shows a sequence of identical actions estimated using the asymmetry of the above-mentioned organ being observed.

26. Images of the subject's organs are acquired in chronological order. The images of the above organs, acquired in chronological order, are divided into operation units, and the operation performed by the first endoscope is determined for each operation unit. For each determined operation unit, the image and information regarding the endoscopic operation in that operation unit are recorded in the recording unit as operation unit information. The above-mentioned operation unit information recorded in the above-mentioned recording unit is output. A program characterized by causing a computer to perform a certain action.

Images