Second endoscope system, first endoscope system, and endoscopy method

Abstract

Provided are a second endoscope system, a first endoscope system, and an endoscope inspection method whereby a target site such as an affected part can be easily accessed. Provided is an endoscopic examination method for observing, using a second endoscopic system, an organ to be observed of a subject who accepts an examination of the organ using a first endoscopic system, the method comprising: acquiring time-series operation content information in the first endoscopic system as operation unit information (S31); and a control unit that estimates an operation procedure when the subject receives an examination using the second endoscope system, compares the estimated operation procedure with the operation unit information, and outputs operation guide information for operating the second endoscope system so as to observe a feature part in the organ to be observed using the second endoscope system (S37). The operation unit information is image change information estimated by using asymmetry of the organ to be observed.

Description

Technical Field

[0001] The present invention relates to a second endoscope system capable of acquiring time-series operation content information from a first endoscope system and performing an operation for inspection based on the operation content information, a first endoscope system used therefor, and an endoscope inspection method. Background Art

[0002] In the past, a technology for guiding and displaying processing steps during treatment has been proposed. For example, Patent Document 1 discloses a dental treatment assisting device, which includes: a camera unit that captures the inside of the oral cavity of a treatment target during dental treatment; a storage unit that stores a plurality of product information and processing step information for each use of each product; a captured image analysis unit that detects the treatment target in the captured image and determines the treatment step; a processing step control unit that selects the processing step corresponding to the treatment step; and a display control unit that arranges and displays the processing step and the captured image.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2020 / 075796 Summary of the invention

[0006] Problems to be solved by the invention

[0007] As described above, Patent Document 1 describes that the treatment steps are displayed during treatment, and dentists and the like can perform treatment according to the displayed treatment steps. When the affected part is treated again after dental treatment, the target part to be treated can be easily found. However, when re-examining the target part such as the affected part found during the first endoscopic examination, it is not easy to reach the target part again.

[0008] For example, in a specialized hospital, even if an endoscopic examination or treatment is performed by an endoscopy specialist (expert) (for example, using the first endoscope system described later), there is an inconvenience if the patient can only receive the examination in the same hospital when the process observation of the part to be examined / treated is performed later. Therefore, if the process observation can be entrusted to other medical institutions such as clinics, it is convenient from the perspective of the burden on the patient. However, in this case, it is necessary to find the part again in other medical institutions. On the other hand, it is not necessarily possible to correctly observe the process observation part in a small hospital such as a clinic. Therefore, it is desirable to assist in a mechanism that enables process observation even in other medical institutions with different resources such as equipment (for example, the second endoscope system described later) or doctors.

[0009] The present invention has been made in view of such circumstances, and an object of the present invention is to provide a second endoscope system, a first endoscope system, and an endoscope inspection method that can easily access a target site such as an affected part.

[0010] Means for solving problems

[0011] In order to achieve the above-mentioned purpose, the second endoscope system of the first invention observes the observation target organ of the subject who has undergone the organ examination using the first endoscope system, and the second endoscope system comprises: an acquisition unit, which acquires the time-series operation content information in the first endoscope system as operation unit information; an insertion operation determination unit, which estimates the operation process of the subject when undergoing the examination using the second endoscope system; and an operation guidance unit, which compares the operation process estimated by the insertion operation determination unit with the operation unit information, and outputs operation guidance information for operating the second endoscope system, so as to use the second endoscope system to observe the characteristic parts in the observation target organ, and the operation unit information is image change information estimated by utilizing the asymmetry of the observation target organ.

[0012] A second endoscope system according to a second invention is, in the above-mentioned first invention, characterized in that the asymmetry information of the observation target organ is determined based on the anatomical positional relationship between a plurality of sites in a specific organ.

[0013] Regarding a second endoscope system of a third invention, in the above-mentioned first invention, the operation unit information is information on an operation that continues within a predetermined time.

[0014] A second endoscope system according to a fourth invention is the third invention, wherein the operation unit information is an operation start image and information on the operation from the start to the end of the operation.

[0015] A second endoscope system of a fifth invention is the first invention, wherein the operation guidance information output by the operation guidance unit is guidance information for observing the characteristic portion of the observation target organ under the same observation conditions as those of the first endoscope system.

[0016] A second endoscope system according to a sixth invention is the first invention, wherein the operation unit information is continuous image change information indicating the same action.

[0017] A second endoscope system according to a seventh aspect of the present invention is the first aspect of the present invention, wherein the first direction is determined when detecting the asymmetry of the observation target organ.

[0018] A second endoscope system according to an eighth invention is characterized in that, when detecting the asymmetry of the observation target organ, the direction of liquid accumulation determined by the gravity direction or the direction determined by the positional relationship of the already detected in vivo structures is referred to.

[0019] Regarding a second endoscope system of a ninth invention, in the above-mentioned first invention, the operation unit information is determined reflecting an angle of rotation of an operation lever or a knob for rotating a distal end portion of the endoscope system.

[0020] A second endoscope system according to a tenth invention is the first invention, wherein the operation unit information is information that uses a process until the observation direction of the distal end portion of the endoscope system is changed as an operation unit.

[0021] Regarding the second endoscope system of the eleventh invention, in the above-mentioned tenth invention, the observation direction of the front end portion of the endoscope system is changed by twisting the endoscope system, applying an angle to the endoscope system, or pushing the endoscope system into the body.

[0022] A second endoscope system according to a twelfth invention is the first invention, wherein the operation unit information is information that uses a process until the shape of the observation target organ changes as an operation unit.

[0023] Regarding the second endoscope system of the thirteenth invention, in the above-mentioned twelfth invention, the operation unit information is information that takes the process until the shape of the estimated organ is changed by using the endoscope system to supply air and / or water and / or suction, or by pushing the endoscope system as the operation unit.

[0024] Regarding the second endoscope system of the fourteenth invention, in the above-mentioned twelfth invention, the operation unit information is information that takes the process until the state of the mucosa of the estimated organ is changed by spreading the pigment and / or dye using the first endoscope system, or by supplying water using the first endoscope system as the operation unit.

[0025] The second endoscope system of the fifteenth invention, in the above-mentioned first invention, has operation guide information for operating the second endoscope system to observe a characteristic part in the organ to be observed. When comparing the operation process estimated by the insertion operation determination unit and the operation unit information, a plurality of the operation unit information are compared. If a repeated part does not require process observation during observation, the operation unit information is corrected to exclude the operation of the repeated part for comparison.

[0026] A second endoscope system of a sixteenth invention is the first invention, wherein the characteristic portion in the observation target organ is observed by automatic operation under the same observation conditions as those of the first endoscope system based on the operation unit information.

[0027] The endoscopic examination method of the seventeenth invention uses a second endoscope system to observe the subject's observation target organ on a subject who has undergone organ examination using a first endoscope system, obtains time-series operation content information in the first endoscope system as operation unit information, estimates the operation process of the subject when undergoing examination using the second endoscope system, compares the estimated operation process with the operation unit information, and outputs operation guidance information for operating the second endoscope system, the operation guidance information is used to observe characteristic parts in the observation target organ using the second endoscope system, and the operation unit information is image change information estimated by utilizing the asymmetry of the observation target organ.

[0028] The first endoscope system of the eighteenth invention comprises: an input unit, which inputs images of the organs of the subject in a time sequence; an operation unit determination unit, which divides the images of the organs obtained in a time sequence into operation units and determines the operation performed according to each operation unit; a recording unit, which records the images in the operation unit and information related to the endoscope operation as operation unit information for each operation unit determined in the operation unit determination unit; and an output unit, which outputs the operation unit information recorded in the recording unit.

[0029] Regarding the first endoscope system of the nineteenth invention, in the above-mentioned eighteenth invention, the operation unit determination unit is divided into the operation units based on whether at least one of the insertion direction, rotation direction, and bending direction of the front end portion of the first endoscope has changed according to the image obtained by the camera unit.

[0030] A first endoscope system according to a twentieth aspect of the present invention is the eighteenth aspect of the present invention, wherein the operation unit determination unit determines the direction of the operation based on asymmetry of the anatomical structure with respect to the image acquired by the imaging unit.

[0031] A first endoscope system according to a twenty-first invention is the eighteenth invention, wherein the recording unit records a start image and an end image of the continuous images belonging to the operation unit, and records operation information indicating an operation state in the operation unit.

[0032] Regarding a first endoscope system of a twenty-second invention, in the eighteenth invention, the recording unit records operation information after a marked object located near the target is found.

[0033] In the endoscopic examination method of the twenty-third invention, images of the subject's organs are acquired in time sequence, the images of the organs acquired in time sequence are divided into operation units, the operation performed by the first endoscope is determined according to each operation unit, and according to each determined operation unit, the image in the operation unit and the information related to the endoscope operation are recorded as operation unit information in a recording unit, and the operation unit information recorded in the recording unit is output.

[0034] The second endoscope system of the twenty-fourth invention observes the internal organs of a subject who has undergone an organ examination using the first endoscope system, and the second endoscope system comprises: an input unit, which inputs recorded operation unit information for the subject who has undergone an organ examination using the first endoscope system; a camera unit, which acquires images of the internal organs of the subject in a time series; and an operation guidance unit, which divides the images acquired in a time series into operation units, estimates the operation state of the second endoscope system according to each operation unit, compares the estimated operation state with the operation unit information, and outputs guidance information for observation under the same observation conditions as those of the first endoscope system.

[0035] The program of the twenty-fifth invention enables a computer to execute the following steps, wherein the computer uses the second endoscope system to observe the observation target organ of the subject who has undergone the organ examination using the first endoscope system: obtains the time-series operation content information in the first endoscope system as operation unit information; estimates the operation process of the subject when undergoing the examination using the second endoscope system, compares the estimated operation process with the operation unit information, and outputs guidance information for observing the characteristic parts in the observation target organ under the same observation conditions as those of the first endoscope system, wherein the operation unit information is continuous image change information representing the same action estimated by utilizing the asymmetry of the observation target organ.

[0036] The program of the twenty-sixth invention enables a computer to execute the following steps: acquiring images of an organ of a subject in time sequence; dividing the images of the organ acquired in time sequence into operation units, and determining the operation performed by the first endoscope according to each operation unit; recording the image in the operation unit and information related to the endoscope operation as operation unit information in a recording unit according to each determined operation unit; and outputting the operation unit information recorded in the recording unit.

[0037] Effects of the Invention

[0038] According to the present invention, it is possible to provide a second endoscope system, a first endoscope system, and an endoscope inspection method that can easily access a target site such as an affected part. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A This is a block diagram showing a main electrical configuration of an endoscope system according to one embodiment of the present invention.

[0040] Figure 1B This is a block diagram showing a main electrical configuration of an endoscope system according to one embodiment of the present invention.

[0041] Figure 2 This is a diagram showing an example of a route to reach a target object such as an affected area, which is a mark, in the endoscope system according to one embodiment of the present invention.

[0042] Figure 3 This is a flowchart showing the operation of the first endoscope system in the endoscope system according to one embodiment of the present invention.

[0043] Figure 4 This is a flowchart showing the operation of the second endoscope system in the endoscope system according to the embodiment of the present invention.

[0044] Figure 5 It is a diagram showing the insertion process of the endoscope in the endoscope system according to one embodiment of the present invention.

[0045] Figure 6 It is a diagram showing the insertion process of the endoscope in the endoscope system according to one embodiment of the present invention.

[0046] Figure 7 FIG. 1 is a diagram showing an example of an image captured when an endoscope is inserted into the endoscope system according to the embodiment of the present invention.

[0047] Figure 8 FIG. 1 is a diagram showing an example of an image captured when an endoscope is inserted into the endoscope system according to the embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following describes an example of applying the present invention to an endoscope system as one embodiment of the present invention. The endoscope system takes the inspection / diagnosis / treatment (in this specification, the inspection / diagnosis / treatment are sometimes collectively referred to as inspection or inspection, etc.) of an endoscope professional as an ideal, and can record / transmit information that serves as a clue for reproducing the inspection, etc. Furthermore, in order to reproduce the inspection, etc. of the endoscope professional, the image during the inspection is recorded, and the image changes and the characteristics of the captured image are used.

[0049] Specifically, the operating status of the endoscopist is determined based on the image changes as described below. (1) When the change pattern of the inspection image is constant (for example, it is like the impression of driving in a tunnel), it is determined that the endoscope is moving straight. (2) When the change pattern becomes a special pattern, it is determined that the endoscope is rotated or twisted. In addition, in endoscopic images, since it is not easy to determine up, down, left, and right, up, down, left, and right are defined using the structure of the organ of the object of endoscopy (for example, when observed from the endoscope, in the throat, the vocal cord side is the bottom, in the stomach, the gastric angle side is the top, etc.). In this specification, the position (direction) in the endoscopic image is displayed using the anatomical frontal view (for anatomical frontal view, the position (direction) is displayed using the anatomical frontal view). Figure 5 (described later) Using this information as clues, even a non-endoscopy specialist can reproduce an ideal examination.

[0050] use Figure 2 This article describes a method for recording the examinations etc. performed by endoscopy specialists and for non-specialized doctors to reach target areas such as affected areas based on the recorded examinations etc. Figure 2 In the above, Tg is the target part such as the affected part found by the professional doctor, and the non-professional doctor operates the endoscope so as to reach the target part Tg. In addition, Ob ​​is an object that becomes a mark on the route to reach the target part Tg. If the mark Ob is found, the mark Ob is used as a clue to find the target part Tg. In addition, the target part is Figure 2 Only one is depicted, but there can be more than one.

[0051] The operation process until the target site Tg is found is described in detail. Figure 2 In the example, the professional doctor operates the first endoscope system and moves straight along route R1, and changes the moving direction by bending or rotating the endoscope at position L1. After that, the endoscope is further moved along route R2, and changes the moving direction to route R3 by bending or rotating the endoscope at position L2. If the endoscope moves forward in this state, the image of the marker Ob is captured at position L3, and the route is changed to route R4 at position L4, and the target part Tg such as the affected part is finally found.

[0052] At this time, the image acquired by the first endoscope system and the operation status at that time are recorded in units of operation. That is, the image until the image changes due to the insertion direction, rotation, bending operation, etc. of the front end portion of the first endoscope system is recorded as an operation unit. Figure 2In the example, from the operation start along route R1 to position L1 is one operation unit, from position L1 along route R2 to position L2 is one operation unit, from position L2 along route R3 to mark Ob at position L3 is one operation unit, from position L3 along route R4 to position L4 is one operation unit, and from position L4 along route R5 to target part Tg is one operation unit. Figure 2 In the description, for simplicity, the movement is described as two-dimensional up, down, left, and right. However, since the movement is actually in a three-dimensional structure, it is also possible to imagine an operation accompanied by rotation of the screen or an operation of changing the viewing position up, down, left, and right.

[0053] As described above, when the professional doctor records information such as the image and operation status until reaching the target site Tg such as the affected part in the recording unit, an operation guide (operation suggestion) can be generated based on the information. The non-professional doctor can easily reach the target site Tg by operating the second endoscope system while receiving the operation guidance. That is, in the case where the non-professional doctor uses the second endoscope system to examine the same patient (subject) as the professional doctor, the non-professional doctor can easily reach the target site Tg by receiving the operation guidance. Figure 2 Target site Tg indicated.

[0054] For example, when a non-professional doctor inserts the front end of the second endoscope system into the body of a patient (subject) and reaches position L1, operation guidance information such as rotation or bending is displayed, and then operation guidance information is displayed at position L2, position L3 (position of mark Ob), position L4, etc. In addition, reference guidance information is also displayed outside this position. In this specification, when guidance display indicating the upper direction, lower direction, right direction, left direction and operation direction is performed at each position, the anatomical normal position is used to display the direction in the image.

[0055] In addition, in the case of only checking the process observation part, it is possible to waste if the process of the inspection performed during the previous diagnosis / inspection is reflected in the operation unit. For example, in the previous diagnosis / inspection, when the observation was performed in the order of A→B, B→C, C→B, and B→D, that is, after observing in the order of A, B, and C, returning to B and then observing D, if it is judged that C does not need process observation after the next time, then in the next diagnosis / inspection, it is sufficient to guide in the order of A→B, B→D. That is, it is also possible to omit the guidance of "B→C, C→B". For example, in the case of observing the lesser curvature side after inserting the stomach, there is a situation where the greater curvature side is temporarily observed from the ventral valve and then goes to the lesser curvature. If the part to be observed in the process is the lesser curvature side, the part where the professional doctor (expert) observes the greater curvature is omitted, and only the entry from the cardia and bending to the lesser curvature side (that is, the action of swinging from the cardia insertion to the greater curvature and then returning to the vicinity of the cardia insertion) is recorded as the operation unit.

[0056] In this way, as a result of the observation, in order to cope with the situation where the process observation target part is not needed, the possibility of omitting the repeated part of the repeated operation is determined, and if it can be omitted, the part that is not the observation target part is not guided. The reason why such repeated operation can be omitted is that it is decomposed into operation units. If there is an identification signal indicating whether it is the process observation target or not at each point, the operation unit for guidance can be narrowed down according to the identification signal, so that the operation guidance can be simplified.

[0057] use Figure 2 The "omission operation" of determining the possibility of omitting the above-mentioned repeated operation and omitting the repeated part of the operation when possible is explained. For example, suppose that in the last inspection, from position L1 through position L2, to the position in front of it, observation is performed at this position (for example, L2a), and then return to position L2, and then observe in the order of L3 (marker Ob), L4, Tg (target object). In the case where it is determined that the process observation at position L2a is not necessary in the last inspection, the operation guidance from position L2 to position L2a can also be omitted in the next and subsequent process observations. In this case, in the next and subsequent process observations, the operation guidance can be changed in the order of position L1→position L2→position L3.

[0058] In the case where the operation unit information remaining as history is "first advance and then return" or "observe the right side and then observe the left side" and a specific guidance starting point can be reset, it is also possible to correct the multiple operation unit information contained in the history, generate the operation unit information for guidance with the guidance starting point reset, and refer to the operation unit information for guidance. That is, when generating the operation guidance information for the operation of the endoscope during the re-inspection or process observation for observing the characteristic part of the observation object organ, the operation process estimated by the insertion operation determination unit and the operation unit information are compared. At this time, based on the operation unit information group obtained during the previous inspection, multiple operation unit information adjacent in time are compared. If the repeated part during observation does not require process observation, the operation unit information corrected to the operation of removing the repeated part can be compared. It is also possible to strictly correct the operation unit information for guidance to generate new data equivalent to the operation unit information for comparison, and guidance can be given in a predictable manner even if new data is not specially generated.

[0059] use Figure 5 The method of displaying directions in an image based on the anatomical normal position is described. The front end of the endoscope is cylindrical, and a camera unit is arranged therein. It is difficult to know which direction is up or down, or right or left in an image obtained by photographing the inside of the digestive tract, etc., which is formed into a complex shape. That is, in general landscape photos or portrait photos, it is possible to understand which direction is up or down, or front or back in the picture from the image. In contrast, in an image photographing the inside of the digestive tract, etc., it is difficult to determine the direction from the image, and some definition for indicating the direction is required.

[0060] Therefore, the anatomical position is used to indicate the direction. The anatomical position is a posture in which the person stands straight with the palm facing the front (the side facing the face), and the direction is expressed based on the anatomical position. This premise is particularly useful in expressing parts such as limbs whose directions are easily changed. However, even if the anatomical position is used as a premise, the expression of the directions of the limbs, brain, etc. is likely to cause confusion, so the easy-to-understand expression described later is preferred.

[0061] In the anatomic frontal view, regarding up and down, the direction where the head is located is set to superior, and the direction where the feet are located is set to inferior. In addition, regarding left and right, it is expressed in terms of left and right as viewed from the person being observed. That is, when a doctor faces a patient, the left half of the patient's body is located on the right side as viewed from the doctor. If the doctor observes the back of the patient, the right half of the patient's body is located on the right side as viewed from the doctor. In addition, regarding front and back, the face is facing the front (anterior), and the back is facing the back (posterior).

[0062] Figure 5 In addition, when performing gastric endoscopy, the entire body is actually placed in the horizontal position (left side lying position) for examination, but in Figure 5 In the figure, for the convenience of drawing, the head is drawn horizontally (leftward), and the lower part of the neck is drawn forward. Figure 5 In the example shown, the insertion route Ro of the endoscope is indicated by a dotted line. The front end of the endoscope is inserted from the oral cavity OC (or sometimes from the nasal cavity NC depending on the model of the endoscope) and enters the stomach St through the esophagus ES. Before passing through the esophagus ES, there is a branch point of the vocal cords Vc and the trachea Tr. If the endoscope mistakenly advances in the direction of the vocal cords Vc, it will not reach the target site.

[0063] Figure 5 The image P5A in the figure is an image of the front end of the endoscope inserted from the oral cavity OC before entering the esophagus ES. In the image P5A, the lower side is the front side in the anatomical position, which is the vocal cords VC and trachea Tr, and the upper side is the back side in the anatomical position, which is the esophagus ES. In order to insert the front end of the endoscope toward the stomach, Figure 5 In the image P5A, the endoscope can move forward in the esophagus ES in the upper direction of the screen and in the posterior direction of the anatomical position. In fact, in the image before the endoscope enters the esophagus, it is difficult to know which direction is up or down, so the direction is expressed according to the anatomical position. In this case, the organ has asymmetry, and this asymmetry can be used to determine the direction of the continuous action. In this regard, use Figure 6 as well as Figure 7 Provide explanation.

[0064] In addition, Figure 5 In the case where the front end of the endoscope is advanced toward the duodenum, it is advanced toward the pylorus Py. When advancing from the stomach St toward the pylorus Py, the front end of the endoscope is advanced along the wall surface of the stomach St, and the direction of the front end of the endoscope is changed to a direction in which the pylorus Py can be seen. Figure 5The image P5B shown is an image of the pylorus Py as viewed from the side. When the image P5B is viewed, the distal end portion of the endoscope may be bent to change the direction of the distal end portion.

[0065] In this way, when a non-professional doctor operates the second endoscope system to reach the target site Tg such as the affected part, the front end of the endoscope must be directed in an appropriate direction at the branch point, etc., which is not easy. In this embodiment, information on the operation of the first endoscope system 10A by the professional doctor to reach the target site Tg such as the affected part is recorded in advance, and when the non-professional doctor examines the same subject, the operation guidance information is displayed based on the recorded information. The non-professional doctor can easily find the target site such as the affected part found by the professional doctor by operating according to the operation guidance information.

[0066] Next, use Figure 6 (a), (b) and Figure 7 , the operation state when the front end portion of the endoscope is inserted into the body cavity and the continuous images obtained at this time are explained. Figure 6 The endoscope EDS is inserted from the subject's oral cavity OC and passed through the subject's stomach St to examine the pylorus Py. Figure 6 (a) and Figure 5 Similarly, for ease of drawing, the head is drawn horizontally (towards the left), and the neck and lower side are drawn forward.

[0067] Figure 6 (a) shows the case where the endoscope EDS is placed into the oral cavity OC. Figure 6 (b) shows a state where the endoscope EDS is inserted into the esophagus ES and the stomach St. Figure 7 The images P6a to P6f obtained when the endoscope EDS is inserted into the digestive organ are shown, and these images change all the time. Images P6a to P6c show the situation where the endoscope EDS is inserted into the esophagus ES at time T1 to T3. At this time, the front end of the endoscope EDS is not rotated or bent, but moves straight forward. Therefore, the elliptical shape (hole shape) of the esophagus ES gradually becomes larger.

[0068] Images P6a to P6c are images in the first operation unit. At time T4, the endoscope EDS is rotated, and the elliptical (hole-shaped) protrusion of the image obtained at this time rotates. Images P6c to P6d are images in the second operation unit. From time T3 to time T4, the front end of the endoscope EDS rotates in order to find the pylorus Py in the stomach St. That is, if the front end of the endoscope EDS moves downward along the wall of the stomach St for a specified distance, it reaches the vicinity of the pylorus Py, so the front end of the endoscope EDS is rotated at this time to find the pylorus Py. If the pylorus Py is found, it is possible to enter the duodenum.

[0069] At time T5, the endoscope EDS performs a bending operation, and the center portion of the elliptical shape (hole shape) of the image acquired at this time moves. In addition, images P6e to P6f are images of the third operation unit.

[0070] In this way, the internal organs have asymmetry, so the asymmetry can be used to determine the continuity of the action, and a series of continuous images until the continuity of the action is interrupted are used as the operation unit. In a tubular endoscope, there are actions such as insertion, withdrawal, bending of the front end (in four directions), twisting, etc., and several components of these actions are mixed. Therefore, if the asymmetry information is not used, it is not easy to analyze as an operation unit. For example, since there is only a symmetrical space but not asymmetry in the tube shape, when inserted into the tube shape, even if it is twisted in the middle, it cannot be decomposed into an operation unit, but becomes an asymmetric space in the internal organs. That is, the operation unit information is the continuous image change information representing the same action estimated by using the asymmetry of the observed organ.

[0071] Specifically, in Figure 7 In the above, as shown in time T1 to T3, when the shape of the organ is similar and the size gradually changes, the image changes when the front end of the endoscope is inserted. In addition, as shown in time T3 to T4, when the shape is the same (same size) and the position of the protrusion is rotated, the image is when the front end of the endoscope is rotated. And, as shown in time T5 to T6, when the shape is the same (same size) and the center position moves, the image is when the front end of the endoscope is bent. If the asymmetry of the internal organs is used, it is possible to determine whether the same action is continuous. That is, the operation unit information is the continuous image change information indicating the same action estimated by using the asymmetry of the observed organ.

[0072] In addition, Figure 7 In FIG. 1 , the straight-movement operation, the rotation operation, and the bending operation are each a separate operation. However, in reality, in addition to separate operations, there are also cases where multiple operations are performed in combination. In this case, the operation information can be obtained by decomposing each operation by utilizing the asymmetry of the internal organs. Figure 7The idea described in the above can make judgments on the insertion direction, extraction direction, twisting direction, front end bending direction, and decomposition of each operation mode (including direction and amount), and can separate and judge what operation mode is being combined at this time. In the future, it is possible to develop an endoscope capable of bending operations other than the front end, an endoscope with specifications that has functions similar to a zoom lens in addition to bending in directions other than up, down, left, and right, and can control the front end to approach and move away, etc., but in this case, of course, the present embodiment can also be applied with the same idea.

[0073] Furthermore, the operation unit information is not necessarily limited to operations related to changes in the observation position of the front end of the endoscope. That is, regarding operations that improve the observation state, visual recognition or detectability by changing the state of the object part or the object that blocks the object part during observation using the endoscope, it is also preferred to hand over information, transfer technology or apply for it. If these operations are not performed, it may also be impossible to compare images with previous inspections during process observation. The following operations are operations that are often performed when observing the object part using the endoscope system, and are preferably considered as operation units. For example, by spreading pigments and dyes, the shape of the concave and convex parts of the object part, the difference between the lesion part and the normal part can sometimes be seen well. In addition, by using the endoscope system to deliver water (for the purpose of cleaning mucus, etc.), visual recognition is sometimes improved. That is, through active actions other than position change, the state of the mucosa of the estimated organ sometimes changes, and it is also important to use the process until the state of the mucosa of the organ changes as an operation unit.

[0074] In addition, as described above, the operation unit information is continuous image information representing the same action. Here, the same action includes inserting the knob by a certain amount, twisting the knob by a certain amount, or bending the front end by a certain amount (in Figure 7 The same action is preferably divided into a time period (for example, several seconds to several tens of seconds) that is easy for the operator of the second endoscope system to refer to for guidance. In addition, for experts who are familiar with the operation, it is possible to perform the guidance in an easy-to-understand manner because the operation is performed in a time range that does not include other operations. That is, it can be divided into two components, namely, insertion and twisting, so that the guidance can be performed in a time period that is difficult to understand. In various cases, it is possible to perform the guidance in a timely manner. Figure 7 The consideration method described in is to separate and determine what operations are compounded.

[0075] The aforementioned Figure 6 The route of the endoscopic EDS when inserted from the oral cavity OC towards the pylorus Py is shown. Figure 8 Examples of images obtained by the endoscope EDS during insertion are shown. Image P11 is an image of the vocal cords VC observed from above, and image P12 is an image of the esophagus ES observed from above. Image P13 is an image of the endoscope entering the stomach St, image P14 is an image of the pylorus Py observed from above, and image P15 is an image of the pylorus Py observed from the side. As can be seen from these figures, the internal organs are not symmetrical but asymmetrical, and therefore, by utilizing this asymmetry, it is possible to estimate the operation transition during continuous action and determine the operation unit based on the interruption of the operation.

[0076] In this embodiment, as described later, when a professional doctor operates an endoscope, continuous images are divided into operation units, and operation unit information (for example, referring to Figure 1A Operation unit information 35b, Figure 3 The operation unit information can be said to be continuous image change information representing the same action estimated by utilizing the asymmetry of the observed organ (for example, referring to Figure 7 ). In addition, the asymmetry information of the observed organ is determined based on the anatomical positional relationship of multiple parts in the specific organ. It can also be consistent with the anatomical direction expression. In addition, the operation unit information is information related to the operation that continues within a specified time. In addition, the operation unit information can also be an operation start image and information related to the operation from the start to the end of the operation. The operation unit information can also include an end image, and / or information that serves as a mark for discovering the target part, and / or information related to the target part, and / or pre-discovery operation information (for example, refer to Figure 4 S41, S48).

[0077] Furthermore, when a professional doctor uses an endoscope to perform an examination, the operation is performed by operating an operating unit such as an operating lever or a knob for rotating the front end of the endoscope system. When the operation is performed, the first operation unit is often changed to the second operation unit. Therefore, the operation unit information may be determined in a manner reflecting the angle of the operating lever or the knob for rotating the front end of the endoscope system. Furthermore, the operation unit information may be information that takes the process until the observation direction of the front end of the endoscope system is changed as the operation unit. The observation direction of the front end of the endoscope system can be changed by twisting the endoscope system, by applying an angle to the endoscope system, or by pushing the endoscope system into the body.

[0078] In addition, the operation unit information may be information that takes as an operation unit a process until the shape of the observed organ changes. The operation unit information may be information that takes as an operation unit a process until the shape of the estimated organ changes by using the endoscope system to supply air, water, suction, or push the endoscope system. The operation unit information is information that takes as an operation unit a process until the state of the mucosa of the estimated organ changes by using the first endoscope system to spread a pigment / stain or using the first endoscope system to supply water (for the purpose of mucosal cleaning). That is, the operation unit information is not limited to operations related to changes in the observation position of the front end of the endoscope, and the operation unit information may also include operations that change the state of the object part or the object that blocks the object part during observation using the endoscope to improve the observation state, visual recognition, and detectability.

[0079] In addition, if you use Figure 5 As described above, in this embodiment, the anatomical normal position is used to indicate the direction. Therefore, the first direction can also be determined when the asymmetry of the observed organ is detected. In addition, when the asymmetry of the observed organ is detected, the direction of the accumulation of the liquid determined by the gravity direction or the direction determined by the positional relationship of the structures in the body that have been detected can also be referred to.

[0080] In addition, in this embodiment, when a professional doctor operates the endoscope, the operation unit information is recorded, and when a non-professional doctor operates the endoscope, the operation guide is displayed according to the operation unit information. That is, the non-professional doctor performs manual operation according to the operation guide, thereby observing the observation target organ. However, this is not limited to this, and the characteristic part in the observation target organ can also be observed by automatic operation according to the operation unit information under the same observation conditions as the first endoscope system.

[0081] Furthermore, in the present embodiment, when a non-professional doctor uses the second endoscope system to perform an examination, etc., guidance information is output in a manner that enables the non-professional doctor to observe the characteristic part of the observation target organ under the same observation conditions as the first endoscope system, similarly to when the non-professional doctor performs the examination, etc. (for example, referring to Figure 4 Here, as the same observation conditions, there are the size of the object captured in the screen, the angle of view, etc., which are conditions for making the positional relationship between the camera unit and the observation object the same when observing the observation object. In addition, the conditions for making the illumination and exposure control of the observation object the same so as to be able to determine the change in color tone, the focus, the field of view angle (including the position of the person observed last time and the person observed this time, etc.) and other optical conditions can also be made consistent.

[0082] Next, use Figure 1A and Figure 1B The structure of an embodiment of an endoscope system to which the present invention is applied is described. The endoscope system is composed of a second endoscope system, which observes the subject's observation target organ of a subject (including a patient) who has undergone an organ examination (including diagnosis / treatment) using a first endoscope system. Specifically, the endoscope system of this embodiment is composed of an endoscope system 10A, an auxiliary device 30 provided in an in-hospital system / server, etc., and a second endoscope system 10B. Here, for the sake of convenience, the endoscope system 10A and the second endoscope system 10B are, for example, endoscopes inserted from the oral cavity through the esophagus to examine the stomach or duodenum, and the case of performing an endoscopic examination of the stomach or duodenum of the subject is described as an example. The endoscope system 10A is an endoscope used for the first examination of the subject, and the second endoscope system 10B is described as an endoscope used for the second and subsequent examinations of the subject. The endoscope system 10A and the second endoscope system 10B may be endoscopes of the same model, but are described herein as different models.

[0083] In addition, when the second endoscope system 10B and the endoscope system 10A are of the same model, they can be the same device or different models / devices. That is, if the examination is performed at different times, it is sometimes impossible to perform exactly the same examination depending on the changes in conditions (including the patient's affected part, physical and mental constraints such as replacement of doctors, fatigue, habits, etc., even in a healthy state, or surrounding conditions such as assistants, peripheral equipment, and environment). Therefore, in this embodiment, when multiple examinations are performed at different times (in most cases, the examination dates are different, but it is also conceivable that there may be another examination on the same day), it is sufficient as long as the information can be inherited.

[0084] The endoscope system 10A is used when a doctor observes the inside of the esophagus, stomach, and duodenum from the pharynx and performs inspection, treatment, surgery, etc. The endoscope system 10A includes a control unit 11A, an imaging unit 12A, a light source unit 13A, a display unit 14A, an ID management unit 15A, a recording unit 16A, an operation unit 17A, a derivation engine 18A, a clock unit 20A, and a communication unit 21A. In addition, the above-mentioned units can be provided in an integrated device, or can be dispersedly configured in multiple devices.

[0085] The control unit 11A is composed of one or more processors including a processing device such as a CPU (Central Processing Unit), a memory storing a program (the program may also be stored in the recording unit 16A), etc., and executes the program to control each unit in the endoscope system 10A. The CPU of the control unit 11A cooperates with the CPU of the control unit 31 of the auxiliary device 30 to execute the program to achieve Figure 3 The control unit 11A performs various controls when the endoscope system 10A performs endoscopic examination of a subject (patient), and performs control for transmitting image data P1 acquired during the examination to a support device 30 installed in an in-hospital system or server.

[0086] The imaging unit 12A is provided at the front end of the insertion portion of the endoscope system 10A inserted into the body, and has an optical lens, an imaging element, an imaging circuit, an image processing circuit, etc. The imaging unit 12A is composed of a small imaging element and an imaging optical system that forms an image of an object on the imaging element, and assumes that the specifications such as the focus position and the focal length of the optical lens have been determined. In addition, the imaging unit 12A can also be provided with an automatic focusing and depth of field expansion function (EDOF function), in which case the distance to the object, the size of the object, etc. can be determined. If the shooting field angle of the imaging unit 12A is about 140 degrees to 170 degrees, it can be photographed in a wide range. The imaging optical system can also have a zoom lens. The imaging unit 12A obtains image data of a dynamic image at a predetermined time interval determined by a frame rate, and records the image data in the recording unit 16A after image processing. In addition, when the release button in the operating unit 17A is operated, the imaging unit 12A obtains static image data and records the static image data in the recording unit 16A. The imaging unit 12A functions as an imaging unit that acquires images of the subject's organs in a time series manner (for example, see Figure 3 of S1).

[0087] Image P1 is an image obtained by the camera unit 12A and is sent to the input unit 32 of the auxiliary device 30 through the communication unit 21A. Image P1 is a time-series image. Image P11 is an image obtained immediately after the front end of the endoscope system 10A is inserted into the oral cavity, and image P20 is an image obtained just before the endoscope system 10A is pulled out of the oral cavity. Images P11 to P16 are continuous images belonging to the operation unit. Similarly, images P15 to P19 are also images belonging to other operation units. If using Figure 2 As described above, the operation unit is a series of image groups until the image pattern is changed by the insertion direction change operation, rotation operation, distal end bending operation, etc. before the professional doctor reaches the target site such as the affected area.

[0088] In addition, Figure 1A In the example, only two operation units are shown, but sometimes there are three or more operation units depending on the inspection content. Figure 1AIn the example, images P11 to P16 are the first operation unit, and images P15 to P19 are the second operation unit. In this example, images P15 and P16 overlap with the first and second operation units. However, images may not overlap between two operation units, and images between two operation units may not belong to the operation unit (the latter case is equivalent to the case where no operation is performed).

[0089] The light source unit 13A includes a light source and a light source control unit. The light source unit 13A illuminates the object with appropriate brightness. The light source is arranged at the front end of the endoscope system 10A in order to illuminate the affected part and the like inside the body, and the light source control unit controls the illumination of the light source. In addition, by adjusting the brightness and measuring the brightness of the reflected light, the distance of the object can be measured. In addition, light consisting of a prescribed pattern (light consisting of a bright part and a dark part) can also be irradiated, and the distance and the concavity and convexity of the object can be measured based on the difference between the irradiated pattern and the photographed pattern. In this case, it is preferable to use light in a wavelength region other than visible light such as infrared light.

[0090] The display unit 14A displays the image inside the body based on the image data obtained by the camera unit 12A. In addition, the display unit 14A can display the operation guidance overlapping with the inspection image. For example, the vicinity of the indicated part (affected part) is displayed. The operation guidance can also be displayed based on the derivation result of the derivation engine 18A. In addition, it can also display a menu screen for the operation or display of the endoscope system 10A.

[0091] When a professional doctor uses the endoscope system 10A to perform an examination, the ID management unit 15A manages the ID for identifying the subject (patient). For example, the professional doctor may input the subject's (patient's) ID through the operation unit 17A of the endoscope system 10A. In addition, the ID management unit 15A may associate the ID with the image data obtained by the imaging unit 12A.

[0092] The recording unit 16A has an electrically rewritable nonvolatile memory, and records adjustment values ​​for operating the endoscope system 10A, programs used in the control unit 11A, and the like. In addition, the image data obtained by the imaging unit 12A is recorded. The operating unit 17A has various operating units, such as an operating unit (also referred to as an interface) for bending the front end of the endoscope system 10A in an arbitrary direction, an operating unit for a light source, an operating unit for capturing images, and an operating unit for treatment instruments. The ID of the subject (patient) can also be input through the operating unit 17A.

[0093] The derivation model is configured in the derivation engine 18A. The derivation model may also be composed of various derivation models such as a derivation model for derivation of a part that may be an affected part, such as a tumor or a polyp, in the image acquired by the imaging unit 12A, and an operation guide when operating the endoscope system 10A. The derivation engine 18A may be composed of hardware, software (program), or a combination of hardware and software.

[0094] The clock unit 20A has a calendar function and a clock function. When the image data is acquired by the camera unit 12A, the acquisition date and time can be output, and the elapsed time from the start of the examination can also be output. When the image data is recorded in the recording unit 16A, the time information can also be recorded. In addition, when the image data is output from the communication unit 21A to the auxiliary device 30, the time information can also be associated and output. In addition, when the image is acquired in the operation unit, the time information output from the clock unit 20A can also be associated with the image data.

[0095] The communication unit 21A has a communication circuit (including a transmission circuit and a reception circuit) and exchanges information with the auxiliary device 30. That is, the image data obtained in the camera unit 12A is sent to the auxiliary device 30. In addition, the communication unit 21A can also communicate information with the second endoscope 30 in addition to the auxiliary device 30. In addition, the communication unit 21A can also communicate with other servers and in-hospital systems. In this case, it is possible to collect information from other servers and in-hospital systems and provide information. In addition, it is also possible to receive a derivation model generated by an external learning device.

[0096] The auxiliary device 30 is set in the in-hospital system, server, etc. The in-hospital system is connected to one or more endoscopes and other equipment, personal computers (PCs), portable devices such as smart phones, etc. in the hospital through wired communication or wireless communication. The server is connected to the endoscope and other equipment, the in-hospital system, etc. through a communication network such as the Internet and an intranet. The endoscope system 10A can be connected to the auxiliary device 30 in the in-hospital system, or directly connected to the auxiliary device 30 in the server, or connected to the auxiliary device 30 through the in-hospital system.

[0097] The auxiliary device 30 includes a control unit 31, an input unit 32, an ID management unit 33, a communication unit 34, a recording unit 35, an inference engine 37 having an inference model set therein, and an operation unit determination unit 37. In addition, the above-mentioned components may be provided in an integrated device, or may be dispersedly configured in a plurality of devices. Furthermore, the components may be connected via a communication network such as the Internet or an intranet.

[0098] The control unit 31 is composed of one or more processors including a processing device such as a CPU (Central Processing Unit), a memory storing a program (the program may also be recorded in the recording unit 35), etc., and executes the program to control each unit in the auxiliary device 30. When a non-professional doctor uses the second endoscope system 10B (which may also be the endoscope system 10A) to perform the same or similar examination on the same subject (patient) after a professional doctor uses the endoscope system 10A, the control unit 31 controls the entire auxiliary device 30 so that it can output an operation guide for finding a part corresponding to the affected part of the subject (patient). The CPU of the control unit 31 of the auxiliary device 30 cooperates with the CPU of the control unit 11A to execute the program to achieve Figure 3 In this embodiment, the CPU in the processor and the program stored in the memory realize functions such as the operation unit determination unit.

[0099] The input unit 32 has an input circuit (communication circuit) that inputs the input image P1 obtained by the camera unit 12A. The operation unit determination unit 37 determines the image group of the operation unit for the image P1 input by the input unit 32. The image group is output to the deduction engine 37, and the deduction engine 37 uses the deduction model to deduce the operation information for reaching the position of the target part such as the affected part, and outputs the operation information Iop. The operation information Iop includes the operation information for operating the operation unit and the endoscopic image at that time, etc. In addition, in the present embodiment, the operation information Iop is output by deduction using the deduction model, but the operation information Iop may also be output based on the similarity determination of the image. The input unit 32 functions as an input unit that inputs the image of the subject's organ in a time series (for example, refer to Figure 3 of S1).

[0100] The ID management unit 33 manages the ID of the subject (patient). As described above, when a professional doctor uses the endoscope system 10A to perform an examination, etc., the ID of the subject (patient) is input, and the image P1 associated with the ID is sent from the endoscope system 10A. The ID management unit 33 associates the ID associated with the image P1 with the ID information of the subject (patient) recorded in the recording unit 35, etc. Furthermore, when a non-professional doctor uses the second endoscope system 10B to perform an examination, etc., the required operation information Iop is output based on the ID information.

[0101] The communication unit 34 has a communication circuit, and exchanges information with the endoscope system 10A and the second endoscope system 10B. In addition, the communication unit 34 can also communicate with other servers and in-hospital systems, in which case it is possible to collect information from other servers and in-hospital systems and provide information. The operation information Iop generated in the derivation unit 36 ​​is sent to the second endoscope system 10B through the communication unit 34. In this case, the operation information Iop corresponding to the ID of the subject examined using the second endoscope system 10B is sent to the communication unit 21B of the second endoscope system 10B through the communication unit 34. The communication unit 34 functions as an output unit that outputs the operation unit information recorded in the recording unit (for example, refer to Figure 3 S23).

[0102] The recording unit 35 has an electrically rewritable nonvolatile memory, and can record image data input from the imaging unit 12A by the input unit 32, information such as the profile / examination history / examination results of the subject (patient), and programs used in the control unit 31. In addition, when the subject (patient) is examined using the endoscope system 10A (which may also include the second endoscope system 10B), the recording unit 35 may also record image data based on the image P1 at that time, and also record operation information Iop derived and output by the derivation engine 37.

[0103] The recording unit 35 records the inspection image 35a and the operation unit information 35b. As described above, when the subject (patient) is inspected using the endoscope system 10A, the recording unit 35 records the image data based on the image P1 at that time. This image data is recorded as the inspection image 35a.

[0104] The operation unit information 35b is recorded for each ID of the subject (patient) who receives the examination (including diagnosis / treatment) using the endoscope system 10A. In this case, there is a case where one subject receives multiple examinations, so it is possible to distinguish them based on the examination date and time. Figure 7 As described above, there are a plurality of operation units in one examination or the like, and therefore the operation unit information 35b records the start image 35ba, the end image 35bb, the operation information 35bc, and the time information 35bd for each operation unit.

[0105] The operation unit information 35b records a start image 35ba, an end image 35bb, operation information 35bc, and time information 35bd. The start image 35ba is the first image that is determined to belong to the operation unit by the operation unit determination unit 37. For example, in the image group P1, image P12 is the start image that belongs to the first operation unit, and image P15 is the start image that belongs to the next operation unit. The end image 35bb is the last image that is determined to belong to the operation unit by the operation unit determination unit 37. For example, in the image group P1, image P16 is the end image that belongs to the last operation unit, and image P19 is the end image that belongs to the next operation unit. In addition, image P11 is an image when the endoscope is inserted, and image P20 is an image when the endoscope is removed.

[0106] The operation information 35bc is information related to the operation state of the endoscope system 10A, and the operation information is recorded according to each image data and / or operation unit. The operation information can also be obtained based on the change of the image obtained by the camera unit 12A. As described above, if the asymmetry of the internal organs is used, it can be determined whether it is the same continuous action, and the same continuous action can be determined as one operation unit. For example, when a professional doctor performs a straight operation, a rotation operation, a bending operation, etc. on the front end of the endoscope system 10A, the image changes according to the operation. In addition, when a water injection operation, a suction operation, etc. are performed, the image also changes. Based on these image changes, the control unit 31 and the like obtain the operation information and record it as the operation information 35bc. In addition, in addition to obtaining the operation information based on the image, for example, if the operation information performed by the operation unit 17A in the endoscope system 10A is associated with the image data and sent to the auxiliary device 30, the associated operation information can also be obtained.

[0107] The time information 35bd is the time information of each image of the operation unit. For example, the time information may be information indicating the time, minute, and second of the year, month, and day. In addition, the start of the operation may be used as the reference time, and the elapsed time from the reference time may be used as the time information.

[0108] In addition, as the operation unit information 35b, an object serving as a mark of the target site such as an affected site is specified near the target site (see Figure 2 Ob), and also records the image of the mark Ob (which may also include position information) (refer to Figure 3 In addition, the image of the target part Tg (which may also include position information) is recorded as the operation unit information 35b. In addition, the operation information performed by the professional doctor from the discovery of the mark to the arrival of the target is also recorded in the recording unit 35 as the operation unit information 35b (refer to Figure 3S19).

[0109] The recording unit 35 functions as a recording unit that records, for each operation unit determined by the operation unit determination unit, an image in the operation unit and information related to the endoscope operation as operation unit information (for example, refer to Figure 3 The recording unit records the start image and the end image in the continuous images belonging to the operation unit, and records the operation information indicating the operation state in the operation unit (for example, referring to Figure 3 The recording unit records the operation information after the mark located near the target is found (for example, referring to Figure 3 S17, S19).

[0110] The operation unit determination unit 36 ​​performs determination for dividing the image into each operation unit for the image inputted in time series by the input unit 32 (for example, referring to Figure 3 S7, S11, etc.). That is, it is determined based on the image whether it is an image of a situation where the same operation / action is being performed continuously. For example, suppose that a professional doctor advances the front end of the endoscope in a straight line, and at a certain time, advances the front end while bending the front end, and then advances the front end in a straight line again after a short time. In this case, the image during the forward operation until the bending operation becomes an operation unit, and then the image from the bending operation to the straight forward operation again becomes an operation unit. In addition, the operation of the professional doctor is not limited to only one, and sometimes a complex operation is performed. For example, there is a situation where a bending operation or a rotation operation is performed while advancing forward. There are cases where such a complex operation is best distinguished from a simple operation and cases where it is best distinguished from a simple operation, so it can be distinguished based on the operation state including the start / end time point of the operation.

[0111] In addition, the operation unit determination unit 36 ​​determines the direction of the operation (for example, referring to Figure 3 As described above, it is not easy to express the direction in the body cavity, such as the front direction, the rear direction, the right direction, and the left direction, in which the front end of the endoscope is facing (for example, referring to Figure 5 ) Therefore, in this embodiment, the direction of the operation is determined based on the asymmetry of the anatomical structure.

[0112] In addition, the determination of the operation unit can be performed based on information such as operation information attached to the image data, in addition to the image and the operation information. In addition, a sensor can be set at the front end and / or the insertion part and / or the operation part of the endoscope, and the operation information is obtained according to the output from the sensor. If a sensor is set at the so-called coiled tube part, the shape of the mirror body can be identified, and as a result, the situation such as pressing the greater curvature of the stomach can be more accurately grasped. In addition, in order to detect button operations (only blocking the hole in the case of air supply) and angle operations with higher accuracy, a sensor can also be installed on the operation part. In addition, a transmission source can be set at the front end of the endoscope, and a sensor for detecting the signal from the transmission source can be set outside the body, and the operation information is obtained according to the output from the sensor. The operation unit information determined by the operation unit determination unit 36 ​​is output to the derivation engine 37.

[0113] The operation unit determination unit 36 ​​may include a hardware circuit for performing the above determination, or may implement the above determination through software. In addition, the control unit 31 may also have its functions. That is, the determination may also be performed through the hardware circuit of the control unit 31 and / or the software based on the CPU. In addition, the operation unit determination unit 36 ​​may also include a derivation model to determine the operation unit through derivation.

[0114] The operation unit determination unit 36 ​​functions as an operation unit determination unit that divides the images of the organ acquired in time series into operation units and determines the operation performed for each operation unit (for example, referring to Figure 3 In addition, the operation unit determination unit divides the image obtained by the camera unit into operation units (for example, referring to Figure 3 S7, Figure 7 The operation unit determination unit determines the direction of the operation based on the asymmetry of the anatomical structure (for example, referring to Figure 3 S13, Figure 5 P5A, P5B, etc.).

[0115] The derivation engine 37 may be composed of hardware, software (program), or a combination of hardware and software. A derivation model is set in the derivation engine 37. In addition, in the present embodiment, the derivation engine 37 is provided in the auxiliary device 30, but it may also be provided in a device such as an endoscope to perform derivation in the device.

[0116] When the image data of the image P1 is input to the input layer of the deduction engine 37, the deduction engine 37 having the deduction model performs deduction and outputs the operation information Iop related to the endoscope operation from the output layer. The operation information Iop is information for displaying the operation guidance (operation suggestion) for reaching the target part such as the affected part by performing the same operation as the operation performed by the professional doctor when the non-professional doctor inserts the second endoscope system 10B into the body cavity. That is, it includes the image of the operation unit obtained when the professional doctor performs the operation and the information indicating the operation state of the operation unit at that time. In addition, even if not all images or information indicating the operation state are included, as long as there are images / information that are the key points of the operation, it will be sufficient.

[0117] The derivation engine 37 may also use time-series images obtained during an examination performed by a professional doctor using the endoscope system 10A (in Figure 1A In order to generate the derivation model, training data based on multiple time-series images is input to the input layer of the derivation engine 37. Figure 1A The image groups P2 and P3 are shown as examples, but a plurality of image groups other than these are input.

[0118] In the image group P2, similarly to the image group P1, for example, image P22 is the start image of the first operation unit, image P25 is the last image of the operation unit, image P26 is the first image of the next operation unit, and image P29 is the last image of the operation unit. In addition, image P21 is an image of the insertion time in a series of sequential images, and image P30 is an image of the removal time. In the image group P3, similarly to the image group P1, image P32 is the start image of the first operation unit, image P35 is the last image of the operation unit, image P36 is the first image of the next operation unit, and image P39 is the last image of the operation unit. In addition, image P31 is an image of the insertion time in a series of sequential images, and image P40 is an image of the removal time.

[0119] In the image group P2, the operation information is given in the same manner as the determination by the operation unit determination unit 36, the information Isa indicates that the images are the same, and the information Idi indicates that the images are different. In the image group P3, the operation information is given in the same manner as the determination by the operation unit determination unit 36, the information Isa indicates that the images are the same, and the information Idi indicates that the images are different.

[0120] A plurality of images such as the image groups P2 and P3 are used as training data, and the training data is used to perform machine learning such as deep learning, thereby generating a derivation model for operation guidance. Here, deep learning is explained. "Deep learning" structures the process of "machine learning" using neural networks into multiple layers. The "forward propagation neural network" that sends information from front to back for judgment is representative. In the simplest case, the forward propagation neural network has three layers: an input layer composed of N1 neurons, an intermediate layer composed of N2 neurons assigned with parameters, and an output layer composed of N3 neurons corresponding to the number of classes to be judged. Each neuron in the input layer and the intermediate layer, and the neurons in the intermediate layer and the output layer are connected by coupling weights, and bias values ​​are applied to the intermediate layer and the output layer, thereby making it easy to form logic gates.

[0121] The neural network only needs to perform simple judgments and can be 3 layers, but by making the middle layer multiple layers, it is also possible to learn the combination of multiple feature quantities in the process of machine learning. In recent years, from the perspective of the time spent on learning, judgment accuracy, and energy consumption, 9 to 152 layers have become practical. In addition, it is also possible to perform a process called "convolution" of the feature quantity of the compressed image, perform the operation with minimal processing, and use a "convolutional neural network" that is strong in pattern recognition. In addition, it is also possible to process more complex information, and to correspond to the analysis of information whose meaning changes according to the sequence and order, use a "recursive neural network" (fully coupled recursive neural network) that allows information to flow in both directions.

[0122] In order to realize these technologies, the general-purpose calculation processing circuits that have existed in the past, such as CPU and FPGA (Field Programmable Gate Array), can also be used. However, it is not limited to this. Most of the processing of neural networks is matrix multiplication operations, so processors called GPU (Graphic Processing Unit) and Tensor Processing Unit (TPU) dedicated to matrix calculations can also be used. In recent years, such "neural network processing units (NPUs)" dedicated to artificial intelligence (AI) hardware have been designed to be integrated and assembled with other circuits such as CPUs, and sometimes become part of the processing circuit.

[0123] In addition, as a method of machine learning, there are methods such as support vector machines and support vector regression. The learning here calculates the weights, filter coefficients, and biases of the identifier. In addition, there is also a method using logistic regression processing. In the case of making a machine determine something, a person needs to tell the machine the method of determination. In this embodiment, a method of deriving the determination of an image through machine learning is used, but in addition to this, a rule-based method that adapts to the rules obtained by humans through empirical rules / heuristics can also be used.

[0124] As mentioned above, Figure 1B The second endoscope system 10B shown is an endoscope used when the subject (patient) is examined by a non-specialist doctor for the second time or later after the professional doctor has examined the subject (patient) using the endoscope system 10A. The second endoscope system 10B may be the same model or a completely identical device as the endoscope system 10A, but in this embodiment, an endoscope of a different model is shown. The second endoscope system 10B functions as a second endoscope system that observes the subject's observation target organ for the subject (including the patient) who has undergone an organ examination (including diagnosis / treatment) using the first endoscope system.

[0125] The auxiliary device 30 outputs the operation assistance image group P4 for operation guidance during re-examination by a non-specialized doctor to the second endoscope system 10B. The operation assistance image group P4 for re-examination is a time-series image from insertion into the body cavity to image P43 corresponding to the position of the target part such as the affected part when the second endoscope system 10B is used for re-examination. The operation assistance image group P4 for re-examination can also be generated based on images P11 to P20 in the image P1 obtained during the first examination. The image P43 in the operation assistance image group P4 for re-examination includes the operation information Iop as the deduction result of the deduction engine 36, and a guidance display such as "Please perform such an operation" can be performed.

[0126] In addition, Figure 1B In the figure, only the image P43 corresponding to the target part is depicted. However, when the angle, distance, up and down, left and right, lighting and other technical know-how are useful when accessing the target part, an image that becomes a mark Ob can also be displayed in front of it ( Figure 2 In this case, the specification may be such that the image is temporarily stopped in front of the mark and a method of accessing from the position to the target part is guided in more detail. Figure 2The example shown is a case where a place that is easy to understand (position L3) is used as a mark (for example, the pylorus) and the object is bent and observed from that place (target), and the image P43 corresponds to the target part Tg. If the target part Tg can be easily observed even without displaying the image of the mark Ob, it is not necessary to display the image of the mark Ob, and only the image P43 can be displayed. Therefore, as the operation auxiliary image group P4, both the image of the mark Ob and the image P43 corresponding to the target Tg can be displayed separately, or only the image P43 of the target part can be displayed.

[0127] The second endoscope system 10B includes a control unit 11B, an imaging unit 12B, a light source unit 13B, a display unit 14B, an ID management unit 15B, a recording unit 16B, and an operation unit 17B. These are the same as the control unit 11A, the imaging unit 12A, the light source unit 13A, the display unit 14A, the ID management unit 15A, the recording unit 16A, and the operation unit 17A of the endoscope system 10A, and therefore, additional configurations and functions of the second endoscope system 10B are supplementally described, and detailed descriptions are omitted.

[0128] The control unit 11B is composed of one or more processors including a processing device such as a CPU (Central Processing Unit), a memory storing a program (the program may also be stored in the recording unit 16B), etc., and executes the program to control each unit in the second endoscope system 10B. The control unit 11B performs various controls when the endoscope system 10B performs a re-examination of the subject (patient). The CPU of the control unit 11B executes the program stored in the recording unit 16B, etc., to realize Figure 5 In the present embodiment, the CPU in the processor and the program stored in the memory realize the functions of the acquisition unit, the operation determination unit, the operation guidance unit, and the like.

[0129] In addition, the control unit 11B uses the image during the re-inspection obtained by the camera unit 12B and the operation auxiliary image group P4 during the re-inspection output from the auxiliary device 30, etc. to make the guidance unit 19B perform operation guidance for reaching the target part such as the affected part. In order to generate the operation guidance and determine whether the front end of the second endoscope 19B is near the object for searching the target part such as the affected part, the guidance unit 19B set with the derivation model can perform deduction, and the similar image determination unit 23B described later can also perform similar image determination. The operation guidance generated by the control unit 11B can also be displayed on the display unit 14B, and the situation that the front end of the endoscope is near the object or the target part can be displayed on the display unit 14B.

[0130] As described above, the imaging unit 12B is the same as the imaging unit 12A, and thus detailed description thereof is omitted. However, the imaging unit 12B functions as an imaging unit for acquiring images of the subject's organs in a time series manner (for example, see Figure 4 S33).

[0131] The communication unit 21B has a communication circuit (including a transmission circuit and a reception circuit) and exchanges information with the auxiliary device 30. For example, the operation information Iop outputted from the auxiliary device 30 is received. As the operation information Iop, there are a start image, an end image, operation information, and time information for each operation unit (these are recorded in the recording unit 35 as operation unit information 35b). In addition, as the operation information Iop, a target part image (P43) and the like may also be included, and a mark image may also be included. In the case of re-examination of the organ that is the object of the examination etc. performed by the professional doctor on the same subject (patient) as the subject (patient) who was examined etc. by the professional doctor using the first endoscope 10A, the ID of the subject (patient) etc. is sent to the auxiliary device, and the operation unit information 35b of the subject (patient) etc. is obtained. As the operation information Iop, of course, only the necessary information of the operation unit information 35b may be used. In addition, the image data obtained in the imaging unit 12B may be sent to the auxiliary device 30.

[0132] In addition, the communication unit 21B can also communicate information with the endoscope system 10A in addition to the auxiliary device 30. In addition, the communication unit 21B can also communicate with other servers and in-hospital systems. In this case, it is possible to collect information from other servers and in-hospital systems and provide information. In addition, it is also possible to receive a derivation model generated by an external learning device.

[0133] The communication unit 21B functions as an acquisition unit that acquires the time-series operation content information in the first endoscope system as operation unit information (for example, refer to Figure 4 The above-mentioned operation unit information is image change information estimated by utilizing the asymmetry of the observed organ (for example, referring to Figure 7 The above-mentioned operation unit information is continuous image change information representing the same action (for example, refer to Figure 1A Image P1, Figure 7 In addition, the asymmetry information of the observed organ is determined based on the anatomical positional relationship of multiple parts in the specific organ (for example, referring to Figure 7 ). In addition, the communication unit 21B functions as an input unit for inputting recorded operation unit information to the subject who has been examined using the first endoscope system (for example, refer to Figure 4 S31).

[0134] When the front end of the second endoscope system 10B reaches the vicinity of the target part such as the object or the affected part, the signal output unit 22B outputs a signal indicating this situation. For example, the position can be notified to the doctor or the like by irradiating the light source from the light source unit 13B so that the irradiated light can be seen from the outside of the digestive tract wall.

[0135] The similar image determination unit 23B compares the image data acquired by the imaging unit 12B with the operation auxiliary image group P4 for reinspection to determine the similarity. The operation auxiliary image group P4 for reinspection includes a start image, an end image, etc. for each operation unit, so these images are compared with the current endoscopic image acquired by the imaging unit 12B to determine whether they are similar images. There are various methods for determining the similarity of images, such as pattern matching, so a method suitable for this embodiment can be appropriately used from these methods.

[0136] When the doctor inserts the second endoscope system 10B into the body cavity of the subject (patient), the similar image determination unit 23B determines whether each image of the operation auxiliary image group P4 is similar to the image obtained by the camera unit 12B. When making this determination, the operation auxiliary image group P4 is divided into operation units, so the similar image determination unit 23B determines which operation unit the currently obtained image group is similar to. When the image obtained by the camera unit 12B is similar to the end of the operation unit, the guide unit 19B displays the operation information on the display unit 14B according to the operation information Iop.

[0137] Furthermore, the similar image determination unit 23B uses Figures 5 to 8 As described above, the operation process of the second endoscope system 10B is determined by detecting the change of the endoscope image pattern. The continuous images obtained by the camera unit 12B are divided into operation units according to the operation unit information Iop, etc., and the operation process of the current operation, such as insertion operation, rotation operation, bending operation, etc., is determined according to each operation unit (refer to Figure 7 ). The similar image determination unit 23B functions as an insertion operation determination unit that estimates the operation process of the subject when receiving an examination (including diagnosis / treatment) using the second endoscope system (refer to Figure 4 S37).

[0138] When the similar image determination unit 23B finds that the image is similar to the indicated object (refer to Figure 2 Ob) of the image P43 with high similarity, there is a target part such as an affected part ( Figure 2The target part Tg) is displayed on the display unit 14B, so the display unit 14B shows the situation near the target part such as the affected part. The doctor can find the target part such as the affected part by carefully checking the vicinity according to the operation information. If it is not found immediately, air can be supplied, and the endoscope can be slightly pulled out to create space for observation. If the target part such as the affected part is found, the process of the target part such as the affected part can be observed. In addition, depending on the state of the target part such as the affected part, surgery or other treatment is sometimes required.

[0139] In addition, similar image determination can be performed by deduction in addition to determination by the similar image determination unit 23B based on the similarity of the images. That is, a deduction engine can be provided in the similar image determination unit 23B, and a deduction model for similar image determination can be set for the deduction engine to determine similarity by deduction. In this case, the deduction engine functions as a similar image estimation unit having a similarity estimation model, and the similarity estimation model estimates the similarity of the images based on the images of the endoscopic examination. Even when a non-professional doctor operates the endoscope, the determination result in the similar image determination unit 23B can be used to guide the endoscope to the vicinity of the target site such as the affected part.

[0140] The guidance unit 24B provides operation guidance (also referred to as operation suggestion) to the non-professional doctor using the second endoscope system 10B based on the determination result of the similar image determination unit 23B. That is, the guidance unit 24B may also use the determination result of the similar image determination unit 23B to divide the time-series images obtained by the camera unit 12B into operation units, compare the operation information contained in the operation unit information with the currently obtained continuous image, and display the goodness or badness of the operation based on the comparison result. That is, the guidance is performed so that the operation is equivalent to the operation performed by the professional doctor, so that the affected part and other organs can be observed under the same observation conditions as the observation conditions observed by the professional doctor.

[0141] In addition, the guide unit 24B can also perform event determination and display accordingly. For example, it is also possible to perform guided operations such as rotation operation, bending operation, water injection operation, and air supply operation in a certain operation unit. The guide unit 24 can also display the operation guidance for entering the next operation unit at the time of switching the operation unit. The guidance display can be displayed overlapping with the endoscopic image displayed on the display unit 14B, or it can be displayed by the display unit 14B through voice guidance, etc. That is, not only can the display unit 14B display the guidance in a way that can be visually recognized, but the guidance information can also be conveyed to non-professional doctors through voice, etc.

[0142] Furthermore, in the present embodiment, when a non-professional doctor uses the second endoscope system 10B to perform an examination, the guide unit 24B outputs guidance information so that the non-professional doctor can observe the characteristic part of the observation target organ under the same observation conditions as the first endoscope system 10A in the same manner as when the professional doctor performs the examination, for example, referring to Figure 4 Here, as the same observation conditions, there are the size of the object captured in the screen, the angle of view, etc., which are conditions for making the positional relationship between the camera unit and the observation object the same when observing the observation object. In addition, the conditions for making the illumination and exposure control of the observation object the same so as to be able to determine the change in color tone, the focus, the field of view angle (including the position of the person observed last time and the person observed this time, etc.) and other optical conditions can also be made consistent.

[0143] In addition, in the second endoscope system, regarding the situation that observation can be performed under the same observation conditions as the first endoscope system, in the description of this embodiment, the situation of making the positional relationship consistent is mainly described. However, it can also be a concept that includes other conditions, as long as the first and second endoscope systems can share this information in the same way. In addition, there is a correlation between exposure and positional relationship (positional relationship between the observation object and the camera unit). When the observation object is close to the camera unit, the reflection of the illumination light becomes larger. In addition, sometimes due to the gloss of the observation object, regular reflection, light spots, etc. are generated, and it is impossible to become a correct exposure. In this sense, the positional relationship is also an element that should be considered when making the observation conditions the same. In addition, there are also situations where it is better to set the same conditions for treatment such as water injection and suction, and the influence of treatment instruments, which can also be applied to inherit this information. By considering the above factors, it is possible to change exposure, appearance or positional relationship, etc. In addition, even if the observation conditions are made the same, they do not need to be strictly consistent. As long as they are within the allowable range considering appropriate thresholds, they can be said to be the same.

[0144] The guidance unit 24B functions as an operation guidance unit that compares the operation process estimated by the insertion operation determination unit with the operation unit information and outputs operation guidance information (for example, referring to the operation unit information) for operating the second endoscope system in order to observe the characteristic part in the observation target organ with the second endoscope system. Figure 4 The operation guidance information output by the operation guidance unit is guidance information for observing the characteristic part of the observation target organ under the same observation conditions as the first endoscope system (for example, referring to Figure 4Regarding the operation guidance information for operating the second endoscope system in order to observe the characteristic part in the observation target organ, when comparing the operation process estimated by the insertion operation determination unit and the operation unit information, a plurality of operation unit information adjacent in time are compared, and if a repeated part does not require process observation during observation, the operation unit information is corrected to exclude the operation of the repeated part and compared.

[0145] Furthermore, the similar image determination unit 23B and the guidance unit 24B function as an operation guidance unit that divides the images acquired in time series into operation units, estimates the operation state of the second endoscope system for each operation unit, compares the estimated operation state with the operation unit information, and outputs guidance information for observing under the same observation conditions as the first endoscope system (for example, referring to Figure 4 S37, S39).

[0146] For example, in Figure 2 In the display, a guidance display is performed so that when a non-professional doctor inserts the second endoscope system 10B into the body cavity and advances straightly on the route R1, if the position L1 is reached, the front end of the second endoscope system 10B is bent or rotated to enter the route R2. In addition, when advancing on the route R3, if the target part Tg such as the affected part is approached, it is displayed that the target part Tg such as the affected part is approached. In this way, the operation of the professional doctor when using the endoscope system 10A to reach the target part Tg such as the affected part is stored in advance, and the operation guidance for reaching the target part Tg is performed, so that even a non-professional doctor can operate the second endoscope system 10B, easily reach the target part Tg, and perform observation and treatment.

[0147] As described above, in this embodiment, after an endoscopy specialist uses the first endoscope system 10A to perform an inspection, diagnosis, treatment, etc., when a non-specialized doctor uses the second endoscope system 10B to perform an inspection, etc., the second endoscope system 10B can be guided so that the observation target part can be observed under the same observation conditions as the observation conditions performed by the endoscopy specialist using the first endoscope system 10A. As the observation target part that requires such a process observation, there are the following parts.

[0148] Benign lesions that may become malignant in the future

[0149] Sites where lesions may occur in the future (e.g., sites where lesions are common during Helicobacter pylori infection, sites where lesions are common when Helicobacter pylori is not infected, etc.)

[0150] · Evidence of treatment (likely to occur again)

[0151] Furthermore, when the second endoscope system 10B is used to perform an inspection, the following auxiliary information is acquired as a clue to the inspection, and the inspection can be performed by using the auxiliary information.

[0152] · The size, shape, unevenness, color, etc. of the lesion used to observe the process and part of the observation

[0153] Information related to the imaging environment such as the following: whether pigments such as indigo carmine and dyes such as methylene blue are used, whether observation light such as WLI (White Light Imaging) and NBI (Narrow Band Imaging) are used, whether image processing settings such as structure emphasis are set, air supply volume, patient position, video processor, equipment information such as the type of scope, the condition of the mucous membrane around the lesion (whether there is a condition that is easy to confuse), the distance between the lesion and the scope / observation angle, and the insertion amount / torsion amount / angle / bending degree of the scope, etc.

[0154] Information on past observations, such as the degree of pharyngeal reflex

[0155] Information related to the time period, such as the time from the start of the inspection, the inspection implementation period, etc.

[0156] When detecting the above-mentioned part requiring process observation, the following AI (Artificial Intelligence) can be used.

[0157] · AI for detecting parts that require ongoing observation, such as AI for part recognition, CADe (Computer Aided Detection) using images, CADx (Computer Aided Diagnosis), and AI for detecting parts that have been treated. In addition, information recorded in electronic medical records can also be used.

[0158] AI for identifying features of sites that require ongoing observation, such as AI for detecting the size, shape, response, and color of lesions, and AI for detecting the condition of surrounding mucosa.

[0159] AI for recognizing the observation environment, such as AI for detecting the use of dyes such as indigo in images.

[0160] AI for estimating the air supply amount, such as AI estimated based on the output of an air pressure sensor, AI estimated based on the accumulated time of air supply, and AI estimated based on an image.

[0161] AI for estimating the distance to the lesion, for example, AI for estimating the distance / observation angle between the lesion and the endoscope tip using the insertion amount of the endoscope tip, the torsion / angle / bending degree of the endoscope tip, etc.

[0162] Next, use Figure 3 The flowchart shown in FIG. 1 illustrates the operation of recording the operation process of the professional doctor using the endoscope system 10A until reaching the target site such as the affected part. The operation of the endoscope 1 is realized by the cooperation of the control unit 11A in the endoscope system 10A and the control unit 31 in the auxiliary device 30. Specifically, the CPU provided in each control unit controls each unit in the endoscope system 10A and the auxiliary device 30 according to the program stored in the memory.

[0163] When the process of the endoscope 1 starts, first, shooting starts (S1). Here, the imaging element in the imaging unit 12A obtains time-series image data at a time interval determined by the frame rate. When shooting starts, image data in the body cavity is obtained, and the image data is processed by the image processing circuit in the imaging unit 12A. The display unit 14A uses the image data after image processing to display the image in the body cavity. While observing the image, the professional doctor operates the endoscope system 10A to move the front end toward the position of the target part such as the affected part. In addition, the image data after image processing is sent to the input unit 32 in the auxiliary device 30 through the communication unit 21A. In this step, it can be said that the image of the subject's internal organs is obtained in a time sequence by the imaging unit.

[0164] Next, it is determined whether the image of the inner wall surface has changed (S3). Here, the operation unit determination unit 36 ​​in the auxiliary device 30 determines whether the image of the inner wall surface in the body cavity obtained by the endoscope system 10A has changed. Figures 5 to 8 As described in the above, when an endoscope is inserted into a body cavity, the organ being observed changes, so the image of the inner wall surface gradually changes. In addition, even for the same organ, the image sometimes changes due to changes in the direction of the front end portion where the imaging element is provided. In this step, the operation unit determination unit 36 ​​makes a determination based on the change in the image. In addition, the determination is not limited to being performed by the operation unit determination unit 36, but can also be performed by other modules such as the control unit 31. In addition, a derivation model for determining the change in the inner wall surface can also be set in the derivation engine 37, and the image change of the inner wall surface can be determined by the derivation engine 37. When the determination result is that there is no change in the image of the inner wall surface, it becomes a standby state until a change occurs.

[0165] If the result of determination in step S3 is that the image of the inner wall surface has changed, the image is temporarily recorded (S5). Here, the image data inputted into the input unit 32 is temporarily recorded as the inspection image 35a in the recording unit 35. In addition, the recording unit 35 is not limited to the recording unit 35, and any memory capable of temporarily recording image data may be used.

[0166] After temporarily recording the image, it is then determined whether the image change pattern has changed due to the insertion direction, rotation, bending of the front end, etc. (S7). The determination here is that if the determination result in step S3 is that the image of the inner wall surface has changed, it is determined whether the cause of the change is an endoscope operation performed by a professional doctor, such as a change in the insertion direction of the front end of the endoscope, a rotation operation of the front end, a bending operation of the front end, etc. In other words, it is determined whether the change in the image change pattern is caused by the operation of the professional doctor, rather than a change in the part of the organ being observed.

[0167] The operation unit determination unit 36 ​​determines the change of the image change pattern based on the image inputted into the input unit 32. Figure 2 In the case of going straight along route R1, the front end bends at position L1, and when going forward in the direction of route R2, the image change pattern usually changes at position L1. The change in the image change pattern may be a case where only the image pattern changes (for example, the image pattern changes from a circle to a quadrilateral, etc.), or a case where the image pattern changes in a different way. In short, it is sufficient to determine whether the image has changed due to the operation of a professional doctor or the like.

[0168] In addition, it is not limited to images, and other information such as operation information associated with image data and sensor output from a sensor provided at the front end of the endoscope may also be used. For example, the determination may be made based on information such as operation information attached to the image data, and also based on information such as images and operation information. Furthermore, a sensor may be provided at the front end of the endoscope, and operation information may be obtained based on the output from the sensor. Furthermore, a transmission source may be provided at the front end of the endoscope, and a sensor for detecting a signal from the transmission source may be provided outside the body, and operation information may be obtained based on the output from the sensor. In addition, in addition to being performed by the operation determination unit 36, the determination may also be performed by a deduction model set in the deduction engine 37 (which may also be the deduction engine 18A).

[0169] If the result of determination in step S7 is that the image pattern does not change due to the insertion direction, rotation, or bending of the front end, other events are executed (S9). Other events include various events performed by professional doctors, such as air supply operation, water injection operation, suction operation, and still image shooting. When a professional doctor performs other events, the location and type of the event are recorded. When a non-professional doctor performs an examination, the recorded event is displayed (see Figure 4 S39).

[0170] Other events are operations or processes other than insertion direction, rotation (upper and lower relationship), and front end bending, such as the use of treatment instruments, changes in photographic parameters such as exposure / focus, image processing including HDR (High Dynamic Range) or depth synthesis, image processing such as special light observation switching of light sources or emphasizing specific structures, pigment dispersion / staining, etc., which are used to find the object by taking some time. Such information as how much time was spent becomes a useful guide. In addition to the observation guide, there can also be a guide that indicates the removal of the polyp when a polyp is found. In order to achieve this guidance, it can also be set to be able to appropriately select the extent to which the record of the first endoscope system is used in the guidance. For example, it can also be selected by the first side, the second side, or a third party whether to focus on "observation relationship" or "including treatment". In the case of including treatment, when a display such as "polyps were removed at this position before" is performed, it becomes a reference when observing the process.

[0171] On the other hand, if the result of determination in step S7 is that the image change pattern changes due to the insertion direction, rotation, bending of the front end, etc., the continuous portion of the image change pattern is taken as an "operation unit", the operation content information is recorded in time sequence, and the operation start and end images are recorded (S11). Here, a series of image groups from the image in which the result of determination in step S7 is that the image change pattern has changed to the next time the image pattern is determined to have changed is taken as an "operation unit", and the image data is recorded in the recording unit 35 in time sequence. The image that becomes the start point of the operation unit in the series of image groups is recorded as the start image 35ba, and the image that becomes the last image in the series of image groups is recorded as the end image 35bb.

[0172] In step S11, the operation unit determination unit 36 ​​performs image analysis of a series of image groups to obtain operation information, and extracts operation information attached to the images, and records them as operation information 35bc. In addition, the operation unit determination unit 36 ​​extracts time information from the moment information of the first and last images for the images included in the operation unit, and records them as time information 35bd. In addition, these information may not be recorded once, but the information may be appropriately extracted and recorded in the process of repeatedly executing steps S3→S21→S3.

[0173] Step S11 can be said to be a determination step of dividing the images of the organ acquired in time series into operation units and determining the operation performed by the first endoscope for each operation unit. Furthermore, step S11 can also be said to be a recording step of recording the images in the operation unit and the information related to the endoscope operation as operation unit information in the recording unit for each determined operation unit.

[0174] After recording various information related to the operation unit in step S11, it is then determined whether there is an asymmetric image change based on the anatomical structure (S13). Figures 5 to 8 As explained above, when the image is only in the tube shape, the up, down, left, and right directions cannot be immediately known. Therefore, in this embodiment, the position (direction) of the front end and the like is found by using the asymmetry of the anatomical structure of the organ. In this step, the operation unit determination unit 36 ​​analyzes the image input by the input unit 32 and determines whether there is an image change based on the asymmetry of the anatomical structure. For example, Figure 7 In FIG. 1 , from time T3 to time T4, the protrusion of the cavity changes from the upper right 1 o'clock direction to the 12 o'clock direction. In this way, the image change can be detected by utilizing the asymmetry of the internal organs.

[0175] If the result of determination in step S13 is that there is an image change, it is determined that there is a change in the front end direction, and the change in the operation direction is recorded (S15). Since there is a change in the operation direction, it is recorded in the recording unit 35. As described above, the direction of the front end of the endoscope can be known by utilizing the asymmetry of the anatomical structure. Since there is an image change in step S13, it can be said that the direction of the front end has changed. Therefore, in this step S15, the changed operation direction is recorded in the recording unit 35. In addition, when the operation direction has changed, the next series of image groups can also be recorded as an "operation unit".

[0176] After recording in step S15, if the result of determination in step S13 is that there is no image change, or if another event is performed in step S9, then it is determined whether a mark is found (S17). Figure 2The guide unit 24B is located near the target site Tg such as the affected part, and serves as a target object for the mark when searching for the target site Tg. The image information and / or position information of the mark Ob is included in the operation information output from the auxiliary device 30, so the guide unit 24B determines whether the mark is found based on the information and the image obtained by the imaging unit 12B.

[0177] If the result of determination in step S17 is that the mark is found, the mark image is recorded, and the operation before the discovery is recorded (S19). First, the mark image is recorded. In addition, since the target part such as the affected part is located near the mark, the target part located near the mark is searched, and the operation performed until the target part image is found is recorded. That is, the operation performed by the professional doctor from the mark to the target part is recorded. If there is a record of the operation, even a non-professional doctor can easily reach the target part by operating the endoscope with reference to the operation record.

[0178] also, Figure 3 The process shown assumes a situation where a method of accessing a target area using a mark as a starting point is useful, and shows an example of processing in the order of finding the mark and finding the target area. However, as mentioned above, sometimes it is easy to guide to the target area even without a mark. In this case, the image of the mark is sometimes not recorded in the operation unit information. In this case, steps S17 and S19 can be omitted and the target area can be found directly. Whether to record it as a mark can be determined by a professional doctor, and it can also be recorded automatically using AI. In addition, sometimes the endoscope is pulled out or removed. Although not specifically stated, there are also difficult extractions, etc. In this case, the difficult part end image can also be recorded in the operation unit information, and the target can be reset when the difficult part end image is detected.

[0179] Next, it is determined whether the target part is found (S20). In this step, it is determined whether the professional doctor has found the target part. Whether it is the target part can be recorded by the professional doctor. Alternatively, it can be determined based on a specific operation such as taking a static image. Alternatively, it can be determined automatically by AI based on images, operations, etc. If the target part is found, the target part image can also be recorded. In the case where the result of this determination is that the target part is not found in the passage, return to step S19.

[0180] When the target part is found in step S20, or when the object is not found in step S17, it is then determined whether to end (S21). If the professional doctor finds the target part and completes the necessary recording, it can also be determined to be ended here. In addition, in the case of multiple target parts, it can also be determined to be ended when the last target part is found. And, for example, it can also be determined to be ended at the stage where all operations such as pulling out the endoscope from the body cavity are completed. If the result of the determination in this step is not ended, return to step S3 and perform the above-mentioned actions.

[0181] When the determination result in step S21 is the end, the associated data related to the mark, target part, etc. are sent (S23). Here, since the operation information of the professional doctor until the target part is reached is recorded, this information is sent to the terminal, server, etc. that need the information. For example, if there is a request to send the associated data related to the mark, target part, etc. from the second endoscope system 10B, the corresponding data can also be sent according to the ID of the designated subject, etc. In addition, the operation unit information 35b recorded in the recording section 35 can also be sent to the outside. This step S23 functions as an output step for outputting the operation unit information recorded in the recording section. When data is sent in step S23, the process ends.

[0182] Thus, in this process, when the image pattern changes due to a change in the insertion direction, a rotation operation, a front end bending operation, etc., the continuous images until the next change in the image pattern are recorded as operation units (refer to S7, S11). At this time, the image that will become the starting point of the operation, the end point image, the operation content, etc. are recorded as the operation unit information 35b. In addition, it is determined whether there is an asymmetric image change based on the anatomical structure. If there is a change, the change in the front end direction is determined, and the change in the operation direction is recorded (refer to S13, S15). Then, when an object is found, the image of the object is recorded (refer to S17, S19). The object exists near the target site such as the affected part. If the object can be found, the target site can be easily found. In addition, if the pre-discovery operation at this time is recorded, it is easier to reach the target site based on the operation record. That is, if the information until the professional doctor reaches the target site such as the affected part is recorded, the non-professional doctor can use the information to easily reach the target site such as the affected part (refer to Figure 4 Flowchart of the .

[0183] In addition, in this process, as operation unit information, in addition to continuous images, operation content information, operation starting point image, and end point image (refer to S11) are also recorded. As operation unit information, as long as it is the information when the non-professional doctor using the second endoscope system 10B reaches the target part such as the affected part, it is not limited to the information recorded in this process. For example, it is not limited to the starting point image (starting image), it can also be an image of the initial stage of the operation unit, and it can also be an image close to the end, and it can also be an image near the middle. In addition, as a determination of the change of the image pattern in step S7, the image change based on the asymmetry of the anatomical structure can also be determined. Usually, in the case where the image pattern changes in step S7, there are mostly image changes based on the asymmetry of the anatomical structure. In addition, in addition to image information, time information such as the elapsed time from the start of the inspection can also be used as operation unit information. For example, instead of the end point image, time information associated with the timing of the end can be used as operation unit information.

[0184] In addition, in this process, an object (having a character as a mark) near a target part such as an affected part is found. By pre-recording the marked object, there is an advantage that when a non-professional doctor finds the marked object, he can be prompted to observe carefully, but the discovery record of the mark can also be omitted, and only the discovery record of the target part such as the affected part can be performed. In addition, in steps 7 and S11, the record as an operation unit is performed according to the change of the image change mode, and in steps S13 and S15, the front end direction is determined and recorded according to the asymmetry of the anatomical structure. These processes can also be performed together instead of in different steps.

[0185] In addition, in the description Figure 3 The process of the endoscope 1 shown in FIG. 1 is based on the premise that the endoscope system 10A and the auxiliary device 30 cooperate to process. However, the process is not limited to this, and the endoscope system 10A can also independently record the operation information of the professional doctor until the target part is reached. In this case, the judgment of the image change in step S3 is performed by the image processing circuit in the control unit 11A and / or the camera unit 12A, and if there is a change in the image, it is temporarily recorded in the memory such as the recording unit 16A in the endoscope system 10A in step S5. In addition, the judgment in steps S3, S7, S13, and S17 is also performed by the image processing circuit in the control unit 11A and / or the camera unit 12A, and if there is a change, it is recorded in the memory such as the recording unit 16A in the endoscope system 10A. Then, if it is judged to be finished in step S21, the various information recorded in the endoscope system 10A so far is summarized and sent to the auxiliary device 30 (refer to S23).

[0186] Next, use Figure 4The flowchart shown in FIG. 1 illustrates the operation of the endoscope 2 when the non-professional doctor uses the second endoscope system 10B to operate until the endoscope reaches the target part such as the affected part. When performing this operation, the non-professional doctor uses the second endoscope system 10B to examine the same target part such as the affected part of the same person as the subject examined by the professional doctor. When performing this examination, the operation information (based on the operation information) of the professional doctor's examination is transmitted to the non-professional doctor. Figure 1A The operation unit information 35b, etc. of the operation information is provided to the second endoscope system 10B, and the non-specialized doctor operates the second endoscope system 10B according to the operation guidance based on the operation information, thereby being able to easily reach the target site such as the affected area.

[0187] The operation of the endoscope 2 is realized by the control unit such as the CPU of the control unit 11B in the second endoscope system 10B controlling each unit in the second endoscope system 10B according to the program recorded in the memory. The process of the endoscope 2 can realize an endoscopic examination method for observing the subject's observation target organ using the second endoscope system for the subject who has undergone the organ examination using the first endoscope system.

[0188] When the process of the endoscope 2 is started, first, the associated data such as the mark and the target part are obtained (S31). Here, the control unit 11B obtains the associated data such as the mark and the target part from the auxiliary device 30 through the communication unit 21B. As described above, the recording unit 35 of the auxiliary device 30 records the operation unit information 35b (for example, refer to Figure 3 S23). Therefore, the control unit 11B sends the ID (recorded in the ID management unit 15B) of the subject who is to be examined (including diagnosis / treatment) using the second endoscope system 10B to the auxiliary device 30, and obtains data related to the inspection mark, target site, etc. from the operation unit information 35b corresponding to the subject ID. At this time, the starting point image (start image 35ba), the end point image (end image 35bb), the operation content information (operation information 35bc), and the operation time information 35bd can be obtained for each operation unit. This step can be said to be a step of obtaining the time-sequential operation content information in the endoscope system 10A as the operation unit information. In addition, this step S1 can be said to be a step of obtaining the time-sequential operation content information in the first endoscope system as the operation unit information.

[0189] When the associated data is obtained, the shooting is then started (S33). Here, the imaging element in the imaging unit 12B obtains time-series image data at a time interval determined by the frame rate. When the shooting is started, the image data in the body cavity is obtained, and the image data is processed by the image processing circuit in the imaging unit 12B. The display unit 14B uses the image data after image processing to display the image in the body cavity. While observing the image, the non-professional doctor operates the endoscope system 10B to move the front end toward the position of the target part such as the affected part.

[0190] When the shooting starts, it is then determined whether the starting image is detected (S35). In step S31, the starting image (starting image 35ba) of each operation unit is obtained. Therefore, in this step, the similar image determination unit 23B compares the image obtained in the camera unit 12B with the starting image to determine whether the starting image is detected. Usually, there are multiple operation units in one examination / diagnosis / treatment, etc. Therefore, the starting image is read out from the input operation unit information in sequence, and the read starting image is compared with the obtained image to determine whether the starting image is consistent with or similar to the obtained image. In the case where the determination result is that the starting image is not detected, the process proceeds to step S53 to determine whether the end point has been reached.

[0191] On the other hand, if the result of the determination in step S35 is that the starting point image is detected, then, the operation content information (insertion, rotation direction, amount, time) is referred to in time sequence, and reference information is displayed (S37). As described above, what operation was performed is recorded in the information unit information 35b for each operation unit. Therefore, in this step, the similar image determination unit 23B and the guidance unit 24B cause the display unit 14B to display the operation content (operation guidance) in the operation unit corresponding to the starting point image detected in step S35.

[0192] In step S37, when the operation guide is displayed, first, the similar image determination unit 23B determines the operation state of the second endoscope system 10B, such as the operation state of the straight insertion operation, the rotation operation, the bending operation, etc., based on the image obtained by the camera unit 12B. That is, in this step, it can be said that the operation process of the subject when the second endoscope system is used for the examination is estimated. In addition, in addition to the image, the similar image determination unit 23B can also determine the operation state based on information from sensors provided at the front end of the second endoscope system 10B, etc., and can also make a determination based on information related to the operation state of the operation unit 17B, etc., and can also make a determination based on a combination of such information.

[0193] When the operation state is determined, the guidance unit 24B then compares the operation state contained in the operation unit information input from the auxiliary device 30 with the operation state determined by the similar image determination unit 23B, generates an operation guide based on the comparison result, and displays the operation guide on the display unit 14B. That is, the operation information recorded in the operation unit information corresponding to the current operation unit and the current operation information of the second endoscope system 10B determined by the similar image determination unit 23B are displayed as reference information. By referring to this reference information, a non-professional doctor can know the operation method of the second endoscope 10B. If the two are the same (even if they are not strictly the same, as long as they are roughly the same), the operation is roughly the same as the examination of the professional doctor, so it can be said that the operation is directed to the target part such as the affected part. In this step, it can be said that the estimated operation process is compared with the operation unit information, and the guidance information for observing the characteristic part in the observation object organ under the same observation conditions as the first endoscope system is output.

[0194] Next, the quality of the operation and other event judgments are performed, and the corresponding display is displayed (S39). Here, the control unit 11B compares the operation information of the professional doctor recorded as the operation unit information with the operation status actually performed by the non-professional doctor, and judges whether the operation is good or insufficient, etc. Based on the judgment result, the display unit 14B displays whether it is good or not. For example, as the operation unit status, a forward bending operation is recorded, while in contrast, when the judgment result in step S37 is a right rotation operation, the operation status is different, and correction of the operation is recommended.

[0195] In addition, based on the information recorded in the operation unit information, etc., it is determined whether an air supply operation, water injection operation, suction operation or other events are required, and the corresponding operation is displayed based on the determination result. As other events, as described above, there are operations or processing other than insertion direction, rotation (up and down relationship), and front end bending, such as the use of treatment instruments, changes in photographic parameters such as exposure / focus, image processing including HDR (High Dynamic Range) or depth synthesis, light source switching such as special light observation or image processing that emphasizes specific structures, pigment dispersion / staining, etc., which are used to discover the object by taking some time. Operations or processing, etc. If other events are performed when a professional doctor examines a subject, the event is recorded (refer to Figure 3 S9), therefore in this step, a corresponding display is performed according to the recorded event information.

[0196] Next, it is determined whether the marker image is detected (S41). As described above, the object of the marker that becomes the search target is determined near the target such as the affected part (refer to Figure 2The marker image (which may also include position information) is recorded in the operation unit information 35b in the auxiliary device 30. Therefore, in this step, the similar image determination unit 23B compares the marker image of the object to be marked with the current image obtained by the camera unit 12B, and determines whether the marker image is detected based on the comparison.

[0197] If the result of determination in step S41 is that the marker image is not detected, it is determined whether the end image of the operation unit has been reached (S49). As described above, the end image 35bb is recorded for each operation unit in the recording unit 35 in the auxiliary device 30. In this step, the similar image determination unit 23B compares the end image (end image 35bb) with the current image acquired by the imaging unit 12B, and determines whether the end image has been detected based on the comparison.

[0198] If the result of the determination in step S49 is that the image is not the end point of the operation unit, it is determined whether to re-operate (S51). The non-professional doctor operates the second endoscope 10B with the mark or target as the target, but sometimes it is difficult to reach the mark or target, and the operation is re-performed. In this step, the control unit 11B determines whether the non-professional doctor has re-operated based on the image obtained by the camera unit 12B, the output of the sensor set in the operation unit 17B and the device, etc. If the result of the determination is not to re-operate, return to step S37 and repeat the above action. On the other hand, if the result of the determination is to re-operate, return to step S35 and repeat the above action.

[0199] Returning to step S41, when the marker image is detected, a discovery display is performed (S43). Here, the control unit 11B or the guidance unit 24B displays on the display unit 14B that the marker for reaching the target part is found. Through this display, the non-specialized doctor knows that the target part such as the affected part exists near the marker, so he carefully observes the periphery of the marker and finds the target part.

[0200] Next, the mark is recorded (S45). Here, since the mark can be found, the image of the mark is recorded in the recording unit 16B. Next, the operation before discovery is displayed (S45). The information about the operation performed from the time when the professional doctor discovers the mark to the time when the target part is reached is recorded in the operation unit information 35b of the recording unit 35 (refer to Figure 3 Therefore, in this step, the control unit 11B or the guidance unit 24B performs operation display as guidance based on the recorded pre-discovery operation information.

[0201] Next, it is determined whether the target part is found (S48). The image of the target part is recorded in the operation unit information 35b, so the similar image determination unit 23B compares the image of the target part with the current image obtained by the camera unit 12B, and determines whether the target part is detected based on the comparison. If the result of the determination is that the target part is not found, the process returns to step S47 and displays the operation before finding. On the other hand, if the target part is found, the finding of the target part is displayed, and the target part is recorded.

[0202] When the target part is found and the finding is displayed in step S48, or when the determination result in step S49 is that the end image of the operation unit is detected, or when the starting image is not detected in step S35, it is then determined whether it is ended (S53). Here, the non-professional doctor uses the second endoscope system 10B to determine whether the prescribed examination is ended. It can also be determined to be ended when a target such as an affected part is found and an inspection, photographing, etc. are performed. In addition, in the case where there are multiple target parts, it can also be determined to be ended when the last target part is found. In the case where the non-professional doctor decides to end the examination, it can also be determined to be ended. In the case where the determination result is not ended, return to step S35 and perform the above-mentioned actions. On the other hand, in the case where the determination result is ended, the end operation of the process is performed.

[0203] Thus, in the process of the endoscope 2, the associated data related to the target and the object during the diagnosis / inspection in the endoscope system 10A is obtained (refer to S31), and when the image obtained by the camera unit 12B in the second endoscope system 10B is the same / similar to the starting point image of the operation unit (S35: Yes), the operation guidance is displayed on the display unit 14B according to the operation content information (refer to S37). Then, when the mark of the target part is detected (refer to S41), the operation for finding the target part is displayed (S47). That is, according to each operation unit, the guidance display is performed according to the content of the operation of the professional doctor, so that even a non-professional doctor can easily reach the target.

[0204] In other words, in the process of endoscope 2, for the operation content information recorded in time series in the first endoscope system, the operation continued within a specified time is obtained as "operation unit information" (refer to S31), the operation process in the second endoscope system for the organ of the subject who is the same as the subject in the first endoscope system, that is, the observation target organ, is estimated, the estimated operation process is compared with the "operation unit information", and the guidance information for observing the characteristic part in the observation target organ under the same observation conditions as the first endoscope system is output (refer to S35~S39). Therefore, even a non-professional doctor can observe the target part such as the affected part in the same way as a professional doctor.

[0205] In addition, the above-mentioned "operation unit information" is information about continuous image changes that represent the same action. Figure 5 As described above, by dividing the continuous images when performing an inspection (including diagnosis / treatment) using the endoscope system 10A into operation units and recording them, it can be used as an operation guide when performing an inspection using the second endoscope system 10B. Figure 5 As described above, segmentation can be easily performed by utilizing the asymmetry of the observation target organ.

[0206] In addition, the asymmetry information of the observed organ is determined based on the anatomical positional relationship of multiple parts in a specific organ (not the direction of gravity during the examination). In an internal organ, since the direction of gravity is unknown, it is difficult to know the positional relationship such as up, down, left, and right, but the asymmetry of the observed organ can be determined based on the positional relationship of multiple parts in a specific organ.

[0207] In addition, in this process, an object (having the characteristics of a marker) near the target part such as the affected part is found (refer to S41). When a non-professional doctor finds the object that becomes a marker, if the target part such as the affected part is found by careful observation, it has the advantage of being able to quickly find the target part. However, the detection of the marker can also be omitted, and only the detection and determination of the target part such as the affected part can be performed.

[0208] Furthermore, in this flow, all processing is performed only in the module within the second endoscope system 10B. However, the second endoscope system 10B can also be implemented in cooperation with an external device such as the auxiliary device 30. In this case, the second endoscope system obtains an endoscopic image, sends the obtained endoscopic image to an external device such as the auxiliary device 30 (including a server, etc.), performs processing such as steps S35 to S51 in the external device, and the second endoscope system 10B displays the processing results based on the external device. In this case, the second endoscope system is composed of the external device and the second endoscope system 10B.

[0209] As described above, one embodiment of the present invention shows a second endoscope system for observing an organ to be observed by a subject who has undergone an organ examination using a first endoscope system. The second endoscope system includes: an acquisition unit (for example, referring to Figure 1B The communication unit 21B shown, Figure 4 S31), which obtains the sequential operation content information in the first endoscope system as the operation unit information; inserts the operation determination unit (for example, referring to Figure 1B The similar image determination unit 23B, Figure 4S37), which estimates the operation process of the subject when the second endoscope system is used to undergo an examination (including diagnosis / treatment); and the operation guide unit (for example, referring to Figure 1B The guide portion 24B, Figure 4 S37) compares the operation process estimated by the insertion operation determination unit with the operation unit information, and outputs guidance information for observing the characteristic part in the observation target organ under the same observation conditions as the first endoscope system. The above-mentioned operation unit information is continuous image change information representing the same action estimated by using the asymmetry of the observation target organ. Due to such a structure, the second endoscope system can easily access the target part such as the affected part. In addition, even in the case where a non-professional doctor performs process observation of the target part such as the affected part, and even in the case where the device used is different, the second endoscope system can perform the same observation as the professional doctor.

[0210] Furthermore, a first endoscope system according to an embodiment of the present invention includes: an input unit (for example, referring to Figure 1A The input unit 32A, Figure 3 S1 of the present invention inputs images of the subject's organs in a time-series manner; an operation unit determination unit (for example, Figure 1A The operation unit determination unit 36, Figure 3 S11), which divides the images of the organ acquired in time series into operation units and determines the operation performed according to each operation unit; a recording unit (for example, Figure 1A Recording section 35, Figure 3 S11), which records the image and information related to the endoscope operation in the operation unit as operation unit information according to each operation unit determined in the operation unit determination unit; and an output unit (for example, Figure 1A Communications Department 34, Figure 3 S23) outputs the operation unit information recorded in the recording unit. Since it has such a structure, the first endoscope system can obtain information for observation by a non-specialist doctor, which is equivalent to that of a professional doctor.

[0211] In addition, in one embodiment of the present invention, an endoscope inserted from the oral cavity through the esophagus to examine the stomach or duodenum (including diagnosis / treatment) is used as an example for explanation. However, it is not limited to a gastric endoscope or a duodenal endoscope, and the present invention can also be applied to various endoscopes such as a laryngoscope, a bronchoscope, a cystoscope, a choledochoscope, a vascular endoscope, an upper gastrointestinal endoscope, a duodenoscope, a small intestinal endoscope, a capsule endoscope, a colon endoscope, a lower gastrointestinal endoscope, a thoracoscope, a laparoscope, an arthroscope, a spinal endoscope, and an epidural endoscope.

[0212] In addition, in one embodiment of the present invention, an example of using an image obtained by using an imaging element is described, but it is not limited to this example. For example, an image using ultrasound can also be used. For example, if it is the upper digestive tract, the pancreas, pancreatic duct, gallbladder, bile duct, liver, etc., which cannot be observed in the optical image of the endoscope, can also be used for inspection / diagnosis / treatment in the ultrasound image. In addition, if it is the lower digestive tract, if it is fistula, prostate cancer, lesions that cannot be seen in the optical image of the colon endoscope (such as adhesions, etc.), small intestine, etc., the lesions that cannot be seen in the optical image can also be used for inspection / diagnosis / treatment in the ultrasound image. In ultrasonic diagnosis, it is sometimes necessary to depict the lesion by tracking, etc. In this case, in order to make the sensor unit (probe) reach the affected part again, etc., the action when using a normal endoscope becomes the same operation as the last time, such as adjustment.

[0213] In addition, when determining the operation unit, it is performed based on the image (refer to Figure 3 However, in addition to the image, the determination may be made based on the output of a sensor provided in the endoscope, or based on the operation information in the operation unit of the endoscope. Furthermore, in the second endoscope system 10B, in addition to the image, the operation unit may be determined based on other information as in the first endoscope 10A.

[0214] In addition, descriptions corresponding to medical use, the body, and especially the lumen of the digestive tract are provided, but there are many devices that observe the interior through the same operation, and the invention described in this application can also be applied to laparoscopes and industrial endoscopes. The invention is not limited to the invention of soft endoscopes, and the so-called rigid endoscopes also have insertion and rotation operations, which can also be applied to guidance during operations in this case. In the case of a rigid endoscope, when inserting into a body cavity, an element such as the insertion angle is added, but if it is also set as an operation guidance based on, for example, approximately vertical insertion, the invention described in this application can be applied. This is because it is possible to use the image obtained during insertion as a reference to determine whether the angle has changed during insertion.

[0215] In addition, in one embodiment of the present invention, the judgment based on logic and the judgment based on deduction are described, but in this embodiment, whether to perform the judgment based on logic or the judgment based on deduction can be appropriately selected and used. In addition, in the judgment process, the advantages of each can also be partially utilized to perform a hybrid judgment.

[0216] In one embodiment of the present invention, the control units 11A, 11B, and 31 are described as devices composed of a CPU, a memory, and the like. However, in addition to being constituted by software using a CPU and a program, a part or all of each unit may be constituted by a hardware circuit, or a hardware structure such as a gate circuit generated by a program language described in Verilog or VHDL (Verilog Hardware Description Language), or a hardware structure using software such as a DSP (Digital Signal Processor). Of course, these may be appropriately combined.

[0217] In addition, the control units 11A, 11B, and 31 are not limited to CPUs, and any element that functions as a controller may be used. The processing of the above-mentioned units may also be performed by one or more processors configured as hardware. For example, each unit may be a processor configured as an electronic circuit, or may be each circuit unit in a processor configured by an integrated circuit such as an FPGA (Field Programmable Gate Array). Alternatively, the functions of each unit may be performed by a processor configured by one or more CPUs reading and executing a computer program recorded in a recording medium.

[0218] In one embodiment of the present invention, the auxiliary device 30 is described as including the control unit 31, the input unit 32, the ID management unit 33, the communication unit 34, the recording unit 35, the operation unit determination unit 36, and the derivation engine 37. However, these need not be provided in an integrated device, and for example, the above-mentioned units may be dispersed if they are connected via a communication network such as the Internet.

[0219] Similarly, the endoscope system 10A and the second endoscope system 10B are described as including the control units 11A, 11B, the imaging units 12A, 12B, the light source units 13A, 13B, the display units 14A, 14B, the ID management units 15A, 15B, the recording units 16A, 16B, the operation units 17A, 17B, the derivation engine 18A, the clock unit 20A, the communication units 21A, 21B, the signal output unit 22B, the similar image determination unit 23B, and the guidance unit 24B. However, they do not need to be provided in an integrated device, and each unit may be dispersed.

[0220] In addition, in recent years, there are many cases where artificial intelligence that can uniformly determine various judgment criteria is used. Of course, the improvement of the branches of the flowchart shown here is also within the scope of the present invention. For such control, if the user can input good or bad, the user's preferences can be learned and the implementation method shown in this application can be customized in a direction suitable for the user.

[0221] Furthermore, the constituent elements of different embodiments may be appropriately combined. In particular, operations using biological responses including voice recognition require sensors, interfaces, and determination circuits that are suitable for each, but are not specifically described to avoid complexity. However, it is noted that various improved technologies and alternative technologies that can replace these manual operations by the user can also be used to implement the present invention.

[0222] In addition, in the technology described in this specification, the control mainly described in the flowchart can be set by a program in many cases, and sometimes it is stored in a recording medium or a recording unit. Regarding the recording method to the recording medium or recording unit, it can be recorded when the product is shipped, it can also be recorded using a published recording medium, or it can be downloaded from the Internet.

[0223] In addition, in one embodiment of the present invention, a flowchart is used to illustrate the actions in this embodiment, but the order of processing can also be changed. In addition, any step can be omitted, steps can be added, and the specific processing content in each step can be changed.

[0224] Furthermore, even if the process flow in the claims, the specification, and the drawings is described using words such as “first” or “next” expressing a sequence for the sake of convenience, it does not necessarily mean that the process must be performed in this sequence unless otherwise specified.

[0225] The present invention is not directly limited to the above-mentioned embodiments, and the constituent elements can be deformed and embodied in the implementation stage without departing from the scope of the main purpose. In addition, various inventions can be formed by appropriate combinations of multiple constituent elements disclosed in the above-mentioned embodiments. For example, several constituent elements of all constituent elements shown in the embodiments can also be deleted. In addition, constituent elements of different embodiments can also be appropriately combined.

[0226] Description of Reference Numerals

[0227] 10A…Endoscope system, 10B…Second endoscope, 11A…Control unit, 11B…Control unit, 12A…Image unit, 12B…Image unit, 13A…Light source unit, 13B…Light source unit, 14A…Display unit, 14B…Display unit, 15A…ID management unit, 15B…ID management unit, 16A…Recording unit, 16B…Recording unit, 17A…Operation unit, 17B…Operation unit, 18A…Derivation engine, 20A…Clock unit, 21A…Communication unit , 21B…communication unit, 22B…signal output unit, 23B…similar image determination unit, 24B…guidance unit, 30…auxiliary device, 31…control unit, 32…input unit, 33…ID management unit, 34…communication unit, 35…recording unit, 35a…inspection image, 35b…operation unit information, 35ba…start image, 35bb…end image, 35bc…operation information, 35be…time information, 36…operation unit information, 37…derivation engine

Claims

1. A second endoscope system for observing an organ to be observed of a subject who has undergone an organ examination using a first endoscope system, characterized in that: The second endoscope system comprises: an acquisition unit that acquires time-series operation content information in the first endoscope system as operation unit information; an insertion operation determination unit for estimating an operation process of the subject when undergoing an examination using the second endoscope system; as well as an operation guidance unit that compares the operation process estimated by the insertion operation determination unit with the operation unit information and outputs operation guidance information for operating the second endoscope system so as to observe a characteristic part in the observation target organ using the second endoscope system, The operation unit information is image change information estimated by utilizing asymmetry of the observation target organ.

2. The second endoscope system according to claim 1, characterized in that: The asymmetry information of the observation target organ is determined based on the anatomical positional relationship between a plurality of sites in a specific organ.

3. The second endoscope system according to claim 1, characterized in that: The operation unit information is information related to operations that continue within a predetermined time.

4. The second endoscope system according to claim 3, characterized in that: The operation unit information is an operation start image and information related to the operation from the start to the end of the operation.

5. The second endoscope system according to claim 1, characterized in that: The operation guidance information output by the operation guidance unit is guidance information for observing the characteristic part of the observation target organ under the same observation conditions as those of the first endoscope system.

6. The second endoscope system according to claim 1, characterized in that: The operation unit information is continuous image change information indicating the same action.

7. The second endoscope system according to claim 1, characterized in that: When detecting the asymmetry of the observation target organ, a first direction is determined.

8. The second endoscope system according to claim 1, characterized in that: When detecting the asymmetry of the observation target organ, the accumulation direction of the liquid determined by the gravity direction or the direction determined by the positional relationship of the in-vivo structures that have been detected is referred to.

9. The second endoscope system according to claim 1, characterized in that: The operation unit information is determined by reflecting the angle of rotation of an operation lever or a knob for rotating the distal end portion of the endoscope system.

10. The second endoscope system according to claim 1, characterized in that: The operation unit information is information that uses a process until the observation direction of the distal end portion of the endoscope system is changed as an operation unit.

11. The second endoscope system according to claim 10, characterized in that: The observation direction of the front end of the endoscope system is changed by twisting the endoscope system, applying an angle to the endoscope system, or pushing the endoscope system into the body.

12. The second endoscope system according to claim 1, characterized in that: The operation unit information is information that uses a process until the shape of the observation target organ is changed as an operation unit.

13. The second endoscope system according to claim 12, characterized in that: The operation unit information is information that uses as an operation unit a process until the shape of the estimated organ is changed by using the endoscope system to supply air and / or water and / or suction, or by pushing the endoscope system.

14. The second endoscope system according to claim 12, characterized in that: The operation unit information is information that uses, as an operation unit, a process until the state of the mucous membrane of the estimated organ is changed by spreading the pigment and / or stain using the first endoscope system or by sending water using the first endoscope system.

15. The second endoscope system according to claim 1, characterized in that: Regarding the operation guidance information for operating the second endoscope system to observe the characteristic part in the organ to be observed, when comparing the operation process estimated by the insertion operation determination unit and the operation unit information, a plurality of the operation unit information are compared. If a repeated part does not require process observation during observation, the operation unit information is corrected to exclude the operation of the repeated part for comparison.

16. The second endoscope system according to claim 1, characterized in that: Based on the operation unit information, the characteristic part in the observation target organ is observed by automatic operation under the same observation conditions as those of the first endoscope system.

17. An endoscopic examination method, comprising: using a second endoscope system to observe an organ to be observed in a subject who has undergone an organ examination using a first endoscope system, wherein: acquiring sequential operation content information in the first endoscope system as operation unit information, estimating the operation process of the subject when undergoing examination using the second endoscope system, comparing the estimated operation process with the operation unit information, outputting operation guidance information for operating the second endoscope system, the operation guidance information being used to observe a characteristic part in the observation target organ using the second endoscope system, The operation unit information is image change information estimated by utilizing asymmetry of the observation target organ.

18. A first endoscope system, characterized in that: The apparatus comprises: an input unit for inputting images of an organ of a subject in time series; an operation unit determination unit that divides the images of the organ acquired in time series into operation units and determines an operation performed in each operation unit; a recording unit that records, for each operation unit determined by the operation unit determination unit, an image in the operation unit and information related to the endoscope operation as operation unit information; as well as An output unit outputs the operation unit information recorded in the recording unit.

19. The first endoscope system according to claim 18, characterized in that: The operation unit determination unit divides the operation unit based on whether at least one of an insertion direction, a rotation direction, and a bending direction of the distal end portion of the first endoscope has changed based on the image acquired by the imaging unit.

20. The first endoscope system according to claim 18, characterized in that: The operation unit determination unit determines the direction of the operation based on the asymmetry of the anatomical structure of the image acquired by the imaging unit.

21. The first endoscope system according to claim 18, characterized in that: The recording section records a start image and an end image in the continuous images belonging to the operation unit, and records operation information indicating an operation state in the operation unit.

22. The first endoscope system according to claim 18, characterized in that: The recording unit records operation information after a marked object located near a target is found.

23. An endoscopic inspection method, characterized in that: Acquire images of the subject's organs in time series, The images of the organ acquired in time series are divided into operation units, and the operation performed by the first endoscope is determined for each operation unit. For each determined operation unit, an image in the operation unit and information related to the endoscope operation are recorded in the recording unit as operation unit information. The operation unit information recorded in the recording unit is output.

24. A second endoscope system for observing an organ of a subject who has undergone an organ examination using a first endoscope system, characterized in that: The second endoscope system comprises: an input unit for inputting the recorded operation unit information to a subject who has been examined using the first endoscope system; an imaging unit for acquiring images of an organ of a subject in time series; as well as An operation guidance unit divides the image acquired in time series into operation units, estimates the operation state of the second endoscope system according to each operation unit, compares the estimated operation state with the operation unit information, and outputs guidance information for observation under the same observation conditions as those of the first endoscope system.

25. A program, characterized in that The computer is caused to execute the following steps, the computer using the second endoscope system to observe the observation target organ of the subject who has undergone the organ examination using the first endoscope system: acquiring sequential operation content information in the first endoscope system as operation unit information; estimating the operation process of the subject when undergoing examination using the second endoscope system, comparing the estimated operation process with the operation unit information, and outputting guidance information for observing a characteristic portion in the observation target organ under the same observation conditions as those of the first endoscope system, The operation unit information is continuous image change information indicating the same action estimated using the asymmetry of the observation target organ.

26. A program, characterized in that Instruct the computer to perform the following steps: Acquiring images of the subject's organs in time sequence; dividing the images of the organ acquired in time series into operation units, and determining the operation performed by the first endoscope for each operation unit; For each determined operation unit, recording an image in the operation unit and information related to the endoscope operation as operation unit information in a recording unit; as well as The operation unit information recorded in the recording unit is output.

Images