Data processing device, endoscope system, data processing method, insertion guide method, and training method of inference model

Abstract

A data processing device disclosed herein comprises: an insertion difficult state determination unit that acquires insertion shape information from an insertion shape observation device for observing the insertion shape of an insertion unit of an endoscope, and determines an insertion difficult state, in which insertion of the insertion unit into a lumen is difficult, on the basis of the acquired insertion shape information; a training image frame determination unit that is given a series of image frames of a lumen obtained by imaging by an imaging device provided in the insertion unit, and determines one or more image frames, coming after an image frame acquired a predetermined time before the timing of the determination of the insertion difficult state, as an insertion difficult state image; a guide information generation unit that generates guide information, for supporting the insertion of the insertion unit, for the insertion difficult state image; and a recording unit that records the insertion difficult state image and the guide information in association with each other.

Description

Data processing device, endoscope system, data processing method, insertion guide method, and inference model learning method

[0001] The present invention relates to a data processing device that assists insertion into a lumen, an endoscope system, a data processing method, an insertion guide method, and a method for learning an inference model.

[0002] An endoscope is a device that enables observation of diseased areas that are difficult to see from outside the body by inserting an imaging unit composed of an imaging device into the body, etc., and using the imaging results (endoscopically acquired images, endoscopic images) obtained by the imaging unit when illuminating light from an attached illumination unit. An endoscope has an insertion section that is inserted into the body, and the imaging device is provided, for example, at the tip of the insertion section. During an examination using an endoscope, a doctor sequentially displays images acquired by the imaging device at the tip of the insertion section inserted into the body, and adjusts the position of the tip of the insertion section while checking the displayed images to diagnose the health and disease state of the body. Image information consisting of a time series of image frames acquired from the start of insertion of the endoscope into the body to the removal of the endoscope from the body may be digitized and recorded.

[0003] However, inserting an endoscope into the body involves various operations, and it is difficult for an unskilled person to insert the endoscope into the target site.

[0004] Therefore, Japanese Patent Application Laid-Open Publication No. 2006-116289 discloses a technique for assisting the insertion of an endoscope by using an insertion shape observation device that generates an image of the insertion shape of the endoscope insertion portion.

[0005] Japanese Patent Application Laid-Open No. 2006-116289

[0006] However, to utilize the proposal in Patent Document 1, a relatively large-scale device is required, such as by arranging multiple coils in the insertion section and arranging a detection coil near the patient, making it difficult to utilize in all endoscopic examinations. Furthermore, the placement of coils in the insertion section makes it difficult to reduce the diameter of the insertion section. The present invention aims to provide a data processing device, an endoscopic system, a data processing method, an insertion guide method, and an inference model learning method that can utilize data such as endoscopic images obtained in situations where the insertion section is difficult to insert, without requiring an insertion shape observation device in medical settings, thereby enabling the construction of an inference model that effectively assists in the insertion of the insertion section.

[0007] A data processing device according to one aspect of the present invention includes a difficult-insertion state determination unit that acquires insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion part of an endoscope, and determines a difficult-insertion state in which it has become difficult to insert the insertion part into a lumen based on the acquired insertion shape information; a learning image frame determination unit that is given a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion part, and determines one or more image frames after an image frame acquired a predetermined time before the timing of determining the difficult-insertion state as difficult-insertion state images; a guide information generation unit that generates guide information for the difficult-insertion state images to provide assistance with inserting the insertion part; and a recording unit that records the difficult-insertion state images in association with the guide information.

[0008] An endoscopic system according to one aspect of the present invention comprises an inference model created by learning training data obtained by annotating guide information for assisting in the insertion of a first insertion part on difficult-insertion state images including one or more image frames obtained by imaging a series of image frames of a lumen by an imaging device provided in a first insertion part of a first endoscope, the difficult-insertion state images including one or more image frames obtained after an image frame obtained a predetermined time before a determination timing of a difficult-insertion state in which insertion of the first insertion part into the lumen has become difficult; a second endoscope that inserts a second insertion part into the lumen and obtains image images of the lumen in chronological order; and a control unit that provides the image images obtained by the second endoscope to the inference model and causes the inference model to output guide information for assisting with insertion as an inference result.

[0009] A data processing method according to one aspect of the present invention acquires insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion portion of an endoscope, determines a difficult-insertion state in which it has become difficult to insert the insertion portion into a lumen based on the acquired insertion shape information, provides a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion portion, and determines one or more image frames after an image frame acquired a predetermined time before the timing of determining the difficult-insertion state as difficult-insertion state images, generates guide information for the difficult-insertion state images to provide assistance with inserting the insertion portion, and records the difficult-insertion state images in association with the guide information.

[0010] In one aspect of the insertion guide method of the present invention, a series of image frames of a lumen obtained by imaging with an imaging device provided in a first insertion section of a first endoscope are provided, and difficult-insertion state images including one or more image frames after an image frame acquired a predetermined time before a determination timing of a difficult-insertion state in which insertion of the first insertion section into the lumen has become difficult are annotated with guide information for providing assistance with the insertion of the first insertion section. The images are provided to an inference model created by learning training data obtained by annotating the image frames, and the second insertion section is inserted into the lumen and acquires image images of the lumen in chronological order. Guide information to assist with insertion is output as an inference result from the inference model, and a guide display based on the guide information is displayed.

[0011] A method for learning an inference model according to one aspect of the present invention acquires insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion portion of an endoscope, and determines a difficult-insertion state in which it is difficult to insert the insertion portion into a lumen based on the acquired insertion shape information. A series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion portion are given, and one or more image frames after an image frame acquired a predetermined time before the timing of determining the difficult-insertion state are determined to be difficult-insertion state images. Guide information is generated for the difficult-insertion state images to provide assistance with inserting the insertion portion, and an inference model is created by learning using training data obtained by annotating the guide information for the difficult-insertion state images.

[0012] A data processing method according to another aspect of the present invention uses an insertion shape observation device and, during an insertion operation of a first endoscope having a first image capturing device at its tip, acquires insertion shape information from the insertion shape observation device for observing the insertion shape in time series based on the positions of each part of the insertion section of the first endoscope, and determines, based on the acquired time-series insertion shape information, an insertion difficulty determination timing at which the insertion section has become difficult to insert into a lumen, and a series of image frames of the lumen obtained by time-series imaging with the first image capturing device are given, and determine the image frames up to the insertion difficulty determination timing as difficult-insertion state frame images, and during an insertion operation of a second endoscope having a second image capturing device at its tip without using an insertion shape observation device, by comparing the series of image frames of the lumen obtained by time-series imaging with the difficult-insertion state frame images, it is determined whether the insertion operation of the second endoscope will result in the difficult-insertion shape.

[0013] According to the present invention, it is possible to construct an inference model that effectively assists the insertion of an insertion part by utilizing data such as endoscopic images obtained in situations where the insertion of the insertion part is difficult, without requiring an insertion shape observation device in the medical field.

[0014] 1. A block diagram showing a data processing device according to an embodiment of the present invention. FIG. 1 is a block diagram showing the configurations of a recording unit and a learning unit in FIG. 1. FIG. 12 is an explanatory diagram showing an example of an endoscopic system including a UPD device. FIG. 13 is an explanatory diagram for explaining an insertion unit employed in the UPD device. FIG. 14 is an explanatory diagram for explaining a method for determining a difficult-insertion state and a good-insertion state by the difficult-insertion state determination unit 14 and the good-insertion state determination unit 15. FIG. 14 is an explanatory diagram showing an example of a guide display Ga indicating that a manual compression technique should be performed. FIG. 15 is an explanatory diagram for explaining learning for constructing an inference model. FIG. 16 is a flowchart for explaining creation of teacher data. FIG. 17 is a flowchart for explaining learning using teacher data. FIG. 18 is an explanatory diagram for explaining a selection range of image frames to be acquired as a difficult-insertion state image or a good-insertion state image. FIG. 19 is a block diagram showing an endoscopic system that uses an inference model constructed based on teacher data obtained by the data processing device of FIG. 1. FIG. 19 is a flowchart for explaining the operation of the endoscopic system of FIG. 11. FIG. 19 is an explanatory diagram showing an example of a display on a display screen 80a of a display device 80. FIG. 19 is an explanatory diagram showing an example of a display on a display screen 80a of a display device 80.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] (Data Processing Device) Fig. 1 is a block diagram showing a data processing device according to an embodiment of the present invention, Fig. 2 is a block diagram showing the configuration of a recording unit and a learning unit in Fig. 1.

[0017] The internal lumen to be examined with an endoscope may be constricted by sphincters or the like, bent three-dimensionally, clogged with metabolites, or have bulging luminal walls due to lesions. For this reason, the endoscopic image acquired by the imaging unit, which is comprised of an imaging device provided at the tip of the insertion section of the endoscope, does not always provide good visibility of the state of the lumen in the longitudinal direction (the direction of the connection of the holes) ahead in the insertion direction. There may be such difficult-to-insert sections along the lumen where visibility is poor, or when entering a lumen from outside the body, entering another lumen from there, or branching, which may also result in the above-mentioned difficult-to-insert sections.

[0018] Therefore, the endoscope operator may need to proceed with insertion while searching for a path for the endoscope insertion section while viewing an image in which the cavity ahead in the insertion direction is not necessarily visible (a dead end that appears to be an area that is difficult to insert.) Furthermore, the state of the lumen varies from person to person, and even for the same subject, the state of the lumen varies depending on changes in the position of the examination site, changes over time, and the situation at the time, making it difficult to insert the endoscope insertion section.

[0019] Considering differences in testing equipment and usage methods, in many cases, smooth insertion of the insertion part into the lumen requires medical professionals to gain sufficient experience. Therefore, when a beginner performs an examination, support from an experienced physician is also required. To address these issues, the use of AI (artificial intelligence) to assist in the insertion of the insertion part can be considered as a more efficient technique transfer and an examination method that even beginners can easily perform. However, building an inference model for such AI has not been easy until now.

[0020] Therefore, in this embodiment, by using image information acquired during the process from the start of insertion of the endoscope into the body to whether insertion is successful or unsuccessful at a difficult-to-insert portion, it is possible to construct an inference model that supports the insertion of the endoscope at a difficult-to-insert portion where, for example, an endoscopic image of a curved lumen portion cannot be used to confirm the cavity in the insertion direction. In this embodiment, the output of a UPD device (endoscopic insertion shape observation device) is used to determine when insertion is difficult.

[0021] 1, medical images such as endoscopic images acquired by a first endoscope (not shown) are input to a data processing device 1. The endoscope has an insertion section (first insertion section) that is inserted into a body cavity, and an imaging device is provided at the tip of the insertion section. The imaging device includes an imaging element such as a CCD or CMOS sensor, and photoelectrically converts an optical image from a subject to obtain an imaging signal.

[0022] There are also endoscopes that have an ultrasonic transmitter and receiver near the tip and use ultrasonic images for observation and diagnosis, but the present invention can be applied to any endoscope that can obtain images without using an imaging element. In this case, the images are generated by imaging the reflection of ultrasonic signals.

[0023] During an endoscopic examination, a doctor operates an endoscope to insert the insertion section of the endoscope into the human body. The tip of the insertion section is provided with a bending section, and the doctor operates a bending knob or the like provided on the control section of the endoscope to bend the bending section or move the insertion section forward or backward, thereby inserting the tip of the insertion section to the site to be observed. Endoscopic images are acquired during the process from inserting the insertion section to removing it.

[0024] Images of the inside of the body acquired by such an endoscopic examination or the like are input to the data processing device 1. Note that medical images other than images acquired by an ultrasonic endoscope and endoscopic images may also be input to the data processing device 1. While Fig. 1 shows an example of a moving image as the image input to the data processing device 1 (input image), not only a series of continuously acquired images but also still images may be input.

[0025] 1 shows an example in which a plurality of images P1A, P2A, ... (hereinafter referred to as images PA when there is no need to distinguish between these images) are input from an operator O1, and a plurality of images P1B, P2B, ... (hereinafter referred to as images PB when there is no need to distinguish between these images) are input from an operator O2. For example, the images PA and PB are captured images (endoscopic images) acquired in time series by imaging the inside of a lumen.

[0026] Each image PA, PB may include accompanying information added to the image in addition to the image (moving or still image) portion. The accompanying information includes information about the image acquisition environment and the subject, operation information, sensor information, etc. For example, various switches are provided on the operation unit of the endoscope (not shown), and operation information such as the operation of these switches may be added to the image data of the endoscopic image and input as accompanying information. Furthermore, in this embodiment, the accompanying information also includes information acquired by the UPD device. These images PA, PB including accompanying information are supplied as input images to the information processing unit 10 of the data processing device 1.

[0027] Fig. 3 is an explanatory diagram showing an example of an endoscope system including a UPD device, and Fig. 4 is an explanatory diagram for explaining an insertion section employed in the UPD device.

[0028] 3 includes an endoscope 51, a UPD coil unit 52 that detects the position of an insertion shape detection coil (hereinafter abbreviated as UPD coil) provided in the endoscope 51, an insertion shape observation device (UPD device) 50 that generates an image of the insertion shape of the endoscope 51 based on a detection signal from the UPD coil unit 52, and a monitor 53. The UPD device 50 can detect the insertion shape of the endoscope and changes in its insertion state by determining the coordinates of each position of the endoscope insertion portion and changes in those coordinates over time.

[0029] The endoscope 51 has a long, thin insertion portion 51a that is inserted into the body cavity, an operating portion 51b provided at the rear end of the insertion portion 51a, and a universal cord 51c that extends from the operating portion 51b. The universal cord 51c enables the endoscope 51 to be detachably connected to a video processor (not shown).

[0030] The insertion section 51a also has a hard tip section 54a at the tip, a bending section 54b adjacent to the rear end of the tip section 54a and capable of being bent freely, and a flexible tube section 54c having flexibility and extending from the rear end of the bending section 54b to the front end of the operating section 51b.

[0031] Illumination light is emitted from the distal end surface of the insertion section 51a toward the subject, and light reflected from the subject forms an image on the imaging surface of an imaging device (not shown) provided at the distal end of the insertion section 51a. The imaging device photoelectrically converts the incident optical image of the subject to obtain an imaging signal. This imaging signal is output to a video processor, which produces an endoscopic image.

[0032] 4, UPD coils 55 are arranged at predetermined intervals within the insertion section 51a, and signal lines (not shown) connected to the UPD coils 55 are connected to a UPD coil drive circuit (not shown) provided in the UPD device 50. The UPD coil drive circuit sequentially applies AC drive signals to each UPD coil 55 via the signal lines, generating an AC magnetic field around each UPD coil 55.

[0033] In addition, a UPD coil unit 52 consisting of multiple UPD coils is placed at a predetermined position, such as around the bed B (not shown) on which a patient lies and into which the insertion section 51a is inserted, and the UPD coil unit 52 detects the magnetic field generated by the UPD coil 55 placed inside the insertion section 51a (dashed arrow).

[0034] The detection signal from the UPD coil unit 52 is input to a coil position calculation circuit (not shown) in the UPD device 50, which calculates the position of each UPD coil 55 from the amplitude and phase values ​​of the signal detected by the UPD coil unit 52. The position information calculated by the coil position calculation circuit is provided to an insertion shape calculation / display processing circuit, which performs processing to estimate the insertion shape of the insertion part 51 a from a shape obtained by connecting the calculated positions of each UPD coil 55, and signal processing to model the estimated insertion shape and display it as a UPD image. The UPD image from the UPD device 50 is supplied to the monitor 53, and an image showing the insertion shape of the insertion part 51 a is displayed on the display screen of the monitor 53.

[0035] Information on the UPD image (UPD information) from the UPD device 50 is provided to the data processing device 1 in FIG. 1 as accompanying information on the endoscopic image acquired by the endoscope 51 .

[0036] The data processing device 1 includes an information processing unit 10, a recording unit 20, and a learning unit 30. The information processing unit 10 may be configured by a processor using a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an NPU (Neural Processing Unit), etc. The information processing unit 10 may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of its functions using a hardware electronic circuit.

[0037] The information processing unit 10 includes a control unit 11, an observation region determination unit 12, a guide information generation unit 13, a difficult-insertion state determination unit 14, a good-insertion state determination unit 15, a learning image frame determination unit 16, and a communication unit 17. The control unit 11 controls each unit of the information processing unit 10 in an integrated manner.

[0038] In order to create training data necessary for training an inference model that effectively supports insertion in difficult-to-insert sections, the difficult-to-insert state determination unit 14 determines a difficult-to-insert state, and the good-insertion state determination unit 15 determines a good-insertion state. For example, the difficult-to-insert state determination unit 14 and the good-insertion state determination unit 15 determine whether the input image is in a difficult-to-insert state or a good-insertion state by analyzing UPD information in accompanying information added to the input image. Using this determination result, the learning image frame determination unit 16 determines whether each frame of the input image is an image obtained during an insertion operation that causes a difficult-to-insert state (hereinafter referred to as a difficult-to-insert state image) or an image obtained during an insertion operation that does not cause a difficult-to-insert state (hereinafter referred to as a good-insertion state image).

[0039] The difficult-to-insert state image and the good-insertion state image may be either moving images or still images, and include a series of one or more image frames. The series of one or more image frames may be a series of multiple consecutive image frames, or may be multiple image frames spaced apart, such as every other image frame, as long as they are aligned in time.

[0040] 5 is an explanatory diagram for explaining the method of determining a difficult-insertion state and a good-insertion state by the difficult-insertion state determination unit 14 and the good-insertion state determination unit 15. In Fig. 5, a series of image frames obtained by imaging with an endoscope are shown in the left column, the shape of the lumen inside the body and the shape of the endoscope insertion portion are shown in a square frame in the center column, and the advancement and retraction of the insertion portion are shown in the right column.

[0041] The upper and lower central columns of Figure 5 show the curved state of a specific lumen within the body using dashed lines. These lumens have curved portions where the bending angle gradually decreases from one end to the other (hereinafter referred to as a bending shape curve). Note that bending shape curves include curves where the rate of change of the bending angle is constant and curves where the change in the bending angle is not constant. The upper and lower central columns of Figure 5 also show the insertion shape of the endoscope insertion section inserted into these lumens using solid lines. The examples in the upper and lower central columns of Figure 5 show the insertion section locally bending relatively significantly in the circled portions. The phenomenon in which the tip of the insertion section bends approximately 180 degrees, as shown in the upper central column of Figure 5, is called the stick phenomenon (the phenomenon in which the flexible portion (tube) of the endoscope insertion section bends sharply and pushes against the intestinal wall), and the phenomenon in which the insertion section attempts to form a loop, as shown in the lower central column, is called the re-loop phenomenon. In addition to the re-loop, other loop phenomena include the alpha loop, the inverted alpha loop, and the N loop.

[0042] When the insertion section is further inserted (Push operation) (thick arrow) while the sticking phenomenon is occurring, the insertion section moves from the solid line to the dashed line state, as shown in the upper right column of Figure 5. That is, the tip of the insertion section retracts in the direction opposite to the tip direction of the insertion section (the direction of the dashed arrow), pushing into the lumen wall W1. Also, as shown in the lower right column of Figure 5, when the insertion section is further inserted (Push operation) (thick arrow) while the re-looping phenomenon is occurring, the insertion section moves from the solid line to the dashed line state. That is, some parts of the insertion section do not move (hold), while other parts retract or remain, pushing into the lumen wall W2.

[0043] As such, the insertion state of the insertion section does not always change or progress according to the surgeon's intended insertion operation. For example, even when the surgeon inserts (advances) the insertion section, the insertion state of the insertion section does not necessarily advance smoothly. For example, only slight advancement may occur due to the endoscope's deflection or friction with the intestinal wall, or the tip may move away from the direction of advancement if a sticking phenomenon occurs in the splenic convexity. Continuing to advance the endoscope under such conditions may further worsen the insertion state or cause pain to the patient. Therefore, although it is difficult to complete endoscopic insertion without constantly monitoring the behavior of the endoscope's insertion section and tip in response to the intended operation and performing appropriate operations according to the situation, this has not been taken into consideration in conventional endoscope insertion assistance devices or automated endoscopes. Therefore, even when using technology such as that disclosed in Japanese Patent No. 3645223, insertion is not always smooth. Furthermore, for example, Japanese Patent Publication No. 4855901 has proposed an endoscope insertion assistance device that detects loops or bends from the endoscope insertion shape and prompts the user to perform a release operation, but although this device can provide assistance information after loops or bends have occurred, it is difficult to detect the insertion status from slight changes caused by a single operation by a doctor or an automatic endoscope insertion device, and it is also difficult to know conditions such as friction with the mucosal surface that do not necessarily appear in the insertion shape.

[0044] As such, in difficult-to-insert sections, for example, sticking and re-looping phenomena, as shown in FIG. 5 , may occur. These phenomena occur due to problems with the insertion method of the insertion part before and during insertion into the difficult-to-insert section. Therefore, by learning endoscopic images obtained when such insertion is performed, i.e., images of difficult insertion states and images of successful insertion states, it is possible to predict difficult insertion states and successful insertion states in difficult-to-insert sections, thereby enabling effective insertion assistance. The upper and lower left columns of FIG. 5 show examples of images of difficult-to-insert sections when sticking or re-looping occurs. Note that in FIG. 5 , the left side of the page shows older image frames, and the right side shows newer image frames. The example in FIG. 5 shows image frames from before insertion of the insertion part into a curved lumen (bent lumen section) to when the insertion part becomes stuck. The solid area in the figure indicates the inner (deep) part of the lumen where the tip of the insertion part should advance.

[0045] Flexible gastrointestinal endoscopes are inserted deep into lumens, such as the digestive tract. The surgeon uses the imaging results obtained by the imaging element at the tip (which captures the image illuminated by the forward illumination light from the tip) to find the insertion direction along the tunnel-like lumen, maneuvering the endoscope toward the depths of the lumen (where the illumination light does not reach, appearing as black holes). The surgeon simply adjusts the direction of the tip and pushes it in each direction: if a hole is seen to the right, move it to the right; if a hole is seen above, move it upward. Therefore, during a successful insertion process, the insertion section is advanced forward while the black hole pattern is visible in the center of the screen, to avoid placing excessive strain on the inner wall of the lumen. The images obtained during this insertion process are recorded as individual image frames. Each image frame continuously changes, revealing parts of the previous image frame. This is similar to the image seen forward from a car traveling through a tunnel, as it flows to the side. The insertion operation is carried out carefully while checking these gradual changes in the image.

[0046] However, as shown in the image frames in the upper left column of Figure 5, initially, a black pattern is visible (sometimes reflecting the folds or wall of the lumen), and even though the tip appears to be bent in that direction, the positional relationship with the wall or other surface in the direction of travel may prevent proper bending control at the bend in the lumen (circled in the upper center column of the figure), causing the black pattern to disappear. This state, as shown by the solid line in the upper center column of Figure 5, occurs when the endoscope can be inserted but the tip is bent and unable to move in the desired direction, resulting in a stuck state. This state can be easily detected by a UPD device. In this state, the endoscope should be moved in the direction of withdrawal and then bent at a gentler angle, as shown by the dashed line in the upper center column of Figure 5.

[0047] Even without information from the UPD device, it is possible to determine from the changes in the image frames that the tip is gradually moving in a direction different from the expected direction (the direction in which the black pattern is in the center) (e.g., the image appears to move away despite the pressure being applied). In this case, the tip of the insertion tube is pushed against the inner wall, causing it to bend and change in a way that exceeds the operator's sense of operation. As a result, not only the initially visible black pattern but also the image pattern of the inner wall structure that was displayed around it suddenly disappears with abrupt changes (e.g., to the point where the motion vector cannot be detected). By detecting and analyzing the temporal changes in image information (e.g., motion vectors) in the image frames captured during this process, the process leading to the state shown in the center column of Figure 5 can be understood.

[0048] When the UPD device determines the endoscope insertion shape, it can be seen that the operating side of the endoscope tube is moving forward while the tip position is retracted, as shown in the upper right column of Figure 5. If the UPD device determines the endoscope shape for the time series of endoscopic images leading up to the stuck state, annotates the shape information, and uses it as training data for learning, it is possible to build an inference model that infers whether or not a stuck state will occur based on the input of the endoscopic image. By using such inference results as guide information, it becomes possible to recognize that a stuck state has occurred without confusion after the stuck state has occurred and to try again.

[0049] The same can be said for endoscopic image frames showing the process leading up to the re-loop phenomenon, as shown in the lower left column of Figure 5. In the case of a lumen bend, as shown by the dashed line in the lower center column of Figure 5, insertion is possible if the insertion tube is gently and uniformly bent at each part without any strain. However, when insertion is performed as shown by the solid line, an unnatural bend (buckling in the example in the lower center column of Figure 5) occurs, as shown in the circled area. The insertion force is applied only to the middle of the tube, causing the tip to stop moving. In this case, the image frames obtained initially show the insertion tube moving smoothly toward the black pattern at the back of the lumen. However, when the insertion tube is subsequently pushed, the tip does not move further. Instead, the insertion tube bends and points in a different direction, causing the black pattern to disappear from the image frames. The process leading up to this situation also involves image changes that differ from the movement of the tip being pushed in response to the endoscope's push. In other words, a sliding image (image changes such as the image shifting left and right, up and down in endoscopic images) is obtained, which shows the inner wall of the lumen relaxing, rather than pushing and pulling or bending the tip. By annotating this time-series image group with information about the endoscope shape obtained by the UPD device and using this as training data for learning, an inference model can be constructed that, when an endoscopic image is input, can obtain an inference result as to whether or not a re-loop state will be reached. When the UPD device determines the endoscope insertion shape, it can be seen that the operating side of the endoscopic tube is moving, but the tip is retracting or stagnating, as shown in the lower right column of Figure 5. By providing such inference results as guide information in the form of text, audio, images, etc., it becomes possible to recognize that a re-loop state has been reached after a re-loop state without any confusion and to try again.

[0050] The guide for retrying the procedure may be based on information obtained from the UPD device, which determines the endoscope shape that requires retrying and how the endoscope's insertion / removal and bending changes from the time the retry is completed until successful insertion (the distal end of the endoscope begins to move further along the lumen). This information may then be verbalized and used as guide information. Alternatively, the changes may be verbalized and displayed from endoscopic image information obtained during the process. Since the endoscope's movements to release the stuck state may be visible in the endoscopic image, examples of endoscopic images of recovery may be displayed in chronological order, such as video or still images. Furthermore, if the insertion is delayed due to a stuck state or a re-loop, the state of the endoscope may be inferred based on the image information from the UPD device. Therefore, displaying a pseudo-shape image (pseudo-UPD image) generated by the UPD device, as shown in the center column of Figure 5, may enable a calm response even if the user does not immediately understand what has happened. In other words, the pseudo-UPD image may also be considered guide information. Furthermore, instructions on the next unstuck movement may be displayed as an endoscopic shape. Furthermore, in a stuck state, excessive force is often applied during insertion, or the tube is inserted too far, causing bending. Therefore, instructions such as "Pull (remove) it slightly until the image starts to move in accordance with the operation, and then locate the lumen" can be given. Furthermore, folds extending perpendicular to the lumen direction may be visible on the intestinal wall. Instructions can be given to locate the pattern of these folds and use this perpendicular direction as a reference for locating the lumen direction. Furthermore, the surface of the large intestine mucosa has a mesh pattern consisting of countless grooves running parallel to the transverse diameter of the intestine, known in endoscopic examinations as the innominate grooves. The direction of these grooves can also be used as a reference for locating the lumen direction. Of course, audio guidance is also possible in addition to visual guidance.

[0051] Furthermore, the shape of the endoscope when it becomes stuck can be obtained from the UPD information, and the UPD information at that time can also be considered information that indicates the position within the lumen that it has reached. For example, in the case of a lower endoscopy, if the reason for the stuckness is a re-loop, it often occurs when the scope near the sigmoid colon deforms as the tip reaches the splenic flexure. In this case, the position of the tip or the location where the deformation occurred can be communicated to the operator, or in the case of a stick, a guide can be provided to the surgeon that the stuck state occurred because the tip reached the sigmoid colon.

[0052] By outputting this information to the operator as information indicating how far the tip of the endoscope has reached during intraluminal insertion, the operator can check the progress of insertion. Once the observation site is known, it is possible to output a guide, such as a commonly recommended right twist until the descending colon and a left twist from the transverse colon onward. This information can be stored in a database that pairs the intraluminal site (location) with "insertion tips," and by searching the database according to the observation location, the above-mentioned guide can be output. It is said that twisting to the right is effective when re-looping, and changing the stiffness is also effective. By preparing a database that pairs the situation when the endoscope is stuck with "insertion tips," and searching this database when the endoscope is stuck, it is possible to output an appropriate guide to smooth insertion.

[0053] Furthermore, UPD information can also be used to predict the pain or discomfort a patient (subject) may experience during an examination due to the insertion of an endoscope into the body. Research has already been conducted into the relationship between the characteristics of a certain area, the shape of the area (e.g., a curved lumen), and the changes in shape (such as contact during endoscope insertion) and the discomfort that may occur during the process, and these relationships have been compiled into a database. UPD information reveals the location and shape of the endoscope inserted within the body, so pain or discomfort can be predicted by searching the databased relationships based on the UPD information. Furthermore, UPD information reveals not only the position of the endoscope's tip but also the relationship between the curvature of the tube and the subject's internal lumen. Therefore, by accumulating user-reported information on where and to what extent the subject felt discomfort, data can be collected from many cases, enabling highly accurate assessment and prediction of discomfort.

[0054] In other words, the present application uses the UPD data of the insertion shape observation device to determine not only the insertion difficulty determination timing when an insertion difficulty shape occurs, making insertion into a lumen difficult, but also the shape of discomfort during insertion. As a result, a data processing method is provided in which, from a series of image frames of the lumen obtained by time-series imaging with the endoscope imaging unit, image frames up to the timing of the insertion difficulty shape are determined as insertion difficulty shape images, and during an insertion operation during another endoscopic examination without using the insertion shape observation device, the series of image frames of the lumen obtained by time-series imaging in this examination are compared with the insertion difficulty shape frame images, thereby determining or predicting whether an insertion operation during an endoscopic examination will cause discomfort to the subject. Furthermore, in addition to the term "insertion difficulty" being used to refer to a situation in which the insertion operation of the endoscope does not physically lead to movement of the tip, a situation in which the insertion operation of the endoscope causes discomfort to the subject can also be considered "insertion difficulty." Therefore, the "insertion difficulty determination" in the present application may be defined as determining a situation in which the distal end of the endoscope cannot be controlled as intended by the operator.

[0055] The image frames in the upper left column of Figure 5 show an example of the image frames leading up to the insertion difficulty determination timing, in which the initially visible black pattern disappears midway. On the other hand, in the buckling event shown in the lower center column of Figure 5, the lumen path leading to insertion difficulty occurs some time after the tip passes through, and buckling occurs in a portion other than the tip. Therefore, the endoscopic image at the moment of buckling is an image further inside the lumen, making it impossible to directly observe the buckling. However, images of buckling occurrence are related to the buckling phenomenon. In other words, the tip also passes through the lumen where buckling occurs, and the image information during passage differs depending on whether buckling occurs or not. Therefore, analyzing the image information acquired at the endoscope tip can identify whether buckling occurs or its characteristics. Furthermore, because the movement of the insertion tip when buckling occurs is also characteristic of buckling, the endoscopic image also undergoes a unique change in image frame. In other words, when endoscope insertion stops (timing for determining insertion is difficult), including due to buckling, it is possible to determine the reason for the change by analyzing the images and image changes during the insertion process. If the image changes or the results of comparing the image changes are used as training data and learned, it becomes possible to determine and infer whether endoscope insertion is difficult.

[0056] As described above, the difficult-insertion state determination unit 14 uses UPD information as insertion shape information to determine a difficult-insertion state in which insertion of the insertion portion into a lumen has become difficult. The difficult-insertion state determination unit 14 stores insertion shape patterns, such as the shape of the insertion portion during a stick phenomenon (hereinafter referred to as a stick shape) and the shape of the insertion portion during a re-loop phenomenon (hereinafter referred to as a loop shape), in a memory (not shown), and determines whether the insertion shape image of the insertion portion obtained from the UPD information forms a shape pattern similar to the stored shape pattern, thereby determining whether the shape of the insertion portion is a stick shape, a loop shape (a re-loop shape in which a loop occurs at a site that has passed once, and other patterns such as an N-loop shape, an alpha loop shape, and an inverted alpha loop depending on the shape of the looped endoscope). When the difficult-insertion state determination unit 14 determines from the UPD information that the insertion portion has become a stick shape, a re-loop shape, or the like, it determines that an insertion difficult state has occurred. The difficult-insertion state determination unit 14 outputs a notification indicating that the difficult-insertion state has been detected to the learning image frame determination unit 16 at the timing of determining the difficult-insertion state.

[0057] Furthermore, by capturing images based on time changes, a difficult-to-insert state can also be determined based on UPD information, such as when the tip of the endoscope does not move along the lumen even when an attempt is made to insert the endoscope. In other words, the difficult-to-insert state determination unit determines the positions of the endoscope insertion portion, making it possible to observe the insertion shape over time, and determines time-series change information from the insertion shape observation device to determine a state in which the tip of the endoscope does not move (no positional change occurs in the longitudinal direction of the tip of the endoscope) in response to the movement of the endoscope insertion / removal operation (positional change on the proximal side), and determines that insertion is difficult. The timing for determining difficult-to-insert may be determined when the movement of the proximal side of the endoscope insertion portion (in UPD information) is a change in the pushing state from the operator, while the movement of the tip only changes by a specific amount or less.

[0058] In this way, by using an insertion shape observation device (UPD device) in combination with an endoscope having an imaging device at its distal end, and acquiring insertion shape information from the insertion shape observation device, which observes the insertion shape over time based on the position of each part of the endoscope's insertion section during the insertion operation of the endoscope having an imaging device at its distal end, and providing a difficult-insertion state determination step that determines the timing at which the insertion section becomes difficult to insert into the lumen based on the acquired time-series insertion shape information, it is possible to correlate the images captured by the distal end of the endoscope in the difficult-insertion state. In other words, by providing a series of images of the lumen obtained by time-series imaging using the endoscopic imaging device, and providing a difficult-insertion state image determination step that defines the images up to the insertion-difficulty determination timing as difficult-insertion state images, it is possible to determine the characteristics of the image changes leading up to the difficult-insertion state without UPD information. While this example shows an example in which inference reflecting various image information is possible through learning, once the characteristics of the image changes leading up to the difficult-insertion state are known, it is possible to determine whether the endoscope is being inserted successfully or not based solely on the endoscopic imaging results, without UPD information, the next time the endoscope is inserted. In other words, a difficult-insertion shape determination step may be provided that, during an insertion operation of an endoscope having an imaging device at its distal end without using an insertion shape observation device, determines whether the endoscope insertion operation is in the difficult-insertion shape by comparing a series of images of the lumen obtained by time-series imaging with the difficult-insertion state images. By implementing the data processing method, it is possible to determine the shape of the endoscope inside the lumen that led to the current image being obtained. By conveying this shape information to the operator, it is possible to provide them with an understanding of how to deal with the difficult state.

[0059] When the learning image frame determination unit 16 receives a notification indicating that a difficult-insertion state has been detected from the difficult-insertion state determination unit 14, it determines one or more image frames from among the image frames that are obtained a predetermined time or a predetermined number of frames before the difficult-insertion state determination timing by the difficult-insertion state determination unit 14 as difficult-insertion state images. Only necessary information may be used as the difficult-insertion state image, and the images may be thinned out and acquired, for example. For example, the difficult-insertion state image may include an image frame before insertion into the curved lumen, an image frame during insertion, and an image frame in a stuck state.

[0060] The insertion difficulty state determination unit 14 may determine the insertion difficulty state by image analysis of the input image. For example, when the insertion portion advances through the lumen, the image portion of the lumen deep inside (deep in the direction of the lumen length) where illumination light from the endoscope tip does not reach has a low-brightness lumen cross-sectional shape (often approximately circular). As the insertion portion advances through the lumen, this image portion is located approximately at the center of the endoscopic image, and continuous images are obtained in which the lumen wall pattern moves toward the periphery of the image. The insertion difficulty state determination unit 14 can determine the state in which the insertion portion advances through the lumen by analyzing the input image. Conversely, it can determine a state in which insertion is difficult and the insertion portion is not advancing through the lumen, such as when there is no cavity ahead of the insertion portion. The insertion difficulty state determination unit 14 may determine the insertion difficulty state when a predetermined number of consecutive images do not include such an image portion of a cavity.

[0061] As described above, the good insertion state determination unit 15 uses the UPD information to determine whether each input image frame is in a good insertion state. For example, during insertion into the splenic curvature, the good insertion state determination unit 15 stores a bent shape pattern in a memory (not shown) and determines whether the insertion shape image of the insertion portion obtained from the UPD information forms a shape pattern similar to the stored shape pattern, thereby determining whether the shape of the insertion portion resembles the bent shape curve. When the good insertion state determination unit 15 determines from the UPD information that the insertion portion has formed a curve similar to the bent shape curve, it determines that the good insertion state has been reached. The good insertion state determination unit 15 outputs a notification indicating that the good insertion state has been detected to the learning image frame determination unit 16 at the timing for determining the good insertion state.

[0062] When the learning image frame determination unit 16 receives a notification from the good insertion state determination unit 15 indicating that a good insertion state has been detected, it determines, as a good insertion state image, one or more image frames from among the image frames that are obtained a predetermined time or a predetermined number of frames before the good insertion state determination timing by the good insertion state determination unit 15. As the good insertion state image, only necessary information may be used, and for example, thinned images may be used. For example, the good insertion state image may include an image frame before insertion into the lumen bending portion, an image frame during insertion, and an image frame after passing through the lumen bending portion.

[0063] The difficult-insertion images and good-insertion images determined by the learning image frame determination unit 16 are, for example, a series of chronologically consecutive images that record the surgeon contemplating how to insert the difficult-insertion portion and determining the insertion position and the angle at which the endoscope insertion portion is brought into contact with the difficult-insertion portion. That is, the features of these images correspond to features (hereinafter referred to as insertion features) that indicate the difficulty of insertion in response to the surgeon's insertion operation. Because such insertion features are related to the insertion state based on the UPD information, learning the difficult-insertion images and good-insertion images is believed to enable effective insertion assistance in response to the surgeon's insertion operation.

[0064] The guide information generator 13 generates guide information for providing such insertion assistance. The guide information generator 13 includes a post-insertion endoscope shape determiner 13a and a shape correction guide information generator 13b. The post-insertion endoscope shape determiner 13a acquires the shape of the insertion section based on the UPD information, and determines the shape of the lumen into which the insertion section is inserted based on information about the observation site (described below) and the shape of the insertion section. In this way, the post-insertion endoscope shape determiner 13a determines the shapes of the lumen and the insertion section present in the lumen, and obtains insertion shape information that indicates the insertion state of the insertion section into the lumen.

[0065] The shape correction guide information unit 13b outputs the insertion shape information determined by the post-insertion endoscope shape determination unit 13a as guide information for displaying a predicted insertion image, and also generates and outputs guide information for displaying an insertion support display corresponding to the insertion shape information. The predicted insertion image is an image predicted as the insertion shape of the insertion part when a difficult-insertion state or a good-insertion state occurs, and is intended to display an image predicted as the insertion shape in advance before the difficult-insertion state or the good-insertion state actually occurs.

[0066] Possible insertion support displays include a display indicating the ease or difficulty of inserting the insertion section. For example, insertion support displays may include a display of text such as "Leave as is" when a good insertion state is predicted, a display of text such as "Caution" when the degree of difficulty in a difficult insertion state is relatively low or the probability of a difficult insertion state occurring is relatively low, and a display of text such as "Try again" when the degree of difficulty in an difficult insertion state is relatively high or the probability of a difficult insertion state occurring is relatively high. Possible insertion support displays may also include a display indicating that a procedure such as "manual compression," "position change," "hardness change," or "axial retention shortening" should be performed to achieve a good insertion state.

[0067] FIG. 6 is an explanatory diagram showing an example of a guide display Ga indicating that manual compression is recommended. The example in FIG. 6 includes a display L1 indicating the lumen, a display I1 indicating the insertion portion, a display Ga1 indicating the direction of manual compression, and a display I2 showing an ideal insertion example of the insertion portion. The shape modification guide information unit 13b can create the displays L1, I1, and I2 based on information about the observation site determined by the observation site determination unit 12 and information about the insertion shape determined by the post-insertion endoscope shape determination unit 13a. The information processing unit 10 includes an input device (not shown), and the control unit 11 can provide the shape modification guide information unit 13b with information based on a user's input operation on the input device. By inputting information about the required procedure by the user, the shape modification guide information unit 13b can create the display Ga1 based on the user's operation.

[0068] The control unit 11 can also access a knowledge database (DB) 40. The knowledge DB 40 is a database in which information on tests, treatments, etc. is organized and recorded by disease symptoms for each body part. A countermeasure information recording area 41 of the knowledge DB 40 stores information on the shape of the insertion part and correction methods that indicate what kind of technique can be used for this shape to easily insert the insertion part. The knowledge DB 40 may also store information indicating the relationship between the force applied to the insertion part and pain in the human body. The control unit 11 can obtain information on the necessary technique, for example, information on a method for quickly recovering after getting stuck, from the knowledge DB 40 based on the insertion shape information. In this case, the shape correction guide information unit 13b can create the display Ga1 based on the information from the control unit 11.

[0069] The observation part determination unit 12 determines the part of the body that was captured in the input image. The observation part determination unit 12 determines the part of the body from the input image. For example, the observation part determination unit 12 may determine the part by referring to a database (not shown) that contains the characteristic color or shape of the part, or patterns of blood vessels or the like visible on the surface. The observation part determination unit 12 may determine the part by using known AI (artificial intelligence) for part determination.

[0070] The control unit 11 can organize information about difficult-insertion state images and good-insertion state images determined by the learning image frame determination unit 16 and guide information generated by the guide information generation unit 13 for a large number of images input to the data processing device 1, and record the information in the recording unit 20. In this case, by using the determination results of the observation region determination unit 12, it is possible to organize and record information for each region.

[0071] The recording unit 20 includes a recording medium (not shown) and multiple areas for storing images provided by the control unit 11. As shown in FIG. 2, the recording unit 20 may include multiple recording areas for each observation site. Furthermore, the recording unit 20 may include multiple recording areas for each case or procedure. The example in FIG. 1 shows an example where the observation sites are classified, with the first observation site, the second observation site, etc. indicating the respective recording areas for each observation site. The recording unit 20 includes recording areas R1, R2, etc. for different types of lumen curvatures (hereinafter, when there is no need to distinguish between the recording areas R1, R2, etc., they are referred to as recording areas R). The control unit 11 associates difficult-insertion images with guide information corresponding to the difficult-insertion images in the recording area R, and records them for each insertion characteristic indicating difficult insertion. The control unit 11 also associates good-insertion images with guide information corresponding to the good-insertion images in the recording area R, and records them for each insertion characteristic indicating good insertion.

[0072] The associated guide information is added as annotation information to the difficult-insertion image, and teacher data is created from the difficult-insertion image and the annotation information. The associated guide information is added as annotation information to the good-insertion image, and teacher data is created from the good-insertion image and the annotation information. That is, the recording unit 20 records teacher data for constructing an inference model that takes a difficult-insertion image as input and outputs, as an inference result, guide information according to insertion features corresponding to the surgeon's insertion operation, and teacher data for constructing an inference model that takes a good-insertion image as input and outputs, as an inference result, guide information according to insertion features corresponding to the surgeon's insertion operation.

[0073] In the above explanation, an example was given in which guide information corresponding to insertion features was classified into, for example, "as is," "care required," and "redo," but the classification of guide information is not limited to this.

[0074] If the accompanying information includes information on the surgeon's skill, the records in the recording unit 20 may be organized so that the training data is changed for each skill and learning is performed, such as "difficult for an inexperienced operator" or "easy for an experienced operator." By performing such learning and using the inference results obtained using the inference model, an inexperienced operator may be guided to learn the procedure of an experienced operator or may be guided to seek assistance from an experienced operator. Furthermore, if a procedure is difficult for an inexperienced operator but relatively easy for an experienced operator, the procedure status (insertion position and angle) of the inexperienced operator may be monitored and compared with the operation status (insertion position and angle) of the experienced operator. The difference between the insertion position and angle may be indicated, for example, by displaying the difference between the current position and the insertion point of the experienced surgeon on an image using text, a graphic (image) including an arrow, audio, or the like, so that the inexperienced operator can understand the difference.

[0075] The learning unit 30 includes a teacher data generation unit 31, a neural network 32, and a reliability determination unit 33. The teacher data generation unit 31 receives data from the recording unit 20, in which images of a difficult-insertion state or an image of a good-insertion state are associated with various guide information. The teacher data generation unit 31 creates teacher data by annotating the images of a difficult-insertion state or an image of a good-insertion state with various guide information. The teacher data generation unit 31 selects one or more image frames to be used for learning from the annotated images of a difficult-insertion state or an image of a good-insertion state. The neural network 32 receives teacher data from the one or more image frames selected by the teacher data generation unit 31. The neural network 32 calculates parameters for the neural network 32 through deep learning using the teacher data.

[0076] Deep learning is a multilayered version of the machine learning process using neural networks. A typical example is a forward propagation neural network, which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of m1 neurons, a hidden layer consisting of m2 neurons determined by parameters, and an output layer consisting of m3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, neural networks with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0077] The network 31 used for machine learning may be any of a variety of well-known networks. For example, R-CNN (Regions with CNN features), FCN (Fully Convolutional Networks), 3DCNN (3D Convolutional Neural Network), etc., which use a CNN (Convolution Neural Network), may be used. These involve a process called "convolution" that compresses image features, operate with minimal processing, and are strong in pattern recognition. Furthermore, a "recurrent neural network" (fully connected recurrent neural network), which can handle more complex information and allows information analysis whose meaning changes depending on the order or sequence, and in which information flows bidirectionally, may also be used.

[0078] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, GPUs and Tensor Processing Units (TPUs), which are specialized for matrix calculations, may also be used.In recent years, such dedicated artificial intelligence (AI) hardware, called "neural network processing units (NPUs)," have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0079] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0080] The reliability determination unit 33 determines the reliability of the inference result obtained by providing test data to the neural network 32. The reliability determination unit 33 controls the teacher data generation unit 31 to repeat deep learning while correcting the teacher data and guide information, which is annotation information, so that a sufficiently reliable inference result is obtained.

[0081] The communication unit 17 of the information processing unit 10 is capable of wireless or wired data communication with the learning unit 30. The control unit 11 exchanges data with the learning unit 30 via the communication unit 17. When the reliability determination unit 33 requests correction of the teacher data, the control unit 11 may control the learning image frame determination unit 16 to correct the image frame of the difficult-insertion state image or the good-insertion state image, and provide the corrected image frame to the learning unit 30 to use as teacher data.

[0082] FIG. 7 is an explanatory diagram for explaining learning for constructing an inference model.

[0083] 7 shows that the series of images acquired in procedures a to c of Case A and procedure c of Case B are classified by the information processing unit 10 into difficult-insertion images and good-insertion images according to their insertion characteristics. Cases A and B are not necessarily images of the same site; the difficult-insertion images may be different from each other, and the good-insertion images may also be different from each other. Furthermore, these difficult-insertion images and good-insertion images are annotated with insertion shape information indicating the shape of the lumen and the insertion part in the lumen as guide information for displaying an expected insertion image. Furthermore, these difficult-insertion images and good-insertion images are annotated with guide information indicating insertion characteristics, such as "As is," "Caution," or "Retry," based on the insertion shape information.

[0084] This information is supplied as training data from the recording unit 20 to the neural network 32 of the learning unit 30. FIG. 7 shows that training data is provided to the network 31. Taking training with training data as an example, the network design is determined so that the network 31 can obtain an output corresponding to each input by training using a large amount of training data, for example, deep learning. An inference model is constructed by the network 31.

[0085] In the example of Figure 7, an endoscopic image is provided to the inference model constructed in this way. When an image of a difficult insertion state or an image of a good insertion state contained in the endoscopic image is input to the inference model, the inference model outputs information indicating the level of difficulty of insertion, such as "Caution required," and an image predicted for insertion if the insertion part is continued as is. For example, the example of Figure 7 shows that the inference result can be displayed as "Easy," indicating ease of insertion.

[0086] Next, the operation of the data processing device configured in this manner will be described with reference to Figures 8 to 10. Figure 8 is a flowchart for explaining the creation of training data, Figure 9 is a flowchart for explaining learning using training data, and Figure 10 is an explanatory diagram for explaining the selection range of image frames to be acquired as difficult-to-insert images or good-insertion images.

[0087] Image organization (organization of training data) will be described with reference to Fig. 8. The information processing unit 10 of the data processing device 1 accesses a group of endoscopic images such as images PA, PB, etc. The information processing unit 10 captures the target image. The difficult-insertion state determination unit 14 and the good-insertion state determination unit 15 of the information processing unit 10 analyze and classify the UPD information of the lumen curvature site (S1).

[0088] UPD information indicates the shape of the endoscope's insertion section when inserted, based on positional information of each part of the insertion section obtained from magnetic field information from UPD coils 55, which generate alternating magnetic fields and are arranged at predetermined intervals along the insertion section 51a as shown in Figure 4. The UPD information can be used to classify whether the insertion section has a specific shape pattern. As described above, shape patterns that result in poor endoscope insertion can be determined by determining whether the insertion section's insertion shape image forms a shape pattern similar to a stored shape pattern. Since detection is possible at approximately 4 to 5 frames per second, it is also possible to detect, for example, a movement of the UPD coil located near the anus while a movement of the UPD coil near the tip does not occur relative to the direction of progression. Since the UPD coil at the tip moves due to angle manipulation, it is also possible to determine whether the movement is due to angle manipulation. Because there are differences in the detection signals of each UPD coil, the location of each UPD coil can be determined. In other words, UPD information represents the pattern representing the endoscope's shape at that time, as well as changes in the shape and the position of each coil over time. This information can be used to determine insertion difficulties and how to resolve them. Of course, information on the process of how the difficult state was resolved is also available, which can serve as guide information when insertion is difficult.

[0089] The difficult-insertion state determination unit 14 and the good-insertion state determination unit 15 determine whether or not they have been able to determine the insertion difficulty level for determining whether the state is difficult-insertion or good-insertion based on the classification of the UPD information (S2). If the difficult-insertion state determination unit 14 and the good-insertion state determination unit 15 cannot determine the insertion difficulty level, they repeat the process of S1, and if they have been able to determine the insertion difficulty level, they output the determination result of whether the state is difficult-insertion or good-insertion to the learning image frame determination unit 16.

[0090] The learning image frame determination unit 16 selects difficult-insertion images or good-insertion images, which are images taken during insertion into the lumen bending portion, according to the classification result of the UPD information (S3). For example, the learning image frame determination unit 16 may select difficult-insertion images or good-insertion images, which include image frames taken before, during, and when the insertion portion is stuck in the lumen bending portion.

[0091] In S4, the control unit 11 starts organizing moving images for each part of the lumen bending portion. Furthermore, in S5, the control unit 11 starts organizing moving images for each type of lumen bending portion.

[0092] The post-insertion endoscope shape determination unit 13a of the guide information generation unit 13 obtains insertion shape information that indicates the insertion state of the insertion portion into the lumen. Based on the insertion shape information, the shape correction guide information unit 13b generates guide information for displaying an expected insertion image, and guide information for an insertion support display that indicates the difficulty of insertion in response to the insertion shape information and that indicates techniques for achieving a good insertion state.

[0093] The control unit 11 determines whether the guide information generating unit 13 has been able to generate guide information (S6), and if not, proceeds to S8. If successful, the control unit 11 associates the guide information with the difficult-insertion image or the good-insertion image (S7). The control unit 11 provides the difficult-insertion image or the good-insertion image associated with the guide information to the recording unit 20 for recording, and then proceeds to S8. The control unit 11 may also organize and record the images by lumen curvature site and by insertion feature.

[0094] In S8, the control unit 11 determines whether image organization by type has been completed. If image organization by type has not been completed, the control unit 11 proceeds to process S9, selects another image of the same type, and returns to process S6. If image organization by type has been completed, the control unit 11 proceeds to process S10, and determines whether image organization by lumen curvature site has been completed. If image organization by lumen curvature site has not been completed, the control unit 11 proceeds to process S11, selects another image of the same lumen curvature site, and returns to process S5. If image organization by lumen curvature site has been completed, the control unit 11 terminates the process.

[0095] Next, learning using training data will be described with reference to FIG.

[0096] By performing learning using a large amount of training data recorded in the recording unit 20, an inference model is constructed that determines whether the insertion is difficult or successful and outputs guide information. Such inference models are employed as first and second inference models 74a and 74b in FIG. 11 , which will be described later. The training data generation unit 31 acquires images for each lumen curvature site from the recording unit 20 (S21). The training data generation unit 31 generates training data by adding, as annotation information, guide information associated with the read images for the difficult-insertion state or the successful-insertion state for each lumen curvature type (S22). The training data generation unit 31 selects one or more image frames to be used for learning from the annotated images for the difficult-insertion state or the successful-insertion state (S23).

[0097] The neural network 32 is provided with training data based on one or more image frames selected from the training data generation unit 31, and calculates parameters for the neural network 32 through deep learning using the training data (S24). Through this learning, the parameters for the neural network 32 are determined, and an inference model is constructed.

[0098] The reliability determination unit 33 provides test data to the constructed inference model and determines the reliability of the inference result (S25). In the next step S26, the reliability determination unit 33 determines whether the reselection of image frames has been completed, and if not, controls the teacher data generation unit 31 to reselect image frames in S27.

[0099] FIG. 10 shows image frames representing difficult-to-insert states or good-insertion states acquired by the learning image frame determination unit 16. In the example of FIG. 10, one pulse shape represents one image frame P1, P2, .... That is, the example of FIG. 10 shows an example in which the number of image frames representing difficult-to-insert states or good-insertion states is eight. For example, if a series of five temporally consecutive image frames is selected as training data, the training data generation unit 31 may use image frames P1 to P5 as training data during the first learning, image frames P2 to P6 as training data during the second learning, image frames P3 to P7 as training data during the third learning, and image frames P4 to P8 as training data during the fourth learning.

[0100] The teacher data generation unit 31 may select image frames by employing various selection methods. For example, assuming that image frame P8 is the image frame at the time of stacking, as shown by the arrow in Figure 10, image frame P8 may be selected first as teacher data, and then image frame P7 may be selected as teacher data, so that image frames obtained earlier in time are selected one by one in sequence.

[0101] In practice, the difficult-to-insert images or good-insertion images acquired by the learning image frame determination unit 16 often include image frames for a duration longer than one second, and assuming a frame rate of 60 fps (frames per second), for example, 30 image frames, which allow for negligible inference delay, may be used as training data. Furthermore, the training data generation unit 31 may generate training data by thinning out the image frames of the series of difficult-to-insert images or good-insertion images acquired by the learning image frame determination unit 16.

[0102] The reliability determination unit 33 controls the teacher data generation unit 31 to repeat deep learning while modifying the teacher data and guide information, which is annotation information, so as to obtain a sufficiently reliable inference result. When the reliability determination unit 33 determines that the reselection of image frames has been completed (YES in S26), it adopts an inference model obtained by learning teacher data with an earlier end of an image frame from among inference models whose inference results for the test data have reliability equal to or greater than a predetermined threshold (S28). This enables the inference result to be obtained at an earlier timing, making it possible to display a guide display, etc., relatively early before the insertion part gets stuck.

[0103] The teacher data generation unit 31 determines whether another video to be learned is selected (S29). If learning has not been completed for another image of the same lumen curvature type, the teacher data generation unit 31 changes the lumen curvature type (S30), returns the process to S22, and annotates guide information for an image of another lumen curvature type. Furthermore, if learning has not been completed for another image of the same lumen curvature site, the teacher data generation unit 31 changes the lumen curvature site (S31), returns the process to S21, and acquires an image of another lumen curvature site. If there is no other video to be learned, the teacher data generation unit 31 ends the process.

[0104] In this way, the learning unit 30 obtains an inference model that can output guide information for endoscopic images acquired in a difficult insertion state or a good insertion state.

[0105] (Endoscope System) FIG. 11 is a block diagram showing an endoscope system that uses an inference model constructed based on training data obtained by the data processing device of FIG.

[0106] The endoscopic system in Fig. 11 includes an endoscope 60, an image processing unit 70, and a display device 80. The endoscope 60, which serves as a second endoscope, has an insertion section 61, which serves as a second insertion section, and an imaging device 62 is provided at the tip of the insertion section 61. The imaging device 62 captures images of the inside of a body into which the insertion section 61 is inserted, acquires captured images in chronological order, and supplies the acquired captured images to the image processing unit 70. Note that Fig. 11 shows only the configuration of the image processing unit 70 related to the inference processing, but the image processing unit 70 also includes, in addition to the image signal processing unit described above, an operation input unit that accepts user operations, various communication functions for communicating with the outside, and the like.

[0107] The image processing unit 70 includes a control unit 71, an image data input unit 72, a data analysis unit 73, first and second inference models 74a and 74b, a guide information generation unit 76, and a display control unit 77. The control unit 71 and each unit of the image processing unit 70 may be configured by a processor using a CPU, GPU, FPGA, NPU, etc., and may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of the functions using a hardware electronic circuit. The control unit 71 comprehensively controls the entire image processing unit 70.

[0108] The image data input unit 72 captures image information from the endoscope 60. The data analysis unit 73 analyzes the received image information to determine the insertion length and acceleration of the insertion unit, and thereby determine the force exerted by the insertion unit. The data analysis unit 73 can determine the insertion length and acceleration of the insertion unit 61 based on image analysis of the input image. The data analysis unit 73 may also determine the insertion length and acceleration of the insertion unit 61 using accompanying information accompanying the input image information. The data analysis unit 73 determines the degree of pain of the subject from the determined force.

[0109] The first and second inference models 74a, 74b (hereinafter, when there is no need to distinguish between these inference models, they will be collectively referred to as the inference model 74) are models constructed based on learning of training data created by the data processing device 1 of FIG. 1, and are models that correspond to different lumen curvature regions or lumen curvature types, for example. The control unit 71 provides the first and second inference models 74a, 74b with images captured in time series by the imaging device 62. Note that, when the first and second inference models 74a, 74b have specifications for each region, the control unit 71, acting as a region determination unit, determines the region of the input image and then provides the input image to the inference model 74 for the corresponding region. Note that the control unit 71 may perform region determination using known AI that performs region determination.

[0110] The inference model 74 is controlled by the control unit 71 to determine whether the input captured image is a difficult-to-insert image or a good-insertion image, and outputs guide information corresponding to the determined difficult-to-insert image or good-insertion image.

[0111] The guide information generation unit 76 generates guide information for displaying an insertion support display based on the degree of pain calculated by the data analysis unit 73. The guide information generation unit 76 provides the generated guide information and guide information obtained by the inference model 74 to the display control unit 77.

[0112] The display control unit 77 provides the image from the imaging device 62 to the display device 80, causing it to be displayed on the display screen 80a. The display control unit 77 also displays various support displays M1 based on the guide information on the endoscopic image displayed on the display screen 80a. In the example of Fig. 11, the endoscopic image is displayed on the display screen 80a, and the support display M1 saying "Please put it back" is also displayed.

[0113] The support display M1 allows the surgeon inserting the endoscope to recognize that the distal end of the insertion portion 61 has reached a difficult-to-insert portion as a result of the insertion operation of the insertion portion 61, and that it would be better to pull back the insertion portion 61 in order to insert the insertion portion 61 into this difficult-to-insert portion. Thus, the endoscopic system of FIG. 11 provides effective support for smooth insertion, and even a relatively inexperienced surgeon can easily learn the operations required for inserting the insertion portion 61. Furthermore, the inference model 74 may output an inference result before the insertion portion 61 actually gets stuck in the difficult-to-insert portion, allowing the surgeon to determine the operations required for smooth insertion into the difficult-to-insert portion before the insertion becomes difficult.

[0114] Next, the operation of the endoscope system configured as above will be described with reference to Fig. 12. Fig. 12 is a flowchart for explaining the operation of the endoscope system of Fig. 11.

[0115] When imaging by the endoscope 60 begins, in S41 of Fig. 12, the image data input unit 72 acquires an endoscopic image from the endoscope 60. The endoscopic image may have accompanying information added thereto. The input endoscopic image is subjected to predetermined image processing by the image processing unit 70, and then displayed on the display screen 80a of the display device 80 (S42). The control unit 71 may also provide the endoscopic image to a recording device (not shown) for recording.

[0116] The control unit 71 determines which part of the body the input endoscopic image was obtained by capturing. For example, the control unit 71 provides the image to a part inference model (S43) and obtains a part determination result (S44). Based on the part determination result, the control unit 71 provides the input image to a part-specific inference model 74 that matches the specifications of the inference model (S45). Note that if inference models for different types of lumen curvature parts are prepared, the control unit 71 determines which type the input image is and provides the input image to the corresponding inference model 74 based on this determination result.

[0117] The inference model 74 determines whether the input image is an image in a difficult-to-insert state or an image in a good-insertion state, and outputs the determination result together with the guide information. The control unit 71 determines the reliability of the determination result from the inference model 74 and determines whether there is an inference result with high reliability (above a predetermined threshold) (S46). For example, when the control unit 71 receives a determination result with reliability above a predetermined threshold from the inference model 74 for each type (YES in S46), the control unit 71 provides the inference result of the inference model 74 to the guide information generation unit 76 to determine the content of the guide display.

[0118] The guide information generation unit 76 determines the display content for insertion assistance and instructs the display control unit 77. That is, the guide information generation unit 76 generates guide information for displaying insertion assistance displays based on not only the guide information from the inference model 74 but also the strength determined by the data analysis unit 73, and provides the generated guide information to the display control unit 77. The display control unit 77 provides the image from the imaging device 62 to the display device 80 to display it on the display screen 80a, and also displays various assistance displays based on the guide information related to the currently displayed endoscopic image (S47). For example, assistance displays using text, images, illustrations, etc. are provided regarding the current situation and recovery measures for smooth insertion. The control unit 71 may record the display content based on the guide information determined by the guide information generation unit 76 in association with the corresponding image frame.

[0119] In the next step S48, the control unit 71 determines the type of the lumen curvature portion based on the inference result of the inference model 74, and records the determination result in association with the endoscopic image (S48). By recording in this manner, the characteristics of the subject's lumen can be effectively used for diagnosis or the next examination.

[0120] Next, the control unit 71 determines whether the display has ended (S49). If the display has not ended, the control unit 71 returns the process to S41 and repeats the process, and if the display has ended, the control unit 71 ends the process.

[0121] If a highly reliable inference result is not obtained in S46, the control unit 71 controls, for example, the display control unit 77 to display a display recommending an examination using a UPD device so that training data that will enable effective inference during the next endoscopic examination can be obtained.

[0122] 13 and 14 are explanatory diagrams showing examples of displays on the display screen 80a of the display device 80. In FIG. 13 , the display screen 80a displays an endoscopic image P11 on the left center, and a pseudo UPD display P12 on the right center. The pseudo UPD display P12 is created using guide information obtained from the inference results of the inference model 74, and is a predicted insertion image based on insertion shape information generated based on the UPD information. That is, the pseudo UPD display P12 shows the shape of the lumen curved portion L11 into which the insertion section 61 is to be inserted and a predicted insertion state I11 of the insertion section 61 obtained from the current image captured by the insertion section 61 (the input image of the inference model 74). By referring to this pseudo UPD display P12, the surgeon can understand, for example, whether there is a risk of the insertion section getting stuck or whether the insertion is proceeding smoothly before the insertion section gets stuck or passes through the lumen curved portion.

[0123] In addition, in FIG. 13 , an insertion support display P13 of "Caution: Pain" is also displayed. The strength of the insertion part is determined through the analysis by the data analysis unit 73, and the degree of pain that the subject is likely to feel is calculated according to this strength. The guide information generation unit 76 determines the content of the guide information (insertion support display) regarding pain based on the analysis results of the data analysis unit 73. In accordance with this determination, in the example of FIG. 13 , the display control unit 77 displays an insertion support display P13 to alert the subject that pain may occur. This display allows the surgeon to recognize that pain will occur in the subject, and can take care in inserting the catheter in a way that does not cause the subject to feel pain.

[0124] In this way, in addition to calling a situation in which the physical insertion operation of the endoscope does not lead to movement of the tip "difficult insertion," a situation in which the subject feels discomfort due to the insertion operation of the endoscope can also be considered "difficult insertion." Therefore, the "insertion difficulty determination" of this application may be defined as determining a situation in which the operator is unable to control the tip of the endoscope as intended. As explained above, this application effectively utilizes data obtained from an examination using an insertion shape observation device such as a UPD, enabling smoother endoscopic examinations even in situations where a shape observation device is not available. In other words, it provides benefits in terms of introduction cost and space.

[0125] 14 shows an example in which display P14, which is the same as guide display Ga in FIG. 6, is used instead of pseudo UPD display P12. That is, display P14 shows how to perform manual compression for smooth insertion, and by displaying display P14, the surgeon can understand effective techniques for smooth insertion. This assists in smooth movement of the tip and is a device to provide intuitive guidance.

[0126] As described above, this embodiment uses an inference model that outputs various guide information in response to input images of difficult-insertion states and images of successful insertion states, enabling effective insertion assistance. This allows the surgeon to grasp the insertion shape of the endoscope in the same way as if a UPD device were used during the examination, without using a UPD device. Furthermore, before the insertion portion becomes stuck, for example, it is possible to present the surgeon with an image of the expected insertion or appropriate techniques, making smooth insertion easy even for surgeons with limited experience in inserting endoscopes.

[0127] Although the above description has primarily focused on displaying the inference results, the inference results may also be presented to the user by voice. Furthermore, in recent years, automatic insertion endoscopes capable of automatically inserting an endoscope into a lumen have been developed. Therefore, in addition to presenting the inference results, such automatic insertion endoscopes may also be controlled based on the inference results. For example, smooth insertion into difficult-to-insert sections can be achieved by automatically adjusting the angle of the bending portion of the insertion section or the insertion depth of the insertion section based on the inference results.

[0128] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0129] Furthermore, among the technologies described herein, many of the controls and functions, mainly those described in the flowcharts, can be set by a program, and the above-described controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored, in whole or in part, as a computer program product on a portable medium such as a flexible disk, CD-ROM, or nonvolatile memory, or on a storage medium such as a hard disk or volatile memory, and can be distributed or provided at the time of product shipment, via a portable medium, or via a communication line. A user can easily realize the data processing device, endoscope system, data processing method, insertion guide method, and inference model learning method of the present embodiments by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. A data processing device comprising: a difficult-insertion state determination unit that acquires insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion part of an endoscope, and determines a difficult-insertion state in which it has become difficult to insert the insertion part into a lumen based on the acquired insertion shape information; a learning image frame determination unit that is given a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion part, and determines one or more image frames after an image frame acquired before a predetermined time before the timing of determining the difficult-insertion state as a difficult-insertion state image; a guide information generation unit that generates guide information for providing assistance with inserting the insertion part based on the difficult-insertion state image; and a recording unit that records the difficult-insertion state image in association with the guide information.

2. The data processing device according to claim 1, further comprising a good insertion state determination unit that determines a good insertion state in which the insertion part has been inserted into the lumen well based on the insertion shape information, wherein the learning image frame determination unit is given a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion part, and determines one or more image frames after an image frame acquired a predetermined time before the timing of determining the good insertion state as a good insertion state image, the guide information generation unit generates guide information for providing assistance with insertion of the insertion part for the difficult insertion state image and the good insertion state image, and the recording unit records the difficult insertion state image and the good insertion state image in association with the guide information.

3. The data processing device according to claim 1, wherein the difficult-to-insert state determination unit determines the difficult-to-insert state by comparing the insertion shape pattern of the insertion part obtained based on the insertion shape information with a pattern stored as an insertion shape pattern that occurs in the difficult-to-insert state.

4. The data processing device according to claim 3, wherein the recorded pattern is in the shape of a stick or a loop.

5. The data processing device according to claim 1, wherein the guide information generation unit acquires, as the guide information, insertion shape information indicating the insertion shape of the insertion part, which is obtained based on the insertion shape information, in order to display an expected insertion image that is predicted as the insertion shape of the insertion part when the difficult insertion state or the good insertion state occurs.

6. The data processing device according to claim 1, wherein the guide information generating unit generates guide information for displaying the ease or difficulty of insertion of the insertion unit.

7. The data processing device according to claim 1, wherein the guide information generating section generates guide information for guiding a procedure for obtaining a good insertion state of the insertion section.

8. An endoscopic system comprising: an inference model created by learning training data obtained by annotating guide information for providing assistance with the insertion of a first insertion part to difficult-insertion state images including one or more image frames obtained by imaging a lumen using an imaging device provided in a first insertion part of a first endoscope, the difficult-insertion state images including one or more image frames obtained after an image frame obtained a predetermined time before a determination timing of a difficult-insertion state in which insertion of the first insertion part into the lumen has become difficult; a second endoscope that inserts a second insertion part into the lumen and obtains image images of the lumen in chronological order; and a control unit that provides the image images obtained by the second endoscope to the inference model and causes the inference model to output guide information to support insertion as an inference result.

9. The endoscope system according to claim 8, wherein the control unit outputs the inference result before the second insertion unit becomes stuck.

10. An endoscopic system as described in claim 8, further comprising a part determination unit that determines the part into which the second insertion part is inserted, the inference model having a plurality of models for each part, and the control unit providing the captured image acquired by the second endoscope to a model corresponding to the result of the part determination.

11. The endoscope system according to claim 8, further comprising a guide information generating section that predicts pain to the subject when the second insertion section is inserted and generates guide information relating to the pain.

12. A data processing method comprising: acquiring insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion part of an endoscope; determining a difficult-insertion state in which it has become difficult to insert the insertion part into a lumen based on the acquired insertion shape information; providing a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion part; determining one or more image frames after an image frame acquired a predetermined time before the timing of determining the difficult-insertion state as difficult-insertion state images; generating guide information for the difficult-insertion state images to provide assistance with inserting the insertion part; and recording the difficult-insertion state images in association with the guide information.

13. An insertion guide method comprising: providing an inference model created by learning training data obtained by annotating a series of image frames of a lumen obtained by imaging with an imaging device provided in a first insertion portion of a first endoscope, and annotating guide information for providing assistance with the insertion of the first insertion portion for difficult-insertion state images including one or more image frames after an image frame obtained a predetermined time before a determination timing of a difficult-insertion state in which insertion of the first insertion portion into the lumen has become difficult; providing the image frames from a second endoscope that inserts a second insertion portion into the lumen and obtains image images of the lumen in chronological order; outputting guide information to assist with insertion from the inference model as an inference result; and displaying a guide display based on the guide information.

14. The insertion guide method according to claim 13, wherein the guide display is a display of an expected insertion image that is expected as the insertion shape of the second insertion part before the second insertion part is stacked.

15. A method for learning an inference model, comprising: acquiring insertion shape information from an insertion shape observation device that observes the insertion shape of an insertion part of an endoscope; determining a difficult-insertion state in which it has become difficult to insert the insertion part into a lumen based on the acquired insertion shape information; providing a series of image frames of the lumen obtained by imaging with an imaging device provided in the insertion part; determining one or more image frames after an image frame acquired a predetermined time before the timing of determining the difficult-insertion state as difficult-insertion state images; generating guide information for providing assistance with inserting the insertion part for the difficult-insertion state images; and creating an inference model by learning using training data obtained by annotating the guide information for the difficult-insertion state images.

16. A method for learning an inference model as described in claim 15, which repeats learning using training data in which the image frames included in the difficult-to-insert state image used for learning are changed, and adopts an inference model created using training data in which the end of the image frame is earlier than the predetermined threshold value among inference models that achieve reliability above the predetermined threshold value.

17. The data processing device according to claim 1, wherein the difficult-to-insert state determination unit determines the positions of the endoscope insertion portion, thereby making it possible to observe the insertion shape over time, and determines time-series change information from the insertion shape observation device, and determines a state in which the tip of the endoscope does not move in response to the movement of the endoscope insertion / removal operation, thereby determining that insertion is difficult.

18. A data processing method comprising the steps of: during insertion of a first endoscope having a first image capturing device at its tip, in conjunction with an insertion shape observation device, acquiring insertion shape information from the insertion shape observation device for observing the insertion shape in time series depending on the position of each part of the insertion section of the first endoscope; determining, based on the acquired time-series insertion shape information, a timing for determining when the insertion section has become difficult to insert into a lumen; providing a series of image frames of the lumen obtained by time-series imaging with the first image capturing device, determining the image frames up to the timing for determining when the insertion is difficult as difficult-insertion state images; and during insertion of a second endoscope having a second image capturing device at its tip without using an insertion shape observation device, comparing the series of image frames of the lumen obtained by time-series imaging with the difficult-insertion state images to determine whether the insertion of the second endoscope will result in the difficult-insertion state.

19. The data processing method according to claim 18, further comprising the step of notifying the endoscope operator that the shape of the second endoscope during insertion is in a state of sticking or re-looping, according to the result of the determination as to whether or not the shape will cause insertion to be difficult.

20. The data processing method of claim 18, wherein the timing for determining that insertion is difficult is determined when the movement of the proximal side of the insertion section of the first endoscope is a change in the pushing state by the operator, while the movement of the tip section changes only below a specific level.

Images