Endoscope operation assistance device, control method, computer-readable medium, and program

Abstract

The invention relates to an endoscope operation assistance device, a control method, a computer-readable medium, and a program. An endoscope operation assistance device (2000) acquires a shape data sequence (50) indicating a temporal change in shape data of an endoscope body (40). Using the shape data sequence (50), the endoscope operation assistance device (2000) specifies an insertion method for inserting the endoscope body (40). The endoscope operation assistance device (2000) outputs insertion method information (20) relating to the specified insertion method.

Description

[0001] Explanation of Divisional Application

[0002] This application is a divisional application of patent application filed on October 21, 2020, with application number 202080106162.7 and entitled "Endoscopic operation auxiliary device, control method, computer-readable medium and program". Background Technology

[0003] This disclosure relates to methods for assisting in the operation of an endoscope.

[0004] Devices are being developed to assist in endoscopic surgeries and examinations. For example, Patent Document 1 discloses a technique that infers the current state related to endoscopic operation and outputs auxiliary information corresponding to that state. Examples of the current state related to endoscopic operation include, for instance, "insertion without major problems" and "the state of excessive proximity continues." Furthermore, the auxiliary information indicates endoscopic operations that are effective in resolving the current state.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: International Publication No. 2018 / 235185 Summary of the Invention

[0008] The information used to assist endoscopic procedures is not limited to the information described above. The purpose of this disclosure is to provide a technique for presenting new information to assist endoscopic procedures.

[0009] According to one aspect of this disclosure, an endoscope operation assistance device is provided, the endoscope operation assistance device comprising: an acquisition unit configured to acquire a shape data sequence representing a change in shape data of an endoscope body over time; a determination unit configured to determine an insertion method for inserting the endoscope body based on the shape data sequence; and an output unit configured to output insertion method information related to the determined insertion method.

[0010] According to another aspect of this disclosure, a control method executed by a computer is provided. The control method includes: an acquisition step of acquiring a shape data sequence representing a change in shape data of an endoscope body over time; a determination step of determining an insertion method for inserting the endoscope body based on the shape data sequence; and an output step of outputting insertion method information related to the determined insertion method.

[0011] According to another aspect of this disclosure, a computer-readable medium is provided that stores a program that causes a computer to execute the control method of the present invention.

[0012] Beneficial technical effects

[0013] According to this disclosure, a technique is provided to present new information to assist in endoscopic procedures. Attached Figure Description

[0014] Figure 1 This is a diagram illustrating the general operation of the endoscope operation aid device according to Embodiment 1.

[0015] Figure 2 This is a block diagram illustrating the functional structure of the endoscope operation assistance device according to Embodiment 1.

[0016] Figure 3 This is a block diagram illustrating the hardware structure of a computer that implements an endoscopic operation aid.

[0017] Figure 4 This is a diagram illustrating a specific example of the environment in which an endoscope operation aid is used.

[0018] Figure 5 This is a flowchart illustrating the process performed by the endoscope operation assistance device of Embodiment 1.

[0019] Figure 6 It is a diagram that illustrates the structure of a shape data sequence in tabular form.

[0020] Figure 7 This is a diagram that conceptually illustrates the method for calculating likelihood using equation (1).

[0021] Figure 8 This is a diagram illustrating the insertion method information. Detailed Implementation

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding elements are labeled with the same reference numerals, and repeated descriptions are omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and thresholds are stored in advance in a storage device accessible from a device utilizing those values.

[0023] Figure 1 This is a diagram illustrating the general operation of the endoscope operation assistance device 2000 according to Embodiment 1. Here, Figure 1 This is a diagram for easy understanding of the outline of the endoscope operation assistance device 2000. The operation of the endoscope operation assistance device 2000 is not limited to... Figure 1 As shown.

[0024] There are various types of operating methods (hereinafter referred to as insertion methods or surgical modes) for inserting the endoscope body into the body. For example, there are methods such as axis retention shortening, push method, or forming α-ring. The endoscope operation assistance device 2000 determines the insertion method for inserting the endoscope body 40 and outputs information related to the determined insertion method, namely insertion method information 20.

[0025] Here, the insertion of the endoscope body 40 is performed while the shape of the endoscope body 40 is being modified in various ways. Moreover, the way the shape of the endoscope body 40 is modified varies depending on the insertion method. Therefore, based on the change in the shape of the endoscope body 40 over time, the insertion method used can be determined.

[0026] Here, the endoscope operation assistance device 2000 acquires a shape data sequence 50 representing the shape of the endoscope body 40 in a time sequence. Next, the endoscope operation assistance device 2000 uses the shape data sequence 50 to determine the insertion method to be used during the insertion of the endoscope body 40. Furthermore, the endoscope operation assistance device 2000 generates and outputs insertion method information 20 related to the determined insertion method. For example, the insertion method information 20 indicates the name of the determined insertion method.

[0027] <Examples of the effects>

[0028] As described above, the endoscope operation assistance device 2000 of this embodiment uses shape data sequence 50, which represents the change of the shape of the endoscope body 40 over time, to determine an insertion method for inserting the endoscope body 40, and outputs insertion method information 20 related to the insertion method. By using the insertion method information 20, the operator of the endoscope body 40 can master the insertion method for inserting the endoscope body 40.

[0029] For example, the actual movement of the endoscope body 40 may sometimes differ from the operator's expected movement. Therefore, it may be impossible for the operator to perform a certain insertion method even if they intend to. With the endoscope operation assistance device 2000 of this embodiment, the operator of the endoscope body 40 can easily determine whether the expected insertion method can be performed. If the expected insertion method has been performed, the operator can confidently continue operating the endoscope body 40 after confirming that the expected insertion method has been performed. Furthermore, if the expected insertion method cannot be performed, the operator can correct the operation to perform the expected insertion method after confirming that the expected insertion method has not been performed.

[0030] Furthermore, if the insertion method information 20 is stored in the storage device, a record of the insertion method used during the operation of the endoscope body 40 can be retained. By retaining such a record, it can be useful in future endoscopic surgeries, endoscopic examinations, etc. For example, by retaining a record of the insertion method used in an endoscopic surgery performed by an experienced physician, other physicians can share the techniques of that experienced physician. In addition, for example, by retaining a record of the insertion method used in association with the case, it can be used in explanations to patients, etc. Furthermore, the endoscope operation assistance device 2000 can also be configured to provide operation guidance to the operator based on the determined insertion method information. Thus, operation guidance corresponding to the insertion method desired by the operator can be provided.

[0031] The endoscope operation assistance device 2000 of this embodiment will be described in more detail below.

[0032] <Examples of functional structures>

[0033] Figure 2 This is a block diagram illustrating the functional structure of the endoscope operation assistance device 2000 according to Embodiment 1. The endoscope operation assistance device 2000 includes an acquisition unit 2020, a determination unit 2040, and an output unit 2060. The acquisition unit 2020 acquires a shape data sequence 50. The determination unit 2040 uses the shape data sequence 50 to determine an insertion method for inserting the endoscope body 40. The output unit 2060 outputs insertion method information 20 related to the determined insertion method.

[0034] <Examples of hardware architecture>

[0035] The functional components of the endoscope operation assistance device 2000 can be implemented by hardware (e.g., hardwired electronic circuits) or by a combination of hardware and software (e.g., a combination of electronic circuits and a program for controlling those electronic circuits). The following will further explain the implementation of the functional components of the endoscope operation assistance device 2000 by a combination of hardware and software.

[0036] Figure 3 This is a block diagram illustrating the hardware structure of a computer 500 that implements an endoscope operation aid 2000. The computer 500 can be any type of computer. For example, the computer 500 can be a stationary computer such as a personal computer (PC) or a server machine. Alternatively, the computer 500 can be a portable computer such as a smartphone or tablet. The computer 500 can be a dedicated computer designed for implementing the endoscope operation aid 2000, or it can be a general-purpose computer.

[0037] For example, by installing a predetermined application program on the computer 500, the various functions of the endoscope operation assistance device 2000 are implemented on the computer 500. The aforementioned application program includes programs for implementing the functional components of the endoscope operation assistance device 2000.

[0038] Computer 500 includes a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. Bus 502 is a data transmission path for the processor 504, memory 506, storage device 508, input / output interface 510, and network interface 512 to send and receive data with each other. However, the methods for connecting the processor 504 and other components to each other are not limited to bus connections.

[0039] Processor 504 is a processor of various types, such as a central processing unit (CPU), graphics processing unit (GPU), or field-programmable gate array (FPGA). Memory 506 is a main storage device implemented using random access memory (RAM) or similar. Storage device 508 is an auxiliary storage device implemented using hard disks, solid-state drives (SSDs), memory cards, or read-only memory (ROM).

[0040] The input / output interface 510 is an interface for connecting the computer 500 to input / output devices. For example, the endoscope control device 60, which will be described later, is connected to the input / output interface 510. In addition, for example, input devices such as keyboards and output devices such as displays are connected to the input / output interface 510.

[0041] Network interface 512 is an interface used to connect computer 500 to a network. This network can be a local area network (LAN) or a wide area network (WAN).

[0042] Storage device 508 stores programs (programs for implementing the aforementioned application programs) for implementing the various functional components of the endoscope operation assistance device 2000. Processor 504 implements the various functional components of the endoscope operation assistance device 2000 by reading the program into memory 506 and executing it.

[0043] The endoscope operation assistance device 2000 can be implemented by one computer 500 or by multiple computers 500. In the latter case, the structures of each computer 500 do not need to be the same, and they can be different.

[0044] <Examples of the application environment for the Endoscopic Operation Assist Device 2000>

[0045] Figure 4This diagram illustrates a specific example of the environment in which the endoscope operation assistance device 2000 is used. For example, the endoscope operation assistance device 2000 is used together with an endoscope body 40 and an endoscope control device 60. The endoscope body 40 is a scope inserted into the body and is equipped with a camera 10. By viewing the video data generated by the camera 10, the condition inside the body can be visually identified.

[0046] The endoscope control device 60 is a control device used during surgery and examinations using the endoscope body 40. For example, the endoscope control device 60 generates video data to be viewed by performing various processes on video data obtained from a camera installed on the endoscope body 40, and outputs the generated video data. These video data processing operations include, for example, adjusting the color or brightness of the video data, and overlaying various information onto the video data.

[0047] For example, the endoscope operation assist device 2000 acquires various information via the endoscope control device 60. For example, the endoscope operation assist device 2000 acquires still image data or video data (hereinafter collectively referred to as image data) generated by the camera 10 from the endoscope control device 60.

[0048] Note that the data obtained from the endoscope body 40 is not limited to image data generated by the camera 10. For example, the endoscope body 40 may also be provided with an element for obtaining three-dimensional coordinates of each of a plurality of positions of the endoscope body 40. In this case, the endoscope control device 60 uses this element to determine the three-dimensional coordinates of each of the plurality of positions of the endoscope body 40. By using the three-dimensional coordinates of each of the plurality of positions of the endoscope body 40 obtained in this way, the shape of the endoscope body 40 can be determined.

[0049] <Processing flow>

[0050] Figure 5 This is a flowchart illustrating the process performed by the endoscope operation assist device 2000 according to Embodiment 1. The acquisition unit 2020 acquires the shape data sequence 50 (S102). The determination unit 2040 determines the insertion method to be used using the shape data sequence 50 (S104). The determination unit 2040 outputs insertion method information 20 related to the determined insertion method (S106).

[0051] <For shape data sequence 50>

[0052] Shape data sequence 50 is data representing the shape of endoscope body 40 in a time series. For example, for each of a plurality of time moments, shape data sequence 50 represents the shape of endoscope body 40 at that time moment. Figure 6This is a diagram illustrating the structure of shape data sequence 50 in tabular form. Figure 6 The shape data sequence 50 includes time points 51 and shape data 52. Each record in the shape data sequence 50 represents the shape of the endoscope body 40 as indicated by the shape data 52 at the time point indicated by time point 51. Time point 51 can be represented by a time interval or by a method other than a time interval. In the latter case, for example, time point 51 can indicate a relative value assigned chronologically to each shape data 52, such as a sequence number.

[0053] There are multiple ways to represent the shape of the endoscope body 40. For example, the shape of the endoscope body 40 can be represented by a combination of the three-dimensional coordinates of each of a plurality of specific locations of the endoscope body 40. Furthermore, the shape of the endoscope body 40 can also be represented by one of a plurality of predefined shape categories. For example, various categories such as "straight line," "small curve," "large curve," or "abdominal protrusion" can be defined as shape categories. Figure 6 In the figure, shape data 52 indicates the shape category.

[0054] The shape category of the endoscope body 40 can be determined, for example, using the three-dimensional coordinates of each of the multiple specific locations of the endoscope body 40. For instance, a recognition model (hereinafter, a shape recognition model) can be used to determine the shape category. The shape recognition model can be implemented using various models such as neural networks and support vector machines.

[0055] The shape recognition model is pre-trained to output a label representing the shape category of the endoscope body 40 in response to input of the three-dimensional coordinates of each of the multiple specific locations of the endoscope body 40. For each of multiple time points, by inputting the three-dimensional coordinates of each of the multiple specific locations of the endoscope body 40 at that time point into the shape recognition model, the shape category of the endoscope body 40 at that time point can be determined. Note that the shape recognition model can be trained using training data that includes a combination of "the three-dimensional coordinates of each of the multiple specific locations of the endoscope body 40 and the label representing the shape category of the ground truth".

[0056] <Acquisition of shape data sequence 50: S102>

[0057] The acquisition unit 2020 acquires the shape data sequence 50 (S102). The acquisition unit 2020 can acquire the shape data sequence 50 in various ways. For example, the acquisition unit 2020 can acquire the shape data sequence 50 stored in a storage device accessible from the acquisition unit 2020. This storage device can be located inside or outside the endoscope operation aid 2000. Alternatively, the acquisition unit 2020 can also acquire the shape data sequence 50 by receiving it from another device.

[0058] When the shape data sequence 50 represents the shape of the endoscope body 40 by shape category, it is necessary to perform a process to determine the shape category of the endoscope body 40 at each time point. This process can be performed by the endoscope operation aid 2000 or by a device other than the endoscope operation aid 2000. In the former case, the acquisition unit 2020 acquires the shape data sequence 50 generated internally by the endoscope operation aid 2000. For example, in this case, the shape data sequence 50 is stored in a storage device internal to the endoscope operation aid 2000. On the other hand, when the shape data sequence 50 is generated by a device other than the endoscope operation aid 2000, for example, the acquisition unit 2020 acquires the shape data sequence 50 by accessing the device that generated the shape data sequence 50, or receives the shape data sequence 50 sent from that device.

[0059] Here, the insertion method used may change over time. Therefore, for example, the endoscope operation aid 2000 preferably acquires a shape data sequence 50 representing the change of shape data 52 over time in an interval (hereinafter, a unit interval) of predetermined length, and determines the insertion method represented by the shape data sequence 50. In this case, the endoscope operation aid 2000 may acquire a shape data sequence 50 that has been pre-divided into each unit interval, or it may acquire multiple portions of shape data 52 that have not been divided into each unit interval.

[0060] In the latter case, for example, the acquisition unit 2020 obtains a shape data sequence 50 for each unit interval by dividing the acquired shape data 52 into multiple parts according to time sequence for each unit interval. However, adjacent unit intervals may partially overlap. For example, if the length of the unit interval is 10 and the length of the overlapping part is 4, the first shape data sequence 50 contains shape data 52 from the 1st to the 10th, and the second shape data sequence 50 contains shape data 52 from the 7th to the 16th.

[0061] <Determination of the insertion method: S104>

[0062] The determining unit 2040 determines the insertion method to be used using the shape data sequence 50 (S104). For example, for each of the multiple usable insertion methods, the determining unit 2040 calculates the likelihood of using that insertion method using the shape data string 50. Then, based on the likelihood calculated for each insertion method, the determining unit 2040 determines the insertion method to be used from the multiple insertion methods. The method for calculating the likelihood will be described later.

[0063] This method can employ various approaches to determine the insertion method to be used based on the likelihood calculated for each insertion method. For example, the determining unit 2040 determines the insertion method with the highest likelihood as the insertion method to be used. Furthermore, for example, the determining unit 2040 determines multiple candidates for the insertion method to be used based on the likelihood calculated for each insertion method, and determines the insertion method to be used from these multiple candidates. For example, the determining unit 2040 determines the n (n>1) insertion methods with the highest likelihood as candidates for the insertion method to be used. Furthermore, for example, the determining unit 2040 determines insertion methods with a likelihood equal to or greater than a threshold as candidates for the insertion method to be used.

[0064] This method allows for the determination of the insertion method to be used from multiple candidates using various approaches. For example, each insertion method can be pre-assigned a priority. In this case, for example, determining unit 2040 determines the insertion method with the highest priority among the candidate insertion methods as the insertion method to be used. Furthermore, for example, determining unit 2040 corrects the likelihood by multiplying the calculated likelihood by a weight based on the priority of each insertion method among the candidate insertion methods. Note that the priority-based weights are pre-determined to be larger as higher priorities increase. Determining unit 2040 determines the insertion method with the highest corrected likelihood as the insertion method to be used.

[0065] Furthermore, for example, when shape data 52 indicates a shape category, for each of the multiple insertion methods, the important shape categories and the transitions between important shape categories (arrangements of two or more shape categories) are predetermined. In this case, the determining unit 2040 determines the insertion method containing the important shape categories and the transitions between important shape categories in the shape data sequence 50 among the insertion methods determined as candidate insertion methods as the insertion method to be used.

[0066] For example, assuming that insertion methods X and Y are determined as candidates based on likelihood. Furthermore, assuming that the important shape category determined for insertion method X is B, and the important shape category determined for insertion method Y is C. Also, assuming that the time series of shape data 52 indicated by shape data sequence 50 is "A, C, D, C, A". In this case, shape data sequence 50 does not contain the important shape category B in insertion method X, but contains the important shape category C in insertion method Y. Therefore, the determining unit 2040 determines insertion method Y as the insertion method to be used.

[0067] Furthermore, it is assumed that there are multiple insertion methods for important shape categories and transitions between shape categories included in the shape data sequence 50. In this case, for example, the determining unit 2040 determines the insertion method with the highest calculated likelihood among the multiple insertion methods as the insertion method to be used. Alternatively, for example, the determining unit 2040 may also determine the insertion method for which there are more important shape categories and transitions between shape categories included in the shape data sequence 50 as the insertion method to be used.

[0068] The determining unit 2040 may also use information related to the operator of the endoscope body 40 to determine the insertion method to be used from the candidate insertion methods. For example, information that associates the operator's identification information with the insertion methods that the operator can use may be prepared in advance. The determining unit 2040 determines the insertion method to be used from the candidate insertion methods that is limited to insertion methods that can be used by the operator of the endoscope body 40. Note that when there are multiple insertion methods among the candidate insertion methods that are determined to be insertion methods that can be used by the operator of the endoscope body 40, the determining unit 2040 determines the insertion method to be used based on, for example, likelihood, the aforementioned priority level, etc.

[0069] Furthermore, for example, information relating to the operator of the endoscope body 40 can also be used, such as information indicating the operator's skill level (e.g., years or number of experiences in endoscopy, or evaluations from experienced physicians). In this case, for each of the multiple insertion methods, a correlation with the operator's skill level is defined (e.g., "used by someone with 5 or more years of experience," etc.). The determining unit 2040 determines the insertion method to be used from multiple candidates based on the relationship between the operator's skill level and the insertion method.

[0070] The determining unit 2040 can determine the insertion method to be used from candidate insertion methods by utilizing image data obtained from the camera 10 disposed on the endoscope body 40. For example, the determining unit 2040 extracts features from the image data obtained from the camera 10 and determines the insertion method that matches the features from the candidate insertion methods as the insertion method to be used. Features extracted from the image data include, for example, the phenomenon that the entire image turns red due to pressing the tip of the endoscope body 40 against the wall of the viscera (called a red ball), characteristic actions of the screen or the operation of the endoscope body 40, etc.

[0071] The determining unit 2040 can calculate the likelihood of using each insertion method in each of the multiple methods (e.g., the method using the recognition model and the method using the state transition model, described later), and determine the insertion method to be used based on the calculation results. For example, the determining unit 2040 calculates a statistical value (e.g., average, maximum) of the likelihood calculated by each of the multiple methods for each insertion method, and determines the insertion method with the largest statistical value as the insertion method to be used.

[0072] Furthermore, for example, the determining unit 2040 can determine the insertion method used for each method of calculating likelihood. For example, suppose the insertion method determined using the identification model is insertion method X, and the insertion method determined using the state transition model is insertion method Y. In this case, the determining unit 2040 determines insertion method X and insertion method Y as the insertion methods used. In this case, it is preferable that the insertion method information 20 indicates the method used to determine the insertion method (the method of calculating likelihood) and the insertion method determined by that method in a mutually related manner. For example, in the aforementioned example, by describing "identification model: insertion method X, state transition model: insertion method Y," etc., the correlation between the method of calculating likelihood and the determined insertion method can be indicated.

[0073] <Specific methods for calculating likelihood>

[0074] As described above, for example, determining unit 2040 calculates the likelihood of using each of the multiple insertion methods to determine the insertion method to be used. Hereinafter, two methods are exemplified as specific examples of how to calculate this likelihood.

[0075] <<Methods Using Recognition Models>>

[0076] In this method, a recognition model is used, which is trained to output the likelihood of using each insertion method in response to the input shape data sequence 50. Hereinafter, this recognition model is referred to as the insertion method recognition model. For example, the insertion method recognition model is implemented using a recurrent neural network (RNN), which is a neural network that processes time series data. However, the type of model used to implement the insertion method recognition model is not limited to RNNs, and any type of model capable of processing time series data can be used.

[0077] The insertion method identification model is trained using training data that includes a combination of "time-series data representing the shape of the endoscope body and forward output data". The forward output data is, for example, data that shows the highest likelihood (e.g., 1) for the insertion method used and the lowest likelihood (e.g., 0) for other insertion methods.

[0078] Note that data other than the shape data sequence 50 can also be input into the insertion method recognition model. Examples of data other than the shape data sequence 50 include still image data or video data obtained from the camera 10, or data representing the operator's movements of the endoscope body 40 (e.g., time-series data of acceleration obtained from an accelerometer worn on the operator's hand). Furthermore, if the shape data sequence 50 represents the shape of the endoscope body 40 by shape category, the three-dimensional coordinates of each of the multiple locations of the endoscope body 40 can also be input into the insertion method recognition model. When this data is used, data of the same type is also included in the training data of the insertion method recognition model. In this way, the insertion method recognition model is trained to calculate likelihood using data other than the shape data sequence 50.

[0079] Note that the insertion method identification model can output a label representing the insertion method used, instead of outputting the likelihood for each insertion method. In this case, for example, the insertion method identification model outputs a label representing the insertion method used, where the maximum likelihood for that insertion method is calculated as the label of the insertion method used. The determination unit 2040 can determine the insertion method used based on the label output from the insertion method identification model.

[0080] <<Methods Using State Transition Models>>

[0081] In this case, for each of the multiple insertion methods, a state transition model representing the pattern of the shape of the endoscope body 40 changing over time when that insertion method is used is predetermined and stored in a storage device accessible from the determination unit 2040. When the method is used, the likelihood of using each insertion method represents, for example, the degree of consistency between the state transition model of that insertion method and the change of the endoscope body 40 over time represented by the shape data sequence 50. That is, for each of the multiple insertion methods, the determination unit 2040 calculates the degree of consistency between the state transition model of that insertion method and the shape data sequence 50, and determines the insertion method to be used based on the calculation result. For example, the insertion method with the highest degree of consistency is determined as the insertion method to be used.

[0082] When shape data 52 represents a shape category, the likelihood is determined, for example, by utilizing the degree of consistency between the state transition model and the shape data sequence 50 as follows.

[0083] Formula 1

[0084]

[0085] In equation (1), the numerator on the right represents the number of shape categories that match between the shape data sequence 50 and the state transition model. Meanwhile, the unit interval length in the denominator represents the total number of shape categories contained in the aforementioned unit interval.

[0086] Figure 7 This is a diagram conceptually illustrating the method for calculating likelihood using Equation (1). In this example, the state transition model of insertion method X is compared with the shape data sequence 50. In this example, the unit interval length is 12.

[0087] The state transition model of insertion method X includes four shape categories: A, C, E, and G. Shape data sequence 50 includes two shape categories A, two shape categories C, and two shape categories E from the shape categories constituting the state transition model of insertion method X. Therefore, the number of shape categories matching between the state transition model of insertion method X and shape data sequence 50 is 6. Thus, according to equation (1), 0.5 (6 / 12) is calculated as the likelihood of using insertion method X.

[0088] The formula for calculating likelihood is not limited to formula (1). For example, likelihood can also be calculated using the following formula (2).

[0089] Formula 2

[0090]

[0091] For example Figure 7In the example, five shape categories A, B, C, D, and E are included in the shape data sequence 50 within the unit interval. Meanwhile, three shape categories A, C, and E are consistent with the state transition model of insertion method X and the shape data sequence 50. Therefore, according to equation (2), 0.6 (3 / 5) is calculated as the likelihood of the insertion method X used.

[0092] Furthermore, when calculating the likelihood for each insertion method, it is necessary not only to consider the degree of matching of the shape categories contained in the unit interval, but also to calculate the likelihood by adding formulas that increase the likelihood in the case of transition paths (sequence relationships) indicated by the state transition model or decrease the likelihood in the case of non-transition paths, taking into account the order of transitions.

[0093] <Output of Insertion Method Information 20: S108>

[0094] Output unit 2060 outputs insertion method information 20. Various methods exist for outputting insertion method information 20. For example, output unit 2060 displays a screen representing insertion method information 20 on a display device that can be viewed by the operator of the endoscope body 40. Furthermore, for example, output unit 2060 can store a file representing insertion method information 20 to a storage device or send it to any other device.

[0095] It can generate various kinds of information as insertion method information 20. For example, insertion method information 20 is information indicating the name of the insertion method used. In addition, for example, insertion method information 20 can indicate the change of the insertion method used (the change of the insertion method used over time).

[0096] Figure 8 This is a diagram illustrating the insertion method information 20. Figure 8 The insertion method information 20 shows two graphs. The solid line represents the change in the insertion method used for inserting the endoscope body 40 over time, which is determined by the endoscope operation aid 2000. As mentioned above, the insertion method used for inserting the endoscope body 40 may change over time. Therefore, in Figure 8 In the example shown, the endoscopic manipulation aid 2000 determines the insertion method to be used within each unit interval. The determined insertion method is then plotted in association with a time point corresponding to that unit interval (e.g., the start time point of that unit interval). Figure 8 In the example, the insertion methods used change in the order of insertion method Z, insertion method X, insertion method Y, and insertion method Z.

[0097] A single-dotted line represents the likelihood of the determined insertion method. For example, in the interval where insertion method Z is determined as the insertion method used, the likelihood of using insertion method Z is represented. Similarly, in the interval where insertion method X is determined as the insertion method used, the likelihood of using insertion method X is represented.

[0098] The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the structure and details of this application within the scope of the present invention.

[0099] Furthermore, in the examples above, the program can be stored and provided to the computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, or hard disk drives), optical-magnetic recording media (e.g., optical discs), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAM). In addition, the program can also be provided to the computer via various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can provide the program to the computer via wireless communication paths or wired communication paths such as wires and optical fibers.

[0100] Some or all of the above embodiments may also be described as follows, but not limited to the following:

[0101] (Postscript 1)

[0102] An endoscope operation aid device, comprising:

[0103] The acquisition unit is configured to acquire a sequence of shape data representing the change of shape data of the endoscope body over time.

[0104] The determining unit is configured to determine, based on the shape data sequence, an insertion method for inserting the endoscope body; and

[0105] The output unit is configured to output insertion method information related to the determined insertion method.

[0106] (Postscript 2)

[0107] According to the endoscope operation aid device described in Appendix 1, for each of the multiple insertion methods of the endoscope body, the determining unit calculates the likelihood of the insertion method being used based on the shape data sequence, and determines the insertion method to be used based on the calculated likelihood.

[0108] (Note 3)

[0109] According to the endoscope operation assistance device described in Appendix 2, for each of the multiple insertion methods, the determining unit acquires a state transition model representing the pattern of the shape of the endoscope body changing over time when the insertion method is used, and calculates the likelihood that the insertion method is used based on the degree of matching between the state transition model and the shape data sequence.

[0110] (Postscript 4)

[0111] According to the endoscopic operation aid device described in Appendix 2, the determining unit determines the likelihood of each insertion method being used by inputting the shape data sequence into a trained recognition model, the trained recognition model being configured to output the likelihood of each of a plurality of insertion methods being used in response to input of a data sequence representing the change of shape data of the endoscope body over time.

[0112] (Note 5)

[0113] According to any one of Appendices 1 to 4, the endoscope operation aid device, wherein the determining unit performs:

[0114] Based on the likelihood of each insertion method being used, multiple candidates are determined for each insertion method used; and

[0115] Based on the priority determined for each insertion method, the insertion method to be used is determined from the multiple candidates of insertion methods.

[0116] (Note 6)

[0117] According to any one of Appendices 1 to 4, the endoscope operation aid device, wherein the determining unit performs:

[0118] Based on the likelihood of each insertion method being used, multiple candidates are determined for each insertion method used; and

[0119] Using information related to the operator of the endoscope, the insertion method to be used is determined from the plurality of candidates for insertion methods.

[0120] (Note 7)

[0121] According to the endoscopic operation aid device described in Appendix 6, information related to the operator of the endoscope body indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope body.

[0122] (Postscript 8)

[0123] According to the endoscopic operation aid device described in Appendix 1, the determining unit determines the insertion method to be used by inputting a shape data sequence into a trained recognition model, the trained recognition model being configured to output data representing the insertion method to be used in response to input of a data sequence representing the change of shape data of the endoscope body over time.

[0124] (Note 9)

[0125] According to any one of Appendices 1 to 8, the endoscopic operation aid device, wherein the output unit outputs insertion method information including the change of the insertion method used over time.

[0126] (Postscript 10)

[0127] Endoscopic operation aid according to any one of Appendices 1 to 9

[0128] Multiple shape categories are defined, which are the types of shapes used for the endoscope body, and

[0129] The shape data contained in this shape data sequence indicates one of several shape categories of the endoscope body.

[0130] (Postscript 11)

[0131] A control method executed by a computer, the control method comprising:

[0132] The acquisition step involves acquiring a sequence of shape data representing the change in shape data of the endoscope body over time.

[0133] The steps involve determining, based on the shape data sequence, an insertion method for inserting the endoscope body; and

[0134] Output steps: Output insertion method information related to the determined insertion method.

[0135] (Postscript 12)

[0136] According to the control method described in Appendix 11, in the determination step, for each of the multiple insertion methods of the endoscope body, the likelihood of the insertion method being used is calculated using the shape data sequence, and the insertion method to be used is determined based on the calculated likelihood.

[0137] (Postscript 13)

[0138] According to the control method described in Appendix 12, in the determination step, for each of the plurality of insertion methods, a state transition model representing the pattern of the shape of the endoscope body changing over time when the insertion method is used is obtained, and the likelihood of the insertion method being used is calculated based on the degree of matching between the state transition model and the shape data sequence.

[0139] (Postscript 14)

[0140] According to the control method described in Appendix 12, in the determination step, the likelihood of each insertion method being used is determined by inputting a sequence of shape data into a trained recognition model, the trained recognition model being configured to output the likelihood of each of a plurality of insertion methods being used in response to an input representing the change of shape data of the endoscope body over time.

[0141] (Postscript 15)

[0142] According to any one of Appendices 11 to 14, in the determining step,

[0143] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0144] Based on the priority defined for each insertion method, the insertion method to be used is determined from multiple candidates of insertion methods.

[0145] (Postscript 16)

[0146] According to any one of Appendices 11 to 14, in the determining step,

[0147] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0148] Using information related to the operator of the endoscope, the insertion method to be used is determined from multiple candidates of insertion methods.

[0149] (Postscript 17)

[0150] According to the control method described in Appendix 16, information related to the operator of the endoscope body indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope body.

[0151] (Postscript 18)

[0152] According to the control method described in Appendix 11, in the determination step, the insertion method to be used is determined by inputting a sequence of shape data into a trained recognition model, the trained recognition model being configured to output data representing the insertion method to be used in response to an input of a data sequence representing the change of shape data of the endoscope body over time.

[0153] (Postscript 19)

[0154] According to any one of Appendix 11 to 18, in the control method, during the output step, insertion method information, including the time-varying insertion method used, is output.

[0155] (Postscript 20)

[0156] According to any one of the control methods in Appendix 11 to 19,

[0157] Several shape categories are defined, which are the types of shapes used for endoscope bodies, and

[0158] The shape data sequence contains shape data that indicates one of several shape categories of the endoscope body.

[0159] (Postscript 21)

[0160] A computer-readable medium storing a program that causes a computer to execute:

[0161] The acquisition step involves acquiring a sequence of shape data representing the shape data of the endoscope body as a function of time.

[0162] The steps involve determining, based on the shape data sequence, an insertion method for inserting the endoscope body; and

[0163] Output steps: Output insertion method information related to the determined insertion method.

[0164] (Postscript 22)

[0165] According to the computer-readable medium described in Appendix 21, in the determining step, for each of a plurality of insertion methods of the endoscope body, the likelihood of the insertion method being used is calculated based on the shape data sequence, and the insertion method to be used is determined based on the calculated likelihood.

[0166] (Postscript 23)

[0167] According to the computer-readable medium described in Appendix 22, in the determining step, for each of a plurality of insertion methods, a state transition model representing a pattern of change of the shape of the endoscope body over time when the insertion method is used is obtained, and the likelihood of the insertion method being used is calculated based on the degree of matching between the state transition model and the shape data sequence.

[0168] (Postscript 24)

[0169] According to the computer-readable medium described in Appendix 22, in the determining step, the likelihood of each insertion method being used is determined by inputting a sequence of shape data into a trained recognition model, the trained recognition model being configured to output the likelihood of each of a plurality of insertion methods being used in response to an input of a data sequence representing a time-varying change in the shape data of the endoscope body.

[0170] (Postscript 25)

[0171] According to any one of Appendices 21 to 24, in the determining step,

[0172] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0173] Based on the priority defined for each insertion method, the insertion method to be used is determined from the multiple candidates of insertion methods.

[0174] (Postscript 26)

[0175] According to any one of Appendices 21 to 24, in the determining step,

[0176] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0177] Using information related to the operator of the endoscope, the insertion method to be used is determined from the plurality of candidates for insertion methods.

[0178] (Postscript 27)

[0179] According to the computer-readable medium described in Appendix 26, information relating to the operator of the endoscope body indicates one or more insertion methods used by the operator, or indicates the operator's skill level in relation to the operation of the endoscope body.

[0180] (Postscript 28)

[0181] According to the computer-readable medium described in Appendix 21, in the determining step, the insertion method to be used is determined by inputting a sequence of shape data into a trained recognition model, the trained recognition model being configured to output data representing the insertion method to be used in response to input of a data sequence representing a time-varying change in the shape data of the endoscope body.

[0182] (Postscript 29)

[0183] According to any one of Appendices 21 to 28, in the output step, insertion method information, including the time-varying insertion method used, is output.

[0184] (Postscript 30)

[0185] According to any one of Appendices 21 to 29, a computer-readable medium

[0186] Several shape categories are defined, which are the types of shapes used for endoscope bodies, and

[0187] The shape data contained in this shape data sequence indicates one of several shape categories of the endoscope body.

[0188] (Postscript 31)

[0189] A program that causes a computer to execute:

[0190] The acquisition step involves acquiring a sequence of shape data representing the shape data of the endoscope body as a function of time.

[0191] The steps involve determining, based on the shape data sequence, an insertion method for inserting the endoscope body; and

[0192] Output steps: Output insertion method information related to the determined insertion method.

[0193] (Postscript 32)

[0194] According to the procedure described in Appendix 31, in the determining step, for each of the multiple insertion methods of the endoscope body, the likelihood of the insertion method being used is calculated based on the shape data sequence, and the insertion method to be used is determined based on the calculated likelihood.

[0195] (Postscript 33)

[0196] According to the procedure described in Appendix 32, in the determining step, for each of the plurality of insertion methods, a state transition model representing the pattern of the shape of the endoscope body changing over time when the insertion method is used is obtained, and the likelihood of the insertion method being used is calculated based on the degree of matching between the state transition model and the shape data sequence.

[0197] (Postscript 34)

[0198] According to the procedure described in Appendix 32, in the determining step, the likelihood of each insertion method being used is determined by inputting a sequence of shape data into a trained recognition model, which is configured to output the likelihood of each of a plurality of insertion methods being used in response to an input of a data sequence representing a change over time in the shape data of the endoscope body.

[0199] (Postscript 35)

[0200] According to any one of Appendices 31 to 34, in the determining step,

[0201] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0202] Based on the priority defined for each insertion method, the insertion method to be used is determined from the multiple candidates of insertion methods.

[0203] (Postscript 36)

[0204] According to any one of Appendices 31 to 34, in the determining step,

[0205] Based on the likelihood of each insertion method being used, multiple candidates are determined for the insertion method used.

[0206] Using information related to the operator of the endoscope, the insertion method to be used is determined from the plurality of candidates for insertion methods.

[0207] (Postscript 37)

[0208] According to the procedure described in Appendix 36, information related to the operator of the endoscope body indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope body.

[0209] (Postscript 38)

[0210] According to the procedure described in Appendix 31, in the determination step described above, the insertion method to be used is determined by inputting a sequence of shape data into a trained recognition model, which is configured to output data representing the insertion method to be used in response to an input of a data sequence representing the time-varying shape data of the endoscope body.

[0211] (Postscript 39)

[0212] According to any one of Appendices 31 to 38, in the output step, insertion method information, including the time-varying insertion method used, is output.

[0213] (Postscript 40)

[0214] According to any one of Appendix 31 to 39, the procedure

[0215] Several shape categories are defined, which are the types of shapes used for endoscope bodies, and

[0216] The shape data contained in this shape data sequence indicates one of several shape categories of the endoscope body.

[0217] Explanation of reference numerals in the attached figures

[0218] 10 cameras

[0219] 20. Insertion Method Information

[0220] 40 Endoscope body

[0221] 50 Shape Data Sequences

[0222] 51 Time Points

[0223] 52 Shape Data

[0224] 60 Endoscope control device

[0225] 500 computers

[0226] 500 computers

[0227] 502 bus

[0228] 504 processor

[0229] 506 memory

[0230] 508 storage device

[0231] 510 Input / Output Interface

[0232] 512 Network Interface

[0233] 2000 Endoscopic Operation Assist Device

[0234] 2020 Acquisition Unit

[0235] 2040 Determining Unit

[0236] 2060 Output Unit

Claims

1. An endoscopic operation auxiliary device, comprising: The acquisition unit is configured to acquire a sequence of shape data representing the shape data of the endoscope body as it changes over time. The determining unit is configured to determine an insertion method for inserting the endoscope body based on the shape data sequence; as well as The output unit is configured to output insertion method information related to the determined insertion method. The determining unit determines the insertion method for inserting the endoscope body based on the likelihood of each insertion method used, and The output unit therein displays the insertion method and the likelihood over time.

2. The endoscope operation auxiliary device according to claim 1, The output unit displays the insertion method in a first display mode, and The output unit therein displays the likelihood in a second display mode.

3. The endoscope operation auxiliary device according to claim 1, The determining unit determines the insertion method for each unit interval. The output unit uses a chart as the first display method to display the insertion method associated with the corresponding unit interval, and The output unit uses another chart as the second display method to display the likelihood associated with the corresponding insertion method, wherein the display method of the other chart is different from the first display method.

4. The endoscope operation auxiliary device according to claim 1, The determining unit determines a plurality of the insertion methods based on the likelihood, and The transformation of the plurality of insertion methods is associated with the likelihood.

5. The endoscope operation auxiliary device according to claim 1, The determining unit determines the insertion method to be used from a pool of candidates of insertion methods based on a priority defined for each insertion method.

6. The endoscope operation auxiliary device according to claim 1, The shape data indicates the shape category, and The determining unit determines the insertion method based on the transformation of the shape category.

7. The endoscope operation auxiliary device according to claim 1, The determining unit determines the insertion method used by calculating the likelihood of the insertion method using a first model and by calculating the likelihood of the insertion method using a second model.

8. The endoscope operation auxiliary device according to claim 1, The determining unit performs the following: For each method of calculating likelihood, determine the insertion method used; and The determined insertion method is associated with the corresponding likelihood.

9. The endoscope operation auxiliary device according to claim 1, The output unit performs the following: Displays the time elapsed since the endoscope body was inserted into the vertical axis; and This shows the insertion method and the likelihood in the horizontal axis.

10. A control method executed by a computer, the control method comprising: The acquisition step involves acquiring a sequence of shape data representing the shape data of the endoscope body as a function of time. The steps involve determining an insertion method for inserting the endoscope body based on the shape data sequence. as well as The output step outputs insertion method information related to the determined insertion method. The determination step includes determining the insertion method for inserting the endoscope body based on the likelihood of each insertion method used, and The output step includes displaying the insertion method and temporary changes in the likelihood.

Images