Endoscope operation assisting device, control method, computer readable medium and program

Abstract

An endoscope operation assisting device (2000) acquires a shape data sequence (50) representing time-varying shape data of an endoscope body (40). The endoscope operation assisting device (2000) uses the shape data sequence (50) to determine an insertion method for inserting the endoscope body (40). The endoscope operation assisting device (2000) outputs insertion method information (20) related to the determined insertion method.

Description

Background Art

[0001] The present disclosure relates to methods for assisting the operation of an endoscope.

[0002] Devices that assist with surgeries and examinations using endoscopes are being developed. For example, Patent Document 1 discloses a technology that infers the current status related to endoscope operation and outputs auxiliary information corresponding to the status. Examples of the current status related to endoscope operation include "insertion in progress without major problems" and "excessive proximity persists." Furthermore, the auxiliary information indicates the endoscope operation that is effective in resolving the current problem.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2018 / 235185 Summary of the Invention

[0006] The information used to assist endoscopic operation is not limited to the information described above. The present disclosure aims to provide a technique for presenting new information to assist endoscopic operation.

[0007] According to one aspect of the present disclosure, an endoscope operation assistance device is provided, comprising: an acquisition unit configured to acquire a shape data sequence representing changes in shape data of an endoscope scope over time; a determination unit configured to determine an insertion method for inserting the endoscope scope based on the shape data sequence; and an output unit configured to output insertion method information related to the determined insertion method.

[0008] According to another aspect of the present disclosure, a computer-implemented control method is provided. The control method includes: an acquisition step of acquiring a shape data sequence representing changes in the shape data of an endoscope scope over time; a determination step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; and an output step of outputting insertion method information related to the determined insertion method.

[0009] According to another aspect of the present disclosure, a computer-readable medium is provided, which stores a program for causing a computer to execute the control method of the present invention.

[0010] Beneficial technical effects

[0011] According to the present disclosure, a technique is provided to present new information to assist in endoscopic procedures. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1This is a diagram illustrating an outline of the operation of the endoscope operation supporting device according to the first embodiment.

[0013] Figure 2 This is a block diagram illustrating the functional configuration of the endoscope operation support device according to the first embodiment.

[0014] Figure 3 This is a block diagram illustrating the hardware configuration of a computer that realizes the endoscope operation support device.

[0015] Figure 4 It is a diagram showing a specific example of the usage environment of the endoscope operation supporting device.

[0016] Figure 5 This is a flowchart illustrating the flow of processing executed by the endoscope operation supporting device according to the first embodiment.

[0017] Figure 6 This figure illustrates the structure of a shape data sequence in a tabular format.

[0018] Figure 7 This is a diagram conceptually illustrating a method for calculating likelihood using equation (1).

[0019] Figure 8 This is a diagram illustrating insertion method information. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present disclosure in detail with reference to the accompanying drawings. In the drawings, identical or corresponding elements are denoted by the same reference numerals, and for the sake of clarity, repeated descriptions are omitted as necessary. Unless otherwise specified, predetermined values ​​such as predetermined values ​​and thresholds are stored in advance in a storage device accessible from a device that utilizes the values.

[0021] Figure 1 1 is a diagram illustrating an outline of the operation of the endoscope operation supporting device 2000 according to the first embodiment. Figure 1 This is a diagram for easily understanding the outline of the endoscope operation assisting device 2000. The operation of the endoscope operation assisting device 2000 is not limited to Figure 1 shown.

[0022] There are various operating methods for inserting the endoscope body into the body (hereinafter referred to as insertion methods or surgical modes). Examples include the shaft-retention shortening method, the push method, and the α-loop formation method. The endoscope operation support device 2000 determines the insertion method for inserting the endoscope body 40 and outputs information related to the determined insertion method, i.e., insertion method information 20.

[0023] Here, the insertion of the endoscope scope 40 is performed while the shape of the endoscope scope 40 is variously changed. Furthermore, how the shape of the endoscope scope 40 is changed varies depending on the insertion method. Therefore, the insertion method to be used can be determined based on the change in the shape of the endoscope scope 40 over time.

[0024] Here, the endoscope operation assistance device 2000 acquires a shape data sequence 50 representing the shape of the endoscope scope 40 in a time series. Next, the endoscope operation assistance device 2000 uses the shape data sequence 50 to determine the insertion method to be used for inserting the endoscope scope 40. Furthermore, the endoscope operation assistance device 2000 generates and outputs insertion method information 20 related to the determined insertion method. For example, the insertion method information 20 indicates the name of the determined insertion method.

[0025] <Examples of Effects>

[0026] As described above, the endoscope operation support device 2000 of this embodiment determines the insertion method for inserting the endoscope scope 40 using the shape data sequence 50 representing the temporal change in the shape of the endoscope scope 40, and outputs the insertion method information 20 related to the insertion method. By using the insertion method information 20, the operator of the endoscope scope 40 and the like can understand the insertion method for inserting the endoscope scope 40.

[0027] For example, the actual movement of the endoscope scope 40 may sometimes differ from the movement intended by the operator. Consequently, even if the operator intends to perform a certain insertion method, that method may not actually be carried out. Using the endoscope operation assisting device 2000 of this embodiment, the operator of the endoscope scope 40 can easily determine whether the intended insertion method can be carried out. If the intended insertion method has been carried out, the operator can confidently continue operating the endoscope scope 40 after confirming that the intended insertion method has been carried out. Furthermore, if the intended insertion method has not been carried out, the operator can correct the operation to carry out the intended insertion method after confirming that the intended insertion method has not been carried out.

[0028] Furthermore, by storing the insertion method information 20 in a storage device, a record of the insertion method used during operation of the endoscope scope 40 can be retained. Retaining such a record can be useful in future endoscopic surgeries, endoscopic examinations, and other operations. For example, by retaining a record of the insertion method used during endoscopic surgery performed by an experienced physician, other physicians can share the experienced physician's skills. Furthermore, by retaining a record of the insertion method used, for example, in association with a case history, this record can be utilized in explanations to patients. Furthermore, the endoscope operation assistance device 2000 can also be configured to provide operator guidance based on the determined insertion method information. This allows for guidance tailored to the insertion method the operator desires.

[0029] Hereinafter, the endoscope operation supporting device 2000 according to this embodiment will be described in more detail.

[0030] <Example of functional structure>

[0031] Figure 2 This is a block diagram illustrating the functional configuration of an endoscope operation assistance device 2000 according to Embodiment 1. The endoscope operation assistance device 2000 includes an acquisition unit 2020, a determination unit 2040, and an output unit 2060. The acquisition unit 2020 acquires a shape data sequence 50. The determination unit 2040 uses the shape data sequence 50 to determine an insertion method for inserting the endoscope scope 40. The output unit 2060 outputs insertion method information 20 related to the determined insertion method.

[0032] <Hardware Configuration Example>

[0033] Each functional component of the endoscope operation assisting device 2000 may be implemented by hardware that implements each functional component (e.g., a hard-wired electronic circuit, etc.), or may be implemented by a combination of hardware and software (e.g., a combination of an electronic circuit and a program for controlling the electronic circuit, etc.). The following further describes the case where each functional component of the endoscope operation assisting device 2000 is implemented by a combination of hardware and software.

[0034] Figure 3 This is a block diagram illustrating the hardware configuration of a computer 500 that implements the endoscope operation assisting device 2000. The computer 500 is any computer. For example, the computer 500 is a stationary computer such as a personal computer (PC) or a server. Furthermore, the computer 500 is a portable computer such as a smartphone or a tablet computer. The computer 500 can be a dedicated computer designed to implement the endoscope operation assisting device 2000 or a general-purpose computer.

[0035] For example, each function of the endoscope operation assisting device 2000 is implemented in the computer 500 by installing a predetermined application program in the computer 500. The application program includes a program for implementing the functional components of the endoscope operation assisting device 2000.

[0036] Computer 500 includes a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. Bus 502 is a data transmission path for transmitting and receiving data between processor 504, memory 506, storage device 508, input / output interface 510, and network interface 512. However, the method for connecting processor 504 and the like is not limited to bus connection.

[0037] Processor 504 is a variety of processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field programmable gate array (FPGA). Memory 506 is a primary storage device implemented using, for example, random access memory (RAM). Storage device 508 is an auxiliary storage device implemented using, for example, a hard disk, a solid-state drive (SSD), a memory card, or a read-only memory (ROM).

[0038] The input / output interface 510 is an interface for connecting the computer 500 and input / output devices. For example, the endoscope control device 60 described below is connected to the input / output interface 510. In addition, input devices such as a keyboard and output devices such as a display device are connected to the input / output interface 510.

[0039] The network interface 512 is an interface for connecting the computer 500 to a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0040] The storage device 508 stores a program (a program for realizing the aforementioned application program) for realizing each functional component of the endoscope operation supporting apparatus 2000. The processor 504 realizes each functional component of the endoscope operation supporting apparatus 2000 by reading the program into the memory 506 and executing the program.

[0041] The endoscope operation supporting device 2000 may be implemented by a single computer 500 or by a plurality of computers 500. In the latter case, the configurations of the computers 500 do not need to be the same and may be different.

[0042] <Example of Usage Environment of Endoscope Operation Support Device 2000>

[0043] Figure 4This figure shows a specific example of the usage environment of the endoscope operation assistance device 2000. For example, the endoscope operation assistance device 2000 is used together with the endoscope scope 40 and the endoscope control device 60. The endoscope scope 40 is inserted into the body and is equipped with a camera 10. By viewing the video data generated by the camera 10, the internal body conditions can be visually recognized.

[0044] The endoscope control device 60 is a control device used when performing surgery or examination using the endoscope body 40. For example, the endoscope control device 60 generates video data to be viewed by performing various processing on video data obtained from a camera installed in the endoscope body 40, and outputs the generated video data. Examples of video data processing include adjusting the color or brightness of the video data and superimposing various information on the video data.

[0045] For example, the endoscope operation assisting device 2000 acquires various information via the endoscope control device 60. For example, the endoscope operation assisting device 2000 acquires still image data or video data (hereinafter collectively referred to as image data) generated by the camera 10 from the endoscope control device 60.

[0046] Note that the data obtained from the endoscope scope 40 is not limited to the image data generated by the camera 10. For example, the endoscope scope 40 may be provided with a component for obtaining three-dimensional coordinates for each of a plurality of positions on the endoscope scope 40. In this case, the endoscope control device 60 uses this component to determine the three-dimensional coordinates for each of the plurality of positions on the endoscope scope 40. By using the three-dimensional coordinates for each of the plurality of positions on the endoscope scope 40 obtained in this manner, the shape of the endoscope scope 40 can be grasped.

[0047] <Processing Flow>

[0048] Figure 5 This is a flowchart illustrating the flow of processing performed by the endoscope operation support device 2000 according to Embodiment 1. The acquisition unit 2020 acquires the shape data sequence 50 (S102). The determination unit 2040 determines the insertion method to be used using the shape data sequence 50 (S104). The determination unit 2040 outputs the insertion method information 20 related to the determined insertion method (S106).

[0049] <For shape data sequence 50>

[0050] The shape data sequence 50 is data that represents the shape of the endoscope scope 40 in time series. For example, for each of a plurality of time points, the shape data sequence 50 represents the shape of the endoscope scope 40 at that time point. Figure 6: is a diagram illustrating the structure of the shape data sequence 50 in a table format. Figure 6 The shape data sequence 50 includes a time point 51 and shape data 52. Each record in the shape data sequence 50 indicates that the shape of the endoscope scope 40 at the time point indicated by the time point 51 is the shape indicated by the shape data 52. The time point 51 can be represented by a time or by other means. In the latter case, for example, the time point 51 can indicate a relative numerical value, such as a serial number, assigned to each shape data 52 in chronological order.

[0051] There are various ways to represent the shape of the endoscope scope 40. For example, the shape of the endoscope scope 40 can be represented by a combination of three-dimensional coordinates of each of a plurality of specific positions of the endoscope scope 40. Furthermore, for example, the shape of the endoscope scope 40 can be represented by one of a plurality of predetermined shape categories. For example, various shape categories such as "straight line," "small curve," "large curve," and "abdominal protrusion" can be defined. Figure 6 , the shape data 52 indicates the shape category.

[0052] The shape category to which the shape of the endoscope scope 40 belongs can be determined, for example, using the three-dimensional coordinates of each of the aforementioned multiple specific positions of the endoscope scope 40. For example, the shape category can be determined using a recognition model (hereinafter referred to as a shape recognition model). The shape recognition model can be implemented using various models such as neural networks and support vector machines.

[0053] The shape recognition model is pre-trained to output a label representing the shape category of the endoscope scope 40 in response to input of the three-dimensional coordinates of each of a plurality of specific positions of the endoscope scope 40. By inputting the three-dimensional coordinates of each of the plurality of specific positions of the endoscope scope 40 at each of a plurality of time points into the shape recognition model, the shape category of the endoscope scope 40 at that time point can be determined. Note that the shape recognition model can be trained using training data that includes a combination of the three-dimensional coordinates of each of the plurality of specific positions of the endoscope scope 40 and labels representing the ground-truth shape category.

[0054] <Acquisition of Shape Data Sequence 50: S102>

[0055] The acquisition unit 2020 acquires the shape data sequence 50 (S102). The acquisition unit 2020 can acquire the shape data sequence 50 in various ways. For example, the acquisition unit 2020 can access a storage device accessible from the acquisition unit 2020 and acquire the shape data sequence 50 stored in the storage device. The storage device can be located inside or outside the endoscope operation assisting device 2000. Furthermore, for example, the acquisition unit 2020 can also acquire the shape data sequence 50 by receiving the shape data sequence 50 transmitted from another device.

[0056] When the shape data sequence 50 represents the shape of the endoscope scope 40 using a shape category, it is necessary to perform processing to determine the shape category of the endoscope scope 40 at each point in time. This processing can be performed by the endoscope operation assisting device 2000 or by a device external to the endoscope operation assisting device 2000. In the former case, the acquisition unit 2020 acquires the shape data sequence 50 generated within the endoscope operation assisting device 2000. For example, in this case, the shape data sequence 50 is stored in a storage device within the endoscope operation assisting device 2000. On the other hand, when the shape data sequence 50 is generated by a device external to the endoscope operation assisting device 2000, the acquisition unit 2020 acquires the shape data sequence 50 by, for example, accessing the device that generated the shape data sequence 50 or receiving the shape data sequence 50 transmitted from the device.

[0057] Here, the insertion method used may change over time. Therefore, for example, it is preferable that the endoscope operation assistance device 2000 acquires a shape data sequence 50 representing the temporal changes in the shape data 52 within a predetermined length interval (hereinafter, a unit interval) and determines the insertion method represented by this shape data sequence 50. In this case, the endoscope operation assistance device 2000 may acquire a shape data sequence 50 that has been pre-divided into each unit interval, or may acquire multiple portions of the shape data 52 that have not been divided into each unit interval.

[0058] In the latter case, for example, the acquisition unit 2020 divides the acquired shape data 52 into multiple sections per unit interval in chronological order, thereby obtaining a shape data sequence 50 for each unit interval. However, adjacent unit intervals may partially overlap. For example, if the length of the unit interval is 10 and the length of the overlapping portion is 4, the first shape data sequence 50 includes shape data 52 from the first to the tenth element, and the second shape data sequence 50 includes shape data 52 from the seventh to the sixteenth element.

[0059] <Determination of Insertion Method: S104>

[0060] The determination unit 2040 uses the shape data sequence 50 to determine the insertion method to be used (S104). For example, for each of a plurality of possible insertion methods, the determination unit 2040 uses the shape data sequence 50 to calculate the likelihood of using that insertion method. The determination unit 2040 then determines the insertion method to be used from the plurality of insertion methods based on the likelihood calculated for each insertion method. The method for calculating the likelihood will be described later.

[0061] Various methods can be used to determine the insertion method to be used based on the likelihood calculated for each insertion method. For example, the determination unit 2040 determines the insertion method with the greatest likelihood as the insertion method to be used. In addition, for example, the determination unit 2040 determines multiple candidates for the insertion method to be used based on the likelihood calculated for each insertion method, and determines the insertion method to be used from the multiple candidates. For example, the determination unit 2040 determines the n (n>1) insertion methods with the highest likelihood as candidates for the insertion method to be used. In addition, for example, the determination unit 2040 determines the insertion method with a likelihood equal to or greater than a threshold as a candidate for the insertion method to be used.

[0062] Various methods can be used to determine the insertion method to be used from a plurality of candidates. For example, a priority is pre-assigned to each of the plurality of insertion methods. In this case, for example, the determination unit 2040 determines the insertion method with the highest priority among the candidate insertion methods as the insertion method to be used. In addition, for example, the determination unit 2040 corrects the likelihood by multiplying the calculated likelihood by a weight based on the priority of each insertion method among the candidate insertion methods. Note that the weight based on the priority is pre-determined to be a larger value as the priority is higher. The determination unit 2040 determines the insertion method with the greatest corrected likelihood as the insertion method to be used.

[0063] Furthermore, for example, when the shape data 52 indicates a shape category, important shape categories and transitions between important shape categories (arrangements of two or more shape categories) are predetermined for each of a plurality of insertion methods. In this case, the determination unit 2040 determines the insertion method to be used as the insertion method that includes important shape categories and transitions between important shape categories in the shape data sequence 50 among the insertion methods determined as candidate insertion methods.

[0064] For example, assume that insertion methods X and Y are determined as candidates based on likelihood. Furthermore, assume that the important shape category determined for insertion method X is B, and the important shape category determined for insertion method Y is C. Furthermore, assume that the time series of shape data 52 indicated by shape data sequence 50 is "A, C, D, C, A." In this case, shape data sequence 50 does not include shape category B, which is an important shape category for insertion method X, but does include shape category C, which is an important shape category for insertion method Y. Therefore, determination unit 2040 determines insertion method Y as the insertion method to be used.

[0065] Furthermore, it is assumed that the shape data sequence 50 includes multiple insertion methods with important shape categories and transitions between shape categories. In this case, for example, the determination unit 2040 determines the insertion method with the highest calculated likelihood among the multiple insertion methods as the insertion method to be used. Alternatively, for example, the determination unit 2040 may determine the insertion method with the most important shape categories and transitions between shape categories included in the shape data sequence 50 as the insertion method to be used.

[0066] The determination unit 2040 may also use information related to the operator of the endoscope scope 40 to determine the insertion method to be used from the candidate insertion methods. For example, information associating the operator's identification information with the insertion methods that the operator can use is prepared in advance. The determination unit 2040 determines the insertion method to be used as the insertion method that is limited to the insertion methods that can be used by the operator of the endoscope scope 40 among the candidate insertion methods. Note that if there are multiple insertion methods among the candidate insertion methods that are determined to be insertion methods that can be used by the operator of the endoscope scope 40, the determination unit 2040 determines the insertion method to be used based on, for example, likelihood, the level of priority described above, etc.

[0067] Furthermore, for example, information indicating the operator's skill level (e.g., years of experience or number of times performing endoscopy, etc., or evaluations by experienced physicians) may be used as information related to the operator of the endoscope scope 40. In this case, a relationship between the operator's skill level and each of the multiple insertion methods is defined (e.g., "used by a person with five or more years of experience"). Based on the relationship between the operator's skill level and the insertion method, the determination unit 2040 determines the insertion method to be used from the multiple candidates.

[0068] The determination unit 2040 can determine the insertion method to be used from the candidate insertion methods by using image data obtained from the camera 10 provided on the endoscope scope 40. For example, the determination unit 2040 extracts features from the image data obtained by the camera 10 and determines the insertion method that matches the features from the candidate insertion methods as the insertion method to be used. Examples of the features extracted from the image data include a phenomenon in which the entire image turns red (referred to as a red ball) due to the tip of the endoscope scope 40 being pressed against the wall of an internal organ, and characteristic movements of the screen or the operation of the endoscope scope 40.

[0069] The determination unit 2040 may calculate the likelihood of using each insertion method in each of a plurality of methods (e.g., a method utilizing a recognition model and a method utilizing a state transition model, described later), and determine the insertion method to be used based on the calculation result. For example, the determination unit 2040 calculates a statistical value (e.g., an average value, a maximum value) of the likelihood calculated for each insertion method in the plurality of methods, and determines the insertion method having the largest statistical value as the insertion method to be used.

[0070] In addition, for example, the determination unit 2040 can determine the insertion method used for each method of calculating likelihood. For example, it is assumed that the insertion method determined using the recognition model is insertion method X, and the insertion method determined using the state transition model is insertion method Y. In this case, the determination unit 2040 determines insertion method X and insertion method Y as the insertion methods used. In this case, it is preferred that the insertion method information 20 indicates the method for determining the insertion method (method for calculating likelihood) and the insertion method determined by the method in association with each other. For example, in the aforementioned example, by describing "recognition model: insertion method X, state transition model: insertion method Y" and the like, it is possible to indicate the association between the method for calculating likelihood and the determined insertion method.

[0071] <Specific method for calculating likelihood>

[0072] As described above, for example, the determination unit 2040 calculates the likelihood of using each of the plurality of insertion methods to determine the insertion method to be used. Two methods for calculating the likelihood are described below as specific examples.

[0073] <<Method using recognition model>>

[0074] In this method, a recognition model is used that is trained to output the likelihood of using each insertion method in response to the shape data sequence 50 being input. Hereinafter, this recognition model is referred to as an insertion method recognition model. For example, the insertion method recognition model is implemented by utilizing a recurrent neural network (RNN), which is a neural network that processes time series data. However, the type of model used to implement the insertion method recognition model is not limited to RNN, and any type of model that can process time series data can be utilized.

[0075] The insertion method recognition model is trained using training data consisting of a combination of data representing time-series changes in the shape of the endoscope body and correct answer output data. Correct answer output data is data that indicates the highest likelihood (e.g., 1) for the used insertion method and the lowest likelihood (e.g., 0) for the other insertion methods.

[0076] Note that data other than the shape data sequence 50 can also be input into the insertion method recognition model. Examples of data other than the shape data sequence 50 include still image data or video data obtained from the camera 10, or data representing the movements of the operator of the endoscope scope 40 (e.g., time series data of acceleration obtained from an accelerometer attached to the operator's hand). Furthermore, if the shape data sequence 50 represents the shape of the endoscope scope 40 by shape category, the three-dimensional coordinates of each of multiple positions of the endoscope scope 40 can also be input into the insertion method recognition model. When such data is utilized, the same type of data is also included in the training data for the insertion method recognition model. In this way, the insertion method recognition model is trained to calculate likelihood using data other than the shape data sequence 50.

[0077] Note that the insertion method identification model may output a label indicating the insertion method being used instead of outputting the likelihood for each insertion method. In this case, for example, the insertion method identification model outputs a label indicating the insertion method for which the maximum likelihood is calculated. Determination unit 2040 can determine the insertion method being used using the label output by the insertion method identification model.

[0078] <<Method using the state transition model>>

[0079] In this case, for each of the multiple insertion methods, a state transition model representing the pattern of temporal changes in the shape of the endoscope scope 40 when that insertion method is used is predetermined and stored in a storage device accessible from the determination unit 2040. When this method is used, the likelihood of each insertion method being used represents, for example, the degree of consistency between the state transition model of that insertion method and the temporal changes in the endoscope scope 40 represented by the shape data sequence 50. Specifically, for each of the multiple insertion methods, the determination unit 2040 calculates the degree of consistency between the state transition model of that insertion method and the shape data sequence 50, and determines the insertion method to be used based on this calculation result. For example, the insertion method with the highest degree of consistency is determined as the insertion method to be used.

[0080] When the shape data 52 indicates a shape type, the likelihood is determined as follows using, for example, the degree of agreement between the state transition model and the shape data sequence 50 .

[0081] Formula 1

[0082]

[0083] In formula (1), the numerator on the right represents the number of shape categories that match the shape data sequence 50 and the state transition model. Meanwhile, the unit interval length in the denominator represents the total number of shape categories contained in the aforementioned unit interval.

[0084] Figure 7 This diagram conceptually illustrates a method for calculating likelihood using equation (1). In this example, a state transition model of insertion method X is compared with a shape data sequence 50. In this example, the unit interval length is 12.

[0085] The state transition model for insertion method X includes four shape categories: shape categories A, C, E, and G. Shape data sequence 50 includes two shape categories A, two shape categories C, and two shape categories E, among the shape categories that constitute the state transition model for insertion method X. Therefore, the number of shape categories that match the state transition model for insertion method X and shape data sequence 50 is 6. Therefore, according to equation (1), 0.5 (6 / 12) is calculated as the likelihood of using insertion method X.

[0086] The calculation formula of likelihood is not limited to formula (1). For example, likelihood can also be calculated using the following formula (2).

[0087] Formula 2

[0088]

[0089] For example Figure 7In the example, five shape categories A, B, C, D, and E are included in the shape data sequence 50 of the unit interval. At the same time, the state transition model of the insertion method X and the shape data sequence 50 are consistent for three shape categories A, C, and E. Therefore, according to equation (2), 0.6 (3 / 5) is calculated as the likelihood of the insertion method X used.

[0090] In addition, when calculating the likelihood for each insertion method, it is necessary not only to consider the degree of matching of the shape categories contained in the unit interval, but also to consider the order of transition by adding calculation formulas that increase the likelihood when following the transition path (sequential relationship) indicated by the state transition model or reduce the likelihood when not following the transition path.

[0091] <Output of Insertion Method Information 20: S108>

[0092] The output unit 2060 outputs the insertion method information 20. There are various methods for outputting the insertion method information 20. For example, the output unit 2060 displays a screen showing the insertion method information 20 on a display device that can be viewed by the operator of the endoscope scope 40. Furthermore, for example, the output unit 2060 can store a file showing the insertion method information 20 in a storage device or transmit it to any other device.

[0093] Various information can be generated as the insertion method information 20. For example, the insertion method information 20 is information indicating the name of the insertion method used. In addition, for example, the insertion method information 20 can indicate the transition of the insertion method used (change of the insertion method used over time).

[0094] Figure 8 2 is a diagram illustrating the insertion method information 20 . Figure 8 The insertion method information 20 shows two graphs. The solid line represents the transition of the insertion method for insertion of the endoscope scope 40 over time, which is determined by the endoscope operation assisting device 2000. As described above, the insertion method for insertion of the endoscope scope 40 may change over time. Figure 8 In the example shown, the endoscope operation support device 2000 determines the insertion method used in each unit interval. The determined insertion method is then plotted in association with the time point corresponding to the unit interval (e.g., the start time point of the unit interval). Figure 8 In the example, the insertion methods used are switched in the order of insertion method Z, insertion method X, insertion method Y, and insertion method Z.

[0095] The dashed-dotted line indicates the likelihood of the determined insertion method. For example, in the interval where insertion method Z is determined and used, the likelihood of insertion method Z being used is indicated. Similarly, in the interval where insertion method X is determined and used, the likelihood of insertion method X being used is indicated.

[0096] While the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the structure and details of the present application within the scope of the present invention.

[0097] In addition, in the above examples, the program can be stored using various types of non-transitory computer readable media and can be provided to the computer. Non-transitory computer readable media include various types of tangible recording media (tangible storage medium). Examples of non-transitory computer readable media include magnetic recording media (such as floppy disks, magnetic tapes, or hard disk drives), optical magnetic recording media (such as optical magnetic disks), CD-ROMs, CD-Rs, CD-R / Ws, semiconductor memories (such as mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, RAMs). In addition, the program can also be provided to the computer via various types of transitory computer readable media (transitory computer readable media). Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer readable media can provide the program to the computer via wireless communication paths or wired communication paths such as electric wires and optical fibers.

[0098] A part or all of the above-mentioned embodiments can also be described as the following supplementary notes, but are not limited to the following contents.

[0099] (Note 1)

[0100] An endoscope operation assisting device, comprising:

[0101] an acquisition unit configured to acquire a shape data sequence representing a change in the shape data of the endoscope scope over time;

[0102] a determining unit configured to determine an insertion method for inserting the endoscope scope based on the shape data sequence; and

[0103] The output unit is configured to output insertion method information related to the determined insertion method.

[0104] (Note 2)

[0105] According to the endoscope operation assistance device described in Supplementary Note 1, for each of the multiple insertion methods of the endoscope body, the determination unit calculates the likelihood that the insertion method is used based on the shape data sequence, and determines the insertion method to be used based on the calculated likelihood.

[0106] (Note 3)

[0107] According to the endoscope operation assistance device described in Appendix 2, for each insertion method among a plurality of insertion methods, the determination unit obtains a state transition model representing a pattern of change over time in the shape of the endoscope body when the insertion method is used, and calculates the likelihood of the insertion method being used based on the degree of matching between the state transition model and the shape data sequence.

[0108] (Note 4)

[0109] According to the endoscope operation assistance device described in Appendix 2, the determination unit determines the likelihood of each insertion method being used by inputting the shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the likelihood of each insertion method being used among a plurality of insertion methods in response to the input of a data sequence representing the change of the shape data of the endoscope body over time.

[0110] (Note 5)

[0111] The endoscope operation assisting device according to any one of Supplementary Notes 1 to 4, wherein the determining unit performs:

[0112] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used; and

[0113] Based on the priorities determined for the respective insertion methods, the insertion method to be used is determined from among the plurality of candidates for the insertion method.

[0114] (Note 6)

[0115] The endoscope operation assisting device according to any one of Supplementary Notes 1 to 4, wherein the determining unit performs:

[0116] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used; and

[0117] The insertion method to be used is determined from the plurality of candidates for the insertion method using information related to the operator of the endoscope scope.

[0118] (Note 7)

[0119] The endoscope operation assisting device according to Supplementary Note 6, wherein the information related to the operator of the endoscope scope indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope scope.

[0120] (Note 8)

[0121] According to the endoscope operation assistance device described in Supplementary Note 1, the determination unit determines the insertion method used by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output data representing the insertion method used in response to the input of a data sequence representing the change of shape data of the endoscope body over time.

[0122] (Note 9)

[0123] The endoscope operation assisting device according to any one of Supplementary Notes 1 to 8, wherein the output unit outputs insertion method information including a change in the insertion method used over time.

[0124] (Note 10)

[0125] The endoscope operation assisting device according to any one of Supplementary Notes 1 to 9,

[0126] A plurality of shape classes are defined, the plurality of shape classes being the types of shapes that the endoscope scope may take, and

[0127] Each piece of shape data included in the shape data sequence indicates one of a plurality of shape categories of the endoscope scope.

[0128] (Note 11)

[0129] A control method executed by a computer, the control method comprising:

[0130] an acquisition step of acquiring a shape data sequence representing a change in the shape of the endoscope body over time;

[0131] a determining step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; and

[0132] The output step outputs insertion method information related to the determined insertion method.

[0133] (Note 12)

[0134] According to the control method described in Supplementary Note 11, in the determination step, for each of the plurality of insertion methods of the endoscope scope, the likelihood that the insertion method is used is calculated using the shape data sequence, and the insertion method to be used is determined based on the calculated likelihood.

[0135] (Note 13)

[0136] According to the control method described in Appendix 12, in the determination step, for each insertion method among the multiple insertion methods, a state transition model representing a pattern of change in the shape of the endoscope body over time when the insertion method is used is obtained, and based on the degree of matching between the state transition model and the shape data sequence, the likelihood of the insertion method being used is calculated.

[0137] (Note 14)

[0138] According to the control method described in Note 12, in the determination step, the likelihood of each insertion method being used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the likelihood of each insertion method being used among a plurality of insertion methods in response to an input representing a change in the shape data of the endoscope body over time.

[0139] (Note 15)

[0140] The control method according to any one of Supplementary Notes 11 to 14, wherein in the determining step,

[0141] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0142] The insertion method to be used is determined from among a plurality of candidates for the insertion method based on the priority defined for each insertion method.

[0143] (Note 16)

[0144] The control method according to any one of Supplementary Notes 11 to 14, wherein in the determining step,

[0145] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0146] The insertion method to be used is determined from a plurality of candidates for the insertion method using information related to the operator of the endoscope scope.

[0147] (Note 17)

[0148] The control method according to Supplementary Note 16, wherein the information related to the operator of the endoscope scope indicates one or more insertion methods used by the operator, or indicates the skill level of the operator related to the operation of the endoscope scope.

[0149] (Note 18)

[0150] According to the control method described in Note 11, in the determination step, the insertion method used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output data representing the insertion method used in response to the input of a data sequence representing the change of the shape data of the endoscope body over time.

[0151] (Note 19)

[0152] The control method according to any one of Supplementary Notes 11 to 18, wherein in the outputting step, insertion method information including a temporal transition of the used insertion method is output.

[0153] (Note 20)

[0154] According to any one of the control methods of Supplementary Notes 11 to 19,

[0155] wherein a plurality of shape classes are defined, the plurality of shape classes being types of shapes that the endoscope scope may take, and

[0156] Each piece of shape data included in the shape data sequence indicates one of a plurality of shape categories of the endoscope scope.

[0157] (Note 21)

[0158] A computer-readable medium storing a program that causes a computer to:

[0159] an acquisition step of acquiring a shape data sequence representing a change in the shape of the endoscope body over time;

[0160] a determining step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; and

[0161] The output step outputs insertion method information related to the determined insertion method.

[0162] (Note 22)

[0163] A computer-readable medium according to Note 21, wherein in the determining step, for each of a plurality of insertion methods of the endoscope body, the likelihood of the insertion method being used is calculated based on the shape data sequence, and based on the calculated likelihood, the insertion method to be used is determined.

[0164] (Note 23)

[0165] The computer-readable medium according to Note 22, wherein in the determining step, for each of a plurality of insertion methods, a state transition model is obtained that represents a pattern of change in the shape of the endoscope body over time when the insertion method is used, and based on the degree of matching between the state transition model and the shape data sequence, the likelihood of the insertion method being used is calculated.

[0166] (Note 24)

[0167] The computer-readable medium according to Note 22, wherein in the determining step, the likelihood of each insertion method being used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the likelihood of each insertion method being used among a plurality of insertion methods in response to an input of a data sequence representing shape data of an endoscope body that changes over time.

[0168] (Note 25)

[0169] The computer-readable medium according to any one of Notes 21 to 24, wherein in the determining step,

[0170] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0171] The insertion method to be used is determined from among the plurality of candidates for the insertion method based on the priority defined for each insertion method.

[0172] (Note 26)

[0173] The computer-readable medium according to any one of Notes 21 to 24, wherein in the determining step,

[0174] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0175] The insertion method to be used is determined from the plurality of candidates for the insertion method using information related to the operator of the endoscope scope.

[0176] (Note 27)

[0177] The computer-readable medium according to supplementary note 26, wherein the information related to the operator of the endoscope indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope.

[0178] (Note 28)

[0179] The computer-readable medium according to Note 21, wherein in the determining step, the insertion method used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output data representing the insertion method used in response to the input of a data sequence representing the time-varying shape data of the endoscope body.

[0180] (Note 29)

[0181] The computer-readable medium according to any one of Supplementary Notes 21 to 28, wherein in the outputting step, insertion method information including a change over time of the insertion method used is output.

[0182] (Note 30)

[0183] The computer-readable medium according to any one of Notes 21 to 29,

[0184] wherein a plurality of shape classes are defined, the plurality of shape classes being types of shapes that the endoscope scope may take, and

[0185] Each piece of shape data included in the shape data sequence indicates one of a plurality of shape categories of the endoscope scope.

[0186] (Note 31)

[0187] A program that causes a computer to:

[0188] an acquisition step of acquiring a shape data sequence representing a change in the shape of the endoscope body over time;

[0189] a determining step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; and

[0190] The output step outputs insertion method information related to the determined insertion method.

[0191] (Note 32)

[0192] The program according to Supplementary Note 31, wherein in the determination step, for each of a plurality of insertion methods of the endoscope scope, the likelihood of the insertion method being used is calculated based on the shape data sequence, and based on the calculated likelihood, the insertion method to be used is determined.

[0193] (Note 33)

[0194] A program according to Note 32, wherein in the determining step, for each of a plurality of insertion methods, a state transition model is obtained that represents a pattern of change over time in the shape of the endoscope body when the insertion method is used, and based on the degree of matching between the state transition model and the shape data sequence, the likelihood of the insertion method being used is calculated.

[0195] (Note 34)

[0196] A program according to Note 32, wherein in the determination step, the likelihood of each insertion method being used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the likelihood of each insertion method being used among a plurality of insertion methods in response to input of a data sequence representing time-varying shape data of an endoscope body.

[0197] (Note 35)

[0198] The program according to any one of Notes 31 to 34, wherein in the determining step,

[0199] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0200] The insertion method to be used is determined from among the plurality of candidates for the insertion method based on the priority defined for each insertion method.

[0201] (Note 36)

[0202] The program according to any one of Notes 31 to 34, wherein in the determining step,

[0203] determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and

[0204] The insertion method to be used is determined from the plurality of candidates for the insertion method using information related to the operator of the endoscope scope.

[0205] (Note 37)

[0206] The program according to supplementary note 36, wherein the information related to the operator of the endoscope scope indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope scope.

[0207] (Note 38)

[0208] A program according to Note 31, wherein in the above-mentioned determination step, the insertion method used is determined by inputting a shape data sequence into a trained recognition model, and the trained recognition model is configured to: output data representing the insertion method used in response to the input of a data sequence representing the time-varying shape data of the endoscope body.

[0209] (Note 39)

[0210] The program according to any one of Supplementary Notes 31 to 38, wherein in the outputting step, insertion method information including a change over time of the insertion method used is output.

[0211] (Note 40)

[0212] According to any one of the procedures of Notes 31 to 39,

[0213] wherein a plurality of shape classes are defined, the plurality of shape classes being types of shapes that the endoscope scope may take, and

[0214] Each piece of shape data included in the shape data sequence indicates one of a plurality of shape categories of the endoscope scope.

[0215] Description of Reference Numerals

[0216] 10. Camera

[0217] 20 Insert method information

[0218] 40 Endoscope body

[0219] 50 shape data sequences

[0220] 51 Time Point

[0221] 52 shape data

[0222] 60 Endoscope control unit

[0223] 500 Computer

[0224] 500 computers

[0225] 502 bus

[0226] 504 processor

[0227] 506 Memory

[0228] 508 storage devices

[0229] 510 input and output interface

[0230] 512 network interfaces

[0231] 2000 Endoscopic operation assist device

[0232] 2020 Acquisition Unit

[0233] 2040 Determine Unit

[0234] 2060 Output Unit

Claims

1. An endoscope operation assisting device, comprising: an acquisition unit configured to acquire a shape data sequence representing a change in shape data of an endoscope scope over time; A determination unit is configured to determine an insertion method for inserting the endoscope scope based on the shape data sequence; wherein the determination unit performs: determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used; as well as determining the insertion method to be used from the plurality of candidates for the insertion method based on a priority defined for each insertion method; as well as The output unit is configured to output insertion method information related to the determined insertion method.

2. The endoscope operation assisting device according to claim 1 , wherein the determining unit calculates, for each of a plurality of insertion methods of the endoscope scope, a likelihood that the insertion method is used based on the shape data sequence, and determines the insertion method to be used based on the calculated likelihood.

3. The endoscope operation assisting device according to claim 2 , wherein the determining unit obtains, for each of the plurality of insertion methods, a state transition model representing a pattern of temporal change in the shape of the endoscope scope when the insertion method is used, and calculates the likelihood that the insertion method is used based on a degree of matching between the state transition model and the shape data sequence.

4. An endoscope operation assistance device according to claim 2, wherein the determination unit determines the likelihood of each insertion method being used by inputting the shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the likelihood of each insertion method being used in response to the input of the data sequence representing the temporal change of the shape data of the endoscope body.

5. The endoscope operation assisting device according to any one of claims 1 to 4, wherein the determining unit performs: determining a plurality of candidates for each insertion method to be used based on the likelihood that the insertion method is to be used; and The insertion method to be used is determined from among the plurality of candidates for the insertion method using information related to an operator of the endoscope scope.

6. The endoscope operation assisting device according to claim 5, wherein the information related to the operator of the endoscope scope indicates one or more insertion methods used by the operator, or indicates the operator's skill level related to the operation of the endoscope scope.

7. An endoscope operation assisting device according to claim 1, wherein the determination unit determines the insertion method used by inputting the shape data sequence into a trained recognition model, and the trained recognition model is configured to: output data representing the insertion method used in response to the input of the data sequence representing the temporal change of the shape data of the endoscope body.

8. A control method, executed by a computer, comprising: an acquisition step of acquiring a shape data sequence representing a change in the shape of the endoscope body over time; a determining step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; In the determination step, determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and determining an insertion method to be used from a plurality of candidates for the insertion method based on a priority defined for each insertion method; as well as An outputting step of outputting insertion method information related to the determined insertion method.

9. A computer-readable medium storing a program, wherein the program causes a computer to execute: an acquisition step of acquiring a shape data sequence representing a change in the shape of the endoscope body over time; a determining step of determining an insertion method for inserting the endoscope scope based on the shape data sequence; In the determination step, determining a plurality of candidates for the insertion method to be used based on the likelihood that each insertion method is used, and determining an insertion method to be used from the plurality of candidates for the insertion method based on a priority defined for each insertion method; as well as An outputting step of outputting insertion method information related to the determined insertion method.

Images