Endoscopic operation support device, control method, computer-readable medium, and program

Abstract

The endoscope operation support device (2000) acquires a shape data sequence (50), and the shape data sequence (50) represents the temporal change of the shape category of the endoscope observation instrument (40). The endoscope operation support device (2000) uses the shape data sequence (50) to determine a target category or a target position, where the target category is suitable as the transition destination of the shape category of the endoscope observation instrument (40), and the target position is a position suitable as the position where the endoscope observation instrument (40) undergoes a transition. The endoscope operation support device (2000) outputs target information, and the target information is information (20) related to the target category or the target position.

Description

Technical Field

[0001] The present disclosure relates to a method for supporting endoscopic operations. Background Art

[0002] Devices for supporting surgeries, examinations, etc. using an endoscope have been developed. For example, Patent Document 1 discloses a technique that programs the rules of thumb of an experienced doctor and, based on an image obtained from an endoscopic camera and the shape of the endoscope, presents operations to be performed on the endoscope.

[0003] References

[0004] Patent Documents

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2019-097661 Summary of the Invention

[0006] Technical Problem

[0007] The present inventors have studied new techniques for supporting endoscopic operations. An object of the present disclosure is to provide a new technique for supporting endoscopic operations.

[0008] Solution to the Problem

[0009] According to one aspect of the present disclosure, there is provided an endoscopic operation support device including: an acquisition unit configured to acquire a sequence of shape data representing a temporal change in a shape category of an endoscopic observation instrument inserted into a body; a determination unit configured to determine a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination of the shape category of the endoscopic observation instrument, the target position being a position suitable as a position where the endoscopic observation instrument is to be transitioned; and an output unit configured to output target information that is information related to the target category or the target position.

[0010] According to another aspect of the present disclosure, there is provided a control method executed by a computer. The control method includes: an acquisition step of acquiring a sequence of shape data representing a temporal change in a shape category of an endoscopic observation instrument inserted into a body; a determination step of determining a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination of the shape category of the endoscopic observation instrument, the target position being a position suitable as a position where the endoscopic observation instrument is to be transitioned; and an output step of outputting target information that is information related to the target category or the target position.

[0011] According to another aspect, a computer-readable medium storing a program is provided, and the program causes a computer to execute the control method of the present invention.

[0012] Advantageous Effects of the Invention

[0013] According to the disclosure of the present disclosure, a new technology for supporting endoscopic operations is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram showing the operation of an endoscopic operation support device according to a first exemplary embodiment.

[0015] Figure 2 is a block diagram showing the functional configuration of an endoscopic operation support device according to a first exemplary embodiment.

[0016] Figure 3 is a block diagram showing the hardware configuration of a computer that implements an endoscopic operation support device.

[0017] Figure 4 is a diagram showing a specific example of the usage environment of an endoscopic operation support device.

[0018] Figure 5 is a flowchart showing the processing flow executed by an endoscopic operation support device according to a first exemplary embodiment.

[0019] Figure 6 is a diagram showing the configuration of a shape data sequence in tabular form.

[0020] Figure 7 is a diagram showing a state transition model of an insertion method of an endoscopic observation instrument.

[0021] Figure 8 is a diagram conceptually showing a method of calculating accuracy using Equation (1).

[0022] Figure 9 is a diagram showing a screen including target information.

[0023] Figure 10 is a diagram showing target information including a plurality of shape categories determined as candidates for a target category.

[0024] Figure 11 is a diagram showing target information including a target position.

[0025] Figure 12 is a diagram showing target information representing the shape of an endoscopic observation instrument in a three-dimensional image.

[0026] Figure 13This is a diagram showing target information, in which the shape of an endoscope observation instrument corresponding to a target category is represented by a three-dimensional image. Detailed implementation manners

[0027] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or corresponding elements, and repeated descriptions are omitted when necessary for clarity of explanation. Additionally, unless otherwise specified, predetermined values such as predetermined values and threshold values are pre-stored in a storage device accessible from a device using these values.

[0028] Figure 1 This is a schematic diagram showing the operation of the endoscope operation support device 2000 according to the first exemplary embodiment. Here, Figure 1 This is a diagram for helping to understand the outline of the endoscope operation support device 2000, and the operation of the endoscope operation support device 2000 is not limited to Figure 1 the operation shown in

[0029] In surgeries, examinations, etc. using an endoscope (hereinafter referred to as endoscope surgeries, etc.), the operator of the endoscope observation instrument 40 inserts the endoscope observation instrument 40 into the body while changing the shape of the endoscope observation instrument 40 in various ways. Here, it is assumed that the shapes that the endoscope observation instrument 40 can take are pre-classified into multiple categories. This classification of the shape of the endoscope observation instrument 40 is called a shape category. For example, as shape categories, various categories such as "straight line", "small curve", "large curve", and "abdominal protrusion" can be defined.

[0030] The manner in which the shape category of the endoscope observation instrument 40 changes can vary depending on the manner of changing the shape category of the endoscope observation instrument 40. Therefore, the endoscope operation support device 2000 acquires a shape data sequence 50 that represents the temporal change of the shape category of the endoscope observation instrument 40 inserted into the body, and the endoscope operation support device 2000 uses this shape data sequence 50 to determine the shape category that the endoscope observation instrument 40 should transform into. The shape category determined in this way (i.e., the shape category suitable as the transformation destination) is called the target category.

[0031] The endoscope operation support device 2000 outputs target information 20 related to the determined target category. For example, this target category indicates the name of the shape category that the endoscope observation instrument 40 should take next. For example, in Figure 1 the example of , the target information 20 includes a message indicating that the target category is "category E" and an image representing the shape of the endoscope observation instrument 40 classified as category E.

[0032] However, in addition to or in place of the target category, the target information 20 may indicate a target position, which is an appropriate position as the transformation destination of the endoscopic observation instrument 40.

[0033] <Example of beneficial effect>

[0034] As described above, the endoscopic operation support device 2000 according to the present exemplary embodiment uses the shape data sequence 50 to determine the target category or the target position, and the shape data sequence 50 represents the temporal change of the shape category of the endoscopic observation instrument 40. That is, it is determined which shape category is appropriate for the endoscopic observation instrument 40 to transform to, or to which position the endoscopic observation instrument 40 should be transformed. Then, the endoscopic operation support device 2000 outputs the target information 20 related to the target category or the target position.

[0035] By using the target information 20, the operator of the endoscopic observation instrument 40 or the like can grasp how the shape of the endoscopic observation instrument 40 should be changed and which position of the endoscopic observation instrument 40 should be changed. Here, when the target shape category and the target position are known, it is possible to determine what operation should be performed on the endoscopic observation instrument 40. Therefore, the operator of the endoscopic observation instrument 40 or the like can grasp the appropriate operation that should be performed on the endoscopic observation instrument 40.

[0036] Hereinafter, the endoscopic operation support device 2000 according to the present exemplary embodiment will be described in more detail.

[0037] <Example of functional configuration>

[0038] Figure 2 is a block diagram showing the functional configuration of the endoscopic operation support device 2000 according to the first exemplary embodiment. The endoscopic operation support device 2000 includes an acquisition unit 2020, a determination unit 2040, and an output unit 2060. The acquisition unit 2020 acquires the shape data sequence 50. The determination unit 2040 uses the shape data sequence 50 to determine the target category or the target position. The output unit 2060 outputs the target information 20 related to the target category or the target position.

[0039] <Example of hardware configuration>

[0040] Each functional component of the endoscopic operation support device 2000 can be implemented by hardware (e.g., hard-wired electronic circuits, etc.) that implements each functional component, or can be implemented by a combination of hardware and software (e.g., a combination of an electronic circuit and a program for controlling the electronic circuit, etc.). Hereinafter, the case where each functional component of the endoscopic operation support device 2000 is implemented by a combination of hardware and software will be further described.

[0041] Figure 3 FIG. 1 is a block diagram showing the hardware configuration of a computer 500 that implements the endoscopic operation support apparatus 2000. The computer 500 is any computer. For example, the computer 500 is a fixed computer such as a personal computer (PC) or a server machine. In another example, the computer 500 is a portable computer such as a smart phone or a tablet terminal. The computer 500 may be a dedicated computer designed to implement the endoscopic operation support apparatus 2000, or may be a general-purpose computer.

[0042] For example, each function of the endoscopic operation support apparatus 2000 is implemented in the computer 500 by installing a predetermined application program. This application includes a program for implementing the functional components of the endoscopic operation support apparatus 2000. The installation of the application program can be achieved by copying the program from a recording medium (e.g., USB memory, DVD, etc.) storing the program or downloading the program from a server that manages the storage device storing the program.

[0043] The computer 500 includes a bus 502, a processor 504, a memory 506, a storage device 508, an input / output interface 510, and a network interface 512. The bus 502 is a data transmission path for the processor 504, the memory 506, the storage device 508, the input / output interface 510, and the network interface 512 to send and receive data to and from each other. However, the method of interconnecting the processor 504, etc. is not limited to bus connection.

[0044] The processor 504 is various processors such as a central processing unit (CPU), a graphics processing unit (GPU), or a field programmable gate array (FPGA). The memory 506 is a main storage device implemented by using a random access memory (RAM), etc. The storage device 508 is an auxiliary storage device implemented by using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), etc.

[0045] The input / output interface 510 is an interface for connecting the computer 500 and an input / output device. For example, an endoscopic control device 60 described later is connected to the input / output interface 510. In another example, an input device such as a keyboard and an output device such as a display device are connected to the input / output interface 510.

[0046] The network interface 512 is an interface for connecting the computer 500 to a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0047] The storage device 508 stores programs for implementing each functional component of the endoscope operation support device 2000 (programs for implementing the above applications). The processor 504 implements each functional component of the endoscope operation support device 2000 by reading and executing the programs in the memory 506. For example, the programs are stored in the storage device 508 by copying the programs from a storage medium (e.g., USB memory) storing the programs.

[0048] The endoscope operation support device 2000 can be implemented by one computer 500, or can also be implemented by multiple computers 500. In the latter case, the configurations of the computers 500 do not need to be the same but can be different from each other.

[0049] <Examples of the usage environment of the endoscope operation support device 2000>

[0050] Figure 4 is a diagram showing a specific example of the usage environment of the endoscope operation support device 2000. For example, the endoscope operation support device 2000 is used together with an endoscope observation instrument 40 and an endoscope control device 60. The endoscope observation instrument 40 is an observation instrument inserted into the body and is provided with a camera 10. By viewing the video data generated by the camera 10, the state inside the body can be visually recognized.

[0051] The endoscope control device 60 is a control device used when performing an endoscope surgery or the like using the endoscope observation instrument 40. For example, the endoscope control device 60 generates video data to be viewed by performing various processes on the video data obtained from the camera provided in the endoscope observation instrument 40, and outputs the generated video data. Examples of the processes performed on the video data include processes for adjusting the color or brightness of the video data and processes for superimposing various types of information on the video data.

[0052] For example, the endoscope operation support device 2000 acquires various types of information via the endoscope control device 60. For example, the endoscope operation support device 2000 acquires still image data or video data (collectively referred to as image data below) generated by the camera 10 from the endoscope control device 60.

[0053] Note that the data obtained from the endoscope observation instrument 40 is not limited to the image data generated by the camera 10. For example, the endoscope observation instrument 40 can be provided with elements for obtaining three-dimensional coordinates for each of a plurality of positions of the endoscope observation instrument 40. In this case, the endoscope control device 60 determines the three-dimensional coordinates for each of the plurality of positions of the endoscope observation instrument 40 by using the elements. By using the three-dimensional coordinates of the plurality of positions of the endoscope observation instrument 40 obtained in this way, the shape of the endoscope observation instrument 40 can be grasped.

[0054] <Processing Flow>

[0055] Figure 5 is a flowchart showing the processing flow executed by the endoscope operation support device 2000 according to the first exemplary embodiment. The acquisition unit 2020 acquires the shape data sequence 50 (S102). The determination unit 2040 uses the shape data sequence 50 to determine the target category or the target position (S104). The determination unit 2040 outputs the target information 20 related to the determined target category or target position (S106).

[0056] <Shape Data Sequence 50>

[0057] The shape data sequence 50 is data representing the shape of the endoscope observation instrument 40 in time series. For example, the shape data sequence 50 indicates the shape of the endoscope observation instrument 40 at each of a plurality of time points. In the shape data sequence 50, the shape of the endoscope observation instrument 40 is represented by one of a plurality of determined shape categories.

[0058] Figure 6 is a diagram showing the configuration of the shape data sequence 50 in tabular form. Figure 6 The shape data sequence 50 in includes the time point 51 and the shape data 52. Each record in the shape data sequence 50 indicates that: at the time point indicated by the time point 51, the shape of the endoscope observation instrument 40 belongs to the shape category indicated in the shape data 52. The time point 51 can be represented by time, or can be represented by a method other than time. In the latter case, for example, the time point 51 indicates a relative value, such as a serial number, assigned to each piece of shape data 52 in chronological order.

[0059] The shape category to which the shape of the endoscope observation instrument 40 belongs can be determined using, for example, the three-dimensional coordinates of each of a plurality of specific positions of the endoscope observation instrument 40. As described above, for example, the three-dimensional coordinates of each of a plurality of positions of the endoscope observation instrument 40 can be determined using elements provided at these positions.

[0060] For example, an identification model (hereinafter referred to as a shape recognition model) is used to determine a shape category. The shape recognition model can be implemented by various types of models such as neural networks or support vector machines. The shape recognition model is pre-trained so as to output a label representing the shape category of the endoscopic observation instrument 40 in response to the input of the three-dimensional coordinates of each of a plurality of specific positions of the endoscopic observation instrument 40. For each of a plurality of time points, the shape category of the endoscopic observation instrument 40 at that time point can be determined by inputting the three-dimensional coordinates of each of the plurality of specific positions of the endoscopic observation instrument 40 at that time point into the shape recognition model. Note that the training of the shape recognition model can be performed by using training data including a combination of "the three-dimensional coordinates of each of the plurality of specific positions of the endoscopic observation instrument 40 and the label representing the ground-truth shape category".

[0061] <Acquisition of the shape data sequence 50: S102>

[0062] The acquisition unit 2020 acquires the shape data sequence 50 (S102). There are various methods for the acquisition unit 2020 to acquire the shape data sequence 50. For example, the acquisition unit 2020 acquires the shape data sequence 50 stored in a storage device accessible from the acquisition unit 2020 by accessing the storage device. The storage device can be provided inside the endoscopic operation support device 2000, or can also be provided outside the endoscopic operation support device 2000. In another example, the acquisition unit 2020 can acquire the shape data sequence 50 by receiving the shape data sequence 50 sent from another device.

[0063] Here, in the shape data sequence 50, the shape of the endoscopic observation instrument 40 is represented by a shape category. Therefore, in order to generate the shape data sequence 50, it is necessary to perform a process of determining the shape category of the endoscopic observation instrument 40 at each time point. This process can be performed by the endoscopic operation support device 2000, or can also be performed by a device other than the endoscopic operation support device 2000. In the former case, the acquisition unit 2020 acquires the shape data sequence 50 generated inside the endoscopic operation support device 2000. For example, in this case, the shape data sequence 50 is stored in a storage device inside the endoscopic operation support device 2000. Meanwhile, in the case where the shape data sequence 50 is generated by a device other than the endoscopic operation support device 2000, for example, the acquisition unit 2020 acquires the shape data sequence 50 by accessing the device that has generated the shape data sequence 50, or receives the shape data sequence 50 sent from that device.

[0064] Here, since the endoscopic observation instrument 40 is inserted into the body while changing its shape, the target category changes over time. Therefore, for example, it is preferable that the endoscopic operation support device 2000 acquires a shape data sequence 50 representing the temporal change of the shape data 52 within a predetermined length of time period (hereinafter referred to as a unit period), and determines the target category corresponding to the shape data sequence 50. In this case, the endoscopic operation support device 2000 can acquire in advance the shape data sequence 50 divided for each unit period, or can acquire a plurality of shape data 52 not divided for each unit period.

[0065] In the latter case, for example, the acquisition unit 2020 acquires the shape data sequence 50 for each unit period by dividing the plurality of acquired shape data 52 for each unit period in chronological order. However, adjacent unit periods may partially overlap each other. For example, when the length of the unit period is 10 and the length of the overlapping period is 4, the first shape data sequence 50 includes the first to tenth shape data 52, and the second shape data sequence 50 includes the seventh to sixteenth shape data 52.

[0066] <Recognition of target category: S104>

[0067] The determination unit 2040 uses the shape data sequence 50 to determine the target category (S104). There are various specific methods for using the shape data sequence 50 to determine the target category. Hereinafter, some specific methods will be illustrated by way of example. The method for determining the target position will be described later.

[0068] <<Method using recognition model>>

[0069] For example, the determination unit 2040 uses a recognition model to determine the target category. For example, the recognition model is trained to output the possibility that each shape category is the target category in response to the input of the shape data sequence 50. Hereinafter, this recognition model will be referred to as a target category recognition model. For example, the target category recognition model is implemented by using a recurrent neural network (RNN), which is a neural network that processes temporal data. However, the model type for implementing the target category recognition model is not limited to RNN, but any type of model capable of processing temporal data can be used.

[0070] The training of the target category recognition model is performed using training data including a combination of "data representing the temporal change of the shape category of the endoscopic observation instrument, and ground truth output data". The ground truth output data is, for example, data representing the maximum possibility (e.g., 1) for the shape category to be treated as the target category, and the minimum possibility (e.g., 0) for other shape categories.

[0071] The determination unit 2040 determines the likelihood that each shape category is the target category by inputting the shape data sequence 50 into the target category recognition model. Then, the determination unit 2040 determines the target category based on the determined likelihoods.

[0072] Various methods can be used as the method for determining the target category based on the likelihoods calculated for each shape category. For example, the determination unit 2040 determines the shape category with the highest likelihood as the target category. In another example, the determination unit 2040 determines as the target category one or more shape categories whose likelihoods are equal to or greater than a threshold, or a plurality of shape categories whose order of likelihoods is equal to or less than a predetermined order.

[0073] Note that the determination unit 2040 can regard a plurality of shape categories whose likelihoods are equal to or greater than a threshold, or a plurality of shape categories whose order of likelihoods is equal to or lower than a predetermined order as candidates for the target category, and determine the target category from the candidate shape categories. There are various methods for determining the target category from the candidate shape categories. For example, priorities are assigned to each shape category in advance. In this case, the determination unit 2040 determines the shape category with the highest priority among the candidate shape categories as the target category.

[0074] Here, which shape category should have priority may vary depending on the situation. For example, there are various insertion methods for inserting the endoscope observation instrument 40 (described in detail below), and the priority of the shape category can vary for each insertion method. Therefore, for example, for each insertion method, the priority of the shape category is determined in advance. Specifically, insertion method information related to the information on the insertion method is pre-stored in a storage device accessible from the determination unit 2040. In the insertion method information, the priority of each shape category is determined for each insertion method.

[0075] In this case, the determination unit 2040 uses the method described later to determine the insertion method for inserting the endoscope observation instrument 40. The determination unit 2040 uses the insertion method information related to the determined insertion method to determine the priority of each shape category. Then, the determination unit 2040 determines the determined shape category with the highest priority among the candidate shape categories as the target category. This enables the appropriate target category to be determined according to the insertion method used.

[0076] In another example, which shape category should have priority can be different for each operator of the endoscope observation instrument 40. Therefore, the priority of the shape category can be determined for each operator of the endoscope observation instrument 40. For example, operator information related to the information on the operator is pre-stored in a storage device accessible from the determination unit 2040. In the operator information, the priority of each shape category is determined for each operator.

[0077] In this case, the determination unit 2040 acquires operator information related to the operator who operates the endoscopic observation instrument 40, and thereby determines the priority of each shape category for the operator. Then, the determination unit 2040 determines the determined shape category with the highest priority among the candidate shape categories as the target category. Therefore, an appropriate target category can be determined according to the characteristics of the operator.

[0078] There are various methods for determining the association between the operator and the shape category priority. For example, records (such as video data) of past endoscopic surgeries performed by the operator can be used to determine the association between the operator and the shape category priority. Specifically, it can be considered that the video data is analyzed to count the number of occurrences of each shape category, and the shape category with a higher number of occurrences has a higher priority order. In another example, the priority of each shape category can be determined for the operator through manual setting by the operator or the operator's supervisor.

[0079] In another example, depending on the skill level of the operator of the endoscopic observation instrument 40, which shape category should be prioritized may be different. Therefore, the priority of each shape category can be determined for each predetermined skill level (for example, five levels from level 1 to level 5, etc.). For example, the skill level information is information in which the skill level and priority of each shape category are correlated with each other, and is pre-stored in a storage device accessible from the determination unit 2040. The above-mentioned operator information includes information indicating the skill level of each operator.

[0080] In this case, the determination unit 2040 uses the operator information related to the operator of the endoscopic observation instrument 40 to determine the skill level of the operator. In addition, the determination unit 2040 determines the priority of each shape category by acquiring the skill level information corresponding to the determined skill level. Then, the determination unit 2040 determines the determined shape category with the highest priority among the candidate shape categories as the target category. This enables an appropriate target category to be determined according to the skill level of the operator.

[0081] There are various methods for determining the skill level of each operator. For example, the numerical range of the years of experience is divided into predetermined stages. In this case, the skill level of the operator is determined by the numerical range to which the operator's years of experience belong. In another example, the skill level of the operator can be evaluated by using the history (such as video data) of past endoscopic surgeries performed by the operator, thereby determining the skill level of the operator. In another example, the skill level of the operator can be determined through manual setting by the operator or the operator's supervisor.

[0082] In another example, information related to a person (hereinafter referred to as the subject) into whom the endoscopic observation instrument 40 is inserted can be used to determine the target category. For example, based on the subject's past surgical history, the locations in the body where adhesions are likely to occur can be determined. Then, depending on the presence or absence of adhesions, the shape category to be selected can be different. Therefore, information associating the type of past surgeries, etc. with the priority of each shape category is pre-stored in a storage device accessible from the determination unit 2040. The determination unit 2040 determines the past surgeries, etc. performed on the subject by acquiring information representing the history of past surgeries, etc. of the subject. In addition, the determination unit 2040 acquires the priority of each shape category associated with the determined surgeries, etc., and uses this priority to determine the target category from the candidate shape categories. This makes it possible to determine an appropriate target category taking into account the influence of past surgeries, etc.

[0083] Furthermore, the priority of the shape category can be determined based on the operation history of the endoscopic observation instrument 40 in past surgeries, etc. with the same content performed on other people. In this case as well, information associating the type of surgeries, etc. with the priority of each shape category is pre-stored in a storage device accessible from the determination unit 2040. For the surgeries, etc. performed on the subject, the determination unit 2040 acquires the priority of each shape category associated with the surgeries, etc. Then, the determination unit 2040 uses the acquired priority to determine the target category from the candidate shape categories. As a result, an appropriate target category can be determined based on the experience of past surgeries, etc.

[0084] Rather than determining one of the candidate shape categories as the target category, the determination unit 2040 can determine the priority order of the candidate shape categories. For example, a higher priority order is assigned to the shape category with a higher possibility. In another example, by determining the priority of the shape category by the various methods described above, a higher priority order can be assigned to the shape category with a higher priority. For example, as will be described later, when emphasizing multiple target categories in the target information 20 based on the priority order, the determined priority order can be used.

[0085] Data other than the shape data sequence 50 can be further input into the target category recognition model. Examples of data other than the shape data sequence 50 include still image data or video data obtained from the camera 10, data representing the movement of the operator of the endoscope observation instrument 40 (e.g., time-series data of acceleration obtained from an acceleration sensor attached to the operator's hand), information related to the operator of the endoscope observation instrument 40 (the operator's technical level and the priority for each shape category defined for the operator), etc. In addition, the three-dimensional coordinates of each of the multiple positions of the endoscope observation instrument 40 can be further input into the target category recognition model. When using this data, the same type of data is also included in the training data of the target category recognition model. In this way, the target category recognition model is trained to calculate the likelihood by further using data other than the shape data sequence 50.

[0086] The target category recognition model can output a label representing the target category instead of outputting the likelihood of each shape category. In this case, for example, the target category recognition model outputs the label representing the shape category with the highest calculated likelihood as the label of the target category. The determination unit 2040 can determine the target category from the label output by the target category recognition model.

[0087] <<Method Using the State Transition Model>>

[0088] For example, the determination unit 2040 determines the insertion method for inserting the endoscope observation instrument 40 and determines the target category based on the state transition model defined for the insertion method. Here, multiple insertion methods can be used to insert the endoscope observation instrument. Examples of the insertion method include the shaft-holding shortening method, the pushing method, and the method of forming an α-loop.

[0089] The temporal change in the shape of the endoscope observation instrument 40 may vary depending on each insertion method used. Therefore, by determining the insertion method for inserting the endoscope observation instrument 40, it is possible to determine how the shape of the endoscope observation instrument 40 should change in the future. Therefore, the determination unit 2040 determines the target category by determining the insertion method for inserting the endoscope observation instrument 40. Note that the specific method for determining the insertion method will be described later.

[0090] For example, for each of the multiple insertion methods, a state transition model representing the temporal change pattern of the shape category of the endoscope observation instrument 40 in the case of using that insertion method is determined in advance and stored in a storage device accessible from the determination unit 2040. Then, the determination unit 2040 uses the state transition model of the insertion method for inserting the endoscope observation instrument 40 to determine the target category.

[0091] Preferably, the shape of the endoscopic observation instrument 40 is changed according to the state transition model of the insertion method for inserting the endoscopic observation instrument 40. Therefore, for example, the determination unit 2040 determines the next shape category of the shape category that is the last in the shape data sequence 50 in the state transition model of the insertion method for inserting the endoscopic observation instrument 40 as the target category.

[0092] A specific example of a method for determining a target category using a state transition model will be described. Figure 7 FIG. is a diagram showing a state transition model of the insertion method of the endoscopic observation instrument 40. Here, it is assumed that the determination unit 2040 determines to insert the endoscopic observation instrument 40 using the insertion method X using the shape data sequence 50. In addition, it is assumed that Figure 7 shows the state transition model of the insertion method X. Then, it is assumed that the last shape category in the shape data sequence 50 is category E.

[0093] The determination unit 2040 determines the category that transitions from category E in the state transition model of the insertion method X as the target category. In this sense, in Figure 7 the state transition model of the insertion method X shown, the destination of the transition from category E is category C. Therefore, the determination unit 2040 determines category C as the target category.

[0094] In the state transition model, multiple shape categories to which a transition can be made from one shape category can be defined. Therefore, in the state transition model of the insertion method used, there can be multiple shape categories to which a transition can be made from the last shape category of the shape data sequence 50. For example, in Figure 7 's example, as the shape categories to which a transition can be made from category C, there are three categories A, E, and G. Therefore, when the last shape category in the shape data sequence 50 is category C, there are three shape categories to which it can transition.

[0095] Therefore, for example, the determination unit 2040 regards multiple shape categories that are likely to transition from the last shape category of the shape data sequence 50 as candidates for the target category. For example, the determination unit 2040 determines to regard one of these candidate shape categories as the target category.

[0096] There are various methods for determining a target category from candidate shape categories. For example, in the state transition model, in a case where a transition from a specific state to multiple states can be made, it is assumed that a transition probability is assigned to each transition. In this case, for example, the determination unit 2040 determines the candidate shape category having the maximum transition probability as the target category.

[0097] In another example, similar to the case of using a target category recognition model to determine a target category, priorities can be determined for shape categories. In this case, as described above, the determination unit 2040 determines the shape category with the highest priority among the candidate shape categories as the target category.

[0098] Alternatively, instead of determining a single target category, all or some of the candidate shape categories can be determined as target categories. For example, in the case of regarding a part of the candidate shape categories as target categories, the determination unit 2040 determines as target categories the shape categories having a transition probability equal to or greater than a threshold value, the shape categories whose order of transition probability is less than or equal to a predetermined order, the shape categories whose order of priority is less than or equal to a predetermined order, and the like.

[0099] In addition, the determination unit 2040 can determine the priority order for the candidate shape categories. For example, a higher priority order can be assigned to the shape categories with higher transition probabilities, or a higher priority order can be assigned to the shape categories with higher priorities. The method of using the priority order is as described above.

[0100] <<<Method of considering changing the insertion method>>>

[0101] The insertion method for inserting the endoscopic observation instrument 40 can preferably be changed. For example, in the case where the state of the endoscopic observation instrument 40 is maintained for a long time, it is considered preferable to change to another insertion method. In another example, when an operator with a low skill level (e.g., a young doctor) uses a difficult insertion method that requires a high skill level, it is considered preferable to use a simpler insertion method.

[0102] Therefore, for example, the determination unit 2040 can determine whether the insertion method needs to be changed, and when the insertion method needs to be changed, a state transition model of the new insertion method can be used to determine the target category. For example, the determination unit 2040 determines whether the insertion method needs to be changed, and in the case of determining that the insertion method needs to be changed, determines another insertion method (hereinafter referred to as the changed insertion method) to be changed from the current insertion method. Then, the determination unit 2040 determines the target category by comparing the state transition model of the changed insertion method with the shape data sequence 50. The method of determining the target category using the state transition model and the shape data sequence 50 is as described above.

[0103] Predetermined conditions that have been determined in advance can be used to determine whether the insertion method needs to be changed. As the predetermined conditions, various conditions can be used. For example, the predetermined conditions are conditions such as "the residence time is equal to or greater than a threshold value" or "the technical level associated with the current insertion method is higher than the technical level defined for the operator of the endoscopic observation instrument 40". When the predetermined conditions are satisfied, it is determined that the insertion method needs to be changed. Meanwhile, when the predetermined conditions are not satisfied, it is determined that the insertion method does not need to be changed.

[0104] For example, the time during which the same insertion method continues, the time during which the transition cycle on the state transition model continues, etc. can be treated as the residence time. Here, for each insertion method, the time required to complete an insertion method may be different. Therefore, in the above insertion method definition information, the time that should be treated as the residence time threshold can be determined for each insertion method. In this case, when the currently used insertion method is determined, the determination unit 2040 acquires the threshold value of the residence time associated with the insertion method. Then, the determination unit 2040 determines whether the duration of the currently used insertion method is equal to or greater than the threshold value, and when the duration is equal to or greater than the threshold value, it is determined that the insertion method needs to be changed.

[0105] When the technical level is treated as a predetermined condition, each insertion method and the technical level required to use this insertion method are associated with each other in the insertion method definition information. The above operator information includes information indicating the technical level of each operator. Using the insertion method definition information and the operator information, the determination unit 2040 determines whether the technical level required for the currently used insertion method exceeds the technical level of the operator of the endoscopic observation instrument 40. Then, in the case where the technical level required for the currently used insertion method exceeds the technical level of the operator of the endoscopic observation instrument 40, the determination unit 2040 determines that the insertion method should be changed to another insertion method.

[0106] In the case where it is determined that the insertion method needs to be changed, the determination unit 2040 determines the changed insertion method. For example, in the insertion method definition information, another insertion method that can replace this insertion method is defined for each insertion method. For example, it is assumed that insertion methods A and B exist as insertion methods for advancing the endoscopic observation instrument from the first part in the body to the second part. In this case, insertion method B can be defined as an alternative insertion method for insertion method A. Similarly, insertion method A can be defined as an alternative insertion method for insertion method B. The determination unit 2040 uses the insertion method definition information to determine the insertion method associated with the currently used insertion method, and determines the determined insertion method as the changed insertion method.

[0107] However, when it is determined that the insertion method needs to be changed because the currently used insertion method exceeds the operator's skill level, it is considered not preferable to change the insertion method to a more difficult insertion method. Thus, for example, when it is determined that the skill level required for the currently used insertion method exceeds the skill level of the operator of the endoscopic observation instrument 40, the determination unit 2040 determines whether the skill level required for another insertion method associated with the currently used insertion method is lower than the skill level required for the currently used insertion method. When the skill level required for another insertion method associated with the currently used insertion method is lower than the skill level required for the currently used insertion method, the determination unit 2040 determines the other insertion method as the changed insertion method. Meanwhile, when the skill level required for another insertion method associated with the currently used insertion method is equal to or higher than the skill level required for the currently used insertion method, the determination unit 2040 determines not to change the insertion method.

[0108] Alternatively, instead of making such a determination, when another insertion method that can replace the insertion method is associated with the insertion method, the latter insertion method can always adopt an insertion method that can be used at a skill level lower than the skill level required for the former insertion method. Thus, when the insertion method associated with the currently used insertion method is obtained from the insertion method definition information, the skill level required for the obtained insertion method is necessarily lower than the skill level required for the currently used insertion method.

[0109] <<Combination of Multiple Methods>>

[0110] The determination unit 2040 can use both the target category recognition model and the state transition model of the currently used insertion method to determine the target category. For example, in the method of using the target category recognition model and the method of using the state transition model, the determination unit 2040 determines multiple shape categories as candidates for the target category. Then, the determination unit 2040 determines the shape categories that overlap between the candidates obtained by these two methods (the shape categories obtained as candidates for the target category by both methods) as the target category.

[0111] In another example, the determination unit 2040 treats multiple shape categories that the shape data sequence 50 in the state transition model of the currently used insertion method is likely to transition from the last shape category to as candidate shape categories. For each candidate shape category, the determination unit 2040 obtains a transition probability from the state transition model. In addition, the determination unit 2040 obtains a likelihood output from the target category recognition model for each candidate shape category. In addition, for each candidate shape category, the determination unit 2040 calculates a statistical value (such as an average value or a maximum value) of the transition probability obtained from the state transition model and the likelihood obtained from the target category recognition model. Then, the determination unit 2040 uses this statistical value to determine the target category. For example, the determination unit 2040 determines the shape category with the largest statistical value as the target category. In another example, the determination unit 2040 may treat any one of the candidate shape categories as the target category and perform priority setting based on this statistical value.

[0112] In another example, the determination unit 2040 may determine the target category in each of the target category recognition model and the state transition model. For example, assume that the target category determined using the target category recognition model is category C, and the target category determined using the state transition model is category E. In this case, the determination unit 2040 determines category C and category E as the target categories. In this case, the target information 20 preferably indicates the determination method and the target category determined by this determination method in an associated manner. For example, in the above example, by setting "recognition model: category C, state transition model: category E", etc., the association between the determination method and the target category determined by this determination method can be indicated.

[0113] <Method for determining the insertion method>

[0114] Now, a method for determining the currently used insertion method will be described. Various methods can be adopted for this method. For example, the determination unit 2040 uses the shape data sequence 50 to calculate the likelihood of the currently used insertion method for each of the multiple possible insertion methods. Then, the determination unit 2040 determines the currently used insertion method from the multiple insertion methods based on the likelihood calculated for each insertion method. Note that the method for calculating the likelihood will be described later.

[0115] Various methods can be adopted to determine the insertion method in use based on the likelihood calculated for each insertion method. For example, the determination unit 2040 determines the insertion method with the highest likelihood as the insertion method in use. In another example, the determination unit 2040 determines multiple candidates for the insertion method in use based on the likelihood calculated for each insertion method, and determines the insertion method in use from among the multiple candidates. For example, the determination unit 2040 determines the top n (n > 1) insertion methods with high likelihood as candidates for the insertion method in use. In another example, the determination unit 2040 determines the insertion methods with a likelihood equal to or greater than a threshold as candidates for the insertion method in use.

[0116] Various methods can be used as the method for determining the insertion method in use from among multiple candidates. For example, priorities are assigned in advance to each of the multiple insertion methods. In this case, for example, the determination unit 2040 determines the insertion method with the highest priority among the candidate insertion methods as the insertion method in use. In another example, the determination unit 2040 corrects the likelihood by multiplying the calculated likelihood by a weight based on the priority of each candidate insertion method. Note that the weight based on the priority is determined in advance as a value that becomes larger as the priority is higher. The determination unit 2040 determines the insertion method with the highest corrected likelihood as the insertion method in use.

[0117] In another example, for each of the multiple insertion methods, an important shape category or a transition between important shape categories (a permutation of two or more shape categories) is determined in advance. In this case, the determination unit 2040 determines as the insertion method in use the insertion method among the insertion methods determined as candidates, for which the important shape category or the transition between important shape categories defined for the insertion method is included in the shape data sequence 50.

[0118] For example, assume that insertion methods X and Y are determined as candidates based on likelihood. Additionally, assume that the important shape category defined for insertion method X is B, and the important shape category defined for insertion method Y is C. Then, assume that the time series of the shape data 52 indicated by the shape data sequence 50 is “A, C, D, C, A”. In this case, the shape data sequence 50 does not include the shape category B, which is the important shape category for insertion method X, but includes the shape category C, which is the important shape category for insertion method Y. Therefore, the determination unit 2040

[0119] determines insertion method Y as the insertion method in use.

[0120] Assume that there are multiple insertion methods in which important shape categories or transitions between shape categories as described above are included in the shape data sequence 50. In this case, for example, the determination unit 2040 determines the insertion method with a high calculated possibility among the multiple insertion methods as the insertion method being used. In another example, the determination unit 2040 may determine the insertion method in which more important shape categories or transitions between shape categories are included in the shape data sequence 50 as the insertion method being used.

[0121] The determination unit 2040 may use information related to the operator of the endoscope observation instrument 40 to determine the insertion method being used from among the candidate insertion methods. For example, in the operator information, the insertion methods that each operator can use are described. The determination unit 2040 determines the insertion method defined as the insertion method that can be used by the operator of the endoscope observation instrument 40 among the candidate insertion methods as the insertion method being used. Note that in the case where there are multiple candidate insertion methods defined as being usable by the operator of the endoscope observation instrument 40, for example, the determination unit 2040 determines the insertion method being used based on likelihood, the degree of the above-mentioned priority, and the like.

[0122] In another example, information indicating the technical level of the operator can be used as information related to the operator of the endoscope observation instrument 40. In this case, the technical level required for each insertion method is determined in the insertion method definition information. The determination unit 2040 determines the insertion method being used from among the multiple candidates based on the relationship between the technical level of the operator and the insertion method.

[0123] The determination unit 2040 may determine the insertion method being used from among the candidate insertion methods by using the image data obtained from the camera 10 provided in the endoscope observation instrument 40. For example, the determination unit 2040 extracts features from the image data obtained from the camera 10, and determines the insertion method that matches the feature among the candidate insertion methods as the insertion method being used. The features extracted from the image data include, for example, the phenomenon that the entire image turns red due to the tip of the endoscope observation instrument 40 pressing against the wall of the internal organ (referred to as a red ball), the characteristic movement of the screen, or the operation of the endoscope observation instrument 40.

[0124] The determination unit 2040 may calculate the likelihood of using each insertion method in each of multiple methods (for example, the method using an identification model and the method using a state transition model, which will be described later), and determine the insertion method being used based on the calculation results. For example, the determination unit 2040 calculates a statistical value (for example, an average value or a maximum value) of the likelihood calculated by each of the multiple methods for each insertion method, and determines the insertion method having the maximum statistical value as the insertion method being used.

[0125] <Specific method for calculating the possibility of the insertion method>

[0126] As described above, for example, the determination unit 2040 calculates the possibility of using each insertion method among a plurality of insertion methods in order to determine the insertion method being used. Hereinafter, as a method for calculating this possibility, two methods will be exemplified as specific examples.

[0127] <<Method using an identification model>>

[0128] In this method, the following identification model is used, which is trained to output the possibility that each insertion method is being used in response to the input sequence of shape data 50. Hereinafter, this identification model will be referred to as the insertion method identification model. For example, the insertion method identification model is implemented using an RNN, which is a neural network that processes time-series data. However, the model type used to implement the insertion method identification model is not limited to an RNN, but any type of model capable of processing time-series data can be used.

[0129] The training of the insertion method identification model is performed using training data including a combination of "data representing the temporal change in the shape of the endoscopic observation instrument and ground truth output data". The ground truth output data is, for example, data that represents the maximum possibility (e.g., 1) for the insertion method being used and the minimum possibility (e.g., 0) for other insertion methods.

[0130] Note that data other than the sequence of shape data 50 can be further input to the insertion method identification model. Examples of data other than the sequence of shape data 50 include still image data or video data obtained from the camera 10, data representing the movement of the operator of the endoscopic observation instrument 40 (e.g., time-series data of acceleration obtained from an acceleration sensor attached to the operator's hand), etc. Further, in the case where the sequence of shape data 50 represents the shape of the endoscopic observation instrument 40 by shape categories, the three-dimensional coordinates of each of a plurality of positions of the endoscopic observation instrument 40 can also be input to the insertion method identification model. When using these data, the same type of data is also included in the training data of the insertion method identification model. In this way, the insertion method identification model is trained to calculate the possibility by further using data other than the sequence of shape data 50.

[0131] Note that the insertion method recognition model can output a tag indicating the insertion method in use, rather than outputting the likelihood of each insertion method. In this case, for example, the insertion method recognition model outputs a tag indicating the insertion method with the highest calculated likelihood as the tag for the insertion method in use. The determination unit 2040 can determine the insertion method in use from the tag output by the insertion method recognition model.

[0132] <<Method Using the State Transition Model>>

[0133] In this case, for each of the multiple insertion methods, a state transition model representing the temporal change pattern of the shape of the endoscope observation instrument 40 when using that insertion method is determined in advance and stored in a storage device accessible by the determination unit 2040. When using this method, the likelihood that each insertion method is being used represents, for example, the degree of match between the state transition model of the insertion method and the temporal change of the endoscope observation instrument 40 represented by the shape data sequence 50. That is, for each of the multiple insertion methods, the determination unit 2040 calculates the degree of match between the state transition model of the insertion method and the shape data sequence 50, and determines the insertion method in use based on the calculation result. For example, the insertion method with the highest degree of match is determined as the insertion method in use.

[0134] In the case where the shape data 52 represents shape categories, for example, the likelihood is determined as follows for the degree of match between the state transition model and the shape data sequence 50.

[0135] Equation 1

[0136] Likelihood = Number of mutually matching shape categories / Unit time period length (1)

[0137] In Equation (1), the numerator on the right side represents the number of shape categories that match between the shape data sequence 50 and the state transition model. At the same time, the unit time period length in the denominator represents the total number of shape categories included in the above unit time period.

[0138] Figure 8 is a diagram conceptually showing the method of calculating the likelihood using Equation (1). In this example, the state transition model of insertion method X is compared with the shape data sequence 50. In this example, the unit time period length is 12.

[0139] The state transition model of insertion method X includes four shape categories A, C, E, and G. The shape data sequence 50 includes two shape categories A, two shape categories C, and two shape categories E among the shape categories that make up the state transition model of insertion method X. Thus, the number of matching shape categories between the state transition model of insertion method X and the shape data sequence 50 is 6. Therefore, according to Equation (1), 0.5(6 / 12) is calculated as the likelihood that insertion method X is being used.

[0140] The calculation formula for likelihood is not limited to Equation (1). For example, the following Equation (2) can be used to calculate the likelihood.

[0141] Equation 2

[0142] Likelihood = Number of mutually matching shape categories / Total number of types of shape categories included in the unit time period (2)

[0143]

[0144] For example, in Figure 8 the example, the shape data sequence 50 in the unit time period includes five types of shape categories A, B, C, D, and E. At the same time, there are three types of shape categories A, C, and E that match each other between the state transition model of insertion method X and the shape data sequence 50. Therefore, according to Equation (2), 0.6(3 / 5) is calculated as the likelihood that insertion method X is being used.

[0145] In addition, when calculating the likelihood of each insertion method, in addition to the degree of matching of the shape categories included in the unit time period, the likelihood can also be calculated by adding a calculation formula that increases the likelihood when following the transition path (sequential relationship) indicated in the state transition model or decreases the likelihood when not following the transition path, taking into account the order of the transitions.

[0146] <Output of target information 20: S108>

[0147] The output unit 2060 outputs the target information 20. There are various methods for outputting the target information 20. For example, the output unit 2060 displays a screen representing the target information 20 on the display device, and this screen can be viewed by the operator of the endoscopic observation instrument 40. In another example, the output unit 2060 can store a file representing the target information 20 in the storage device or send the file to another arbitrary device.

[0148] The target information 20 is various types of information that enable the target category to be grasped. For example, the target information 20 indicates the name of the target category and a photo or drawing representing the shape of the endoscopic observation instrument 40 belonging to the target category. Figure 1 ​In the example of , the target information 20 regarding category E of the target category indicates an image representing the name of "category E" and the shape of the endoscopic observation instrument 40 belonging to category E.

[0149] The target information 20 can be output together with information other than the information representing the target category. For example, the target information 20 is output together with the image from the camera 10 (video data obtained from the camera 10) and the information representing the shape category to which the current shape of the endoscopic observation instrument 40 belongs. Figure 9 FIG. is a diagram showing a screen including the target information 20. The screen 70 includes an area 72 representing the image from the camera 10 and an area 74 representing the information related to the shape of the endoscopic observation instrument 40. In the area 74, the shape category and the target category to which the current shape of the endoscopic observation instrument 40 belongs are shown.

[0150] Here, the target category included in the target information 20 is not limited to one. For example, the output unit 2060 can include information related to multiple candidates of the target category determined by the determination unit 2040. Figure 10 FIG. is a diagram showing the target information 20, which includes multiple shape categories determined as candidates for the target category. Figure 9 Similar to the screen 70 of , the screen 80 includes an area 82 representing the image of the camera 10 and an area 84 representing the information related to the shape of the endoscopic observation instrument 40.

[0151] In Figure 10 In the example of , the determination unit 2040 determines category C and category E as the target categories. Therefore, in the area 84, category C and category E are shown as the target categories. In addition, for each target category, the possibility calculated for the target category is indicated.

[0152] When multiple target categories are indicated, their display modes can be different from each other. For example, in the case where priorities can be set for multiple target categories, the target category with a higher priority order can be displayed in a more emphasized manner. As a method of emphasis, methods such as increasing the size or thickening the border can be adopted. Note that in the case of determining multiple target categories, the method of setting priorities for the target categories is as described above.

[0153] In addition to or instead of the target category, the endoscopic operation support device 2000 may output other information as information related to the operation to be applied to the endoscopic observation instrument 40. For example, the target information 20 may indicate a target position of an appropriate position as a transition destination for the endoscopic observation instrument 40. For example, the determination unit 2040 determines the target position of the endoscopic observation instrument 40 based on the target category. The target position of the endoscopic observation instrument 40 is, for example, the position where the end of the endoscopic observation instrument 40 should reach. In addition to or instead of outputting the target category, the output unit 2060 outputs the target information 20 indicating the target position of the endoscopic observation instrument 40.

[0154] Figure 11 FIG. is a diagram showing the target information 20, which includes the target position. In Figure 11 In the screen 80 of, in the region 84, a marker 86 indicating the target position is shown on the image representing the current shape category of the endoscopic observation instrument 40. In addition to this, Figure 11 the screen 80 of is the same as Figure 10 the screen 80 of.

[0155] The target position of the endoscopic observation instrument 40 can be determined based on the position of the endoscopic observation instrument 40 in the target category. For example, for the shape category with the highest possibility in the target category, the determination unit 2040 determines which part of the organ to be inserted (in this case, the large intestine) is the target position of the endoscopic observation instrument 40. For example, in Figure 11 In the example of, the part where the end of the endoscopic observation instrument 40 is located in the category C is the sigmoid colon. Therefore, the determination unit 2040 displays the marker 86 in the sigmoid colon part on the image representing the current situation of the endoscopic observation instrument 40.

[0156] Here, for each shape category, the position to be treated as the target position of the endoscopic observation instrument 40 (for example, the part of the organ to be inserted) is determined in advance. For example, for each shape category, the name of the shape category, an image representing the shape of the endoscopic observation instrument 40 in the shape category (for example, an image of the shape category for the screen 70, etc.), and information indicating the target position of the endoscopic observation instrument 40 (hereinafter referred to as category information) are stored in a storage device accessible from the endoscopic operation support device 2000. The endoscopic operation support device 2000 acquires the category information from the storage device and uses the category information to generate the target information 20.

[0157] The method for determining the target position is not limited to the method based on the target category. For example, an identification model (hereinafter referred to as the target position identification model) trained to output the target position in response to the input of data representing the temporal change of the shape of the endoscope observation instrument 40 can be used. The target position represents, for example, any one of a plurality of predetermined parts in the body. The determination unit 2040 determines the target position by inputting the shape data sequence 50 into the target position identification model.

[0158] Similar to the target category identification model, the target position identification model can be implemented by various models such as RNNs that can process temporal data. In addition, the target position identification model can be trained with training data configured by a combination of, for example, "data representing the temporal change of the shape of the endoscope observation instrument, the ground truth target position".

[0159] Note that the target position identification model can be configured to output the possibility of the target position for each predetermined position. In this case, the determination unit 2040 obtains the possibility of each predetermined position by inputting the shape data sequence 50 into the target position identification model, and determines the target position based on the possibility obtained for each predetermined position. For example, the determination unit 2040 treats the predetermined position with the highest possibility as the target position. In another example, the determination unit 2040 can use, as candidates for the target position, a plurality of predetermined positions whose possibility is equal to or greater than a threshold value or a plurality of predetermined positions whose order of possibility is less than or equal to a predetermined order, and determine the target position from the candidates. As a method for determining the target position from the candidate predetermined positions, a method similar to the method for determining the target category from the candidate shape categories can be used.

[0160] In addition, the determination unit 2040 can treat each candidate predetermined position among the candidate predetermined positions as the target position. At this time, a priority order can be determined for these predetermined positions. As a method for determining the priority order for the predetermined positions, a method similar to the method for determining the priority order for the candidate target categories can be used.

[0161] In the target information 20 in the previous example, the shape of the endoscope observation instrument 40 is represented by a plane. However, the target information 20 can represent the shape of the endoscope observation instrument 40 in a three-dimensional manner.

[0162] Figure 12It is a diagram showing target information 20, where the target information 20 represents the shape of the endoscopic observation instrument 40 in a three-dimensional image. The screen 90 includes an area 92 representing the video from the camera 10 and an area 94 representing the shape of the endoscopic observation instrument 40 as a three-dimensional image. In the area 94, the shape of the endoscopic observation instrument 40 is represented by the trajectory of the distal position of the endoscopic observation instrument 40. Further, a marker 96 indicating the target position of the endoscopic observation instrument 40 is shown in the area 94. Note that the three-dimensional coordinates of the specific position (e.g., the tip) of the endoscopic observation instrument 40 that should be set as the target position can be grasped by providing the above-described elements for determining the three-dimensional coordinates of each position of the endoscopic observation instrument 40 at that specific position.

[0163] The target information 20 can indicate the shape of the endoscopic observation instrument 40 corresponding to the target category in a three-dimensional manner. Figure 13 It is a diagram showing target information 20, in which the shape of the endoscopic observation instrument 40 corresponding to the target category is represented by a three-dimensional image. The screen 100 includes an area 102 representing the video from the camera 10 and an area 104 representing the shape of the endoscopic observation instrument 40 as a three-dimensional image. In the area 104, similar to Figure 12 the area 94, the shape of the endoscopic observation instrument 40 is represented by the trajectory of its tip position. Further, in the area 104, the shape of the endoscopic observation instrument 40 corresponding to the target category is represented by a three-dimensional image 106. In other words, the shape of the endoscopic observation instrument 40 to be changed is represented by the three-dimensional image 106.

[0164] Here, in order to generate the three-dimensional image 106, for each shape category, three-dimensional data (a set of three-dimensional coordinates) representing the shape of the endoscopic observation instrument 40 belonging to that shape category is required. Thus, for example, the three-dimensional data for each shape category is included in the above-described category information. The output unit 2060 acquires the three-dimensional data representing the shape of the endoscopic observation instrument 40 from the category information obtained for the target category determined by the determination unit 2040, and generates the three-dimensional image 106 using this three-dimensional data.

[0165] However, the three-dimensional coordinates of each position of the endoscopic observation instrument 40 in a specific shape may vary according to the body size of the person into whom the endoscopic observation instrument 40 is inserted (even when the shapes of the trajectories at the ends of the endoscopic observation instrument 40 are similar, they are not congruent). Thus, for example, the determination unit 2040 generates the three-dimensional image 106 of the target category by performing a similarity transformation of the shape represented by the three-dimensional data corresponding to the target category according to the size of the trajectory of the end position of the endoscopic observation instrument 40. For example, the determination unit 2040 acquires the three-dimensional data corresponding to the shape category to which the current shape of the endoscopic observation instrument 40 belongs, and calculates the ratio of the size of the trajectory of the endoscopic observation instrument 40 to the size of the shape represented by the pre-prepared three-dimensional data by comparing the three-dimensional data with the trajectory of the end of the endoscopic observation instrument 40. Then, the output unit 2060 generates the three-dimensional image 106 of the target category by performing a similarity transformation on the size of the shape represented by the three-dimensional data corresponding to the target category according to this ratio.

[0166] The target information 20 may further include information for determining the target category and the target position. For example, the name of the currently used insertion method and the diagram of the state transition model of the insertion method may be included in the target information 20. In addition, in the case where the determination unit 2040 determines that the insertion method needs to be changed, the target information 20 may include the name of the changed insertion method and the diagram of the state transition model of the changed insertion method.

[0167] Although the present invention has been described above with reference to the exemplary embodiments, the present invention is not limited to the above exemplary embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0168] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, or hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM, CD-R, CD-R / W, and semiconductor memories (e.g., mask ROM, PROM (programmable ROM), EPROM (erasable PROM), flash ROM, and RAM). In addition, the program can be provided to the computer through various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to the computer via wired communication paths such as wires and optical fibers or wireless communication paths.

[0169] Some or all of the above exemplary embodiments can be described as the following supplementary explanations, but are not limited to the following.

[0170] (Supplementary Explanation 1)

[0171] An endoscope operation support device, comprising:

[0172] An acquisition unit configured to acquire a sequence of shape data, the sequence of shape data representing a temporal change in the shape category of an endoscope observation instrument inserted into a body;

[0173] A determination unit configured to determine a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination of the shape category of the endoscope observation instrument, and the target position being a position suitable as a position where the endoscope observation instrument makes a transition;

[0174] An output unit configured to output target information, the target information being information related to the target category or the target position.

[0175] (Supplementary Explanation 2)

[0176] The endoscope operation support device according to Supplementary Explanation 1, wherein the determination unit determines the target category or the target position by inputting the sequence of shape data to a trained recognition model, the trained recognition model being configured to: in response to an input of a data sequence representing a temporal change in the shape of the endoscope observation instrument, output the target category or the target position.

[0177] (Supplementary Explanation 3)

[0178] The endoscope operation support device according to Supplementary Explanation 1, wherein the determination unit obtains, from a plurality of state transition models, a state transition model corresponding to the currently used insertion method, the plurality of state transition models representing temporal transition patterns of the shape category of the endoscope observation instrument defined for each of a plurality of insertion methods of the endoscope observation instrument, and the determination unit uses the obtained state transition model to determine the target category.

[0179] (Supplementary Explanation 4)

[0180] The endoscope operation support device according to Supplementary Explanation 3, wherein the determination unit determines the last shape category in the sequence of shape data from the obtained state transition model, and determines, as the target category, a shape category that is likely to transition from the determined shape category in the obtained state transition model.

[0181] (Supplementary Explanation 5)

[0182] The endoscopic operation support device according to Supplementary Explanation 3, wherein the determination unit performs:

[0183] Determine whether it is necessary to change the insertion method of the endoscopic observation instrument;

[0184] In the case where it is necessary to change the insertion method of the endoscopic observation instrument, determine the changed insertion method; and

[0185] Use the state transition model corresponding to the changed insertion method to determine the target category.

[0186] (Supplementary Explanation 6)

[0187] The endoscopic operation support device according to Supplementary Explanation 5, wherein the determination unit determines whether it is necessary to change the insertion method of the endoscopic observation instrument based on information related to the operator of the endoscopic observation instrument.

[0188] (Supplementary Explanation 7)

[0189] The endoscopic operation support device according to any one of Supplementary Explanations 1 to 6, wherein the determination unit performs:

[0190] Determine candidates for the target category or candidates for the target position; and

[0191] Based on the priorities determined for each shape category or each position, determine the target category or the target position from the multiple candidates.

[0192] (Supplementary Explanation 8)

[0193] The endoscopic operation support device according to any one of Supplementary Explanations 1 to 7, wherein the priority for each shape category or each position is determined based on the insertion method of the endoscopic observation instrument or information related to the operator of the endoscopic observation instrument.

[0194] (Supplementary Explanation 9)

[0195] The endoscopic operation support device according to any one of Supplementary Explanations 1 to 8, wherein the target information includes information indicating the current shape category of the endoscopic observation instrument.

[0196] (Supplementary Explanation 10)

[0197] The endoscopic operation support device according to Supplementary Explanation 9, wherein the target information includes an image of the interior of the body, an image representing the current shape of the endoscopic observation instrument, and an image representing the target position superimposed on the image of the interior of the body.

[0198] (Supplementary Explanation 11)

[0199] The endoscopic operation support device according to any one of Supplementary Explanations 1 to 10, wherein the target information indicates a plurality of candidates for the target category or the target position and the likelihood of each candidate being the target category or the target position.

[0200] (Supplementary Explanation 12)

[0201] A control method executed by a computer, the control method comprising:

[0202] An acquisition step of acquiring a sequence of shape data, the sequence of shape data representing a temporal change in the shape category of an endoscopic observation instrument inserted into a body;

[0203] A determination step of determining a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination for the shape category of the endoscopic observation instrument, and the target position being a position suitable as a position for the endoscopic observation instrument to transition;

[0204] An output step of outputting target information, the target information being information related to the target category or the target position.

[0205] (Supplementary Explanation 13)

[0206] The control method according to Supplementary Explanation 12, wherein in the determination step, the target category or the target position is determined by inputting the sequence of shape data to a trained recognition model, the trained recognition model being configured to output the target category or the target position in response to an input of a data sequence representing a temporal change in the shape of the endoscopic observation instrument.

[0207] (Supplementary Explanation 14)

[0208] The control method according to Supplementary Explanation 12, wherein in the determination step, a state transition model corresponding to the currently used insertion method is acquired from a plurality of state transition models, the plurality of state transition models representing temporal transition patterns of the shape category of the endoscopic observation instrument defined for each of a plurality of insertion methods of the endoscopic observation instrument, and the acquired state transition model is used to determine the target category.

[0209] (Supplementary Explanation 15)

[0210] The control method according to Supplementary Note 14, wherein in the determining step, the last shape category in the shape data sequence is determined from the obtained state transition model, and the shape category that is likely to transition from the determined shape category in the obtained state transition model is determined as the target category.

[0211] (Supplementary Note 16)

[0212] The control method according to Supplementary Note 14, wherein in the determining step,

[0213] it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument,

[0214] in the case where it is necessary to change the insertion method of the endoscopic observation instrument, the changed insertion method is determined, and

[0215] the state transition model corresponding to the changed insertion method is used to determine the target category.

[0216] (Supplementary Note 17)

[0217] The control method according to Supplementary Note 16, wherein in the determining step, it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument based on information related to the operator of the endoscopic observation instrument.

[0218] (Supplementary Note 18)

[0219] The control method according to any one of Supplementary Notes 12 to 17, wherein in the determining step,

[0220] candidates for the target category or candidates for candidates for the target position are determined; and

[0221] the target category or the target position is determined from the multiple candidates based on the priority determined for each shape category or each position.

[0222] (Supplementary Note 19)

[0223] The control method according to any one of Supplementary Notes 12 to 18, wherein the priority for each shape category or each position is determined based on the insertion method of the endoscopic observation instrument or information related to the operator of the endoscopic observation instrument.

[0224] (Supplementary Note 20)

[0225] The control method according to any one of Supplementary Notes 12 to 19, wherein the target information includes information indicating the current shape category of the endoscopic observation instrument.

[0226] (Supplementary Explanation 21)

[0227] The control method according to Supplementary Explanation 20, wherein the target information includes an image inside the body, and an image representing the current shape of the endoscopic observation instrument and an image representing the target position are superimposed on the image inside the body.

[0228] (Supplementary Explanation 22)

[0229] The control method according to any one of Supplementary Explanations 12 to 21, wherein the target information indicates a plurality of candidates for the target category or the target position and the possibility of each candidate being the target category or the target position.

[0230] (Supplementary Explanation 23)

[0231] A computer-readable medium storing a program, the program causing a computer to execute:

[0232] An acquisition step of acquiring a sequence of shape data, the sequence of shape data representing the temporal change of the shape category of an endoscopic observation instrument inserted into the body;

[0233] A determination step of determining a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination of the shape category of the endoscopic observation instrument, and the target position being a position suitable as a position for the endoscopic observation instrument to make a transition; and

[0234] An output step of outputting target information, the target information being information related to the target category or the target position.

[0235] (Supplementary Explanation 24)

[0236] The computer-readable medium according to Supplementary Explanation 23, wherein in the determination step, the target category or the target position is determined by inputting the sequence of shape data to a trained recognition model, and the trained recognition model is configured to: in response to the input of a data sequence representing the temporal change of the shape of the endoscopic observation instrument, output the target category or the target position.

[0237] (Supplementary Explanation 25)

[0238] The computer-readable medium according to Supplementary Note 23, wherein in the determining step, the state transition model corresponding to the currently used insertion method is obtained from a plurality of state transition models, the plurality of state transition models representing the time transition patterns of the shape categories of the endoscopic observation instrument defined for each of the plurality of insertion methods of the endoscopic observation instrument, and the obtained state transition model is used to determine the target category.

[0239] (Supplementary Note 26)

[0240] The computer-readable medium according to Supplementary Note 25, wherein in the determining step, the last shape category in the shape data sequence is determined from the obtained state transition model, and the shape category that is likely to transition from the determined shape category in the obtained state transition model is determined as the target category.

[0241] (Supplementary Note 27)

[0242] The computer-readable medium according to Supplementary Note 25, wherein in the determining step,

[0243] it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument;

[0244] In the case where it is necessary to change the insertion method of the endoscopic observation instrument, the changed insertion method is determined, and

[0245] the state transition model corresponding to the changed insertion method is used to determine the target category.

[0246] (Supplementary Note 28)

[0247] The computer-readable medium according to Supplementary Note 27, wherein in the determining step, it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument based on information related to the operator of the endoscopic observation instrument.

[0248] (Supplementary Note 29)

[0249] The computer-readable medium according to any one of Supplementary Notes 23 to 28, wherein in the determining step,

[0250] candidates for the target category or candidates for candidates for the target position are determined; and

[0251] the target category or the target position is determined from the plurality of candidates based on the priorities determined for each shape category or each position.

[0252] (Supplementary Note 30)

[0253] A computer-readable medium according to any one of Supplementary Notes 23 to 29, wherein priorities for each shape category or each position are determined based on an insertion method of the endoscopic observation instrument or information related to an operator of the endoscopic observation instrument.

[0254] (Supplementary Note 31)

[0255] A computer-readable medium according to any one of Supplementary Notes 23 to 30, wherein the target information includes information indicating a current shape category of the endoscopic observation instrument.

[0256] (Supplementary Note 32)

[0257] A computer-readable medium according to Supplementary Note 31, wherein the target information includes an image of the interior of the body, an image representing a current shape of the endoscopic observation instrument, and an image representing the target position superimposed on the image of the interior of the body.

[0258] (Supplementary Note 33)

[0259] A computer-readable medium according to any one of Supplementary Notes 23 to 32, wherein the target information indicates a plurality of candidates for the target category or the target position and a likelihood that each candidate is the target category or the target position.

[0260] (Supplementary Note 34)

[0261] A program that causes a computer to perform:

[0262] An acquisition step of acquiring a sequence of shape data that represents a temporal change in shape categories of an endoscopic observation instrument inserted into a body;

[0263] A determination step of determining a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination for the shape category of the endoscopic observation instrument, the target position being a position suitable as a position for the endoscopic observation instrument to transition; and

[0264] An output step of outputting target information that is information related to the target category or the target position.

[0265] (Supplementary Note 35)

[0266] The program according to Supplementary Note 34, wherein in the determining step, the target category or the target position is determined by inputting the shape data sequence into a trained recognition model, and the trained recognition model is configured to: output the target category or the target position in response to the input of a data sequence representing the temporal change in the shape of the endoscopic observation instrument.

[0267] (Supplementary Note 36)

[0268] The program according to Supplementary Note 34, wherein in the determining step, the state transition model corresponding to the currently used insertion method is obtained from a plurality of state transition models, the plurality of state transition models representing the temporal transition patterns of the shape categories of the endoscopic observation instrument defined for each of a plurality of insertion methods of the endoscopic observation instrument, and the obtained state transition model is used to determine the target category.

[0269] (Supplementary Note 37)

[0270] The program according to Supplementary Note 36, wherein in the determining step, the last shape category in the shape data sequence is determined from the obtained state transition model, and the shape category that is likely to transition from the determined shape category in the obtained state transition model is determined as the target category.

[0271] (Supplementary Note 38)

[0272] The program according to Supplementary Note 36, wherein in the determining step,

[0273] it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument,

[0274] in the case where it is necessary to change the insertion method of the endoscopic observation instrument, the changed insertion method is determined, and

[0275] the state transition model corresponding to the changed insertion method is used to determine the target category.

[0276] (Supplementary Note 39)

[0277] The program according to Supplementary Note 38, wherein in the determining step, it is determined whether it is necessary to change the insertion method of the endoscopic observation instrument based on information related to the operator of the endoscopic observation instrument.

[0278] (Supplementary Note 40)

[0279] The program according to any one of Supplementary Notes 34 to 39, wherein in the determining step,

[0280] Determine candidates for the multiple candidates of the target category or candidates for the target position; and

[0281] Based on the priorities determined for each shape category or each position, determine the target category or the target position from the multiple candidates.

[0282] (Supplementary Note 41)

[0283] According to the procedure described in any one of Supplementary Notes 34 to 40, wherein the priority for each shape category or each position is determined based on the insertion method of the endoscopic observation instrument or information related to the operator of the endoscopic observation instrument.

[0284] (Supplementary Note 42)

[0285] According to the procedure described in any one of Supplementary Notes 34 to 41, wherein the target information includes information indicating the current shape category of the endoscopic observation instrument.

[0286] (Supplementary Note 43)

[0287] According to the procedure described in Supplementary Note 42, wherein the target information includes an image of the interior of the body, an image representing the current shape of the endoscopic observation instrument, and an image representing the target position superimposed on the image of the interior of the body.

[0288] (Supplementary Note 44)

[0289] According to the procedure described in any one of Supplementary Notes 34 to 43, wherein the target information indicates multiple candidates for the target category or the target position and the likelihood of each candidate being the target category or the target position.

[0290] List of Reference Numerals

[0291] 10 Camera

[0292] 20 Target Information

[0293] 40 Endoscopic Observation Instrument

[0294] 50 Shape Data Sequence

[0295] 51 Time Point

[0296] 52 Shape Data

[0297] 60 Endoscopic Control Device

[0298] 70 Screen

[0299] 72 Region

[0300] 74 Region

[0301] 80 Screen

[0302] 82 Area

[0303] 84 Area

[0304] 86 Marker

[0305] 90 Screen

[0306] 92 Area

[0307] 94 Area

[0308] 96 Marker

[0309] 100 Screen

[0310] 102 Area

[0311] 104 Area

[0312] 106 3D Image

[0313] 500 Computer

[0314] 502 Bus

[0315] 504 Processor

[0316] 506 Memory

[0317] 508 Storage Device

[0318] 510 Input / Output Interface

[0319] 512 Network Interface

[0320] 2000 Endoscope Operation Support Device

[0321] 2020 Acquisition Unit

[0322] 2040 Determination Unit

[0323] 2060 Output Unit

Claims

1. An endoscope operation support device, comprising: an acquisition unit configured to acquire a sequence of shape data representing temporal changes in the shape categories of an endoscope observation instrument inserted into a body; a determination unit configured to determine a target category or a target position based on the sequence of shape data, the target category being suitable as a transition destination for the shape category of the endoscope observation instrument, and the target position being a position suitable as a position where the endoscope observation instrument undergoes a transition; and an output unit configured to output target information, the target information being information related to the target category or the target position.

2. The endoscope operation support device according to claim 1, wherein the determination unit determines the target category or the target position by inputting the sequence of shape data to a trained recognition model, the trained recognition model being configured to output the target category or the target position in response to an input of a data sequence representing temporal changes in the shape of the endoscope observation instrument.

3. The endoscope operation support device according to claim 1, wherein the determination unit obtains, from a plurality of state transition models, a state transition model corresponding to the currently used insertion method, the plurality of state transition models representing temporal transition patterns of the shape categories of the endoscope observation instrument defined for each of a plurality of insertion methods of the endoscope observation instrument, and the determination unit uses the obtained state transition model to determine the target category.

4. The endoscope operation support device according to claim 3, wherein the determination unit determines the final shape category in the sequence of shape data from the obtained state transition model, and determines the shape category that is likely to be transitioned from the determined shape category in the obtained state transition model as the target category.

5. The endoscope operation support device according to claim 3, wherein the determination unit performs: determining whether it is necessary to change the insertion method of the endoscope observation instrument; in the case where it is necessary to change the insertion method of the endoscope observation instrument, determining the insertion method after the change; and using the state transition model corresponding to the changed insertion method to determine the target category.

6. The endoscope operation support device according to claim 5, wherein the determination unit determines whether it is necessary to change the insertion method of the endoscope observation instrument based on information related to the operator of the endoscope observation instrument.

7. The endoscope operation support device according to any one of claims 1 to 6, wherein the determination unit performs: determining a plurality of candidates for the target category or a plurality of candidates for the target position; and determining the target category or the target position from the plurality of candidates based on the priorities determined for each shape category or each position.

8. The endoscopic operation support device according to any one of claims 1 to 6, wherein the priority of each shape category or each position is determined based on the insertion method of the endoscopic observation instrument or information related to the operator of the endoscopic observation instrument.

9. A control method executed by a computer, the control method comprising: an acquisition step of acquiring a shape data sequence representing a temporal change in the shape category of an endoscopic observation instrument inserted into the body; a determination step of determining a target category or a target position based on the shape data sequence, the target category being suitable as a transition destination of the shape category of the endoscopic observation instrument, and the target position being a position suitable as a position where the endoscopic observation instrument makes a transition; and an output step of outputting target information, the target information being information related to the target category or the target position.

10. A computer-readable medium storing a program that causes a computer to execute: an acquisition step of acquiring a shape data sequence representing a temporal change in the shape category of an endoscopic observation instrument inserted into the body; a determination step of determining a target category or a target position based on the shape data sequence, the target category being suitable as a transition destination of the shape category of the endoscopic observation instrument, and the target position being a position suitable as a position where the endoscopic observation instrument makes a transition; and an output step of outputting target information, the target information being information related to the target category or the target position.

Images