Learning data creation device, information support device, endoscope system, learning data creation method, and information storage medium

Abstract

Providing training data creation devices and other equipment that achieve both improved accuracy in machine learning and an increase in training data. [Solution] This embodiment relates to a learning data creation device 100 including a processor 110 that performs processing to generate learning data for machine learning. The processor 110 plots multiple endoscopic images of the learning device in a feature space based on individual features, determines a class for the multiple endoscopic images, and sets a set region for each determined class. The processor 110 also generates new learning data such that the number of newly generated learning data in the first region R1, which is the intersection region of the set regions for each class, is greater than the number of newly generated learning data in the second region R2, which is the region obtained by subtracting the first region from the union region of the set regions for each class.

Description

Technical Field

[0001] The present invention relates to a learning data creation device, an information support device, an endoscope system, a learning data creation method, and an information storage medium.

Background Art

[0002] In fields such as medicine, a method of automatically recognizing surgical scenes by machine learning is known. Since the required time is different for each surgical scene, when learning image data is created based on still images sampled at regular intervals from the moving images captured by an endoscope imager, the learning amount for each scene will not be uniform. Patent Document 1 discloses a method of additionally generating learning data by transforming existing captured image data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the method of Patent Document 1, there is a possibility that the accuracy of machine learning may not be sufficiently improved because learning data different from the original captured image data is created. Also, although it is conceivable to delete learning data in order to make the learning amount uniform, depending on the deletion method, the accuracy of machine learning may not be improved. Therefore, a method that achieves both an improvement in the accuracy of machine learning and an increase in learning data is desired.

Means for Solving the Problems

[0005] One aspect of the present disclosure relates to a learning data creation device including a processor that performs processing to generate learning data for machine learning, wherein the processor plots a plurality of endoscopic images of the learning device in a feature space based on individual features, determines a class for the plurality of endoscopic images, sets a set region for each determined class, and generates new learning data such that the number of newly generated learning data in a first region, which is the intersection region of the set regions for each class, is greater than the number of newly generated learning data in a second region, which is the region obtained by subtracting the first region from the union region of the set regions for each class.

[0006] Another aspect of the present disclosure relates to an information support device including a memory for storing a trained model that has been machine-learned using the training data generated by the training data creation device described above, and a processor, wherein the processor outputs results inferred from endoscopic images based on the trained model.

[0007] Other aspects of this disclosure relate to an endoscopic system including the information support device described above and an endoscope.

[0008] Another aspect of this disclosure relates to a method for creating learning data, which causes a computer to perform the following steps: plotting a plurality of endoscopic images of a learning device in a feature space based on individual features; determining a class for the plurality of endoscopic images; setting a set region for each determined class; and generating new learning data such that the number of newly generated learning data in a first region, which is the intersection region of the set regions for each class, is greater than the number of newly generated learning data in a second region, which is the union region of the set regions for each class excluding the first region.

[0009] Another aspect of this disclosure relates to a non-temporary information storage medium for storing a program to be executed by a computer as a step of generating new learning data such that the number of newly generated learning data in a first region, which is the intersection of the set regions for each class, is greater than the number of newly generated learning data in a second region, which is the region obtained by removing the first region from the union of the set regions for each class. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram illustrating an example of a system configuration including a learning data creation device and an endoscope system. [Figure 2] A diagram illustrating an example of an endoscopic system used in laparoscopic surgery. [Figure 3] A schematic diagram illustrating the arteries related to the large intestine. [Figure 4] A diagram illustrating the input and output data of a trained model. [Figure 5] A diagram illustrating an example of a procedure schedule for sigmoid colectomy. [Figure 6] A flowchart illustrating an example of a treatment flow applying the method of this embodiment. [Figure 7] A diagram illustrating an example of a given table. [Figure 8] A diagram explaining the timing of endoscopic image acquisition. [Figure 9] A flowchart illustrating an example of the processing method according to this embodiment. [Figure 10] A diagram illustrating an example of a feature space. [Figure 11] A diagram illustrating the set domains of the first and second classes. [Figure 12] A diagram illustrating the first and second regions. [Figure 13] A flowchart illustrating an example of the process involved in creating training data. [Figure 14] A flowchart for explaining a processing example related to the creation of an interpolated image. [Figure 15] (A) is a diagram for explaining a method of determining that the selected instance is located in the second region. (B) is a diagram for explaining a method of determining that the selected instance is located in the first region. [Figure 16] A flowchart for explaining a processing example related to the generation of an interpolated image based on a third instance. [Figure 17] A diagram for explaining a third instance, a fourth instance, and a fifth instance. [Figure 18] A diagram for explaining a method of determining the time related to a third instance. [Figure 19] A diagram for explaining another configuration example of the learning data creation device. [Figure 20] A flowchart for explaining another processing example related to the creation of an interpolated image. [Figure 21] A diagram for explaining the generation of an extended image. [Figure 22] A diagram for explaining another example of the method of this embodiment. [Figure 23] A diagram for explaining another example of the method of this embodiment.

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail. Note that the embodiments described below do not unduly limit the content described in the claims, and not all of the configurations described in the embodiments are essential constituent elements.

[0012] FIG. 1 is a block diagram for explaining a configuration example of a system including an endoscope system 1 and a learning device 90 of this embodiment. And it is a block diagram for explaining a configuration example of the endoscope system 1 including the learning data creation device 100. In FIG. 1, the learning device 90 includes the learning data creation device 100. The learning data creation device 100 includes a processor 110.

[0013] The processor 110 (hereinafter simply referred to as "processor 110") of the learning data creation device 100 of this embodiment is composed of the following hardware. The hardware may include at least one of a circuit that processes digital signals and a circuit that processes analog signals. For example, the hardware may consist of one or more circuit devices or one or more circuit elements mounted on a circuit board. One or more circuit devices are, for example, ICs. One or more circuit elements are, for example, resistors, capacitors, etc. Furthermore, for example, the learning data creation device 100 of this embodiment may be composed of a memory (not shown in Figure 1) and a processor 110 that operates based on the information stored in the memory. As a result, the processor 110 can function as a class determination unit 112, a data generation unit 114, etc. Note that, as will be described later in Figure 9 and beyond, the main component of the processing etc related to the method of this embodiment will be consistently referred to as the processor 110 for the sake of explanation. The information stored in the memory is, for example, a program and various data. The processor 110 can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), etc. The memory may be volatile memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory), or non-volatile memory such as ROM (Read Only Memory), or a magnetic storage device such as a hard disk drive, or an optical storage device such as an optical disk drive. For example, the memory stores instructions that can be read by the computer, and when these instructions are executed by the processor 110, the functions of each part are realized as processing. The instructions here may be instructions from an instruction set that makes up a program, or they may be instructions that instruct the hardware circuit of the processor 110 to operate.

[0014] Furthermore, although not shown in the diagram, the above-mentioned program can be stored in a non-temporary information storage medium, such as a computer-readable medium. This information storage medium can be implemented as, for example, an optical disc, a memory card, a hard disk drive, or non-volatile memory.

[0015] The learning device 90 includes, in addition to the learning data creation device 100, a processor (not shown, hereinafter conveniently referred to as the "learning device processor") and memory. The memory of the learning device 90 stores machine learning programs and training data, and the learning device processor functions as a machine learning unit to generate or update the trained model 22, which will be described later.

[0016] In Figure 1, the learning device 90 is configured to include the learning data creation device 100, but the learning device 90 may also function as the learning data creation device 100. In this case, the memory of the learning device 90 may further include a program related to the processing performed by the learning data creation device 100, and the learning device processor may function as a machine learning unit and a learning data generation unit.

[0017] The trained model 22 generated by the learning device 90 configured in this way is stored in the memory 20 of the information support device 10 by data transmission and reception via a communication interface (not shown). Note that the system example shown in Figure 1 is an example configuration at the training stage. When performing inference based on the trained model 22, the learning device 90 can be separated from the system shown in Figure 1, and only the endoscope system 1 can be used for actual endoscopic surgery, etc., as shown in Figure 2.

[0018] The endoscope system 1 of this embodiment includes an endoscope 3 and an information support device 10. The information support device 10 includes a memory 20 and a processor 30. The memory 20 stores a trained model 22. The processor 30 of the information support device 10 (hereinafter simply referred to as "processor 30") can be configured with hardware similar to the processor 110 included in the training data creation device 100 described above. Because the information support device 10 includes the memory 20 and the processor 30, the processor 30 functions as an inference unit. In other words, the memory 20 stores programs and the like that cause the processor 30 to function as an inference unit. The memory 20 is a non-temporary information storage medium that can be implemented by a disk, memory card, hard disk drive, or non-volatile memory.

[0019] Furthermore, the information support device 10 of this embodiment may further include an input unit 40 and an output unit 50, as shown in Figure 1. Although Figure 1 illustrates one input unit 40 and one output unit 50, there may be multiple types of input units 40 and output units 50. In this embodiment, for the sake of explanation, hardware that functions as an input interface, as described later, is collectively referred to as the input unit 40, and hardware that functions as an output interface is collectively referred to as the output unit 50.

[0020] Furthermore, although Figure 1 shows that one trained model 22 is stored in memory 20, the information support device 10 of this embodiment may include multiple types of trained models, and details will be described later in Figures 6 and 7. In other words, the processor 30 of this embodiment may be capable of simultaneously executing processes using multiple trained models. For the sake of explanation in this embodiment, the trained model that is trained by machine learning as described later in Figure 4 and is involved in the process of step S1 in Figure 6, which will be described later, will be referred to as "trained model 22" with a reference numeral.

[0021] In this embodiment, the endoscope 3 is, for example, a rigid endoscope in which the majority of the insertion portion is rigid. Although detailed illustration of the configuration is omitted because rigid endoscopes are well known, a configuration in which an imager is included at the tip of the endoscope 3 is widely adopted. In this way, the input unit 40 of the information support device 10 can receive an image signal from the imager via a cable (not shown), and the processor 30 can generate a display image based on the received image signal. As a result, by connecting the information support device 10 to the endoscope 3 and connecting the information support device 10 to the display shown in A1 of Figure 2, an endoscope system 1 can be constructed that displays the display image captured by the imager on the display via the output unit 50. This allows the user to perform procedures on biological tissue with instruments while observing the biological tissue in the body cavity during endoscopic surgery as shown in Figure 2. In this embodiment, the user refers to, for example, the surgeon handling the instruments, the scopist operating the endoscope 3, and all persons involved in the procedure.

[0022] In this embodiment, still images acquired by the imager at regular intervals of 1 hour in each case are called "endoscopic images." Furthermore, the collection of still images captured by the imager is conveniently referred to as "endoscopic video." In other words, the processor 30 in this embodiment acquires time-series endoscopic images during procedures using the endoscope 3. Specifically, for example, in this embodiment, when a video signal of an endoscopic image is input to the processor 30 from the imager via the input unit 40, the processor 30 performs a first process of generating an endoscopic image based on the received video signal and outputting the endoscopic image via the output unit 50, and a second process of using the generated endoscopic image as input data and performing inference based on the trained model 22. That is, although not specifically illustrated, the information support device 10 includes an input interface as hardware for inputting endoscopic image data to the trained model 22, and an output interface for outputting output data based on the inference results. This allows the user to perform procedures smoothly while observing the endoscopic images, as will be described later. The second process here corresponds to the process recognition (step S1) described later in Figure 6.

[0023] One example of a procedure to which the method of this embodiment can be applied is sigmoid colectomy. Sigmoid colectomy is a procedure aimed at curing cancer and other lesions in the sigmoid colon, shown as B1 in the large intestine schematically shown in Figure 3, by removing the sigmoid colon. In sigmoid colectomy, it is necessary to consider not only the sigmoid colon but also the handling of related tissues such as arteries, veins, and nerves. Related arteries include, for example, the abdominal aorta shown as B10 in Figure 3, the inferior mesenteric artery (hereinafter referred to as IMA) shown as B20, the left colic artery (hereinafter referred to as LCA) shown as B31, the sigmoid colic artery shown as B32, and the superior rectal artery (hereinafter referred to as SRA) shown as B33. Although not shown in the illustration, related veins include the inferior mesenteric vein (hereinafter referred to as IMV). How these tissues are treated is determined on a case-by-case basis. For example, if the lesion is cancerous, the extent of lymph nodes to be resected along with the sigmoid colon is determined according to the stage of cancer progression, and the prescribed lymph node dissection and vascular treatment are performed.

[0024] It should be noted that the endoscope 3 in this embodiment is not limited to the rigid endoscope described above, but may be other types of endoscopes such as flexible endoscopes. In other words, the method of this embodiment can be broadly applied to procedures using the endoscope 3. For example, although the following may describe the steps related to sigmoid colectomy using a rigid endoscope, the procedures to which the method of this embodiment can be applied are not limited to sigmoid colectomy, but can be broadly applied to procedures using the endoscope 3.

[0025] In this embodiment, machine learning refers to, for example, supervised learning, and the trained model 22 is generated by supervised learning based on a dataset that associates input data with correct labels. At least a portion of the trained model 22 in this embodiment includes a neural network. Although detailed illustrations are omitted, the neural network has an input layer into which data is input, an intermediate layer that performs calculations based on the output from the input layer, and an output layer that outputs data based on the output from the intermediate layer. The number of intermediate layers is not particularly limited. Also, the number of nodes included in each layer of the intermediate layer is not particularly limited. Nodes included in a given layer of the intermediate layer are connected to nodes in adjacent layers. A weighting coefficient is set for each connection. Each node multiplies the output of the preceding node by the weighting coefficient and obtains the sum of the multiplication results. Furthermore, each node adds a bias to the sum and applies an activation function to the sum result to obtain the output of that node. By sequentially executing this process from the input layer to the output layer, the output of the neural network is obtained. Various functions such as the sigmoid function and the ReLU function are known as activation functions, and these can be widely applied in this embodiment.

[0026] Learning in a neural network is the process of determining appropriate weighting coefficients. These weighting coefficients include bias. In the example shown in Figure 1, the learning device processor functions as the machine learning unit and performs the process of generating or updating the trained model 22. The processor 30 inputs the input data from the training data into the neural network and obtains an output by performing a forward operation using the weighting coefficients at that time. The processor 30 calculates an error function based on this output and the correct labels from the training data. Then, it updates the weighting coefficients to minimize the error function. For updating the weighting coefficients, for example, backpropagation can be used, in which the weighting coefficients are updated from the output layer to the input layer.

[0027] Various neural network configurations are known, and these can be broadly applied in this embodiment. For example, the neural network may be a CNN (Convolutional Neural Network), an RNN (Recurrent Neural Network), or any other model. When using a CNN or the like, the input data from the training data is input to the model, and the output is obtained by performing forward operations according to the model configuration using the weight coefficients at that time. An error function is calculated based on this output and the correct label, and the weight coefficients are updated to minimize this error function. When updating the weight coefficients of a CNN or the like, for example, backpropagation can be used.

[0028] During a single endoscopic surgical procedure performed by a given user, time-series endoscopic image data is acquired. For example, processor 30 uses the data extracted from the time-series endoscopic image data as input to a neural network.

[0029] The output of a neural network is information representing the stage of a procedure when, for example, the procedures in endoscopic surgery are classified into N stages. N is an integer greater than or equal to 2. For example, the output layer of a neural network has N nodes. The first node represents the likelihood that the procedure corresponding to the input endoscopic image belongs to class 1. The second to the Nth nodes are similar, with each node representing the likelihood that the input data belongs to class 2 to class N. For example, if the output layer is a known softmax layer, the N outputs are a set of probability data that sum to 1. Classes 1 to N correspond to procedures 1 to N, respectively.

[0030] Thus, the trained model 22 of this embodiment is generated by supervised learning based on a dataset that associates input data consisting of endoscopic images with correct labels representing process stages. It should be noted that the method of this embodiment does not preclude the application of other learning methods such as semi-supervised learning or self-supervised learning.

[0031] For example, in the sigmoid colectomy described above, the procedure is classified into stages as shown in Figure 5. In Figure 5, "Step 1" is "retrorectal dissection," "Step 2" is "medial mobilization before vascular treatment," "Step 3" is "vascular treatment," "Step 4" is "medial mobilization after vascular treatment," "Step 5" is "lateral mobilization," "Step 6" is "rectomesentery treatment," "Step 7" is "rectal transection," "Step 8" is "rectal anastomosis," and "Step 9" is "IMV treatment, LCA treatment." Note that the steps shown in Figure 5 are examples and are not limited to these. The "IMA treatment" in parentheses in "Step 3" is an example of "vascular treatment," and which blood vessels are treated in Step 3 is determined by the surgical procedure.

[0032] The trained model 22, which has undergone machine learning in this manner, can be applied to procedures including the processes shown in the flowchart in Figure 6, for example. In Figure 6, the processor 30 performs process recognition (step S1), and then sets whether to enable or disable the functions described later based on a predetermined table (step S2). The processor 30 then performs image recognition related to the enabled functions (step S3), overlays the recognition results onto the endoscopic image and displays them (step S4), and then determines whether the procedure is complete or not (step S5). If the processor 30 determines that the procedure is not complete (NO in step S5), it performs step S1 again, and if it determines that the procedure is complete (YES in step S5), it terminates the flow.

[0033] In step S1, the processor 30 reads the trained model 22 and infers the process stage based on the endoscopic image input from the endoscope 3. In other words, as described above in Figure 4, step S1 is a process in which the processor 30 functions as an inference unit.

[0034] In step S2, the processor 30 enables the functions necessary for the process related to the class inferred in step S1. Specifically, for example, a predetermined table shown in Figure 7 is stored in memory 20. Figure 7 is an example of a part of the predetermined table when the procedure is sigmoid colectomy. For example, if the processor 30 estimates that the class is 1 in step S1, the processor 30 enables the ureter recognition function, nerve recognition function, and SRA recognition function in step S2. The ureter recognition function here refers to a function that performs image processing on the endoscopic image displayed on the display so that the user can recognize the region where the ureter is located. The region where the ureter is located refers to the region where the ureter can be directly seen from the endoscopic image, as well as the region where the ureter is located when it is assumed to be located behind the tissue surface displayed in the endoscopic image. The same applies to the nerve recognition function, IMA recognition function, IMV recognition function, and SRA recognition function in Figure 7.

[0035] Image processing in the ureter recognition function may use a pre-trained model that has been machine-learned, such as the aforementioned CNN. In other words, a pre-trained model that has been machine-learned using a dataset in which endoscopic images are input data and the region where the ureter is located is output data may be stored in memory 20. As a result, in step S3, the processor 30 reads out the pre-trained model that has been machine-learned for the ureter recognition function and performs processing on the endoscopic image, such as semantic segmentation. Similarly, in step S3, the processor 30 reads out the pre-trained model that has been machine-learned for the nerve recognition function and the pre-trained model that has been machine-learned for the SRA recognition function and performs processing on the endoscopic image, such as semantic segmentation.

[0036] Then, in step S4, the processor 30 overlays and displays the segmented marker information for the ureter, nerve, and SRA onto the endoscopic image. Step S5 is NO until the procedure related to process 1 is completed, so steps S1 to S4 are repeated each time an endoscopic image is acquired.

[0037] Subsequently, the user completes the procedure related to step 1 and proceeds to the procedure related to step 2. As a result, the captured endoscopic image changes, and the processor 30 estimates that the endoscopic image acquired in step S1 is class 2. Note that intervals may be further included between each step in the table shown in Figure 5, and the processor 30 may infer from the endoscopic image that it is within the period related to the interval. Furthermore, the processor 30 may set the functions shown in Figure 6 to be disabled during the period related to the interval.

[0038] If the processor 30 infers process 2 from the endoscopic image in step S1, it enables the ureter recognition function, nerve recognition function, IMA recognition function, and IMV recognition function in step S2. In other words, it reads the trained models related to the ureter recognition function, the nerve recognition function, the IMA recognition function, and the IMV recognition function. Then, in step S3, the processor 30 performs processing such as semantic segmentation on the endoscopic image for the ureter, nerve, IMA, and IMV regions, and in step S4, it overlays and displays marker information for the ureter, nerve, IMA, and IMV on the endoscopic image. Similar processing is performed for processes 3 and beyond. Figure 7 shows that, as mentioned above in Figure 6, when "IMA processing" is performed as an example of "vascular processing", only the IMA recognition function is enabled.

[0039] In sigmoid colectomy, it is desirable to be able to determine the location of the ureter, nerves, IMA, IMV, SRA, etc. from the endoscopic image. However, if all of these tissues are segmented at every stage of the procedure, the user will be unable to perform the procedure smoothly. Therefore, it is desirable that the segmented tissues switch according to the stage of the procedure. However, if the user has to set the image processing settings each time at each stage of the procedure, the efficiency of the procedure will decrease. In this respect, by using the information support device 10 of this embodiment, when the stage of the procedure changes, the tissues targeted for image processing on the endoscopic image are automatically switched, so the user can perform the procedure efficiently. In order to improve the inference accuracy of the stages of such endoscopic surgery, it is desirable to be able to acquire a large amount of endoscopic images to be used as training data at each stage.

[0040] However, the time required for each process is not uniformly the same. For example, as shown in Figure 8, suppose that in a certain endoscopic surgical procedure, process (K-1) is performed between timing t1 and timing t2, process K is performed between timing t3 and timing t4, and process (K+1) is performed between timing t5 and timing t6. K is a natural number greater than or equal to 2. As mentioned above, since endoscopic images are acquired at regular intervals of 1 hour, the number of endoscopic images that can be acquired as training data is proportional to the time required for each process. In the case of Figure 8, for example, the ratio of the number of endoscopic images acquired as training data is process (K-1):process K:process (K+1)=5:2:6, so in order to improve the accuracy of learning, it is necessary to acquire more training data for process K, for example. This embodiment relates to a method for generating training data that leads to improved learning accuracy for processes such as process K, for which less training data has been acquired compared to other processes.

[0041] For example, when an endoscopic surgical procedure is performed, the time-series endoscopic image data (endoscopic video) related to that procedure is stored in memory (not shown) included in the learning device 90 in a directory-type file management structure having a hierarchical structure based on the procedure. The processor 110 then creates new training data for the multiple endoscopic images stored in the memory as training data using the method described below. The learning device processor then updates the directory to add the newly created training data. Alternatively, a text-type management file may be created based on the image file names of each endoscopic image stored in the memory (not shown) included in the learning device 90, and the user may manage this management file. The learning device processor may then update this management file to add text based on the image file names of the newly created training data. The user may also manage the acquired endoscopic images by associating them with the time information in which they were acquired. Specifically, for example, the endoscopic image data may be organized into a directory-type file management structure based on the procedure and time information, or a management file may be created that associates text names based on the image file names of the endoscopic images with text names based on the time information.

[0042] In the following explanation, we will use a process with limited training data as an example, but the methods described below can also be applied to creating more training data related to the interval period, for example. The periods shown in A11 and A12 in Figure 8 correspond to the interval period. The period shown in A11 is from timing t2, when process (K-1) is completed, to timing t3, when process K is started. Similarly, the period shown in A12 is from timing t4, when process K is completed, to timing t5, when process (K+1) is started.

[0043] Furthermore, while the following primarily describes methods for creating new training data in processes where the initially acquired training data is relatively small, unless otherwise specified, the methods of this embodiment may also be applied to processes where the acquired training data is relatively large.

[0044] An example of the processing method of this embodiment will be explained using the flowchart in Figure 9. After acquiring an endoscopic image (step S10), the processor 110 functions as a class determination unit 112 to perform class determination (step S20). Subsequently, the processor 110 functions as a data generation unit 114 to set a collection area (step S30) and then generates training data (step S100). Details of step S100 will be described later.

[0045] In step S10, more specifically, for example, the processor 110 acquires time-series endoscopic images for a predetermined case from the aforementioned database. The data acquired in step S10 includes time information on when the endoscopic images were acquired, and process stage information based on that time information.

[0046] In step S20, for example, the processor 110 determines the class of the acquired multiple endoscopic images based on acquired time information, process information, image similarity, etc. In the following, in order to facilitate understanding of the method of this embodiment, a class with a relatively large number of endoscopic images acquired as training data will be referred to as the first class, and a class with a relatively small number of endoscopic images acquired as training data will be referred to as the second class, and an example using these two classes, the first class and the second class, will be described. In the following, the relationship between the first class and the second class will be assumed to be based on the difference in the number of endoscopic images acquired as training data, and other relationships will not be considered unless otherwise specified. Other relationships include, for example, the relationship between the process related to the first class and the process related to the second class, which will be described in detail in Figure 22.

[0047] The processor 110 then sets up a set region (step S30). For example, the processor 110 extracts features from the endoscopic images that have been classified into classes in step S20, plots the extracted features in the feature space, and associates the classified classes with these features. The features here are also called feature vectors. The features are, but are not limited to, edges, gradients, or their statistics in the endoscopic image. For feature extraction, methods such as SIFT (Scale Invariant Feature Transform), SURF (Speeded-Up Robust Features), and HOG (Histograms of Oriented Gradients) can be used, but features may also be extracted from the endoscopic image using machine learning methods. Specifically, machine learning methods here include CNNs, but methods that are further developments of CNNs, such as VGG and ResNET, may also be used.

[0048] In the following explanation, to facilitate understanding of the method of this embodiment, the feature quantity will be assumed to be a two-dimensional quantity consisting of feature quantity A and feature quantity B. However, the method of this embodiment is not limited to this, and the dimension of the feature quantity can be extended to three or more. Alternatively, the method of this embodiment may include processing to reduce the multidimensional feature space to a lower dimension such as two or three dimensions using methods such as Principal Component Analysis, so that the relationships between classes are easier for the user to understand. Specifically, for example, in the feature space shown in Figure 10, instances related to endoscopic images classified as the first class (hereinafter referred to as "instances belonging to the first class") are represented by white circles. Similarly, in the feature space shown in Figure 10, instances related to endoscopic images classified as the second class (hereinafter simply referred to as "instances belonging to the second class") are represented by black circles.

[0049] Furthermore, in step S30 of Figure 9, the processor 110 sets up a set region based on the set of instances associated with each class. The shape of the set region is not particularly limited, but in this embodiment, as shown in A21 of Figure 11, the processor 110 sets up a set region in the shape of a convex polygon that includes the set of instances belonging to the first class. Similarly, as shown in A22 of Figure 11, the processor 110 sets up a set region in the shape of a convex polygon that includes the set of instances belonging to the second class. In other words, step S30 of this embodiment includes the process of finding the convex hull for each of the sets of instances belonging to the first class and the sets of instances belonging to the second class. The method of using the convex hull is not particularly limited, and a wide range of known convex hull formation algorithms can be adopted.

[0050] Furthermore, in step S30 of Figure 9, the processor 110 may set the set region by further considering, for example, the distance in the feature space of the endoscopic images. Specifically, the distance here refers to the distance from the centroid of the plotted positions of the endoscopic images in the feature space to the plotted position of each endoscopic image. For example, the centroid of the plotted positions of the endoscopic images in the feature space can be determined by using a method similar to the k-means method. Note that the k-means method is well known, so a detailed explanation will be omitted. For convenience, the distance from the centroid of the plotted positions of the endoscopic images in the feature space to the plotted position of each endoscopic image is called the first distance. The first distance may differ by the number of plotted endoscopic images.

[0051] Furthermore, for example, in step S30 of Figure 9, the processor 110 may set a set region by further setting a second distance. The second distance is, for example, a fixed distance in the feature space set for each class. For example, the processor 110 compares the first distance and the second distance for each instance belonging to the same class, and if the first distance is shorter than the second distance for all instances, it can set a set region for all instances. Alternatively, for example, if there are instances where the first distance is longer than the second distance, the processor 110 can exclude those instances and set a set region for the instances that have not been excluded. In other words, the second distance is a distance that serves as a criterion for allowing an instance to belong to a set region.

[0052] In the following, as shown in Figure 12, the intersection of the two sets of domains will be called the first domain R1, and the region obtained by excluding the first domain R1 from the union of the two sets of domains will be called the second domain R2. Furthermore, if necessary, the region of the second domain R2 consisting of the set of instances belonging to the first class will be distinguished and called "the first second domain R2-1," and the region of the second domain R2 consisting of the set of instances belonging to the second class will be distinguished and called "the second second domain R2-2." When there is no need to distinguish between "the first second domain R2-1" and "the second second domain R2-2," they will be collectively referred to simply as "the second domain R2." It is not easy to determine the overlap between the first and second classes based on the set of instances (set of points), but as described above, by using the convex hull, the first domain R1 can be set, making it easy to grasp the overlap between the first and second classes.

[0053] In this embodiment, new training data is generated by step S100, which will be described later. The training data is generated such that the features related to the newly generated training data are more often located in the first region R1 than in the second region R2. Note that "the features related to the newly generated training data are more often located in the first region R1 than in the second region R2" may include cases where the features related to the newly generated training data are not located in the second region R2. In other words, in this embodiment, the number of newly generated training data located in the second region R2 may be zero, and all the features related to the newly generated training data may be located in the first region R1. Details will be described later. Based on the above, this embodiment relates to a training data creation device 100 including a processor 110 that performs processing to generate training data for machine learning. The processor 110 plots multiple endoscopic images of the learning device in the feature space based on individual features, determines a class for the multiple endoscopic images, and sets a set region for each determined class. Furthermore, the processor 110 generates new training data such that the number of training data newly generated in the first region R1, which is the intersection region of the sets of each class, is greater than the number of training data newly generated in the second region R2, which is the region obtained by subtracting the first region from the union region of the sets of each class.

[0054] As described above, the training data creation device 100 of this embodiment determines a class based on the endoscopic images which are the training data, and plots the features based on the endoscopic images in the feature space, thereby setting a set region for each determined class within the feature space. Furthermore, since more training data is generated in the first region R1 than in the second region R2, more training data based on features belonging to different classes can be generated in regions with similar features. This allows for more machine learning based on endoscopic images with similar features, so that more accurate classification results can be output when endoscopic images are input during the inference stage after machine learning. This makes it possible to construct a training data creation device 100 that achieves both improved machine learning accuracy and an increase in training data.

[0055] For example, if the features of the endoscopic image related to process K and the endoscopic image related to process (K+1) are similar, as mentioned above in Figure 8, there is little training data for the endoscopic image of process K. Therefore, even if the endoscopic image that should be classified as process K is input and inference is performed, there is a possibility that it will be incorrectly classified as another process, such as process (K+1). Furthermore, even if additional training is performed by increasing the training data as in conventional methods, it is not guaranteed that the training data with appropriate features will increase, so the accuracy of classification at the inference stage will not necessarily improve. In this respect, by applying the method of this embodiment, it is possible to generate a large amount of training data in the first region R1 where the features are similar in the two classes, so that the number of instances increases. As a result, by performing inference using the trained model 22 that has been machine-learned with such training data, the accuracy of endoscopic image classification can be reliably improved.

[0056] Furthermore, the method of this embodiment may be implemented as an information support device 10. That is, this embodiment relates to an information support device 10 including a memory 20 for storing a trained model 22 that has been machine-learned using the training data generated by the training data creation device 100 described above, and a processor 30. The processor 30 outputs the results inferred from the endoscopic image based on the trained model 22. By doing so, the same effects as described above can be obtained.

[0057] Furthermore, the method of this embodiment may be implemented as an endoscope system 1. That is, the endoscope system 1 of this embodiment includes the information support device 10 described above and the endoscope 3. By doing so, the same effects as described above can be obtained.

[0058] Furthermore, the method of this embodiment may also be implemented as a method for creating training data. That is, the training data creation method of this embodiment involves having a computer perform the following steps: plotting multiple endoscopic images from a learning device in a feature space based on individual features; determining a class for the multiple endoscopic images; and setting a set region for each determined class. The training data creation method also involves having the computer perform a step of generating new training data such that the number of newly generated training data in the first region R1, which is the intersection region of the set regions for each class, is greater than the number of newly generated training data in the second region R2, which is the region obtained by subtracting the first region R1 from the union region of the set regions for each class. By doing so, the same effect as described above can be obtained.

[0059] Furthermore, the method of this embodiment may be implemented as a non-temporary information storage medium that stores a program. That is, the non-temporary information storage medium of this embodiment stores a program that causes a computer to execute the following steps: plotting multiple endoscopic images of a learning device in a feature space based on individual features; determining a class for the multiple endoscopic images; and setting a set region for each determined class. The non-temporary information storage medium of this embodiment also stores a program that causes a computer to execute the step of generating new learning data such that the number of newly generated learning data in the first region R1, which is the intersection region of the set regions for each class, is greater than the number of newly generated learning data in the second region R2, which is the region obtained by removing the first region R1 from the union region of the set regions for each class. By doing so, the same effects as described above can be obtained.

[0060] Furthermore, in the training data creation device 100 of this embodiment, the processor 110 may set up a set region for each class by forming a convex hull. In this way, it is possible to easily set up the first region R1, which is the overlapping region of the set regions of each class in the feature space.

[0061] Furthermore, in the training data creation device 100 of this embodiment, the processor 110 may generate training data for the second class set region in the first region R1, where the first class set region and the second class set region, which has less training data than the first class, overlap. By doing so, a large amount of training data related to features belonging to the second class can be generated in the first region R1, which is a region with similar features. As a result, the trained model 22 can improve the accuracy of classifying endoscopic images that should be classified as the second class into the second class as output data when they are input as input data during the inference stage.

[0062] Furthermore, in the learning data creation device 100 of this embodiment, the processor 110 may set a set region based on a first distance, which is a distance in the feature space, and is the distance from the centroid of the feature quantities of multiple endoscope images belonging to the same class to the feature quantities of each endoscope image. By doing so, the set region can be set appropriately.

[0063] Furthermore, in the learning data creation device 100 of this embodiment, the processor 110 may further set a second distance for each class, compare the first distance and the second distance for each instance of the endoscopic image, and set a set region for the set of instances where the first distance is shorter than the second distance. By doing so, the set region can be set more appropriately.

[0064] Figure 13 is a flowchart that explains step S100 in Figure 9 in more detail. In Figure 13, the processor 110 generates an interpolated image (step S110) and determines whether the number of instances belonging to the first class is the same as the number of instances belonging to the second class (step S190). The details of step S110 will be described later in Figure 14. The processor 110 then terminates the flow if the number of instances belonging to the first class is the same as the number of instances belonging to the second class (YES in step S190). On the other hand, the processor 110 performs step S110 again if the number of instances belonging to the first class is different from the number of instances belonging to the second class (NO in step S190). In other words, as will be described later, step S110 is a process that increases the number of instances belonging to the second class, and step S110 is repeated until the number of instances belonging to the second class is equal to the number of instances belonging to the first class.

[0065] The criterion for step S190 is that the number of instances belonging to the first class is the same as the number of instances belonging to the second class, but it is not limited to this. For example, if the number of instances belonging to the second class is within a predetermined percentage range of the number of instances belonging to the first class, the processor 110 may determine YES in step S190. The predetermined percentage can be appropriately determined according to the accuracy of the inference obtained as a result of the learning process.

[0066] Step S110 will be explained in more detail using the flowchart in Figure 14. The processor 110 arbitrarily selects an instance belonging to the first class (step S111). Hereafter, the instance belonging to the first class selected by the process in step S111 will be called the first instance. The processor 110 determines whether or not the first instance is located in the first second region R2-1 (step S112). If the processor 110 is not located in the first second region R2-1 (NO in step S112), it repeats step S111. On the other hand, if the processor 110 is located in the first second region R2-1 (YES in step S112), it arbitrarily selects an instance belonging to the second class (step S121). Hereafter, the instance belonging to the second class selected by the process in step S112 will be called the second instance. The first and second instances play a role in determining the location of the third instance, as will be described later in Figure 17.

[0067] The processor 110 then determines whether the second instance is located in the first region R1 (step S122). If the processor 110 determines that the second instance is not located in the first region R1 (NO in step S122), it repeats step S121. On the other hand, if the processor 110 determines that the second instance is located in the first region R1 (YES in step S122), it generates an interpolated image based on the third instance (step S130).

[0068] Thus, in the learning data creation device 100 of this embodiment, the set region for each class includes a set region for the first class and a set region for the second class. The processor 110 arbitrarily selects a first instance from the set region for the first class and arbitrarily selects a second instance from the set region for the second class. The processor 110 also generates learning data if the selected first instance is included in the first second region R2-1, which is the set region obtained by removing the first region R1 from the set region for the first class, and the selected second instance is included in the first region R1. In this way, the positions of the first and second instances can be appropriately set. This makes it possible to determine the position of the instance (third instance, described later) that is related to a suitable interpolated image.

[0069] In step S112, whether or not the first instance is located in the first second region R2-1 can be determined by, for example, the following method. For example, in Figure 15(A), the convex hull shown in A30 is the set region of the first class, and the first instance selected in step S111 is located at the position shown in A31. Then the processor 110 draws the search vector shown in A32. The search vector is a vector that starts from the first instance shown in A31 and searches the outer perimeter line of the set region of the first class. In Figure 15(A), the direction of the search vector is shown as parallel to the right direction of the paper, but it may also be to the left, upward, or downward. In other words, the direction of the search vector may be parallel to either the horizontal or vertical direction. The search vector shown in A32 can also be considered as a half-line with the first instance shown in A31 as its end.

[0070] Alternatively, one axis in the feature space (hereinafter referred to as the "first axis" for convenience) may be arbitrarily selected, and the search vector may be set in a direction parallel to the selected first axis. In Figure 15(A), the horizontal direction of the paper is assumed to be parallel to the direction along the first axis selected in the feature space shown in Figure 10, etc., and the vertical direction of the paper is assumed to be parallel to the direction along the second axis, which is arbitrarily selected from among the axes perpendicular to the first axis. The same applies to Figure 15(B), which will be described later.

[0071] The processor 110 then determines how many times the search vector shown at A32 in Figure 15(A) intersects with the outer perimeter line of the first class set region. Specifically, at the location shown at A33, the search vector intersects with the outer perimeter line of the first class set region once, so it can be determined that the first instance shown at A31 is located inside the first class set region. In other words, the processor 110 determines that the first instance is not located inside the first class set region if the search vector does not intersect with the outer perimeter line of the first class set region even once, or if the search vector intersects with the outer perimeter line of the first class set region two or more times.

[0072] In step S112, the processor 110 also determines how many times the search vector shown in A32 intersects the outer perimeter line of the second class set region. Although not shown in Figure 15(A), the processor 110 determines that the first instance is not located inside the second class set region if the search vector does not intersect the outer perimeter line of the second class set region even once, or if the search vector intersects the outer perimeter line of the first class set region two or more times. If the first instance is located inside the first class set region and not inside the second class set region, the processor 110 determines YES in step S121 because the first instance is located in the first second region R2-1.

[0073] In step S122, the method described above in step S112 can also be used to determine whether the second instance is located in the first region R1. For example, in Figure 15(B), suppose the convex hull shown in A40-1 is the region of the first class set, the convex hull shown in A40-2 is the region of the second class set, and the second instance selected in step S112 is located at the position shown in A41. Then the processor 110 draws the search vector shown in A42. The search vector intersects the outer perimeter line of the first class set region once at the location shown in A43, and the search vector intersects the outer perimeter line of the second class set region once at the location shown in A44. From this, it can be determined that the first instance shown in A41 is located inside the first region R1.

[0074] Furthermore, as described above in Figure 15(A), the search vector shown at A42 in Figure 15(B) can also be considered as a half-line with the second instance shown at A41 as its endpoint. Based on the above, in the training data creation device 100 of this embodiment, the processor 110 virtually sets a half-line parallel to an arbitrarily selected axis in the feature space and with the selected second instance as its endpoint. The processor 110 also determines that the second instance exists inside the first region R1 and generates training data when the set half-line intersects once with the outer perimeter of the first class set region and once with the outer perimeter of the second class set region. In this way, it is easy to determine whether the first and second instances are located within suitable regions in the feature space.

[0075] Step S130 will be explained in more detail using the flowchart in Figure 16. The processor 110 sets up the third instance based on the first and second instances (step S132), and then sets up the fourth and fifth instances (step S134). Then the processor 110 sets the time related to the third instance (step S136), and acquires the endoscopic image based on the determined time (step S138).

[0076] For example, as shown in Figure 17, the first region R1, the first second region R2-1, and the second second region R2-2 are partitioned by the superposition of a part of the convex hull of the first class of set region and a part of the convex hull of the second class of set region. Then, assume that the first instance shown in A51 is selected by step S111 in Figure 14. In this case, since the first instance shown in A51 in Figure 17 is located inside the first second region R2-1, the processor 110 determines YES in step S112 in Figure 14. Similarly, assume that the second instance shown in A52 in Figure 17 is selected by step S121 in Figure 14. In this case, since the second instance shown in A52 is located inside the first region R1, the processor 110 determines YES in step S122 in Figure 14.

[0077] Then, in step S132, the processor 110 sets a third instance at the position shown at A53 in Figure 17. Step S132 is the process of determining the position of the newly interpolated feature. The position shown at A53 is on the line segment connecting the position of the first instance shown at A51 and the position of the second instance shown at A52, but is set to be inside the first region R1. For example, the processor 110 may control the probability of setting the third instance in the first region R1 to be higher than the probability of setting the third instance in the first second region R2-1. Also, the position shown at A53 is set to be near the boundary line between the first region R1 and the first second region R2-1, and also inside the first region R1. In other words, even if the feature space is a nonlinear space, the third instance is set so that the feature space can be treated locally as a linear space. This makes the set region of the second class clearer. To determine whether linearity holds within a desired range of the feature space, one can, for example, randomly select several instances within that range and check whether additivity and homogeneity hold for the selected instances.

[0078] Subsequently, the processor 110 sets up the fourth instance shown at A54 and the fifth instance shown at A55 in Figure 17 by step S134 in Figure 16. The fourth instance shown at A54 and the fifth instance shown at A55 are instances belonging to the second class that are based on the feature closest to the position shown at A53. As mentioned above, the endoscopic images in this embodiment are images acquired at regular intervals of 1 hour based on endoscopic video, so the time when the endoscopic image based on the fourth instance shown at A54 was acquired and the time when the endoscopic image based on the fifth instance shown at A55 was acquired are known. Similarly, the time when the endoscopic image based on the second instance shown at A52 was acquired is also known.

[0079] Step S136 will be explained in more detail. For example, as shown in Figure 18, suppose the endoscopic image for the second instance is acquired from the endoscopic video at timing t12, the endoscopic image for the fourth instance is acquired from the endoscopic video at timing t14, which is later than timing t12, and the endoscopic image for the fifth instance is acquired from the endoscopic video at timing t15, which is later than timing t14. Note that Figure 18 is an example and does not limit the temporal order of timings t12, t14, and t15.

[0080] In step S136, for example, the processor 110 generates time information according to a uniform random number for a time width of a second time unit as the minimum unit, from timing t12 to timing t15. The calculation of the uniform random number can be done using a known method. It is desirable that the second time is shorter than the first time mentioned above. More specifically, for example, if endoscopic images are acquired every 2 seconds (= first time), in step S136, the processor 110 generates time information according to a uniform random number for a time width of a second time unit as the minimum unit, from timing t12 to timing t15. As a result, for example, the time shown at A60 in Figure 18 is determined as the time based on the newly created third instance. In this way, new training data that is considered valid as endoscopic images relating to features interpolated in the feature space can be created.

[0081] Then, in step S138 of Figure 16, the processor 110 acquires an endoscopic image based on the time shown at A60 in Figure 18. This interpolates the training data based on the third instance.

[0082] In step S136, the processor 110 may generate time information according to a uniform random number for the period from timing t14 to timing t15, with the second time being the smallest unit. This is because the third instance is located in the nearest neighbor to the fourth and fifth instances in the feature space, and therefore the time related to the third instance is expected to be close to that of the fourth and fifth instances.

[0083] Furthermore, if the third instance is set inside the first second region R2-1, the fourth and fifth instances should be set as instances belonging to the first class, based on the feature closest to the position of the set third instance. Then, the processor 110 should obtain the time related to the third instance based on the times related to the first, fourth, and fifth instances. Thus, in the learning data creation device 100 of this embodiment, the processor 110 creates learning data corresponding to the interpolated feature by interpolating a new feature inside the first region R1 based on the acquisition times of endoscopic images related to multiple instances. In this way, the acquisition time related to the endoscopic image related to the newly interpolated feature can be estimated within a reasonable range. This makes it possible to create new endoscopic images as appropriate learning data.

[0084] In this way, by applying the method of this embodiment, new training data based on instances belonging to the second class can be created. In the sigmoid colectomy described above, for example, step 3 (vascular processing) in Figure 5 takes less time than steps 2 and 4, which are preparation and post-processing steps for processing blood vessels, and therefore less training data is obtained by endoscopic surgery. Therefore, it is conceivable to apply the method of this embodiment to step 3 as the second class. In other words, in the training data creation device 100 of this embodiment, the processor 110 arbitrarily selects a second instance from the set region of the second class, which is the class related to the process of processing blood vessels in sigmoid colectomy. By doing so, the trained model 22 can improve the accuracy of classifying the process of processing blood vessels and outputting it as output data when endoscopic images are input as input data in sigmoid colectomy.

[0085] Furthermore, for example, step 7 (rectal resection) in Figure 5 takes less time than steps 6 (rectomesentery processing) and 8 (rectal anastomosis), which are preparation and post-processing steps for resection, so less learning data is obtained through endoscopic surgery. Therefore, it is conceivable to apply the method of this embodiment to step 7 as the second class. In other words, in the learning data creation device 100 of this embodiment, the processor 110 arbitrarily selects a second instance from the set region of the second class, which is the class related to the step of resectioning the rectum in sigmoid colectomy. By doing so, when endoscopic images in sigmoid colectomy are input as input data, the accuracy of classifying the step of resectioning the rectum and outputting it as output data can be improved.

[0086] Furthermore, the learning data creation device 100 of this embodiment may be configured as shown in the example in Figure 19. The learning data creation device 100 shown in Figure 19 differs from the learning data creation device 100 shown in Figure 1 in that it further includes a data expansion unit 116. In other words, the learning data creation device 100 of this embodiment may further store a program in which the processor 110 functions as the data expansion unit 116 in a memory (not shown) or the like.

[0087] Furthermore, when using the learning data creation device 100 shown in Figure 19, the flowchart shown in Figure 13 may be modified to resemble the flowchart shown in Figure 20. The flowchart in Figure 20 differs from the flowchart in Figure 13 in that a process (step S150) in which the processor 110 generates an augmented image is added between steps S110 and S190.

[0088] Step S150 more specifically performs processing on the endoscopic image acquired in step S10 of Figure 9, such as scaling, shearing, horizontal inversion, vertical inversion, and rotation. Here, scaling means enlarging or reducing the endoscopic image within a specified range. Shearing means transforming a rectangular endoscopic image into a parallelogram. Horizontal inversion means inverting the endoscopic image using a line passing through the center of the endoscopic image and parallel to the vertical direction of the endoscopic image as a reference. Vertical inversion means inverting the endoscopic image using a line passing through the center of the endoscopic image and parallel to the horizontal direction of the endoscopic image as a reference. Note that when applying the method of this embodiment to sigmoid colectomy, the process of vertically inverting the endoscopic image does not need to be included in step S150. This is because the possibility of acquiring a vertically inverted image is low in sigmoid colectomy using a rigid endoscope.

[0089] Furthermore, step S150 in Figure 20 may be applied to the interpolated image generated in step S110. For example, suppose that when step S10 in Figure 9 is performed, the number of endoscopic images belonging to the first class corresponds to the length shown in A71 in Figure 21, and the number of endoscopic images belonging to the second class corresponds to the length shown in A72. In this case, the processor 110 creates a new interpolated image by step S110 in Figure 20, a new deformed image by step S150, and an image that has undergone further processing in step S150 on the endoscopic images belonging to the second class acquired by step S10 in Figure 9. As a result, the number of endoscopic images belonging to the second class increases to the number corresponding to the length shown in A73 in Figure 21. When the length shown in A73 becomes equal to the length shown in A71, the processor 110 determines YES in step S190 described above.

[0090] Based on the above, in the learning data creation device 100 of this embodiment, the processor 110 further generates learning data by transforming multiple endoscopic images from the learning device and the endoscopic images related to the generated learning data. In this way, the number of learning data related to endoscopic images belonging to the second class can be increased.

[0091] The above explanation described an example where, as long as the learning data for the process related to the first class is greater than the learning data for the process related to the second class, other relationships are not particularly important. However, in addition to the relationships described above, if, for example, the process related to the first class and the process related to the second class are in a transitional relationship, the method of this embodiment may be modified as follows. More specifically, although the flowchart is omitted, step S110 may be modified as follows.

[0092] In this embodiment, a process transition refers to the transition of processes according to a predetermined endoscopic surgical procedure plan, and includes transitions between processes that are valid as procedures. For example, while procedures generally proceed in the order of process numbers, there may be cases where processes proceed in reverse order for predetermined reasons, and even in this case, the processes that proceed can be said to have transitioned. Note that the process to which one returns is not limited to one step back, but may be two or more steps. Predetermined reasons include, for example, that the procedure on the target of the procedure was difficult, or that the procedure in a process was insufficient. For example, suppose there is a predetermined endoscopic surgical procedure that includes process M, process (M+1), process (M+2), and process (M+3). In principle, the processes proceed in the order of process M, process (M+1), and process (M+2), but if, for reasons such as insufficient procedure in process M, the procedure is performed to return from process (M+2) to process M, then process M and process (M+2) can be said to be in a transitional relationship. For example, the processor 110 stores in memory (not shown) the predetermined process transition patterns and the actual process transition patterns performed during each endoscopic surgical procedure. Then, based on the stored process transition patterns, the processor 110 creates new training data only from processes that are in a transitional relationship. In this way, it is possible to prevent the creation of new training data from processes that are not in a transitional relationship. This prevents a decrease in the performance of the trained model 22.

[0093] For example, processor 110 arbitrarily selects an instance that is included in the first region R1 and belongs to the second class, and then arbitrarily selects an instance that is included in the first second region R2-1 and belongs to the first class. Specifically, as shown in Figure 22, for example, the instance shown at A80 is an instance that is included in the first region R1 and belongs to the second class (i.e., the second instance), and the instance shown at A81 is an instance that is included in the first second region R2-1 and belongs to the first class (i.e., the first instance). Note that in Figure 22, the first region R1, the first second region R2-1, and the second second region R2-2 are shown in the same way as in Figure 17.

[0094] Subsequently, processor 110 connects the instance shown in A80 (i.e., the second instance) and the instance shown in A81 (i.e., the first instance) with a virtual line segment. Then, processor 110 sets up the third instance within the range of the set of points related to the line segment shown by the solid line in A82. In other words, in the example shown in Figure 22, processor 110 does not set up the third instance within the range of the set of points shown by the dotted line in A83. In other words, if the process related to the first class and the process related to the second class are in a transitional relationship, processor 110 generates new training data such that the features are located in the first region R1, but does not generate new training data such that the features are located in the second region R2-1 of the first class. This is because if the process related to the first class and the process related to the second class are transitional processes, new training data related to the second class should be generated. In the example shown in Figure 22, the location of the newly set third instance does not necessarily have to be near the boundary line between the first region R1 and the first second region R2-1; it can be anywhere on the solid line shown in A82. Furthermore, after setting the third instance, the fourth and fifth instances are set, the time for the third instance is set, and the endoscopic image based on the set time is acquired, as described above.

[0095] Alternatively, new instances may be continuously set on the solid line portion shown at A82 in Figure 22. Specifically, for example, processor 110 sets a third instance near the boundary line between the first region R1 and the first second region R2-1 in Figure 22, and sets the time associated with the third instance. Then, processor 110 calculates the difference between the time associated with the set third instance and the times associated with the other instances, and sets a new sixth instance, which takes this difference into account, at a desired position on the solid line portion shown at A82. The other instances can be any of the second, fourth, or fifth instances shown at A80 as appropriate. Furthermore, by repeating the same method, three or more instances, not limited to the third and sixth instances described above, may be newly set on the solid line portion shown at A82.

[0096] Furthermore, when repeatedly generating training data using the method shown in Figure 22, instead of re-selecting an instance belonging to the second class, it is also possible to arbitrarily select a new instance that is included in the first second region R2-1 and belongs to the first class. In other words, for example, if it is desired to generate five new training data sets, the processor 110 may select the instance shown at A90 in Figure 23, as well as the instances shown at A91, A92, A93, A94, and A95, respectively. The instance shown at A90 is an instance included in the first region R1 and belonging to the second class, while the instances shown at A91 to A95 are instances included in the first second region R2-1 and belonging to the first class.

[0097] The processor 110 then connects, for example, the instance shown at A90 and the instance shown at A91 in Figure 23 with a line segment, sets a third instance for a point on the solid line portion of the connected line segment, sets a fourth instance and a fifth instance, sets the time related to the third instance, and acquires an endoscopic image based on the set time, as described above. The processor 110 can repeat the same process for the instance shown at A90 and the instance shown at A92, the instance shown at A90 and the instance shown at A93, the instance shown at A90 and the instance shown at A94, and the instance shown at A90 and the instance shown at A95.

[0098] Furthermore, as mentioned above in Figure 22, the processor 110 may continuously set multiple instances on the solid portion of the line segment connecting the instance shown at A90 and the instance shown at A91 in Figure 23. Similarly, the processor 110 may continuously set multiple instances on the solid portion of the line segment connecting the instance shown at A90 and the instance shown at A92, or on the solid portion of the line segment connecting the instance shown at A90 and the instance shown at A93. Similarly, the processor 110 may continuously set multiple instances on the solid portion of the line segment connecting the instance shown at A90 and the instance shown at A94, or on the solid portion of the line segment connecting the instance shown at A90 and the instance shown at A95.

[0099] Based on the above, in the training data creation device 100 of this embodiment, the processor 110 sets a line segment where a straight line connecting the first instance and the second instance in the feature space overlaps with the first region R1, and generates training data corresponding to instances located on the set line segment. In this way, new training data based on instances located within the first region R1 can be created. This makes it possible to increase the amount of training data related to instances belonging to the second class.

[0100] Furthermore, the steps related to the first class and the steps related to the second class may be steps that can be transitioned between in a procedure using the endoscope 3. In this way, a method can be constructed to increase the amount of training data belonging to the second class that can be transitioned between the steps related to the first class. As a result, when endoscopic images with similar features in the first and second classes are input to the trained model 22 as input data, the output data will be a result that more accurately classifies the first and second classes. This allows the user to perform the procedure without confusing the steps.

[0101] Furthermore, the processor 110 may select one second instance and multiple first instances, set multiple line segments based on the one second instance and each of the first instances, and generate training data on the set multiple line segments. In this way, multiple new training data can be created without having to perform the process of selecting a second instance again. This reduces the processing load on the training data creation device 100 while enabling the creation of more training data.

[0102] Although this embodiment has been described in detail above, it will be readily apparent to those skilled in the art that many modifications are possible without substantially departing from the novelty and effects of this disclosure. Therefore, all such modifications are included within the scope of this disclosure. For example, any term that appears at least once in the specification or drawings together with a broader or synonymous term may be replaced with that different term anywhere in the specification or drawings. Furthermore, all combinations of this embodiment and its modifications are also included within the scope of this disclosure. In addition, the configuration and operation of the learning data creation device, information support device, endoscope system, learning data creation method, and information storage medium are not limited to those described in this embodiment, and various modifications are possible. [Explanation of symbols]

[0103] 1...Endoscope system, 3...Endoscope, 10...Information support device, 20...Memory, 22...Trained model, 30...Processor, 40...Input unit, 50...Output unit, 90...Learning device, 100...Learning data creation device, 110...Processor, 112...Class determination unit, 114...Data generation unit, 116...Data expansion unit, R1...First domain, R2...Second domain, R2-1...First second domain, R2-2...Second second domain, t1, t2, t3, t4, t5, t6, t12, t14, t15...Timing

Claims

1. A training data creation device including a processor that performs processing to generate training data for machine learning, The aforementioned processor, The learning device plots multiple endoscopic images in a feature space based on their individual features. The class is determined for multiple endoscopic images, A set area is set for each of the determined classes. A learning data creation device characterized by generating new learning data such that the number of learning data newly generated in a first region, which is the intersection region of the sets of each class, is greater than the number of learning data newly generated in a second region, which is the region obtained by removing the first region from the union region of the sets of each class.

2. In the learning data creation device according to claim 1, The aforementioned processor, A learning data creation device characterized by setting the set region based on a first distance, which is a distance in the feature space, where the distance is the distance from the centroid of the feature quantities of a plurality of endoscope images belonging to the same class to the feature quantities of each of the endoscope images.

3. In the learning data creation device according to claim 2, The aforementioned processor, A second distance is further set for each of the aforementioned classes, For each instance of the endoscopic image, the first distance and the second distance are compared. A learning data creation device characterized by setting the set region for a set of instances in which the first distance is shorter than the second distance.

4. In the learning data creation device according to claim 1, The aforementioned processor, A learning data creation device characterized by setting the set region for each class by forming a convex hull.

5. In the learning data creation device according to claim 1, The aforementioned processor, A learning data creation device characterized by generating learning data corresponding to the interpolated feature quantities by interpolating new feature quantities within the first region based on the acquisition time of the endoscopic images relating to multiple instances.

6. In the learning data creation device according to claim 1, The aforementioned processor, A learning data creation device characterized by generating the learning data in the second class's set region in the first region where the first class's set region and the second class's set region, which has less learning data than the first class, overlap.

7. In the learning data creation device according to claim 6, The set region for each class includes the set region for the first class and the set region for the second class, The aforementioned processor, A first instance is arbitrarily selected from the set area of ​​the first class, and a second instance is arbitrarily selected from the set area of ​​the second class. A learning data creation device characterized in that it generates the learning data when the selected first instance is included in a first second region, which is the set region obtained by removing the first region from the set region of the first class, and the selected second instance is included in the first region.

8. In the learning data creation device of claim 7, The aforementioned processor, In the feature space, a half-line is virtually set parallel to an arbitrarily selected axis and with the selected second instance as its end. A learning data creation device characterized in that, when the set half-line intersects once with the outer perimeter of the first class collection area and once with the outer perimeter of the second class collection area, it is determined that the second instance exists inside the first area and generates the learning data.

9. In the learning data creation device of claim 7, The aforementioned processor, A learning data creation device characterized by arbitrarily selecting a second instance from the set region of the second class, which is the class relating to the process of processing blood vessels in sigmoid colectomy.

10. In the learning data creation device of claim 7, The aforementioned processor, A learning data creation device characterized by arbitrarily selecting a second instance from the set region of the second class, which is the class relating to the process of resecting the rectum in sigmoid colectomy.

11. In the learning data creation device of claim 7, The aforementioned processor, A learning data creation device characterized by setting a line segment in the feature space where a straight line connecting the first instance and the second instance overlaps with the first region, and generating the learning data corresponding to the instance located on the set line segment.

12. In the learning data creation device of claim 11, A learning data creation device characterized in that the process relating to the first class and the process relating to the second class are processes that can be transitioned to in a procedure using an endoscope.

13. In the learning data creation device of claim 12, The aforementioned processor, Select one of the second instances and select multiple of the first instances, Based on the first instance and each of the first instances, a plurality of line segments are set. A learning data creation device characterized by generating the learning data on a plurality of predetermined line segments.

14. In the learning data creation device according to claim 1, The aforementioned processor, A learning data creation device characterized by generating further learning data by deforming a plurality of endoscopic images of the learning device and the endoscopic images related to the generated learning data.

15. A memory for storing a trained model that has been machine-learned using the training data generated by the training data creation device of claim 1, Processor and An information support device including, The aforementioned processor, An information support device characterized by outputting results inferred from the endoscopic image based on the trained model.

16. The information support device of claim 15, Endoscope and, An endoscopic system characterized by including the following.

17. The steps include plotting multiple endoscopic images from the learning device in a feature space based on their individual features, A step of determining a class for multiple endoscopic images, The steps include setting a set area for each of the determined classes, The step of generating new training data such that the number of training data newly generated in the first region, which is the intersection region of the sets of each class, is greater than the number of training data newly generated in the second region, which is the region obtained by subtracting the first region from the union region of the sets of each class, is: A method for creating training data, characterized by having a computer perform the task.

18. The steps include plotting multiple endoscopic images from the learning device in a feature space based on their individual features, A step of determining a class for multiple endoscopic images, The steps include setting a set area for each of the determined classes, The number of new training data generated in the first region, which is the intersection region of the sets of each class, is greater than the number of new training data generated in the second region, which is the region obtained by subtracting the first region from the union region of the sets of each class, as a step in generating new training data. A non-temporary information storage medium characterized by its ability to store programs to be executed by a computer.

Images