Image processing system, endoscope system, and image processing method

Abstract

An image processing system (100) includes an image acquisition unit (110) that acquires, as a processing target image sequence, time-series images obtained by capturing an inside of a living body with an endoscope camera device, and a processing unit (120) that performs processing on the processing target image sequence in accordance with a database generated from a plurality of images of the inside of the living body captured at a time point earlier than the processing target image sequence. When bleeding has occurred in the inside of the living body, the processing unit (120) performs processing of deciding a hemostatic treatment for a blood vessel in which the bleeding has occurred, based on the processing target image sequence and the database, and prompting the decided hemostatic treatment to a user.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing system, an endoscope system, and an image processing method. BACKGROUND

[0002] In the past, in surgery using an endoscope system or the like, a method of performing image processing based on an image obtained by photographing inside a living body is known. For example, a method of detecting a bleeding region from dynamic image information in surgery is disclosed in Patent Literature 1.

[0003] PRIOR ART DOCUMENTS

[0004] PATENT LITERATURE

[0005] Patent Literature 1: Japanese Patent Application Publication No. 2011-36371 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] The method of Patent Literature 1 requires detection of a bleeding region, a change in area of the bleeding region, a bleeding amount, and the like, but the disposal of bleeding is left to the user as a surgical staff to cope with.

[0008] According to several aspects of the present disclosure, it is possible to provide an image processing system, an endoscope system, and an image processing method, and the like, which can appropriately support a user by presenting a recommended disposal method for bleeding generated inside a living body.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] One aspect of the present disclosure relates to an image processing system including: an image acquisition unit that acquires, as a processing target image sequence, a time-series image obtained by photographing inside a living body by an endoscope imaging device; and a processing unit that performs processing in accordance with a database and the processing target image sequence, the database being generated from a plurality of images of the inside of a living body that are photographed at a time point earlier than the processing target image sequence, and when bleeding has occurred in the living body, the processing unit performs processing of deciding a hemostatic disposal for a blood vessel in which the bleeding has occurred, based on the processing target image sequence and the database, and presenting the decided hemostatic disposal to a user.

[0011] Another aspect of the present disclosure relates to an endoscope system including: an imaging unit configured to image an inside of a living body; an image acquisition unit configured to acquire a time-series image captured by the imaging unit as a processing target image sequence; and a processing unit configured to perform processing based on a database and the processing target image sequence, the database being generated based on a plurality of images of the inside of the living body captured at a time point earlier than the processing target image sequence, and when bleeding occurs in the inside of the living body, the processing unit is configured to determine a hemostasis treatment for a blood vessel in which the bleeding occurs based on the processing target image sequence and the database, and present the determined hemostasis treatment to a user.

[0012] Another aspect of the present disclosure relates to an image processing method including: acquiring a time-series image obtained by imaging the inside of a living body by an endoscope imaging device as a processing target image sequence; when bleeding occurs in the inside of the living body, determining a hemostasis treatment for a blood vessel in which the bleeding occurs based on a database and the processing target image sequence, the database being generated based on a plurality of images of the inside of the living body captured at a time point earlier than the processing target image sequence; and presenting the determined hemostasis treatment to a user. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a schematic view showing a blood vessel related to a liver.

[0014] Figure 2 is a schematic view showing a blood vessel related to a liver.

[0015] Figure 3 is a schematic view showing a blood vessel related to a liver.

[0016] Figure 4 is a schematic view showing a blood vessel related to a liver.

[0017] Figure 5 is a schematic view showing a blood vessel related to a liver.

[0018] Figure 6 is a schematic view showing a blood vessel related to a liver.

[0019] Figure 7 is a schematic view showing a blood vessel related to a liver.

[0020] Figure 8 is a schematic view showing a blood vessel related to a liver.

[0021] Figure 9 is a schematic view showing a blood vessel related to a liver.

[0022] Figure 10 is a schematic view showing a blood vessel related to a liver.

[0023] Figure 11 Fig. 1 is a flowchart illustrating a procedure of a bleeding point detection process targeting a given bleeding region.

[0024] Figure 12 (A) of Fig. 2, Figure 12 (B) of Fig. 2 is a schematic diagram illustrating a bleeding point detection process.

[0025] Figure 13 Fig. 3 is an example of data acquired through the bleeding point detection process.

[0026] Figure 14 (A) of Fig. 4, Figure 14 (B) of Fig. 4 is an example of a database.

[0027] Figure 15 (A) of Fig. 5, Figure 15 (B) of Fig. 5 is an example of a neural network.

[0028] Figure 16 Fig. 6 is an example of input and output of a neural network.

[0029] Figure 17 Fig. 7 is a flowchart illustrating a decision process of a hemostatic treatment.

[0030] Figure 18 Fig. 8 is a flowchart illustrating a display process.

[0031] Figure 19 Fig. 9 is an example of a display image for prompting a hemostatic treatment.

[0032] Figure 20 (A) of Fig. 10 is another example of a database, Figure 20 (B) of Fig. 10 is another example of input and output of a neural network.

[0033] Figure 21 (A) of Fig. 11 is another example of a database, Figure 21 (B) of Fig. 11 is another example of input and output of a neural network.

[0034] Figure 22 Fig. 12 is a flowchart illustrating a decision process of a hemostatic treatment.

[0035] Figure 23 (A) of Fig. 13, Figure 23 (B) of Fig. 13 is another example of a database. DETAILED DESCRIPTION

[0036] Hereinafter, the present embodiment will be described. Note that the present embodiment described below is not intended to unduly limit the content recited in the claims. Also, all the structures described in the present embodiment are not necessarily essential constituent elements.

[0037] 1. Method of the present embodiment

[0038] In surgery using a laparoscope, even if a physician performs surgery in accordance with a preoperative plan, it is possible that an unexpected bleeding occurs during surgery. For example, the course of a blood vessel differs from patient to patient, and thus it is difficult to reliably avoid damage to a blood vessel. In the case where bleeding occurs, the physician needs to promptly perform a treatment. For example, in the case of a liver partial resection in which a part of a liver is removed, if the treatment for bleeding is wrong, it can be related to a prognosis, and thus it is important to promptly perform a treatment.

[0039] In Patent Literature 1, a method of detecting and prompting information related to bleeding based on a captured image is disclosed. However, the existing method of Patent Literature 1 and the like can possibly be able to detect the presence or absence of bleeding and the like, but the decision of a specific treatment based on this information is left to the physician.

[0040] Figure 1 is a schematic view showing a liver and blood vessels related to the liver. As shown in Figure 1 Among the blood vessels related to the liver, there are multiple blood vessels of different types including a portal vein (Al), a hepatic artery (A2), and a hepatic vein (A3). Moreover, depending on the type of blood vessel, the desired hemostatic treatment when bleeding occurs differs. Therefore, depending on the physician's proficiency, it is difficult to promptly decide an appropriate treatment for bleeding. As a result, the surgery time increases, and the burden on the patient and the surgical staff becomes large.

[0041] Figure 2 is a view showing a configuration example of an image processing system 100 of the present embodiment. The image processing system 100 includes an image acquisition unit 110 that acquires, as a processing target image sequence, a time-series image obtained by capturing an inside of a living body by an endoscope imaging device, and a processing unit 120 that performs processing on the basis of a database generated from a plurality of images of the inside of the living body captured at a point in time before the processing target image sequence and the processing target image sequence. Moreover, when bleeding has occurred in the inside of the living body, the processing unit 120 performs processing of deciding a hemostatic treatment for a blood vessel in which bleeding has occurred and prompting the decided hemostatic treatment to a user on the basis of the processing target image sequence and the database. In addition, the endoscope imaging device refers to an imaging device included in an endoscope system 200, such as an endoscope scope 210 described later. When bleeding has occurred in the inside of the living body refers to, specifically, for example, using a blood flow detection device described later, it is determined that bleeding has occurred in the inside of the living body. Figure 9 As described later, indicates a case where it is detected that bleeding has occurred by a bleeding detection process. However, the image processing system 100 can also not perform the bleeding detection process, but periodically perform a decision process of a hemostatic treatment. In this case, the decision process result of the hemostatic treatment differs depending on the occurrence condition of bleeding, and a specific hemostatic treatment such as suturing is decided and prompted when bleeding has occurred, and it is determined that a hemostatic treatment is not needed when bleeding has not occurred.

[0042] Here, the processing target image sequence represents images of a time series that become a processing target of deciding a hemostasis treatment. Specifically, the processing target image sequence is a real-time dynamic image that is captured in a laparoscopic surgery in progress. In contrast, the database contains a plurality of images that are captured in a surgery performed before a point in time at which the processing of deciding the hemostasis treatment is performed. In addition, the surgery performed before can be a surgery performed on the same patient as a patient who is the object of the laparoscopic surgery performed in real time, or can be a surgery performed on a different patient. For example, in the case of deciding a hemostasis treatment in a liver partial resection, the database contains a dynamic image that is captured in a liver partial resection performed before. In addition, if the relationship with the processing target image sequence is considered, the surgery here specifically is a surgery using a laparoscope. However, a case image obtained by capturing a surgery progress in an open surgery can also be contained in the database. In addition, if the accuracy of the processing based on the database is considered, it is preferable that the amount of information contained in the database be large. For example, in the case where a dynamic image of a period corresponding to the occurrence of one bleeding in a liver partial resection is counted as one dynamic image, the database contains a plurality of dynamic images. That is, one dynamic image corresponding to one bleeding contains a plurality of in-vivo images, and the database contains a plurality of such dynamic images.

[0043] According to the method of the present embodiment, not only the presence or absence of bleeding, the bleeding point, but also a recommended hemostasis treatment can be prompted to the user. The user here specifically is a doctor who is a surgery staff. Thereby, the hemostasis treatment performed by the user can be appropriately supported.

[0044] In addition, in the in-vivo images, a region in which blood exists is captured as a red region. Therefore, the amount of bleeding can be estimated from the area of the red region. However, if the distance between the imaging unit and the living body changes, even if the size of the subject is the same in reality, the size on the image changes. Furthermore, the structure of the imaging optical system such as a lens, image processing such as digital zooming becomes a factor that causes the size of a given subject on the image to change. Therefore, it is not easy to accurately grasp the state of bleeding, for example, to estimate the amount of bleeding or its time series change with high precision, from only the processing target image sequence. Therefore, it is also difficult to estimate the hemostasis treatment recommended for bleeding with high precision from only the processing target image sequence. For example, even if it is intended to decide a hemostasis treatment by a process of comparing a given threshold value with the area of the red region, it is not possible to set an appropriate threshold value. In addition, regarding the size of an organ such as a liver, a fine shape, there are individual differences due to patients, so it can also be said that it is difficult to set a threshold value that can be applied to many patients in general.

[0045] On the other hand, in the method of the present embodiment, when deciding the recommended hemostatic treatment, not only the processing target image sequence is used, but also database-based processing is performed. Since the processing is performed based on information obtained in a surgery performed previously, the recommended hemostatic treatment can be estimated with high accuracy. In addition, the database-based processing can be processing based on a learned model generated by using machine learning using the database, or can be comparison processing using information included in the database and the processing target image sequence. Details will be described later.

[0046] 2. System configuration example

[0047] First, the configuration of the entire system including the image processing system 100 will be described, and then the detailed configuration of the image processing system 100 and the configuration of the endoscope system 200 will be described.

[0048] 2.1 Entire configuration example

[0049] Figure 3 is an example of the configuration of a system including the image processing system 100 of the present embodiment. As shown in Figure 3 , the system includes the image processing system 100, the endoscope system 200, the database server 300, the database generation apparatus 400, the learning apparatus 500, and the image collection endoscope system 600. However, the system is not limited to the configuration of Figure 3 , and various modifications such as omission of part of the configuration elements or addition of other configuration elements can be implemented.

[0050] The image collection endoscope system 600 is an endoscope system that captures a plurality of in-vivo images used for generating the database of the present embodiment. On the other hand, the endoscope system 200 is a system that captures a processing target image sequence that is the object of the hemostatic treatment decision processing, and is a system that is performing a surgery using a laparoscope in a narrow sense. If the update processing of the database described later is considered, the processing target image sequence captured by the endoscope system 200 can be used as part of the database for deciding a hemostatic treatment in a future surgery. That is, the endoscope system 200 can also function as the image collection endoscope system 600 at other points in time. In addition, the image collection endoscope system 600 can also function as the endoscope system 200 of the present embodiment at other points in time.

[0051] The database server 300 can be a server provided in a dedicated network such as an intranet, or can be a server provided in a public communication network such as the Internet.

[0052] The database server 300 first collects a plurality of in-vivo images, i.e., a surgery image sequence, taken in a previous surgery from the image collection endoscope system 600. However, the surgery image sequence also contains images unrelated to bleeding and the like. For example, an image taken during a period in which bleeding does not occur in the surgery image sequence is less useful in the determination processing of the hemostatic treatment than an image at the time of bleeding. In addition, in order to utilize the determination processing of the hemostatic treatment in the image processing system 100, it is necessary to correspond a plurality of in-vivo images taken to have bleeding to information related to the hemostatic treatment that should be performed with respect to the bleeding.

[0053] Therefore, the database generation apparatus 400 performs the following processing: the database server 300 acquires the collected surgery image sequence from the image collection endoscope system 600, and generates the database of the present embodiment.

[0054] Figure 4 is a flowchart illustrating the database generation processing in the database generation apparatus 400. When the processing is started, in step S11, the database generation apparatus 400 performs processing of extracting an image sequence of a bleeding pattern from the surgery image sequence. The image sequence of the bleeding pattern indicates, for example, a plurality of in-vivo images corresponding to a period from a bleeding start frame to a frame at which bleeding is detected. For example, the database generation apparatus 400 performs processing of determining an image Pi at which bleeding is detected and an image Ps corresponding to a bleeding start frame by performing the processing described later. The database generation apparatus 400 extracts images corresponding to a period from Pi to Ps as the image sequence of the bleeding pattern. Furthermore, all of the images corresponding to the period from Pi to Ps can be extracted as extraction targets, or a part of the images can be omitted. Alternatively, the image sequence of the bleeding pattern can be, for example, a plurality of in-vivo images corresponding to a period from bleeding start to emergency hemostatic treatment described later. In this case, it is assumed that the amount of bleeding increases as time elapses from the start point, and decreases if emergency hemostatic treatment is started. In this way, the temporal change in the bleeding condition in the in-vivo images is defined as the bleeding pattern in advance, and the database generation apparatus 400 performs processing of extracting an image sequence having high similarity to the bleeding pattern from the surgery image sequence. Furthermore, the extraction of the image sequence can be performed based on user input, or automatically using image processing. Figure 9-11 The processing described later. The database generation apparatus 400 performs processing of determining an image Pi at which bleeding is detected and an image Ps corresponding to a bleeding start frame by performing the processing described later. The database generation apparatus 400 extracts images corresponding to a period from Pi to Ps as the image sequence of the bleeding pattern. Furthermore, all of the images corresponding to the period from Pi to Ps can be extracted as extraction targets, or a part of the images can be omitted. Alternatively, the image sequence of the bleeding pattern can be, for example, a plurality of in-vivo images corresponding to a period from bleeding start to emergency hemostatic treatment described later. In this case, it is assumed that the amount of bleeding increases as time elapses from the start point, and decreases if emergency hemostatic treatment is started. In this way, the temporal change in the bleeding condition in the in-vivo images is defined as the bleeding pattern in advance, and the database generation apparatus 400 performs processing of extracting an image sequence having high similarity to the bleeding pattern from the surgery image sequence. Furthermore, the extraction of the image sequence can be performed based on user input, or automatically using image processing.

[0055] Next, in step S12, the database generation device 400 performs annotation. Annotation refers to a process of adding metadata related to recommended hemostatic treatment to the bleeding captured in the image sequence of the bleeding pattern. Annotation is performed by a user with expertise such as a doctor, for example. The database generation device 400 receives the annotation input by the user and performs a process of associating the received information with the image sequence extracted in step S11. Note that the information added here can be information for determining the type of blood vessel, information for determining the hemostatic treatment, or both.

[0056] In addition, in step S13, the database generation device 400 can also perform a process of associating other additional information with the image sequence of the bleeding pattern. The additional information here can include, for example, information such as the name of the hospital, the name of the surgical staff, the surgical method, and the like, and information such as matters to be noted during surgery. In addition, the additional information can be information related to the patient such as age, sex, height, weight, and the like, and information obtained using a CT (Computed Tomography) or MRI (Magnetic Resonance Imaging) before surgery.

[0057] In step S14, the database generation device 400 performs a process of registering metadata containing information for determining the hemostatic treatment in association with the image sequence of the bleeding pattern in the database of the present embodiment. Specifically, the process of step S14 is a write process to the table data of the database server 300.

[0058] The learning device 500 generates a learned model by performing machine learning using the database. For example, the learning device 500 acquires the database from the database server 300 and generates a learned model by performing machine learning as described later. The learning device 500 transmits the generated learned model to the database server 300.

[0059] The image processing system 100 acquires the learned model from the database server 300. In addition, the image processing system 100 acquires the image sequence to be processed from the endoscope system 200. Then, the image processing system 100 performs a process of determining the hemostatic treatment based on the learned model generated based on the database and the image sequence to be processed.

[0060] In addition, the database generation device 400 can be included in the database server 300 and perform the processes illustrated in the database server 300. Alternatively, the database generation device 400 can be included in the endoscope system 600 for image collection. In this case, Figure 4 the processes illustrated in the database server 300. Alternatively, the database generation device 400 can be included in the endoscope system 600 for image collection. In this case, Figure 4The illustrated processing is executed in the image collection endoscope system 600, and the processed data is transmitted to the database server 300.

[0061] In addition, the database generation apparatus 400 and the learning apparatus 500 can also be provided as the same apparatus. In this case, the processing of generating a database from a surgical image sequence and the machine learning based on the generated database can be executed in the same apparatus. Alternatively, the learning apparatus 500 and the image processing system 100 can also be integrated. In this case, the learning processing of generating a learned model and the inference processing using the learned model can be executed in the same apparatus.

[0062] In addition, in Figure 3 the example in which the learned model generated by the learning apparatus 500 is temporarily stored in the database server 300 is illustrated, but the learned model can also be directly transmitted from the learning apparatus 500 to the image processing system 100. In addition, as described later, in the method of the present embodiment, machine learning is not necessarily required. In the case where machine learning is not performed, the learning apparatus 500 can be omitted.

[0063] As described above, Figure 3 is an example of a system structure, and the structure of a system including the image processing system 100 can be implemented in various modifications.

[0064] 2.2 Image processing system

[0065] Figure 5 is a diagram illustrating an example of a detailed structure of the image processing system 100. The image processing system 100 includes an image acquisition section 110, a processing section 120, and a storage section 130. The processing section 120 includes a bleeding detection section 121, a bleeding point detection section 123, a treatment decision section 125, and a display processing section 127. However, the image processing system 100, the processing section 120 are not limited to Figure 5 the structure of, and various modifications such as omission of a part of the structural elements or addition of other structural elements can be implemented.

[0066] The image acquisition section 110 is an interface circuit that acquires an in-vivo image captured by a photographing device of the endoscope system 200. In the case where the image processing system 100 is included in the processor unit 220 of the endoscope system 200, the image acquisition section 110 corresponds to a photographing data reception section 221 that acquires an image signal from the photographing device via a cable. The interface circuit is, for example, a circuit having a function of acquiring an in-vivo image from the endoscope system 200 and transmitting the acquired in-vivo image to the processing section 120. In this case, the image acquisition section 110 can also receive an in-vivo image as digital data from the photographing device. Alternatively, the image acquisition section 110 can also receive an analog signal from the photographing device, and acquire an in-vivo image as digital data by A / D converting the analog signal. That is, the interface circuit can also be an A / D conversion circuit. Further, in the case where the image processing system 100 is provided separately from the endoscope system 200, the image acquisition section 110 is realized as a communication interface that receives an in-vivo image from the endoscope system 200 via a network. That is, the interface circuit can also be a circuit provided to a communication chip, a communication device. The network here can be a private network such as an intranet, or a public communication network such as the Internet. Further, the network can be wired or wireless. Further, the image acquisition section 110 acquires, for example, an image obtained by photographing an in-vivo object per 1 frame. However, the image acquisition section 110 can also acquire a plurality of images corresponding to a plurality of frames collectively.

[0067] The processing section 120 is constituted by hardware described below. The hardware can include at least one of a control circuit that processes a digital signal and a control circuit that processes an analog signal. The hardware can be constituted by one or a plurality of circuit devices mounted on a circuit board, one or a plurality of circuit elements, for example. The one or a plurality of circuit devices are, for example, an IC (Integrated Circuit), an FPGA (field-programmable gate array), or the like. The one or a plurality of circuit elements are, for example, a resistor, a capacitor, or the like.

[0068] Further, the processing section 120 can also be realized by a processor described below. The image processing system 100 includes a memory that stores information and a processor that works based on the information stored in the memory. The memory here can be the storage section 130 or a different memory. The information is, for example, programs and various data and the like. The processor includes hardware. The processor can use various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), and the like. The memory can be a semiconductor memory such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), and the like, can be a register, and can also be a magnetic storage device such as an HDD (Hard Disk Drive), and can also be an optical storage device such as an optical disk device. For example, the memory stores instructions that can be read by a computer, and by executing the instructions by the processor, the functions of each section of the processing section 120 are realized as processing. Specifically, each section of the processing section 120 is the bleeding detection section 121, the bleeding point detection section 123, the treatment decision section 125, and the display processing section 127. The instructions here can be instructions that constitute an instruction set of a program, or can be instructions that instruct the hardware circuit of the processor to work. Further, all or a part of each section of the processing section 120 can also be realized by cloud computing, and each processing described later is performed on the cloud computing.

[0069] The storage section 130 becomes a work area of the processing section 120 and the like, and its functions can be realized by a semiconductor memory, a register, a magnetic storage device, and the like. The storage section 130 stores the processing target image sequence acquired by the image acquisition section 110. In addition, the storage section 130 stores the learned model generated using the database. Further, the storage section 130 can store the database itself together with or instead of the learned model.

[0070] The bleeding detection section 121 performs a bleeding detection process in which whether bleeding has occurred in a living body that is a photographic subject is detected from the processing target image sequence. Details of the bleeding detection process will be described later with reference to Figure 9 Details of the bleeding detection process will be described later with reference to

[0071] The bleeding point detection section 123 performs a bleeding point detection process in which a bleeding point is detected from the processing target image sequence. The bleeding point here indicates a position where bleeding has occurred. The bleeding point detection process uses the learned model stored in the storage section 130. Details of the bleeding point detection process will be described later with reference to Figure 10 , Figure 11Details of the bleeding point detection processing will be described later.

[0072] The treatment decision unit 125 performs processing of deciding a recommended hemostatic treatment for bleeding. The treatment decision unit 125 performs processing of reading out a learned model from, for example, the storage unit 130, processing of inputting an image sequence in which a prescribed interval of the processing target image sequence is extracted to the learned model, processing of deciding a blood vessel type in which bleeding has occurred on the basis of an output of the learned model, and processing of deciding a hemostatic treatment on the basis of the decided blood vessel type. The processing of deciding a hemostatic treatment using the learned model is described in detail later. Figure 14 Figure 17 Details of the processing using the learned model will be described later. Note that, as a modification, the processing unit 120 can also directly decide a hemostatic treatment on the basis of an output of the learned model, as described later. Alternatively, the processing unit 120 can also decide a hemostatic treatment on the basis of a comparison processing of the processing target image sequence and a database without using the learned model.

[0073] The display processing unit 127 performs processing of displaying a hemostatic treatment decided by the treatment decision unit 125 on a display unit. The display unit here is, for example, the display unit 230 of the endoscope system 200. The display processing unit 127 performs processing of generating a display image, for example, by superimposing information on the hemostatic treatment on each image of the processing target image sequence, and processing of transmitting the generated display image to the display unit 230. Alternatively, the display processing unit 127 can also perform processing of transmitting a display image to the processor unit 220 of the endoscope system 200. In this case, display control in the display unit 230 is performed by the processor unit 220. Alternatively, the display processing unit 127 can also perform processing of transmitting information on the decided hemostatic treatment to the endoscope system 200 or the like and instructing display. In this case, the generation processing of a display image and the control processing of the display unit 230 are performed by the processor unit 220.

[0074] Alternatively, the processing performed by the image processing system 100 of the present embodiment can also be implemented as an image processing method. The image processing method of the present embodiment performs processing of acquiring a time-series image obtained by photographing an inside of a living body by an endoscope as a processing target image sequence, deciding a hemostatic treatment for a blood vessel in which bleeding has occurred on the basis of a database generated on the basis of a plurality of images of the inside of the living body photographed at a time point before the processing target image sequence when bleeding has occurred in the inside of the living body, and the processing target image sequence, and prompting the decided hemostatic treatment to a user.

[0075] ​Further, each of the sections of the processing section 120 of the present embodiment can be realized as a module of a program that works on a processor. For example, the bleeding detection section 121 and the bleeding point detection section 123 are realized as image processing modules that perform bleeding detection processing and bleeding point detection processing, respectively. The treatment decision section 125 is realized as a processing module for deciding a hemostatic treatment. In the case of using a learned model, the treatment decision section 125 is realized as an inference processing module for calculating an output in accordance with the learned model. In the case of using the database itself, the treatment decision section 125 is realized as a comparison processing module that compares a processing target image sequence and the database. The display processing section 127 is realized as an image processing module that generates a display image, a control module that controls the display section 230.

[0076] Further, a program that realizes the processing performed by each of the sections of the processing section 120 of the present embodiment can be stored in an information storage device that is a medium readable by a computer, for example. The information storage device can be realized by an optical disk, a memory card, an HDD, a semiconductor memory, or the like, for example. The semiconductor memory is a ROM, for example. The processing section 120 performs various processing of the present embodiment on the basis of a program stored in the information storage device. That is, the information storage device stores a program for causing a computer to function as each of the sections of the processing section 120. The computer is a device that is provided with an input device, a processing section, a storage section, and an output device. Specifically, the program of the present embodiment is a program for causing a computer to perform each of the steps described later using the program. Figure 7 Further, a program that realizes the processing performed by each of the sections of the processing section 120 of the present embodiment can be stored in an information storage device that is a medium readable by a computer, for example. The information storage device can be realized by an optical disk, a memory card, an HDD, a semiconductor memory, or the like, for example. The semiconductor memory is a ROM, for example. The processing section 120 performs various processing of the present embodiment on the basis of a program stored in the information storage device. That is, the information storage device stores a program for causing a computer to function as each of the sections of the processing section 120. The computer is a device that is provided with an input device, a processing section, a storage section, and an output device. Specifically, the program of the present embodiment is a program for causing a computer to perform each of the steps described later using the program.

[0077] 2.3 Endoscope system

[0078] Figure 6 is a configuration example of an endoscope system 200. The endoscope system 200 includes an endoscope scope 210, a processor unit 220, and a display section 230. The endoscope system 200 can further include an operation section 240.

[0079] A photographing device is provided at a distal end portion of the endoscope scope 210, and the distal end portion is inserted into an abdominal cavity. The photographing device photographs an image inside the abdominal cavity, and photographing data thereof is transmitted from the endoscope scope 210 to the processor unit 220.

[0080] The processor unit 220 is a device that performs various processing in the endoscope system 200. For example, the processor unit 220 performs control and image processing of the endoscope system 200, and the like. The processor unit 220 includes a photographing data reception section 221, a processing section 222, and a storage section 223.

[0081] The photographing data reception section 221 receives photographing data from the endoscope scope 210. The photographing data reception section 221 is a connector that is connected to a cable of the endoscope scope 210, an interface circuit that receives photographing data, or the like, for example.

[0082] The processing unit 222 processes the captured data and displays the processing results on the display unit 230. The processing in the processing unit 222 includes, for example, white balance adjustment, noise reduction, and other correction processing. Furthermore, the processing unit 222 can also perform detection processing, such as detecting a given subject from the captured data. Specifically, the processing unit 222 is an image processing circuit.

[0083] The storage unit 223 becomes the working area of ​​the processing unit 222, etc., and is a storage device such as a semiconductor memory, hard disk drive, or optical disk drive.

[0084] The display unit 230 is a monitor that displays images output from the processing unit 222, such as a liquid crystal display or an organic EL display.

[0085] The operation unit 240 is a device for the operator to operate the endoscope system 200. For example, the operation unit 240 is a button, dial, foot switch, touch panel, etc. As described later, the processing unit 222 can also change the display mode of the object based on the input information from the operation unit 240.

[0086] Figure 2 and Figure 5 The image processing system 100 shown can, for example, be with... Figure 6 The endoscope system 200 shown is a separate unit. For example, the image processing system 100 is included in an information processing device connected to the processor unit 220. This information processing device can be a PC (Personal Computer) or a server system. The processor unit 220 and the information processing device can be connected via cable or wireless network.

[0087] The processing unit 222 of the processor unit 220 processes the image data acquired by the image data receiving unit 221 and transmits it to the information processing device via a communication unit (not shown). The image acquisition unit 110 of the image processing system 100 acquires the timing image data transmitted from the processor unit 220 as a processing target image sequence. The processing unit 120 performs hemostasis treatment decision processing on the processing target image sequence. The image processing system 100 returns the hemostasis treatment decision processing result to the processor unit 220. The processing unit 222 of the processor unit 220 displays the display image received from the image processing system 100, which contains information related to the hemostasis treatment, on the display unit 230.

[0088] Alternatively, the image processing system 100 can also be included in the endoscope system 200. In this case, the image acquisition section 110 corresponds to the captured data reception section 221. The processing section 120 corresponds to the processing section 222. The storage section 130 corresponds to the storage section 223. That is, the method of the present embodiment can also be applied to the endoscope system 200. The endoscope system 200 of the present embodiment includes: a photographing section that photographs the inside of a living body; an image acquisition section 110 that acquires a time-series image photographed by the photographing section as a processing target image sequence; and a processing section 120 that performs processing on the processing target image sequence in accordance with a database generated from a plurality of images of the inside of the living body photographed at a point in time prior to the processing target image sequence. The processing section 120 performs processing of deciding a hemostatic treatment for a blood vessel in which bleeding has occurred on the basis of the processing target image sequence and the database when bleeding has occurred in the inside of the living body, and prompts the user with the decided hemostatic treatment.

[0089] The photographing section is, for example, a photographing device included in the endoscope scope 210 as described above. The endoscope scope 210 irradiates illumination light guided to an illumination lens of the endoscope scope 210 from a light source device not shown by light guiding to an object. The photographing section is an image pickup element that receives reflected light from the object via a lens system including an objective lens, a focusing lens, and the like. In addition, the photographing section can also include an A / D conversion circuit that A / D-converts an analog signal output as the image pickup element. Alternatively, the captured data reception section 221 can also include the A / D conversion circuit.

[0090] 3. Details of processing

[0091] Next, the processing performed in the image processing system 100 will be described. First, after the overall processing flow is described, the details of each processing will be described.

[0092] 3.1 Overall processing

[0093] Figure 7 is a flowchart that explains the processing in the image processing system 100. When the processing is started, first in step S101, the image acquisition section 110 acquires a processing target image sequence. For example, the image acquisition section 110 acquires an image of the latest frame from the endoscope system 200, and reads out an image of a prescribed frame before that stored in the storage section 130. Here, the latest frame is denoted as frame i, and the image in this frame i is denoted as Pi. i is a variable that indicates the frame.

[0094] In step S102, the processing section 120 performs bleeding-related determination, that is, bleeding determination processing. The bleeding determination processing includes bleeding detection processing performed by the bleeding detection section 121, bleeding point detection processing performed by the bleeding point detection section 123, and decision processing of a treatment performed by the treatment decision section 125.

[0095] In step S103, the display processing unit 127 performs the following processing: it generates a display image by overlaying information related to the determined bleeding point onto the image of the latest frame, and displays the display image on the display unit. Additionally, in step S103, the display processing unit 127 may also display information related to the determined hemostasis treatment.

[0096] Next, in step S104, the processing unit 120 performs database update processing. Furthermore, the processing in step S104 does not need to be performed during surgery; for example, it can be performed after the surgery is completed.

[0097] Next, in step S105, after incrementing the variable i representing the frame, the process returns to step S101. That is, the image of the next frame is taken as the object, and the processing of steps S101 to S104 continues.

[0098] 3.2 Bleeding Assessment and Management

[0099] Figure 8 This is an explanation Figure 7 The flowchart below shows the bleeding determination process in step S102. When this process begins, in step S201, the bleeding detection unit 121 performs bleeding detection processing. In step S202, the bleeding point detection unit 123 performs bleeding point detection processing. In step S203, the treatment decision unit 125 performs hemostasis treatment decision processing. The processing of each step will be described in detail below.

[0100] 3.2.1 Bleeding Detection and Management

[0101] Figure 9 This is a flowchart illustrating the bleeding detection process in step S201. First, in step S301, the bleeding detection unit 121 performs a comparison process between the image Pi to be processed and images Pi to n that are n frames before Pi. Images Pi to n are, for example, images acquired one second before image Pi, but various modifications can be applied to the acquisition time of the images used as comparison objects.

[0102] Next, in step S302, the bleed detection unit 121 determines whether the change in red areas between images is significant. For example, the bleed detection unit 121 uses a known matching method such as block matching to determine the areas where changes occur between images. Furthermore, if the area of ​​the changed region is above a predetermined threshold, it determines whether that region is a red region. For example, the bleed detection unit 121 determines regions with hues within a predetermined angular range including 0 degrees as red regions.

[0103] In a case where it is determined that the area of the region in which a change occurs between images is equal to or greater than the threshold value and the region is a red region, in step S303, the bleeding detection unit 121 determines that there is bleeding in the image Pi. In other cases, in step S304, the bleeding detection unit 121 determines that there is no bleeding in the image Pi.

[0104] 3.2.2 Bleeding point detection processing

[0105] Figure 10 、 Figure 11 is a flowchart illustrating the bleeding point detection processing in step S202. First, in step S401 of Figure 10 , the bleeding point detection unit 123 determines whether or not bleeding is detected in the image Pi. In a case where the determination in step S401 is NO, the bleeding point detection unit 123 ends the processing without performing the processing after step S402.

[0106] In a case where the determination in step S401 is YES, in step S402, the bleeding point detection unit 123 determines a bleeding region S in the image Pi. The bleeding region here is, for example, a region in which a predetermined number of pixels or more in a region that is a set of continuous red pixels in the image Pi. Further, in a case where a plurality of bleeding regions exist in the image Pi, the bleeding point detection unit 123 distinguishes each of the bleeding regions as S1, S2,....

[0107] In step S403, the bleeding point detection unit 123 initializes a variable k that determines a bleeding region to 1. Then, in step S404, the bleeding point detection unit 123 executes the bleeding point detection processing with the bleeding region Sk as a target. After the processing in step S404, in step S405, the bleeding point detection unit 123 increments k and returns to the processing in step S404. That is, the bleeding point detection unit 123 sequentially executes the bleeding point detection processing on one or a plurality of bleeding regions detected in the image Pi. In a case where the bleeding point detection processing for all of the bleeding regions ends, Figure 10 the processing illustrated in Fig. 7 ends.

[0108] Figure 11This is a flowchart illustrating the bleed point detection process targeting the bleed region Sk in step S404. In step S501, the bleed point detection unit 123 initializes the variable j used for bleed point search to 1. In step S502, the bleed point detection unit 123 determines whether a bleed region Sk, the target of processing, exists in image Pi-j, which is j frames earlier than image Pi. For example, the bleed point detection unit 123 detects the bleed region in image Pi-j and calculates the centroid of the bleed region, and compares the centroid with the centroid of Sk in image Pi-j+1. If the difference is below a predetermined threshold, the bleed point detection unit 123 determines that the bleed region in images Pi-j is the bleed region Sk, the target of processing. Furthermore, if it is assumed that the subject moves significantly in the image within a period equivalent to one frame, the bleed point detection unit 123 may perform the centroid comparison processing after performing motion vector-based correction.

[0109] If the condition is "yes" in step S502, then in step S503, the bleed point detection unit 123 increments the variable j. After processing in step S503, the bleed point detection unit 123 returns to step S502 to continue processing. That is, it further backtracks in the previous direction to determine whether the bleed area Sk exists.

[0110] Figure 12 (A) is a schematic diagram illustrating the bleeding point detection process. For example... Figure 12 As shown in (A), the bleeding point detection unit 123 searches for the bleeding region Sk detected in the image Pi in the previous direction. Figure 12 Illustration (A) shows the case where "No" is true in step S502, i.e., Sk is detected in image Pi-j+1 but not in image Pi-j. In this case, in step S505, the bleed point detection unit 123 sets image Pi-j+1 as the bleed start image Ps corresponding to the bleed start frame. That is, the bleed start frame is the frame j-1 frames earlier than the latest frame i.

[0111] After determining the bleeding start image Ps, in step S505, the bleeding point detection unit 123 determines the bleeding point. Figure 12 (B) is a schematic diagram illustrating the process of determining the bleeding point (xs, ys). Figure 12 As shown in (B), the bleed detection unit 123 performs the process of detecting the centroid (xs, ys) of the bleed region Sk in image Ps as a bleed point. Additionally, in Figure 12 In (B), the upper left endpoint of the image is set as the origin, but the coordinate system is set arbitrarily.

[0112] Figure 13 This is to explain the passage. Figure 10-12Fig. 2 is a diagram showing the structure of data acquired by the bleeding point detection process shown in (B) of Fig. 1. As shown in (A) of Fig. 2, the bleeding detection process is performed on the basis of the image Pi in the frame i, and one or a plurality of bleeding regions are detected. The information indicating the bleeding start frame is associated with each of the bleeding regions. The bleeding start frame is determined as i-j+1, for example, as shown in (B) of Fig. 2. In addition, as shown in (C) of Fig. 2, the coordinate (xs, ys) of the bleeding start point is associated with each of the bleeding regions. Figure 13 Figure 11 Figure 12

[0113] As described above, the processing section 120 performs the bleeding detection process of detecting whether or not bleeding has occurred in the living body on the basis of the processing target image sequence. Further, the processing section 120 performs the bleeding point detection process of determining the bleeding point, i.e., the position at which bleeding has occurred, when bleeding is detected. In this way, it is possible to detect the occurrence of bleeding and determine the bleeding point, and thus it is possible to appropriately support the hemostatic treatment performed by the user.

[0114] 3.2.3 Decision process of hemostatic treatment

[0115] The decision process of the hemostatic treatment in step S203 will be described. Note that a specific example of the database and a learning process using the database will also be described as a premise of the decision process of the hemostatic treatment.

[0116] Hereinafter, an example in which the image of the living body included in the processing target image sequence and the database is an image of the liver will be described. The processing section 120 decides the hemostatic treatment for any blood vessel among the hepatic artery, the hepatic vein, and the portal vein on the basis of the processing target image sequence and the database. In this way, in a surgery in which the liver is the target, such as a liver partial resection, it is possible to appropriately support the user's response to unexpected bleeding. In particular, the liver is prone to bleeding due to the high density of blood vessels, and if the hemostatic treatment is not appropriately performed, it can become a poor prognosis. Therefore, it is very useful to support the hemostatic treatment related to the liver.

[0117] The processing section 120 can also determine the type of the blood vessel in which bleeding has occurred on the basis of the processing target image sequence and the database, and decide the hemostatic treatment on the basis of the determined type. In this way, by determining the type of the blood vessel in which bleeding has occurred, it is possible to prompt the user of an appropriate hemostatic treatment corresponding to the type.

[0118] Here, the type of the blood vessel indicates a structural division or a functional division in the field of anatomy, vasculature, or the like, for example. As shown in (A) of Fig. 3, the type of the blood vessel is determined on the basis of the image Pi in the frame i. The type of the blood vessel is determined as the hepatic artery, for example, as shown in (B) of Fig. 3. Figure 1 ​​​As types of blood vessels related to the liver, as exemplified, the portal vein that guides blood from the digestive organ system to the liver, the hepatic artery that branches from the abdominal aorta, and the hepatic vein that branches from the inferior vena cava are considered. However, the types in the present embodiment can also divide blood vessels more finely. For example, the relatively thick hepatic artery and the thin hepatic artery can also be set as different types. Or, in a liver partial resection, the liver is divided into the right lobe, the left lobe, and the caudate lobe, and further the right lobe and the left lobe are each divided into a plurality of regions. The types of blood vessels in the present embodiment can also consider which region a blood vessel is connected to. For example, in the present embodiment, the right hepatic vein, the middle hepatic vein, and the left hepatic vein can also be set as different types, respectively.

[0119] Figure 14 (A) of FIG. 12, Figure 14 (B) is a diagram that shows an example of the database of the present embodiment. In the database generation processing described above with reference to Figure 4 (B) of FIG. 12, as Figure 14 (A) shows, information of the blood vessel type of the blood vessel from which bleeding has occurred is associated with the image sequence of the bleeding pattern. That is, in step S12 of Figure 4 (A) of FIG. 12, the database shown in Figure 14 (A) is acquired by performing annotation that adds the blood vessel type as metadata.

[0120] In addition, as shown in (B) of FIG. 12, the database contains data that associates the blood vessel type and the hemostatic treatment corresponding to the blood vessel type. For example Figure 14 (B) is a table that contains data of the number of rows corresponding to the blood vessel type that is the object, and one hemostatic treatment corresponds to one blood vessel type. Note that, as the hemostatic treatment, suturing that sutures a blood vessel, sealing that uses an energy device to cauterize a blood vessel, clip hemostasis that mechanically performs compression by clamping the vicinity of a bleeding point with a clip, injection, and the like that disperses a medicament having a blood vessel constricting action can be considered. Note that the number of blood vessel types does not need to coincide with the number of types of hemostatic treatments, and it is also no problem that one hemostatic treatment is associated with a plurality of blood vessel types. Figure 14

[0121] ​The determination processing of the hemostatic treatment of the present embodiment can also use machine learning. Specifically, the learning device 500 generates a learned model by performing machine learning based on a database. The processing section 120 of the image processing system 100 operates in accordance with the learned model generated by the learning device 500, whereby the hemostatic treatment is determined. Hereinafter, machine learning using a neural network will be described, but the method of the present embodiment is not limited thereto. In the present embodiment, for example, machine learning using other models such as a support vector machine (SVM) can be performed, and machine learning using a method obtained by developing various methods such as a neural network and an SVM can also be performed.

[0122] Figure 15 (A) of FIG. 10 is a schematic diagram illustrating a neural network. The neural network has an input layer that inputs data, an intermediate layer that performs an operation based on an output from the input layer, and an output layer that outputs data based on an output from the intermediate layer. In Figure 15 In (A) of FIG. 10, a network in which the intermediate layer is two layers is exemplified, but the intermediate layer can be one layer or three or more layers. In addition, the number of nodes (neurons) included in each layer is not limited to Figure 15 Various modifications can be implemented with respect to the example of (A) of FIG. 10. In addition, if the accuracy is considered, the learning of the present embodiment preferably uses deep learning (deep learning) using a multilayer neural network. The multilayer here is four or more layers in the narrow sense.

[0123] As illustrated in (A) of FIG. 10, the nodes included in a given layer are coupled with the nodes of the adjacent layer. A weighting coefficient is set for each coupling. Each node multiplies the output of the node of the previous stage by the weighting coefficient, and calculates a sum value of the multiplication result. Further, each node adds a bias to the sum value, and applies an activation function to the addition result, whereby the output of the node is calculated. By sequentially performing this processing from the input layer toward the output layer, the output of the neural network is calculated. In addition, as the activation function, various functions such as a sigmoid function and a ReLU function are known, and these functions can be widely applied in the present embodiment. Figure 15 The learning in the neural network is processing of determining appropriate weighting coefficients (including a bias). Specifically, the learning device 500 inputs the input data in the training data to the neural network, and calculates the output by performing a forward operation using the weighting coefficients at that time. The learning device 500 calculates an error function based on the output and the correct answer label in the training data. Then, the weighting coefficients are updated so as to reduce the error function. In the update of the weighting coefficients, for example, an error backpropagation method in which the weighting coefficients are updated from the output layer toward the input layer can be used.

[0124]

[0125] ​In addition, a neural network can be, for example, a CNN (Convolutional Neural Network). Figure 15 (B) is a schematic diagram illustrating a CNN. A CNN contains convolutional layers and pooling layers that perform convolutional operations. Convolutional layers are layers that perform filtering. Pooling layers are layers that perform pooling operations to reduce the vertical and horizontal dimensions. Figure 15 The example shown in (B) is a network that, after multiple operations based on convolutional and pooling layers, yields its output through operations based on fully connected layers. A fully connected layer is a layer that performs operations that map all nodes from the previous layer to nodes in the given layer, and is used in conjunction with the operations described above. Figure 15 The operations at each level described in (A) correspond to those in the previous section. Additionally, in Figure 15 In (B), the activation function-based computation is omitted. CNNs are known to have various architectures, and these architectures can be widely applied in this implementation.

[0126] When using CNNs, the processing steps are also different. Figure 15 The same applies to (A). That is, the learning device 500 inputs the input data from the training data into the CNN, and obtains the output by performing filtering or pooling operations using the filtering characteristics at that time. Based on the output and the positive label, the error function is calculated, and the weighting coefficients containing the filtering characteristics are updated to reduce the error function. When updating the weighting coefficients of the CNN, for example, the backpropagation method can also be used.

[0127] Figure 16 This is an example of use. Figure 14 The graph in (A) shows the input and output of the neural network in the case of the database. Figure 16 As shown, the input data of the neural network is an image sequence, and the output data is information used to determine the blood vessel type. The input image sequence can be of fixed length, but depending on the structure of the neural network, a variable length is also acceptable. In the case where the output layer of the neural network is a known softmax layer, the output data represents the probability of each blood vessel type. Figure 16 In the example, the output data consists of three data points: "probability data indicating the portal vein", "probability data indicating the hepatic artery", and "probability data indicating the hepatic vein" for the type of bleeding vessel in the image sequence that was input.

[0128] Learning device 500 from Figure 14The database shown in (A) retrieves image sequences of bleeding patterns and corresponding vessel type information. Next, the image sequences are input into the neural network, and a forward operation is performed using the current weighting coefficients, thereby obtaining three probability data points. In the case where the vessel type information is "portal vein," the positive solution data is data where the probability of "portal vein" is 1, and both the probability of "hepatic artery" and "hepatic vein" are 0. The learning device 500 calculates the error between the three probability data points obtained through the forward operation and the positive solution as an error function. In the update processing of the weighting coefficients used to reduce the error function, the error backpropagation method, as described above, is used. The above is based on... Figure 14 The processing is performed on one row of data contained in the database (A). The learning device 500 generates a learned model by repeatedly performing this processing. In addition, various methods such as batch learning and mini-batch learning are known in learning processing, and these methods can be widely applied in this embodiment.

[0129] Figure 17 This is a flowchart illustrating the decision-making process for hemostasis in step S203. Furthermore, it is assumed that at the beginning... Figure 17 Before further processing, the learned model was generated through the processing described above.

[0130] First, in step S601, the processing decision unit 125 determines whether bleeding has occurred in the latest image Pi. For example, the processing decision unit 125 performs the processing in step S601 by obtaining the processing result of the bleeding detection unit 121 in step S201. If no bleeding has occurred, the processing decision unit 125 skips the steps after step S602 and ends the processing.

[0131] If bleeding is determined to have occurred, in step S602, the processing decision unit 125 performs processing to extract an image sequence to be input into the learned model from the processing object image sequence. Here, the image sequence refers to images representing the bleeding condition. The starting point of the image sequence is, for example, image Ps. The ending point of the image sequence is, for example, image Pi. However, the range of the extracted image sequence is not limited to this; at least one of the starting point and the ending point may be different. Furthermore, it is not limited to including all frames from the starting point to the ending point in the image sequence; a portion of the images along the way may be omitted.

[0132] In step S603, the treatment decision unit 125 acquires the learned model generated by the learning device 500. The processing of step S603 can also be processing of receiving the learned model from the database server 300. Alternatively, the treatment decision unit 125 can also acquire the learned model from the database server 300 in advance and store it in the storage unit 130. In this case, step S603 corresponds to processing of reading out the stored learned model. In addition, the processing order of steps S602 and S603 is not limited to this, and step S603 can be performed before S602, or the two processes can be performed in parallel.

[0133] In step S604, the treatment decision unit 125 performs inference processing based on the learned model. Specifically, the treatment decision unit 125 acquires output data by inputting the extracted image sequence to the learned model. The processing of step S604 is forward operation including, for example, filter processing corresponding to a convolution layer, and the like. The output data is, for example, three pieces of probability data as described above using Figure 16 .

[0134] In step S605, the treatment decision unit 125 determines the type of the blood vessel in which bleeding has occurred, based on the output data. The processing of step S605 is, for example, processing of selecting the data having the largest value among the three pieces of probability data.

[0135] In step S606, the treatment decision unit 125 decides the hemostatic treatment based on the determined type of the blood vessel. For example, the treatment decision unit 125 decides the hemostatic treatment by performing comparison processing of the database shown in (B) of Figure 14 with the type of the blood vessel determined in step S605. If it is an example of (B) of Figure 14 , in a case where it is determined that the type of the blood vessel is a portal vein, the hemostatic treatment is suturing.

[0136] As described above, the database of the present embodiment can also include a plurality of data sets that are data sets obtained by corresponding, to the blood vessel type information indicating in which type of blood vessel bleeding photographed in the in-vivo image sequence occurs, the in-vivo image sequence that is the in-vivo image in time series. The database is, for example, (A) of Figure 14 . The processing unit 120 determines the type of the blood vessel in which bleeding has occurred, based on the learned model and the processing target image sequence. The learned model is a learned model that has performed machine learning on the relationship between the in-vivo image sequence and the blood vessel type information based on a plurality of the above-described data sets. Further, the processing unit 120 decides the hemostatic treatment based on the determined type.

[0137] By thus using the learned model to determine the blood vessel type, the blood vessel type of a blood vessel in which bleeding has occurred can be estimated with high precision. In addition, the user can be prompted about the hemostasis treatment corresponding to the blood vessel type.

[0138] Further, the operation in the processing section 120 in accordance with the learned model, that is, the operation for outputting output data based on input data, can be executed by software or by hardware. In other words, in the case of software, the processing section 120 can be implemented by a CPU or the like. In the case of hardware, the processing section 120 can be implemented by a circuit device such as an FPGA or the like. Figure 15 The product-sum operation performed in each node of (A) of the CNN, the filter processing performed in the convolution layer of the CNN, and the like can also be executed in a software manner. Alternatively, the above operations can also be executed by a circuit device such as an FPGA or the like. Further, the above operations can also be executed by a combination of software and hardware. In this way, the operation of the processing section 120 in accordance with the instructions from the learned model stored in the storage section 130 can be implemented in various ways. For example, the learned model includes a derivation algorithm and parameters used in the derivation algorithm. The derivation algorithm is an algorithm that performs product-sum operations and the like based on input data. The parameters are parameters obtained through learning processing, and are, for example, weighting coefficients. In this case, both the derivation algorithm and the parameters can be stored in the storage section 130, and the processing section 120 performs derivation processing in a software manner by reading out the derivation algorithm and the parameters. Alternatively, the derivation algorithm can be implemented by an FPGA or the like, and the storage section 130 stores the parameters.

[0139] 3.3 Display processing

[0140] Figure 18 is a flowchart illustrating the display processing in step S103. First, in step S701, the display processing section 127 obtains the bleeding point (xs, ys) in the bleeding start image Ps of the bleeding point that is the object in the surgery being currently performed. For example, the display processing section 127 determines the coordinates of the bleeding point in the image Ps by reading out the data shown in Figure 13 .

[0141] In step S702, the display processing section 127 performs processing of correcting the bleeding point based on the amount of movement of the subject between images. Alternatively, the bleeding point detection section 123 can perform the correction processing of the bleeding point, and the display processing section 127 can obtain the correction processing result.

[0142] For example, the display processing section 127 calculates the movement amount of the subject between the image Ps and the image Pi that is the latest frame to be displayed. The movement amount here is, for example, a motion vector. The motion vector can be found, for example, by block matching or the like. By performing correction processing of the position of the bleeding point in the image Ps based on the found motion vector, the position of the bleeding point in the image Pi can be determined. In addition, the correction processing of the bleeding point is not limited to processing that directly performs correction processing between the image Ps and the image Pi. For example, the display processing section 127 performs processing of finding a motion vector between the image Ps and the image Ps+1 and determining a bleeding point in the image Ps+1 based on the motion vector. Thereafter, the display processing section 127 can also determine a bleeding point in the image Pi by correcting the bleeding point frame by frame. In addition, the display processing section 127 can also correct a bleeding point by performing processing of determining a bleeding point every 2 frames, and various modifications can be implemented as a specific processing.

[0143] In step S703, the display processing section 127 generates a display image from the image to be displayed and information of the determined bleeding point for the image. Specifically, the display processing section 127 performs display related to the bleeding point by attaching a result of correction of the bleeding point (xs, ys) in the bleeding start image Ps based on the motion vector to the image Pi. The display processing section 127, for example, performs processing of overlapping a marker object that explicitly shows the position of the bleeding point on the image to be displayed.

[0144] Figure 19 is an example of a display image for prompting the user of the determined hemostasis treatment. As shown in Figure 19 , the display image is, for example, an image in which information indicating the hemostasis treatment is overlapped on an image obtained by photographing the inside of the living body. In Figure 19 , by overlapping a text such as "clip hemostasis", the clip hemostasis is prompted as the hemostasis treatment. However, the overlapped information is not limited to the text, and can be an image, an icon. In addition, the prompting processing of the hemostasis treatment can also be performed by a method other than display. For example, the processing section 120 can also prompt the user of the hemostasis treatment using sound or the like.

[0145] In addition, as shown in Figure 19 , information indicating the bleeding point and information indicating the type of the blood vessel in which bleeding has occurred can also be overlapped on the display image. By displaying such additional information, the user can be prompted of the bleeding condition in an easily understandable manner. For example, by displaying the bleeding point, the user can be made to grasp the position at which the hemostasis treatment should be performed.

[0146] As described above, the processing unit 120 can also perform bleeding notification processing, which involves informing the user of information related to bleeding points identified through the bleeding point detection processing. By alerting the user to the bleeding points, it becomes easier for the user to perform hemostasis procedures. Furthermore, in Figure 18 The example shown illustrates the display of bleeding points, but bleeding notification processing can also be included in the bleeding point detection process, where information related to the occurrence of bleeding is reported when bleeding is detected. Alternatively, bleeding notification processing can be a process that reports both information related to the occurrence of bleeding and information related to the bleeding point itself.

[0147] 3.4 Database Update Processing

[0148] In addition, such as Figure 7 As shown in step S104, the processing unit 120 performs a process of adding a dataset to the database that corresponds to the processing object image sequence used to determine hemostasis treatment and the hemostasis treatment for bleeding detected in the processing object image sequence. In this way, the data stored in the database is expanded as the surgery is performed. This improves the accuracy of processing using the database. In the case of machine learning, by increasing the amount of training data, a learned model capable of performing high-precision derivation processing can be generated.

[0149] The database update process is based on Figure 4 The database generation process shown is used as a reference. In step S602 of the hemostasis treatment decision process, the image sequence is extracted, so the processing in step S11 can use the processing result of S602. For example, the image sequence stored in the database is an image sequence starting from image Ps and ending at image Pi. However, it is acceptable for the user to readjust the range of the image sequence or perform image sequence extraction processing again.

[0150] Furthermore, in step S606 of the hemostasis treatment decision process, a hemostasis treatment is determined. If hemostasis is appropriately achieved through this hemostasis treatment, the metadata assigned in step S12 becomes information representing the hemostasis treatment actually performed. Additionally, if the hemostasis effect of the performed hemostasis treatment is insufficient, processing can be performed to correlate information indicating this, information representing other hemostasis treatments deemed more ideal by a skilled physician, etc., with the image sequence.

[0151] The association of additional information in step S13, the registration process to the database in step S14, and... Figure 4 The examples are the same.

[0152] 4. Variations

[0153] The following are some examples of variations.

[0154] 4.1 Variant of liver partial resection

[0155] As described above, the processing target image sequence and the in-vivo image included in the database can also be images of the liver. Specifically, the surgery envisaged in the present embodiment can also be surgery in the liver, or in a narrow sense, liver partial resection.

[0156] In liver partial resection, a method called Pringle's maneuver is used. This is a method of blocking blood flow by clamping the hepatic artery and portal vein with a clamp forceps. By using Pringle's method, bleeding during surgery can be suppressed. Massive bleeding that occurs during the implementation of Pringle's maneuver is highly likely to be bleeding from the hepatic vein. On the other hand, in the case where bleeding occurs after Pringle's maneuver is released, the likelihood of bleeding from the portal vein or hepatic artery increases.

[0157] Therefore, the processing section 120 determines whether Pringle's maneuver of blocking blood flow of the hepatic artery and portal vein is being performed, and decides the treatment of hemostasis based on the determination result. In this way, it is possible to make a determination considering whether Pringle's maneuver is in progress or after Pringle's maneuver is released, and thus it is possible to perform the decision processing of the treatment of hemostasis with high accuracy.

[0158] For example, input indicating whether Pringle's maneuver is in progress or after Pringle's maneuver is released can also be made by the user. The processing section 120 determines whether Pringle's maneuver is in progress or not according to the user input. However, the processing section 120 can also determine whether Pringle's maneuver is in progress or not based on image processing.

[0159] For example, the database can include two tables, i.e., a first table that corresponds vascular type information to image sequences in the case where bleeding occurs during Pringle's maneuver, and a second table that corresponds vascular type information to image sequences in the case where bleeding occurs after Pringle's maneuver is released. The learning device 500 generates a first learned model based on the first table, and a second learned model based on the second table.

[0160] The processing section 120 determines whether Pringle's maneuver of blocking blood flow of the hepatic artery and portal vein is in progress, and decides the treatment of hemostasis based on the first learned model in the case where it is determined that Pringle's maneuver is in progress, and decides the treatment of hemostasis based on the second learned model in the case where it is determined that Pringle's maneuver is released. In this way, it is possible to output a treatment result corresponding to Pringle's maneuver.

[0161] Alternatively, the processing unit 120 can determine the first hemostasis procedure when the Pringle maneuver is performed and the second hemostasis procedure when the Pringle maneuver is not performed, based on the image sequence of the processed object and a database. For example, the display processing unit 127 displays both hemostasis procedures. For example, it records two pieces of information together: "During Pringle maneuver: suture" and "After Pringle maneuver is removed: clamp hemostasis". The user during the operation can then determine whether the Pringle maneuver is in progress or after it has been removed. Therefore, even when two pieces of information are displayed, the user can easily understand which information to refer to.

[0162] For example, the processing unit 120 inputs the image sequence of the object to be processed into both the first learned model and the second learned model. The processing unit 120 determines the first hemostasis treatment based on the output of the first learned model and determines the second hemostasis treatment based on the output of the second learned model.

[0163] 4.2 Variations of Machine Learning

[0164] Figure 20 (A) is a diagram illustrating other structures of the database. For example... Figure 20 As shown in (A), vascular type information, representing the type of blood vessel causing bleeding, and information representing recommended hemostasis are mapped to an image sequence of the bleeding pattern. Specifically, in Figure 4 In step S12, additional vessel type and hemostasis treatment are annotated as metadata to obtain... Figure 20 The database shown in (A) is as follows.

[0165] Figure 20 (B) is an example of use Figure 20 The graph in (A) shows the input and output of the neural network in the case of the database. Figure 20 As shown in (B), the input data of the neural network is an image sequence, and the output data consists of information for determining the blood vessel type and information for determining hemostasis treatment. The information used to determine the blood vessel type includes, for example, information related to... Figure 16 Similarly, the example is probabilistic data representing the likelihood of different blood vessel types. Information used to determine hemostasis procedures is, for example, probabilistic data representing the likelihood of recommending each hemostasis procedure. Information used to determine hemostasis procedures includes, for example, "probability data for recommending sutures," "probability data for recommending clip hemostasis," and "probability data for recommending cauterization sealing," etc.

[0166] Learning device 500 Figure 20the image sequence in (A) as input, performs machine learning of information indicating a blood vessel type and information indicating a recommended hemostatic treatment as a correct answer label, thereby generating a learned model. The treatment decision unit 125 of the image processing system 100 inputs the processing target image sequence to the learned model. In the case of using the neural network of (B), both the blood vessel type and the hemostatic treatment can be decided as output of the learned model. Figure 20

[0167] It is not limited to determining the blood vessel type and deciding the hemostatic treatment in stages, but can be performed in parallel as shown in (B). In this case, the blood vessel type and the hemostatic treatment suitable for the blood vessel type can also be prompted to the user. Note that, as shown in (A), different hemostatic treatments can be associated with the same type of blood vessel in the database here. Therefore, even the same type of blood vessel can be flexibly changed in the hemostatic treatment according to the situation. Figure 20 Figure 20

[0168] Note that the method of the present embodiment is a method of prompting the recommended hemostatic treatment to the user. Therefore, the decision and the prompt of the blood vessel type are not necessary. Figure 21 (A) is a diagram illustrating another structure of the database. As shown in (A), the information indicating the recommended hemostatic treatment can be associated with the image sequence of the bleeding pattern. Figure 21

[0169] Figure 21 (B) is a diagram illustrating the input and the output of the neural network in the case of using the database shown in (A). As shown in (B), the input data of the neural network is the image sequence, and the output data is the information for determining the hemostatic treatment. Even in the case where the blood vessel type is omitted like this, the decision and the prompt of the recommended hemostatic treatment can be performed. Figure 21 Figure 21 As described using (A) and (A), the database can include a plurality of data sets, the data set being a data set obtained by associating the hemostatic treatment information indicating the recommended hemostatic treatment for the bleeding photographed in the in-vivo image sequence with the in-vivo image sequence which is the in-vivo image in time series. The processing unit 120 performs processing of deciding the hemostatic treatment based on a learned model obtained by performing machine learning of the relationship between the in-vivo image sequence and the hemostatic treatment information based on the plurality of data sets and the processing target image sequence. In this way, the information indicating the hemostatic treatment can be obtained as output of the learned model.

[0170] Figure 20 Figure 21

[0171] ​​​​​​​​Further, the above describes an example in which the input to the learned model is an image sequence, but the method of the present embodiment is not limited thereto. For example, a feature quantity calculated based on each image included in the image sequence can be used as the input to the learned model. The feature quantity can be information related to hue, saturation, brightness, and the like. Alternatively, the feature quantity can be information indicating a time-series change in bleeding speed. The bleeding speed indicates the amount of bleeding per unit time. As described above, a region in which blood exists is captured as a red region. Although it is difficult to accurately estimate the absolute amount of bleeding, the area of the red region is information that is an index of the amount of bleeding. Thus, a change in the area of the red region can be used as an index of the bleeding speed. By using the bleeding speed itself, it is possible to determine the intensity of bleeding. Further, by observing the degree of change in the bleeding speed, it is possible to determine whether there is pulsation. For example, the feature quantity indicating the bleeding state is a time change waveform of the bleeding speed. However, the feature quantity can be a statistical quantity such as the average value, maximum value, or the like of the bleeding speed, a statistical quantity of the change in the bleeding speed, or a combination thereof, and various modifications can be implemented for the specific manner.

[0172] Note that the output data obtained by inputting the feature quantity to the neural network can be the blood vessel type, the hemostasis treatment, or both, as described above.

[0173] Figure 22 is a flowchart illustrating a decision process of the hemostasis treatment when the feature quantity is used as the input data.

[0174] Figure 22 Steps S801 and S802 of Figure 17 are the same as steps S601 and S602 of Next, in step S803, the treatment decision unit 125 calculates a feature quantity from the extracted image sequence. For example, the treatment decision unit 125 calculates a feature quantity corresponding to the bleeding speed from the time change of the red region in the image.

[0175] In step S804, the treatment decision unit 125 acquires the learned model generated by the learning device 500. In step S805, the treatment decision unit 125 performs an inference process based on the learned model. Specifically, the treatment decision unit 125 acquires output data by inputting the calculated feature quantity to the learned model. The processes of steps S806 and S807 are the same as those of steps S606 and S607 of Figure 17

[0176] 4.3 Modification of the Database-Based Process

[0177] The above describes an example in which the decision process of the hemostasis treatment based on the database is a process using machine learning. However, the method of the present embodiment is not limited to machine learning.

[0178] The processing section 120 can also perform the decision processing of the hemostatic treatment by comparing the in-vivo image sequence included in the database with the processing target image sequence. As for the comparison of the images with each other, for example, there is a method of calculating the hue, saturation, and luminance histograms of the images respectively, and comparing the histograms. Here, since the in-vivo image sequence and the processing target image sequence each have a plurality of images, the processing section 120, for example, calculates the similarity between the image sequences by repeating the comparison processing of 2 images a plurality of times.

[0179] The processing section 120 decides the image sequence most similar to the processing target image sequence from among the plurality of in-vivo image sequences included in the database. In the case of using the database shown in (A) of FIG. 10, the processing section 120 decides the in-vivo image sequence most similar to the processing target image sequence. The processing section 120 then determines the blood vessel type corresponding to the decided in-vivo image sequence, and determines the hemostatic treatment corresponding to the blood vessel type. Figure 14 In the case of using the database shown in (A) of FIG. 10, the processing section 120 determines that bleeding has occurred in the blood vessel type corresponding to the decided in-vivo image sequence. The processing after the blood vessel type decision is the same as in the above-described example. Also in the case of using the database shown in (A) of FIG. 10, the processing section 120 determines the blood vessel type corresponding to the decided in-vivo image sequence, and determines the hemostatic treatment corresponding to the blood vessel type. Figure 20 Also in the case of using the database shown in (A) of FIG. 10, the processing section 120 determines the blood vessel type corresponding to the decided in-vivo image sequence, and determines the hemostatic treatment corresponding to the blood vessel type. Figure 21 Also in the case of using the database shown in (A) of FIG. 10, the processing section 120 determines the blood vessel type corresponding to the decided in-vivo image sequence, and determines the hemostatic treatment corresponding to the blood vessel type.

[0180] In the case of comparing the database and the processing target image sequence, the above-described feature quantity or the like can be calculated each time. However, in the generation processing of the database, the feature quantity can be calculated from the in-vivo image sequence, and a database in which the feature quantity is associated with the blood vessel type, the hemostatic treatment, or the like can be generated. For example, the database generation device 400 generates a database in which the blood vessel type information is associated with the feature quantity such as the histogram, instead of the database shown in (A) of FIG. 10, and transmits the database to the database server 300. Figure 14

[0181] Also, the comparison of the in-vivo image sequence and the processing target image sequence is processing of searching for an image sequence similar in bleeding condition. Therefore, the feature quantity used for the comparison can also use a feature quantity indicating bleeding in the image. The feature quantity indicating bleeding is the amount of bleeding, the bleeding speed, the time variation of the bleeding speed, or the like.

[0182] Figure 23 (A) of FIG. 11 is a diagram illustrating another structure of the database. As shown in (A) of FIG. 11, information indicating the blood vessel type of the blood vessel in which bleeding has occurred is associated with the time variation waveform of the bleeding speed. For example, the database generation device 400 performs the following processing: the area of the red region is calculated from each image of the in-vivo image sequence, and the time variation waveform of the bleeding speed is calculated from the differential information. The differential information is, for example, the difference between 2 adjacent images. Figure 23

[0183] ​​The treatment decision unit 125 calculates the area of ​​the red region and the differential information in the image sequence of the processing object to obtain the time-varying waveform of the bleeding velocity. Next, the treatment decision unit 125 calculates the similarity between the time-varying waveforms of the bleeding velocity. Then, it determines that bleeding has occurred in the blood vessel type corresponding to the time-varying waveform with the highest similarity. The processing after determining the blood vessel type is the same as in the example described above. For example, as... Figure 23 As shown in (B), the database contains data that maps blood vessel types to the corresponding hemostatic treatments. Figure 23 (B) and Figure 14 (B) is the same. The decision-making department 125 is based on... Figure 23 The type of blood vessel determined by (A) and Figure 23 The comparison of the treatments shown in table (B) determines the hemostatic treatment.

[0184] 4.4 Emergency Hemostasis Treatment

[0185] Furthermore, the above example illustrates how, when bleeding is detected in the latest frame's image Pi, the hemostasis treatment is determined based on the image sequence starting from the bleeding in image Ps and ending at image Pi. However, the hemostasis treatment can also be determined based on whether the user performs emergency hemostasis procedures.

[0186] For example, the disposal decision department 125 replaced Figure 17 Step S601 or Figure 22 The processing in step S801 determines whether display processing related to bleeding points was performed in the previous frame, and whether the user performed emergency hemostasis.

[0187] pass Figure 7 The display of bleeding points in step S103 prompts the user to perform emergency hemostasis. Emergency hemostasis refers to a procedure that is emergency compared to the hemostasis procedure determined through the decision-making process of this embodiment. Specifically, emergency hemostasis refers to compression hemostasis using a covering of a treatment instrument or gauze inserted into the abdominal cavity. This method uses pressure to obstruct blood flow and suppress bleeding, and its hemostatic effect is relatively low compared to methods such as clamping, cauterization, and suturing, which are typically indicated as hemostasis procedures. However, since emergency hemostasis can suppress massive bleeding, it is easier to perform highly effective hemostatic procedures such as clamping. Therefore, it is effective to first display the bleeding point to prompt emergency hemostasis.

[0188] If the treatment decision unit 125 determines that emergency hemostasis has been performed, it executes processing steps after S602 or S802. For example, the determination of whether emergency hemostasis has been performed can be based on user input. For instance, in the display screen of step S103 or in the guidance of the endoscope system 200, the user may be prompted to perform a prescribed input operation after the emergency hemostasis has been completed. Alternatively, the determination of whether emergency hemostasis has been performed can be made through image processing. Since bleeding is temporarily suppressed by emergency hemostasis, for example, if the processing unit 120 determines that emergency hemostasis has been performed when the area of ​​the red region is sufficiently smaller compared to the point when bleeding was determined to have occurred, then emergency hemostasis has been performed.

[0189] In this way, when the processing unit 120 determines that the user has performed emergency hemostasis for bleeding, it can determine the hemostasis treatment based on the image sequence of the processing object and the database, and decide the treatment with higher hemostasis ability than the emergency hemostasis treatment as the hemostasis treatment.

[0190] Bleeding was suppressed through emergency hemostasis. However, the effectiveness of this emergency hemostasis varies depending on the type and extent of the damaged blood vessel. By determining the appropriate hemostasis treatment after emergency hemostasis, we can consider not only the extent of bleeding but also the degree to which the emergency hemostasis brought the bleeding to a stop. This improves the accuracy of hemostasis estimation. Depending on the situation, if the emergency hemostasis has sufficiently stopped the bleeding, it may be determined that no further hemostasis is necessary.

[0191] In addition, regarding the use of the endoscope system 200, such as Figure 7 As shown in step S103, consider that the processing unit 120 first performs bleeding notification processing, and the user performs emergency hemostasis based on the bleeding report processing. Therefore, the processing unit 120 can also perform processing based on the image sequence of the processing object and the database to determine the hemostasis treatment after the bleeding report processing and the emergency hemostasis treatment. However, in this embodiment, the bleeding notification processing is not necessary and can be omitted.

[0192] More specifically, the processing unit 120 can also perform processing based on a sequence of images of the processing object, including at least those captured during or after the execution of emergency hemostasis by the user, to determine the hemostasis procedure.

[0193] In this way, images depicting the suppression of bleeding through emergency hemostasis are used to determine the appropriate hemostasis treatment. Therefore, processing accuracy is improved compared to cases that only consider the increase in bleeding volume from the onset of bleeding. In the example described above, the accuracy of processing that estimates the vessel type based on the sequence of images of the object being processed is further improved.

[0194] Furthermore, as an action of both the organism and the user, bleeding first occurs in the organism, and the user identifies this bleeding. The user's identification of bleeding can be performed through the aforementioned bleeding notification processing, or it can be done spontaneously by the user observing an image within the organism. Then, the user performs emergency hemostasis, resulting in the suppression of bleeding. As described above, the processing unit 120 determines, for example, that emergency hemostasis has been performed based on the degree of reduction in the red area. In this case, at the point when the processing unit 120 determines that emergency hemostasis has been performed, although it is unclear whether the emergency hemostasis is still in progress or has already been completed, at least the bleeding has decreased. Therefore, by using the point when the processing unit 120 determines that emergency hemostasis has been performed, or a point thereafter, as the endpoint of the processed image sequence, the hemostasis treatment can be determined considering the bleeding convergence. Additionally, when emergency hemostasis is determined to have been performed based on user input, for example, it is sufficient to prompt the user to input after the execution of the emergency hemostasis has begun. In this case, the hemostasis treatment can also be determined considering the bleeding convergence. In other words, by using the point at which the processing unit 120 determines that emergency hemostasis has been performed or a point thereafter as the endpoint of the processing target image sequence, it is possible to obtain a processing target image sequence that includes at least images taken during or after the user performs emergency hemostasis.

[0195] For example, as mentioned above, the starting point for processing the object image sequence can be... Figure 11 The time point corresponding to Ps shown can also be in Figure 9 The bleeding detection process shown indicates the point at which bleeding is determined. Additionally, when the processing unit 120 performs bleeding notification processing, the point at which this bleeding notification processing begins can also be used as the starting point. It should be noted that the entire image from the starting point to the ending point does not require the decision processing for hemostasis, and some parts can be omitted. For example, when the input image sequence of the CNN is of a fixed length, thinning processing to adjust the length of the image sequence can also be performed.

[0196] Furthermore, while this embodiment has been described in detail above, those skilled in the art will readily understand that various modifications can be made without substantially departing from the present embodiment and its effects. Therefore, all such modifications are included within the scope of this disclosure. For example, in the specification or drawings, a term described at least once with a broader or synonymous term can be replaced with that different term anywhere in the specification or drawings. Additionally, the scope of this disclosure also includes all combinations of this embodiment and its modifications. Moreover, the structure and operation of image processing systems, endoscope systems, etc., are not limited to those described in this embodiment, and various modifications can be implemented.

[0197] Label Explanation

[0198] 100 image processing system, 110 image acquisition section, 120 processing section, 121 bleeding detection section, 123 bleeding point detection section, 125 treatment decision section, 127 display processing section, 130 storage section, 200 endoscope system, 210 endoscope scope, 220 processor unit, 221 photographed data reception section, 222 processing section, 223 storage section, 230 display section, 240 operation section, 300 database server, 400 database generation device, 500 learning device, 600 image collection endoscope system

Claims

1. An image processing system, characterized by, The image processing system includes: an image acquisition unit that acquires, as a processing target image sequence, time-series images obtained by an endoscope imaging device that captures an inside of a living body; a processing unit that performs processing based on a database and the processing target image sequence, the database being generated based on a plurality of inside-of-living-body images captured at time points earlier than the processing target image sequence, wherein the database includes a plurality of data sets including a plurality of first data sets that correspond hemostatic treatment information to an inside-of-living-body image sequence of the inside-of-living-body images that is time-series, the hemostatic treatment information indicating a kind of the hemostatic treatment recommended for a bleeding captured in the inside-of-living-body image sequence, when a bleeding occurs in the inside of the living body, the processing unit determines a kind of ideal hemostatic treatment for a blood vessel in which the bleeding occurs based on a first learned model and the processing target image sequence, and presents the determined kind of the hemostatic treatment to a user, wherein the first learned model is a model that has learned a relationship between the inside-of-living-body image sequence and the hemostatic treatment information based on the plurality of first data sets.

2. The image processing system according to claim 1, wherein the processing unit determines a type of the blood vessel in which the bleeding occurs based on the processing target image sequence and the database, and determines the kind of the hemostatic treatment based on the determined type.

3. The image processing system according to claim 2, wherein the plurality of data sets further include a plurality of second data sets that correspond vessel type information to the inside-of-living-body image sequence of the inside-of-living-body images that is time-series, the vessel type information indicating in which type of the blood vessel the bleeding captured in the inside-of-living-body image sequence occurs, the processing unit determines the type of the blood vessel in which the bleeding occurs based on a second learned model and the processing target image sequence, the second learned model being a model that has learned a relationship between the inside-of-living-body image sequence and the vessel type information based on the plurality of second data sets, and determines the kind of the hemostatic treatment based on the determined type.

4. The image processing system according to claim 1, wherein the processing unit performs processing of adding another data set to the database, the another data set being a third data set that corresponds the processing target image sequence used in the determination of the kind of the hemostatic treatment to the kind of the hemostatic treatment for the bleeding detected in the processing target image sequence.

5. The image processing system according to claim 1, wherein the processing unit performs bleeding detection processing of detecting whether the bleeding occurs in the inside of the living body based on the processing target image sequence, the processing unit performs bleeding point detection processing of determining a bleeding point that is a position where the bleeding occurs when the bleeding is detected.

6. The image processing system according to claim 5, wherein the processing portion performs a bleeding notification process of notifying the user of information about at least one of occurrence of the bleeding detected by the bleeding detection process and the bleeding point determined by the bleeding point detection process.

7. The image processing system according to claim 1, wherein the processing portion, when it is determined that the user performed an emergency hemostasis treatment on the bleeding, performs a process of deciding a kind of treatment for hemostasis, which is higher in hemostasis ability than a kind of the emergency hemostasis treatment, based on the processing target image sequence and the database.

8. The image processing system according to claim 7, wherein the processing portion performs a process of deciding the kind of treatment for hemostasis based on the processing target image sequence including an image taken during or after the user performs the emergency hemostasis treatment.

9. The image processing system according to claim 1, wherein the processing target image sequence and the in-vivo image are images taken of a liver, the processing portion performs a process of deciding the kind of treatment for hemostasis for any blood vessel of a hepatic artery, a hepatic vein, and a portal vein based on the processing target image sequence and the database.

10. The image processing system according to claim 9, wherein the processing portion determines whether a Pringle maneuver of blocking blood flow of the hepatic artery and the portal vein is performed, and performs a process of deciding the kind of treatment for hemostasis based on a result of the determination.

11. The image processing system according to claim 9, wherein the processing portion decides a kind of first treatment for hemostasis in a case where the Pringle maneuver of blocking blood flow of the hepatic artery and the portal vein is performed and a kind of second treatment for hemostasis in a case where the Pringle maneuver is not performed based on the processing target image sequence and the database.

12. An endoscope system characterized by comprising: the endoscope system includes: a photographing portion that photographs an in-vivo; an image acquisition portion that acquires a time-series image photographed by the photographing portion as a processing target image sequence; and a processing portion that performs processing according to a database and the processing target image sequence, the database being generated from a plurality of in-vivo images taken at time points earlier than the processing target image sequence, the database including a plurality of data sets including a plurality of first data sets in which information on a treatment for hemostasis is associated with an in-vivo image sequence of the in-vivo images that is time-series, the information on the treatment for hemostasis indicating a kind of the treatment for hemostasis recommended for a bleeding photographed in the in-vivo image sequence, When bleeding has occurred in the living body, the processing unit performs processing of deciding, based on a first learned model and the processing target image sequence, a kind of ideal hemostatic treatment for a blood vessel in which the bleeding has occurred, and presenting the decided kind of hemostatic treatment to a user, wherein the first learned model is a model that has learned a relationship between the in-vivo image sequence and the hemostatic treatment information based on the plurality of first data sets.

13. An image processing method, characterized by, The image processing method performs processing of: acquiring, as a processing target image sequence, a time-series image obtained by imaging an in-vivo using an endoscope imaging device; when bleeding has occurred in the living body, deciding, from a first learned model and the processing target image sequence, a kind of ideal hemostatic treatment for a blood vessel in which the bleeding has occurred; and presenting the decided kind of hemostatic treatment to a user; wherein a database is generated from a plurality of in-vivo images imaged at a time point earlier than the processing target image sequence, the database includes a plurality of data sets including a plurality of first data sets in which hemostatic treatment information and an in-vivo image sequence of in-vivo images that are time-series are associated, the hemostatic treatment information indicating a kind of hemostatic treatment recommended for bleeding imaged in the in-vivo image sequence, and the first learned model is a model that has learned a relationship between the in-vivo image sequence and the hemostatic treatment information based on the plurality of first data sets.

Images