Computer programs, methods for generating learning models, image processing devices, and surgical assistance systems

Abstract

This enables easy monitoring of the surgical subject's condition. The computer performs the following processing: acquiring surgical field images obtained from endoscopic surgery, and learning a model that, given an input surgical field image, outputs information about the connective tissue between the preserved and resected organs. The acquired surgical field image is input, information about the connective tissue contained in the surgical field image is obtained, and based on the information obtained from the learning model, navigation information is output when handling the connective tissue between the preserved and resected organs.

Description

Technical Field

[0001] This invention relates to computer programs, methods for generating learning models, image processing apparatus, and surgical assistance systems. Background Technology

[0002] The following endoscopic surgical procedures are performed: multiple small incisions of approximately 3mm to 10mm are made in the body to reach the abdominal and thoracic cavities, and the endoscope and surgical instruments are inserted through these incisions without cutting the body. As a technique for performing endoscopic surgical procedures, a surgical system is proposed that allows for the manipulation of surgical instruments using a medical robotic arm to perform surgical procedures while simultaneously viewing images acquired by the endoscope on a monitor (see, for example, Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Publication No. 2013-531538 Summary of the Invention

[0006] The problem the invention aims to solve

[0007] In surgical procedures, it is important to lay the target organ in a position suitable for its placement. However, in endoscopic surgery, it is difficult to grasp the overall image of the organ, and the lack of tactile feedback makes it challenging to determine the direction and extent of traction to achieve the optimal placement.

[0008] This invention was made in view of this background, and its purpose is to provide a technique that enables easy monitoring of the condition of the surgical subject.

[0009] Solution for solving the problem

[0010] In the main invention of the present invention used to solve the above-mentioned problems, a computer performs the following process: acquiring a surgical field image obtained by photographing the surgical field of an endoscopic surgery, and learning a learning model that learns to output information about the connective tissue between the preserved and resected organs when given a surgical field image, inputting the acquired surgical field image, acquiring information about the connective tissue contained in the surgical field image, and outputting navigation information when processing the connective tissue between the preserved and resected organs based on the information acquired from the learning model.

[0011] Invention Effects

[0012] According to the present invention, the condition of the surgical subject can be easily grasped. Attached Figure Description

[0013] Figure 1This is a diagram illustrating a surgical procedure using the surgical assistance system of this embodiment.

[0014] Figure 2 This is a conceptual diagram illustrating a general outline of the surgical assistance system of this embodiment.

[0015] Figure 3 This is a diagram illustrating an example of the hardware structure of an image processing device.

[0016] Figure 4 This is a diagram illustrating an example of the software structure of an image processing device.

[0017] Figure 5 This diagram illustrates the learning process in the surgical assistance system of this embodiment.

[0018] Figure 6 This diagram illustrates the output processing flow of the score in the surgical assistance system of this embodiment.

[0019] Figure 7 This is a diagram showing an example of the fraction of loose connective tissue expansion treatment.

[0020] Figure 8 This is a diagram showing an example of the fraction of loose connective tissue expansion treatment.

[0021] Figure 9 This is a diagram showing an example of the fraction of loose connective tissue expansion treatment.

[0022] Figure 10 This is a diagram showing an example of the fraction of traction on a blood vessel.

[0023] Figure 11 This is a diagram showing an example of the fraction of traction on a blood vessel.

[0024] Figure 12 This is a diagram showing an example of the fraction of traction on a blood vessel.

[0025] Figure 13 This is a schematic diagram illustrating an example of a surgical field image in Embodiment 2.

[0026] Figure 14 This is a schematic diagram illustrating a structural example of a learning model.

[0027] Figure 15 This is a schematic diagram showing the recognition results from the learning model.

[0028] Figure 16 This is a flowchart illustrating the process of generating the learning model.

[0029] Figure 17This is a flowchart illustrating the steps of processing performed by the image processing apparatus in embodiment 2 during the application phase.

[0030] Figure 18 This is a schematic diagram showing an example of implementation 2.

[0031] Figure 19 This is a schematic diagram illustrating an example of a surgical field image that needs to be expanded.

[0032] Figure 20 This is a diagram showing an example of the output of navigation information related to the unfolding operation.

[0033] Figure 21 This is a diagram showing an example of the output of navigation information related to the cut-off operation.

[0034] Figure 22 This is a flowchart illustrating the steps of processing performed by the image processing apparatus in embodiment 3 during the application phase.

[0035] Figure 23 This is a schematic diagram illustrating the structure of the learning model in Implementation Method 4.

[0036] Figure 24 This is a flowchart illustrating the steps of processing performed by the image processing apparatus in embodiment 4 during the application phase.

[0037] Figure 25 This is a schematic diagram illustrating the structure of the learning model in Implementation Method 5.

[0038] Figure 26 This is an explanatory diagram illustrating the steps involved in creating correct answer data.

[0039] Figure 27 This is a flowchart illustrating the steps of processing performed by the image processing apparatus in embodiment 5 during the application phase.

[0040] Figure 28 This is a schematic diagram showing an example of a cut-off section.

[0041] Figure 29 This is a schematic diagram illustrating the structure of the learning model in Implementation Method 6.

[0042] Figure 30 This is a flowchart illustrating the steps of processing performed by the image processing apparatus in Embodiment 6.

[0043] Figure 31 This is an explanatory diagram illustrating the state of the area being processed.

[0044] Figure 32This is an explanatory diagram illustrating the state of the area being processed.

[0045] Figure 33 This is an explanatory diagram illustrating the state of the area being processed.

[0046] Figure 34 This is an explanatory diagram illustrating the learning model in Implementation Method 8.

[0047] Figure 35 This is a flowchart illustrating the steps of processing performed by the image processing apparatus in Embodiment 8. Detailed Implementation

[0048] <Summary of the Invention>

[0049] The embodiments of the present invention will be described in detail. For example, the present invention has the following structure.

[0050] [Project 1]

[0051] A surgical assistance system, which is a system for assisting surgery, has the following features:

[0052] Anatomical status acquisition unit, which acquires information showing the anatomical status during the surgery;

[0053] A fraction calculation unit, which determines the fraction of the anatomical state based on the information; and

[0054] The fraction output unit outputs information based on the fraction.

[0055] [Project 2]

[0056] In the surgical assistance system described in Project 1

[0057] The anatomical state includes the correlation of multiple organs.

[0058] [Project 3]

[0059] In the surgical assistance system described in Project 1 or 2

[0060] The anatomical state indicates the appropriate state, or degree, for the treatment of the surgical subject.

[0061] [Project 4]

[0062] In any of the surgical assistance systems described in Projects 1 to 3

[0063] The information includes at least images of the surgical field.

[0064] [Project 5]

[0065] In the surgical assistance system described in Project 4

[0066] The image includes at least one of a visible image, an infrared image, and a depth map.

[0067] [Project 6]

[0068] In any of the surgical assistance systems described in items 1 to 5,

[0069] The fraction calculation unit determines the fraction based on the size of the area of ​​the surgical object.

[0070] [Project 7]

[0071] In any of the surgical assistance systems described in items 1 to 5,

[0072] The score calculation unit determines the score based on the degree of deformation of the surgical object.

[0073] [Project 8]

[0074] In any of the surgical assistance systems described in items 1 to 5,

[0075] The score calculation unit determines the score based on information including the color of the surgical object.

[0076] [Project 9]

[0077] In any of the surgical assistance systems described in items 1 to 8

[0078] The score output unit displays the score on the surgical monitor.

[0079] [Project 10]

[0080] In the surgical assistance system described in Project 3

[0081] The score output unit displays whether the action is appropriate based on the score.

[0082] [Project 11]

[0083] In any of the surgical assistance systems described in items 1 to 10,

[0084] The fraction output unit outputs the fraction when the change in the fraction becomes above a predetermined value.

[0085] [Project 12]

[0086] Surgical assistance system as described in any one of items 1 to 11,

[0087] It also features a treatable area display unit, which overlays and displays markings indicating treatable areas of the surgical subject on the surgical monitor.

[0088] [Project 13]

[0089] In the surgical assistance system described in Project 12

[0090] The disposal area display unit uses the identifier to display the score.

[0091] [Project 14]

[0092] In the surgical assistance system described in item 12 or 13

[0093] The identifier indicates the approximate outer edge of the treatable area.

[0094] [Project 15]

[0095] In any of the surgical assistance systems described in items 1 to 14,

[0096] The treatment of the surgical subject includes at least one of the following: excision, dissection, suturing, ligation, or anastomosis.

[0097] [Project 16]

[0098] In the surgical assistance systems described in items 3, 6, 7, 8, 12, 13, 14, or 15,

[0099] The surgical target is loose connective tissue.

[0100] (Implementation Method 1)

[0101] <Outline>

[0102] Hereinafter, a surgical assistance system 1 according to one embodiment of the present invention will be described. The surgical assistance system 1 of this embodiment assists, for example, surgeries performed remotely, such as endoscopic surgery or robotic surgery.

[0103] Figure 1 This diagram illustrates the surgical assistance system 1 of this embodiment. (See diagram for reference.) Figure 1 As shown, in this embodiment, it is envisioned that the organ 2 is preserved and the organ 3 is removed.

[0104] like Figure 1As shown, since the proper blood vessel 6 passes through the resected organ 3 and exists within the connective tissue 4, it is necessary to cut the connective tissue 4 to treat the proper blood vessel 6. However, in order to avoid damage to the preserved organ 2, the resected organ 3, the proper blood vessel 6, and other living tissues, it is best to cut the connective tissue 4 while it is stretched. However, if it is overstretched, it may also cause laceration to the other living tissues of the preserved organ 2 connected to the connective tissue 4. Therefore, in the surgical assistance system 1 of this embodiment, the system learns in advance by receiving input from the judgment of a doctor or the like, which is a judgment about the suitability of the stretched state of the connective tissue 4 for performing incisions or other treatments. This allows the system to calculate a score based on the image showing the degree to which the state of the connective tissue 4 is suitable for treatment (the degree to which the connective tissue 4 can be safely resected or treated). Furthermore, during the operation, the score is calculated based on the image of the surgical field and displayed on a monitor to assist the doctor 110 (hereinafter also referred to as operator 110) in treating the connective tissue 4. In addition, in this embodiment, the main consideration for treatment is excision, but in addition to excision, treatment may also include dissection, suturing, clamping or other ligation, or anastomosis.

[0105] As the organ to be removed (3), it may include, for example, the stomach, large intestine, esophagus, pancreas, lung, prostate, ovary, etc. If the stomach is the organ to be removed (3), the organ to be retained (2) may include, for example, the pancreas, transverse mesocolon, etc. If the large intestine is the organ to be removed (3), the organ to be retained (2) may include, for example, the ureter, arterioles and veins, seminal vesicle, pelvic nerve plexus, etc. If the esophagus is the organ to be removed (3), the organ to be retained (2) may include, for example, the trachea, recurrent laryngeal nerve, etc. If the pancreas is the organ to be removed (3), the organ to be retained (2) may include, for example, the left renal vein, etc. If the lung is the organ to be removed (3), the organ to be retained (2) may include, for example, the recurrent laryngeal nerve, aorta, esophagus, etc. If the prostate is the organ to be removed (3), the organ to be retained (2) may include, for example, the rectum.

[0106] There is connective tissue 4 (loose connective tissue) between the preserved organ 2 and the resected organ 3. Blood flows from the main blood vessels 5 through the proper blood vessels 6 to the resected organ 3. When resecting the resected organ 3, the proper blood vessels 6 need to be managed. First, the connective tissue 4 is removed to expose the proper blood vessels 6, making management of the proper blood vessels 6 easier. Here, if the preserved organ 2 is the stomach, the main blood vessels 5 contain, for example, the hepatic artery and celiac artery, and the proper blood vessels 6 contain, for example, the left gastric artery. If the preserved organ 2 is the large intestine, the main blood vessels 5 contain, for example, the portal vein and aorta, and the proper blood vessels 6 contain, for example, the ileocolic artery and vein, and the inferior mesenteric artery. If the preserved organ 2 is the esophagus, the main blood vessels 5 contain, for example, the aorta. If the preserved organ 2 is the pancreas, the main blood vessels 5 contain, for example, the portal vein and superior mesenteric artery, and the proper blood vessels 6 contain, for example, the IPD. If organ 2 is the lung, the main blood vessels 5 include, for example, the trachea, pulmonary artery, and pulmonary vein, and the proper blood vessels 6 are, for example, proper branches of the main blood vessels 5.

[0107] In the surgical assistance system 1 of this embodiment, information showing the anatomical state is acquired, the degree to which the anatomical state is suitable for treatment (score) is determined, and the score is output. The anatomical state includes, for example, the relationships between multiple organs (e.g., the distance between the preserved organ 2 and the resected organ 3), the state of the organs (e.g., the state of the preserved organ 2 and / or the resected organ 3), and the state of the loose connective tissue 4 connecting multiple tissues (stretching state, extended state, etc.). In this embodiment, as an example, when resecting the resected organ 3, the resected organ 3 is pulled to separate from the preserved organ 2, stretched to a degree that does not rupture, thereby determining whether it is suitable for resection. The score of the anatomical state can be pre-determined by an experienced physician based on surgical video images, and the result determined by the physician is learned using machine learning methods.

[0108] <System Architecture>

[0109] Figure 2 This is a conceptual diagram illustrating a general outline of the surgical assistance system 1 according to this embodiment. Furthermore, the structure of the diagram is an example and may include elements beyond these.

[0110] In the surgical assistance system 1, the surgical subject (patient 100) is placed in the surgical unit 10, and the surgical field of the patient 100 is captured by a camera 21 (e.g., inserted into the abdomen). The image captured by the camera 21 is transmitted to the control unit 22, and the image output from the control unit 22 is displayed on the main monitor 23. In the surgical assistance system 1 of this embodiment, the image processing device 30 acquires the image of the surgical field from the control unit 22, analyzes the image, and outputs a fraction of the anatomical state to the secondary monitor 31.

[0111] <Hardware>

[0112] The image processing device 30 can be a general-purpose computer such as a workstation or personal computer, or it can be logically implemented through cloud computing. Alternatively, the image processing device 30 can also be a logic circuit such as an FPGA (Field Programmable Gate Array). Figure 3 This diagram illustrates an example of the hardware structure of an image processing apparatus 30. The image processing apparatus 30 includes a CPU 301, a memory 302, a storage device 303, a communication interface 304, an input device 305, and an output device 306. The storage device 303 stores various data and programs, such as a hard disk drive, a solid-state drive, or a flash memory. The communication interface 304 is an interface for connecting to a communication network, such as an adapter for connecting to an Ethernet network, a modem for connecting to a public telephone network, a radio communication device for radio communication, a USB (Universal Serial Bus) connector for serial communication, or an RS232C connector. The input device 305 is for inputting data, such as a keyboard, mouse, touch panel, contactless panel, buttons, or microphone. The output device 306 is for outputting data, such as a monitor, printer, or speaker.

[0113] <Software>

[0114] Figure 4 This is a diagram showing an example of the software structure of the image processing apparatus 30. The image processing apparatus 30 includes an image acquisition unit 311, an anatomical state acquisition unit 312, a handleable area display unit 313, a learning processing unit 314, a score calculation unit 315, a score output unit 316, and a model storage unit 331.

[0115] Furthermore, in the image processing device 30, the image acquisition unit 311, the anatomical state acquisition unit 312, the treatable area display unit 313, the learning processing unit 314, the score calculation unit 315, and the score output unit 316 can be implemented by reading and executing the program stored in the storage device 303 by the CPU 301 of the image processing device 30, and the model storage unit 331 can be implemented as part of the storage area of ​​the storage device 303 and the storage area of ​​the storage device 302 of the image processing device 30.

[0116] The image acquisition unit 311 acquires images of the surgical field captured by the camera 21. In this embodiment, the image acquisition unit 311 acquires video images captured by the camera 21 and output from the control unit 22.

[0117] The anatomical state acquisition unit 312 acquires the anatomical state of the surgical field. In this embodiment, the anatomical state acquisition unit 312 can analyze the image acquired by the image acquisition unit 311 and identify living tissues such as organs, connective tissues, and blood vessels, and can acquire, for example, the characteristic quantities of each living tissue.

[0118] The treatable area display unit 313 outputs a graphic showing a pre-set target tissue for treatment (conceived as connective tissue in this embodiment). The treatable area display unit 313 may, for example, change the color of the pre-set target tissue (e.g., connective tissue) in the living tissue identified by the anatomical state acquisition unit 312, or extract the outline and overlay it onto the image. Furthermore, the treatable area display unit 313 may also output the target tissue only if the score calculated by the score calculation unit 315 described below is above a threshold.

[0119] The learning processing unit 314 learns a score corresponding to the anatomical state. In the learning processing unit 314, for example, it receives an input from a doctor viewing an image acquired by the image acquisition unit 311, indicating the degree to which the state of the tissue to be treated is suitable for treatment (e.g., any range such as a maximum score of 5, 10, or 100). The input score is used as a teacher signal, and the anatomical state (features of the living tissue) analyzed from the image is used as an input signal. For example, learning can be performed using machine learning methods such as neural networks. Furthermore, the features provided as input signals are not limited to the tissue to be treated; for example, features such as preserved organs 2, resected organs 3, major blood vessels 5, and proper blood vessels 6 can also be used. That is, for example, the score can include the positional relationship between preserved organs 2 and resected organs 3. The learning processing unit 314 registers the learned model in the model storage unit 331. The model storage unit 331 stores the model in a known format, and detailed descriptions are omitted here.

[0120] The score calculation unit 315 calculates a score indicating the degree to which the treatment target area is suitable for treatment based on the image acquired by the image acquisition unit 311. The score calculation unit 315 is able to calculate the score by providing information (features) of each living tissue acquired by the anatomical state acquisition unit 312 to the model stored in the model storage unit 331.

[0121] The score output unit 316 outputs the score calculated by the score calculation unit 315. In this embodiment, the score output unit 316 displays the score on the sub-monitor 31 by overlaying it with the image of the surgical field captured by the camera 21. The score output unit 316 can output the score as text information such as numbers on the sub-monitor 31, or it can output a meter representing the score near the treatment object on the sub-monitor 31, or it can change the color of the graphic displayed by the treatment area display unit 313 according to the score. In addition, the score output unit 316 can also output the sound of the score from a speaker (for example, it can be the voice reading the score through speech synthesis, or it can be the sound corresponding to the score).

[0122] <Learning Processing>

[0123] Figure 5 This is a diagram illustrating the learning processing steps of the surgical assistance system 1 of this embodiment. Figure 5 The learning process shown can be performed, for example, using either pre-captured images of the surgical field or images being captured.

[0124] The image acquisition unit 3119 acquires images of the surgical field captured by the camera 21 (either real-time images or previously recorded images) (S401). The anatomical state acquisition unit 312 identifies living tissues such as organs 2 to be preserved, organs 3 to be removed, and connective tissue 4 from the images (S402). The learning processing unit 314 receives input from doctors or others showing the suitability of the treatment of the tissue to be treated (connective tissue 4 in this embodiment) (S403). It uses information (features, etc.) showing the anatomical state acquired by the anatomical state acquisition unit 312 as input signals, uses the received scores as teacher signals for learning (S404), and registers the learned model in the model storage unit 331 (S405).

[0125] As described above, based on images of the surgical field, it is possible to learn the suitability (score) of the treatment of connective tissue 4 from the state of each living tissue, such as the preserved organ 2, the resected organ 3, and the connective tissue 4, as displayed in the images.

[0126] <Learning Processing>

[0127] Figure 6 This diagram illustrates the steps of the fraction output processing in the surgical assistance system 1 of this embodiment. Figure 6 The output processing of the fractions shown is assumed to be based on images taken during surgery.

[0128] The image acquisition unit 311 acquires images of the surgical field captured by the camera 21 from the control unit 22 (S421). The anatomical state acquisition unit 312 identifies living tissues such as preserved organs 2, resected organs 3, and connective tissue 4 from the images (S422). The score calculation unit 315 provides the characteristic quantities of the identified tissues to the model stored in the model storage unit 331 and calculates the score (S423). The score output unit 316 overlays the calculated scores onto the image of the surgical field on the sub-monitor 31 (S424).

[0129] As described above, the suitability of the treatment object (connective tissue 4) can be output to the sub-monitor 31 based on the images of the surgical field taken during surgery.

[0130] Figures 7 to 9 This is a diagram showing an example of the fractionation of loose connective tissue expansion treatment. Figures 7 to 9 The example shown illustrates a case where loose connective tissue is used as connective tissue 4. The sub-monitor 31 displays the preserved organ 2, the removed organ 3, and the connective tissue 4 between the preserved organ 2 and the removed organ 3 (not hidden within the preserved organ 2 and the removed organ 3). Furthermore, the sub-monitor 31 displays a score indicating the degree to which the connective tissue 4 is suitable for treatment, shown in the vicinity of the connective tissue 4 as the object of treatment, in the form of a meter 51 and a text display 52.

[0131] exist Figures 7 to 9 In the example, it is shown that the operator 110 remotely operates the surgical instrument 50 and holds the organ to be removed 3 in a posture that pulls it up towards the top of the screen. The organ to be removed is... Figure 7 , Figure 8 , Figure 9 The order is pulled up, so it can be seen that among these, Figure 9 The current state is the most suitable for handling. For example... Figure 7 As shown, when organ 3 is separated from preserved organ 2 not far away, the score (treatment suitability) is 10%. Figure 8 In the middle, if organ 3 is pulled up, it indicates a score of 55%. Figure 9 In this state, if the resected organ 3 is further pulled up, the suitability score for the procedure is displayed as 98%. Referring to this score, the surgical operator 110 (doctor) is able to perform procedures such as pulling up the resected organ 3 and removing the connective tissue 4. If the resected organ 3 is separated from the preserved organ 2, and the loose connective tissue is exposed and fully stretched, it indicates that the loose connective tissue can be safely removed. Since the safety of this removal can be visually identified using the measuring instrument 51 and the text display 52, the connective tissue 4 can be safely handled.

[0132] In addition, such as Figure 8 as well as Figure 9As shown, the treatable area display unit 313 can overlay a graphic representing the connective tissue 4 onto the image displayed on the sub-monitor 31. This allows the surgical operator 110 to immediately and easily identify the object to be treated.

[0133] Furthermore, the score can be displayed either by the meter 51 or the text display 52, or it can be represented by color using a graphic of the connective tissue 4 displayed by the handleable area display unit 313. For example, it can be displayed in blue or green when the score is above a first threshold, in red when the score is below a second threshold (less than the first threshold), and in yellow when the score is below the first threshold but above the second threshold. Alternatively, a color gradient corresponding to the score can also be formed.

[0134] exist Figures 7 to 9 The example shown is loose connective tissue as connective tissue 4, but connective tissue 4 is not limited to loose connective tissue; it can be any tissue that connects two organs. For example, dense connective tissue is also connective tissue 4, and blood vessels can also be considered as connective tissue 4. Furthermore, while the above embodiment envisions the removal of connective tissue 4, the treatment of connective tissue 4 is not limited to removal. For example, for blood vessels 4, clamping, removal, suturing, and other treatments are possible.

[0135] Figures 10 to 12 This is a diagram showing an example of a fractional display of vascular traction. The sub-monitor 31 displays the preserved organ 2, the resected organ 3, and the blood vessel 4 that runs between the preserved organ 2 and the resected organ 3. The fractional degree to which the blood vessel 4 is suitable for treatment is displayed near the blood vessel (or preserved organ 2 or resected organ 3) in the form of a meter 51 and a text display 52. ​​Additionally, in... Figures 10 to 12 In this example, surgical instrument 50 is used to stretch connective tissue 4 (blood vessels) instead of removing organ 3. It can be seen that blood vessel 4... Figure 10 , Figure 11 , Figure 12 The order is stretched. It is possible to grasp the... Figure 12 The state is the most suitable for treatment. In this way, even blood vessel 4 can be safely treated by showing its suitability for treatment.

[0136] As described above, the surgical assistance system 1 according to this embodiment can display, based on the captured images of the surgical field, whether the organs or other parts of the patient are in a suitable state for treatment, i.e., a state in which safe treatment can be performed. Therefore, surgery on the patient can be performed safely.

[0137] The following describes a structure that uses a learning model to acquire information about the connective tissue 4 between the preserved organ 2 and the resected organ 3, and outputs navigation information when processing the connective tissue 4 between the preserved organ 2 and the resected organ 3 based on the acquired information. In the following embodiments, loose connective tissue is mainly used as an example of connective tissue 4, but connective tissue 4 may also include membranes, layers, adipose tissue, etc., that exist between the preserved organ 2 and the resected organ 3.

[0138] (Implementation Method 2)

[0139] In Implementation 2, a learning model is used to identify the loose connective tissue between the preserved organ 2 and the resected organ 3. Based on the identification results, the following structure is described: the cutting time point of the loose connective tissue is output as navigation information.

[0140] Figure 13 This is a schematic diagram illustrating an example of a surgical field image in Embodiment 2. Figure 13 The surgical field shown includes the ORG tissue constituting the preserved organ 2, the NG tissue constituting the resected organ 3, and the LCT loose connective tissue connecting these tissues. The LCT loose connective tissue is an example of connective tissue 4. Figure 13 In the example, loose connective tissue (LCT) is shown with a dashed line.

[0141] In laparoscopic surgery, such as for removing lesions like malignant tumors formed in a patient's body, the surgeon uses forceps 50A to grasp the tissue NG containing the lesion and unfolds it in the appropriate direction, thereby exposing the loose connective tissue LCT between the tissue NG containing the lesion and the tissue ORG to be retained. The surgeon then uses an energy treatment tool 50B to remove the exposed loose connective tissue LCT, thereby separating the tissue NG containing the lesion from the tissue ORG to be retained. Forceps 50A and energy treatment tool 50B are examples of surgical instruments 50.

[0142] Furthermore, from the viewpoint of ease of resection of loose connective tissue LCT, the loose connective tissue LCT to be resected preferably has elasticity. Additionally, it is preferable that there is space inside the loose connective tissue LCT to allow movement of the forceps 50A and the energy treatment tool 50B. Moreover, it is preferable that the loose connective tissue LCT to be resected is kept in a stretched state. Figure 13 An example shows the following posture: there is a space SP inside the loose connective tissue LCT, and at least a portion of it remains in a stretched state.

[0143] Because a fluid matrix and various cells surround the fibers that constitute loose connective tissue (LCT), it is not easy for surgeons to detect LCT from surgical field images. Therefore, the image processing apparatus 30 of this embodiment utilizes a learning model 410 (see reference 410). Figure 14 The system identifies loose connective tissue portions from the surgical field image and outputs auxiliary information about laparoscopic surgery based on the identification results.

[0144] The following describes a structural example of the learning model 410 used in the image processing apparatus 30.

[0145] Figure 14 This is a schematic diagram illustrating a structural example of the learning model 410. The learning model 410 is a learning model for image segmentation, constructed, for example, using a neural network with convolutional layers such as SegNet. Figure 14 The diagram shows an example of the SegNet structure, but it is not limited to SegNet. Any neural network capable of image segmentation, such as FCN (Fully Convolutional Network), U-Net (U-Shaped Network), or PSPNet (Pyramid Scene Parsing Network), can be used to construct the learning model 410. Alternatively, object detection neural networks such as YOLO (You Only Look Once) or SSD (SingleShot Multi-Box Detector) can be used to replace the neural network used for image segmentation when constructing the learning model 410.

[0146] In this embodiment, the input image to the learning model 410 is a surgical field image obtained from the camera 21. The surgical field image is not limited to video; it can also be a still image. Furthermore, the surgical field image input to the learning model 410 does not need to be the original image obtained from the camera 21; it can be an image with appropriate image processing, data showing the frequency components of the image, etc. The learning model 410 learns in such a way that, for the input surgical field image, it outputs an image showing the recognition results of the loose connective tissue portions contained in the surgical field image.

[0147] The learning model 410 in this embodiment includes, for example, an encoder 411, a decoder 421, and a softmax layer. The encoder 411 is configured with alternating convolutional layers and pooling layers. The convolutional layers are divided into multiple layers of 2 to 3 layers. Figure 14 In the example, the convolutional layer is shown without profile lines, while the pooling layer is shown with profile lines.

[0148] In the convolutional layer, convolution operations are performed by applying a convolution operation between the input data and each filter of a defined size (e.g., 3×3, 5×5, etc.). Specifically, for each feature, the input value at the corresponding position in each filter is multiplied by a preset weight coefficient on the filter, and the linear sum of the products of these features is calculated. The output of the convolutional layer is obtained by adding the set bias to the calculated linear sum. Furthermore, the result of the convolution operation can be transformed using an activation function. For example, ReLU (Rectified LinearUnit) can be used as the activation function. The output of the convolutional layer represents a feature map that extracts features from the input data.

[0149] In the pooling layer, local statistics of the feature map output from the convolutional layer, which is an upper layer connected to the input, are calculated. Specifically, a window of a predetermined size (e.g., 2×2, 3×3) corresponding to the position of the upper layer is set, and local statistics are calculated based on the input values ​​within the window. For example, the maximum value can be used as the statistic. The size of the feature map output from the pooling layer is reduced (downsampled) according to the size of the window. Figure 14 The example shows that by repeatedly performing convolutional and pooling operations in encoder 411, the 224-pixel × 224-pixel input image is successively downsampled into feature maps of 112×112, 56×56, 28×28, ..., 1×1.

[0150] The output of encoder 411 (in) Figure 14 In the example, a 1×1 feature map is input to decoder 412. Decoder 412 consists of alternating deconvolutional layers and inverse pooling layers. The deconvolutional layers are divided into multiple layers of 2 to 3 layers. Figure 14 In the example, the deconvolution layer is shown without profile lines, while the inverse pooling layer is shown with profile lines.

[0151] In the deconvolutional layer, a deconvolution operation is performed on the input feature map. The deconvolution operation is defined as follows: it reconstructs the feature map before the convolution operation by applying specific filters to the input feature map. In this operation, when the specific filters are represented as matrices, the product of the transpose of that matrix and the input feature map is calculated, thereby generating the output feature map. Furthermore, the result of the deconvolutional layer can also be transformed using activation functions such as ReLU.

[0152] The inverse pooling layer of the decoder 412 corresponds one-to-one with the pooling layer of the encoder 411, and the corresponding pair has essentially the same size. The inverse pooling layer further amplifies the size of the downsampled feature map (upsampling) in the pooling layer of the encoder 411. Figure 14 The example shows that by repeatedly performing convolutional and pooling operations in the decoder 412, feature maps are upsampled sequentially to 1×1, 7×7, 14×14, ..., 224×224.

[0153] The output of decoder 412 (in) Figure 14 In the example, a 224×224 feature map is input to the softmax layer 413. The softmax layer 413 outputs the probability of identifying a label for each location (pixel) by applying a softmax function to the input values ​​from the deconvolution layer connected to the input side. In this embodiment, a label for identifying loose connective tissue is set, and identification of whether it belongs to loose connective tissue is performed on a pixel-by-pixel basis. By extracting pixels whose label probability output from the softmax layer 413 is above a threshold (e.g., above 50%), an image showing the identification result of the loose connective tissue portion (hereinafter referred to as the identification image) is obtained.

[0154] In addition, Figure 14 In the example, a 224-pixel × 224-pixel image is used as the input image to the learning model 410. However, the size of the input image is not limited to the above, but can be appropriately set according to the processing capability of the image processing device 30, the size of the surgical field image obtained from the camera 21, etc. Furthermore, the input image to the learning model 410 does not need to be the entire surgical field image obtained from the camera 21; it can also be a partial image generated by cropping out the region of interest from the surgical field image. Since the region of interest containing the object to be treated is mostly located near the center of the surgical field image, for example, a rectangular partial image can be cropped from the center of the surgical field image at approximately half the original size. By reducing the size of the image input to the learning model 410, processing speed and recognition accuracy can be improved.

[0155] Figure 15 This is a schematic diagram showing the recognition results from the learning model 410. Figure 15In the example, the loose connective tissue portion identified by the learning model 410 is shown with a thick solid line, while other organs and tissues are shown with dashed lines as references. The CPU 301 of the image processing apparatus 30 generates an identification image of the loose connective tissue in order to display the identified loose connective tissue portion in a discriminative manner. The identification image is an image of the same size as the surgical field image, and it is an image in which specific colors are assigned to the pixels identified as loose connective tissue. The colors assigned to the pixels of loose connective tissue are preferably colors that do not exist inside the human body, so as to distinguish them from organs, blood vessels, etc. Colors that do not exist inside the human body are, for example, cool colors (blue family) such as blue or light blue. Furthermore, information indicating transmittance is added to each pixel constituting the identification image; an opaque value is set in the pixels identified as loose connective tissue, and a transmittance value is set in the pixels other than those identified as loose connective tissue. By overlaying and displaying the identification image generated in this way on the surgical field image, the loose connective tissue portion can be displayed as a structure with a specific color on the surgical field image.

[0156] The image processing device 30 generates a learning model 410, for example, during a learning phase before the operation begins. As a preparation phase for generating the learning model 410, in this embodiment, annotation is performed by manually segmenting the loose connective tissue portion from the surgical field image obtained from the camera 21. Furthermore, the annotation can be performed using the surgical field image stored in the storage device 303, etc.

[0157] During annotation, the operator (such as a doctor or expert) displays the surgical field image sequentially on the secondary monitor 31, simultaneously identifying loose connective tissue that is easily resectable between the preserved and resected organs. Specifically, the tissue containing the lesion is unfolded to identify the exposed loose connective tissue. The exposed loose connective tissue is elastic and preferably kept in a stretched state. Furthermore, there is space inside the loose connective tissue, preferably space for the surgical instrument 50 to move. When the operator identifies loose connective tissue that is easily resectable, they annotate the portion corresponding to the loose connective tissue in the surgical field image using a mouse, stylus, or similar input device 305, selecting it in pixels. Alternatively, a suitable movement pattern for the loose connective tissue can be selected, and the amount of data can be increased through perspective transformation, reflection, and other processing. Furthermore, if learning progresses, the amount of data can be increased using recognition results from the learning model 410.

[0158] Training data is prepared by annotating multiple surgical field images, consisting of surgical field images and correct data showing the loose connective tissue portion of the surgical field images. The training data is stored in a storage device (e.g., storage device 303 of the image processing device 30).

[0159] Figure 16 This is a flowchart illustrating the generation process of the learning model 410. The CPU 301 of the image processing device 30 reads a pre-prepared learning program from the storage device 303 and executes the following steps to generate the learning model 410. Furthermore, in the stage before learning begins, initial values ​​are assigned to the definition information describing the learning model 410.

[0160] CPU 301 accesses storage device 303 and selects a set of training data for learning (step S201). CPU 301 inputs the surgical field image into learning model 410, and the selected training data includes the surgical field image (step S202), and performs operations from learning model 410 (step S203). That is, CPU 301 performs the following operations: generating feature maps from the input surgical field image, and sequentially downsampling the generated feature maps by encoder 411; sequentially upsampling the feature maps input from encoder 411 by decoder 412; and recognizing each pixel of the feature map finally obtained from decoder 412 by softmax layer 413.

[0161] CPU 301 obtains the computation results from the learning model 410 and evaluates the obtained computation results (step S204). For example, CPU 301 evaluates the computation results by calculating the similarity between the image data of the loose connective tissue portion obtained as the computation result and the correct data contained in the training data. The similarity is calculated using, for example, the Jaccard coefficient. The Jaccard coefficient is given by A∩B / A∪B×100 (%) when the loose connective tissue portion extracted by the learning model 410 is set as A and the loose connective tissue portion contained in the correct data is set as B. Instead of the Jaccard coefficient, the Dice coefficient, the Simpson coefficient, or other existing methods can be used to calculate the similarity.

[0162] Based on the evaluation of the calculation results, CPU301 determines whether the learning has ended (step S205). If the similarity obtained is above a preset threshold, CPU301 can determine that the learning has ended.

[0163] If the learning process is determined to be incomplete (S205: No), the CPU 301 updates the weight coefficients and biases of each layer of the learning model 410 sequentially from the output side to the input side according to the backpropagation method (step S206). After updating the weight coefficients and biases of each layer, the CPU 301 returns to step S201 and executes the processing from step S201 to step S205.

[0164] If the learning is determined to be complete in step S205 (S205: Yes), since the learning model 410 has been obtained, the CPU 301 stores the obtained learning model 410 in the model storage unit 331 and ends the processing of this flowchart.

[0165] In this embodiment, the learning model 410 is generated by the image processing apparatus 30, but the learning model 410 can also be generated by an external computer such as a server device. The image processing apparatus 30 acquires the learning model 410 generated by the external computer using communication or other methods, and stores the acquired learning model 410 in the model storage unit 331.

[0166] In the application phase after obtaining the learned model 410, the image processing device 30 performs the following processing. Figure 17 This is a flowchart illustrating the processing steps performed by the image processing apparatus 30 in embodiment 2 during the application phase. The CPU 301 of the image processing apparatus 30 performs the following steps by reading and executing a pre-prepared recognition processing program from the storage device 303. If laparoscopic surgery begins, an image of the surgical field is captured and output from the camera 21 to the control unit 22 at any time. The CPU 301 of the image processing apparatus 30 acquires the surgical field image output from the control unit 22 via the image acquisition unit 311 (step S221). Each time the CPU 301 acquires a surgical field image, it performs the following processing.

[0167] CPU 301 inputs the acquired surgical field image into learning model 410 and performs calculations using learning model 410 (step S222) to identify the loose connective tissue portion contained in the surgical field image (step S223). Specifically, CPU 301 performs the following calculations: generating a feature map from the input surgical field image; sequentially performing downsampling operations on the generated feature map using encoder 411; sequentially performing upsampling operations on the feature map input from encoder 411 using decoder 412; and performing softmax layer 413 operations to identify each pixel in the final feature map obtained from decoder 412. Furthermore, CPU 301 identifies pixels whose label output from softmax layer 413 has a probability of above a threshold (e.g., above 50%) as loose connective tissue portions.

[0168] Next, based on the recognition results of the learning model 410, the CPU 301 calculates the exposed area of ​​the loose connective tissue portion (step S224). The CPU 301 calculates the exposed area using the number of pixels identified as loose connective tissue, or the ratio of pixels identified as loose connective tissue in the total pixels of the surgical field image. In addition, to standardize the size (area) of each pixel, the CPU 301 can, for example, derive distance information up to the focal point by referring to the focal length of the camera 21, and correct the exposed area based on the derived distance information.

[0169] Next, based on the calculated exposed area, CPU301 determines whether it is a cutting time point for loose connective tissue (step S225). Specifically, CPU301 determines the relationship between the calculated exposed area and a preset threshold. If the exposed area is greater than the threshold, it is determined to be a cutting time point. If it is determined not to be a cutting time point (S225: No), CPU301 ends the processing of this flowchart.

[0170] If the time point is determined to be a cutting point (S225: Yes), the CPU 301 notifies that it is a cutting point for loose connective tissue (step S226). For example, the CPU 301 can overlay and display text information indicating the cutting point for loose connective tissue on the surgical field image. Alternatively, instead of displaying text information, an icon or marker indicating the cutting point for loose connective tissue can be overlaid and displayed on the surgical field image. Furthermore, the CPU 301 can output the cutting point as voice using the microphone provided with the output device 306, or, if a vibrator is mounted on a laparoscope or surgical instruments, it can notify the cutting point by vibrating the vibrator.

[0171] Furthermore, since CPU 301 identifies loose connective tissue in step S223, it can also generate the identification image and overlay and display the generated identification image on the surgical field image. For example, CPU 301 can generate the identification image by assigning a color that does not exist inside the human body (e.g., a cool color such as blue or light blue) to the pixels identified as loose connective tissue, and setting the transmittance of the background for pixels outside the loose connective tissue. Moreover, since CPU 301 calculates the exposed area of ​​the loose connective tissue in step S224, it can also generate an image of an indicator showing the extent of the exposed area and overlay and display the generated indicator image on the surgical field image.

[0172] Figure 18 This is a schematic diagram showing an example of implementation 2. Figure 18 The following example is shown, which is aimed at Figure 13The surgical field image shown overlays and displays an identification image 53 of loose connective tissue, textual information 54 indicating the cutting time of the loose connective tissue, and an indicator 55 indicating the extent of exposure of the loose connective tissue. The image processing device 30 may also be structured such that this information is not required to be displayed frequently, but is displayed only when indicated by a display command from the surgeon via an input device 305 and a foot switch (not shown).

[0173] As described above, in Embodiment 2, the learning model 410 identifies the loose connective tissue portion included in the surgical field image, and based on the identification result, outputs information showing the cutting time point of the loose connective tissue. By outputting this navigation information, the image processing device 30 can provide visual assistance to the surgeon for the process involved in cutting the loose connective tissue between the tissue ORG constituting the preserved organ 2 and the tissue NG constituting the resected organ 3.

[0174] In Implementation 2, the decision on whether to cut is based on the exposed area of ​​loose connective tissue is made. However, the learning model 410 can also be used to identify loose connective tissue in a stretched state, and the appropriateness of the cut can be determined based on the identification result. For example, by selecting and labeling the portion of loose connective tissue in a stretched state at pixel-level, a learning model 410 for identifying loose connective tissue in a stretched state can be generated using training data obtained from such labeling. The CPU 301 of the image processing device 30 can identify loose connective tissue in a stretched state by inputting the surgical field image captured by the camera 21 into the learning model 410 and performing calculations using the learning model 410. When the CPU 301 identifies loose connective tissue in a stretched state using the learning model 410, it can notify the user with text information indicating the cut time point.

[0175] The image processing apparatus 30 in Embodiment 2 has a structure that includes a learning model 410, but it may also have a structure that includes learning models 420 to 450 as described below. That is, the image processing apparatus 30 may also be a structure that combines multiple learning models to generate navigation information.

[0176] (Implementation Method 3)

[0177] In Implementation 3, a learning model 410 is used to identify the loose connective tissue between the preserved organ 2 and the resected organ 3. Based on the identification results, a structure is described that outputs information related to the operation in handling the loose connective tissue as navigation information.

[0178] The image processing device 30 inputs the surgical field image into the learning model 410 and performs calculations using the learning model 410, thereby enabling it to identify the loose connective tissue portion contained in the surgical field image. In Embodiment 3, the image processing device 30, referring to the recognition results from the learning model 410, determines the operating direction of the surgical instrument 50 when unfolding or cutting the loose connective tissue, and outputs information related to the determined operating direction.

[0179] Figure 19 This is a schematic diagram illustrating an example of a surgical field image required for the deployment operation. Figure 19 The surgical field image shows the state of the loose connective tissue not being fully expanded. The image processing device 30 inputs this surgical field image into the learning model 410 and performs calculations from the learning model 410 to identify the presence of the loose connective tissue LCT exposed from the edge of the resected organ 3. However, as indicated by the indicator 55, since the loose connective tissue LCT does not have a sufficient exposed area, it is known that further expansion of the loose connective tissue LCT is necessary during dissection.

[0180] The image processing device 30 determines the unfolding direction of the loose connective tissue LCT based on the direction of its extension. For example, in Figure 19 In this example, the image processing device 30 can infer, based on the recognition results of the learning model 410, that the loose connective tissue LCT extends along the vertical direction of the image. Since there is a resected organ 3 above the loose connective tissue LCT and a retained organ 2 below it, the loose connective tissue LCT can be unfolded by pulling the resected organ 3 upwards using forceps 50A, or by pulling the retained organ 2 downwards using forceps 50A. The image processing device 30 outputs the unfolding direction of the loose connective tissue LCT as navigation information. Figure 20 This is a diagram showing an example of the output of navigation information related to the unfolding operation. Figure 20 The arrow marked M1 indicates the unfolding direction of the loose connective tissue (LCT) (i.e., the operating direction of the forceps 50A).

[0181] On the other hand, when the loose connective tissue LCT is fully expanded, the image processing device 30 outputs the cutting direction of the loose connective tissue LCT as navigation information. For example, in Figure 13 , Figure 18 In the example shown, since the image processing device 30 can infer from the recognition results of the learning model 410 that the loose connective tissue LCT is fully expanded in the vertical direction, it determines that the direction of cutting the loose connective tissue LCT is the horizontal direction of the image. The image processing device 30 outputs the cutting direction of the loose connective tissue LCT as navigation information. Figure 21 This is a diagram showing an example of the output of navigation information related to the cut-off operation. Figure 21The arrow marked M2 indicates the cutting direction of the loose connective tissue LCT (i.e., the operating direction of the energy treatment tool 50B).

[0182] exist Figure 20 as well as Figure 21 In this design, the markings M1 and M2 are displayed in larger sizes to emphasize the unfolding and cutting directions. However, since these markings M1 and M2 sometimes obscure a portion of the surgical field, it can be difficult to perform the surgery. Therefore, the display format (size, color, display time, etc.) of the markings M1 and M2 should be set in a way that allows the surgeon to ensure a clear view. Furthermore, the markings M1 and M2 are not limited to arrows; solid lines or dashed lines can also be used to indicate the unfolding and cutting directions. In this embodiment, the markings M1 and M2 are displayed overlapping with the surgical field image, but they can also be displayed as other images or on other monitors. Moreover, the CPU 301 can notify the unfolding and cutting directions via voice using the microphone provided with the output device 306, or, when a vibrator is mounted on a surgical instrument, it can notify the cutting time point by vibrating the vibrator.

[0183] Figure 22 This is a flowchart illustrating the steps performed by the image processing apparatus 30 in embodiment 3 during the application phase. Each time a surgical field image is acquired, the CPU 301 of the image processing apparatus 30 performs the following processing: The CPU 301 inputs the acquired surgical field image into the learning model 410 through the same steps as in embodiment 2, and performs calculations from the learning model 410 to identify loose connective tissue portions and calculate the exposed area of ​​the identified loose connective tissue portions (steps S301 to S304).

[0184] Based on the recognition results from the learning model 410, CPU 301 determines whether the loose connective tissue needs to be expanded (step S305). For example, CPU 301 compares the exposed area calculated in step S304 with a first threshold. If the calculated exposed area is less than the first threshold, it determines that the loose connective tissue needs to be expanded. Here, the first threshold is preset and stored in the storage device 303. Alternatively, when a surgical field image is input, CPU 301 can also use a learning model that learns by inputting information about whether the expansion of loose connective tissue is needed to determine whether the expansion of loose connective tissue is needed.

[0185] If it is determined that loose connective tissue needs to be unfolded (S305: Yes), CPU301 specifies the unfolding direction of the loose connective tissue (step S306). For example, CPU301 estimates the direction of the extension of loose connective tissue between the preserved organ 2 and the resected organ 3, and specifies the estimated direction as the unfolding direction of the loose connective tissue.

[0186] If it is determined that the unfolding of loose connective tissue is unnecessary (S305: No), CPU 301 determines whether the loose connective tissue can be cut (step S307). For example, CPU 301 compares the exposed area calculated in step S304 with a second threshold. If the calculated exposed area is greater than the second threshold, it is determined that the loose connective tissue can be cut. Here, the second threshold is set to a value greater than the first threshold and stored in the storage device 303. If it is determined that the loose connective tissue cannot be cut (S307: No), CPU 301 ends the processing of this flowchart.

[0187] If it is determined that the loose connective tissue can be cut (S307: Yes), CPU301 specifies the cutting direction of the loose connective tissue (step S308). For example, CPU301 may presuppose the direction in which the loose connective tissue extends between the preserved organ 2 and the resected organ 3, and specify the direction that intersects the presupposed direction as the cutting direction of the loose connective tissue.

[0188] Next, CPU 301 outputs navigation information when processing the connective tissue between the preserved organ 2 and the resected organ 3 (step S309). For example, in the case of a specific unfolding direction of the loose connective tissue in step S306, CPU 301 outputs navigation information by overlaying and displaying the marker M1 indicating the unfolding direction on the surgical field image. Additionally, in the case of a specific cutting direction of the loose connective tissue in step S308, CPU 301 outputs navigation information by overlaying and displaying the marker M2 indicating the cutting direction on the surgical field image.

[0189] As described above, in Embodiment 3, the unfolding direction and cutting direction of loose connective tissue can be used as navigation information to prompt the surgeon.

[0190] (Implementation Method 4)

[0191] In Implementation 4, the structure for learning the cutting time point of loose connective tissue will be described.

[0192] Figure 23This is a schematic diagram illustrating the structure of the learning model 420 in Embodiment 4. The learning model 420 is a learning model such as CNN (Convolutional Neural Networks) or R-CNN (Region-based CNN), and includes an input layer 421, an intermediate layer 422, and an output layer 423. The learning model 420 learns by outputting information about the time points of transection of loose connective tissue in response to an input surgical field image. The learning model 420 is generated using the image processing device 30 or an external server that can communicate with the image processing device 30, and stored in the model storage unit 331 of the image processing device 30.

[0193] Furthermore, the surgical field image input to the learning model 420 is not limited to video, but can also be a still image. Additionally, the surgical field image input to the learning model 420 does not need to be the original image obtained from the camera 21, but can also be an image with appropriate image processing, data showing the frequency components of the image, etc.

[0194] The surgical field image is input into input layer 421. Intermediate layer 422 includes convolutional layers, pooling layers, and fully connected layers. Multiple convolutional and pooling layers can be used alternately. The convolutional and pooling layers extract feature parts of the surgical field image by utilizing the operations of each layer's nodes. The fully connected layer combines the feature parts extracted by the convolutional and pooling layers at a single node, outputting feature variables transformed by an activation function. The feature variables are then output to the output layer through the fully connected layer.

[0195] The output layer 423 has one or more nodes. Based on the feature variables input from the fully connected layer of the intermediate layer 422, the output layer 423 converts them into probabilities using a softmax function, and outputs the converted probabilities from each node. In this embodiment, the output layer 423 simply outputs the probability indicating whether the current time point is the cutoff point of loose connective tissue.

[0196] The learning model 420 is generated by using a group of data, including surgical field images and data indicating whether the loose connective tissue contained in the surgical field image is at the time point of transection, as training data, and performing learning using appropriate learning algorithms such as CNN and R-CNN.

[0197] Figure 24This is a flowchart illustrating the steps performed by the image processing apparatus 30 in the application phase of embodiment 4. When the CPU 301 of the image processing apparatus 30 acquires a surgical field image captured by the camera 21 (step S451), the acquired surgical field image is input into the learning model 420, and the calculation of the learning model 420 is performed (step S452). The calculation from the learning model 420 is performed, thereby obtaining from the output layer 423 the probability indicating whether the current time point is a cutoff point for loose connective tissue.

[0198] Based on the probability obtained from the output layer 423 of the learning model 420, CPU 301 determines whether the current time is a cutoff point for loose connective tissue (step S453). For example, if the probability output by the output layer 423 of the learning model 420 exceeds a threshold (e.g., 50%), CPU 301 determines that the current time is a cutoff point for loose connective tissue; if it is below the threshold, it determines that it is not a cutoff point. If it is determined that it is not a cutoff point (S453: No), CPU 301 ends the processing of this flowchart.

[0199] If the time point for cutting is determined to be the cutting point (S453: Yes), the CPU301 notifies the cutting point of the loose connective tissue (step S454). The method of notifying the cutting point is the same as in embodiment 2, which can be to overlay and display text information, icons, marks, etc. on the surgical field image, or to notify via voice or vibration.

[0200] As described above, in Embodiment 4, by using the learning model 420 to determine the cutting time point of the loose connective tissue and output navigation information, visual assistance can be provided for the surgeon regarding the cutting of the loose connective tissue between the tissue ORG constituting the preserved organ 2 and the tissue NG constituting the resected organ 3.

[0201] (Implementation Method 5)

[0202] In Implementation 5, the structure for outputting information about the cut site of loose connective tissue will be described.

[0203] Figure 25 This is a schematic diagram illustrating the structure of the learning model 430 in Embodiment 5. The learning model 430 is a learning model of a neural network used for image segmentation or object detection, such as SegNet, FCN, U-Net, PSPNet, YOLO, or SSD, and includes, for example, an encoder 431, a decoder 432, and a softmax layer 433. Since the structure of the learning model 430 is the same as that of the learning model 410 in Embodiment 2, its description is omitted.

[0204] In this embodiment, the input image to the learning model 430 is a surgical field image obtained from the camera 21. The surgical field image input to the learning model 430 is not limited to video; it can also be a still image. Furthermore, the surgical field image input to the learning model 430 does not need to be the original image obtained from the camera 21; it can also be an image with appropriate image processing, data showing the frequency components of the image, etc. The learning model 430 learns in the following manner: for the input surgical field image, it outputs an image showing the excision site of the loose connective tissue portion included in the surgical field image.

[0205] When generating the learning model 430, training data is prepared, which consists of a surgical field image and correct data showing the cut site of the loose connective tissue contained in the surgical field image.

[0206] Figure 26 This is an explanatory diagram illustrating the steps for creating correct data. The operator (such as a doctor or specialist) annotates the surgical field image containing the loose connective tissue LCT. For example, the operator designates the portion corresponding to the loose connective tissue in the surgical field image, in pixels, and designates the cutting site of the loose connective tissue LCT using a band-shaped region CR. The image processing device 30 stores the pixels designated as the loose connective tissue LCT within this region CR as the cutting site of the loose connective tissue LCT (correct data) in the storage device 303.

[0207] In addition, if a learning model 410 for identifying loose connective tissue has been obtained, the identification image generated by the learning model 410 can be displayed on the sub-monitor 31 together with the surgical field image, and the region CR showing the cutting site can be designated in the displayed identification image.

[0208] Furthermore, the image processing device 30 can also be structured as follows: by analyzing the video of a doctor cutting loose connective tissue and the trajectory of a surgical instrument 50 such as a specific electrosurgical scalpel, the cutting site of the loose connective tissue in the surgical field image can be specified.

[0209] The image processing device 30 uses training data to generate a learning model 430, which consists of a surgical field image and correct data showing the cutting site in the loose connective tissue contained in the surgical field image. Since the learning process is the same as in Embodiment 2, its description is omitted.

[0210] In the application phase after obtaining the learning model 430 where learning has ended, the image processing device 30 performs the following processing. Figure 27This is a flowchart illustrating the steps performed by the image processing apparatus 30 of Embodiment 5 during the application phase. Each time the CPU 301 of the image processing apparatus 30 acquires a surgical field image, it performs the following processing: The CPU 301 inputs the acquired surgical field image into the learning model 430 according to the same procedure as in Embodiment 2, performs calculations derived from the learning model 430, and thereby identifies the cutting site of loose connective tissue (steps S501 to S503). That is, the CPU 301 determines whether each pixel corresponds to a cutting site by referring to the probability output from the softmax layer 433 of the learning model 430.

[0211] CPU 301 generates a recognition image showing the recognition result of the identified cut portion (step S504). CPU 301 can generate a recognition image showing the recognition result of the cut portion by extracting pixels whose probability of the label output from the softmax layer 433 is above a threshold (e.g., above 50%). CPU 301 assigns a color that does not exist inside the human body (e.g., a cool color such as blue or light blue) to the pixels identified as cut portions, and sets the transmittance of the background to the pixels other than those pixels.

[0212] CPU 301 overlays and displays the generated recognition image on the surgical field image (step S505). Thus, the incision site identified by the learning model 430 is displayed on the surgical field image as a region with a specific color. Additionally, CPU 301 can also display information indicating that the region shown in the specific color is the site that should be removed on the surgical field image.

[0213] Figure 28 This is a schematic diagram showing an example of a cut-off portion. For ease of drawing, in... Figure 28 In the example, the cutting site 53a identified by the learning model 430 is shown using a thick solid line. In fact, since the portion corresponding to the cutting site is painted with colors such as blue or light blue, which do not exist inside the human body, on a pixel-by-pixel basis, the surgeon can clearly grasp the cutting site of the loose connective tissue by observing the display on the sub-monitor 31.

[0214] As described above, in Embodiment 5, surgical assistance can be provided by pointing out the cutting site of the loose connective tissue to the surgeon.

[0215] Furthermore, in this embodiment, the learning model 430 is used to identify the structure of the cut site of loose connective tissue. However, it is also possible to distinguish between loose connective tissue contained in the surgical image and the cut site of that loose connective tissue, and display them in different colors. For example, the part identified as loose connective tissue is displayed in green, and the part identified as the cut site in the loose connective tissue is displayed in blue. In addition, the density and transmittance can be changed according to the confidence level of the identification result (the probability output from the softmax layer 433), using a light color (a color with higher transmittance) to display the part with lower confidence level, and a dark color (a color with lower transmittance) to display the part with higher confidence level.

[0216] Alternatively, it can be a combination of Embodiment 2 (or Embodiment 4) and Embodiment 5, where, when the cutting time point is identified, the identification image of the cutting part is overlaid.

[0217] Furthermore, in this embodiment, the structure is to color and display the parts identified as cut sites in pixel units. However, it is also possible to generate a region or line containing pixels identified as cut sites, and to overlay the generated region or line as an identification image of the cut site on the surgical field image.

[0218] (Implementation Method 6)

[0219] In Implementation 6, the structure for scoring anatomical status based on surgical field images will be described.

[0220] The so-called anatomical condition score refers to using numerical values ​​to indicate the quality of the anatomical condition. In this embodiment, the score is defined as follows: if the anatomical condition is good, the score is high; if the anatomical condition is poor, the score is low. The image processing device 30 calculates the anatomical condition score by inputting the surgical field image captured by the camera 21 into the learning model 440 described below, after acquiring the surgical field image.

[0221] Furthermore, the scores of Embodiment 1 and Embodiment 6 (and Embodiments 7-8 below) together show the scores related to the anatomical state, but in the scores of Embodiment 1, the scores of Embodiments 6-8 indicate the degree to which the state of the connective tissue is suitable for treatment, and the scores of Embodiments 6-8 indicate the quality of the anatomical results.

[0222] Figure 29This is a schematic diagram illustrating the structure of the learning model 440 in Embodiment 6. The learning model 440 is a learning model such as CNN or R-CNN, and includes an input layer 441, an intermediate layer 442, and an output layer 443. The learning model 440 learns from an input surgical field image by inputting information related to the score of the anatomical state. The learning model 440 is generated using the image processing device 30 or an external server that can communicate with the image processing device 30, and is stored in the model storage unit 331 of the image processing device 30.

[0223] Furthermore, the surgical field image input to the learning model 440 is not limited to video, but can also be a still image. Additionally, the surgical field image input to the learning model 440 does not need to be the original image obtained from the camera 21, but can also be an image with appropriate image processing, data showing the frequency components of the image, etc.

[0224] The surgical field image is input into input layer 441. Intermediate layer 442 includes convolutional layers, pooling layers, and fully connected layers. Multiple convolutional and pooling layers can be used alternately. The convolutional and pooling layers extract feature parts of the surgical field image by utilizing the node operations of each layer. The fully connected layer combines the data extracted by the convolutional and pooling layers with a node, and outputs the transformed feature variable through an activation function. The feature variable is then output to the output layer through the fully connected layer.

[0225] The output layer 443 has one or more nodes. Based on the feature variables input from the fully connected layer of the intermediate layer 442, the output layer 443 converts them into probabilities using the softmax function, and outputs the converted probabilities from each node. For example, the output layer 443 can consist of n nodes from the first node to the Nth node. The first node outputs the probability P1 of score S1, the second node outputs the probability P2 of score S2 (>S1), the third node outputs the probability P3 of score S3 (>S2), ..., and the Nth node outputs the probability Pn of score Sn (>Sn-1).

[0226] The learning model 440 shown in Implementation 6 has an output layer 443 with a probability of outputting each score. However, in the case of input surgical field images, a regression model that learns by calculating the scores of the anatomical state can be used instead of this learning model 440.

[0227] The learning model 440 is generated by using a combination of a surgical field image and a score (correct data) determined for the anatomical state of that surgical field image as training data, and learning based on an appropriate learning algorithm. The score for the anatomical state used in the training data is determined, for example, by a physician. The physician confirms the anatomical state based on the surgical field image and determines the score based on factors such as the exposed area of ​​connective tissue, the stretching state of the connective tissue, the amount of structures (attachment) such as blood vessels and fat surrounding the connective tissue, and the degree of damage to the preserved organs. Alternatively, the score for the anatomical state used in the training data can also be determined using the image processing device 30. For example, the image processing device 30 uses the aforementioned learning model 410 to identify loose connective tissue contained in the surgical field image, and evaluates the exposed area, stretching state, and amount of surrounding structures of the loose connective tissue based on the identification result, determining the score based on the evaluation result. Furthermore, the image processing device 30 can also determine the score by identifying the preserved organs and evaluating differences in color (burn marks), shape, etc., compared to normal.

[0228] The image processing device 30 uses the generated learning model 440 to score the anatomical state shown in the surgical field image. Figure 30 This is a flowchart illustrating the steps of processing performed by the image processing apparatus 30 in Embodiment 6. When the CPU 301 of the image processing apparatus 30 acquires a surgical field image captured by the camera 21 (step S601), it inputs the acquired surgical field image into the learning model 440 and performs calculations on the learning model 440 (step S602). By performing calculations from the learning model 440, a probability for each score is obtained from each node constituting the output layer 443.

[0229] CPU 301 refers to the output of learning model 440, and the score of a specific anatomical state (step S603). Since the probability for each score is obtained from each node of output layer 443, the score with the highest probability is selected by CPU 301, thus obtaining the score of the specific anatomical state.

[0230] CPU 301 associates the surgical field image with a specific score and stores it in storage device 303 (step S604). At this time, CPU 301 may also associate all surgical field images with scores and store them in storage device 303. Alternatively, CPU 301 may only extract surgical field images with scores greater than (or less than) a specified value and store them in storage device 303.

[0231] As described above, in Embodiment 6, the anatomical state contained in the surgical field image can be scored, and the surgical field image is associated with the score and stored in the storage device 303. The surgical field image associated with the score and stored can be used for surgical evaluation, educational assistance for resident physicians, etc.

[0232] (Implementation Method 7)

[0233] In Embodiment 7, a structure for instructing the assistant personnel (helpers) who assist in endoscopic surgery will be described.

[0234] The learning model 440 in Implementation 7 learns by outputting a score related to the state of the target region when given an input surgical field image. The score is defined as follows: for example, a higher score indicates a suitable state for resection, while a lower score indicates an unsuitable state. Here, a suitable state for resection refers to a state where the target region is stretched at three points, creating a triangular planar portion. Conversely, an unsuitable state for resection refers to a state where the target region is not sufficiently expanded, resulting in relaxation.

[0235] The internal structure of the learning model 440 is the same as that of implementation method 6, including an input layer 441, an intermediate layer 442, and an output layer 443. The learning model 440 is generated by using a surgical field image and a set of scores (correct data) representing the state of the target region in that surgical field image as training data, and learning based on an appropriate learning algorithm. Furthermore, the scores representing the state of the target region used in the training data are determined, for example, by a physician. The physician determines the score based on the state of the target region in the surgical field image, for example, from the perspective of whether it is suitable for resection.

[0236] Figures 31-33 This is an explanatory diagram illustrating the state of the area being processed. Figure 31 The illustration shows two different portions of fat tissue (FAT) being held with forceps 50A and slightly lifted. In this example, the area being treated is not fully expanded, resulting in laxity and making it difficult to remove. Because when... Figure 31 The image processing device 30 provides operational assistance to the assistant operator in the case of the surgical field image input learning model 440, which has a low score. The image processing device 30 assists the assistant operator by, for example, by expanding the processing area and eliminating slack. Instructions to the assistant operator are given through displays on the secondary monitor 31, voice output, etc.

[0237] Figure 32The illustration shows two different portions of adipose tissue (FAT) being held using clamps 50A, with the FAT being fully lifted. However, the narrow spacing between the two clamps 50A creates a slack between them. [The illustration continues with further details about the technique.] Figure 32 In the case of the surgical field image input learning model 440 shown, although the score is improved, insufficient resection leads the image processing device 30 to provide operational assistance to the assistant in order to further improve the score. The image processing device 30 provides operational assistance, for example, by instructing the assistant to expand the area to be processed and eliminate slack. Instructions to the assistant are made through displays on the sub-monitor 31, voice output, etc.

[0238] Furthermore, the image processing device 30 can also evaluate the state of the area to be processed and provide instructions to an assistant based on the evaluation results. For example, when holding adipose tissue (FAT) in two locations, the image processing device 30 can evaluate the resulting ridgelines. Figure 31 as well as Figure 32 In the diagram, the edges are shown as dashed lines. Because... Figure 31 The ridges shown are straight, resulting in less relaxation in the lateral direction. However, due to insufficient upward lifting (the distance between the ridges and the organs is short), relaxation occurs in the vertical direction. Therefore, the image processing device 30 can also instruct the assistant to lift the adipose tissue (FAT) using forceps 50A. Figure 32 The upward lifting amount is sufficient, but because the ridge is arc-shaped, it causes relaxation in the left-right direction. Therefore, the image processing device 30 can also instruct the assistant to spread the adipose tissue (FAT) in the left-right direction using forceps 50A.

[0239] Figure 33 The following state is shown: two different portions of the fat tissue (FAT) are held with forceps 50A, the FAT is fully lifted, and spread out in the left-right direction. Since the score is sufficiently improved when this surgical field image is input into the learning model 440, it is a suitable state for resection, and therefore the image processing device 30 provides operational assistance to the surgeon. The image processing device 30 provides operational assistance by instructing the surgeon to excise the area to be treated. Instructions to the surgeon are made through display on the sub-monitor 31, voice output, etc.

[0240] As described above, in Implementation 7, the learning model 440 can be used to grasp the state of the processing object area, and based on the grasped state of the processing object area, operational assistance can be provided to the surgical personnel and assistants.

[0241] (Implementation Method 8)

[0242] In Implementation 8, the following structure will be described: the structure generates a predicted image of the surgical field and outputs the generated predicted image as navigation information.

[0243] Figure 34 This is an explanatory diagram illustrating the learning model of Embodiment 8. In Embodiment 8, for an input surgical field image, an image generation model is used to generate a surgical field image (predicted image) with a slightly improved score.

[0244] The learning model 450 is an image generation model that is learned in the following way: for example, given a surgical field image with a score range of 0-10 (0 to 10), it outputs a predicted image with a score range of 10-20. As an image generation model, it can utilize GAN (Generative Adversarial Network), VAE (Variational Autoencoder), autoencoder, stream-based generative models, etc. This learning model 450 is generated in the following way: for example, it can read surgical field images with a score range of 0-10 (input data) and surgical field images with a score range of 10-20 (correct data) from storage device 303, and use these sets as training data for learning.

[0245] Similarly, an image generation model is prepared in advance, which is an example of a learning model that generates a surgical field image with a slightly improved score for an input surgical field image in the following manner: when the input surgical field image has a score range of 10-20, it learns by outputting a predicted image with a score range of 20-30; when the input surgical field image has a score range of 20-30, it learns by outputting a predicted image with a score range of 30-40.

[0246] Furthermore, the surgical field image input to the image generation model is not limited to video; it can also be a still image. Additionally, the surgical field image input to the image generation model does not need to be the original image obtained from camera 21; it can be an image with appropriate image processing, data showing the frequency components of the image, etc. Furthermore, the scoring range of the surgical field image input to the image generation model and the scoring range of the surgical field image output from the image generation model are not limited to the above ranges and can be appropriately set. Moreover, while this embodiment uses actually captured surgical field images for learning correct data, virtual surgical field images drawn using 3D graphics can also be used as correct data.

[0247] In the application stage after the image generation model is generated, the image processing device 30 uses the image generation model to generate a predicted image to provide prompts to the surgical personnel. Figure 35This is a flowchart illustrating the processing steps performed by the image processing apparatus 30 of Embodiment 8. When the CPU 301 of the image processing apparatus 30 acquires a surgical field image captured by the camera 21 (step S801), it inputs the acquired surgical field image into the learning model 440 described in Embodiment 6 and performs calculations from the learning model 440 (step S802). By performing calculations from the learning model 440, a probability for each score is obtained from each node constituting the output layer 443.

[0248] CPU301 refers to the output learning model 440, and the score of a specific anatomical state (step S803). Since the probability for each score can be obtained from each node of the output layer 443, the score with the highest probability is selected by CPU301, thus obtaining the score of the specific anatomical state.

[0249] CPU 301 selects an image generation model based on a specific score (step S804). For example, if the score specified in step S803 is a scoring range of 0-10, then CPU 301 selects learning model 450. The same applies if the score specified in step S803 is another scoring range; CPU 301 selects the learning model prepared for each scoring range.

[0250] CPU 301 inputs the surgical field image to the selected image generation model and performs calculations from the image generation model to generate a predicted image (step S805). CPU 301 displays the generated predicted image on the sub-monitor 31 (step S806). At this time, CPU 301 can display the predicted image separately from the surgical field image.

[0251] As described above, in embodiment 8, since a predicted image of the anatomical state can be displayed, it is possible to provide operational assistance to the surgeon.

[0252] The above description of this embodiment is intended to make the present invention easier to understand and is not intended to limit the scope of the invention. The present invention can be modified and improved without departing from its purpose, and its equivalents are also included.

[0253] Explanation of reference numerals in the attached figures:

[0254] 311: Image Acquisition Department

[0255] 312: Anatomical Status Acquisition Department

[0256] 313: Display unit for manageable areas

[0257] 314: Learning Processing Department

[0258] 315: Fraction Calculation Department

[0259] 316: Fraction Output Department

[0260] 331: Model Storage Department

Claims

1. A computer program product comprising a computer program, characterized in that, This computer program is used to cause the computer to perform the following processes: To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information identifying the connective tissue between the preserved and resected organs contained in the surgical field image. The learning model is designed to learn by outputting information identifying the connective tissue between the preserved and resected organs given an input surgical field image. Based on the information obtained from the learning model, the exposed area of ​​the connective tissue is calculated. Based on the calculated exposed area, the time point for transection of the connective tissue is determined. Information related to the determined cut time point is output as navigation information when dealing with the connective tissue between the preserved and removed organs.

2. A computer program product comprising a computer program, characterized in that, This computer program is used to cause the computer to perform the following processes: To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information identifying the connective tissue between the preserved and resected organs contained in the surgical field image. The learning model is designed to learn by outputting information identifying the connective tissue between the preserved and resected organs given an input surgical field image. Based on the information obtained from the learning model, the exposed area of ​​the connective tissue is calculated. An indicator showing the calculated amount of exposed area will be output as navigation information when dealing with connective tissue between preserved and resected organs.

3. A computer program product comprising a computer program, characterized in that, This computer program is used to cause the computer to perform the following processes: To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image and acquires information identifying connective tissue contained within the surgical field image that exists between the preserved and resected organs and is in a stretched state. The learning model learns by outputting information identifying connective tissue existing between the preserved and resected organs and in a stretched state, given an input surgical field image. Based on the information acquired from the learning model, it is determined whether the transection of the connective tissue is appropriate. Information related to whether the transection of the connective tissue was appropriate is output as navigation information when dealing with connective tissue between preserved and resected organs.

4. A computer program product comprising a computer program, characterized in that, This computer program is used to cause the computer to perform the following processes: To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information about the cutting time points of the connective tissue contained in the surgical field image. The learning model is designed to learn by outputting information about the cutting time points of the connective tissue between the preserved and resected organs, given an input surgical field image. The information related to the cutting time point obtained from the learning model will be output as navigation information when dealing with the connective tissue between preserved and resected organs.

5. A computer program product comprising a computer program, characterized in that, This computer program is used to cause the computer to perform the following processes: To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information about the connective tissue contained in the surgical field image. The learning model learns by outputting a score related to the anatomical state, based on the information about the connective tissue between the preserved and resected organs, given the input surgical field image. Based on the information obtained from the learning model, navigation information is output when dealing with connective tissue between organs that are preserved and those that are removed.

6. The computer program product according to claim 5, characterized in that, The learning model learns by varying the score based on at least one of the following: the exposed area of ​​the connective tissue, the tensile state of the connective tissue, the amount of structures present around the connective tissue, and the degree of damage to the preserved organs.

7. The computer program product according to claim 5, characterized in that, The computer program is used to cause the computer to perform the following processes: The surgical field image input to the learning model is associated with the score obtained by inputting the surgical field image into the learning model and stored in a storage device.

8. The computer program product according to claim 5, characterized in that, The computer program is used to cause the computer to perform the following processes: The learning model learns by changing the score based on the state of the processing object region. Based on the score output from the learning model, instructions are output to the assistants assisting in the endoscopic surgery.

9. The computer program product according to any one of claims 1 to 8, characterized in that, The computer program is used to cause the computer to perform the following processes: An image generation model, trained to generate a predicted image of the anatomical state given an input surgical field image, takes the acquired surgical field image as input and obtains a predicted image for the surgical field image. The acquired predicted image is output as the navigation information.

10. An image processing apparatus, characterized in that, The image processing device includes: CPU The CPU, Obtain surgical field images from endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information identifying the connective tissue between the preserved and resected organs contained in the surgical field image. The learning model is one that learns by outputting information identifying the connective tissue between the preserved and resected organs given an input surgical field image. Based on the information obtained from the learning model, the exposed area of ​​the connective tissue is calculated. Based on the calculated exposed area, the time point for transection of the connective tissue is determined. Information related to the determined cut time point is output as navigation information when dealing with the connective tissue between the preserved and removed organs.

11. An image processing apparatus, characterized in that, The image processing device includes: CPU The CPU, To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information identifying the connective tissue between the preserved and resected organs contained in the surgical field image. The learning model is designed to learn by outputting information identifying the connective tissue between the preserved and resected organs given an input surgical field image. Based on the information obtained from the learning model, the exposed area of ​​the connective tissue is calculated. An indicator showing the calculated amount of exposed area will be output as navigation information when dealing with connective tissue between preserved and resected organs.

12. An image processing apparatus, characterized in that, The image processing device includes: CPU The CPU, To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image and obtains information identifying connective tissue contained within the surgical field image that exists between the preserved and resected organs and is in a stretched state. The learning model learns by outputting information identifying connective tissue existing between the preserved and resected organs and in a stretched state, given an input surgical field image. Based on the information obtained from the learning model, it is determined whether the cutting of the connective tissue is appropriate. Information related to whether the transection of the connective tissue was appropriate is output as navigation information when dealing with connective tissue between preserved and resected organs.

13. An image processing apparatus, characterized in that, The image processing device includes: CPU The CPU, To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information about the cutting time points of the connective tissue contained in the surgical field image. The learning model is designed to learn by outputting information about the cutting time points of the connective tissue between the preserved and resected organs, given an input surgical field image. The information related to the cutting time point obtained from the learning model will be output as navigation information when dealing with the connective tissue between preserved and resected organs.

14. An image processing apparatus, characterized in that, The image processing device includes: CPU The CPU, To obtain surgical field images obtained by photographing the surgical field during endoscopic surgery. The learning model is fed with an acquired surgical field image to obtain information about the connective tissue contained in the surgical field image. The learning model learns by outputting a score related to the anatomical state, based on the information about the connective tissue between the preserved and resected organs, given the input surgical field image. Based on the information obtained from the learning model, navigation information is output when dealing with connective tissue between organs that are preserved and those that are removed.

Images