Image processing device, surgery support system, image processing method, and program

Abstract

An image processing device according to the present invention comprises: an image acquisition unit that acquires a surgical field image obtained by imaging a surgical field in endoscopic surgery; a region recognition unit that inputs the surgical field image acquired by the image acquisition unit into a trained model, which has learned training data obtained by adding a region indicating a tissue classification to a training image obtained by imaging a surgical field, and thereby recognizes a region included in the surgical field image; a structure detection unit that detects a fibrous structure which is included in the surgical field image acquired by the image acquisition unit; and an image output unit that outputs a surgical field image indicating the fibrous structure which is included in the region recognized by the region recognition unit.

Description

Image processing device, surgery support system, image processing method and program

[0001] The present invention relates to an image processing device, a surgery assistance system, an image processing method, and a program.

[0002] In endoscopic surgery, an image of the surgical field taken inside the patient's body by an endoscope is output to a monitor installed in the operating room, and the surgeon operates various surgical instruments while viewing the surgical field image. In endoscopic surgery, it is important for the surgeon to properly grasp from the surgical field image which tissues to be removed and which tissues to be preserved and which tissues to dissect.

[0003] A surgical support device is used to teach a surgeon the location of loose connective tissue, known as an appropriate separation layer. For example, Patent Literature 1 discloses a surgical support device that inputs an image of a surgical field obtained by imaging the surgical field of laparoscopic surgery into a learning model and recognizes loose connective tissue portions included in the surgical field image.

[0004] International Publication No. 2022 / 25151

[0005] However, conventional techniques have the problem that generating a learning model to output highly visible surgical field images is time-consuming. For example, because loose connective tissue has a fibrous structure, generating a learning model requires preparing training data in which labels indicating each fiber are attached to the surgical field image.

[0006] In view of the above technical problems, one aspect of the present invention aims to output a surgical field image with high visibility.

[0007] An image processing device according to one aspect of the present invention includes an image acquisition unit that acquires an operative field image captured from the operative field of laparoscopic surgery, a region recognition unit that recognizes regions included in the operative field image by inputting the operative field image acquired by the image acquisition unit into a trained model that has learned from training data in which regions indicating tissue classification are added to training images of the operative field, a structure detection unit that detects fibrous structures included in the operative field image acquired by the image acquisition unit, and an image output unit that outputs an operative field image showing the fibrous structures included in the region recognized by the region recognition unit.

[0008] According to one aspect of the present invention, it is possible to output a surgical field image with high visibility.

[0009] 1 is a diagram showing an example of a navigation image in which a region is filled in. FIG. 2 is a diagram showing an example of a navigation image showing a fibrous structure within a region. FIG. 3 is a block diagram showing an example of the overall configuration of a surgery assistance system. FIG. 4 is a block diagram showing an example of the hardware configuration of a computer. FIG. 5 is a block diagram showing an example of the functional configuration of a learning device in the first embodiment. FIG. 6 is a block diagram showing an example of the functional configuration of an image processing device in the first embodiment. FIG. 7 is a diagram showing an example of the flow of image processing. FIG. 8 is a flowchart showing an example of learning processing. FIG. 9 is a flowchart showing an example of model learning processing. FIG. 10 is a flowchart showing an example of recognition processing. FIG. 11 is a diagram showing an example of a differential filter. FIG. 12 is a diagram showing an example of thinning processing. FIG. 13 is a diagram showing an example of parameters of a Canny edge detector. FIG. 14 is a diagram showing an example of a navigation image for each edge detector. FIG. 14 is a diagram showing an example of the detection speed for each edge detector. FIG. 15 is a block diagram showing an example of the functional configuration of a learning device in the second embodiment. FIG. 16 is a block diagram showing an example of the functional configuration of an image processing device in the second embodiment.

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0011] [First Embodiment] One embodiment of the present invention is a surgery support system that supports a surgeon performing endoscopic surgery. The surgery support system in this embodiment outputs an image (hereinafter also referred to as a "navigation image") showing fibrous structures included within a predetermined region in an image (hereinafter also referred to as a "surgical field image") captured inside a patient's body using an endoscope to a display device such as a monitor installed in an operating room. The surgery support system in this embodiment allows a surgeon to easily grasp the position, range, shape, etc. of tissue having a fibrous structure (hereinafter also referred to as "fibrous tissue") in the surgical field by viewing the navigation image output on the display device.

[0012] In endoscopic surgery, multiple drilling instruments called trocars are attached to the surface of the patient's body, such as the abdomen or chest, and surgical instruments such as an endoscope, an electric scalpel, and forceps are inserted into the patient's body through the holes in the trocars. The surgeon performs procedures such as resecting the affected area using an electric scalpel while viewing the operative field image captured by the endoscope in real time on a monitor. The surgeon is a medical professional involved in endoscopic surgery, and includes the surgeon, assistants, nurses, and doctors monitoring the surgery.

[0013] In laparoscopic surgery, the surgeon must properly select the tissue to be dissected and proceed with the surgery. In particular, it is important to accurately visually confirm the location, extent, and shape of the fibrous tissue present in the surgical field. Examples of fibrous tissue include loose connective tissue, nerves, and blood vessels. Loose connective tissue is known as an appropriate dissection layer. Resection of nerves and blood vessels is tissue that should be avoided as much as possible.

[0014] One example of a method for improving the visibility of a surgical field image is to recognize regions corresponding to the classification of tissues present in the surgical field based on a trained region segmentation model and display them on the surgical field image. Here, the classification of tissues may include, for example, a region indicating the range of tissue to be excised (hereinafter also referred to as an "excision region"), a region indicating the range of tissue to be preserved (hereinafter also referred to as a "preservation region"), a region indicating the range of tissue to be dissected (hereinafter also referred to as a "recommended dissection region"), and a region indicating the range of other tissues that do not belong to any of the above (hereinafter also referred to as an "other region").

[0015] One example of a method for indicating a region on a surgical field image is to fill the region with a predetermined color. FIG. 1 is a diagram showing an example of a navigation image in which the region is filled. FIG. 1 shows an example in which a recommended dissection region 911 included in a navigation image 910 is displayed by filling it with a transparent fluorescent color. Note that in FIG. 1, the fluorescent color is indicated by hatching. As shown in FIG. 1, because the recommended dissection region 911 is filled in on the navigation image 910, the visibility of the color, shape, etc. of the tissue present within that range is reduced.

[0016] For example, since loose connective tissue is known as an appropriate dissection layer, displaying the loose connective tissue present in the recommended dissection area as fibers allows the surgeon to accurately visually identify the areas to be dissected.Furthermore, since nerves and blood vessels are tissues that should be avoided from being resected, displaying the nerves and blood vessels present in the preservation area as fibers allows the surgeon to accurately identify the areas to be resected with particular care.

[0017] Fig. 2 is a diagram showing an example of a navigation image showing a fibrous structure within a region. Fig. 2 shows an example in which loose connective tissue within a recommended dissection region 921 included in a navigation image 920 is displayed in a fibrous form. As shown in Fig. 2, in the navigation image 920, the recommended dissection region 921 is displayed in a fibrous form, thereby improving the visibility of the color, shape, etc. of the tissue within that range.

[0018] To recognize fibrous structures contained in surgical field images based on a region segmentation model, the region segmentation model must be able to accurately recognize each fiber that constitutes the fibrous structure. To achieve this, correct labels indicating each fiber must be assigned to the surgical field images (hereinafter also referred to as "training images") used to train the region segmentation model. Since correct labeling is typically performed manually, it requires significant effort and time to accurately assign correct labels to regions with complex shapes. Furthermore, because it is difficult to assign correct labels to regions with complex shapes using a pointing device such as a general mouse, specialized input devices for image editing, such as pen tablets, are sometimes used. Therefore, creating training data to train a region segmentation model capable of recognizing fibrous structures requires significant costs.

[0019] The present embodiment aims to improve the visibility of fibrous structures in surgical field images captured by an endoscope. In particular, it aims to reduce the cost of training a region segmentation model used to recognize regions containing fibrous structures. In one aspect, this embodiment allows a surgeon performing endoscopic surgery to properly grasp the position, range, shape, etc. of fibrous structures present within a specified region. This enables the surgeon to perform endoscopic surgery safely and efficiently.

[0020] <Overall Configuration of Surgery Support System> The overall configuration of the surgery support system in this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the overall configuration of the surgery support system in this embodiment.

[0021] 3, the surgery assistance system 1 in this embodiment includes an endoscope 2, an imaging control device 3, a display device 4, a learning device 5, and an image processing device 6. The endoscope 2 and the display device 4 are installed in the same operating room. The imaging control device 3, the learning device 5, and the image processing device 6 may be installed in the same operating room as the endoscope 2 and the display device 4, or may be installed in a different facility.

[0022] The endoscope 2 and the imaging control device 3 are electrically connected by a universal cord or the like. The display device 4 and the image processing device 6 are electrically connected by a multimedia interface or the like. The display device 4 may be electrically connected to the imaging control device 3. The imaging control device 3, the learning device 5, and the image processing device 6 are connected to each other so as to be able to communicate data with each other via a communication network such as a LAN (Local Area Network).

[0023] The endoscope 2 is a medical device that captures images of the inside of a patient's body. The endoscope 2 has an insertion section that is inserted into the patient's body, an imaging device such as a camera located at the tip of the insertion section, an operation section that is located at the rear end of the insertion section, and a connection section that connects to the imaging control device 3. By operating the operation section, the surgeon can control the curvature of the insertion section up, down, left, and right, and move the insertion section to a desired position inside the patient's body.

[0024] The imaging device of the endoscope 2 captures images of the inside of the patient's body constantly or at desired timing under the control of the imaging control device 3, and inputs surgical field images via a connection unit to the imaging control device 3. The imaging device may capture still images or moving images (i.e., a time series of still images).

[0025] The imaging control device 3 is an electronic device or information processing device that controls the imaging device of the endoscope 2. The imaging control device 3 may be connected to a light source device or an image recording device, or may be a device integrated with the light source device and image recording device. The light source device is an optical device that emits illumination light that is irradiated onto the surgical field from the tip of the insertion portion of the endoscope 2. The image recording device is a storage device that records images of the surgical field captured by the imaging device of the endoscope 2.

[0026] The imaging control device 3 transmits control signals to the imaging device of the endoscope 2 via the connection part of the endoscope 2. The control signals are signals that instruct the start or stop of imaging, the emission or stop of illumination light, change of imaging magnification, etc. The imaging control device 3 transmits the operative field image input from the endoscope 2 to the image processing device 6 via the communication network. The imaging control device 3 may output the operative field image to the display device 4.

[0027] The learning device 5 is an information processing device such as a personal computer, workstation, or server that learns a region segmentation model that recognizes regions according to tissue classification from operative field images captured by the imaging device of the endoscope 2. The learning device 5 generates training data by assigning correct labels indicating the range of a predetermined region to training images collected in advance. The learning device 5 learns the region segmentation model based on the training data including training images assigned correct labels.

[0028] The image processing device 6 is an information processing device such as a personal computer, workstation, or server that performs predetermined image processing on the operative field image input from the imaging control device 3. The image processing device 6 recognizes areas included in the operative field image by inputting the operative field image into a trained region segmentation model. The image processing device 6 also detects fibrous structures from the operative field image using an edge detector. The image processing device 6 superimposes the region recognition results and fibrous structure detection results on the operative field image. In other words, the image processing device 6 generates a navigation image that shows the fibrous structures included in the predetermined regions on the operative field image. The image processing device 6 outputs the navigation image to the display device 4.

[0029] The display device 4 displays on the display the navigation image input from the image processing device 6. When a surgical field image is input from the imaging control device 3, the display device 4 displays the navigation image and the surgical field image so that they can be compared. For example, the display device 4 may display the navigation image and the surgical field image side by side in the vertical or horizontal direction, or may display them in a switchable manner in response to an operation by the surgeon or the like.

[0030] The overall configuration of the surgery assistance system 1 shown in Figure 3 is an example, and various system configurations are possible depending on the application and purpose. For example, the learning device 5 and the image processing device 6 may be implemented using multiple computers, or may be implemented as a cloud computing service. Furthermore, for example, the surgery assistance system 1 may be implemented as a standalone information processing device that combines the functions that two or more of the imaging control device 3, learning device 5, and image processing device 6 should have. Furthermore, for example, the surgery assistance system 1 may be implemented as an image display system in which the functions that the image processing device 6 should have are incorporated into the display device 4. The division of devices such as the endoscope 2, imaging control device 3, display device 4, learning device 5, and image processing device 6 shown in Figure 3 is an example.

[0031] <Hardware Configuration of Surgery Support System> The hardware configuration of the surgery support system 1 in this embodiment will be described with reference to FIG.

[0032] <Hardware Configuration of Computer> The imaging control device 3, learning device 5, and image processing device 6 in this embodiment are realized by, for example, a computer. Fig. 4 is a block diagram showing an example of the hardware configuration of a computer 500 in this embodiment.

[0033] 4, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The hardware components of the computer 500 are connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0034] The CPU 501 is a computing device that reads programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executes processing to realize overall control and functions of the computer 500. The computer 500 may have a GPU (Graphics Processing Unit) in addition to or instead of the CPU 501.

[0035] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs, data, etc. required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 starts up, as well as data such as OS (Operating System) settings and network settings.

[0036] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when executed by the CPU 501.

[0037] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses flash memory as a storage medium (e.g., an SSD (Solid State Drive)) instead of the HDD 504.

[0038] The input device 505 includes a touch panel, operation keys and buttons, a keyboard and mouse, a microphone for inputting sound data such as voice, and the like, which are used by the user to input various signals.

[0039] The display device 506 is composed of a display such as a liquid crystal display or organic electroluminescence (EL) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0040] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0041] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0042] The drive device 510 is a device for loading a recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 511 may also include semiconductor memories that record information electrically, such as ROMs and flash memories. This allows the computer 500 to read from and / or write to the recording medium 511 via the external I / F 508.

[0043] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading the various programs recorded on the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded via the communication I / F 507 from a network different from the communication network.

[0044] <Functional Configuration of Surgery Support System> The functional configuration of the surgery support system in this embodiment will be described with reference to FIGS. 5 and 6. FIG.

[0045] 5 is a block diagram showing an example of the functional configuration of the learning device 5 according to this embodiment. As shown in FIG. 5, the learning device 5 according to this embodiment includes an image storage unit 101, a label assignment unit 102, and a model learning unit 103.

[0046] The image storage unit 101 is realized by the HDD 504 shown in Fig. 4. The labeling unit 102 and the model learning unit 103 are realized by the processing that the CPU 501 executes by a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 4.

[0047] Training images for training a region segmentation model are stored in the image storage unit 101. The training images are stored in the image storage unit 101 by receiving image data of the training images input to the input device 505 by a user operation. The user is, for example, an administrator of the surgery assistance system 1.

[0048] The labeling unit 102 assigns correct labels to the training images read from the image storage unit 101. The correct labels in this embodiment include correct values ​​for regions corresponding to tissue classifications. Specifically, the correct labels are two-dimensional data that assign a label indicating the classification of the region to each pixel of the surgical field image. The labels may include an excision region, a preservation region, a region recommended for removal, or other regions. The labels may be expressed, for example, by the name of the tissue. The types of labels are not limited to these, and any label that can classify tissues may be used. The labeling unit 102 stores training data including a plurality of training images to which correct labels have been assigned in a storage unit such as the HDD 504.

[0049] The model learning unit 103 learns a region segmentation model based on the training data generated by the labeling unit 102. The learning algorithm may be selected depending on the type of region segmentation model. The model learning unit 103 transmits the learned region segmentation model to the image processing device 6.

[0050] 6 is a block diagram showing an example of the functional configuration of the image processing device 6 in this embodiment. As shown in Fig. 6, the image processing device 6 in this embodiment includes a model storage unit 201, a detector setting unit 202, an image acquisition unit 203, a region recognition unit 204, a preprocessing unit 205, a structure detection unit 206, a result integration unit 207, and an image output unit 208.

[0051] The model storage unit 201 is realized by the HDD 504 shown in Fig. 4. The detector setting unit 202, the image acquisition unit 203, the region recognition unit 204, the preprocessing unit 205, the structure detection unit 206, the result integration unit 207, and the image output unit 208 are realized by the processing that the CPU 501 executes by a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 4.

[0052] A trained region segmentation model is stored in the model storage unit 201. The region segmentation model is trained by the training device 5 and transferred to the image processing device 6.

[0053] An example of a segmentation model is a machine learning model that performs semantic segmentation based on deep learning. The network structure of the segmentation model is, for example, a convolutional neural network. Specifically, a segmentation model such as SegNet or U-Net can be used.

[0054] The region segmentation model is not limited to the above model, and any machine learning model that can recognize a predetermined region from an input image can be used. Another example of the region segmentation model is a machine learning model that performs instance segmentation or panoptic segmentation based on deep learning.

[0055] The detector setting unit 202 sets an edge detector to be used for detecting a fibrous structure. The detector setting unit 202 may set the edge detector in accordance with an instruction from an operator, etc. The detector setting unit 202 may set each parameter of the edge detector as necessary.

[0056] The image acquisition unit 203 acquires an operative field image. The operative field image acquired by the image acquisition unit 203 is an operative field image (hereinafter referred to as a "target image") for generating a navigation image. The image acquisition unit 203 acquires the target image by receiving image data of the target image from the imaging control device 3.

[0057] The region recognition unit 204 recognizes a region included in the target image by inputting the target image acquired by the image acquisition unit 203 into the trained region segmentation model read out from the model storage unit 201. The region recognition unit 204 may perform preprocessing such as normalization before recognizing a region from the target image. After recognizing a region from the target image, the region recognition unit 204 may also perform postprocessing such as determining that a region whose area is smaller than a threshold is a false detection and deleting it.

[0058] The preprocessing unit 205 performs predetermined preprocessing on the target image acquired by the image acquisition unit 203. The preprocessing includes, for example, binarization, noise removal, expansion or contraction, etc. The preprocessing is not limited to these, and any preprocessing used in image processing may be performed. The preprocessing may be selected depending on the state of the target image or the type of edge detector.

[0059] The structure detection unit 206 detects fibrous structures from the target image acquired by the image acquisition unit 203 using the edge detector set by the detector setting unit 202. In this embodiment, the edge detector includes, for example, a gradient filter, a Canny edge detector, or a Gabor filter. The differential filter includes, for example, a Sobel filter, a Laplacian filter, a Scharr filter, or a Prewitt filter. The edge detector is not limited to these, and any method capable of detecting edges from an image can be applied.

[0060] The result integration unit 207 generates an integrated recognition result by integrating the recognition result from the area recognition unit 204 and the detection result from the structure detection unit 206. Specifically, the result integration unit 207 calculates the logical product of the recognition result indicating the range of the specified area and the detection result of the fibrous structure. In other words, the result integration unit 207 masks the detection result of the fibrous structure with the recognition result indicating the range of the specified area. Therefore, the integrated recognition result becomes two-dimensional data indicating the fibrous structure included in the specified area.

[0061] The image output unit 208 generates a navigation image by displaying the integrated recognition result generated by the result integration unit 207 on the target image acquired by the image acquisition unit 203. The image output unit 208 outputs the generated navigation image to the display device 4.

[0062] <<Outline of Image Processing>> An outline of the image processing executed by the image processing device 6 in this embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the image processing in this embodiment.

[0063] 7, an operative field image 931 input to the image processing device 6 is input to a trained region segmentation model 932 and a preset edge detector 933. Note that the image processing device 6 may perform predetermined preprocessing on the operative field image 931 before inputting the operative field image 931 to the edge detector 933. The preprocessing may be, for example, two-level gradation.

[0064] The region segmentation model 932 recognizes regions included in the operative field image 931 and outputs a region recognition result 934. The edge detector 933 detects fibrous structures included in the operative field image 931 and outputs a fibrous structure detection result 935. Note that the image processing device 6 may perform predetermined post-processing on the detection result 935 output from the edge detector 933. The post-processing may be, for example, thinning.

[0065] The image processing device 6 calculates the logical product of the recognition result 934 and the detection result 935. That is, the image processing device 6 masks the detection result 935 with the recognition result 934. As a result, the image processing device 6 obtains an integrated recognition result 936 by integrating the recognition result 934 and the detection result 935.

[0066] The image processing device 6 superimposes the integrated recognition result 936 on the surgical field image 931. As a result, the image processing device 6 obtains a navigation image 937 showing the fibrous structure detected by the edge detector 933 within the area recognized by the area segmentation model 932. The image processing device 6 then outputs the navigation image 937 to the display device 4.

[0067] <Processing Procedure of Surgery Support System> The processing procedure of the surgery support method executed by the surgery support system 1 in this embodiment will be described with reference to Figures 8 to 10. The surgery support method in this embodiment includes a learning process (see Figure 8) that learns an area segmentation model, and a recognition process (see Figure 10) that outputs a navigation image based on a recognition result using the learned area segmentation model.

[0068] Learning Process FIG. 8 is a flowchart showing an example of the learning process in this embodiment.

[0069] In step S1, the labeling unit 102 of the learning device 5 reads out training images stored in the image storage unit 101. As a result, the labeling unit 102 acquires a plurality of training images. The number of training images may be any number sufficient for training a region segmentation model.

[0070] Next, the labeling unit 102 assigns a correct label to each training image. Subsequently, the labeling unit 102 generates training data including the training images to which the correct labels have been assigned. Then, the labeling unit 102 stores the training data in a storage unit.

[0071] Specifically, the label assignment unit 102 assigns correct labels as follows: First, the label assignment unit 102 selects a training image to which a correct label is to be assigned from the training images read out from the image storage unit 101. Next, the label assignment unit 102 displays the selected training image on the display device 506. Subsequently, the label assignment unit 102 accepts an input of a correct label input to the input device 505 by a user operation. By repeatedly performing the above operation for each training image, training data in which correct labels are assigned to all training images is generated.

[0072] The method of assigning correct labels is not limited to a method based on user operation. For example, correct labels may be assigned using a rule-based method or a broader machine learning method (other than deep learning). Furthermore, training data may be generated by correcting correct labels assigned using a rule-based method or the like through user operation.

[0073] The regions indicated by the correct answer labels may differ depending on the dissection location captured in the training image, or may differ depending on the type of surgery even if the dissection location is the same.

[0074] For example, assuming that the laparoscopic surgery is laparoscopic rectal cancer surgery, the training images may include a surgical field image of the anterior wall of the rectum, a surgical field image of the lateral wall of the rectum, and a surgical field image of the posterior wall of the rectum. When the training image is a surgical field image of the anterior wall of the rectum, the tissue to be removed is the resected organ. However, the resected organ is the rectum. The resected organ may include loose connective tissue that appears near these tissues. The tissue to be preserved is the Denonvilliers fascia and the preserved organ. However, the preserved organs are the hypogastric nerve, pelvic plexus, ureter, seminal vesicles, prostate, vagina, and neurovascular bundle (NVB). The preserved organ may include loose connective tissue that appears near these tissues.

[0075] When the training image is a surgical field image of the rectal side wall, the tissues to be removed are the rectal fascia proper and the resected organ. However, the resected organs are the rectum and mesorectum. The resected organs may include loose connective tissue that appears near these tissues. The tissues to be preserved are the ureterohypogastric prefascia and the preserved organ. However, the preserved organs are the hypogastric nerve, pelvic plexus, pelvic splanchnic nerves, ureter, and neurovascular bundle. The preserved organs may include loose connective tissue that appears near these tissues.

[0076] When the training image is a surgical field image of the posterior wall of the rectum, the tissues to be removed are the rectal fascia proper and the resected organ. However, the resected organs are the rectum and mesorectum. The resected organs may include loose connective tissue that appears near these tissues. The tissues to be preserved are the ureteral hypogastric prefascia and the preserved organ. However, the preserved organs are the hypogastric nerve, pelvic plexus, pelvic splanchnic nerves, ureter, and median sacral vessels. The preserved organs may include loose connective tissue that appears near these tissues.

[0077] The tissues to be removed and the tissues to be preserved are not limited to these, and can be determined according to classification rules based on basic anatomical aspects and clinical surgical anatomical aspects.

[0078] In step S3, the model learning unit 103 of the learning device 5 reads out the training data stored in the storage unit. Next, the model learning unit 103 uses the read out training data to learn a region segmentation model. The model learning unit 103 optimizes model parameters based on the error between the recognition result when a training image is input into the region segmentation model and the correct label. The optimization method may be selected according to the network structure of the region segmentation model.

[0079] The learning process of the region segmentation model (step S3 in Fig. 8) will now be described in detail with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the model learning process.

[0080] In step S3-1, the model learning unit 103 sets initial values ​​of the model parameters of the region segmentation model. The initial values ​​of the model parameters may be set to predetermined values ​​or may be set to random values.

[0081] In step S3-2, the model learning unit 103 selects a training image to be used for training from the teacher data. In step S3-3, the model learning unit 103 inputs the selected training image into a region segmentation model to recognize regions included in the training image.

[0082] In step S3-4, the model learning unit 103 calculates the error between the recognition result of the region included in the training image and the correct label assigned to the selected training image. In step S3-5, the model learning unit 103 updates the model parameters of the region segmentation model based on the error between the recognition result and the correct label, using the backpropagation method or the like.

[0083] In step S3-6, the model learning unit 103 determines whether learning is complete. Whether learning is complete may be determined based on whether the model parameters have converged, or whether a predetermined number of repetitions has been exceeded. If learning is not complete (NO), the model learning unit 103 returns the process to step S3-2. Thereafter, the model learning unit 103 repeatedly executes the processes of steps S3-2 to S3-5 for different training images. If learning is complete (YES), the model learning unit 103 ends the model learning process.

[0084] The region segmentation model may be trained using time-series data consisting of multiple training images with different time information. Such time-series data can be generated, for example, by extracting multiple frames from video data captured by an endoscope. In this case, the region segmentation model has a network structure capable of handling time-series data. Examples of networks capable of handling time-series data include a recurrent neural network (RNN) or a long short-term memory (LSTM).

[0085] Returning to Fig. 8 , in step S4, the model learning unit 103 of the learning device 5 transmits the trained region segmentation model trained in step S3 to the image processing device 6. The image processing device 6 receives the trained region segmentation model from the learning device 5 and stores it in the model storage unit 201.

[0086] <Recognition Processing> FIG. 10 is a flowchart showing an example of the recognition processing in this embodiment.

[0087] In step S11, the detector setting unit 202 of the image processing device 6 accepts an edge detector setting operation by the user. Next, the detector setting unit 202 selects the edge detector specified in the setting operation. Subsequently, the detector setting unit 202 generates setting information indicating the selected edge detector. If the user inputs parameters to be set in the edge detector, the detector setting unit 202 includes the input parameters in the setting information. Then, the detector setting unit 202 stores the edge detector setting information in a storage unit.

[0088] Note that step S11 may be executed at least once at any timing. For example, if an edge detector that has already been set is to be continuously used, step S11 may be skipped and the recognition process may start from step S12. Furthermore, if setting information indicating the initial setting of the edge detector is stored in advance in the storage unit, step S11 may not be executed.

[0089] In step S12, the image acquisition unit 203 of the image processing device 6 receives image data of the target image from the imaging control device 3. As a result, the image acquisition unit 203 acquires the target image. The image acquisition unit 203 sends the acquired target image to the area recognition unit 204, the preprocessing unit 205, and the image output unit 208.

[0090] In step S13, the area recognition unit 204 of the image processing device 6 receives the target image from the image acquisition unit 203. Next, the area recognition unit 204 reads out the area division model stored in the model storage unit 201. Subsequently, the area recognition unit 204 inputs the received target image into the read area division model. As a result, the area recognition unit 204 obtains a recognition result output from the area division model. The recognition result includes information indicating which area each pixel of the target image is included in. Then, the area recognition unit 204 sends the area recognition result to the result integration unit 207.

[0091] In step S14, the preprocessing unit 205 of the image processing device 6 receives the target image from the image acquisition unit 203. Next, the preprocessing unit 205 performs predetermined preprocessing on the target image. In this example, the preprocessing performed by the preprocessing unit 205 is assumed to be binarization. The preprocessing unit 205 then sends the preprocessed target image to the structure detection unit 206.

[0092] In step S15, the structure detection unit 206 of the image processing device 6 receives the preprocessed target image from the preprocessing unit 205. Next, the structure detection unit 206 reads setting information for the edge detector from the storage unit. Subsequently, the structure detection unit 206 detects fibrous structures from the preprocessed target image using the edge detector indicated in the setting information. Then, the structure detection unit 206 sends the detection result of the fibrous structures to the result integration unit 207.

[0093] In step S16, the result integration unit 207 of the image processing device 6 receives the recognition result from the region recognition unit 204. The result integration unit 207 also receives the detection result from the structure detection unit 206. Next, the result integration unit 207 integrates the recognition result and the detection result. Specifically, the result integration unit 207 calculates the logical product of the recognition result indicating the range of a predetermined region and the detection result of the fibrous structure. Here, the predetermined region is the region recommended for separation. Therefore, the integrated recognition result is two-dimensional data indicating the fibrous structure (e.g., loose connective tissue) included in the region recommended for separation. The result integration unit 207 then sends the integrated recognition result to the image output unit 208.

[0094] In step S17, the image output unit 208 of the image processing device 6 receives the target image from the image acquisition unit 203. The image output unit 208 also receives the integrated recognition result from the result integration unit 207. Next, the image output unit 208 displays the integrated recognition result on the received target image. This generates a navigation image. The integrated recognition result indicates the position, range, shape, etc. of the fibrous structure (here, loose connective tissue) contained in a predetermined region (here, the recommended ablation region). Therefore, the navigation image indicates the fibrous structure contained in the predetermined region.

[0095] The image output unit 208 may display the integrated recognition result in the target image in a predetermined color. The predetermined color may be a transparent color. The predetermined color may be set in advance or may be set by a user instruction. The predetermined color may be changeable at any time by a user instruction. The predetermined color may have, for example, an RGB value of (0, 0, 255) and an opacity of 0.25.

[0096] The image output unit 208 outputs the generated navigation image to the display device 4. The display device 4 displays the navigation image on a display. The display device 4 may also display the target image input from the imaging control device 3 so that the target image can be contrasted with the navigation image.

[0097] The surgeon views the navigation image displayed on the display device 4 and determines the dissection site while taking into consideration the position, area, and shape of the fibrous structure present in the current surgical field. The surgeon then operates the endoscope 2 or surgical instruments to perform the dissection operation at the determined dissection site.

[0098] <Characteristics of each edge detector> In this embodiment, various edge detectors can be used, but the processing procedure and detection results differ depending on the type of edge detector. Since the detection results affect the visibility of the navigation image, in this embodiment, the edge detector can be set by user instructions. The characteristics of each edge detector will be described below.

[0099] Differential Filter: A differential filter is an image processing filter that detects contours based on a brightness gradient. In this embodiment, in order to detect fibrous structures contained in an operative field image, contours in the x direction and contours in the y direction are detected in the operative field image converted to grayscale, and the detection results are converted to two-tone and then logically summed.

[0100] Fig. 11 is a diagram illustrating an example of a differential filter. As shown in Fig. 11, in this embodiment, a Sobel filter, a Laplacian filter, a Scharr filter, or a Prewitt filter can be selected as the differential filter.

[0101] Since the Sobel filter, Scharr filter, and Prewitt filter are based on first-order derivatives, it can be seen that the detection results tend to be similar. Among these, the Scharr filter shows good detection results. On the other hand, the Laplacian filter is based on second-order derivatives, so it cannot well represent the gradient of the contour.

[0102] Note that the detection results using a differential filter may have thick contours. In this case, the visibility can be improved by thinning the contours included in the detection results. For thinning, the hit-miss transform, which is a morphological technique, can be used.

[0103] 12 is a diagram showing an example of thinning processing. As shown in Fig. 12, when hit-miss conversion is performed on a detection result 951 obtained by a differential filter, a detection result 952 with a thinned outline is obtained. Because the detection result 952 has a thin outline, the visibility of the background tissue is improved when it is superimposed on an operative field image.

[0104] <Canny Edge Detector> The Canny edge detector uses two thresholds as parameters to detect strong and weak boundary pixels. Next, it traces the neighboring pixels from the strong boundary pixel to form a boundary line. At this time, among the weak boundary pixels, those neighboring the strong boundary pixel are treated as a correct edge. The first threshold is a parameter that controls the level of image contrast that is detected as an edge. The second threshold is a parameter that controls the level of detail that is retained.

[0105] 13 is a diagram showing an example of parameters of the Canny edge detector. A detection result 961 is a detection result by the Canny edge detector when the first threshold (threshold1) is set to 20 and the second threshold (threshold2) is set to 20. A detection result 962 is a detection result when the first threshold is set to 20 and the second threshold is set to 100. A detection result 963 is a detection result when the first threshold is set to 40 and the second threshold is set to 40. A detection result 964 is a detection result when the first threshold is set to 40 and the second threshold is set to 100.

[0106] As shown in Fig. 13, the detection result changes depending on the combination of the first threshold and the second threshold. In this embodiment, the parameters of the Canny edge detector can be set by user instructions.

[0107] Gabor Filter: A Gabor filter is an image processing filter that detects highlights in an image. In this embodiment, to detect fibrous structures contained in an operative field image, the operative field image is converted to grayscale, and the x- and y-axis contours are detected. These detection results are then converted to two-tone and then logically summed.

[0108] When using a Gabor filter, the results of detection by the Gabor filter from multiple surgical field images rotated at different angles may be ANDed. For example, contours may be detected from surgical field images rotated at 0°, 45°, 90°, and 135°, and the ANDing of the results may be performed. Furthermore, as with the differential filter, the detection results may be subjected to hit-miss transformation to thin the contours included in the detection results.

[0109] <<Comparison of Detection Results>> Fig. 14 is a diagram showing an example of a navigation image for each edge detector. Image 971 (Original Image) is an original image captured by the imaging device of the endoscope 2. Image 972 is a navigation image obtained when a contour is detected for image 971 using a gradient filter. Image 973 is a navigation image obtained when a contour is detected for image 971 using a Canny edge detector. Image 974 is a navigation image obtained when a contour is detected for image 971 using a Gabor filter.

[0110] As shown in Figure 14, the visibility of fibrous structures in the navigation image is improved regardless of which edge detector is used. Note that navigation images using the Canny edge detector may show areas where the contours are connected or where the shadows are easily distorted, creating an optical illusion. Furthermore, navigation images using the Gabor filter may show areas where the contours overlap with the fibers captured in the surgical field image.

[0111] According to the results shown in Fig. 14, it is considered that the Canny edge detector is suitable for enhancing the contours of fibers, and the Gabor filter is suitable for enhancing the extent of fibers.

[0112] Comparison of Detection Speeds Fig. 15 is a diagram showing an example of the detection speed for each edge detector. As shown in Fig. 15, the differential filter (Gradient) completed edge detection in 0.003 seconds per run. The Canny edge detector (Canny) completed edge detection in 0.006 seconds per run. The Gabor filter (Gabor) completed edge detection in 0.015 seconds per run.

[0113] Considering the time required for recognition processing using a region segmentation model and other image processing, it is preferable to be able to process approximately 30 frames per second (i.e., within 0.033 seconds per frame). Differential filters and Canny edge detectors have been shown to be capable of edge detection within a sufficiently practical processing time. Gabor filters tend to have long processing times because they perform filtering twice, in the x and y directions. However, because differential filters and Gabor filters can be optimized through hardware implementation, it is possible to achieve practical processing times even with Gabor filters.

[0114] Effect of First Embodiment The image processing device 6 in this embodiment recognizes regions included in a surgical field image based on a trained region segmentation model, detects fibrous structures included in the surgical field image, and outputs a navigation image showing the fibrous structures included in the recognized regions on the surgical field image. The navigation image displays the fibrous structures included in a specified region superimposed on the surgical field image, allowing the surgeon to easily grasp the position, range, shape, etc. of the fibrous tissue present in the surgical field. Therefore, this embodiment makes it possible to output a surgical field image with high visibility.

[0115] The image processing device 6 in this embodiment may detect fibrous structures using an edge detector. The edge detector may include a differential filter, a Canny edge detector, or a Gabor filter. These edge detectors do not require training, so the user only needs to prepare training data for training the region segmentation model. The training data for training the region segmentation model only needs to be assigned a correct answer label indicating the range of the region, which reduces the cost of preparing training data. Therefore, this embodiment makes it possible to efficiently output surgical field images with high visibility.

[0116] The image processing device 6 in this embodiment may detect fibrous structures using an edge detector specified by the user. Because the detection results for fibrous structures vary depending on the type of edge detector, it is convenient for the surgeon to select a suitable edge detector depending on the dissection location, surgical environment, or personal preference. Therefore, this embodiment can output a surgical field image showing a fibrous structure suitable for the user.

[0117] The image processing device 6 in this embodiment may display thinned fibrous structures in the surgical field image. When the contour extracted by the edge detector is thick, thinning the contour may further improve visibility.

[0118] The image processing device 6 in this embodiment may output a navigation image showing the loose connective tissue included in the recommended dissection region. Therefore, according to this embodiment, the user can accurately grasp the position, range, shape, etc. of the loose connective tissue within the recommended dissection region, and can proceed with the surgery efficiently.

[0119] The image processing device 6 in this embodiment may output a navigation image showing the nerves or blood vessels included in the region to be preserved. Therefore, according to this embodiment, the user can accurately grasp the position, range, shape, etc. of the nerves or blood vessels in the region to be preserved, and can proceed with the surgery safely.

[0120] [Second Embodiment] The image processing method of the present disclosure can be used to create training data used for training a region segmentation model. In the second embodiment of the present invention, the region segmentation model is trained using training data to which fibrous structures detected by an edge detector are assigned as correct labels. Therefore, the region segmentation model in this embodiment can recognize fibrous structures from surgical field images.

[0121] The surgery assistance system of this embodiment will be described below, focusing on the differences from the first embodiment.

[0122] <Functional Configuration of Surgery Support System> The functional configuration of the surgery support system in this embodiment will be described with reference to FIGS. 16 and 17. FIG.

[0123] 16 is a block diagram showing an example of the functional configuration of the learning device 7 in this embodiment. As shown in Fig. 16, the learning device 7 in this embodiment includes an image storage unit 101, a label assignment unit 102, a model learning unit 103, a detector setting unit 202, a preprocessing unit 205, a structure detection unit 206, and a label receiving unit 301.

[0124] That is, the learning device 7 includes the image storage unit 101, label assignment unit 102, and model learning unit 103 that are included in the learning device 5 in the first embodiment. The learning device 7 also includes the detector setting unit 202, preprocessing unit 205, and structure detection unit 206 that are included in the image processing device 6 in the first embodiment. The learning device 7 further includes a label acceptance unit 301.

[0125] The label receiving unit 301 receives input of a label (hereinafter also referred to as an "area label") indicating the range of a predetermined area included in the learning image read from the image storage unit 101. The predetermined area may be, for example, an excision area, a sparing area, a recommended excision area, or other area. Here, the predetermined area is the recommended excision area.

[0126] Specifically, the label accepting unit 301 accepts input of region labels as follows: First, the label accepting unit 301 selects a training image for which a region label is to be accepted from the training images read out from the image storage unit 101. Next, the label accepting unit 301 displays the selected training image on the display device 506.

[0127] Next, the label accepting unit 301 accepts input of a region label input to the input device 505 by user operation. The region label is information indicating the range of a predetermined region. Specifically, the region label is two-dimensional data that assigns a label to each pixel of the surgical field image indicating whether or not it is included in the predetermined region. By repeatedly performing the above operation for each training image, the label accepting unit 301 accepts input of region labels corresponding to all training images.

[0128] The labeling unit 102 in this embodiment assigns a correct label to a training image read out from the image storage unit 101. Specifically, the labeling unit 102 assigns a correct label as follows: The labeling unit 102 first calculates the logical product of the region label input to the label receiving unit 301 and the detection result of the fibrous structure by the structure detection unit 206. In other words, the labeling unit 102 masks the detection result of the fibrous structure with a region label that indicates the range of a predetermined region.

[0129] Next, the labeling unit 102 assigns the masked fibrous structure detection results as correct labels to the training images. Therefore, in this embodiment, the correct labels are two-dimensional data indicating the fibrous structures contained in a predetermined region. The labeling unit 102 then stores training data including multiple training images to which the correct labels have been assigned in a storage unit.

[0130] <Image Processing Device> Fig. 17 is a block diagram showing an example of the functional configuration of an image processing device 8 in this embodiment. As shown in Fig. 17, the image processing device 6 in this embodiment includes a model storage unit 201, an image acquisition unit 203, a region recognition unit 204, and an image output unit 208. That is, the image processing device 8 in this embodiment differs from the image processing device 6 in the first embodiment in that it does not include the detector setting unit 202, the preprocessing unit 205, the structure detection unit 206, and the result integration unit 207.

[0131] The region recognition unit 204 in this embodiment recognizes fibrous structures included in a predetermined region of the target image by inputting the target image acquired by the image acquisition unit 203 into a trained region segmentation model read from the model storage unit 201. The region segmentation model in this embodiment is trained using training data including training images to which the fibrous structures included in the predetermined region are assigned as correct labels. Therefore, the region recognition unit 204 in this embodiment can accurately recognize the fibrous structure (here, loose connective tissue) included in the predetermined region (here, the recommended region for exfoliation).

[0132] The image output unit 208 in this embodiment generates a navigation image by displaying the recognition result obtained by the area recognition unit 204 on the target image acquired by the image acquisition unit 203. The image output unit 208 outputs the generated navigation image to the display device 4.

[0133] Effect of Second Embodiment The learning device 7 in this embodiment detects fibrous structures contained in a surgical field image and assigns the fibrous structures contained in a predetermined region to the surgical field image. The surgical field image to which the fibrous structures have been assigned can be used to train a region segmentation model for recognizing the fibrous structures from the surgical field image. Therefore, according to this embodiment, training data for training a region segmentation model for recognizing the fibrous structures can be efficiently created.

[0134] [Variation 1] In the above embodiment, a configuration has been described in which a common region segmentation model is used to recognize regions included in a surgical field image regardless of the dissection location. Here, a region segmentation model according to the dissection location may be used to recognize regions included in a surgical field image. In this case, training data may be generated for each dissection location in the learning process, and a region segmentation model for each dissection location may be learned using the training data. In this case, in the recognition process, the region segmentation model to be used may be switched depending on the dissection location by an operation by the surgeon or based on the recognition results of the surgical field image.

[0135] The region segmentation model to be used may be selected according to the progress of the surgery, and the region included in the target image may be recognized by inputting the target image into the selected region segmentation model. Alternatively, the region segmentation model to be used may be selected based on the recognition result by inputting the target image into a trained classification model that has been trained to classify surgical field images by dissection location.

[0136] This configuration allows for learning of a region segmentation model specialized for the dissection site. Therefore, according to this modification, it is possible to output an operative field image showing a more appropriate fibrous structure.

[0137] Here, an example has been described in which multiple region division models are generated for each ablation location, but multiple region division models may be generated from different perspectives. For example, different region division models may be generated depending on the surgical procedure. Also, for example, different region division models may be generated depending on the model of endoscope or the specifications of the imaging device provided in the endoscope.

[0138] [Variation 2] In the above embodiments, the image processing device recognizes each region from the surgical field image using a machine learning model based on deep learning. However, the method for recognizing each region from the surgical field image is not limited to the machine learning model based on deep learning. For example, the image processing device 6 may recognize each region from the surgical field image using a method such as template matching or a support vector machine (SVM).

[0139] [Supplementary Note] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0140] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

Claims

1. An image processing device comprising: an image acquisition unit configured to acquire a surgical field image of a surgical field in laparoscopic surgery; a region recognition unit configured to recognize a region included in the surgical field image acquired by the image acquisition unit by inputting the surgical field image acquired by the image acquisition unit into a trained model that has learned from training data in which a training image of the surgical field is assigned a region indicating a tissue classification; a structure detection unit configured to detect a fibrous structure included in the surgical field image acquired by the image acquisition unit; and an image output unit configured to output the surgical field image indicating the fibrous structure included in the region recognized by the region recognition unit.

2. An image processing device according to claim 1, wherein the structure detection unit is configured to detect the fibrous structure using an edge detector.

3. An image processing device according to claim 2, wherein the edge detector includes a differential filter, a Canny edge detector or a Gabor filter.

4. An image processing device according to claim 2, wherein the structure detection unit is configured to detect the fibrous structure using the edge detector designated by a user.

5. An image processing device according to claim 1, wherein the image output unit is configured to thin the fibrous structures detected by the structure detection unit and display them in the surgical field image.

6. An image processing device according to any one of claims 1 to 5, wherein the region is a region indicating an area of ​​tissue to be dissected, and the fibrous structure is loose connective tissue.

7. An image processing device according to any one of claims 1 to 5, wherein the region is a region indicating an area of ​​tissue to be preserved, and the fibrous structure indicates a nerve or a blood vessel.

8. An image processing device comprising: an image acquisition unit configured to acquire a surgical field image captured of a surgical field during laparoscopic surgery; a label receiving unit configured to accept input of a label indicating the range of a predetermined area included in the surgical field image; a structure detection unit configured to detect a fibrous structure included in the surgical field image acquired by the image acquisition unit; and a label assignment unit configured to assign the fibrous structure included in the area indicated in the label to the surgical field image.

9. A surgical support system including an endoscope used in endoscopic surgery, a display device installed in an operating room where the endoscopic surgery is performed, and an image processing device, wherein the image processing device comprises: an image acquisition unit configured to acquire a surgical field image of the surgical field of the endoscopic surgery from the endoscope; a region recognition unit configured to recognize the region contained in the surgical field image acquired by the image acquisition unit by inputting the surgical field image acquired by the image acquisition unit into a trained model that has learned from training data in which a region indicating a tissue classification is added to a learning image of the surgical field; a structure detection unit configured to detect a fibrous structure included in the surgical field image acquired by the image acquisition unit; and an image output unit configured to output the surgical field image showing the fibrous structure included in the region recognized by the region recognition unit to the display device.

10. An image processing method in which a computer executes the following steps: an image acquisition step for acquiring a surgical field image of a surgical field in laparoscopic surgery; a region recognition step for recognizing a region contained in the surgical field image acquired by the image acquisition step by inputting the surgical field image into a trained model that has learned from training data in which a training image of the surgical field is assigned a region indicating a tissue classification; a structure detection step for detecting a fibrous structure contained in the surgical field image acquired by the image acquisition step; and an image output step for outputting the surgical field image showing the fibrous structure contained in the region recognized by the region recognition step.

11. An image processing method in which a computer executes the following steps: an image acquisition step for acquiring a surgical field image captured of a surgical field during laparoscopic surgery; a label acceptance step for accepting input of a label indicating the range of an area corresponding to a tissue classification contained in the surgical field image; a structure detection step for detecting a fibrous structure contained in the surgical field image acquired by the image acquisition step; and a label assignment step for assigning the fibrous structure contained in the area indicated by the label to the surgical field image.

12. A program for causing a computer to execute the following steps: an image acquisition step for acquiring a surgical field image of a surgical field in laparoscopic surgery; a region recognition step for recognizing a region contained in the surgical field image acquired by the image acquisition step by inputting the surgical field image into a trained model that has learned from training data in which a training image of the surgical field is assigned a region indicating a tissue classification; a structure detection step for detecting a fibrous structure contained in the surgical field image acquired by the image acquisition step; and an image output step for outputting the surgical field image showing the fibrous structure contained in the region recognized by the region recognition step.

13. A program for causing a computer to execute the following steps: an image acquisition step for acquiring a surgical field image captured of a surgical field during laparoscopic surgery; a label acceptance step for accepting input of a label indicating the range of an area corresponding to a tissue classification contained in the surgical field image; a structure detection step for detecting a fibrous structure contained in the surgical field image acquired by the image acquisition step; and a label assignment step for assigning the fibrous structure contained in the area indicated by the label to the surgical field image.

Images