Computer program, method of generating learning model, and assistance device

Abstract

To provide a computer program, a method of generating a learning model, and an assistance device.SOLUTION: A computer is caused to execute a process of acquiring a surgical field image obtained by imaging a surgical field of an arthroscopic surgery, and by using a learning model that has been learned to output information on an object tissue upon input of a surgical field image, recognizing a tissue part of interest included in the acquired surgical field image distinctively from a blood vessel tissue part appearing on the surface of the object tissue part.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present invention relates to a computer program, a method for generating a learning model, and a support device. [Background technology]

[0002] In laparoscopic surgery, for example, surgery is performed to remove a lesion such as a malignant tumor formed inside the patient's body. During this operation, images of the inside of the patient's body are captured using a laparoscope, and the obtained images of the surgical field are displayed on a monitor (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-287839 A Summary of the Invention [Problem to be solved by the invention]

[0004] Conventionally, it has been difficult to recognize tissues such as nerves and ureters that require the surgeon's attention from an image of the surgical field and to notify the surgeon of these tissues.

[0005] The present invention aims to provide a computer program, a method for generating a learning model, and an assistance device that can output recognition results of tissues such as nerves and ureters from surgical field images. [Means for solving the problem]

[0006] A computer program in one aspect of the present invention is a computer program for causing a computer to acquire a surgical field image obtained by imaging the surgical field of laparoscopic surgery, and execute a process of recognizing a target tissue portion included in the acquired surgical field image and distinguishing it from a vascular tissue portion appearing on the surface of the target tissue portion using a learning model that has been trained to output information about a target tissue when the surgical field image is input.

[0007] In one aspect of the present invention, a method for generating a learning model involves a computer acquiring training data including a surgical field image obtained by imaging the surgical field of laparoscopic surgery and ground truth data in which a target tissue portion included in the surgical field image is labeled to distinguish it from a vascular tissue portion appearing on the surface of the target tissue portion, and generating a learning model that outputs information about the target tissue when a surgical field image is input based on the acquired set of training data.

[0008] An assistance device in one aspect of the present invention includes an acquisition unit that acquires a surgical field image obtained by imaging the surgical field of endoscopic surgery, a recognition unit that recognizes a target tissue portion included in the acquired surgical field image by distinguishing it from a vascular tissue portion appearing on the surface of the target tissue portion using a learning model that has been trained to output information regarding a target tissue when a surgical field image is input, and an output unit that outputs assistance information regarding the endoscopic surgery based on the recognition result of the recognition unit. Effect of the Invention

[0009] According to the present application, it is possible to output the recognition results of tissues such as nerves and ureters from the surgical field image. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating a schematic configuration of a laparoscopic surgery support system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an internal configuration of the support device. [Diagram 3] FIG. 2 is a schematic diagram showing an example of an operative field image. [Figure 4] FIG. 2 is a schematic diagram showing an example of the configuration of a learning model. [Diagram 5] FIG. 13 is a schematic diagram showing a recognition result based on a learning model. [Figure 6] FIG. 4 is a partially enlarged view showing a recognition result according to the first embodiment. [Figure 7] FIG. 13 is a partially enlarged view showing a recognition result of a comparative example. [Figure 8] 11 is a flowchart illustrating a procedure for generating a learning model. [Figure 9]13 is a flowchart illustrating a procedure for executing surgery assistance. [Figure 10] FIG. 2 is a schematic diagram showing a display example on a display device. [Figure 11] FIG. 11 is a schematic diagram showing an example of an operative field image in the second embodiment. [Figure 12] FIG. 11 is an explanatory diagram illustrating the configuration of a learning model in the second embodiment. [Figure 13] FIG. 11 is a schematic diagram showing an example of display of a recognition result in the second embodiment. [Figure 14] FIG. 13 is a schematic diagram showing an example of an operative field image in the third embodiment. [Figure 15] FIG. 11 is an explanatory diagram illustrating the configuration of a learning model in embodiment 3. [Figure 16] FIG. 13 is a schematic diagram showing an example of display of a recognition result in the third embodiment. [Figure 17] FIG. 13 is a schematic diagram showing a display example in the fourth embodiment. [Figure 18] FIG. 13 is an explanatory diagram illustrating a display method in the fifth embodiment. [Figure 19] 23 is a flowchart illustrating a procedure of a process executed by the support device according to the sixth embodiment. [Figure 20] FIG. 23 is a schematic diagram showing a display example in the sixth embodiment. [Figure 21] FIG. 23 is an explanatory diagram illustrating the configuration of a learning model in embodiment 7. [Figure 22] FIG. 23 is a schematic diagram showing an example of display of a recognition result in the seventh embodiment. [Figure 23] FIG. 23 is an explanatory diagram illustrating the configuration of a learning model in the eighth embodiment. [Figure 24] FIG. 11 is an explanatory diagram for explaining a method for identifying organ boundaries. [Diagram 25] 23 is a flowchart illustrating a procedure of a process executed by a support device according to the eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention is specifically described below with reference to the drawings with respect to an embodiment in which the present invention is applied to a support system for laparoscopic surgery. Note that the present invention is not limited to laparoscopic surgery, but can be applied to all types of endoscopic surgery using imaging devices, such as thoracoscopes, gastrointestinal endoscopes, cystoscopes, arthroscopes, robot-assisted endoscopes, spinal endoscopes, surgical microscopes, neuroendoscopes, and exoscopes. (Embodiment 1) FIG. 1 is a schematic diagram for explaining the outline configuration of a laparoscopic surgery support system according to the first embodiment. In laparoscopic surgery, instead of performing an open abdominal surgery, a plurality of perforation instruments called trocars 10 are attached to the abdominal wall of a patient, and instruments such as a laparoscope 11, an energy treatment instrument 12, and forceps 13 are inserted into the patient's body through perforations provided in the trocars 10. The surgeon performs treatment such as resecting an affected area using the energy treatment instrument 12 while viewing an image (operative field image) of the inside of the patient's body captured by the laparoscope 11 in real time. The surgical instruments such as the laparoscope 11, the energy treatment instrument 12, and the forceps 13 are held by the surgeon or a robot. The surgeon is a medical professional involved in laparoscopic surgery, and includes the surgeon, assistant, nurse, doctor monitoring the surgery, and the like.

[0012] The laparoscope 11 includes an insertion section 11A that is inserted into the patient's body, an imaging device 11B built into the tip portion of the insertion section 11A, an operating section 11C that is provided at the rear end portion of the insertion section 11A, and a universal cord 11D for connecting to a camera control unit (CCU) 110 and a light source device 120.

[0013] The insertion section 11A of the laparoscope 11 is formed of a rigid tube. A bending section is provided at the tip of the rigid tube. The bending mechanism in the bending section is a well-known mechanism incorporated in general laparoscopes, and is configured to bend in four directions, for example, up, down, left, and right, by pulling an operating wire linked to the operation of the operating section 11C. Note that the laparoscope 11 is not limited to a flexible scope having a bending section as described above, and may be a rigid scope having no bending section.

[0014] The imaging device 11B includes a solid-state imaging element such as a complementary metal oxide semiconductor (CMOS), and a driver circuit including a timing generator (TG) and an analog signal processing circuit (AFE). The driver circuit of the imaging device 11B captures RGB color signals output from the solid-state imaging element in synchronization with a clock signal output from the TG, and performs necessary processing such as noise removal, amplification, and AD conversion in the AFE to generate image data in a digital format. The driver circuit of the imaging device 11B transmits the generated image data to the CCU 110 via the universal code 11D.

[0015] The operation unit 11C includes an angle lever, a remote switch, and the like that are operated by the surgeon. The angle lever is an operating tool that receives an operation for bending the bending portion. A bending operation knob, a joystick, and the like may be provided instead of the angle lever. The remote switch includes, for example, a changeover switch that switches the observed image between a moving image display and a still image display, and a zoom switch that enlarges or reduces the observed image. A specific function that is determined in advance may be assigned to the remote switch, or a function that is set by the surgeon may be assigned to the remote switch.

[0016] Furthermore, the operation unit 11C may have a built-in vibrator configured by a linear resonant actuator, a piezoelectric actuator, etc. When an event occurs that should be notified to the surgeon operating the laparoscope 11, the CCU 110 may vibrate the operation unit 11C by activating the vibrator built in the operation unit 11C, thereby notifying the surgeon of the occurrence of the event.

[0017] Inside the insertion section 11A, the operation section 11C, and the universal cord 11D of the laparoscope 11, there are arranged a transmission cable for transmitting a control signal output from the CCU 110 to the imaging device 11B and image data output from the imaging device 11B, a light guide for guiding illumination light emitted from the light source device 120 to the tip portion of the insertion section 11A, and the like. The illumination light emitted from the light source device 120 is guided to the tip portion of the insertion section 11A through the light guide, and is irradiated to the surgical field through an illumination lens provided at the tip portion of the insertion section 11A. In this embodiment, the light source device 120 is described as an independent device, but the light source device 120 may be configured to be built into the CCU 110.

[0018] The CCU 110 includes a control circuit for controlling the operation of the imaging device 11B included in the laparoscope 11, an image processing circuit for processing image data from the imaging device 11B input through the universal code 11D, and the like. The control circuit includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and outputs control signals to the imaging device 11B in response to the operation of various switches included in the CCU 110 and the operation of the operation unit 11C included in the laparoscope 11, and controls the start and stop of imaging, zooming, and the like. The image processing circuit includes a DSP (Digital Signal Processor), an image memory, and the like, and performs appropriate processing such as color separation, color interpolation, gain correction, white balance adjustment, and gamma correction on the image data input through the universal code 11D. The CCU 110 generates frame images for a moving image from the processed image data, and sequentially outputs each generated frame image to the support device 200 described later. The frame rate of the frame images is, for example, 30 FPS (Frames Per Second).

[0019] The CCU 110 may generate video data conforming to a predetermined standard such as NTSC (National Television System Committee), PAL (Phase Alternating Line), or DICOM (Digital Imaging and COmmunication in Medicine). The CCU 110 can output the generated video data to the display device 130 to display an operative field image (video) on the display screen of the display device 130 in real time. The display device 130 is a monitor equipped with a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like. The CCU 110 may also output the generated video data to the recording device 140 and cause the recording device 140 to record the video data. The recording device 140 includes a recording device such as an HDD (Hard Disk Drive) that records the video data output from the CCU 110 together with an identifier for identifying each surgery, the date and time of surgery, the location of surgery, the patient's name, the surgeon's name, and the like.

[0020] The support device 200 generates support information for laparoscopic surgery based on image data input from the CCU 110 (i.e., image data of an operative field image obtained by imaging the operative field). Specifically, the support device 200 performs processing to distinguish and recognize the target tissue to be recognized from vascular tissue (surface blood vessels) appearing on the surface of the target tissue, and to display information about the recognized target tissue on the display device 130. In the first to sixth embodiments, a configuration for recognizing nerve tissue as the target tissue will be described, and in the seventh embodiment described later, a configuration for recognizing ureteral tissue as the target tissue will be described. The target tissue is not limited to nerve tissue or ureteral tissue, but may be any organ having surface blood vessels, such as an artery, a spermatic duct, a bile duct, a bone, or a muscle.

[0021] In this embodiment, a configuration in which the support device 200 executes the neural tissue recognition process will be described. However, a configuration in which the CCU 110 is provided with a function equivalent to that of the support device 200 and the CCU 110 executes the neural tissue recognition process may also be used.

[0022] The internal configuration of the support device 200, and the recognition process and display process executed by the support device 200 will be described below.

[0023] 2 is a block diagram for explaining the internal configuration of the support device 200. The support device 200 is a dedicated or general-purpose computer including a control unit 201, a storage unit 202, an operation unit 203, an input unit 204, an output unit 205, a communication unit 206, and the like. The support device 200 may be a computer installed in an operating room, or may be a computer installed outside the operating room. In addition, the support device 200 may be a server installed in a hospital where laparoscopic surgery is performed, or may be a server installed outside the hospital.

[0024] The control unit 201 includes, for example, a CPU, a ROM, and a RAM. The ROM included in the control unit 201 stores a control program that controls the operation of each hardware unit included in the support device 200. The CPU in the control unit 201 executes the control program stored in the ROM and various computer programs stored in a storage unit 202 (described later) and controls the operation of each hardware unit, thereby causing the entire device to function as the support device of the present application. The RAM included in the control unit 201 temporarily stores data used during the execution of calculations.

[0025] In this embodiment, the control unit 201 is configured to include a CPU, a ROM, and a RAM, but the configuration of the control unit 201 is arbitrary, and may be, for example, an arithmetic circuit or a control circuit including one or more GPUs (Graphics Processing Units), quantum processors, volatile or non-volatile memories, etc. Furthermore, the control unit 201 may have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until when a measurement end instruction is given, and a counter that counts numbers.

[0026] The storage unit 202 includes a storage device using a hard disk, a flash memory, etc. The storage unit 202 stores computer programs executed by the control unit 201, various data acquired from the outside, various data generated within the device, and the like.

[0027] The computer programs stored in the storage unit 202 include a recognition processing program PG1 that causes the control unit 201 to execute a process for distinguishing a target tissue portion included in an operative field image from a vascular tissue portion and recognize the same, a display processing program PG2 that causes the control unit 201 to execute a process for displaying support information based on the recognition result on the display device 130, and a learning processing program PG3 for generating a learning model 310. Note that the recognition processing program PG1 and the display processing program PG2 do not need to be independent computer programs, and may be implemented as one computer program. These programs are provided, for example, by a non-transitory recording medium M on which the computer programs are readably recorded. The recording medium M is a portable memory such as a CD-ROM, a USB memory, or an SD (Secure Digital) card. The control unit 201 reads a desired computer program from the recording medium M using a reading device not shown in the figure, and stores the read computer program in the storage unit 202. Alternatively, the computer programs may be provided by communication using the communication unit 206.

[0028] The storage unit 202 also stores a learning model 310 used in the above-mentioned recognition processing program PG1. The learning model 310 is a learning model that has been trained to output a recognition result regarding a target tissue in response to an input of an operative field image. The learning model 310 is described by its definition information. The definition information of the learning model 310 includes information on layers included in the learning model 310, information on nodes constituting each layer, and parameters such as weighting and bias between nodes. These parameters are learned using a predetermined learning algorithm with the operative field image obtained by imaging the operative field and the correct answer data indicating the target tissue portion in the operative field image as training data. The configuration and generation procedure of the learning model 310 will be described in detail later.

[0029] The operation unit 203 includes operation devices such as a keyboard, a mouse, a touch panel, a non-contact panel, a stylus pen, and a voice input device using a microphone. The operation unit 203 receives an operation by an operator or the like, and outputs information related to the received operation to the control unit 201. The control unit 201 executes appropriate processing according to the operation information input from the operation unit 203. Note that, although the support device 200 is configured to include the operation unit 203 in this embodiment, the operation may be received via various devices such as the CCU 110 connected externally.

[0030] The input unit 204 includes a connection interface for connecting an input device. In the present embodiment, the input device connected to the input unit 204 is the CCU 110. The input unit 204 receives image data of an operative field image captured by the laparoscope 11 and processed by the CCU 110. The input unit 204 outputs the input image data to the control unit 201. The control unit 201 may store the image data acquired from the input unit 204 in the storage unit 202.

[0031] The output unit 205 includes a connection interface for connecting an output device. In this embodiment, the output device connected to the output unit 205 is the display device 130. When the control unit 201 generates information to be notified to an operator, such as a recognition result by the learning model 310, the control unit 201 outputs the generated information from the output unit 205 to the display device 130, thereby causing the display device 130 to display the information. In this embodiment, the display device 130 is connected to the output unit 205 as an output device, but an output device such as a speaker that outputs sound may be connected to the output unit 205.

[0032] The communication unit 206 includes a communication interface for transmitting and receiving various types of data. The communication interface included in the communication unit 206 is a communication interface conforming to a wired or wireless communication standard used in Ethernet (registered trademark) or WiFi (registered trademark). When data to be transmitted is input from the control unit 201, the communication unit 206 transmits the data to be transmitted to a specified destination. When data transmitted from an external device is received, the communication unit 206 outputs the received data to the control unit 201.

[0033] The support device 200 does not have to be a single computer, but may be a computer system including a plurality of computers and peripheral devices. Furthermore, the support device 200 may be a virtual machine that is virtually constructed by software.

[0034] Next, the operative field image input to the support device 200 will be described. 3 is a schematic diagram showing an example of an operative field image. The operative field image in this embodiment is an image obtained by imaging the inside of a patient's abdominal cavity with the laparoscope 11. The operative field image does not need to be a raw image output by the imaging device 11B of the laparoscope 11, but may be an image (frame image) that has been processed by the CCU 110 or the like.

[0035] The surgical field imaged by the laparoscope 11 includes tissues that constitute organs, blood vessels, nerves, etc., connective tissues present between tissues, tissues containing lesions such as tumors, and tissues such as membranes and layers that cover the tissues. The surgeon dissects the tissues containing the lesions using instruments such as forceps and energy treatment tools while grasping the relationship between these anatomical structures. The surgical field image shown as an example in FIG. 3 shows a scene in which the forceps 13 is used to pull the membrane 32 that covers the organ 31, and the energy treatment tool 12 is used to dissect the tissues containing the lesions 33. The example in FIG. 3 also shows a state in which a nerve 34 runs in the vertical direction in the vicinity of the lesions 33. In laparoscopic surgery, if a nerve is damaged during the process of traction or dissection, functional disorders may occur after surgery. For example, damage to the hypogastric nerve in colon surgery can cause urinary disorders. Damage to the semi-perineal nerve in esophagectomy or lung resection can cause swallowing disorders.

[0036] In order to avoid nerve damage during surgery, it is important to understand the direction of nerves. However, nerves are rarely completely exposed and often overlap with other tissues such as blood vessels, so it is not always easy for surgeons to understand the direction of nerves. Therefore, the support device 200 according to this embodiment uses the learning model 310 to recognize nerve tissue parts included in the surgical field image, distinguishing them from blood vessel tissue parts, and outputs support information related to laparoscopic surgery based on the recognition result.

[0037] An example of the configuration of the learning model 310 will be described below. 4 is a schematic diagram showing a configuration example of the learning model 310. The learning model 310 is a learning model for performing image segmentation, and is constructed by a neural network having a convolution layer such as SegNet. The learning model 310 is not limited to SegNet, and may be constructed by any neural network capable of performing image segmentation, such as FCN (Fully Convolutional Network), U-Net (U-Shaped Network), and PSPNet (Pyramid Scene Parsing Network). Furthermore, the learning model 310 may be constructed by using a neural network for object detection, such as YOLO (You Only Look Once) and SSD (Single Shot Multi-Box Detector), instead of a neural network for image segmentation.

[0038] In this embodiment, the input image to the learning model 310 is a surgical field image obtained from the laparoscope 11. The learning model 310 is trained to output an image indicating the recognition result of nerve tissue included in the surgical field image in response to the input of the surgical field image.

[0039] The learning model 310 includes, for example, an encoder 311, a decoder 312, and a softmax layer 313. The encoder 311 is configured by alternately arranging convolutional layers and pooling layers. The convolutional layers are multi-layered into two or three layers. In the example of FIG. 4, the convolutional layers are shown without hatching, and the pooling layers are shown with hatching.

[0040] In the convolution layer, a convolution operation is performed between the input data and a filter of a predetermined size (for example, 3×3 or 5×5). That is, the input value input to the position corresponding to each element of the filter is multiplied by a weighting coefficient preset in the filter for each element, and a linear sum of the multiplied values ​​for each element is calculated. The output in the convolution layer is obtained by adding a set bias to the calculated linear sum. The result of the convolution operation may be transformed by an activation function. For example, ReLU (Rectified Linear Unit) can be used as the activation function. The output of the convolution layer represents a feature map that extracts the features of the input data.

[0041] In the pooling layer, local statistics of the feature map output from the convolution layer, which is the upper layer connected to the input side, are calculated. Specifically, a window of a predetermined size (e.g., 2×2, 3×3) corresponding to the position of the upper layer is set, and local statistics are calculated from the input values ​​in the window. For example, the maximum value can be used as the statistic. The size of the feature map output from the pooling layer is reduced (downsampled) according to the size of the window. The example in FIG. 4 shows that an input image of 224 pixels×224 pixels is sequentially downsampled to feature maps of 112×112, 56×56, 28×28, ..., 1×1 by sequentially repeating the calculation in the convolution layer and the calculation in the pooling layer in the encoder 311.

[0042] The output of the encoder 311 (a 1×1 feature map in the example of FIG. 4) is input to the decoder 312. The decoder 312 is configured by alternately arranging deconvolution layers and depooling layers. The deconvolution layers are multi-layered into 2 to 3 layers. In the example of FIG. 4, the deconvolution layers are shown without hatching, and the depooling layers are shown with hatching.

[0043] In the deconvolution layer, a deconvolution operation is performed on the input feature map. The deconvolution operation is an operation to restore the feature map before the convolution operation under the assumption that the input feature map is the result of a convolution operation using a specific filter. In this operation, when the specific filter is expressed as a matrix, the transpose matrix of this matrix is ​​calculated and the input feature map is multiplied to generate an output feature map. The operation result of the deconvolution layer may be converted by an activation function such as the above-mentioned ReLU.

[0044] The inverse pooling layers of the decoder 312 correspond to the pooling layers of the encoder 311 in a one-to-one correspondence, and the corresponding pairs have substantially the same size. The inverse pooling layers re-expand (upsample) the size of the feature maps downsampled in the pooling layers of the encoder 311. The example of Fig. 4 shows that the decoder 312 sequentially upsamples feature maps to 1x1, 7x7, 14x14, ..., 224x224 by sequentially repeating the operations in the convolutional layers and the pooling layers.

[0045] The output of decoder 312 (a 224×224 feature map in the example of FIG. 4) is input to softmax layer 313. Softmax layer 313 applies a softmax function to the input values ​​from the deconvolution layer connected to the input side, thereby outputting the probability of the label that identifies the part at each position (pixel). In this embodiment, a label that identifies nervous tissue is set, and whether or not a pixel belongs to nervous tissue may be identified on a pixel-by-pixel basis. An image showing the recognition result of the nervous tissue part (hereinafter referred to as a recognized image) is obtained by extracting pixels for which the probability of the label output from softmax layer 313 is equal to or higher than a threshold (e.g., 70% or higher).

[0046] In the example of FIG. 4, an image of 224 pixels x 224 pixels is used as an input image to the learning model 310, but the size of the input image is not limited to the above, and can be appropriately set according to the processing capacity of the support device 200, the size of the surgical field image obtained from the laparoscope 11, etc. In addition, the input image to the learning model 310 does not need to be the entire surgical field image obtained from the laparoscope 11, and may be a partial image generated by cutting out a region of interest in the surgical field image. Since a region of interest that includes a treatment target is often located near the center of the surgical field image, for example, a partial image obtained by cutting out a rectangular shape near the center of the surgical field image to be about half the size of the original may be used. By reducing the size of the image input to the learning model 310, the processing speed can be increased while the recognition accuracy can be improved.

[0047] FIG. 5 is a schematic diagram showing the recognition result by the learning model 310. In the example of FIG. 5, the nerve tissue portion 51 recognized using the learning model 310 is shown with hatching, and other organs, membranes, and surgical tools are shown with dashed lines for reference. The control unit 201 of the support device 200 generates a recognition image of the nerve tissue to display the recognized nerve tissue portion in a distinguishable manner. The recognition image has the same size as the surgical field image, and is an image in which a specific color is assigned to pixels recognized as nerve tissue. The color assigned to the nerve tissue is set arbitrarily. For example, the color assigned to the nerve tissue may be a white color similar to nerves, or a blue color that does not exist inside the human body. In addition, information indicating the transparency is added to each pixel constituting the recognition image, and a non-transparent value is set to pixels recognized as nerve tissue, and a transparent value is set to other pixels. The support device 200 can display the nerve tissue portion on the surgical field image as a structure having a specific color by superimposing the recognition image generated in this way on the surgical field image.

[0048] Fig. 6 is a partially enlarged view showing the recognition result of the first embodiment. Fig. 6 shows the result of distinguishing and recognizing a nerve tissue portion 61 included in the operative field image from a blood vessel tissue portion 62 appearing on the surface of the nerve tissue portion 61. In the example of Fig. 6, only the recognized nerve tissue portion 61 excluding the blood vessel tissue portion 62 is shown with hatching. From this recognition result, it is possible to confirm the presence of two nerves running parallel to each other in the directions of the two arrows shown in the figure.

[0049] Fig. 7 is a partially enlarged view showing the recognition result of a comparative example. The comparative example of Fig. 7 shows the result of recognizing a nerve tissue portion 71 in the same region as Fig. 6 without distinguishing it from a blood vessel tissue portion that appears on the surface. This recognition result does not clearly indicate the presence of two nerves running parallel to each other, so there is a risk that it will be interpreted as a single relatively thick nerve running in the direction of the arrow.

[0050] In this embodiment, in order to distinguish and recognize a nerve tissue portion included in an operative field image from a blood vessel tissue portion appearing on the surface of the nerve tissue portion, a learning model 310 is generated that recognizes whether or not a pixel corresponds to nerve tissue. As a preparation step for generating the learning model 310, annotation is performed on an already captured operative field image.

[0051] In the preparation stage of generating the learning model 310, an operator (an expert such as a doctor) performs annotation by displaying the operative field image recorded in the recording device 140 on the display device 130 and specifying the part corresponding to the nerve tissue in pixel units using a mouse, a stylus pen, or the like provided as the operation unit 203. At this time, it is preferable that the operator specifies the part corresponding to the nerve tissue in pixel units, excluding vascular tissue that appears on the surface of the nerve tissue. A set of a large number of operative field images used for annotation and data (correct answer data) indicating the positions of pixels corresponding to the specified nerve tissue in each operative field image are stored in the storage unit 202 of the support device 200 as training data for generating the learning model 310. In order to increase the number of training data, a set of an operative field image generated by applying a perspective transformation, a mirroring process, or the like, and correct answer data for the operative field image may be included in the training data. Furthermore, as the learning progresses, a set of an operative field image and a recognition result (correct answer data) of the learning model 310 obtained by inputting the operative field image may be included in the training data.

[0052] Furthermore, when performing annotation, the operator may label pixels corresponding to vascular tissue to be excluded as incorrect data. A set of the operative field image used for annotation, data (correct data) indicating the positions of pixels corresponding to the specified nerve tissue in each operative field image, and data (incorrect data) indicating the positions of pixels corresponding to the specified vascular tissue may be stored in the storage unit 202 of the support device 200 as training data for generating the learning model 310.

[0053] The support device 200 generates the learning model 310 by using the above-mentioned training data. Fig. 8 is a flowchart explaining the procedure for generating the learning model 310. The control unit 201 of the support device 200 reads out the learning processing program PG3 from the storage unit 202, and executes the following procedure to generate the learning model 310. Note that, before the start of learning, it is assumed that initial values ​​are given to the definition information describing the learning model 310.

[0054] The control unit 201 accesses the storage unit 202 and selects a set of training data from training data prepared in advance to generate the learning model 310 (step S101). The control unit 201 inputs an operative field image included in the selected training data to the learning model 310 (step S102) and executes a calculation by the learning model 310 (step S103). That is, the control unit 201 executes a calculation by the encoder 311 that generates a feature map from the input operative field image, sequentially downsamples the generated feature map, a calculation by the decoder 312 that sequentially upsamples the feature map input from the encoder 311, and a calculation by the softmax layer 313 that identifies each pixel of the feature map finally obtained by the decoder 312.

[0055] The control unit 201 acquires the calculation result from the learning model 310 and evaluates the acquired calculation result (step S104). For example, the control unit 201 may evaluate the calculation result by calculating the similarity between the image data of the nerve tissue obtained as the calculation result and the correct answer data included in the training data. The similarity is calculated, for example, by the Jaccard coefficient. The Jaccard coefficient is given by A∩B / A∪B×100(%), where A is the nerve tissue part extracted by the learning model 310 and B is the nerve tissue part included in the correct answer data. Instead of the Jaccard coefficient, the Dice coefficient or the Simpson coefficient may be calculated, or the similarity may be calculated using other existing methods. When the training data includes incorrect answer data, the control unit 201 may proceed with learning by referring to the incorrect answer data. For example, when the nerve tissue part extracted by the learning model 310 corresponds to the blood vessel tissue part included in the incorrect answer data, the control unit 201 may perform a process of subtracting the similarity.

[0056] The control unit 201 determines whether or not the learning is completed based on the evaluation of the calculation result (step S105). The control unit 201 can determine that the learning is completed when a similarity equal to or greater than a preset threshold is obtained.

[0057] When it is determined that the learning is not completed (S105: NO), the control unit 201 uses the backpropagation method to sequentially update the weighting coefficients and biases in each layer of the learning model 310 from the output side to the input side of the learning model 310 (step S106). After updating the weighting coefficients and biases in each layer, the control unit 201 returns the process to step S101 and executes the processes from step S101 to step S105 again.

[0058] When it is determined in step S105 that the learning is completed (S105: YES), the control unit 201 ends the process according to this flowchart since the learned learning model 310 is obtained.

[0059] In this embodiment, the support device 200 is configured to generate the learning model 310, but the learning model 310 may be generated using an external computer such as a server device. In this case, the support device 200 may acquire the learning model 310 generated by the external computer using a means such as communication, and store the acquired learning model 310 in the storage unit 202.

[0060] The support device 200 performs surgery support in the operation phase after the learning model 310 is generated. FIG. 9 is a flowchart explaining the execution procedure of surgery support. The control unit 201 of the support device 200 executes the following procedure by reading out the recognition processing program PG1 and the display processing program PG2 from the storage unit 202 and executing them. When laparoscopic surgery is started, an operative field image obtained by capturing an image of the operative field by the imaging device 11B of the laparoscope 11 is output to the CCU 110 via the universal code 11D as needed. The control unit 201 of the support device 200 acquires the operative field image output from the CCU 110 from the input unit 204 (step S121). The control unit 201 executes the following process every time it acquires an operative field image.

[0061] The control unit 201 inputs the acquired operative field image to the learning model 310, executes a calculation using the learning model 310 (step S122), and recognizes a nervous tissue portion included in the operative field image (step S123). That is, the control unit 201 executes a calculation by an encoder 311 that generates a feature map from the input operative field image and sequentially downsamples the generated feature map, a calculation by a decoder 312 that sequentially upsamples the feature map input from the encoder 311, and a calculation by a softmax layer 313 that identifies each pixel of the feature map finally obtained by the decoder 312. The control unit 201 also recognizes a pixel whose label probability output from the softmax layer 313 is equal to or higher than a threshold (e.g., 70% or higher) as a nervous tissue portion.

[0062] When generating the learning model 310, if annotation is performed to recognize nerve tissue in the central visual field of the surgeon, only nerve tissue present in the central visual field of the surgeon is recognized in step S123. Also, if annotation is performed to recognize nerve tissue not in the central visual field of the surgeon, only nerve tissue not in the central visual field of the surgeon is recognized in step S123. Furthermore, if annotation is performed to recognize nerve tissue in a tensioned state, in step S123, the nerve tissue is recognized as nerve tissue at the stage where the nerve tissue transitions from a pre-tensioned state to a tensioned state. Furthermore, if annotation is performed to recognize nerve tissue that has begun to be exposed in the surgical field, the nerve tissue is recognized as nerve tissue at the stage where the nerve tissue begins to be exposed by retracting or cutting a membrane or layer covering a tissue such as an organ.

[0063] The control unit 201 generates a recognition image of the nervous tissue in order to distinguishably display the nervous tissue portion recognized using the learning model 310 (step S124). The control unit 201 assigns a specific color, such as a white color similar to nerves or a bluish color not present in the human body, to pixels recognized as nervous tissue, and sets a transparency for pixels other than the nervous tissue such that the background is visible through the pixels.

[0064] The control unit 201 outputs the recognition image of the nerve tissue generated in step S124 from the output unit 205 to the display device 130 together with the operative field image acquired in step S121, and displays the recognition image on the display device 130 by superimposing it on the operative field image (step S125). As a result, the nerve tissue portion recognized using the learning model 310 is displayed on the operative field image as a structure having a specific color.

[0065] Fig. 10 is a schematic diagram showing a display example on the display device 130. For convenience of drawing, in the display example of Fig. 10, the nervous tissue portion 101 recognized using the learning model 310 is shown as a hatched area. In reality, the nervous tissue portion 101 is painted in a specific color such as white or blue on a pixel-by-pixel basis, so that the surgeon can clearly recognize the nervous tissue portion 101 as being distinct from the blood vessel tissue portion 102 by viewing the display screen of the display device 130.

[0066] In the present embodiment, the pixels corresponding to the nerve tissue are displayed in a white or blue color, but a preset display color (white or blue) and a display color of the background surgical field image may be averaged, and the averaged color may be displayed. For example, if the display color set for the nerve tissue portion is (R1, G1, B1), and the display color of the nerve tissue portion in the background surgical field image is (R2, G2, B2), the control unit 201 may display the recognized nerve tissue portion in the colors ((R1+R2) / 2, (G1+G2) / 2, (B1+B2) / 2). Alternatively, weighting coefficients W1 and W2 may be introduced, and the recognized blood vessel portion may be displayed in the colors (W1×R1+W2×R2, W1×G1+W2×G2, W1×B1+W2×B2).

[0067] Also, the recognized target tissue portion (nerve tissue portion in this embodiment) may be displayed in a blinking manner. That is, the control unit 201 may periodically switch between displaying and hiding the target tissue portion by alternately and repeatedly executing a process of displaying the recognized target tissue portion for a first set time (for example, 2 seconds) and a process of hiding the recognized target tissue portion for a second set time (for example, 2 seconds). The display time and non-display time of the target tissue portion may be set appropriately. Also, the display and non-display of the target tissue portion may be switched in synchronization with biological information such as the patient's heart rate and pulse. Furthermore, instead of the configuration of displaying the target tissue portion in a blinking manner, the blood vessel tissue portion may be displayed in a blinking manner. By displaying only the target tissue portion excluding the blood vessel tissue portion in a blinking manner or by displaying only the blood vessel tissue portion in a blinking manner, the target tissue portion can be highlighted and distinguished from the blood vessel tissue portion.

[0068] 9, when a nerve tissue included in an operative field image is recognized, the recognized image of the nerve tissue portion is superimposed on the operative field image. However, the superimposed display may be performed only when a display instruction is given. The display instruction may be given by the operation unit 203 of the support device 200, or may be given by the operation unit 11C of the laparoscope 11. The display instruction may also be given by a foot switch (not shown) or the like.

[0069] Furthermore, in this embodiment, the recognized image of the nerve tissue is displayed superimposed on the operative field image, but the detection of the nerve tissue may be notified to the surgeon by sound or voice.

[0070] Furthermore, the control unit 201 of the assistance device 200 may be configured to generate a control signal for controlling a medical device such as an energy treatment tool 12 or a surgical robot (not shown) based on the recognition result of the nervous tissue, and output the generated control signal to the medical device.

[0071] As described above, in this embodiment, neural tissue can be recognized using learning model 310 and the recognized neural tissue can be displayed in a distinguishable manner on a pixel-by-pixel basis, thereby providing visual support in laparoscopic surgery.

[0072] If a nerve recognition method were to be adopted that recognized nerve tissue and the vascular tissue that appears on the surface of the nerve tissue as a single region without distinguishing between them, the area containing the nerves in the surgical field image would be covered with a solid image, making the nerves themselves difficult to see, and there was a possibility that information necessary for the surgeon performing the surgery would be lost.

[0073] In contrast, in the present embodiment, the recognized nerve tissue can be displayed in a distinguishable manner on a pixel-by-pixel basis, so that the recognized nerve tissue can be displayed in an easily visible manner. In addition, the nerve tissue and the blood vessel tissue that appears on the surface of the nerve tissue are displayed separately, so that the running direction of the nerve is emphasized. By grasping the running direction of the nerve, the surgeon can predict the presence of an invisible nerve.

[0074] In this embodiment, in order to distinguish and recognize nerve tissue from vascular tissue appearing on the surface of the nerve tissue, pixels corresponding to nerve tissue and pixels corresponding to vascular tissue appearing on the surface of the nerve tissue are distinguished and annotated, and learning is performed using training data obtained by this annotation to generate a learning model 310. The surface blood vessels appearing on the surface of the nerve have a pattern unique to the nerve, which is different from the pattern of the surface blood vessels appearing on other organs. By using the above-mentioned training data during learning, learning progresses not only with the position information of the nerve, but also with information on the pattern of the surface blood vessels appearing on the surface of the nerve, improving the accuracy of nerve recognition.

[0075] The images generated by the support device 200 may be used not only for surgical support but also for educational support for trainees and the like, and may be used for evaluation of laparoscopic surgery. For example, the laparoscopic surgery can be evaluated by comparing the images recorded by the recording device 140 during surgery with the images generated by the support device 200 to determine whether the traction operation and dissection operation in the laparoscopic surgery were appropriate.

[0076] (Embodiment 2) In the second embodiment, a configuration will be described in which nerve tissue running in a first direction and nerve tissue running in a second direction different from the first direction are distinguished and recognized.

[0077] Fig. 11 is a schematic diagram showing an example of an operative field image in embodiment 2. Fig. 11 shows an operative field image including an organ 111 appearing in the lower region (dotted region) of the operative field image, nerve tissue 112 running in a direction along the organ 111 (in the direction of the black arrow in the figure), and nerve tissue 113 branching off from nerve tissue 112 and running in a direction toward the organ 111 (in the direction of the white arrow in the figure).

[0078] In the following, the nerve tissue running in a direction along the organ is referred to as the first nerve tissue, and the nerve tissue running in a direction toward the organ is referred to as the second nerve tissue. In this embodiment, the first nerve tissue represents a nerve that should be preserved in laparoscopic surgery. For example, the vagus nerve and the recurrent laryngeal nerve correspond to the first nerve tissue. In contrast, the second nerve tissue represents a nerve that can be dissected in laparoscopic surgery, and is dissected as necessary when unfolding an organ or resecting a diseased part. The first nerve tissue and the second nerve tissue do not need to be a single nerve tissue, and may be tissue such as a nerve plexus or a nerve fiber bundle.

[0079] It would be useful for the surgeon if it were possible to distinguish and recognize a first nerve tissue running in one direction (first direction) from a second nerve tissue running in the other direction (second direction) and provide the recognition result to the surgeon. Therefore, the support device 200 according to the second embodiment uses a learning model 320 (see FIG. 12) to distinguish and recognize the first nerve tissue running in the first direction from the second nerve tissue running in the second direction.

[0080] FIG. 12 is an explanatory diagram for explaining the configuration of the learning model 320 in the second embodiment. In FIG. 12, for simplification, only the softmax layer 323 of the learning model 320 is shown. The configuration other than the softmax layer 323 is the same as that of the learning model 310 shown in the first embodiment. The softmax layer 323 included in the learning model 320 in the second embodiment outputs the probability of the label set corresponding to each pixel. In the second embodiment, a label identifying the first nerve tissue, a label identifying the second nerve tissue, and a label indicating the other are set. The control unit 201 of the support device 200 recognizes that the pixel is the first nerve tissue if the probability of the label identifying the first nerve tissue is equal to or greater than a threshold, and recognizes that the pixel is the second nerve tissue if the probability of the label identifying the second nerve tissue is equal to or greater than a threshold. In addition, the control unit 201 recognizes that the pixel is neither the first nerve tissue nor the second nerve tissue if the probability of the label indicating the other is equal to or greater than a threshold.

[0081] The learning model 320 for obtaining such a recognition result is generated by learning using a set including an operative field image and ground truth data indicating the respective positions (pixels) of the first nerve tissue and the second nerve tissue included in the operative field image as training data. The method of generating the learning model 320 is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0082] 13 is a schematic diagram showing a display example of the recognition result in the second embodiment. The control unit 201 of the support device 200 inputs the surgical field image to the trained learning model 320, thereby acquiring the recognition result by the learning model 320. The control unit 201 generates a recognition image capable of distinguishing the first nerve tissue from the second nerve tissue, with reference to the recognition result by the learning model 320. For example, the control unit 201 can generate the recognition image by assigning a specific color such as white or blue to pixels recognized as the first nerve tissue, and assigning a different color to pixels recognized as the second nerve tissue.

[0083] 13, for convenience of drawing, only the first nerve tissue portion 131 (the nerve tissue portion running along the organ) is shown with hatching. In reality, the portion corresponding to the first nerve tissue may be displayed by applying a specific color such as white or blue to each pixel.

[0084] In addition, while FIG. 13 shows only the first nerve tissue portion 131, it is also possible to display only the second nerve tissue portion 132 (the nerve tissue portion running in the direction toward the organ), and the first nerve tissue portion 131 and the second nerve tissue portion 132 may be displayed in different display modes.

[0085] As described above, in embodiment 2, it is possible to distinguish and recognize nerve tissue running in a first direction from nerve tissue running in a second direction, and therefore it is possible to inform the surgeon of, for example, the presence of nerve tissue that should be preserved and the presence of nerve tissue that can be peeled off.

[0086] In this embodiment, both the nerve tissue running in the first direction and the nerve tissue running in the second direction are recognized, but only the nerve tissue running in the first direction (or the second direction) may be recognized. In this case, the learning model 320 may be generated using training data including pixels corresponding to the nerve tissue running in the first direction (or the second direction) as correct data and pixels corresponding to the nerve tissue running in the second direction (or the first direction) as incorrect data. By recognizing the nerve tissue using such a learning model 320, only the nerve tissue running in the first direction (or the second direction) can be recognized.

[0087] (Embodiment 3) In the third embodiment, a configuration for distinguishing and recognizing nervous tissue and loose connective tissue will be described.

[0088] Fig. 14 is a schematic diagram showing an example of an operative field image in embodiment 3. Fig. 14 shows an operative field image including an organ 141 appearing in a central region (a dotted region) of the operative field image, nerve tissue 141 running in the left-right direction (the direction of the black arrow in the figure) on the surface of the organ 141, and loose connective tissue 143 running in a direction intersecting with the nerve tissue 141 (the direction of the white arrow in the figure).

[0089] Loose connective tissue is a fibrous connective tissue that fills the spaces between tissues and organs, and has a relatively small amount of fibers (collagen fibers and elastic fibers) that make up the tissue. Loose connective tissue is peeled off as necessary when exposing organs or removing lesions.

[0090] Since the nervous tissue and the loose connective tissue appearing in the surgical field image are both white in color and linearly extending, it is often difficult to distinguish them from each other in appearance. Therefore, it would be useful for the surgeon if the nervous tissue and the loose connective tissue could be distinguished and recognized, and the recognition result could be provided to the surgeon. Therefore, the support device 200 according to the third embodiment distinguishes the nervous tissue from the loose connective tissue and recognizes them using a learning model 330 (see FIG. 15).

[0091] FIG. 15 is an explanatory diagram for explaining the configuration of the learning model 330 in the third embodiment. In FIG. 15, for simplification, only the softmax layer 333 of the learning model 330 is shown. The configuration other than the softmax layer 333 is the same as that of the learning model 310 shown in the first embodiment. The softmax layer 333 included in the learning model 330 in the third embodiment outputs the probability of the label set corresponding to each pixel. In the third embodiment, a label identifying nervous tissue, a label identifying loose connective tissue, and a label indicating other are set. The control unit 201 of the support device 200 recognizes that the pixel is nervous tissue if the probability of the label identifying nervous tissue is equal to or higher than a threshold, and recognizes that the pixel is loose connective tissue if the probability of the label identifying loose connective tissue is equal to or higher than a threshold. In addition, the control unit 201 recognizes that the pixel is neither nervous tissue nor loose connective tissue if the probability of the label indicating other is equal to or higher than a threshold.

[0092] The learning model 330 for obtaining such a recognition result is generated by learning using a set including the surgical field image and the ground truth data indicating the respective positions (pixels) of the nerve tissue and the loose connective tissue contained in the surgical field image as training data. The method of generating the learning model 330 is the same as that of the first embodiment, and therefore the description thereof will be omitted.

[0093] 16 is a schematic diagram showing a display example of the recognition result in the third embodiment. The control unit 201 of the support device 200 inputs the surgical field image to the trained learning model 330, thereby acquiring the recognition result by the learning model 330. The control unit 201 generates a recognition image capable of distinguishing between nervous tissue and loose connective tissue, with reference to the recognition result by the learning model 330. For example, the control unit 201 can generate the recognition image by assigning a specific color such as white or blue to pixels recognized as nervous tissue, and assigning a different color to pixels recognized as loose connective tissue.

[0094] 16, for convenience of drawing, only the nerve tissue portion 161 is shown with hatching. In reality, the portion corresponding to the nerve tissue may be displayed by applying a specific color such as white or blue to each pixel.

[0095] Further, in FIG. 16, only the nervous tissue portion 161 is displayed, but only the loose connective tissue portion 162 may be displayed, or the nervous tissue portion 161 and the loose connective tissue portion 162 may be displayed in different display modes.

[0096] As described above, in embodiment 3, it is possible to distinguish and recognize nervous tissue and loose connective tissue, and therefore it is possible to inform the surgeon of, for example, the presence of nervous tissue that should be preserved and the presence of loose connective tissue that can be dissected.

[0097] In this embodiment, both nervous tissue and loose connective tissue are recognized, but only nervous tissue (or loose connective tissue) may be recognized. In this case, the learning model 330 may be generated using training data including pixels corresponding to nervous tissue (or loose connective tissue) as correct data and pixels corresponding to loose connective tissue (or nervous tissue) as incorrect data. By using such a learning model 330, the control unit 201 can recognize only nervous tissue (or loose connective tissue) by distinguishing it from loose connective tissue (or nervous tissue).

[0098] (Fourth embodiment) In the fourth embodiment, a configuration will be described in which the display mode is changed depending on the degree of certainty of the recognition result for the nerve tissue.

[0099] As described in the fourth embodiment, the softmax layer 313 of the learning model 310 outputs a probability for the label set corresponding to each pixel. This probability represents the confidence of the recognition result. The control unit 201 of the support device 200 changes the display mode of the nervous tissue part according to the confidence of the recognition result.

[0100] Fig. 17 is a schematic diagram showing a display example in the fourth embodiment. Fig. 17 shows an enlarged view of an area including nervous tissue. In this example, the nervous tissue portion is displayed with different densities for the reliability of the recognition result of the nervous tissue, which are 70% to 80%, 80% to 90%, 90% to 95%, and 95% to 100%. In this example, the display mode is changed so that the higher the reliability, the higher the density.

[0101] 17, the density is changed according to the confidence level, but the color or transparency may be changed according to the confidence level. When changing the color, for example, the lower the confidence level, the whiter the color displayed, and the higher the confidence level, the bluer the color displayed. When changing the transparency, the display mode may be changed so that the higher the confidence level, the lower the transparency.

[0102] In the example of FIG. 17, the density is changed in four stages according to the degree of certainty, but the density may be set in finer increments to provide a gradation display according to the degree of certainty.

[0103] (Embodiment 5) In the fifth embodiment, a configuration will be described in which an estimated position of a nerve tissue portion that is hidden behind an object such as a surgical tool and cannot be visually recognized is displayed.

[0104] 18 is an explanatory diagram for explaining a display method in the fifth embodiment. As described above, the support device 200 recognizes a nerve tissue portion included in the surgical field image by using the learning model 310. However, when an object such as a surgical tool or gauze is present in the surgical field to be imaged, the support device 200 cannot recognize the nerve tissue hidden behind the object from the surgical field image even if the support device 200 uses the learning model 310. Therefore, when a recognition image of the nerve tissue is displayed superimposed on the surgical field image, the nerve tissue portion hidden behind the object cannot be displayed in a distinguishable manner.

[0105] Therefore, the support device 200 of embodiment 5 stores in the memory unit 202 a recognition image of the nervous tissue recognized when it is not hidden behind an object, and when the nervous tissue part is hidden behind an object, it reads out the recognition image stored in the memory unit 202 and displays it superimposed on the surgical field image.

[0106] 18, time T1 shows an operative field image in a state where the nervous tissue is not hidden behind the surgical tools, and time T2 shows an operative field image in a state where part of the nervous tissue is hidden behind the surgical tools. However, between time T1 and time T2, the laparoscope 11 is not moved, and there is no change in the area being imaged.

[0107] From the operative field image at time T1, it is possible to recognize the nervous tissue appearing in the operative field, and a recognized image of the nervous tissue is generated from the recognition result of the learning model 310. The generated recognized image of the nervous tissue is stored in the storage unit 202.

[0108] On the other hand, from the surgical field image at time T2, among the nerve tissues appearing in the surgical field, nerve tissues that are not hidden by surgical tools can be recognized, but nerve tissues hidden by surgical tools cannot be recognized. Therefore, the support device 200 displays a recognition image of the nerve tissue generated from the surgical field image at time T1 superimposed on the surgical field image at time T2. In the example of FIG. 18, the part indicated by the dashed line is a nerve tissue part hidden by the surgical tools and cannot be seen, but the support device 200 can display the part in a distinguishable manner by reusing the recognition image recognized at time T1.

[0109] As described above, in embodiment 5, it is possible to notify the surgeon of the presence of nerve tissue that is hidden behind objects such as surgical tools and gauze and cannot be seen, thereby improving safety during surgery.

[0110] In the present embodiment, the recognition image of the nerve tissue recognized in a state where it is not hidden behind an object is used to display the nerve tissue part hidden behind the object in a distinguishable manner, but the support device 200 may display the nerve tissue part hidden behind the object in a distinguishable manner by estimating the nerve tissue part hidden behind the object using a mathematical method such as interpolation or extrapolation. In addition, the support device 200 may display the nerve tissue part not hidden behind the object and the nerve tissue part hidden behind the object in different display modes (color, density, transparency, etc.). Furthermore, the support device 200 may generate a recognition image including both the nerve tissue part not hidden behind the object and the nerve tissue part hidden behind the object using a learning model of an image generation system such as GAN (Generative Adversarial Network) or VAE (Variational AutoEncoder), and display the generated recognition image superimposed on the surgical field image.

[0111] (Embodiment 6) In the sixth embodiment, a configuration will be described in which the running pattern of the nerve tissue is predicted, and a nerve part estimated based on the predicted running pattern of the nerve tissue is displayed in a distinguishable manner.

[0112] FIG. 19 is a flowchart for explaining the procedure of the process executed by the support device 200 according to the sixth embodiment. The control unit 201 of the support device 200 acquires an operative field image (step S601) in the same manner as in the first embodiment, inputs the acquired operative field image to the learning model 310, and executes a calculation by the learning model 310 (step S602). The control unit 201 predicts the running pattern of the nerve tissue based on the calculation result by the learning model 310 (step S603). In the first embodiment, a recognition image of the nerve tissue portion is generated by extracting pixels whose probability of the label output from the softmax layer 313 of the learning model 310 is equal to or higher than a threshold (e.g., 70% or higher). In the sixth embodiment, however, the threshold is lowered to predict the running pattern of the nerve tissue. Specifically, the control unit 201 predicts the running pattern of the nerve tissue by extracting pixels whose probability of the label output from the softmax layer 313 of the learning model 310 is equal to or higher than a first threshold (e.g., 40% or higher) and less than a second threshold (e.g., 70%).

[0113] Control unit 201 displays the nerve tissue portion estimated from the predicted running pattern in a distinguishable manner (step S604). Fig. 20 is a schematic diagram showing a display example in embodiment 6. In Fig. 20, recognized nerve tissue portion 201A is shown by hatching, and nerve tissue portion 201B estimated from the predicted running pattern is shown by a thick dashed line. For convenience of drawing, in the example of Fig. 20, recognized nerve tissue portion 201A is shown by hatching, and nerve tissue portion 201B estimated from the running pattern is shown by a thick dashed line, but it is sufficient to display them by changing the display mode such as color, density, transparency, etc.

[0114] As described above, in the sixth embodiment, the nerve tissue portion estimated from the travel pattern can be displayed together, so that visual support can be provided in laparoscopic surgery.

[0115] In this embodiment, the running pattern of the nervous tissue is predicted by lowering the threshold for recognizing the nervous tissue, but the support device 200 may use a learning model of an image generation system such as GAN or VAE to generate a recognition image including the running pattern of the nervous tissue that cannot be clearly seen from the surgical field image, and display the generated recognition image superimposed on the surgical field image.

[0116] (Embodiment 7) In the first to sixth embodiments, the configuration for recognizing nerve tissue as the target tissue has been described, but the target tissue is not limited to the nerve tissue and may be the ureter. In the seventh embodiment, a configuration for recognizing the ureter instead of the nerve tissue will be described.

[0117] FIG. 21 is an explanatory diagram for explaining the configuration of the learning model 340 in the seventh embodiment. For simplification, FIG. 21 shows only the softmax layer 343 of the learning model 340. The configuration other than the softmax layer 343 is the same as that of the learning model 310 shown in the first embodiment. The softmax layer 343 included in the learning model 340 in the seventh embodiment outputs the probability of the label set corresponding to each pixel. In the seventh embodiment, a label identifying ureteral tissue and a label indicating other than that are set. The control unit 201 of the assistance device 200 recognizes that the pixel is ureteral tissue if the probability of the label identifying ureteral tissue is equal to or higher than a threshold. In addition, the control unit 201 recognizes that the pixel is not ureteral tissue if the probability of the label indicating other than that is equal to or higher than a threshold.

[0118] A learning model 340 for obtaining such a recognition result is generated by learning using a set including an operative field image and ground truth data indicating the position (pixel) of the ureteral tissue contained in the operative field image as training data. That is, the learning model 340 in the seventh embodiment is trained to recognize the ureteral tissue separately from the blood vessel tissue. The method of generating the learning model 340 is the same as that in the first embodiment, and therefore the description thereof will be omitted.

[0119] 22 is a schematic diagram showing a display example of the recognition result in the seventh embodiment. The control unit 201 of the support device 200 inputs the surgical field image to the trained learning model 340, thereby acquiring the recognition result by the learning model 340. The control unit 201 generates a recognition image capable of distinguishing between ureteral tissue and other tissues including vascular tissue, with reference to the recognition result by the learning model 340. For example, the control unit 201 can generate the recognition image by assigning a specific color such as white or blue to pixels recognized as ureteral tissue.

[0120] 22, for convenience of drawing, only the ureteral tissue portion 221 is shown with hatching. In reality, the portion corresponding to the ureteral tissue may be displayed by applying a specific color such as white or blue to each pixel.

[0121] As described above, in the seventh embodiment, the learning model 340 is used to recognize ureteral tissue, and the recognized ureteral tissue can be displayed in a distinguishable manner on a pixel-by-pixel basis, thereby providing visual support in laparoscopic surgery.

[0122] If a method for recognizing the ureter were adopted that does not distinguish between ureteral tissue and vascular tissue that appears on the surface of the ureteral tissue and recognizes them as a single region, the region including the ureter in the surgical field image would be covered with a solid image, making it difficult to see the ureter itself and potentially causing the loss of information necessary for the surgeon performing the surgery. For example, the ureter performs peristaltic movement to transport urine from the renal pelvis to the bladder, but if the region including the ureter is covered with a solid image, it may be difficult to recognize the peristaltic movement.

[0123] In contrast, in this embodiment, the recognized ureteral tissue can be displayed in a distinguishable manner on a pixel-by-pixel basis, so that the recognized ureteral tissue can be displayed in an easily visible manner. In particular, in this embodiment, the ureteral tissue and the vascular tissue (surface blood vessels) that appear on the surface of the ureteral tissue are displayed separately, so that the presence of the surface blood vessels that move with the peristaltic movement of the ureter is emphasized. As a result, the surgeon can easily recognize the peristaltic movement of the ureter. Furthermore, by displaying the vascular tissue that appears on the surface of the ureteral tissue without including it, the running direction of the ureter is further emphasized. By grasping the running direction of the ureter, the surgeon can predict the presence of an unseen ureter.

[0124] In this embodiment, in order to distinguish between ureteral tissue and vascular tissue appearing on the surface of the ureteral tissue, pixels corresponding to the ureteral tissue and pixels corresponding to vascular tissue appearing on the surface of the ureteral tissue are distinguished and annotated, and learning is performed using training data obtained by this annotation to generate a learning model 340. The surface blood vessels appearing on the surface of the ureter have a pattern unique to the ureter and are different from the patterns of surface blood vessels appearing on other organs. By using the above-mentioned training data during learning, learning progresses not only with the position information of the ureter but also with information on the pattern of the surface blood vessels appearing on the surface of the ureter, improving the recognition accuracy of the ureter.

[0125] (Embodiment 8) In the eighth embodiment, a configuration will be described in which the boundary of an organ is identified by recognizing surface blood vessels appearing on the surface of the organ and specifying the positions of the ends of the recognized surface blood vessels.

[0126] FIG. 23 is an explanatory diagram for explaining the configuration of the learning model 350 in the eighth embodiment. For simplification, FIG. 23 shows only the softmax layer 353 of the learning model 350. The configuration other than the softmax layer 353 is the same as that of the learning model 310 shown in the first embodiment. The softmax layer 353 included in the learning model 350 in the eighth embodiment outputs the probability of the label set corresponding to each pixel. In the eighth embodiment, a label for identifying a surface blood vessel and a label indicating the other are set. If the probability of the label for identifying a surface blood vessel is equal to or higher than a threshold, the control unit 201 of the support device 200 recognizes that the pixel corresponds to a surface blood vessel. Also, if the probability of the label indicating the other is equal to or higher than a threshold, the control unit 201 recognizes that the pixel is not a surface blood vessel.

[0127] The learning model 350 for obtaining such a recognition result is generated by learning using a set including an operative field image and ground truth data indicating the positions (pixels) of the surface blood vessels included in the operative field image as training data. That is, the learning model 350 in the eighth embodiment is trained to recognize the surface blood vessels separately from other tissues. The method of generating the learning model 350 is the same as that in the first embodiment, and therefore the description thereof will be omitted.

[0128] Fig. 24 is an explanatory diagram for explaining a method for identifying organ boundaries. The control unit 201 of the support device 200 inputs an operative field image to a trained learning model 350, thereby acquiring a recognition result by the learning model 350. The control unit 201 generates a recognition image of surface blood vessels appearing on the surface of the organ, referring to the recognition result by the learning model 350. In Fig. 24, solid lines represent surface blood vessels recognized by the learning model 350.

[0129] The control unit 201 specifies the position coordinates of the ends of the surface blood vessels in the generated recognition image. For example, the control unit 201 can specify the position coordinates of the ends by determining the number of adjacent pixels belonging to the same segment for each pixel constituting a segment of the surface blood vessel and specifying a pixel with the number of adjacent pixels being 1. FIG. 24 shows an example in which the coordinates of four points P1 to P4 are specified as the position coordinates of the ends of the surface blood vessels. The control unit 201 specifies the boundary of the organ where the surface blood vessels appear by deriving an approximation curve passing through the specified points P1 to P4 (or the vicinity of the points P1 to P4). A known method such as the least squares method can be used to derive the approximation curve. The control unit 201 may also specify the boundary of the organ where the surface blood vessels appear by deriving a closed curve that includes all the specified end points.

[0130] The control unit 201 does not need to identify all of the organ boundaries, and may be configured to identify a portion of the organ boundaries.

[0131] 25 is a flowchart for explaining the procedure of processing executed by the support device 200 according to embodiment 8. As in embodiment 1, the control unit 201 of the support device 200 acquires an operative field image (step S801), inputs the acquired operative field image to the learning model 350, and executes a calculation by the learning model 350 (step S802). The control unit 201 recognizes surface blood vessels appearing on the surface of the organ based on the calculation result by the learning model 350 (step S803).

[0132] Next, the control unit 201 identifies the position coordinates of the ends of the surface blood vessels (step S804). At this time, the control unit 201 may identify the position coordinates of the ends of all surface blood vessels, or may extract only surface blood vessels whose length is equal to or greater than a threshold and identify the position coordinates of the ends.

[0133] Next, the control unit 201 identifies the border of the organ based on the position coordinates of the ends of the identified surface blood vessels (step S805). As described above, the control unit 201 can identify the border of the organ where the surface blood vessels appear by deriving an approximation curve that passes through the position coordinates (or the vicinity of the position coordinates) of the ends of the identified surface blood vessels.

[0134] As described above, in the eighth embodiment, the border of an organ can be identified by using the surface blood vessels that appear on the surface of the organ as a clue. The support device 200 can support the surgery by presenting the information of the identified border to the surgeon.

[0135] In the eighth embodiment, the border of an organ is identified. However, the target tissue for which the border is to be identified is not limited to an organ, and may be a membrane or layer covering an organ.

[0136] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0137] 10 Trocca 11 Laparoscopy 12 Energy treatment device 13 forceps 110 Camera Control Unit (CCU) 120 Light source device 130 Display device 140 Recording Device 200 Support equipment 201 Control section 202 Storage section 203 Operation section 204 Input section 205 Output section 206 Communications Department 310,320,330,340,350 Learning Model PG1 Recognition Processing Program PG2 Display processing program PG3 Learning Processing Program

Claims

1. On the computer, Acquire an image of the surgical field obtained by imaging the surgical field of laparoscopic surgery; When an image of the surgical field is input, a learning model is used that is trained to output information for identifying the surface blood vessels of the target tissue from the acquired surgical field image to identify the positions of the surface blood vessels of the target tissue; Identifying the boundary of the target tissue based on the location of the identified surface blood vessels. A computer program for executing a process.

2. The target tissue includes a membrane or layer that exists within the surgical field as an anatomical structure.

2. The computer program product of claim 1.

3. The computer Acquire an image of the surgical field obtained by imaging the surgical field of laparoscopic surgery; When an image of the surgical field is input, a learning model is used that is trained to output information for identifying the surface blood vessels of the target tissue from the acquired surgical field image to identify the positions of the surface blood vessels of the target tissue; Identifying the boundary of the target tissue based on the location of the identified surface blood vessels. Information processing methods.

4. an acquisition unit for acquiring an operative field image obtained by imaging an operative field of laparoscopic surgery; a control unit that, when an operative field image is input, identifies the positions of the surface blood vessels in the target tissue from the acquired operative field image using a learning model that has been trained to output information for identifying the surface blood vessels of the target tissue, and identifies the boundary of the target tissue based on the identified positions of the surface blood vessels; an output unit that outputs information on the boundary of the target tissue identified by the control unit; A support device comprising:

Images