Information processing device, information processing method, and computer program
The information processing device integrates multiple learning models to derive comprehensive recognition results from surgical field images, addressing the limitations of single-class inference in pathological diagnosis.
Patent Information
- Application Number
- JP2024160255
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-05-24
- Filing Date
- 2024-09-17
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-05-24
AI Technical Summary
Existing pathological diagnosis methods require specialized knowledge and often use single-class training data, leading to single-class inference limitations and the inability to derive integrated recognition results from multiple types of learning models.
An information processing device and method that utilize multiple learning models to perform calculations on surgical field images, integrating recognition results through a derivation unit and outputting integrated recognition information based on the calculation results of these models.
Enables the derivation of integrated recognition results from multiple types of learning models, improving diagnostic accuracy and overcoming limitations of single-class inference.
Smart Images

Figure 2024169523000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device, an information processing method, and a computer program. [Background technology]
[0002] In the medical field, pathological diagnosis requires specialized knowledge and experience, and is therefore performed by removing a portion of body tissue and examining it outside the body using a microscope or the like.
[0003] Meanwhile, in recent years, an invention has been proposed in which the inside of the body is directly observed using an endoscope or the like, and the observed images are analyzed by a computer to perform a pathological diagnosis (see, for example, Patent Document 1). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2006-153742 A Summary of the Invention [Problem to be solved by the invention]
[0005] The creation of learning models for recognizing lesions and other conditions from observed images requires the knowledge of medical specialists, but because medical specialists' fields of expertise are highly specialized, single-class training data is often created. A learning model created with single-class training data can only perform single-class inference.
[0006] An object of the present invention is to provide an information processing device, an information processing method, and a computer program for deriving an integrated recognition result from the calculation results of a plurality of types of learning models. [Means for solving the problem]
[0007] An information processing device in one embodiment of the present invention includes a first calculation unit that performs calculations using a first learning model in response to input of a operative field image, a second calculation unit that performs calculations using a second learning model in response to input of the operative field image, a derivation unit that derives an integrated recognition result for the operative field image based on the calculation results output from the first calculation unit and the calculation results output from the second calculation unit, and an output unit that outputs information based on the derived recognition result.
[0008] An information processing method in one embodiment of the present invention includes a computer executing a calculation using a first learning model in response to input of a surgical field image, executing a calculation using a second learning model in response to input of the surgical field image, deriving an integrated recognition result for the surgical field image based on the calculation result using the first learning model and the calculation result using the second learning model, and outputting information based on the derived recognition result.
[0009] A computer program in one embodiment of the present invention is a computer program for causing a computer to execute a process of performing a calculation using a first learning model in response to input of a surgical field image, performing a calculation using a second learning model in response to input of the surgical field image, deriving an integrated recognition result for the surgical field image based on the calculation result using the first learning model and the calculation result using the second learning model, and outputting information based on the derived recognition result. Effect of the Invention
[0010] According to the present application, an integrated recognition result can be derived from the computation results of a plurality of types of learning models. [Brief description of the drawings]
[0011] [Figure 1] 1 is a schematic diagram illustrating a schematic configuration of a surgery support system according to embodiment 1. FIG. [Diagram 2] FIG. 2 is a block diagram illustrating an internal configuration of an information processing device. [Diagram 3] FIG. 2 is a schematic diagram showing an example of an operative field image. [Figure 4]FIG. 2 is a schematic diagram showing an example of the configuration of a first learning model. [Diagram 5] FIG. 2 is a schematic diagram showing an example of the configuration of a second learning model. [Figure 6] 4 is a flowchart showing a procedure of a process executed by the information processing device according to the first embodiment. [Figure 7] FIG. 4 is a schematic diagram showing an example of display of a recognition result in the first embodiment. [Figure 8] 4 is a schematic diagram showing an example of a display of warning information in the first embodiment. FIG. [Figure 9] 11A and 11B are schematic diagrams showing examples of displaying recognition results according to confidence levels; [Figure 10] FIG. 13 is a schematic diagram showing an example of the configuration of a third learning model. [Figure 11] 11 is a flowchart showing a procedure of a process executed by an information processing device according to the second embodiment. [Figure 12] FIG. 11 is a schematic diagram showing an example of display of a recognition result in the second embodiment. [Figure 13] FIG. 13 is a schematic diagram showing an example of the configuration of a fourth learning model. [Figure 14] 11 is a flowchart showing a procedure of a process executed by an information processing device according to the third embodiment. [Figure 15] 13 is a flowchart showing a procedure for deriving dimensional information. [Figure 16] A schematic diagram showing an example of the configuration of a fifth learning model. [Figure 17] 13 is a flowchart showing a procedure of a process executed by an information processing device according to the fourth embodiment. [Figure 18] 13 is a flowchart showing a processing procedure in Modification Example 4-1. [Figure 19] FIG. 13 is a conceptual diagram illustrating an example of a prior information table. [Figure 20] 13 is a flowchart showing a processing procedure in Modification Example 4-2. [Figure 21] 13 is a flowchart showing a processing procedure in Modification Example 4-3. [Figure 22] 13 is a flowchart showing a processing procedure in Modification Example 4-4. [Diagram 23] FIG. 13 is a conceptual diagram illustrating an example of a proper name table. [Figure 24] FIG. 13 is a schematic diagram showing an example of display of organ names. [Diagram 25] FIG. 13 is a conceptual diagram illustrating an example of a structure table. [Figure 26] FIG. 13 is a schematic diagram showing a display example of a structure. [Figure 27] FIG. 13 is a conceptual diagram illustrating an example of a case table. [Figure 28] FIG. 13 is a schematic diagram showing an example of displaying an event. [Figure 29] 13 is a flowchart showing a procedure of a process executed by an information processing device according to the fifth embodiment. [Diagram 30] FIG. 13 is a conceptual diagram illustrating an example of a scene recording table. [Diagram 31] 13 is a flowchart showing a procedure of processing executed by an information processing device according to the sixth embodiment. [Diagram 32] FIG. 23 is an explanatory diagram illustrating an estimation method in the seventh embodiment. [Diagram 33] 23 is a flowchart showing an estimation procedure in the seventh embodiment. [Diagram 34] FIG. 11 is an explanatory diagram for explaining an analysis method of the calculation results. [Diagram 35] FIG. 13 is a diagram illustrating an example of an evaluation coefficient table. [Diagram 36] FIG. 11 is a diagram showing an example of a calculation result of a score. [Figure 37] 23 is a flowchart showing a procedure of a process executed by an information processing device according to the eighth embodiment. [Figure 38] FIG. 23 is a sequence diagram showing an example of processing executed by an information processing device according to the ninth embodiment. [Figure 39] 6 is a flowchart showing a procedure of a process executed by a first calculation unit. [Diagram 40] 10 is a flowchart showing a procedure of a process executed by a second calculation unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] The present invention is specifically described below with reference to the drawings with respect to an embodiment in which the present invention is applied to a support system for laparoscopic surgery. Note that the present invention is not limited to laparoscopic surgery, but can be applied to all types of endoscopic surgery using imaging devices such as thoracoscopes, gastrointestinal endoscopes, cystoscopes, arthroscopes, robot-assisted endoscopes, surgical microscopes, and endoscopes. (Embodiment 1) FIG. 1 is a schematic diagram for explaining the schematic configuration of a surgery support system according to the first embodiment. In laparoscopic surgery, instead of performing an open abdominal surgery, a plurality of perforation instruments called trocars 10 are attached to the abdominal wall of a patient, and instruments such as a laparoscope 11, an energy treatment instrument 12, and forceps 13 are inserted into the patient's body through perforations provided in the trocars 10. The surgeon performs treatment such as resecting an affected area using the energy treatment instrument 12 while viewing an image (operative field image) of the inside of the patient's body captured by the laparoscope 11 in real time. The surgical instruments such as the laparoscope 11, the energy treatment instrument 12, and the forceps 13 are held by the surgeon or a robot. The surgeon is a medical professional involved in laparoscopic surgery, and includes the surgeon, assistant, nurse, doctor monitoring the surgery, and the like.
[0013] The laparoscope 11 includes an insertion section 11A that is inserted into the patient's body, an imaging device 11B built into the tip portion of the insertion section 11A, an operating section 11C that is provided at the rear end portion of the insertion section 11A, and a universal cord 11D for connecting to a camera control unit (CCU) 110 and a light source device 120.
[0014] The insertion section 11A of the laparoscope 11 is formed of a rigid tube. A bending section is provided at the tip of the rigid tube. The bending mechanism in the bending section is a well-known mechanism incorporated in general laparoscopes, and is configured to bend in four directions, for example, up, down, left, and right, by pulling an operating wire linked to the operation of the operating section 11C. Note that the laparoscope 11 is not limited to a flexible scope having a bending section as described above, but may be a rigid scope having no bending section, or may be an imaging device having no bending section or rigid tube.
[0015] The imaging device 11B includes a solid-state imaging element such as a complementary metal oxide semiconductor (CMOS), and a driver circuit including a timing generator (TG) and an analog signal processing circuit (AFE). The driver circuit of the imaging device 11B captures RGB color signals output from the solid-state imaging element in synchronization with a clock signal output from the TG, and performs necessary processing such as noise removal, amplification, and AD conversion in the AFE to generate image data in a digital format. The driver circuit of the imaging device 11B transmits the generated image data to the CCU 110 via the universal code 11D.
[0016] The operation unit 11C includes an angle lever, a remote switch, and the like that are operated by the surgeon. The angle lever is an operating tool that receives an operation for bending the bending portion. A bending operation knob, a joystick, and the like may be provided instead of the angle lever. The remote switch includes, for example, a changeover switch that switches the observed image between a moving image display and a still image display, and a zoom switch that enlarges or reduces the observed image. A specific function that is determined in advance may be assigned to the remote switch, or a function that is set by the surgeon may be assigned to the remote switch.
[0017] Furthermore, the operation unit 11C may have a built-in vibrator configured by a linear resonant actuator, a piezoelectric actuator, etc. When an event occurs that should be notified to the surgeon operating the laparoscope 11, the CCU 110 may vibrate the operation unit 11C by activating the vibrator built in the operation unit 11C, thereby notifying the surgeon of the occurrence of the event.
[0018] Inside the insertion section 11A, the operation section 11C, and the universal cord 11D of the laparoscope 11, there are arranged a transmission cable for transmitting a control signal output from the CCU 110 to the imaging device 11B and image data output from the imaging device 11B, a light guide for guiding illumination light emitted from the light source device 120 to the tip portion of the insertion section 11A, and the like. The illumination light emitted from the light source device 120 is guided to the tip portion of the insertion section 11A through the light guide, and is irradiated to the surgical field through an illumination lens provided at the tip portion of the insertion section 11A. In this embodiment, the light source device 120 is described as an independent device, but the light source device 120 may be configured to be built into the CCU 110.
[0019] The CCU 110 includes a control circuit for controlling the operation of the imaging device 11B included in the laparoscope 11, an image processing circuit for processing image data from the imaging device 11B input through the universal code 11D, and the like. The control circuit includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and outputs control signals to the imaging device 11B in response to the operation of various switches included in the CCU 110 and the operation of the operation unit 11C included in the laparoscope 11, and controls the start and stop of imaging, zooming, and the like. The image processing circuit includes a DSP (Digital Signal Processor), an image memory, and the like, and performs appropriate processing such as color separation, color interpolation, gain correction, white balance adjustment, and gamma correction on the image data input through the universal code 11D. The CCU 110 generates frame images for a moving image from the processed image data, and sequentially outputs each generated frame image to the information processing device 200 described later. The frame rate of the frame images is, for example, 30 FPS (Frames Per Second).
[0020] The CCU 110 may generate video data conforming to a predetermined standard such as NTSC (National Television System Committee), PAL (Phase Alternating Line), or DICOM (Digital Imaging and COmmunication in Medicine). The CCU 110 can output the generated video data to the display device 130 to display an operative field image (video) on the display screen of the display device 130 in real time. The display device 130 is a monitor equipped with a liquid crystal panel, an organic EL (Electro-Luminescence) panel, or the like. The CCU 110 may also output the generated video data to the recording device 140 and cause the recording device 140 to record the video data. The recording device 140 includes a recording device such as an HDD (Hard Disk Drive) that records the video data output from the CCU 110 together with an identifier for identifying each surgery, the date and time of surgery, the location of surgery, the patient's name, the surgeon's name, and the like.
[0021] The information processing device 200 acquires image data of the operative field image from the CCU 110, and inputs the acquired image data of the operative field image to each of the multiple learning models, thereby executing calculations using each learning model. The information processing device 200 derives an integrated recognition result for the operative field image from the calculation results using the multiple learning models, and outputs information based on the derived recognition result.
[0022] FIG. 2 is a block diagram for explaining the internal configuration of the information processing device 200. The information processing device 200 is a dedicated or general-purpose computer including a control unit 201, a storage unit 202, an operation unit 203, an input unit 204, a first calculation unit 205, a second calculation unit 206, an output unit 207, a communication unit 208, and the like. The information processing device 200 may be a computer installed in an operating room, or may be a computer installed outside the operating room. The information processing device 200 may be a server installed in a hospital where laparoscopic surgery is performed, or may be a server installed outside the hospital. The information processing device 200 is not limited to a single computer, and may be a computer system consisting of multiple computers and peripheral devices. The information processing device 200 may be a virtual machine virtually constructed by software.
[0023] The control unit 201 includes, for example, a CPU, a ROM, and a RAM. The ROM included in the control unit 201 stores a control program that controls the operation of each hardware unit included in the information processing device 200. The CPU in the control unit 201 executes the control program stored in the ROM and various computer programs stored in a storage unit 202 (described later) and controls the operation of each hardware unit, thereby causing the entire device to function as the information processing device of the present application. The RAM included in the control unit 201 temporarily stores data used during the execution of calculations, etc.
[0024] In this embodiment, the control unit 201 is configured to include a CPU, a ROM, and a RAM, but the configuration of the control unit 201 is arbitrary, and may be an arithmetic circuit or control circuit including a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), a quantum processor, a volatile or non-volatile memory, etc. Furthermore, the control unit 201 may have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until when a measurement end instruction is given, and a counter that counts numbers.
[0025] The storage unit 202 includes a storage device such as a hard disk, a flash memory, etc. The storage unit 202 stores computer programs executed by the control unit 201, various data acquired from the outside, various data generated within the device, and the like.
[0026] The computer programs stored in the storage unit 202 include a recognition processing program PG1 that causes the control unit 201 to execute a process for recognizing a recognition target included in an operative field image, and a display processing program PG2 that causes the control unit 201 to execute a process for displaying information based on a recognition result on the display device 130. The recognition processing program PG1 and the display processing program PG2 do not need to be independent computer programs, and may be implemented as one computer program. These programs are provided, for example, by a non-transitory recording medium M on which the computer programs are readably recorded. The recording medium M is a portable memory such as a CD-ROM, a USB memory, or an SD (Secure Digital) card. The control unit 201 reads a desired computer program from the recording medium M using a reading device not shown in the figure, and stores the read computer program in the storage unit 202. Alternatively, the computer program may be provided by communication. In this case, the control unit 201 may download a desired computer program through the communication unit 208 and store the downloaded computer program in the storage unit 202.
[0027] The storage unit 202 includes a first learning model 310 and a second learning model 320. The first learning model 310 is a trained learning model that has been trained to output information about a first organ included in an operative field image in response to an input of the operative field image. The first organ includes specific organs such as the esophagus, stomach, large intestine, pancreas, spleen, ureter, lungs, prostate, uterus, gallbladder, liver, and vas deferens, or non-specific organs such as connective tissue, fat, nerves, blood vessels, muscles, and membranous structures. The first organ is not limited to a specific organ, and may be any structure present in the body. Similarly, the second learning model 320 is a trained learning model that has been trained to output information about a second organ included in an operative field image in response to an input of the operative field image. The second organ includes the above-mentioned specific organ or non-specific organ. The second organ is not limited to a specific organ, and may be any structure present in the body. In the following description, when there is no need to distinguish between specific organs and non-specific organs, they will also be simply referred to as organs.
[0028] In the first embodiment, the second organ to be recognized has similar characteristics to the first organ, and an organ that may be mistaken for the first organ in the recognition process is selected. For example, when the recognition target by the first learning model 310 is loose connective tissue, the nerve tissue (or membrane structure) is selected as the recognition target by the second learning model 320. The combination of the first organ and the second organ is not limited to (a) loose connective tissue and nerve tissue (or membrane structure), but may be combinations such as (b) fat and pancreas, (c) fat to be excised and fat to be preserved, (d) two of bleeding, bleeding marks, and blood vessels, (e) two of ureters, arteries, and membrane structures, (f) stomach and intestinal tract, and (g) liver and spleen.
[0029] In the first embodiment, a case will be described in which the first organ is loose connective tissue and the second organ is nervous tissue. That is, the first learning model 310 is trained to output information about the loose connective tissue contained in the surgical field image in response to the input of the surgical field image. Such a first learning model 310 is generated by learning using an appropriate learning algorithm with a surgical field image obtained by imaging the surgical field and correct answer data indicating the first organ part (loose connective tissue part in the first embodiment) in the surgical field image as training data. The correct answer data indicating the first organ part is generated by an expert such as a doctor manually annotating the data. The same applies to the second learning model 320.
[0030] The first and second learning models 310, 320 may be generated inside the information processing device 200 or may be generated by an external server. In the latter case, the information processing device 200 may download the first and second learning models 310, 320 generated by the external server via communication, and store the downloaded first and second learning models 310, 320 in the storage unit 202. The storage unit 202 stores information on the layers included in the first and second learning models 310, 320, information on the nodes constituting each layer, and information on parameters determined by learning, such as weight coefficients and biases between nodes, as definition information of the first and second learning models 310, 320.
[0031] The operation unit 203 includes operation devices such as a keyboard, a mouse, a touch panel, a non-contact panel, a stylus pen, and a microphone for voice input. The operation unit 203 receives operations by an operator or the like, and outputs information related to the received operations to the control unit 201. The control unit 201 executes appropriate processing according to the operation information input from the operation unit 203. Note that, although the information processing device 200 is configured to include the operation unit 203 in this embodiment, it may also be configured to receive operations via various devices such as the CCU 110 connected externally.
[0032] The input unit 204 includes a connection interface for connecting an input device. In the present embodiment, the input device connected to the input unit 204 is the CCU 110. The input unit 204 receives image data of an operative field image captured by the laparoscope 11 and processed by the CCU 110. The input unit 204 outputs the input image data to the control unit 201. The control unit 201 may store the image data acquired from the input unit 204 in the storage unit 202.
[0033] In the embodiment, a configuration for acquiring image data of the operative field image from the CCU 110 through the input unit 204 will be described, but the image data of the operative field image may be acquired directly from the laparoscope 11, or may be acquired from an image processing device (not shown) detachably attached to the laparoscope 11. In addition, the information processing device 200 may acquire image data of the operative field image recorded in the recording device 140.
[0034] The first calculation unit 205 includes a processor, a memory, and the like. An example of the processor is a GPU (Graphics Processing Unit), and an example of the memory is a VRAM (Video RAM). When an operative field image is input, the first calculation unit 205 executes a calculation by the first learning model 310 using a built-in processor, and outputs the calculation result to the control unit 201. In addition, the first calculation unit 205 draws an image to be displayed on the display device 130 on the built-in memory in response to an instruction from the control unit 201, and outputs the image to the display device 130 via the output unit 207, thereby causing the display device 130 to display a desired image.
[0035] The second calculation unit 206 includes a processor, a memory, and the like, similar to the first calculation unit 205. The second calculation unit 206 may be equivalent to the first calculation unit 205, or may have a lower calculation capability than the first calculation unit 205. When an operative field image is input, the second calculation unit 206 executes a calculation by the second learning model 320 using a built-in processor and outputs the calculation result to the control unit 201. In addition, the second calculation unit 206 draws an image to be displayed on the display device 130 on a built-in memory in response to an instruction from the control unit 201, and outputs the image to the display device 130 through the output unit 207, thereby displaying a desired image on the display device 130.
[0036] The output unit 207 includes a connection interface for connecting an output device. In this embodiment, the output device connected to the output unit 207 is the display device 130. When the control unit 201 generates information to be notified to the surgeon, such as an integrated recognition result derived from the calculation results of the learning models 310 and 320, the control unit 201 outputs the generated information from the output unit 207 to the display device 130, thereby displaying the information on the display device 130. Alternatively, the output unit 207 may output the information to be notified to the surgeon by voice or sound.
[0037] The communication unit 208 includes a communication interface for transmitting and receiving various types of data. The communication interface included in the communication unit 208 is a communication interface conforming to a wired or wireless communication standard used in Ethernet (registered trademark) or WiFi (registered trademark). When data to be transmitted is input from the control unit 201, the communication unit 208 transmits the data to be transmitted to a specified destination. When the communication unit 208 receives data transmitted from an external device, the communication unit 208 outputs the received data to the control unit 201.
[0038] Next, the operative field image input to the information processing device 200 will be described. 3 is a schematic diagram showing an example of an operative field image. The operative field image in this embodiment is an image obtained by imaging the inside of a patient's abdominal cavity with the laparoscope 11. The operative field image does not need to be a raw image output by the imaging device 11B of the laparoscope 11, but may be an image (frame image) that has been processed by the CCU 110 or the like.
[0039] The surgical field imaged by the laparoscope 11 includes tissues constituting specific organs, blood vessels, nerves, etc., connective tissues present between tissues, tissues including lesions such as tumors, and tissues such as membranes and layers covering the tissues. The surgeon dissects the tissues including the lesions using surgical tools such as the energy treatment tool 12 and the forceps 13 while grasping the relationship between these anatomical structures. The surgical field image shown as an example in FIG. 3 shows a scene in which the forceps 13 is used to pull the membrane covering the organ 501, thereby applying an appropriate tension to the loose connective tissue 502, and the loose connective tissue 502 is about to be dissected using the energy treatment tool 12. The loose connective tissue 502 is a fibrous connective tissue that fills the spaces between tissues and organs, and refers to tissues that have a relatively small amount of fibers (elastic fibers) constituting the tissue. The loose connective tissue 502 is dissected as necessary when the organ 501 is expanded or when a lesion is excised. In the example of Figure 3, loose connective tissue 502 runs in the up-down direction (in the direction of the white arrow in the figure), and nerve tissue 503 runs in the left-right direction (in the direction of the black arrow in the figure) so as to intersect with loose connective tissue 502.
[0040] In general, the loose connective tissue and the nerve tissue that appear in the surgical field image are both white in color and run linearly, so that it is often difficult to distinguish them from each other by appearance. The loose connective tissue is a tissue that is dissected as necessary, but if the nerve is damaged during the traction or dissection process, functional disorders may occur after surgery. For example, damage to the hypogastric nerve in colon surgery may cause urinary disorders. Also, damage to the recurrent laryngeal nerve in esophagectomy or lung resection may cause swallowing disorders. Therefore, it would be useful for the surgeon if information regarding the recognition results of the loose connective tissue and the nerve tissue could be provided to the surgeon.
[0041] The information processing device 200 of embodiment 1 performs calculations using the first learning model 310 and calculations using the second learning model 320 for the same surgical field image, derives an integrated recognition result for the surgical field image from the results of the two calculations, and outputs information based on the derived recognition result, thereby providing the surgeon with information about loose connective tissue and nervous tissue.
[0042] The configurations of the first learning model 310 and the second learning model 320 will be described below. FIG. 4 is a schematic diagram showing a configuration example of the first learning model 310. The first learning model 310 is a learning model for performing image segmentation, and is constructed by a neural network having a convolution layer such as SegNet. The learning model 310 is not limited to SegNet, and may be constructed by using any neural network capable of performing image segmentation, such as FCN (Fully Convolutional Network), U-Net (U-Shaped Network), and PSPNet (Pyramid Scene Parsing Network). In addition, the learning model 310 may be constructed by using a neural network for object detection, such as YOLO (You Only Look Once) and SSD (Single Shot Multi-Box Detector), instead of a neural network for image segmentation.
[0043] The calculation by the first learning model 310 is executed in the first calculation unit 205. When a surgical field image is input, the first calculation unit 205 executes the calculation according to the definition information of the first learning model 310 including the learned parameters.
[0044] The first learning model 310 includes, for example, an encoder 311, a decoder 312, and a softmax layer 313. The encoder 311 is configured by alternately arranging convolutional layers and pooling layers. The convolutional layers are multi-layered into two or three layers. In the example of FIG. 4, the convolutional layers are shown without hatching, and the pooling layers are shown with hatching.
[0045] In the convolution layer, a convolution operation is performed between the input data and a filter of a predetermined size (for example, 3×3 or 5×5). That is, the input value input to the position corresponding to each element of the filter is multiplied by a weighting coefficient preset in the filter for each element, and a linear sum of the multiplied values for each element is calculated. The output in the convolution layer is obtained by adding a set bias to the calculated linear sum. The result of the convolution operation may be transformed by an activation function. For example, ReLU (Rectified Linear Unit) can be used as the activation function. The output of the convolution layer represents a feature map that extracts the features of the input data.
[0046] In the pooling layer, local statistics of the feature map output from the convolution layer, which is the upper layer connected to the input side, are calculated. Specifically, a window of a predetermined size (e.g., 2×2, 3×3) corresponding to the position of the upper layer is set, and local statistics are calculated from the input values in the window. For example, the maximum value can be used as the statistic. The size of the feature map output from the pooling layer is reduced (downsampled) according to the size of the window. The example in FIG. 4 shows that an input image of 224 pixels×224 pixels is sequentially downsampled to feature maps of 112×112, 56×56, 28×28, ..., 1×1 by sequentially repeating the calculation in the convolution layer and the calculation in the pooling layer in the encoder 311.
[0047] The output of the encoder 311 (a 1×1 feature map in the example of FIG. 4) is input to the decoder 312. The decoder 312 is configured by alternately arranging deconvolution layers and depooling layers. The deconvolution layers are multi-layered into 2 to 3 layers. In the example of FIG. 4, the deconvolution layers are shown without hatching, and the depooling layers are shown with hatching.
[0048] In the deconvolution layer, a deconvolution operation is performed on the input feature map. The deconvolution operation is an operation to restore the feature map before the convolution operation under the assumption that the input feature map is the result of a convolution operation using a specific filter. In this operation, when the specific filter is expressed as a matrix, the transpose matrix of this matrix is calculated and the input feature map is multiplied to generate an output feature map. The operation result of the deconvolution layer may be converted by an activation function such as the above-mentioned ReLU.
[0049] The inverse pooling layers of the decoder 312 correspond to the pooling layers of the encoder 311 in a one-to-one correspondence, and the corresponding pairs have substantially the same size. The inverse pooling layers re-expand (upsample) the size of the feature maps downsampled in the pooling layers of the encoder 311. The example of Fig. 4 shows that the decoder 312 sequentially upsamples feature maps to 1x1, 7x7, 14x14, ..., 224x224 by sequentially repeating the operations in the convolutional layers and the pooling layers.
[0050] The output of the decoder 312 (a 224×224 feature map in the example of FIG. 4) is input to the softmax layer 313. The softmax layer 313 outputs the probability of a label that identifies a site at each position (pixel) by applying a softmax function to the input value from the deconvolution layer connected to the input side. The first learning model 310 according to the first embodiment only needs to output the probability indicating whether or not each pixel corresponds to loose connective tissue in response to the input of the surgical field image from the softmax layer 313. The calculation result by the first learning model 310 is output to the control unit 201.
[0051] An image (recognized image) showing the recognition result of the loose connective tissue part is obtained by extracting pixels whose probability of the label output from the softmax layer 313 is equal to or higher than a threshold (for example, 60% or higher). The first calculation unit 205 may draw the recognition image of the loose connective tissue part in an internal memory (VRAM) and output it to the display device 130 via the output unit 207, thereby displaying the recognition result of the first learning model 310 on the display device 130. The recognized image is an image of the same size as the surgical field image, and is generated as an image in which a specific color is assigned to pixels recognized as loose connective tissue. The color assigned to the pixels of the loose connective tissue is preferably a color that does not exist inside the human body so that it can be distinguished from organs, blood vessels, etc. The color that does not exist inside the human body is, for example, a cool color (blue-based color) such as blue or light blue. In addition, information indicating the transparency is added to each pixel constituting the recognized image, and a non-transparent value is set to the pixels recognized as loose connective tissue, and a transparent value is set to the other pixels. When the recognition image thus generated is displayed superimposed on the surgical field image, the loose connective tissue portion can be displayed on the surgical field image as a structure having a specific color.
[0052] In the example of FIG. 4, an image of 224 pixels x 224 pixels is used as an input image to the first learning model 310, but the size of the input image is not limited to the above, and can be appropriately set according to the processing capacity of the information processing device 200, the size of the surgical field image obtained from the laparoscope 11, etc. In addition, the input image to the first learning model 310 does not need to be the entire surgical field image obtained from the laparoscope 11, but may be a partial image generated by cutting out a region of interest in the surgical field image. Since a region of interest that includes a treatment target is often located near the center of the surgical field image, for example, a partial image cut out in a rectangular shape near the center of the surgical field image to be about half the size of the original may be used. By reducing the size of the image input to the first learning model 310, the processing speed can be increased while the recognition accuracy can be improved.
[0053] 5 is a schematic diagram showing a configuration example of the second learning model 320. The second learning model 320 includes an encoder 321, a decoder 322, and a softmax layer 323, and is configured to output information about a nerve tissue portion included in an operative field image in response to an input of the operative field image. The configurations of the encoder 321, the decoder 322, and the softmax layer 323 included in the second learning model 320 are similar to those of the first learning model 310, and therefore detailed description thereof will be omitted.
[0054] The calculation by the second learning model 320 is executed by the second calculation unit 206. When an operative field image is input, the second calculation unit 206 executes the calculation according to the definition information of the second learning model 320 including the learned parameters. The second learning model 320 according to the first embodiment may output the probability indicating whether each pixel corresponds to nerve tissue or not from the softmax layer 323 in response to the input of the operative field image. The calculation result by the second learning model 320 is output to the control unit 201.
[0055] An image (recognition image) showing the recognition result of the nervous tissue part is obtained by extracting pixels whose probability of the label output from the softmax layer 323 is equal to or higher than a threshold (e.g., 60% or higher). The second calculation unit 206 may render the recognition image of the nervous tissue part in an internal memory (VRAM) and output it to the display device 130 via the output unit 207, thereby displaying the recognition result of the second learning model 320 on the display device 130. The configuration of the recognition image showing the nervous tissue is similar to that of the loose connective tissue, but the color assigned to the pixels of the nervous tissue is preferably a color (e.g., a greenish or yellowish color) that can be distinguished from the loose connective tissue.
[0056] The operation of the information processing device 200 will now be described. 6 is a flowchart showing the procedure of processing executed by the information processing device 200 according to the first embodiment. The control unit 201 of the information processing device 200 reads out and executes the recognition processing program PG1 and the display processing program PG2 from the storage unit 202, thereby performing processing according to the following procedure. When laparoscopic surgery is started, an operative field image obtained by capturing an image of the operative field by the imaging device 11B of the laparoscope 11 is output to the CCU 110 via the universal code 11D as needed. The control unit 201 of the information processing device 200 acquires the operative field image in frame units output from the CCU 110 from the input unit 204 (step S101). The control unit 201 executes the following processing every time it acquires an operative field image in frame units.
[0057] The control unit 201 sends the operative field images acquired in frame units through the input unit 204 to the first calculation unit 205 and the second calculation unit 206, and instructs the first calculation unit 205 and the second calculation unit 206 to start calculation (step S102).
[0058] When the first calculation unit 205 receives a calculation start instruction from the control unit 201, the first calculation unit 205 executes calculation by the first learning model 310 (step S103). That is, the first calculation unit 205 executes calculation by the encoder 311 that generates a feature map from the input operative field image and sequentially downsamples the generated feature map, calculation by the decoder 312 that sequentially upsamples the feature map input from the encoder 311, and calculation by the softmax layer 313 that identifies each pixel of the feature map finally obtained by the decoder 312. The first calculation unit 205 outputs the calculation result by the learning model 310 to the control unit 201 (step S104).
[0059] When the second calculation unit 206 receives an instruction to start calculation from the control unit 201, the second calculation unit 206 executes calculation by the second learning model 320 (step S105). That is, the second calculation unit 206 executes calculation by the encoder 321 that generates a feature map from the input operative field image and sequentially downsamples the generated feature map, calculation by the decoder 322 that sequentially upsamples the feature map input from the encoder 321, and calculation by the softmax layer 323 that identifies each pixel of the feature map finally obtained by the decoder 322. The second calculation unit 206 outputs the calculation result by the learning model 320 to the control unit 201 (step S106).
[0060] In the flowchart of FIG. 6, for convenience, the procedure is such that the calculation by the first calculation unit 205 is performed first, and then the calculation by the second calculation unit 206 is performed. However, it is preferable that the calculation by the first calculation unit 205 and the calculation by the second calculation unit 206 are performed simultaneously in parallel.
[0061] The control unit 201 derives an integrated recognition result for the surgical field image based on the calculation result by the first learning model 310 and the calculation result by the second learning model 320. Specifically, the control unit 201 executes the following processing.
[0062] The control unit 201 executes a process of recognizing loose connective tissue by referring to the calculation result by the first learning model 310 (step S107). The control unit 201 can recognize the loose connective tissue included in the surgical field image by extracting pixels whose probability of the label output from the softmax layer 313 of the first learning model 310 is equal to or higher than a threshold (e.g., 60% or higher).
[0063] The control unit 201 executes a neural tissue recognition process by referring to the calculation result by the second learning model 320 (step S108). The control unit 201 can recognize the neural tissue included in the surgical field image by extracting pixels whose labels output from the softmax layer 323 of the second learning model 320 have a probability of a threshold or more (e.g., 60% or more).
[0064] The control unit 201 judges whether the recognition result of the loose connective tissue and the recognition result of the nervous tissue overlap (step S109). In this step, it is confirmed whether a specific structure included in the operative field image is recognized as loose connective tissue on the one hand and as nervous tissue on the other hand. Specifically, the control unit 201 judges that the recognition results overlap when one pixel in the operative field image is recognized as loose connective tissue on the one hand and as nervous tissue on the other hand. In addition, the control unit 201 may compare an area in the operative field image recognized as loose connective tissue with an area in the operative field image recognized as nervous tissue, and judge the recognition results to overlap. For example, if the overlap between the two is a predetermined ratio or more (e.g., 40% or more) in area ratio, it may be judged that the recognition results overlap, and if it is less than the predetermined ratio, it may be judged that the recognition results do not overlap.
[0065] When it is determined that the recognition results are not overlapped (S109: NO), the control unit 201 outputs the recognition results of the loose connective tissue and the recognition results of the nervous tissue (step S110). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the loose connective tissue on the surgical field image, and instructs the second calculation unit 206 to superimpose the recognized image of the nervous tissue on the surgical field image. In response to the instruction from the control unit 201, the first calculation unit 205 and the second calculation unit 206 respectively draw the recognized images of the loose connective tissue and the nervous tissue in the built-in VRAM and output them to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized images of the loose connective tissue and the nervous tissue on the surgical field image.
[0066] 7 is a schematic diagram showing a display example of the recognition result in the first embodiment. In the display example of FIG. 7, for convenience of drawing, the part recognized as the loose connective tissue 502 is shown as a hatched area, and the part recognized as the nerve tissue 503 is shown as a different hatched area. In reality, the pixels recognized as the loose connective tissue 502 are displayed in, for example, a blue color, and the pixels recognized as the nerve tissue 503 are displayed in, for example, a green color. By viewing the recognition result displayed on the display device 130, the surgeon can distinguish and recognize the loose connective tissue 502 and the nerve tissue 503, and can, for example, use the energy treatment tool 12 to peel off the loose connective tissue 502 while grasping the presence of the nerve tissue 503 that must not be damaged.
[0067] In this embodiment, both the loose connective tissue recognized based on the calculation result of the first calculation unit 205 and the nerve tissue recognized based on the calculation result of the second calculation unit 206 are displayed, but only one of them may be displayed. Furthermore, the tissue to be displayed may be selected by the surgeon or may be switched by the surgeon's operation.
[0068] In step S109 of the flowchart shown in FIG. 6, when it is determined that the recognition results are duplicated (S109: YES), the control unit 201 outputs warning information indicating that a similar structure has been recognized (step S111).
[0069] FIG. 8 is a schematic diagram showing a display example of warning information in the first embodiment. FIG. 8 shows a display example of warning information in a case where a structure 504 included in an operative field image is recognized as loose connective tissue based on the calculation result of the first calculation unit 205 and as nervous tissue based on the calculation result of the second calculation unit 206. The first learning model 310 used by the first calculation unit 205 is trained to output information about loose connective tissue in response to an input of an operative field image, and the second learning model 320 used by the second calculation unit 206 is trained to output information about nervous tissue in response to an input of an operative field image. However, since loose connective tissue and nervous tissue have similar characteristics in appearance, when the above two learning models 310 and 320 are used to perform recognition processing independently, there is a possibility that the recognition results will overlap. When the recognition results overlap, if only one of the recognition results is presented to the surgeon, there is a possibility that the surgeon will mistakenly recognize the actual loose connective tissue as nervous tissue, or the surgeon will mistakenly recognize the actual nervous tissue as loose connective tissue. When the recognition results overlap, the control unit 201 can prompt the operator to check by displaying warning information as shown in FIG.
[0070] 8, the control unit 201 is configured to display text information indicating a warning by superimposing it on the operative field image, but the text information indicating a warning may be displayed outside the display area of the operative field image, or may be displayed on another display device (not shown). Instead of displaying text information indicating a warning, the control unit 201 may display a figure indicating a warning, or issue a warning by outputting voice or sound.
[0071] (Variation 1-1) The control unit 201 of the information processing device 200 may stop outputting information based on the recognition results when the recognition results of loose connective tissue by the first learning model 310 overlap with the recognition results of nervous tissue by the second learning model 320.
[0072] (Variation 1-2) When the recognition result of the loose connective tissue by the first learning model 310 and the recognition result of the nervous tissue by the second learning model 320 overlap, the control unit 201 of the information processing device 200 may select the recognition result with the higher confidence level and output information based on the selected recognition result. The confidence level of the recognition result by the first learning model 310 is calculated based on the probability output from the softmax layer 313. For example, the control unit 201 may calculate the confidence level by averaging the probability values for each pixel recognized as loose connective tissue. The same applies to the confidence level of the recognition result by the second learning model 320. For example, when the structure 504 shown in FIG. 8 is recognized as loose connective tissue with a confidence level of 95% by the first learning model 310 and the same structure 504 is recognized as nervous tissue with a confidence level of 62% by the second learning model 320, the control unit 201 may present the recognition result that the structure 504 is loose connective tissue to the operator.
[0073] (Variation 1-3) When the recognition result of the loose connective tissue by the first learning model 310 and the recognition result of the nervous tissue by the second learning model 320 overlap, the control unit 201 of the information processing device 200 may derive a recognition result according to the confidence level and display the recognition result in a display mode according to the confidence level. Fig. 9 is a schematic diagram showing a display example of the recognition result according to the confidence level. For example, when the structure 505 included in the surgical field image is recognized as loose connective tissue with a confidence level of 95% by the first learning model 310, and the same structure 505 is not recognized as nervous tissue by the second learning model 320, the control unit 201 presents the structure 505 to the surgeon by coloring it, for example, in a blue color (black in the drawing). Similarly, when a structure 506 included in the surgical field image is recognized by the second learning model 320 as nervous tissue with a certainty of 90%, and the same structure 506 is recognized by the first learning model 310 as loose connective tissue but not as loose connective tissue, the control unit 201 colors the structure 506, for example, in a greenish color (white in the drawing) and presents it to the surgeon. On the other hand, when a structure 507 included in the surgical field image is recognized by the first learning model 310 as loose connective tissue with a certainty of 60%, and the same structure 507 is recognized by the second learning model 320 as nervous tissue with a certainty of 60%, the control unit 201 colors the structure 507, for example, in an intermediate color between blue and green (gray in the drawing) and presents it to the surgeon. Instead of changing the color according to the certainty, a configuration of changing the saturation, transparency, etc. may be adopted.
[0074] (Variation 1-4) When a structure recognized as loose connective tissue and a structure recognized as nervous tissue are present within the surgical field image, the control unit 201 of the information processing device 200 may derive information such as the appropriate positional relationship between the two structures, the distance to a feature point, the distance to other structures, and the area of the other structures from the relationship between the two structures.
[0075] As described above, in the first embodiment, it is possible to obtain an integrated recognition result for organs included in the surgical field based on the calculation result by the first learning model 310 and the calculation result by the second learning model 320. When the recognition results for similar structures overlap, the information processing device 200 issues a warning or stops outputting information, thereby making it possible to avoid presenting the surgeon with a result that may be an erroneous recognition.
[0076] (Embodiment 2) In the second embodiment, a configuration will be described in which organ recognition and event recognition are combined to derive an integrated recognition result.
[0077] The information processing device 200 according to the second embodiment includes a first learning model 310 for recognizing an organ and a third learning model 330 for recognizing an event. The organ recognized by the first learning model 310 is not limited to loose connective tissue, but may be any organ that is set in advance. The event recognized by the third learning model 330 is an event such as bleeding, damage, and pulsation. Other configurations of the information processing device 200 are the same as those of the first embodiment, and therefore description thereof will be omitted.
[0078] FIG. 10 is a schematic diagram showing a configuration example of the third learning model 330. The third learning model 330 includes an encoder 331, a decoder 332, and a softmax layer 333, and is configured to output information about an event occurring in an operative field image in response to an input of the operative field image. The event information output by the third learning model 330 is information about events such as bleeding, damage (burn marks caused by the energy treatment tool 12), and pulsation. The third learning model 330 is not limited to a learning model for image segmentation or object detection, and may be a learning model using CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks), LSTM (Long Short Term Memory), GAN (Generative Adversarial Network), etc.
[0079] The calculation by the third learning model 330 is executed by the second calculation unit 206. When an operative field image is input, the second calculation unit 206 executes the calculation according to the definition information of the third learning model 330 including the learned parameters. The third learning model 330 may output a probability indicating whether or not an event has occurred for the input of the operative field image from the softmax layer 333. The calculation result by the third learning model 330 is output to the control unit 201. When the probability of the label output from the softmax layer 333 is equal to or greater than a threshold (e.g., 60% or greater), the control unit 201 determines that an event has occurred in the operative field image. The control unit 201 may determine whether or not an event has occurred on a pixel-by-pixel basis of the operative field image, or may determine whether or not an event has occurred on a per-operative field image basis.
[0080] 11 is a flowchart showing the procedure of processing executed by the information processing device 200 according to the second embodiment. Every time the information processing device 200 acquires an operative field image, the information processing device 200 executes the procedure of steps S201 to S206 in the same manner as in the first embodiment. The control unit 201 of the information processing device 200 acquires the calculation result by the first learning model 310 and the calculation result by the third learning model 330, and derives an integrated recognition result for the operative field image based on these calculation results. Specifically, the control unit 201 executes the following processing.
[0081] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S207). The control unit 201 can recognize the organ included in the surgical field image by extracting pixels whose probability of the label output from the softmax layer 313 of the first learning model 310 is equal to or higher than a threshold (e.g., 60% or higher).
[0082] The control unit 201 executes an event recognition process by referring to the calculation result by the third learning model 330 (step S208). The control unit 201 can determine whether an event has occurred for each pixel by extracting pixels whose probability of the label output from the softmax layer 333 of the third learning model 330 is equal to or higher than a threshold (e.g., 60% or higher).
[0083] The control unit 201 judges whether an event has occurred in the organ recognized in step S207 (step S209). The control unit 201 compares the pixel recognized as the organ in step S207 with the pixel recognized as the event occurring in step S208, and if the two match, it judges that an event has occurred in the recognized organ.
[0084] When it is determined that no event has occurred in the recognized organ (S209: NO), the control unit 201 ends the process according to this flowchart. Note that the control unit 201 may display the recognition result of the organ separately without associating it with an event, or may display the recognition result of the event separately without associating it with the organ.
[0085] When it is determined that an event has occurred in the recognized organ (S209: YES), the control unit 201 outputs information about the organ in which the event has occurred (step S210). For example, the control unit 201 may display the name of the organ in which the event has occurred as text information superimposed on the operative field image. Instead of displaying the name superimposed on the operative field image, the name of the organ in which the event has occurred may be displayed outside the operative field image, or may be output by sound or voice.
[0086] Fig. 12 is a schematic diagram showing a display example of the recognition result in the second embodiment. The display example in Fig. 12 shows an example in which text information indicating that bleeding has occurred from the surface of the stomach is superimposed on an operative field image. The control unit 201 may display information of an organ damaged by the energy treatment tool 12 or the like, not limited to the organ from which bleeding has occurred, and may display information of an organ in which pulsation is occurring. When displaying information of an organ such as a blood vessel in which pulsation is occurring, for example, the organ may be displayed in a blinking manner in synchronization with the pulsation. The synchronization does not necessarily have to be perfectly consistent with the pulsation, and a periodic display close to the pulsation may be used.
[0087] When the control unit 201 recognizes bleeding from a specific organ (for example, an important blood vessel), it may acquire data on the patient's vital signs (pulse, blood pressure, respiration, body temperature) that are constantly detected by a sensor not shown in the figure, as needed, and display the acquired data on the display device 130. Furthermore, when the control unit 201 recognizes bleeding from a specific organ (for example, an important blood vessel), it may notify an external device via the communication unit 208. The external device to which the notification is sent may be a terminal carried by an anesthesiologist, or may be an in-hospital server that supervises and manages events within the hospital.
[0088] Furthermore, when the control unit 201 recognizes bleeding from an organ, the control unit 201 may change the threshold value used in organ recognition or stop organ recognition. Furthermore, when the control unit 201 recognizes bleeding from an organ, organ recognition may be difficult, so the control unit 201 may switch to a learning model (not shown) improved for bleeding and continue organ recognition using this learning model.
[0089] Furthermore, when there is a risk of anemia due to bleeding, the control unit 201 may automatically estimate the amount of bleeding or the bleeding rate and suggest a blood transfusion. The control unit 201 can estimate the amount of bleeding by calculating the bleeding area on the image, and can estimate the bleeding rate by calculating the change over time in the bleeding area.
[0090] As described above, in the second embodiment, an integrated recognition result obtained by combining the learning model 310 for organ recognition and the learning model 320 for event recognition can be presented to the surgeon.
[0091] (Embodiment 3) In the third embodiment, a configuration will be described in which organ recognition and device recognition are combined to derive an integrated recognition result.
[0092] The information processing device 200 according to the third embodiment includes a first learning model 310 for recognizing an organ and a fourth learning model 340 for recognizing a device. The organ recognized by the first learning model 310 is not limited to loose connective tissue, but may be any organ that has been set in advance. The device recognized by the fourth learning model 340 is a surgical tool used during surgery, such as an energy treatment tool 12 and forceps 13. Other configurations of the information processing device 200 are the same as those of the first embodiment, and therefore description thereof will be omitted.
[0093] FIG. 13 is a schematic diagram showing a configuration example of the fourth learning model 340. The fourth learning model 340 includes an encoder 341, a decoder 342, and a softmax layer 343, and is configured to output information about devices included in an operative field image in response to an input of an operative field image. The device information output by the fourth learning model 340 is information about surgical tools used during surgery, such as the energy treatment tool 12 and the forceps 13. The fourth learning model 340 is not limited to a learning model for image segmentation or object detection, and may be a learning model based on CNN, RNN, LSTM, GAN, or the like. A plurality of fourth learning models 340 may be prepared according to the type of device.
[0094] The calculation by the fourth learning model 340 is executed by the second calculation unit 206. When a surgical field image is input, the second calculation unit 206 executes the calculation according to the definition information of the fourth learning model 340 including the learned parameters. The fourth learning model 340 may output a probability indicating whether each pixel corresponds to a specific device or not from the softmax layer 343 in response to the input of the surgical field image. The calculation result by the fourth learning model 340 is output to the control unit 201. When the probability of the label output from the softmax layer 343 is equal to or higher than a threshold (e.g., 60% or higher), the control unit 201 determines that a specific device included in the surgical field image has been recognized.
[0095] 14 is a flowchart showing the procedure of processing executed by the information processing device 200 according to the third embodiment. The information processing device 200 executes the procedure of steps S301 to S306 in the same manner as in the first embodiment every time it acquires an operative field image. The control unit 201 of the information processing device 200 acquires the calculation result by the first learning model 310 and the calculation result by the fourth learning model 340, and derives an integrated recognition result for the operative field image based on these calculation results. Specifically, the control unit 201 executes the following processing. It is assumed that the storage unit 202 stores information on organs and devices recognized in the most recent operative field image (for example, one frame before).
[0096] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S307). The control unit 201 can recognize the organ included in the surgical field image by extracting pixels whose probability of the label output from the softmax layer 313 of the first learning model 310 is equal to or higher than a threshold (e.g., 60% or higher).
[0097] The control unit 201 executes device recognition processing by referring to the calculation result by the fourth learning model 340 (step S308). The control unit 201 can recognize devices included in the surgical field image by extracting pixels whose labels output from the softmax layer 343 of the fourth learning model 340 have a probability of a threshold or more (e.g., 60% or more).
[0098] The control unit 201 judges whether the device is moving on the organ recognized in step S307 (step S309). The control unit 201 reads out information on the organ and device recognized from the most recent operative field image from the storage unit 202, compares the read out information on the organ and device with the newly recognized information on the organ and device, and detects a change in the relative position, thereby being able to judge whether the device is moving on the organ.
[0099] When it is determined that the device has not moved on the organ (S309: NO), the control unit 201 ends the process according to this flowchart. Note that the control unit 201 may display the recognition result of the organ independently of the device, or may display the recognition result of the device independently of the organ.
[0100] When it is determined that the device is moving on the organ (S309: YES), the control unit 201 generates display data in which the display mode of the organ is changed, and outputs the display data to the display device 130 through the output unit 207 (step S310). The control unit 201 may change the display mode by changing the display color, saturation, or transparency of the organ, or by blinking the organ. The organ with the changed display mode is displayed on the display device 130. When the device is moving on the organ, there is a possibility that the information processing device 200 may erroneously recognize the organ. Therefore, by changing and displaying the display mode, the surgeon can be prompted to make a decision. The control unit 201 may instruct the first calculation unit 205 to change the display mode, and the display mode may be changed by the processing of the first calculation unit 205.
[0101] (Variation 3-1) The control unit 201 of the information processing device 200 may stop the organ recognition process when it determines that the device is moving (not stopped) on the organ. When the organ recognition process is stopped, the control unit 201 may continuously perform the device recognition process and resume the organ recognition process when it determines that the device has stopped.
[0102] (Variation 3-2) The control unit 201 of the information processing device 200 may stop the output process of the organ recognition result when it determines that the device is moving (not stopped) on the organ. In this case, the calculation by the first learning model 310 and the fourth learning model 340, the organ recognition process based on the calculation result of the first learning model 310, and the device recognition process based on the calculation result of the fourth learning model 340 are continuously executed, and the display of the recognition image showing the organ recognition result is stopped. The control unit 201 may resume the output process at the timing when it determines that the device has stopped.
[0103] (Variation 3-3) The control unit 201 of the information processing device 200 may derive information on the organ being processed by the device based on the organ recognition process and the device recognition process. The control unit 201 can derive information on the organ being processed by the device by comparing the pixels recognized as the organ in step S307 with the pixels recognized as the device in step S308. The control unit 201 may output information on the organ being processed by the device and display it on the display device 130, for example.
[0104] (Variation 3-4) The control unit 201 of the information processing device 200 may acquire dimensional information of the recognized device, and derive dimensional information of the recognized organ based on the acquired dimensional information of the device.
[0105] 15 is a flowchart showing the procedure for deriving dimensional information. The control unit 201 acquires dimensional information of the recognized device (step S321). The dimensional information of the device may be stored in advance in the storage unit 202 of the information processing device 200, or may be stored in an external device. In the former case, the control unit 201 may acquire the dimensional information by reading out desired information from the storage unit 202, and in the latter case, the control unit 201 may acquire the dimensional information by accessing the external device. Note that the dimensional information does not need to be the dimensions of the entire device, and may be the dimensions of a part of the device (for example, the cutting edge).
[0106] The control unit 201 calculates the ratio between the dimensions of the image information of the device portion indicated by the acquired dimension information and the dimensions of the recognized organ on the image (step S322).
[0107] The control unit 201 calculates the size of the organ based on the device size information acquired in step S321 and the size ratio calculated in step S322 (step S323). The control unit 201 may output the calculated organ size information and display it on the display device 130, for example.
[0108] (Variation 3-5) The control unit 201 of the information processing device 200 may derive information on an organ damaged by a device based on the organ recognition process and the device recognition process. For example, when the control unit 201 recognizes a device on an organ and determines that a part of the organ has discolored, it determines that damage to the organ caused by a device has been detected. The device on the organ is recognized by a procedure similar to that shown in the flowchart of FIG. 14. Discoloration of the organ is recognized by a change in pixel value over time. When the control unit 201 detects organ damage caused by a device, it outputs information indicating that fact and causes the display device 130 to display the information, for example.
[0109] (Variation 3-6) The control unit 201 of the information processing device 200 may derive information indicating whether or not the device used for the organ is appropriate based on the organ recognition process and the device recognition process. The storage unit 202 of the information processing device 200 has a definition table that defines the relationship between the type of organ and the device that can be used for each organ (or the device that should not be used). For example, this definition table defines that sharp forceps should not be used for the intestinal tract. The control unit 201 recognizes the organ and the device from the surgical field image, and judges whether or not the device used for the organ is appropriate by referring to the above-mentioned definition table. If it is judged to be inappropriate (for example, when sharp forceps are used for the intestinal tract), the control unit 201 outputs information that an incorrect device is being used, and causes the display device 130 to display it, for example. The control unit 201 may also issue a warning by voice or alarm sound.
[0110] (Variation 3-7) When the control unit 201 of the information processing device 200 recognizes a device, the control unit 201 may output operation support information for the device and display it, for example, on the display device 130. The operation support information for the device is a user manual for the device, and may be stored in the storage unit 202 of the information processing device 200 or an external device.
[0111] As described above, in the third embodiment, an integrated recognition result obtained by combining the learning model 310 for organ recognition and the learning model 340 for device recognition can be presented to the surgeon.
[0112] (Embodiment 4) In the fourth embodiment, a configuration will be described in which organ recognition and scene recognition are combined to derive an integrated recognition result.
[0113] The information processing device 200 according to the fourth embodiment includes a first learning model 310 for recognizing an organ and a fifth learning model 350 for recognizing a scene. The organ recognized by the first learning model 310 is not limited to loose connective tissue, but may be any pre-set organ. The first learning model 310 may be prepared for each type of organ so as to be compatible with various organs. The scene recognized by the fifth learning model 350 is, for example, a characteristic scene showing a characteristic scene of a surgery. Other configurations of the information processing device 200 are the same as those of the first embodiment, and therefore description thereof will be omitted.
[0114] FIG. 16 is a schematic diagram showing a configuration example of the fifth learning model 350. The fifth learning model 350 includes an input layer 351, an intermediate layer 352, and an output layer 353, and is configured to output information about a scene shown by an operative field image in response to an input of the operative field image. The scene information output by the fifth learning model 350 includes information such as the probability that the scene includes a specific organ such as a blood vessel, an important nerve, or a specific organ (such as a ureter or a spleen), the probability that the scene is a scene in which a characteristic procedure is performed in a specific surgery such as a blood vessel dissection or lymph node dissection, and the probability that the scene is a scene in which a characteristic operation (such as blood vessel ligation, intestinal dissection, or anastomosis) is performed using a specific surgical device (such as a blood vessel clip or an automatic anastomosis device). The fifth learning model 350 is constructed by, for example, a CNN. Alternatively, the fifth learning model 350 may be a learning model constructed by an RNN, LSTM, GAN, or the like, and may be a learning model for image segmentation or object detection.
[0115] The calculation by the fifth learning model 350 is executed by the second calculation unit 206. When an operative field image is input, the second calculation unit 206 executes calculation according to the definition information of the fifth learning model 350 including the learned parameters. The fifth learning model 350 outputs the probability that the input operative field image corresponds to a specific scene from each node constituting the output layer 353. The calculation result by the fifth learning model 350 is output to the control unit 201. The control unit 201 performs scene recognition by selecting the scene with the highest probability from the probabilities of each scene output from the output layer 353.
[0116] 17 is a flowchart showing the procedure of processing executed by the information processing device 200 according to embodiment 4. The control unit 201 of the information processing device 200 acquires a frame-by-frame operative field image output from the CCU 110 through the input unit 204 (step S401). The control unit 201 executes the following processing every time it acquires a frame-by-frame operative field image.
[0117] The control unit 201 sends the operative field image in frame units acquired through the input unit 204 to the first calculation unit 205 and the second calculation unit 206, and instructs the second calculation unit 206 to start calculation (step S402).
[0118] When the second calculation unit 206 receives an instruction to start calculation from the control unit 201, it executes calculation by the fifth learning model 350 (step S403). That is, the first calculation unit 205 executes each calculation in the input layer 351, the intermediate layer 352, and the output layer 353 constituting the fifth learning model 350, and outputs the probability that each node of the output layer 353 corresponds to a specific scene. The second calculation unit 206 outputs the calculation result by the fifth learning model 350 to the control unit 201 (step S404).
[0119] The control unit 201 executes scene recognition processing based on the calculation result by the second calculation unit 206 (step S405). That is, the control unit 201 selects the scene with the highest probability among the probabilities of each scene output from the output layer 353, thereby identifying the scene shown by the current operative field image.
[0120] The control unit 201 selects a learning model for organ recognition according to the identified scene (step S406). For example, when the scene recognized in step S405 includes a ureter, the control unit 201 selects a learning model for ureter recognition. When the scene recognized in step S405 is a lymph node dissection scene at the upper edge of the pancreas in a gastric cancer surgery, the control unit 201 selects a learning model for lymph node recognition, a learning model for pancreas recognition, a learning model for stomach recognition, or the like. When the scene recognized in step S405 is a scene of ligation using a vascular clip, the control unit 201 selects a learning model for blood vessel recognition, a learning model for a surgical device, or the like. The control unit 201 can select a learning model for organ recognition according to the identified scene, not limited to a ureter, lymph node, pancreas, or stomach. Hereinafter, the learning model for organ recognition selected in step S406 is referred to as a first learning model 310. The control unit 201 gives the first calculation unit 205 an instruction to start calculation together with information on the selected first learning model 310.
[0121] When the first calculation unit 205 receives a calculation start instruction from the control unit 201, the first calculation unit 205 executes calculation by the first learning model 310 (step S407). That is, the first calculation unit 205 executes calculation by the encoder 311 that generates a feature map from the input operative field image and sequentially downsamples the generated feature map, calculation by the decoder 312 that sequentially upsamples the feature map input from the encoder 311, and calculation by the softmax layer 313 that identifies each pixel of the feature map finally obtained by the decoder 312. The first calculation unit 205 outputs the calculation result by the learning model 310 to the control unit 201 (step S408).
[0122] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S409). The control unit 201 can recognize the organ included in the surgical field image by extracting pixels whose labels output from the softmax layer 313 of the first learning model 310 have a probability of a threshold or more (e.g., 60% or more).
[0123] The control unit 201 outputs the recognition result of the organ (step S410). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the organ on the surgical field image. In response to the instruction from the control unit 201, the first calculation unit 205 renders the recognized image of the organ in a built-in VRAM and outputs it to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized image of the organ on the surgical field image. In addition, an instruction to end the calculation of the first learning model or start a different learning model may be given using a learning model that recognizes that a specific scene has ended.
[0124] (Variation 4-1) When the information processing device 200 is configured to perform recognition processing of a specific organ (i.e., when it has only one first learning model 310), it may be configured not to perform organ recognition processing until it recognizes a scene that includes this specific organ.
[0125] Fig. 18 is a flowchart showing a processing procedure in Modification 4-1. The control unit 201 of the information processing device 200 executes a scene recognition process every time an operative field image is acquired in the same procedure as that shown in Fig. 17 (steps S421 to S425).
[0126] The control unit 201 determines whether or not a specific scene has been recognized by the scene recognition process (step S426). The control unit 201 may determine whether or not the scene is a preset scene, such as a scene including a ureter, a scene of lymph node dissection, or a scene of ligation using a vascular clip. If it is determined that a specific scene has not been recognized (S426: NO), the control unit 201 ends the processing according to this flowchart. On the other hand, if it is determined that a specific scene has been recognized (S426: YES), the control unit 201 gives an instruction to the first calculation unit 205 to start calculation.
[0127] When the first calculation unit 205 receives an instruction to start calculation from the control unit 201, the first calculation unit 205 executes calculation using the first learning model 310 (step S427), and outputs the calculation result using the learning model 310 to the control unit 201 (step S428).
[0128] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S429). The control unit 201 can recognize the organ included in the surgical field image by extracting pixels whose probability of the label output from the softmax layer 313 of the first learning model 310 is equal to or higher than a threshold (e.g., 60% or higher).
[0129] The control unit 201 outputs the recognition result of the organ (step S430). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the organ on the operative field image. In response to the instruction from the control unit 201, the first calculation unit 205 renders the recognized image of the organ in a built-in VRAM and outputs it to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized image of the organ on the operative field image.
[0130] (Variation 4-2) The control unit 201 of the information processing device 200 may acquire prior information according to the recognition result of the scene recognition process, and may perform the organ recognition process by referring to the acquired prior information. FIG. 19 is a conceptual diagram showing an example of a prior information table. In the prior information table, prior information is registered according to the scene of the surgery. When lymph node dissection is performed on the upper edge of the pancreas in a gastric cancer surgery, the pancreas is located under the lymph node. In the prior information table, for example, prior information that the pancreas is located under the lymph node with respect to the scene of lymph node dissection on the upper edge of the pancreas in a gastric cancer surgery is registered. In the prior information table, various prior information corresponding to various scenes is registered, not limited to the information shown in FIG. 19. The prior information table may be prepared in the storage unit 202 of the information processing device 200, or may be prepared in an external device.
[0131] Fig. 20 is a flowchart showing a processing procedure in Modification 4-2. The control unit 201 of the information processing device 200 executes a scene recognition process every time an operative field image is acquired in the same procedure as that shown in Fig. 17 (steps S441 to S445).
[0132] The control unit 201 accesses the storage unit 202 or an external device, and acquires prior information according to the recognized scene (step S446). After acquiring the prior information, the control unit 201 gives an instruction to the first calculation unit 205 to start calculation.
[0133] When the first calculation unit 205 receives an instruction to start calculation from the control unit 201, the first calculation unit 205 executes calculation using the first learning model 310 (step S447) and outputs the calculation result using the first learning model 310 to the control unit 201 (step S448).
[0134] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S449). The control unit 201 can recognize an organ included in the surgical field image by extracting pixels whose probability of the label output from the softmax layer 313 of the first learning model 310 is equal to or higher than a threshold (e.g., 60% or higher).
[0135] The control unit 201 judges whether the organ recognition result is consistent with the prior information (step S450). For example, in a scene of lymph node dissection at the upper edge of the pancreas in a stomach cancer operation, even though prior information that the pancreas is located below the lymph node is obtained, if the pancreas located above the lymph node is recognized, the control unit 201 can judge that the organ recognition result is not consistent with the prior information.
[0136] When it is determined that the organ recognition result and the prior information do not match (S450: NO), the control unit 201 outputs warning information (step S451). The control unit 201 outputs text information indicating that the organ recognition result and the prior information do not match from the output unit 207, and causes the output unit 207 to superimpose the text information on the display area of the operative field image. The control unit 201 may display the text information indicating the warning outside the display area of the operative field image, or may display the text information indicating the warning on another display device (not shown). Instead of displaying text information indicating the warning, the control unit 201 may display a graphic indicating the warning, or issue a warning by outputting voice or sound.
[0137] In this embodiment, when the organ recognition result and the prior information are not consistent, warning information is output, but when the two are not consistent, there is a possibility of erroneous recognition, so the output process of the organ recognition result may be stopped. In this case, the calculation by the first learning model 310 and the fifth learning model 350, the organ recognition process based on the calculation result of the first learning model 310, and the scene recognition process based on the calculation result of the fifth learning model 350 are continuously executed, but the display of the recognition image showing the organ recognition result is stopped. Moreover, the control unit 201 may be configured to stop the organ recognition process instead of stopping the output process.
[0138] When it is determined that the organ recognition process matches the prior information (S450: YES), the control unit 201 outputs the organ recognition result (step S452). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the organ on the operative field image. In response to the instruction from the control unit 201, the first calculation unit 205 renders the recognized image of the organ in a built-in VRAM and outputs it to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized image of the organ on the operative field image.
[0139] In the flowchart of Fig. 20, the prior information is referred to after the organ recognition process is performed, but the prior information may be referred to when the organ recognition process is performed. For example, when the prior information that the pancreas is located below the lymph nodes is obtained, the control unit 201 may generate mask information to exclude the upper part of the lymph nodes from the recognition target, and may give an instruction to start the calculation to the first calculation unit 205 together with the generated mask information. The first calculation unit 205 may apply a mask to the surgical field image, and perform calculation by the first learning model 310 based on the partial image other than the masked area.
[0140] (Variation 4-3) The control unit 201 of the information processing device 200 may change the threshold value used for organ recognition depending on the recognized scene. Fig. 21 is a flowchart showing a processing procedure in Modification 4-3. The control unit 201 of the information processing device 200 executes scene recognition processing every time an operative field image is acquired in the same procedure as that shown in Fig. 17 (steps S461 to S465).
[0141] The control unit 201 determines whether or not a specific scene has been recognized by the scene recognition process (step S466). The control unit 201 may determine whether or not the scene is a preset scene, such as a scene including a ureter or a scene of lymph node dissection.
[0142] When it is determined that a specific scene has been recognized (S466: YES), the control unit 201 sets the threshold value used for organ recognition to a relatively low first threshold value (<second threshold value) (step S467). That is, the control unit 201 sets the threshold value so that the organ to be recognized is easily detected. After setting the threshold value, the control unit 201 gives an instruction to start calculation to the first calculation unit 205.
[0143] On the other hand, when it is determined that a specific scene has not been recognized (S466: NO), the control unit 201 sets the threshold value used for organ recognition to a relatively high second threshold value (>first threshold value) (step S468). That is, the control unit 201 sets the threshold value so that the organ to be recognized is difficult to detect. After setting the threshold value, the control unit 201 gives an instruction to start calculation to the second calculation unit 206.
[0144] When the first calculation unit 205 receives an instruction to start calculation from the control unit 201, the first calculation unit 205 executes calculation using the first learning model 310 (step S469), and outputs the calculation result using the learning model 310 to the control unit 201 (step S470).
[0145] The control unit 201 executes an organ recognition process by referring to the calculation result by the first learning model 310 (step S471). The control unit 201 compares the probability of the label output from the softmax layer 313 of the first learning model 310 with the threshold (first threshold or second threshold) set in step S467 or step S468, and extracts pixels equal to or greater than the threshold, thereby recognizing the organ included in the surgical field image.
[0146] The control unit 201 outputs the recognition result of the organ (step S472). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the organ on the operative field image. In response to the instruction from the control unit 201, the first calculation unit 205 renders the recognized image of the organ in a built-in VRAM and outputs it to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized image of the organ on the operative field image.
[0147] (Variation 4-4) The control unit 201 of the information processing device 200 may change the threshold value between the period until the target organ is recognized and the period after the target organ is recognized. Fig. 22 is a flowchart showing a processing procedure in Modification 4-4. The control unit 201 of the information processing device 200 executes a scene recognition process and an organ recognition process (steps S481 to S488) every time an operative field image is acquired in the same procedure as that shown in Fig. 17. It is assumed that a third threshold value (<first threshold value) is set in advance as the threshold value used in the organ recognition process.
[0148] The control unit 201 determines whether or not an organ has been recognized by the organ recognition process of step S488 (step S489). For example, when the number of pixels determined to have a probability of the label output from the softmax layer 313 of the first learning model 310 equal to or greater than the third threshold (e.g., equal to or greater than 30%) is equal to or greater than a predetermined number, the control unit 201 can determine that an organ has been recognized from the surgical field image.
[0149] When it is determined that the organ has not been recognized (S489: NO), the control unit 201 sets a third threshold value (step S490). That is, the control unit 201 maintains the preset threshold value. By maintaining the threshold value at a relatively low value until the organ is recognized, the organ becomes easier to detect, and the information processing device 200 can function as a sensor for organ detection.
[0150] When it is determined that the organ has been recognized (S489: YES), the control unit 201 sets the first threshold higher than the third threshold (step S491). That is, when the control unit 201 starts recognizing the organ, the control unit 201 changes the threshold used for organ recognition from the third threshold to the first threshold (>the third threshold), thereby improving the accuracy of organ recognition.
[0151] The control unit 201 outputs the recognition result of the organ (step S492). Specifically, the control unit 201 instructs the first calculation unit 205 to superimpose the recognized image of the organ on the operative field image. In response to the instruction from the control unit 201, the first calculation unit 205 renders the recognized image of the organ in a built-in VRAM and outputs it to the display device 130 via the output unit 207, thereby superimposing and displaying the recognized image of the organ on the operative field image.
[0152] (Variation 4-5) The control unit 201 of the information processing device 200 may derive the proper name of the organ recognized from the calculation result of the first learning model 310 based on the information of the scene recognized by the scene recognition. FIG. 23 is a conceptual diagram showing an example of the proper name table. The proper name table registers the proper name of the organ in association with the surgical scene. For example, the inferior mesenteric artery and the inferior mesenteric plexus often appear in the scene of the sigmoid colon cancer surgery. In the proper name table, the inferior mesenteric artery and the inferior mesenteric plexus are registered as the proper names of the organs in the scene of the sigmoid colon cancer surgery. The proper name table may be prepared in the memory unit 202 of the information processing device 200 or in an external device.
[0153] The control unit 201 can present the proper name of the organ identified by the organ recognition to the surgeon by accessing the storage unit 202 or an external device according to the scene recognized by the scene recognition and reading out the proper name of the organ registered in the proper name table. FIG. 24 is a schematic diagram showing an example of display of the organ name. When the control unit 201 recognizes a sigmoid colon cancer surgery scene by the scene recognition and recognizes a blood vessel using the first learning model 310 for blood vessel recognition, the control unit 201 estimates that the blood vessel is the inferior mesenteric artery by referring to the proper name table. The control unit 201 may cause the display unit 130 to display text information indicating that the proper name of the recognized blood vessel is the inferior mesenteric artery. The control unit 201 may also cause the display unit 130 to display the proper name only when the control unit 201 receives an instruction from the surgeon via the operation unit 203 or the like.
[0154] In addition, the control unit 201 recognizes the sigmoid colon cancer surgery scene by scene recognition, and when it recognizes a nerve using the first learning model 310 for nerve recognition, it infers that the nerve is the inferior mesenteric plexus by referring to the proper name table, and displays it on the display device 130.
[0155] (Variation 4-6) The control unit 201 of the information processing device 200 may derive information on the structure based on information on the scene recognized by the scene recognition and information on the organ recognized by the organ recognition. The information on the structure derived in the modification examples 4-6 is information on the organ not recognized by the organ recognition, information on a lesion such as a cancer or a tumor, etc.
[0156] The control unit 201 derives information on structures by referring to the structure table. FIG. 25 is a conceptual diagram showing an example of the structure table. Information on known structures for each scene is registered in the structure table. The information on structures is, for example, textbook-like information on organs, and includes information on the organ's name, location, running direction, and the like. For example, by referring to the structure table shown in FIG. 25, when the control unit 201 recognizes the portal vein in stomach surgery, it can present to the surgeon information that the left gastric vein branches off from the portal vein. Similarly, when the control unit 201 recognizes the right gastric artery in stomach surgery, it can present to the surgeon information that the root of the right gastric artery has a shape of a human character.
[0157] A structure table may be prepared for each patient. In the structure table for each patient, information on the lesion for each patient obtained in advance using other medical images or examination means is registered. For example, information on the lesion obtained in advance using CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, ultrasound tomographic images, optical coherence tomographic images, angiographic images, etc. is registered. By using these medical images, information on the lesion inside the organ that does not appear in the observation image (operative field image) of the laparoscope 11 can be obtained. When the control unit 201 of the information processing device 200 recognizes a scene or an organ, it can refer to the structure table and present information on the lesion that does not appear in the operative field image.
[0158] Fig. 26 is a schematic diagram showing an example of display of structures. Fig. 26A shows an example of display when the right gastric artery is recognized in a stomach operation. The control unit 201 recognizes the right gastric artery by scene recognition or organ recognition, and refers to the structure table to read out information that the root of the right gastric artery has a shape of the Chinese character 'human'. Based on the information read out from the structure table, the control unit 201 can predict and display on the display device 130 text information that the root of the right gastric artery has a shape of the Chinese character 'human' and the course of blood vessels that have not yet been confirmed.
[0159] 26B shows an example of displaying a lesion that does not appear in the surgical field image. When information on a lesion obtained in advance for a specific patient is registered in the structure table, the control unit 201, upon recognizing the scene and organ, reads out information on the lesion that does not appear in the surgical field image by referring to the structure table for each patient. Based on the information read out from the structure table, the control unit 201 can display on the display device 130 an image of the lesion that does not appear in the surgical field image and text information indicating that a lesion exists inside the organ.
[0160] The control unit 201 may display the above-mentioned structure as a three-dimensional image. The control unit 201 can display the structure as a three-dimensional image by reconstructing a plurality of tomographic images obtained in advance by CT, MRI, or the like using a technique such as surface rendering or volume rendering. The control unit 201 may display the three-dimensional image as an object in augmented reality (AR) by superimposing it on the operative field image, or may display it as an object in virtual reality (VR) separately from the operative field image. The object in augmented reality or virtual reality may be presented to the surgeon through a head-mounted display not shown in the figure. The information processing device 200 according to the present embodiment can recognize and display structures such as organs that appear in the operative field image using the first learning model 310, and can display structures that do not appear in the operative field image using AR technology or VR technology. The surgeon can visually recognize structures such as organs that appear in the operative field image and internal structures that do not appear in the operative field image, and can easily grasp the overall picture.
[0161] (Variation 4-7) The control unit 201 of the information processing device 200 may predict possible events based on scene information recognized by scene recognition and organ information recognized by organ recognition. A case table that collects cases that occurred in past surgeries is used to predict events. FIG. 27 is a conceptual diagram showing an example of the case table. In the case table, information on cases that occurred during surgery is registered in association with the scene recognized by scene recognition and the organ recognized by organ recognition. The table in FIG. 27 shows an example in which many cases of bleeding from the inferior mesenteric artery are registered during sigmoid colon cancer surgery.
[0162] 28 is a schematic diagram showing an example of displaying an event. When the control unit 201 recognizes a scene of a sigmoid colon cancer surgery by scene recognition and recognizes the inferior mesenteric artery by organ recognition, it can grasp that there have been frequent bleedings in the past by referring to a case table, and can display this fact as text information on the display device 130. Note that the control unit 201 may display not only cases that have occurred frequently in the past, but also special cases that have occurred in the past and cases that should be notified to the surgeon as advance information on the display device 130.
[0163] As described above, in embodiment 4, based on the calculation results obtained from the first learning model 310 for organ recognition and the calculation results obtained from the fifth learning model 350 for scene recognition, an integrated recognition result for the surgical field image can be derived, and information based on the derived recognition result can be provided to the surgeon.
[0164] (Embodiment 5) In the fifth embodiment, a configuration will be described in which event recognition and scene recognition are combined to derive an integrated recognition result.
[0165] The information processing device 200 according to the fifth embodiment derives information on a characteristic scene in an event according to a recognition result of the event. As a specific example, a configuration will be described in which, when the information processing device 200 recognizes organ damage as an event, a scene where the damage occurs and a scene where the damage is repaired are derived as characteristic scenes. The information processing device 200 includes a third learning model 330 that is trained to output information on organ damage according to an input of an operative field image, and a fifth learning model 350 that is trained to output information on a scene where the damage occurs and a scene where the damage is repaired according to an input of an operative field image. The information processing device 200 executes a calculation by the third learning model 330 in the first calculation unit 205 and executes a calculation by the fifth learning model 350 in the second calculation unit 206 every time an operative field image is acquired. In addition, the information processing device 200 temporarily records the operative field image (video) input from the input unit 204 in the storage unit 202.
[0166] 29 is a flowchart showing the procedure of processing executed by the information processing device 200 according to embodiment 5. Every time an operative field image is acquired, the control unit 201 executes event (organ damage) recognition processing by referring to the calculation result by the third learning model 330, and judges the presence or absence of organ damage based on the recognition result. When it is judged that there is organ damage, the control unit 201 executes the following processing.
[0167] Each time an operative field image is acquired, the control unit 201 executes a scene recognition process by referring to the calculation result by the fifth learning model 350 (step S501), and judges whether or not a scene where damage has occurred has been recognized based on the execution result of the scene recognition process (step S502). The control unit 201 compares the scene of the previous frame with the scene of the current frame, and judges that a scene where damage has occurred has been recognized if the scene has switched from a scene where an organ is not damaged to a scene where an organ is damaged. If it is judged that a scene where damage has occurred has not been recognized (S502: NO), the control unit 201 transitions the process to step S505.
[0168] When it is determined that the damage occurrence scene has been recognized (S502: YES), the control unit 201 extracts a partial video including the occurrence scene (step S503). The control unit 201 extracts a partial video including the occurrence scene from the surgical field image (video) temporarily recorded in the storage unit 202, for example, by specifying a time point before the damage occurrence time point (e.g., a time point several seconds before) as the start point of the partial video and the time point of the damage occurrence as the end point of the partial video. Alternatively, the control unit 201 may further proceed with recording of the surgical field image (video), and extract a partial video including the occurrence scene by specifying a time point before the damage occurrence time point as the start point of the partial video and a time point after the damage occurrence time point (e.g., a time point several seconds after) as the end point of the partial video.
[0169] The control unit 201 stores the extracted partial video in the storage unit 202 (step S504). The control unit 201 cuts out the partial video between the start point and end point specified in step S503 from the recorded video, and stores it separately in the storage unit 202 as a video file.
[0170] The control unit 201 judges whether or not a damage repair scene has been recognized based on the execution result of the scene recognition process in step S501 (step S505). The control unit 201 compares the scene in the previous frame with the scene in the current frame, and judges that a damage repair scene has been recognized if the scene has switched from one in which the organ is damaged to one in which the organ is not damaged. If it is judged that a damage repair scene has not been recognized (S505: NO), the control unit 201 ends the process according to this flowchart.
[0171] When it is determined that the damage repair scene has been recognized (S505: YES), the control unit 201 extracts a partial video including the repair scene (step S506). The control unit 201 extracts a partial video including the repair scene from the surgical field image (video) temporarily recorded in the storage unit 202, for example, by specifying a time point before the repair time (e.g., a time point several seconds before) as the start point of the partial video and the repair time as the end point of the partial video. Alternatively, the control unit 201 may further proceed with recording of the surgical field image (video), and extract a partial video including the repair scene by specifying a time point before the repair time as the start point of the partial video and a time point after the repair (e.g., a time point several seconds later) as the end point of the partial video.
[0172] The control unit 201 stores the extracted partial video in the storage unit 202 (step S507). The control unit 201 cuts out the partial video between the start point and end point specified in step S506 from the recorded video, and stores it separately in the storage unit 202 as a video file.
[0173] Each time a scene is recognized, the control unit 201 registers information about the recognized scene in a scene recording table. The scene recording table is prepared in the storage unit 202. FIG. 30 is a conceptual diagram showing an example of a scene recording table. The scene recording table stores, for example, the date and time when a scene is recognized, a name that identifies the recognized scene, and a video file of a partial video extracted when the scene is recognized, in association with each other. The example of FIG. 30 shows a state in which, for damage occurrence scenes and repair scenes recognized from the start to the end of surgery, a video file showing the damage occurrence scene and a video file showing the damage repair scene are registered in the scene recording table in association with the date and time.
[0174] The control unit 201 may cause the display device 130 to display information on the scenes registered in the scene record table. For example, the control unit 201 may cause the display device 130 to display information on the scenes in a table format so that the surgeon or the like can select any scene. Alternatively, the control unit 201 may arrange objects (UI), such as thumbnails and icons indicating each scene, on the display screen and accept the selection of a scene by the surgeon or the like. When the control unit 201 accepts the selection of a scene by the surgeon or the like, it reads out a video file of the corresponding scene from the storage unit 202 and plays the read out video file. The played video file is displayed on the display device 130.
[0175] Also, a configuration may be adopted in which a video file of a scene selected by the control unit 201 is played. The control unit 201 may refer to the scene recording table, and when a scene in which damage occurred is registered but a scene in which repair was performed for that damage is not registered, play the scene in which the damage occurred to inform the surgeon that repair was not performed.
[0176] In this embodiment, the configuration is such that damage occurrence scenes and repair scenes are recognized, partial images of each recognized scene are extracted, and stored as a moving image file in the storage unit 202, but the scenes to be recognized are not limited to damage occurrence scenes and repair scenes. The control unit 201 may recognize characteristic scenes in various events that may occur during surgery, extract partial images of the recognized characteristic scenes, and store them as a moving image file in the storage unit 202.
[0177] For example, the control unit 201 may recognize a bleeding occurrence scene and a bleeding stop scene in response to recognition of a bleeding event, extract a partial image for each recognized scene, and store it as a video file in the storage unit 202. Furthermore, when the control unit 201 recognizes a bleeding event, the control unit 201 may recognize a scene where an artificial object such as gauze is framed in the surgical field image and a scene where the artificial object is framed out of the surgical field image, extract a partial image for each recognized scene, and store it as a video file in the storage unit 202. Note that, since the introduction of gauze into the patient's body is not limited to when bleeding occurs, the control unit 201 may extract a partial image for each scene and store it as a video file in the storage unit 202 when it recognizes a scene where the gauze is framed in the surgical field image and a scene where the gauze is framed out of the surgical field image during surgery, regardless of recognition of a bleeding event. In addition, the control unit 201 may recognize scenes in which artificial objects such as hemostatic clips, restraining bands, and suture needles, not limited to gauze, are framed in the surgical field image and scenes in which they are framed out of the surgical field image, and may store video files corresponding to partial images of these scenes in the memory unit 202.
[0178] As described above, in the fifth embodiment, partial images (moving image files) showing characteristic scenes regarding events that occurred during surgery are stored in the storage unit 202, making it easy to review the events.
[0179] (Embodiment 6) The control unit 201 of the information processing device 200 may have both the first calculation unit 205 and the second calculation unit 206 perform calculations using the first learning model 310 on the same surgical field image, and derive error information in the first calculation unit 205 or the second calculation unit 206 by evaluating the calculation results by the first calculation unit 205 and the calculation results by the second calculation unit 206.
[0180] 31 is a flowchart showing a procedure of processing executed by the information processing device 200 according to embodiment 6. When the control unit 201 acquires an operative field image (step S601), the control unit 201 causes both the first calculation unit 205 and the second calculation unit 206 to execute calculations using the first learning model 310 (step S602).
[0181] The control unit 201 acquires the calculation result of the first calculation unit 205 and the calculation result of the second calculation unit 206 (step S603) and evaluates the calculation result (step S604). The control unit 201 judges whether the calculation result of the first calculation unit 205 and the calculation result of the second calculation unit 206 differ by a predetermined amount or more (step S605). If the difference is less than the predetermined amount (S605: NO), the control unit 201 ends the process according to this flowchart. If it is judged that the difference is the predetermined amount or more (S605: YES), the control unit 201 judges that an error is detected (step S606) and outputs warning information (step S607). The warning information may be displayed on the display device 130, or may be output by sound or voice.
[0182] (Embodiment 7) In the seventh embodiment, a configuration will be described in which the use state of a surgical tool is estimated by combining recognition results from a plurality of learning models.
[0183] The information processing device 200 according to embodiment 7 includes a first learning model 310 for recognizing an organ, a fourth learning model 340 for recognizing a device, and a fifth learning model 350 for recognizing a scene.
[0184] The first learning model 310 is a learning model that is trained to output a probability indicating whether each pixel constituting the surgical field image corresponds to the organ to be recognized in response to an input of the surgical field image. The control unit 201 of the information processing device 200 acquires the calculation results from the first learning model 310 at any time and analyzes them in a time series manner to estimate the effect of the surgical tool on the organ. For example, the control unit 201 calculates the number of pixels recognized as an organ by the first learning model 310, the change in area, or the amount of pixels recognized as an organ that have been lost, and when it is determined that a predetermined amount of pixels have been lost, it estimates that the surgical tool has moved onto the organ.
[0185] The fourth learning model 340 is a learning model that is trained to output information about devices appearing in an operative field image in response to an input of the operative field image. The fourth learning model 340 in the seventh embodiment is trained to include information about the open / closed state of the forceps 13 as information about the device.
[0186] The fifth learning model 350 is a learning model that is trained to output information about a scene shown by a surgical field when a surgical field image is input. The fifth learning model 350 in the seventh embodiment is trained to include information about a scene in which an organ is grasped as information about the scene.
[0187] Fig. 32 is an explanatory diagram for explaining the estimation method in the seventh embodiment. The surgical field image shown in Fig. 32 shows a scene in which the connective tissue filling between the tissue ORG1 constituting an organ and the tissue ORG2 containing a lesion such as a malignant tumor is about to be excised. At this time, the surgeon grasps the tissue ORG2 containing the lesion with the forceps 13 and spreads it in an appropriate direction to expose the connective tissue existing between the tissue ORG2 containing the lesion and the tissue ORG1 to be left. The surgeon excises the exposed connective tissue using the energy treatment tool 12, thereby detaching the tissue ORG2 containing the lesion from the tissue ORG1 to be left.
[0188] The control unit 201 of the information processing device 200 obtains the calculation result obtained by inputting the operative field image as shown in Fig. 32 into the fourth learning model 340, and recognizes that the forceps 13 are in a closed state. However, based only on the calculation result of the fourth learning model 340, the control unit 201 cannot determine whether the forceps 13 are closed with an organ grasped or closed without grasping an organ.
[0189] Therefore, in the seventh embodiment, the use state of the forceps 13 is estimated by further combining the calculation results of the first learning model 310 and the fifth learning model 350. That is, when the control unit 201 recognizes that the forceps 13 is in a closed state based on the calculation results of the fourth learning model 340, recognizes that the forceps 13 are on an organ based on the calculation results of the first learning model 310, and recognizes that the scene is one in which the organ is being grasped based on the calculation results of the fifth learning model 350, it can recognize that the forceps 13 is closed while grasping the organ.
[0190] 33 is a flowchart showing an estimation procedure in the seventh embodiment. The control unit 201 of the information processing device 200 acquires a frame-by-frame operative field image output from the CCU 110 through the input unit 204 (step S701). The control unit 201 executes the following process every time it acquires a frame-by-frame operative field image.
[0191] The control unit 201 acquires the calculation result by the fourth learning model 340 (step S702) and determines whether the forceps 13 are closed or not (step S703). The calculation by the fourth learning model 340 is executed by, for example, the second calculation unit 206. The control unit 201 may acquire the calculation result by the fourth learning model 340 from the second calculation unit 206. If it is determined that the forceps 13 are not closed (S703: NO), the control unit 201 ends the processing according to this flowchart.
[0192] When it is determined that the forceps 13 are closed (S703: YES), the control unit 201 acquires the calculation result by the first learning model 310 (step S704) and determines whether the forceps 13 are present on the organ (step S705). The calculation by the first learning model 310 is executed by, for example, the first calculation unit 205. The control unit 201 may acquire the calculation result by the first learning model 310 from the first calculation unit 205. When it is determined that the forceps 13 are not present on the organ (S705: NO), the control unit 201 ends the processing according to this flowchart.
[0193] When it is determined that the forceps 13 are present on the organ (S705: YES), the control unit 201 acquires the calculation result by the fifth learning model 350 (step S706) and determines whether or not the scene is one in which the organ is to be grasped (step S707). The calculation by the fifth learning model 350 is executed by the second calculation unit 206, for example, while the calculation by the fourth learning model 340 is not being executed. The control unit 201 may acquire the calculation result by the fifth learning model 350 from the second calculation unit 206. When it is determined that the scene is not one in which the organ is to be grasped (S707: NO), the control unit 201 ends the processing according to this flowchart.
[0194] When it is determined that the scene is one in which the organ is to be grasped (step S707: YES), the control unit 201 estimates that the forceps are grasping the organ (step S708). The estimation result may be displayed on the operative field image. For example, when the forceps 13 have completed grasping the organ, the color of the forceps 13 may be brightened to display the forceps 13.
[0195] In this embodiment, a configuration for estimating the state in which an organ is grasped by forceps 13 has been described. However, by combining the recognition results of multiple learning models, it is possible to estimate not only grasping by forceps 13, but also procedures using various surgical devices (grasping, cutting, dissection, etc.).
[0196] (Embodiment 8) In the eighth embodiment, a configuration will be described in which an optimal learning model is selected depending on an input operative field image.
[0197] The information processing device 200 according to the eighth embodiment includes a plurality of learning models for recognizing the same recognition target. As an example, a configuration for recognizing the same organ using the first learning model 310 and the second learning model 320 will be described. The organs recognized by the first learning model 310 and the second learning model 320 are not limited to loose connective tissue or nervous tissue, and may be any organ that has been set in advance.
[0198] In one example, the first learning model 310 and the second learning model 320 are constructed using different neural networks. For example, the first learning model 310 is constructed by SegNet, and the second learning model 320 is constructed by U-Net. The combination of neural networks that construct the first learning model 310 and the second learning model 320 is not limited to the above, and any arbitrary neural network may be used.
[0199] Alternatively, the first learning model 310 and the second learning model 320 may be learning models with different internal configurations. For example, the first learning model 310 and the second learning model 320 may be learning models constructed using the same neural network, but may differ in the type of layers, the number of layers, the number of nodes, the connection relationship of the nodes, and the like.
[0200] The first learning model 310 and the second learning model 320 may be learning models trained using different training data. For example, the first learning model 310 may be a learning model trained using training data including correct answer data annotated by a first expert, and the second learning model 320 may be a learning model trained using training data including correct answer data annotated by a second expert different from the first expert. The first learning model 310 may be a learning model trained using training data including an operative field image captured at a certain medical institution and annotation data (correct answer data) for the operative field image, and the second learning model 320 may be a learning model trained using training data including an operative field image captured at another medical institution and annotation data (correct answer data) for the operative field image.
[0201] When an operative field image is input, the information processing device 200 executes a calculation by the first learning model 310 in the first calculation unit 205, and executes a calculation by the second learning model 320 in the second calculation unit 206. The control unit 201 of the information processing device 200 analyzes the calculation results by the first calculation unit 205 and the calculation results by the second calculation unit 206, and selects a learning model (in this embodiment, either the first learning model 310 or the second learning model) optimal for organ recognition based on the analysis results.
[0202] FIG. 34 is an explanatory diagram for explaining a method of analyzing the calculation results. From each learning model for recognizing an organ, a probability (certainty) indicating whether each pixel corresponds to the organ to be recognized is output as a calculation result. When the number of pixels is counted for each confidence, for example, a distribution as shown in FIG. 34A to FIG. 34C is obtained. The horizontal axis of each graph shown in FIG. 34A to FIG. 34C represents the confidence, and the vertical axis represents the number of pixels (proportion of the entire image). Ideally, each pixel is classified into a confidence of 1 (when the probability that it is an organ is 100%) or a confidence of 0 (when the probability that it is an organ is 0), so when the distribution of confidence is examined based on the calculation result obtained from an ideal learning model, a polarized distribution as shown in FIG. 34A is obtained.
[0203] When the control unit 201 of the information processing device 200 obtains the calculation results from the first learning model 310 and the second learning model 320, it counts the number of pixels for each confidence level and selects a learning model having a distribution close to the ideal distribution. For example, if the distribution obtained from the calculation results of the first learning model 310 is the distribution shown in FIG. 34B and the distribution obtained from the calculation results of the second learning model 320 is the distribution shown in FIG. 34C, the latter is closer to the ideal distribution, so the control unit 201 selects the second learning model 320.
[0204] The control unit 201 evaluates each distribution using an evaluation coefficient such that the evaluation value becomes higher as the confidence level approaches 1 or 0, and determines whether the distribution is close to an ideal distribution. Fig. 35 is a diagram showing an example of an evaluation coefficient table. Such an evaluation coefficient table is prepared in advance in the storage unit 202. In the example of Fig. 35, the evaluation coefficient is set so that the confidence level becomes higher as the confidence level approaches 1 or 0.
[0205] When the control unit 201 obtains the counted result of the number of pixels for each confidence level, it calculates a score indicating the quality of the distribution by multiplying the evaluation coefficient. FIG. 36 is a diagram showing an example of the score calculation result. FIG. 36A to FIG. 36C show the results of calculating the scores for each distribution shown in FIG. 34A to FIG. 34C. The score calculated from the ideal distribution is the highest. When the score is calculated for the distribution obtained from the calculation result of the first learning model 310, the total score is 84, and when the score is calculated for the distribution obtained from the calculation result of the second learning model 320, the total score is 188. That is, since the second learning model 320 has a higher score than the first learning model 310, the control unit 201 selects the second learning model 320 as an appropriate learning model.
[0206] 37 is a flowchart showing a procedure of a process executed by the information processing device 200 according to the eighth embodiment. When the control unit 201 acquires an operative field image (step S801), the control unit 201 causes the first calculation unit 205 to execute a calculation by the first learning model 310 (step S802), and acquires a calculation result by the first learning model 310 (step S803). The control unit 201 counts the number of pixels for each confidence level for the first learning model 310 (step S804), and multiplies each by an evaluation coefficient to calculate a distribution score (first score) (step S805).
[0207] Similarly, the control unit 201 causes the second calculation unit 206 to execute calculations using the second learning model 320 for the operative field image acquired in step S801 (step S806), and acquires the calculation results using the second learning model 320 (step S807). The control unit 201 tallies the number of pixels for each confidence level for the second learning model 320 (step S808), and multiplies each by an evaluation coefficient to calculate a distribution score (second score) (step S809).
[0208] In this flowchart, for convenience, the procedure is such that calculations for the first learning model 310 (S802 to S805) are performed first, and then calculations for the second learning model 320 (S806 to S809) are performed, but these procedures may be performed in any order, or may be performed simultaneously in parallel.
[0209] The control unit 201 compares the first score with the second score, and determines whether the first score is equal to or greater than the second score (step S810).
[0210] When it is determined that the first score is equal to or greater than the second score (S810: YES), the control unit 201 selects the first learning model 310 as an appropriate learning model (step S811). After that, the control unit 201 executes the organ recognition process using the selected first learning model 310.
[0211] When it is determined that the first score is less than the second score (S810: NO), the control unit 201 selects the second learning model 320 as an appropriate learning model (step S812). After that, the control unit 201 executes the organ recognition process using the selected second learning model 320.
[0212] As described above, in the eighth embodiment, it is possible to select a more appropriate learning model and execute the organ recognition process.
[0213] The information processing device 200 may execute an organ recognition process using the calculation result of the first learning model 310 in the foreground, and execute a calculation by the second learning model 320 in the background. The control unit 201 may evaluate the first learning model 310 and the second learning model 320 at regular intervals, and switch the learning model used for organ recognition according to the evaluation result. The control unit 201 may also evaluate the first learning model 310 and the second learning model 320 at a timing when an instruction is given by an operator or the like, and switch the learning model used for organ recognition according to the evaluation result. Furthermore, in combination with the scene recognition described in the fourth embodiment, when a characteristic scene is recognized, the first learning model 310 and the second learning model 320 may be evaluated, and the learning model used for organ recognition may be switched according to the evaluation result.
[0214] In the eighth embodiment, an example of application to the learning models 310 and 320 for organ recognition has been described, but default models and optional models may also be prepared for the learning model 330 for event recognition, the learning model 340 for device recognition, and the learning model 350 for scene recognition, and the model to be used for recognition may be switched by evaluating the calculation results of these.
[0215] In the eighth embodiment, a method using an evaluation coefficient has been described as a method for evaluating the first learning model 310 and the second learning model 320. However, the evaluation can be performed using various statistical indexes, not limited to the method using the evaluation coefficient. For example, the control unit 201 may obtain the variance and standard deviation of the distribution, and determine that the distribution is polarized when the variance and standard deviation are high. The control unit 201 may also evaluate the calculation results of each model by taking the value of 100-pixel ratio (%) as the value of the vertical axis of the graph and obtaining the kurtosis and skewness of the graph. Furthermore, the control unit 201 may evaluate the calculation results of each model using the mode, percentile, etc.
[0216] (Embodiment 9) In the ninth embodiment, a configuration will be described in which organ recognition is performed in the foreground and bleeding events are recognized in the background.
[0217] FIG. 38 is a sequence diagram showing an example of processing executed by the information processing device 200 according to the ninth embodiment. Image data of the operative field image output from the CCU 110 is input (video input) to the information processing device 200 at any time through the input unit 204. The input operative field image is constantly recorded in the storage unit 202, for example. The control unit 201 expands the operative field image on a frame-by-frame basis on the memory, and instructs the first calculation unit 205 and the second calculation unit 206 to perform calculations. If the first calculation unit 205 has the calculation capacity to process the operative field image at 30 FPS, the control unit 201 may instruct the first calculation unit 205 to perform calculations every time the operative field image is expanded on the memory at a frame rate of 30 FPS. In response to the instruction from the control unit 201, the first calculation unit 205 executes calculations (inference processing) based on the first learning model 310, and executes processing to draw a recognized image of an organ recognized based on the calculation result on a built-in VRAM. The recognized image of the organ drawn on the VRAM is output through the output unit 207 and displayed on the display device 130. The information processing device 200 according to the ninth embodiment can constantly display the recognized image of the organ included in the operative field image by executing the organ recognition (inference) process and the drawing process by the first calculation unit 205 in the foreground.
[0218] The second calculation unit 206 executes in the background a recognition process for events that do not need to be constantly recognized. In the ninth embodiment, a configuration for recognizing a bleeding event will be described as an example, but a configuration for recognizing an organ damage event or scene instead of a bleeding event may be used. The computing power of the second calculation unit 206 may be lower than that of the first calculation unit 205. For example, if the second calculation unit 206 has the computing power to process an operative field image at 6 FPS, as shown in FIG. 38, the second calculation unit 206 may process an operative field image input at 30 FPS once every five times.
[0219] If the second calculation unit 206 has a higher calculation capacity, the second calculation unit 206 may execute the recognition processing of multiple events in the background. For example, if the second calculation unit 206 has the same calculation capacity as the first calculation unit 205, the second calculation unit 206 can execute the recognition processing of up to five types of events, and events such as bleeding and organ damage may be executed sequentially.
[0220] FIG. 39 is a flowchart showing the procedure of the process executed by the first calculation unit 205. The first calculation unit 205 judges whether or not the current time is the inference mode (step S901). If the current time is the inference mode (S901: YES), the first calculation unit 205 acquires the latest frame expanded on the memory by the control unit 201 (step S902) and executes the inference process (step S903). In the inference process, the first calculation unit 205 executes a calculation according to the first learning model 310 for organ recognition. A recognition image of the organ is obtained by the calculation using the first learning model 310. The first calculation unit 205 executes a drawing process to draw the recognition image of the organ on the built-in VRAM (step S904). The recognition image of the organ drawn on the VRAM is output to the display device 130 through the output unit 207 and is displayed on the display device 130. After finishing the drawing process, the first calculation unit 205 returns the process to step S901.
[0221] If the current time is not the inference mode (S901: NO), the first calculation unit 205 judges whether the current time is the playback mode (step S905). If the current time is not the playback mode (S905: NO), the first calculation unit 205 returns the process to step S901.
[0222] If the current time is the playback mode (S905: YES), the first calculation unit 205 acquires the bleeding log created by the second calculation unit 206 (step S906) and acquires the designated frame from the stored partial image (step S907). Here, the partial image is an image of the surgical field having frames in a time range from the bleeding start time to the bleeding end time. Alternatively, the partial image may be an image of the surgical field including the bleeding start time and having frames from a time before the bleeding start to a time when a predetermined time range has elapsed. The second calculation unit 206 executes a drawing process to draw the partial image on the built-in VRAM (step S908). The partial image drawn on the VRAM is output to the display device 130 through the output unit 207 and is displayed on the display device 130. After completing the drawing process, the first calculation unit 205 returns the process to step S905.
[0223] 40 is a flowchart showing the procedure of the process executed by the second calculation unit 206. The second calculation unit 206 acquires the latest frame expanded in the memory by the control unit 201 (step S921), and executes an inference process (step S922). In the inference process, the second calculation unit 206 executes a calculation according to the third learning model 330 for event recognition.
[0224] The second calculation unit 206 executes bleeding determination processing and log recording processing based on the calculation results by the third learning model 330 (step S923). The second calculation unit 206 determines that the start of bleeding has been recognized when the number of pixels recognized as bleeding (or the proportion in the surgical field image) is equal to or greater than a threshold based on the calculation results by the third learning model 330. Furthermore, the second calculation unit 206 determines that the end of bleeding has been recognized when the number of pixels recognized as bleeding (or the proportion in the surgical field image) is less than a threshold after recognizing the start of bleeding. When the second calculation unit 206 recognizes the start or end of bleeding, it records the start or end of bleeding in a log.
[0225] The second calculation unit 206 judges whether or not a set of bleeding start and bleeding end is complete (step S924). If a set is not complete (S924: NO), the second calculation unit 206 returns the process to step S921. If a set is complete, the second calculation unit 206 saves the partial image (step S925). That is, the second calculation unit 206 may store an image of the surgical field having frames in a time range from the bleeding start time to the bleeding end time in the storage unit 202. The second calculation unit 206 may also store an image of the surgical field having frames including the bleeding start time and from a time before the bleeding start to a time when a predetermined time range has elapsed in the storage unit 202 as a partial image.
[0226] As described above, in the ninth embodiment, organ recognition is performed in the foreground, and event recognition is performed in the background. When an event such as bleeding is recognized by the second calculation unit 206, the first calculation unit 205 can execute a drawing process of the event recorded as a log, instead of the organ recognition process and drawing process.
[0227] The control unit 201 may constantly monitor the loads on the first calculation unit 205 and the second calculation unit 206, and may cause the first calculation unit 205 to execute the calculation that should be executed by the second calculation unit 206 according to the loads of each calculation unit.
[0228] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims.
[0229] For example, the information processing device 200 according to the first to eighth embodiments is configured to derive an integrated recognition result from the calculation results of two types of learning models, but may be configured to derive an integrated recognition result from the calculation results of three or more types of learning models. For example, by combining the first and second embodiments, the information processing device 200 may derive an integrated recognition result based on the calculation results of the two types of learning models 310 and 320 for organ recognition and the calculation result of the learning model 330 for event recognition. In addition, by combining the second and third embodiments, the information processing device 200 may derive an integrated recognition result based on the calculation result of the learning model 310 for organ recognition, the calculation result of the learning model 330 for event recognition, and the calculation result of the learning model 340 for device recognition. The information processing device 200 may derive an integrated recognition result by appropriately combining the first to eighth embodiments, without being limited to these combinations.
[0230] In addition, in this embodiment, the first calculation unit 205 executes calculations using one learning model, and the second calculation unit 206 executes calculations using the other learning model, but the hardware that executes calculations using the learning models is not limited to the first calculation unit 205 and the second calculation unit 206. For example, calculations using two or more types of learning models may be executed in the control unit 201, or a virtual machine may be prepared and calculations using two or more types of learning models may be executed on the virtual machine.
[0231] In addition, in this embodiment, the order of the processes is defined by the flowchart, but some of the processes may be executed simultaneously in parallel, and the order of the processes may be changed.
[0232] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that cites at least one multiple claim may also be used. [Explanation of symbols]
[0233] 10 Trocca 11 Laparoscopy 12 Energy treatment device 13 forceps 110 Camera Control Unit (CCU) 120 Light source device 130 Display device 140 Recording Device 200 Information processing device 201 Control section 202 Storage section 203 Operation section 204 Input section 205 1st calculation section 206 2nd calculation section 207 Output section 208 Communications Department 310 First Learning Model 320 Second Learning Model PG1 Recognition Processing Program PG2 Display processing program
Claims
1. a first calculation unit that executes a calculation using a first learning model in response to an input of a surgical field image; A second calculation unit that executes a calculation using a second learning model in response to an input of the operative field image; a derivation unit that derives an integrated recognition result for the operative field image based on a calculation result output from the first calculation unit and a calculation result output from the second calculation unit; an output unit that outputs information based on the derived recognition result; An information processing device comprising:
2. The first learning model and the second learning model are learning models that are trained to output information about organs included in the surgical field image when the surgical field image is input. The information processing device according to claim 1 .
3. the derivation unit recognizes a specific structure included in the operative field image as a first organ based on the calculation result of the first calculation unit, and derives a recognition result that a similar structure has been recognized when the structure included in the operative field image is recognized as a second organ different from the first organ based on the calculation result of the second calculation unit; The output unit outputs warning information based on the recognition result. The information processing device according to claim 2 .
4. When the derivation unit derives a recognition result that a specific structure included in the operative field image is recognized as a first organ based on the calculation result of the first calculation unit, and that the structure included in the operative field image is recognized as a second organ different from the first organ based on the calculation result of the second calculation unit, the output unit stops outputting information based on the recognition result. The information processing device according to claim 2 .
5. the derivation unit, when recognizing a specific structure included in the operative field image as a first organ based on the calculation result of the first calculation unit and recognizing the structure included in the operative field image as a second organ different from the first organ based on the calculation result of the second calculation unit, selects a recognition result based on the calculation result with a higher degree of certainty; The output unit outputs information based on the selected recognition result. The information processing device according to claim 2 .
6. the derivation unit recognizes a specific structure included in the operative field image as a first organ based on the calculation result of the first calculation unit, and when the derivation unit recognizes the structure included in the operative field image as a second organ different from the first organ based on the calculation result of the second calculation unit, derives a recognition result according to a certainty of each calculation result; The output unit generates display information of the recognition result so as to display the recognition result in a display mode corresponding to the degree of certainty, and outputs the generated display information of the recognition result. The information processing device according to claim 2 .
7. The first learning model is a learning model that is trained to output information about organs included in the surgical field image when the surgical field image is input, The second learning model is a learning model that is trained to output information regarding an event that occurs in the surgical field image when the surgical field image is input. The information processing device according to claim 1 .
8. The information output by the second learning model includes information regarding bleeding or injury, The derivation unit derives information on an organ in which bleeding or damage has occurred based on a calculation result of the first calculation unit and a calculation result of the second calculation unit. The information processing device according to claim 7.
9. The information output by the second learning model includes information regarding pulsation, The derivation unit derives information about a pulsating organ based on a calculation result of the first calculation unit and a calculation result of the second calculation unit, The output unit generates display information of the organ to display the organ in synchronization with the pulsation, and outputs the generated display information of the organ. The information processing device according to claim 7.
10. The first learning model is a learning model that is trained to output information about organs included in the surgical field image when the surgical field image is input, The second learning model is a learning model that is trained to output information about a surgical tool included in the surgical field image when the surgical field image is input. The information processing device according to claim 1 .
11. The derivation unit derives information indicating whether or not a surgical tool recognized from a calculation result of the second calculation unit is moving on an organ recognized from a calculation result of the first calculation unit, The output unit generates display information of the organ so as to change a display mode of the organ depending on whether the surgical tool is moving on the organ, and outputs the generated display information of the organ. The information processing device according to claim 10.
12. The derivation unit determines whether or not the surgical tool recognized from the calculation result of the second calculation unit has stopped, and when it determines that the surgical tool has stopped, performs an organ recognition process based on the calculation result of the first calculation unit, and when it determines that the surgical tool has not stopped, stops the recognition process. The information processing device according to claim 10.
13. When the derivation unit determines that the surgical tool recognized from the calculation result of the second calculation unit is not stopped, the output unit stops outputting information based on the recognition result of the organ. The information processing device according to claim 10.
14. The derivation unit derives information on an organ being treated by a surgical tool based on a calculation result of the first calculation unit and a calculation result of the second calculation unit. The information processing device according to claim 10 or 11.
15. The derivation unit acquires dimensional information of a surgical tool recognized from a calculation result of the second calculation unit, and derives dimensional information of an organ recognized from a calculation result of the first calculation unit based on the acquired dimensional information. The information processing device according to claim 10 or 11.
16. The output unit outputs operation support information for a surgical tool recognized from a calculation result of the second calculation unit.
16. The information processing device according to claim 10.
17. The first learning model is a learning model that is trained to output information about organs included in the surgical field image when the surgical field image is input, The second learning model is a learning model that is trained to output information about a scene shown in the surgical field image when the surgical field image is input. The information processing device according to claim 1 .
18. The derivation unit determines whether or not the scene recognized from the calculation result of the second calculation unit is a characteristic scene, and when it is determined that the scene is a characteristic scene, executes a recognition process of the organ based on the calculation result of the first calculation unit. The information processing device according to claim 17.
19. The derivation unit acquires prior information according to a scene to be recognized from a calculation result of the second calculation unit, and performs an organ recognition process based on the calculation result of the first calculation unit by referring to the acquired prior information.
19. The information processing device according to claim 17 or 18.
20. The derivation unit determines a consistency between a recognition result of an organ based on a calculation result of the first calculation unit and prior information, and when determining that there is no consistency, derives warning information indicating that the recognition result and the prior information are not consistent.
20. An information processing device according to any one of claims 17 to 19.
21. The derivation unit determines a consistency between an organ recognition result based on a calculation result of the first calculation unit and prior information, and when determining that there is no consistency, stops the organ recognition process.
21. The information processing device according to claim 17.
22. When the derivation unit derives a recognition result indicating that the organ recognition result based on the calculation result of the first calculation unit is not consistent with the prior information, the output unit stops outputting information based on the organ recognition result.
21. The information processing device according to claim 17.
23. The first learning model includes a plurality of learning models corresponding to a plurality of types of organs, The derivation unit selects one learning model from a plurality of learning models included as the first learning model according to a scene recognized from a calculation result of the second calculation unit, and instructs the first calculation unit to execute a calculation using the selected one learning model.
23. An information processing device according to any one of claims 17 to 22.
24. The derivation unit changes a threshold value used in recognizing an organ based on a calculation result of the first calculation unit according to a scene recognized based on a calculation result of the second calculation unit.
24. An information processing device according to any one of claims 17 to 23.
25. The derivation unit sets the threshold to a first threshold in a scene including an organ to be detected, and sets the threshold to a second threshold higher than the first threshold in a scene including no organ to be detected. The information processing device according to claim 24.
26. The derivation unit sets the threshold to a third threshold lower than the first threshold until an organ to be detected is recognized, and sets the threshold to the first threshold when the organ to be detected is recognized. The information processing device according to claim 25.
27. The derivation unit derives a proper name of an organ recognized from the calculation result of the first calculation unit based on information of a scene recognized from the calculation result of the second calculation unit.
27. An information processing device according to any one of claims 17 to 26.
28. The derivation unit derives information of a structure not recognized from the calculation result of the first calculation unit by referring to prior information including information of a known structure based on information of an organ recognized from the calculation result of the first calculation unit and information of a scene recognized from the calculation result of the second calculation unit.
28. An information processing device according to any one of claims 17 to 27.
29. The derivation unit derives information on a possible event by referring to prior information including cases that have occurred in the past, based on information on an organ recognized from the calculation result of the first calculation unit and information on a scene recognized from the calculation result of the second calculation unit.
29. An information processing device according to any one of claims 17 to 28.
30. The first learning model is a learning model that is trained to output information regarding an event in the surgical field image when the surgical field image is input, The second learning model is a learning model that is trained to output information about a scene shown in the surgical field image when the surgical field image is input. The information processing device according to claim 1 .
31. The derivation unit derives information on a characteristic scene in the event in response to the recognition of the event. The information processing device according to claim 30.
32. an extraction unit that extracts a partial video showing the characteristic scene; a storage unit that stores the partial video extracted by the extraction unit; The information processing device according to claim 31 .
33. the event is organ damage; The extraction unit extracts a partial video of a scene in which damage to an organ occurs and a partial video of a scene in which the damage is repaired. The information processing device according to claim 32.
34. The event involves the introduction of an artificial object into the body, The extraction unit extracts a partial image of a scene where the artificial object is framed in the surgical field image and a partial image of a scene where the artificial object is framed out of the surgical field image. The information processing device according to claim 32.
35. The output unit outputs the partial video stored in the storage unit.
35. An information processing device according to any one of claims 32 to 34.
36. The derivation unit combines calculation results of a plurality of learning models including the first learning model and the second learning model, and estimates a use state of a surgical tool appearing in the surgical field image.
36. An information processing apparatus according to any one of claims 1 to 35.
37. The derivation unit evaluates the first learning model and the second learning model based on a calculation result using the first learning model and a calculation result using the second learning model, and derives a recognition result using a learning model selected based on the evaluation result. The information processing device according to claim 1 .
38. the first calculation unit, when the operative field image is input, executes a calculation using the first learning model and a drawing process of an organ recognized from a calculation result of the first learning model; The second calculation unit, when the operative field image is input, executes a calculation using the second learning model and a recording process for recording a recognition result recognized based on the calculation result of the second learning model as a log.
30. An information processing apparatus according to any one of claims 1 to 29.
39. The first calculation unit executes each process in the foreground, and the second calculation unit executes each process in the background.
39. The information processing device according to claim 38.
40. When a specific recognition target is recognized from the calculation result of the second learning model, the first calculation unit executes a drawing process of an image showing the recognition target instead of the calculation by the first learning model and the drawing process of the organ.
40. The information processing device according to claim 38 or 39.
41. The image is a moving image of a predetermined time range including the time when the recognition target is recognized. The information processing device according to claim 40.
42. The derivation unit acquires calculation results by the first calculation unit and the second calculation unit for operative field images including the same object, and derives error information in the first calculation unit or the second calculation unit by evaluating the acquired calculation results.
42. The information processing device according to claim 1, further comprising:
43. Executing a calculation using a first learning model in response to an input of an operative field image; Executing a calculation using a second learning model in response to the input of the operative field image; Deriving an integrated recognition result for the surgical field image based on a calculation result by the first learning model and a calculation result by the second learning model; Output information based on derived recognition results An information processing method in which processing is carried out by a computer.
44. Executing a calculation using a first learning model in response to an input of an operative field image; Executing a calculation using a second learning model in response to the input of the operative field image; Deriving an integrated recognition result for the surgical field image based on a calculation result by the first learning model and a calculation result by the second learning model; Output information based on derived recognition results A computer program for causing a computer to execute a process.
Citation Information
Patent Citations
Information processing device, processor for endoscope, information processing method, and program
JP2020089710A
Vessel recognition apparatus, vessel recognition method and vessel recognition system
JP2020156860A
Medical image processing device, medical image processing method, and medical image processing program
WO2019054045A1
Operation assistance system, information processing device, and program
WO2019181432A1
Blood vessel recognition device, blood vessel recognition method, and blood vessel recognition system
WO2020194942A1
Cited By
Learning model creation device, assistance device, learning model creation method, assistance method, and program
WO2026197366A1
