Surgery support device, surgery support program

Abstract

A surgical support device is provided that enables medical personnel to perform surgery while referring to images captured inside a patient's organs. [Solution] The system comprises a first image processing unit that segments medical images to distinguish between parts where abnormality detection is performed and parts where it is not performed, a second image processing unit that monitors medical images and detects abnormalities within segments determined by the first image processing unit to be subject to abnormality detection, and an alarm unit that alerts an abnormality when the second image processing unit detects an abnormality. The second image processing unit has multiple layers that perform convolution on medical images input from the imaging unit, and one of the multiple layers performs the convolution processing calculation. The learning model is determined by weighted learning using supervised data in which multiple medical images, which have been taken in advance with areas whose anatomical structures match the segments as subjects, are divided into abnormal values ​​and normal values ​​using internal parameters that represent the appropriateness of each of the multiple medical images.

Description

[Technical field]

[0001] The present invention relates to a surgery support device that photographs the inside of a patient's organs and allows a medical professional to perform surgery while referring to the photographed images. [Background technology]

[0002] In fields that use medical images such as MRI, CT, and ultrasound diagnostic imaging devices, a technique is widely known that uses pattern recognition enhanced by machine learning to extract and display characteristic parts from medical images in order to prevent lesions or diseases from being overlooked in the acquired medical images (see, for example, Patent Documents 1 to 3, etc.). As one type of medical technology, devices such as endoscopes that are inserted into a patient's body cavity to perform procedures within the body cavity are known (see, for example, Patent Documents 4 to 6, etc.). Such devices transmit medical images from a camera attached to the tip to provide visual assistance to medical practitioners, enabling medical practitioners to perform work while visually checking the medical images. Now, when medical professionals perform procedures inside a body cavity, it is best if they can actually work while visually checking the condition inside the cavity, and if any abnormality occurs, it is often necessary to immediately confirm the abnormality and take action.

[0003] However, when working in tight spaces, particularly inside the eye socket or paranasal sinuses, not only is the task itself difficult, but it also requires a high level of experience to distinguish between changes such as bleeding that occur during medical procedures and abnormalities that require the procedure to be stopped, creating a need for support devices that would reduce the burden on medical professionals during surgery. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7302988 [Patent Document 2] Patent No. 7115114 [Patent Document 3] Patent No. 7366583 [Patent Document 4] Patent No. 7221016 [Patent Document 5] Patent No. 7350470 [Patent Document 6] JP 2021-166706 A Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention is made to solve such technical problems, and aims to provide a surgery support device that can quickly detect any abnormality when it occurs, thereby reducing the burden on medical personnel. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention provides a medical image processing method comprising: an imaging unit for acquiring a medical image; a first image processing unit for discriminating between portions for which abnormality detection is to be performed and portions for which abnormality detection is not to be performed by performing segmentation on the medical image obtained by the imaging unit based on an anatomical structure; a second image processing unit for monitoring the medical image and detecting an abnormality in a segment determined by the first image processing unit to be subjected to abnormality detection; and a notification unit for notifying an operator of an abnormality when the second image processing unit detects an abnormality, the medical image including color information of at least three parameters of RGB, and the second image processing unit The medical image processing system has a plurality of layers that perform convolution on the medical image input from a computer, and one of the plurality of layers performs a convolution processing operation using a learning model that is a filter layer incorporating a plurality of predetermined linear or nonlinear filters, and the learning model is determined by weighted learning using supervised data in which a plurality of medical images, which are previously captured with a site where the segment and the anatomical structure coincide as a subject, are divided into abnormal values ​​and normal values ​​using an internal parameter that represents the appropriateness of each of the plurality of medical images, and the appropriateness in the learning model of the filter layer is determined by a degree of exposure of fat in the segment, The appropriateness varies depending onThe color information in the segment is calculated based on a feature vector that varies depending on at least one of the two parameters other than R. Effect of the Invention

[0007] According to the surgery support device of the present invention, any abnormality can be detected quickly, reducing the burden on medical personnel. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the overall configuration of a surgery support device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of an endoscope used in the surgery support device. [Diagram 3] 2 is a diagram illustrating an example of a functional configuration of a control unit of the dispensing support system illustrated in FIG. 1. [Figure 4] FIG. 2 is a diagram showing an example of a medical image captured by an endoscope. [Diagram 5] FIG. 5 is a diagram showing an example of segmentation of the medical image shown in FIG. 4. [Figure 6] FIG. 1 is a schematic diagram showing an example of machine learning performed by a surgery support device. [Figure 7] FIG. 13 is a diagram showing an example of how to distinguish between mucous membrane and fat. [Figure 8] FIG. 11 is a diagram illustrating an example of the operation of the surgery support device. [Figure 9] FIG. 1 shows an example of a tool that can be used with the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 shows, as an embodiment of the present invention, a surgery support device 100 which is an integrated system having an endoscope 10 which is an imaging unit, a display device 20 for displaying an image captured by the endoscope 10 on a screen, and a control unit 50 which operates as a control device for controlling the operations of the endoscope 10 and the display device 20. In the following explanatory diagrams, for the sake of simplification, etc., the illustrations focus on those necessary for explaining the invention and those related to it.

[0010] The endoscope 10 is a medical device that has a tip 10a that can be inserted into the body cavity of a patient 110 as shown in FIG. 2, and a camera 11 that captures images within the field of view visible from the tip 10a, and that allows a doctor 120 to arbitrarily control the magnification and focal position within the field of view of the tip 10a using an operation unit 12 that is operated by hand.The endoscope 10 is a medical instrument for visualizing the ear, nose and throat areas of the patient 110, such as the paranasal sinuses 111, using medical images captured by the camera 11. The tip portion 10a functions as an optical system including an objective lens and a light guide made of optical glass fiber, and is a member whose viewing direction and viewing angle can be changed by changing the cutting of the tip according to the detachable attachment. The camera 11 is a video head capable of forming an image of light from the tip portion 10a and communicating the image as a medical image to the display device 20, and functions as a part of the imaging section in this embodiment. The image captured by the camera 11 is displayed in real time on the display device 20 via a display image control section 51 in the control section 50 as shown in FIG.

[0011] The control unit 50 is an information processing device such as a computer connected to the endoscope 10, and has a display image control unit 51 that adds information as described below to images from the camera 11 and displays them, an image learning unit 52 that performs labeled machine learning to learn images when normal processing is being performed and images when an abnormality is detected from images including known moving images that have been input up to that point, and an image segmentation unit 53 that uses anatomical information to segment each part of the image captured by the endoscope 10. The control unit 50 also has an alarm unit 55 for notifying the user of an abnormality by an error sound or the like when an abnormality is detected.

[0012] The display image control unit 51 can display, in addition to the video data captured by the camera 11 at the tip of the endoscope 10, the segmentation by the image segmentation unit 53 and abnormalities detected by the image learning unit 52, as described below, as an overlay on the video data. Also, for example, caption information indicating the position of an anatomical structure may be added to each of the segmented parts as shown in Fig. 3. In this manner, the display image control unit 51 displays on the display device 20 a moving image to which various information created by the display image control unit 51 has been added in addition to the real-time moving image data of the endoscope 10. For the doctor 120, the advantage is that by viewing the display device 20, the caption information and various overlays make it easier to grasp visual information than if the doctor 120 were simply working based on the video data from the endoscope 10.

[0013] The image learning unit 52 generates a learning model by machine learning from a plurality of labeled teacher data 61 held in a database 60 provided inside or outside the surgery support device 100. The learning model here is an optimized artificial intelligence algorithm, and is a program having a function similar to a function that outputs specific data in response to input data. In other words, the image learning unit 52 may be a program stored in hardware such as a memory within the control unit 50, and in this embodiment in particular, when image data is input, it performs the operation of outputting a judgment as to whether the image data is to be labeled as normal data, as described below, or as abnormal data. The labeled training data 61 stores, for example, imaging data of the endoscope 10 when surgery is performed normally and confirmed by multiple doctors as normal data, and also stores imaging data of the endoscope 10 that has been shared in advance as failure cases as abnormal data. These multiple image data are stored in a database 60 with pre-labeled data as either normal or abnormal, and these data sets are treated as "labeled training data 61."

[0014] In this embodiment, the surgery support device 100 handles sinus surgery using an endoscope 10. Therefore, the database 60 stores image data of a plurality of insides of the sinuses, including images captured by the endoscope 10, as labeled teacher data 61, and a series of data sets relating to sinus surgery in which each image data is labeled as normal data determined by a professional such as a doctor or abnormal data in which an abnormality has been found.

[0015] The image learning unit 52 generates a learning model 56 from the labeled training data 61 in the database 60. The learning model 56 is described by its definition information. The definition information of the learning model 56 includes information on the layers included in the learning model 56, information on the nodes constituting each layer, and parameters such as weighting and bias between nodes. The configuration of the learning model 56 and the procedure for generating the learning model 56 will be described in detail later. The learning model 56 thus has a plurality of layers that perform convolution on the medical image input from the camera 11, and one of the plurality of layers is a filter layer incorporating a plurality of predetermined linear or nonlinear filters. The image learning unit 52 also functions as a second image processing unit that performs a convolution process calculation on the medical image input from the camera 11 using the learning model 56, and separates a plurality of medical images captured in advance with a site where the segment and the anatomical structure match as the subject into abnormal values ​​and normal values ​​using an internal parameter that indicates the appropriateness of each of the plurality of medical images. The learning model 56 is determined by weighted learning using labeled training data 61 stored in a database 60.

[0016] The image segmentation unit 53 is an image division means that classifies and labels each pixel of the image data obtained from the camera 11 into a group of parts based on anatomical structure, and divides the area into each type of subject. The image segmentation unit 53 may use a program that uses a trained general-purpose large-scale segmentation model such as Segment-Anything as the general-purpose learning model 57. Alternatively, such a general-purpose learning model 57 may be fine-tuned in advance to further specialize it for medical images. Such pre-fine-tuning can be implemented by adding the steps shown in Figure 4 to the dataset and architecture used for training. First, the image data used in the dataset is given a segmentation mask that indicates which class each pixel belongs to based on anatomical structures. Next, a pre-trained architecture such as a convolutional neural network (CNN) is transfer trained to suit the segmentation task. Typically, such transfer learning is performed by adjusting the final output layer to the number of classes in the segmentation mask, freezing the weights, and re-learning. By repeating this re-learning while evaluating, the existing general-purpose large-scale segmentation model is tuned into a learning model specialized for medical images, such as the medical images showing the surgical field of a sinus surgery in this embodiment.

[0017] 4 and 5 respectively show an example of a medical image Q captured by the camera 11 and a medical image R obtained by segmenting the medical image Q by the image segmentation unit 53. In FIG. The image segmentation unit 53 performs such segmentation on the input image, and defines image regions 531a, 531b, 531c, etc., which are each segmented object. Ideally, such image regions are segmented according to differences in anatomical structures. In other words, a set of anatomical structures in the human body of the patient 110 is divided into a plurality of image regions as the same part. The image segmentation unit 53 further classifies the image regions 531a, 531b, 531c, ... that are highlighted with diagonal lines as a priority monitoring region 531d in Fig. 5. The priority monitoring region 531d will be described.

[0018] It is generally known that there is a risk of damaging the thin bone called the lamina propria, which is located at the border with the orbit, during sinus surgery. If such damage to the lamina propria occurs and surgery is continued without noticing, it can lead to mild eye swelling or severe blindness. Therefore, if the surgery support device 100 can detect damage to the paper board early, this risk can be reduced, contributing to the safety of the patient 110. Such a paper-like plate is located inside the orbital portion, and therefore exists at a specific position in the surgical field that can be confirmed when the endoscope 10 is inserted, for example, into the nose of the patient 110. The position of such a paper-like plate can be identified with high accuracy by segmentation using machine learning by the image segmentation unit 53, since the positions where the endoscope 10 can be inserted during paranasal sinus surgery are somewhat limited due to the anatomical structure of the human body.

[0019] Therefore, in this embodiment, the surgery support device 100 performs early detection of such damage by analyzing in real time image data captured by the endoscope 10. Specifically, the image learning unit 52 monitors the key monitoring area 531d segmented by the image segmentation unit 53 according to the learning model 56. In this embodiment, this key monitoring area 531d is a position on the anatomical structure including the placode. In other words, the image segmentation unit 53 functions as a first image processing unit that segments the medical image obtained by the camera 11 based on anatomical structures, thereby determining which parts will undergo abnormality detection by the image learning unit 52 and which parts will not.

[0020] It is known that in cases of minor damage to the lamina propria, the fatty tissue within the orbit is first exposed. This means that if the exposure of fatty tissue can be detected with high accuracy in the priority monitoring area 531d, serious damage to the cardboard plate can be prevented, thereby contributing to the safety of the patient 110. Therefore, the learning model 56 of the image learning unit 52 in this embodiment analyzes each pixel in each frame of the image acquired by the endoscope 10 as a parameter to detect such exposure of fatty tissue, and if exposure of fatty tissue is found in the key monitoring area 531d of the surgical field image, it detects this as an abnormality. The learning model 56 has an encoder 561, a decoder 562, and a softmax layer 563. The learning model 56 generates a feature map from the input operative field image, and executes a calculation by an encoder 561 that sequentially downsamples the generated feature map, a calculation by a decoder 562 that sequentially upsamples the feature map input from the encoder 561, and a calculation by a softmax layer 563 that identifies each pixel of the feature map finally obtained by the decoder 562. The image learning unit 52 outputs the calculation results by the learning model 56 to the control unit 50.

[0021] The encoder 561 is multi-layered, with two or three convolutional layers and pooling layers arranged alternately. Figure 6 shows a schematic diagram of the structure. The encoder 561 performs a convolution operation using multiple filters to extract image features and convert them into feature vectors. A filter is a small matrix that weights a portion of an image. The encoder 561 outputs the result of the convolution operation as a feature map. A feature map is a matrix that represents the features of an image, and is expressed as a feature vector. The encoder 561 sequentially downsamples the feature map to reduce the resolution and abstract the features. Downsampling is a process that reduces the size of a feature map by removing a portion of it.

[0022] The learning model 56 then inputs the feature map output from the encoder 561 to the decoder 562 . The decoder 562 gives a part of the target sequence to the feature map and upsamples it by predicting the next element to improve the resolution and refine the features. Upsampling means to increase the size of the feature map by adding new elements. The target sequence may be text such as a caption or tag for an image, or a different representation or transformed image of the image, and the addition of these enhances the characteristics of the image input to the encoder 561. The decoder 562 further adds the feature map from the encoder 561 to the upsampled feature map to complement the image information. Addition here means arranging and concatenating feature maps side by side. For example, when the encoder 561 performs multiple convolution operations, the image learning unit 52 extracts features of "what the object is" in the surgical field image, but the position information of "where the object is in the surgical field image" is lost. Such a loss of position information may not be restored by upsampling alone, which is called a deconvolution operation. Therefore, in order to complement such a loss of position information, the decoder 562 complements such position information by adding the image after the feature map is upsampled to the feature map from the encoder 561, i.e., the feature map at the time of input that was held. Through these processes, the decoder 562 finally outputs a feature map of the same size as the input image.

[0023] The learning model 56 inputs the feature map output from the decoder 562 to the softmax layer 563. The softmax layer 563 performs probabilistic calculations for each pixel of the feature map to identify the object class. The softmax layer 563 outputs a probability value indicating which class each pixel belongs to. The probability value is a numerical value between 0 and 1. The softmax layer 563 is a part that converts the output of the last layer of the decoder 562 into a probability distribution using a softmax function, which is an activation function, and can select the most probable one from among the candidates for each element of the target sequence. In this way, the softmax layer 563 selects the class with the highest probability value from the results of the output layer, thereby classifying the image. By repeating these data processes, the image can be learned. In this embodiment, when the highest probability value is close to the normal data group, the learning model 56 returns a value that the image is normal data with a high appropriateness. When the image is different from the normal data or close to the abnormal data group, the appropriateness is treated as low.

[0024] Now, learning whether or not fatty tissue is exposed during sinus surgery using such a learning model 56 can be rephrased as learning the labeled training data 61 in the database 60 as already described, and using the presence or absence of fatty tissue as a feature vector, a probabilistic determination is made as to whether or not an input image has such a feature vector. The type of feature vector that the presence or absence of such adipose tissue will result in will vary greatly depending on the selection of the dataset of labeled training data 61 and the algorithm that characterizes it, so it is difficult to discuss it in terms of specific numerical values, etc. However, it is known that when adipose tissue is exposed, the RGB ratio between the mucosal area and the exposed fat area in the image changes significantly, as shown in Figure 7.

[0025] Therefore, by running the above-mentioned learning loop including the inter-pixel hue, i.e., color information of three or more RGB parameters, as the target sequence of the learning model 56, it is possible to estimate that the probability value of the final output result will be generated by a feature vector that further depends on two parameters other than R among the RGB color information. In this case, it can be said that the appropriateness of the judgment result of the above-mentioned learning model 56 varies depending on at least one of the two parameters other than R among the color information within each segment determined by the image segmentation unit 53.

[0026] The learning model 56 outputs the output results of the softmax layer 563 to the image learning unit 52. The image learning unit 52 can display, store, and analyze the results of the learning model 56. For example, the image learning unit 52 can use the learning model 56 to determine whether the appropriateness of a group of pixels within the priority monitoring area 531d is above a threshold, thereby checking whether or not exposure of fatty tissue is occurring in the priority monitoring area 531d.

[0027] FIG. 8 shows a schematic flow diagram of the operation performed using such a surgery support device 100, an endoscope 10, and a microdebritter 30 (not shown). When a doctor 120 uses an endoscope 10 to perform sinus surgery, the surgery support device 100 displays an image captured by a camera 11 at the tip of the endoscope 10 as a real-time image on a display device 20. Specifically, frame-by-frame images of the surgical field output from the camera 11 are acquired from the camera 11 and displayed on the display device 20 (step S101). While checking the surgical field on the display device 20, the doctor 120 operates the endoscope 10 and the microdebritter 30 illustrated in Figs. 9(a) to (c) to perform work using the microdebritter 30. The microdebritter 30 is a medical instrument that can be inserted into the paranasal sinus 111, and is equipped with a tip blade 31. The microdebritter 30 removes mucous membranes, tissues, and the like within the paranasal sinus 111 using the tip blade 31. In addition to the above configuration, other medical instruments such as forceps may be used as necessary, and the medical instruments used together with the endoscope 10 are not limited. The shape of the tip blade 31 may be of various types other than those shown in Figs. 9(b) and (c). At this time, the surgery support device 100 uses each part of the control unit 50 to instruct the start of a calculation operation for various images acquired from the endoscope 10 (step S102). The start of such a calculation operation may be performed by the doctor 120 by operating a button in the display device 20, or the control unit 50 may automatically start operation upon detecting that the endoscope 10 has reached the surgical field. The control unit 50 executes the following process every time the operative field image is updated in frame units.

[0028] The control unit 50 causes the image segmentation unit 53 to segment the images acquired by the endoscope 10 in real time (step S103). At this time, step S103 is a process of generating a feature map from one frame of the input operative field image, inputting the input operative field image and instructions (position, text, etc.) to the Segment-Anything model, and performing image segmentation. The Segment-Anything model is composed of an image encoder and an instruction encoder and decoder, and integrates the features of the image and instructions to identify the object class for each pixel of the image. Step S103 identifies the class of such an object, and performs abnormality detection for the part classified into the key monitoring area 531d, as will be described later in step S105. Therefore, step S103 can be said to be a first image processing step for distinguishing between parts for which abnormality detection is performed and parts for which it is not performed, by segmenting the medical image obtained by the imaging means based on the anatomical structure.

[0029] The image segmentation unit 53 adds, as an output result, the object class for each pixel of the image obtained by such segmentation to the image, and outputs an image such as that shown in FIG. 5 (step S104).

[0030] Similarly, the control unit 50 causes the image learning unit 52 to perform a calculation on the image acquired by the endoscope 10 using the learning model 56 (step S105), and outputs the calculation result (step S106).

[0031] In addition, for convenience, the flowchart in Figure 9 shows a procedure in which an operation is performed by the image segmentation unit 53 and then an operation is performed by the image learning unit 52, but the operation by the image segmentation unit 53 and the operation by the image learning unit 52 may be performed simultaneously in parallel. The control unit 50 derives an integrated recognition result for the surgical field image based on the segmentation calculation result by the image segmentation unit 53 and the calculation result by the image learning unit 52. The control unit 50 refers to the calculation result by the image segmentation unit 53 and executes a recognition process of the anatomical structure of each element in the surgical field (step S107). At the same time, the image learning unit 52 extracts pixels with a probability of the output label being equal to or higher than a threshold (e.g., 60% or higher) for each pixel in the key monitoring area 531d classified by the image segmentation unit 53, and can recognize the fatty tissue contained in the surgical field image. Alternatively, the threshold may be determined by back-calculating the actual hit rate for the input initial test data. In this way, step S107 can be said to be a second image processing step in which, for a segment determined in the first image processing step S103 to be subjected to abnormality detection, the medical image P is monitored to perform abnormality detection within that segment.

[0032] When fatty tissue is recognized in the operative field image in step S107, the notification unit 55 generates an overlay display or a warning sound for the key monitoring region 531d in which fatty tissue is recognized as an error (step S108) to alert the doctor 120. That is, step S108 functions as a notification step. If fatty tissue cannot be recognized in the surgical field image, the doctor 120 continues to work while referring to the image from the endoscope 10, and the surgical support device 100 continues the calculations shown in the above-mentioned steps S103 to S107 on the image data captured by the camera 11 of the endoscope 10 until fatty tissue is recognized or the doctor 120 finishes his work.

[0033] <1> In this way, the surgical support device 100 is equipped with an image segmentation unit 53 that distinguishes between key monitoring areas 531d where abnormality detection is performed and areas where it is not performed by segmenting the image obtained by the camera 11 based on anatomical structure, an image learning unit 52 that monitors the image and detects abnormalities within the key monitoring areas 531d that have been determined by the image segmentation unit 53 to be subject to abnormality detection, and a notification unit 55 that notifies the operator of an abnormality when the image learning unit 52 detects an abnormality. Moreover, the image learning unit 52 performs calculations for convolution processing using a learning model 56 having an encoder 561 composed of multiple layers that performs convolution on the image input from the camera 11. The learning model 56 is determined by weighted learning using supervised data in which multiple images captured in advance of a site whose anatomical structure coincides with the key monitoring area 531d are classified into abnormal values ​​and normal values ​​using an internal parameter that indicates the appropriateness of each of the multiple medical images. According to this configuration, if an abnormality occurs during work using the endoscope 10, it can be detected quickly, thereby reducing the burden on the doctor 120.

[0034] <2> In addition, the surgery support device 100 <1> In addition to the configuration shown in , it has the following configuration. The suitability of the surgery support device 100 varies depending on the degree of exposure of fat cells within the priority monitoring area 531d. According to this configuration, exposure of the fatty tissue can be promptly detected, thereby preventing damage to the patient 110, and thus reducing the burden on the doctor 120.

[0035] <3> More specifically, the surgery support device 100 includes: <1> , <2> In addition to the configuration shown in , it has the following configuration. The image captured by the camera 11 includes color information of at least three parameters of RGB, and the appropriateness varies depending on at least one of the two parameters other than R among the color information within the key monitoring area 531d. This is because learning model 56 uses the color information of such pixels as the target series and contains color information of three or more parameters of RGB, and as shown in Figure 9, it is clear that the color information of pixels that contain fat cells and the color information of pixels that do not contain fat cells (in the mucosal area) differ in two parameters other than R.

[0036] <4> In addition, the surgery support device 100 includes: <1> ~ <3> In addition to the configuration shown in any one of the above, it has an alarm unit 55 for notifying the doctor 120 of an abnormality, and when the alarm unit 55 notifies the doctor of the abnormality, the image learning unit 52 draws attention by overlaying the position on the priority monitoring area 531d where it is determined that an abnormality has occurred. According to this configuration, if an abnormality occurs during work using the endoscope 10, it can be detected quickly, thereby reducing the burden on the doctor 120.

[0037] <5> In addition, the surgery support device 100 in this embodiment has <1> ~ <4> In addition to the configuration shown in any one of the above, a camera 11 is attached to an endoscope 10 that can be inserted into a body cavity of the human body.

[0038] Although a preferred embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the spirit of the present invention described in the claims, unless otherwise specifically limited in the above description.

[0039] For example, in this embodiment, the surgical support device 100 is described as including a control unit 50 connected to an endoscope 10, and an image segmentation unit 53 and an image learning unit 52 provided within the control unit 50, which perform machine learning. However, the present invention is not limited to this configuration, and the terminals and devices for executing these may be placed in separate locations and connected to each other via a network.

[0040] The effects described in the embodiments of the present invention are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments of the present invention. [Explanation of symbols]

[0041] P…Medical image (surgical field image) 10. Endoscope 11...Camera (imaging unit) 30…Microdebrider 52...Image learning unit (second image processing unit) 53...Image segmentation unit (first image processing unit) 55…Information Department 56…Learning Model 61…Labeled training data 100…Surgical support equipment 120…Operator (doctor) 531a, 531b, 531c...(segments) 531d…Key monitoring areas (segments) 561...Encoder (filter layer)

Claims

1. an imaging unit for acquiring medical images; A first image processing unit that performs segmentation based on an anatomical structure on the medical image obtained by the imaging unit to distinguish between a portion for which abnormality detection is performed and a portion for which abnormality detection is not performed; a second image processing unit that monitors the medical image and detects abnormalities in a segment determined by the first image processing unit to be subjected to abnormality detection; a notification unit that notifies an operator of an abnormality when the second image processing unit detects the abnormality; Equipped with The medical image includes color information of at least three parameters of RGB, The second image processing unit is performing a calculation of the convolution process using a learning model having a plurality of layers that perform convolution on the medical image input from the imaging unit, any one of the plurality of layers being a filter layer incorporating a plurality of predetermined linear or nonlinear filters; The learning model is determined by weighted learning using supervised data in which a plurality of medical images, which have been captured in advance with a region where the segment and the anatomical structure coincide as subjects, are divided into abnormal values ​​and normal values ​​using an internal parameter that indicates the appropriateness of each of the plurality of medical images; a surgical support device, characterized in that the appropriateness in the learning model of the filter layer changes depending on a degree of exposure of fat within the segment, and the appropriateness is calculated based on a feature vector that changes depending on at least one of two parameters other than R among color information within the segment.

2. 2. The surgery support device according to claim 1, A surgical support device characterized in that the notification unit notifies the operator of an abnormality, and the second image processing unit overlays and displays the position of the segment in which it is determined that the abnormality has occurred.

3. 2. The surgery support device according to claim 1, A surgery support device, wherein the imaging unit is attached to an endoscope that can be inserted into a body cavity of a human body.

4. A first image processing step of segmenting a medical image obtained by using an imaging means based on an anatomical structure to distinguish between portions for which abnormality detection is performed and portions for which abnormality detection is not performed; a second image processing step of monitoring the medical image and detecting an abnormality in a segment determined to be subjected to abnormality detection in the first image processing step; a notification step of notifying an operator of an abnormality when the second image processing step detects an abnormality; A surgery assistance program that executes the above. The medical image includes color information of at least three parameters of RGB, The second image processing step includes: performing a calculation of the convolution process using a learning model having a plurality of layers that perform convolution on the medical image input from the imaging means, any one of the plurality of layers being a filter layer incorporating a plurality of predetermined linear or nonlinear filters; The learning model is determined by weighted learning using supervised data in which a plurality of medical images, which have been taken in advance of subjects being areas where the segment and the anatomical structure coincide, are divided into abnormal values ​​and normal values ​​using an internal parameter that represents the appropriateness of each of the plurality of medical images, and the appropriateness in the learning model of the filter layer changes depending on the degree of exposure of fat within the segment, and the appropriateness is calculated based on a feature vector that changes depending on at least one of two parameters other than R among the color information within the segment.

Images