Surgery support device and surgery support program

Abstract

To provide a surgery support device for allowing a medical person to perform surgery while referring to an image photographed in an organ of a patient.SOLUTION: A surgery support device includes: a first image processing unit for discriminating between a part for performing abnormality detection and a part for not performing abnormality detection by performing segmentation for a medical image; a second image processing unit for monitoring the medical image and performing abnormality detection in a segment for the segment in which it is determined that the abnormality detection is performed in the first image processing unit; and a notification unit for notifying an abnormality when the second image processing unit detects the abnormality. The second image processing unit includes a plurality of layers that perform convolution on the medical image input from an imaging unit, and one of the plurality of layers performs calculation of convolution processing. A learning model is determined by weighting learning based on data with a teacher in which a plurality of medical images captured in advance with a site having an anatomical structure matching the segment as a subject are classified into an abnormal value and a normal value using an internal parameter representing a degree of appropriateness of each of the plurality of medical images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a surgery support device that takes images of the inside of a patient's organs and allows a medical professional to perform surgery while referring to the taken images. [Background technology]

[0002] In fields that use medical images such as MRI, CT, and ultrasound diagnostic imaging devices, a technique is widely known that uses pattern recognition enhanced by machine learning to extract and display characteristic features from medical images in order to prevent overlooking of lesions or diseases from the acquired medical images (see, for example, Patent Documents 1 to 3, etc.). Also, as one type of medical technology, devices such as endoscopes that are inserted into a patient's body cavity to perform procedures within the body cavity are known (see, for example, Patent Documents 4 to 6). Such devices transmit medical images from a camera attached to the tip to provide visual assistance to medical professionals, allowing medical professionals to perform their work while visually checking the medical images. When medical professionals perform procedures inside a body cavity, it is best to be able to actually see the condition inside the cavity while working, and if any abnormalities occur, it is often necessary to immediately confirm the abnormality and take action.

[0003] However, when working in tight spaces, such as inside the eye socket or sinuses, not only is the work itself difficult, but it also requires a high level of experience to distinguish between changes such as bleeding that occur during medical procedures and abnormalities that require the work to be stopped, and there was a need for support devices that would reduce the burden on medical professionals during surgery. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7302988 [Patent Document 2] Patent No. 7115114 [Patent Document 3] Patent No. 7366583 [Patent Document 4] Patent No. 7221016 [Patent Document 5] Patent No. 7350470 [Patent Document 6] Patent Publication No. 2021-166706 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention is intended to solve such technical problems, and aims to provide a surgery support device that can quickly detect any abnormalities that occur and reduce the burden on medical professionals. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention provides a medical image processing system comprising: an imaging unit that acquires medical images; a first image processing unit that segments the medical images obtained from the captured images based on anatomical structures to distinguish between areas where abnormality detection is to be performed and areas where it is not to be performed; a second image processing unit that monitors the medical images and detects abnormalities within the segments determined by the first image processing unit to be subject to abnormality detection; and a notification unit that notifies an operator of an abnormality when the second image processing unit detects an abnormality, wherein the second image processing unit has a plurality of layers that perform convolution on the medical images input from the imaging unit, and one of the plurality of layers performs the convolution processing calculation using a learning model that is a filter layer incorporating a plurality of predetermined linear or nonlinear filters, and the learning model is determined by weighted learning using supervised data that classifies a plurality of medical images, which have been previously taken of areas where the anatomical structures match the segments, into abnormal values ​​and normal values ​​using internal parameters that represent the appropriateness of each of the plurality of medical images. [Effects of the Invention]

[0007] According to the surgery support device of the present invention, any abnormality can be detected quickly, reducing the burden on medical personnel. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of the overall configuration of a surgery assistance device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of the configuration of an endoscope used in a surgery assistance device. [Figure 3] 2 is a diagram illustrating an example of a functional configuration of a control unit of the dispensing support system shown in FIG. 1. FIG. [Figure 4] FIG. 1 is a diagram illustrating an example of a medical image captured by an endoscope. [Figure 5] FIG. 5 is a diagram showing an example of segmentation of the medical image shown in FIG. 4. [Figure 6] FIG. 1 is a schematic diagram illustrating an example of machine learning performed by a surgery assistance device. [Figure 7] FIG. 10 is a diagram showing an example of how to distinguish between mucous membrane and fat. [Figure 8] FIG. 10 is a diagram illustrating an example of the operation of the surgery assistance device. [Figure 9] FIG. 1 shows an example of a tool that can be used with the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Fig. 1 shows, as an embodiment of the present invention, a surgery support device 100, which is an integrated system having an endoscope 10 as an imaging unit, a display device 20 for displaying images captured by the endoscope 10 on a screen, and a control unit 50 that operates as a control device for controlling the operations of the endoscope 10 and the display device 20. For the sake of simplicity, the following explanatory drawings mainly illustrate elements necessary for explaining the invention and related components.

[0010] As shown in Figure 2, the endoscope 10 is a medical device that has a tip 10a that can be inserted into the body cavity of a patient 110 and a camera 11 that captures images within the field of view visible from the tip 10a, and can arbitrarily control the magnification and focal position within the field of view of the tip 10a using an operation unit 12 that is operated by a doctor 120 at hand.The endoscope 10 is a medical instrument that visualizes the ear, nose, and throat areas of the patient 110, such as the paranasal sinuses 111, using medical images captured by the camera 11. The tip 10a functions as an optical system including an objective lens and a light guide made of optical glass fiber, and is a component whose viewing direction and viewing angle can be changed by changing the cutting of the tip according to the detachable attachment. The camera 11 is a video head that can image light from the tip 10a and transmit the image to the display device 20 as a medical image, and functions as a part of the imaging unit in this embodiment. The image captured by the camera 11 is displayed in real time on the display device 20 via a display image control unit 51 in the control unit 50 as shown in FIG.

[0011] The control unit 50 is an information processing device such as a computer connected to the endoscope 10, and includes a display image control unit 51 that adds information described below to images from the camera 11 and displays them; an image learning unit 52 that performs labeled machine learning to distinguish between images when normal processing is being performed and images when an abnormality is detected from images including known moving images that have been input up to that point; and an image segmentation unit 53 that uses anatomical information to segment each part of the image captured by the endoscope 10. The control unit 50 also has a notification unit 55 for notifying the user of an abnormality by sounding an error or the like when an abnormality is detected.

[0012] In addition to the video data captured by the camera 11 at the tip of the endoscope 10, the display image control unit 51 can also display the segmentation by the image segmentation unit 53 and abnormalities detected by the image learning unit 52, as described below, as an overlay on the video data. Furthermore, for example, caption information indicating the position of the anatomical structure may be attached to each segmented part as shown in Fig. 3. In this way, the display image control unit 51 displays on the display device 20 a moving image to which various information created by the display image control unit 51 has been added in addition to the real-time moving image data of the endoscope 10. For the doctor 120, the advantage is that by viewing the display device 20, the caption information and various overlays make it easier to grasp visual information than simply working based on the video data from the endoscope 10.

[0013] The image learning unit 52 generates a learning model by machine learning from a plurality of labeled training data 61 stored in a database 60 provided inside or outside the surgery assistance device 100. The learning model here is an optimized artificial intelligence algorithm, and is a program having a function similar to a function that outputs specific data in response to input data. In other words, the image learning unit 52 may be a program stored in hardware such as a memory within the control unit 50, and in this embodiment, particularly, when image data is input, it performs the operation of outputting a judgment as to whether the image data should be labeled as normal data (described later) or abnormal data. The labeled training data 61 stores, for example, the photographed data of the endoscope 10 when surgery is performed normally as confirmed by multiple doctors as normal data, and also stores the photographed data of the endoscope 10 that has been shared in advance as a failed case as abnormal data. These multiple image data are stored in a database 60 with pre-labeled data as normal or abnormal, and these data sets are treated as "labeled training data 61."

[0014] In this embodiment, the surgery assistance device 100 handles sinus surgery using an endoscope 10. Therefore, the database 60 stores, as labeled training data 61, image data of the inside of a plurality of sinuses, including images captured by the endoscope 10, and a series of datasets relating to sinus surgery, in which each image data is labeled as normal data determined by a medical professional such as a doctor or abnormal data in which an abnormality has been discovered.

[0015] The image learning unit 52 generates a learning model 56 from the labeled training data 61 in the database 60. The learning model 56 is described by its definition information. The definition information of the learning model 56 includes information on the layers included in the learning model 56, information on the nodes that make up each layer, and parameters such as weighting and bias between nodes. The configuration of the learning model 56 and the procedure for generating the learning model 56 will be described in detail later. The learning model 56 has a plurality of layers that perform convolution on the medical images input from the camera 11, and one of the plurality of layers is a filter layer incorporating a plurality of predetermined linear or nonlinear filters. The image learning unit 52 also functions as a second image processing unit that performs convolution processing calculations on the medical images input from the camera 11 using the learning model 56, and classifies a plurality of medical images, which have been captured in advance with the subject being a region whose anatomical structure matches the segment, into abnormal values ​​and normal values ​​using an internal parameter that indicates the appropriateness of each of the plurality of medical images. The learning model 56 is determined by weighted learning using labeled training data 61 stored in a database 60.

[0016] The image segmentation unit 53 is an image dividing means that classifies and labels each pixel of the image data obtained from the camera 11 into groups of parts based on anatomical structure, and divides the area into groups of objects. The image segmentation unit 53 may use a program that uses a trained general-purpose large-scale segmentation model such as Segment-Anything as the general-purpose learning model 57. Alternatively, such a general-purpose learning model 57 may be fine-tuned in advance to further specialize it for medical images. Such pre-fine-tuning can be implemented by adding the steps shown in Figure 4 to the dataset and architecture used for training. First, a segmentation mask is added to the image data used in the dataset, which indicates which class each pixel belongs to based on anatomical structures. Next, a pre-trained architecture such as a convolutional neural network (CNN) is transferred to a form suitable for the segmentation task. Typically, this type of transfer learning is performed by adjusting the final output layer to match the number of classes in the segmentation mask, freezing the weights, and then retraining. By repeating this retraining process while evaluating, an existing general-purpose large-scale segmentation model is tuned into a learning model specialized for medical images, such as medical images showing the surgical field of sinus surgery in this embodiment.

[0017] 4 and 5 show an example of a medical image Q captured by the camera 11 and a medical image R obtained by segmenting the medical image Q by the image segmentation unit 53, respectively. The image segmentation unit 53 performs this segmentation on the input image and defines image regions 531a, 531b, 531c, etc., which are each segmented object. Ideally, these image regions are segmented based on differences in anatomical structure. In other words, a single part of the anatomical structure of the patient 110's body is divided into multiple image regions. The image segmentation unit 53 further classifies the image areas 531a, 531b, 531c, etc., as a priority monitoring area 531d, which is particularly highlighted with diagonal lines, from among the image areas 531a, 531b, 531c, etc. in Fig. 5. The priority monitoring area 531d will be described below.

[0018] It is generally known that sinus surgery carries the risk of damaging the thin bone called the lamina propria, which is located at the border with the orbit. If such damage to the lamina propria occurs and surgery is continued without noticing, it can lead to mild eye swelling or severe blindness. Therefore, if the surgery assistance device 100 can detect damage to the paper board early, it can reduce such risks and contribute to the safety of the patient 110. Such a paper-like plate is located inside the orbital region, and therefore exists at a specific position in the surgical field that can be confirmed when the endoscope 10 is inserted into, for example, the nose of the patient 110. The position of such a paper-like plate can be identified with high accuracy by segmentation using machine learning by the image segmentation unit 53, because the positions where the endoscope 10 can be inserted during sinus surgery are somewhat limited due to the anatomical structure of the human body.

[0019] Therefore, in this embodiment, the surgery support device 100 performs early detection of such damage by analyzing in real time image data captured by the endoscope 10. Specifically, the image learning unit 52 monitors the priority monitoring area 531d segmented by the image segmentation unit 53 in accordance with the learning model 56. In this embodiment, this priority monitoring area 531d is a position on the anatomical structure including the placoderm. In other words, the image segmentation unit 53 functions as a first image processing unit that segments the medical image obtained by the camera 11 based on anatomical structures, thereby determining which parts will be subjected to abnormality detection by the image learning unit 52 and which parts will not.

[0020] It is known that in the case of minor damage to the lamina propria, the fatty tissue inside the orbit is exposed first. This means that if the exposure of fatty tissue can be detected with high accuracy in the priority monitoring area 531d, serious damage to the cardboard plate can be prevented, thereby contributing to the safety of the patient 110. Therefore, the learning model 56 of the image learning unit 52 in this embodiment analyzes each pixel of the image acquired by the endoscope 10 on a frame-by-frame basis as a parameter to detect such exposure of fatty tissue, and if exposure of fatty tissue is found in the key monitoring area 531d of the surgical field image, it detects this as an abnormality. The learning model 56 includes an encoder 561, a decoder 562, and a softmax layer 563. The learning model 56 generates a feature map from the input surgical field image and executes calculations by an encoder 561 that sequentially downsamples the generated feature map, calculations by a decoder 562 that sequentially upsamples the feature map input from the encoder 561, and calculations by a softmax layer 563 that identifies each pixel of the feature map finally obtained from the decoder 562. The image learning unit 52 outputs the calculation results by the learning model 56 to the control unit 50.

[0021] The encoder 561 is multi-layered, with two or three convolutional layers and pooling layers arranged alternately. Figure 6 shows a schematic diagram of this structure. The encoder 561 performs convolution operations using multiple filters to extract image features and convert them into feature vectors. A filter is a small matrix that weights a portion of an image. The encoder 561 outputs the result of the convolution operation as a feature map. A feature map is a matrix that represents the features of an image and is expressed as a feature vector. The encoder 561 reduces the resolution and abstracts the features by sequentially downsampling the feature map. Downsampling is a process that reduces the size by removing part of the feature map.

[0022] Next, the learning model 56 inputs the feature map output from the encoder 561 to the decoder 562 . The decoder 562 adds a part of the target sequence to the feature map and upsamples it by predicting the next element to improve the resolution and refine the features. Upsampling means increasing the size of the feature map by adding new elements. The target sequence may be text such as a caption or tag for an image, or a different representation or converted image of the image, and adding these enhances the characteristics of the image input to the encoder 561. The decoder 562 further complements the image information by adding the feature map from the encoder 561 to the upsampled feature map. Addition here refers to arranging and concatenating feature maps side by side. For example, if the encoder 561 performs multiple convolution operations, the image learning unit 52 extracts features of "what an object is" in the surgical field image, but loses positional information of "where the object is located in the surgical field image." This loss of positional information may not be restored by upsampling alone, a process known as deconvolution. Therefore, to complement this loss of positional information, the decoder 562 complements the positional information by adding the image after upsampling the feature map to the feature map from the encoder 561, i.e., the feature map stored at the time of input. Through these processes, the decoder 562 finally outputs a feature map of the same size as the input image.

[0023] The learning model 56 inputs the feature map output from the decoder 562 to the softmax layer 563. The softmax layer 563 performs probabilistic calculations for each pixel in the feature map to identify the object class. The softmax layer 563 outputs a probability value indicating which class each pixel belongs to. The probability value is a number between 0 and 1. The softmax layer 563 converts the output of the last layer of the decoder 562 into a probability distribution using a softmax function, which is an activation function, and can select the one with the highest probability from among the candidates for each element of the target sequence. In this way, the softmax layer 563 classifies the image by selecting the class with the highest probability value from the results of the output layer. By repeating these data processes, the image can be learned. In this embodiment, when the highest probability value is close to the normal data group, the learning model 56 returns a value indicating that the image is normal data, with a high appropriateness. On the other hand, when the image is different from the normal data or close to the abnormal data group, the appropriateness is treated as low.

[0024] Now, learning whether or not adipose tissue is exposed during sinus surgery using such a learning model 56 can be rephrased as learning the labeled training data 61 in the database 60, as already mentioned, and then determining whether or not an input image has such a feature vector, based on the presence or absence of adipose tissue. The type of feature vector that the presence or absence of such adipose tissue will result in will vary greatly depending on the selected dataset of labeled training data 61 and the algorithm that characterizes it, so it is difficult to discuss it in terms of specific numerical values, etc. However, when adipose tissue is exposed, the RGB ratio between the mucosal area and the exposed fat area in the image is known to change significantly, as shown in Figure 7.

[0025] Therefore, by running the learning loop described above including the color information between pixels, i.e., three or more RGB parameters, as the target sequence of the learning model 56, it is possible to estimate that the probability value that will be the final output result will be generated by generating a feature vector that further depends on two parameters other than R among the RGB color information. In this case, it can be said that the appropriateness of the judgment result of the above-mentioned learning model 56 varies depending on at least one of the two parameters other than R among the color information within each segment determined by the image segmentation unit 53.

[0026] The learning model 56 outputs the output results of the softmax layer 563 to the image learning unit 52. The image learning unit 52 can display, save, and analyze the results of the learning model 56. For example, the image learning unit 52 can use the learning model 56 to determine whether the appropriateness of a group of pixels within the priority monitoring area 531d is above a threshold, thereby confirming whether or not fatty tissue is exposed in the priority monitoring area 531d.

[0027] FIG. 8 shows a schematic flow diagram of the operation performed using such a surgery assistance device 100, an endoscope 10, and a microdebritter 30 (not shown). When a doctor 120 performs sinus surgery using an endoscope 10, the surgery support device 100 displays an image captured by a camera 11 at the tip of the endoscope 10 as a real-time image on a display device 20. Specifically, the surgery support device 100 acquires frame-by-frame images of the surgical field output from the camera 11 and displays them on the display device 20 (step S101). While checking the surgical field on the display device 20, the surgeon 120 operates the endoscope 10 and the microdebritter 30, an example of which is shown in Figures 9(a) to (c), to perform work using the microdebritter 30. The microdebritter 30 is a medical instrument that can be inserted into the paranasal sinus 111 and is equipped with a tip blade 31. The microdebritter 30 removes mucous membranes, tissue, and the like within the paranasal sinus 111 using this tip blade 31. Note that in addition to this configuration, other medical instruments such as forceps may be used as needed, and there is no limitation on the medical instruments that can be used with the endoscope 10. Furthermore, various types of tip blade 31 may be used in addition to the shapes shown in Figures 9(b) and (c). At this time, the surgery support device 100 uses each part of the control unit 50 to issue an instruction to start a calculation operation for the various images acquired from the endoscope 10 (step S102). The start of such a calculation operation may be performed by the doctor 120 by operating a button on the display device 20, or the control unit 50 may automatically start operation upon detecting that the endoscope 10 has reached the surgical field. The control unit 50 executes the following process every time the operative field image is updated in frame units.

[0028] The control unit 50 causes the image segmentation unit 53 to segment the images acquired by the endoscope 10 in real time (step S103). In this case, step S103 is a process of generating a feature map from one frame of the input surgical field image, inputting the input surgical field image and instructions (position, text, etc.) into the Segment-Anything model, and performing image segmentation. The Segment-Anything model is composed of an image encoder and an instruction encoder and decoder, and integrates the features of the image and instructions to identify the object class for each pixel of the image. Step S103 identifies the class of such an object and performs abnormality detection for the part classified into the priority monitoring area 531d, as will be performed in step S105 described later. Therefore, step S103 can be said to be a first image processing step that segments the medical image obtained using the imaging means based on the anatomical structure to distinguish between parts for which abnormality detection is to be performed and parts for which it is not to be performed.

[0029] The image segmentation unit 53 adds the object class for each pixel of the image obtained by the segmentation to the image as an output result, and outputs the image as shown in FIG. 5 (step S104).

[0030] Similarly, the control unit 50 causes the image learning unit 52 to perform calculations on the images acquired by the endoscope 10 using the learning model 56 (step S105), and outputs the calculation results (step S106).

[0031] Also, for convenience, the flowchart in Figure 9 shows the procedure as one in which calculations are performed by the image segmentation unit 53 and then by the image learning unit 52, but the calculations by the image segmentation unit 53 and the calculations by the image learning unit 52 may be performed simultaneously in parallel. The control unit 50 derives an integrated recognition result for the surgical field image based on the segmentation calculation result by the image segmentation unit 53 and the calculation result by the image learning unit 52. The control unit 50 refers to the calculation result by the image segmentation unit 53 and executes a recognition process of the anatomical structure of each element in the surgical field (step S107). At the same time, the image learning unit 52 extracts pixels in the priority monitoring area 531d classified by the image segmentation unit 53, for which the probability of the output label is equal to or greater than a threshold (for example, 60% or greater), thereby recognizing the fatty tissue contained in the surgical field image. Alternatively, the threshold may be determined by back-calculating the actual hit rate for the input initial test data. In this way, step S107 can be said to be a second image processing step in which, for a segment determined in the first image processing step S103 to be subjected to abnormality detection, the medical image P is monitored to perform abnormality detection within the segment.

[0032] If fatty tissue is recognized in the surgical field image in step S107, the notification unit 55 generates an overlay display or a warning sound for the key monitoring region 531d in which fatty tissue is recognized as an error (step S108) to alert the doctor 120. That is, step S108 functions as a notification step. If fatty tissue cannot be recognized in the surgical field image, the doctor 120 continues to work while referring to the image from the endoscope 10, and the surgical support device 100 continues the calculations shown in steps S103 to S107 described above on the image data captured by the camera 11 of the endoscope 10 until fatty tissue is recognized or the doctor 120 finishes his work.

[0033] <1> In this way, the surgical support device 100 is equipped with an image segmentation unit 53 that segments the image obtained by the camera 11 based on anatomical structure to distinguish between priority monitoring areas 531d where abnormality detection will be performed and areas where it will not be performed, an image learning unit 52 that monitors the image and detects abnormalities within the priority monitoring areas 531d that have been determined by the image segmentation unit 53 to be where abnormality detection will be performed, and an alarm unit 55 that alerts the operator to an abnormality when the image learning unit 52 detects an abnormality. Furthermore, the image learning unit 52 performs calculations for convolution processing using a learning model 56 having an encoder 561 made up of multiple layers that performs convolution on the image input from the camera 11. The learning model 56 is determined by weighted learning using supervised data in which multiple images captured in advance of subjects whose anatomical structure matches the key monitoring area 531d are classified into abnormal values ​​and normal values ​​using internal parameters that represent the appropriateness of each of the multiple medical images. With this configuration, if an abnormality occurs during work using the endoscope 10, it can be detected quickly, thereby reducing the burden on the doctor 120.

[0034] <2> In addition, the surgery support device 100 <1> In addition to the configuration shown in , it has the following configuration. The suitability of the surgery support device 100 varies depending on the degree of exposure of fat cells within the priority monitoring region 531d. According to this configuration, exposure of the fatty tissue can be detected quickly, thereby preventing injury to the patient 110, and thus reducing the burden on the doctor 120.

[0035] <3> More specifically, the surgery assistance device 100 includes: <1> , <2> In addition to the configuration shown in , it has the following configuration. The image captured by the camera 11 includes color information of at least three parameters of RGB, and the appropriateness changes depending on at least one of the two parameters other than R among the color information within the priority monitoring area 531d. This is because the learning model 56 uses the color information of such pixels as the target series and contains color information of three or more parameters of RGB, and as shown in Figure 9, it is clear that the color information of pixels containing fat cells differs from the color information of pixels (in the mucosal area) that do not contain fat cells in two parameters other than R.

[0036] <4> In addition, the surgery assistance device 100 <1> ~ <3> In addition to the configuration shown in any one of the above, it has a notification unit 55 for notifying the doctor 120 of an abnormality, and while the notification unit 55 notifies the doctor 120 of the abnormality, the image learning unit 52 draws attention by overlaying the position on the priority monitoring area 531d where it is determined that an abnormality has occurred. With this configuration, if an abnormality occurs during work using the endoscope 10, it can be detected quickly, thereby reducing the burden on the doctor 120.

[0037] <5> Furthermore, the surgery assistance device 100 in this embodiment includes: <1> ~ <4> In addition to the configuration shown in any one of the above, the camera 11 is attached to an endoscope 10 that can be inserted into a body cavity of the human body.

[0038] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to such a specific embodiment, and unless otherwise specifically limited in the above description, various modifications and changes are possible within the spirit and scope of the present invention as set forth in the claims.

[0039] For example, in this embodiment, the surgical support device 100 is described as including a control unit 50 connected to the endoscope 10, and an image segmentation unit 53 and an image learning unit 52 provided within the control unit 50, which perform machine learning. However, the present invention is not limited to such a configuration, and the terminals and devices for executing these may be placed in separate locations and connected to each other via a network.

[0040] The effects described in the embodiments of the present invention are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments of the present invention. [Explanation of symbols]

[0041] P...Medical image (surgical field image) 10...Endoscope 11...Camera (imaging unit) 30...Microdebrider 52...Image learning unit (second image processing unit) 53...Image segmentation unit (first image processing unit) 55…Information Department 56...Learning Model 61...Labeled training data 100…Surgical support equipment 120...Operator (doctor) 531a, 531b, 531c... (segments) 531d...Key monitoring area (segment) 561...Encoder (filter layer)

Claims

1. an imaging unit for acquiring medical images; a first image processing unit that segments the medical image obtained by the imaging unit based on an anatomical structure to distinguish between portions for which abnormality detection is to be performed and portions for which abnormality detection is not to be performed; a second image processing unit that monitors the medical image and performs abnormality detection within a segment determined by the first image processing unit to be subjected to abnormality detection; a notification unit that notifies an operator of an abnormality when the second image processing unit detects the abnormality; Equipped with The second image processing unit performing a convolution process using a learning model having a plurality of layers that perform convolution on the medical image input from the imaging unit, any one of the plurality of layers being a filter layer incorporating a plurality of predetermined linear or nonlinear filters; A surgical support device characterized in that the learning model is determined by weighted learning using supervised data in which multiple medical images taken in advance of areas where the segment and the anatomical structure match are divided into abnormal values ​​and normal values ​​using internal parameters that represent the appropriateness of each of the multiple medical images.

2. The surgery assistance device according to claim 1, A surgical assistance device, characterized in that the appropriateness changes depending on the degree of exposure of fat within the segment.

3. The surgery assistance device according to claim 1, The medical image includes color information of at least three parameters of RGB, A surgery support device characterized in that the appropriateness changes depending on at least one of two parameters other than R among the color information within the segment.

4. The surgery assistance device according to claim 1, A surgery support device characterized in that the notification unit notifies the operator of an abnormality, and the second image processing unit overlays and displays the position of the segment in which it is determined that the abnormality has occurred.

5. The surgery assistance device according to claim 1, The surgery support device is characterized in that the imaging unit is attached to an endoscope that can be inserted into a body cavity of a human body.

6. a first image processing step of segmenting a medical image obtained using an imaging means based on anatomical structures to distinguish between portions for which abnormality detection is to be performed and portions for which abnormality detection is not to be performed; a second image processing step of monitoring the medical image and detecting abnormalities in a segment determined to be subject to abnormality detection in the first image processing step; a notifying step of notifying an operator of an abnormality when the second image processing step detects an abnormality; A surgery assistance program that executes the above. The second image processing step includes: performing a convolution process using a learning model having a plurality of layers that perform convolution on the medical image input from the imaging means, any one of the plurality of layers being a filter layer incorporating a plurality of predetermined linear or nonlinear filters; A surgical assistance program characterized in that the learning model is determined by weighted learning using supervised data in which multiple medical images taken in advance of areas where the segment and the anatomical structure match are divided into abnormal values ​​and normal values ​​using internal parameters that represent the appropriateness of each of the multiple medical images.

Images