Image processing system, image processing method, and information storage medium

Abstract

To provide an image processing system, etc. capable of presenting a traction state of a tissue by image deformation analysis.SOLUTION: An image processing system 100 includes a processor 110 for sequentially acquiring time-series images captured by an endoscope 500 and performing deformation analysis processing for the time-series images. In the deformation analysis processing, the processor 110 arranges an evaluation mesh including a plurality of analysis points for an image at optional timing of the time-series images, deforms the evaluation mesh in each image so that each analysis point in each image of the time-series images tracks the feature point of a subject positioned at each analysis point in the image at the optional timing where the evaluation mesh is arranged, calculates a deformation amount of each cell in the evaluation mesh on the basis of the magnitude of a moving amount and the direction of each analysis point in each image, and presents information on the deformation of the evaluation mesh on the basis of the calculated deformation amount.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing system, an image processing method, an information storage medium, and the like. [Background technology]

[0002] Patent Document 1 discloses a method for remote control in surgery. In this method, a sensor detects a force applied to tissue, and the system uses the force detected by the sensor to provide haptic feedback to the user's operation steps. Patent Document 2 discloses a visual-haptic system for a robotic surgery platform. The system uses a visual-touch model to refer to an image and classify the image into a set of force levels. The visual-haptic model is machine-learned to classify the image into a set of force levels using a force level applied to the tissue and a video showing the tissue. This document also discloses that the system maps the visual appearance of the image to a force level, the force level includes the tightness of the knot of the thread, the force level includes a tension level, and the system generates a skill score. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2013 / 0041368 [Patent Document 2] US Patent Application Publication No. 2021 / 0322121 Summary of the Invention [Problem to be solved by the invention]

[0004] In a treatment using an endoscope, it is desirable that the treatment is performed with an appropriate tension applied to the treatment target. At this time, there is a problem of how to detect whether or not an appropriate tension is applied to the treatment target. The above two documents disclose a method for detecting the force itself. However, the method disclosed in Patent Document 1 is premised on a sensor detecting the force applied to the tissue, and it is not possible to detect the force or return tactile feedback without using a sensor. In addition, Patent Document 2 requires a huge learning cost for machine learning to classify the force level from an image. In addition, this document does not disclose a specific presentation method that is useful to the surgeon. In addition, in this document, it is necessary to use force information that cannot be obtained from an image alone in the learning stage. [Means for solving the problem]

[0005] One aspect of the present disclosure relates to an image processing system that includes a processor that sequentially acquires time-series images taken by an endoscope and performs a deformation analysis process on the time-series images, wherein in the deformation analysis process, the processor places an evaluation mesh including a plurality of analysis points on an image of an arbitrary timing among the time-series images, deforms the evaluation mesh in each of the images so that each analysis point in each image of the time-series images tracks feature points of a subject located at each analysis point in the image of the arbitrary timing in which the evaluation mesh is placed, calculates the amount of deformation of each cell of the evaluation mesh based on the magnitude and direction of movement of each analysis point in each image, and presents information regarding the deformation of the evaluation mesh based on the calculated amount of deformation.

[0006] Another aspect of the present disclosure relates to an image processing method comprising the steps of: sequentially acquiring time-series images taken by an endoscope; placing an evaluation mesh including a plurality of analysis points on an image of an arbitrary timing among the time-series images; deforming the evaluation mesh in each of the images so that each analysis point in each image of the time-series images tracks feature points of a subject located at each analysis point in the image of the arbitrary timing in which the evaluation mesh is placed; calculating the amount of deformation of each cell of the evaluation mesh based on the magnitude and direction of movement of each analysis point in each image; and presenting information regarding the deformation of the evaluation mesh based on the calculated amount of deformation.

[0007] Yet another aspect of the present disclosure relates to a non-transitory information storage medium storing a program that causes a computer to execute the following steps: sequentially acquiring time-series images taken by an endoscope; placing an evaluation mesh including a plurality of analysis points on an image at an arbitrary timing among the time-series images; deforming the evaluation mesh in each image so that each analysis point in each image of the time-series images tracks feature points of the subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is placed; calculating the amount of deformation of each cell of the evaluation mesh based on the magnitude and direction of movement of each analysis point in each image; and presenting information regarding the deformation of the evaluation mesh based on the calculated amount of deformation. [Brief description of the drawings]

[0008] [Figure 1] 1 is an example of an endoscopic surgery scene to which the method of this embodiment can be applied. [Diagram 2] 1A to 1C are diagrams for explaining a basic method for presenting the traction state of tissue in this embodiment. [Diagram 3] 1A to 1C are diagrams for explaining a basic method for presenting the traction state of tissue in this embodiment. [Figure 4] 1A to 1C are diagrams for explaining a basic method for presenting the traction state of tissue in this embodiment. [Diagram 5] An example of a medical system configuration. [Figure 6] An example of the configuration of an image processing system. [Figure 7] 3 is a flow chart showing a process performed by the image processing system. [Figure 8] 11 is a table showing a determination target and a determination method in a towing scene determination step. [Figure 9] An example of signal behavior. [Figure 10] FIG. 13 is a diagram illustrating deformation analysis when recognizing that tissue has begun to move. [Figure 11] 11 is a table showing a towing range estimation and a determination method in the towing range estimation step. [Figure 12] 1A to 1C are diagrams illustrating a method for estimating an area where tissue is under tension. [Figure 13] 11A and 11B are diagrams for explaining a method for detecting an area where excessive tension is applied. [Figure 14] FIG. 13 is a diagram for explaining a method for estimating the traction range when non-homogeneous tissues are mixed. [Figure 15] The relationship between deformation and stress in stretchable and non-stretchable tissues. [Figure 16] 11A and 11B are diagrams illustrating another method for estimating the towing range. [Figure 17] FIG. 13 is a diagram for explaining the other technique in detail. [Figure 18] FIG. 13 is a diagram for explaining the other technique in detail. [Figure 19] FIG. 13 is a diagram for explaining the other technique in detail. [Figure 20] FIG. 13 is a diagram for explaining the other technique in detail. [Figure 21] 11 is a table showing the processing and presentation method for presentation in the presentation information processing step. [Figure 22] A concrete example of how to present it. [Diagram 23] A concrete example of how to present it. [Figure 24] A concrete example of how to present it. [Diagram 25] A concrete example of how to present it. [Figure 26] A concrete example of how to present it. [Figure 27] A concrete example of how to present it. [Figure 28] A concrete example of how to present it. [Figure 29] A concrete example of how to present it. [Diagram 30] A concrete example of how to present it. [Diagram 31] A concrete example of how to present it. [Diagram 32] A concrete example of how to present it. [Diagram 33] A concrete example of how to present it. [Diagram 34] A concrete example of how to present it. [Diagram 35] A concrete example of how to present it. [Diagram 36] A diagram explaining a situation in which the evaluation area spans the traction tissue and the background. [Figure 37] FIG. 11 is a diagram for explaining a method according to a second embodiment. [Figure 38] 13 shows an example of the configuration of a medical system according to a second embodiment. [Figure 39] 13 is a processing flow of an image processing system according to a second embodiment. [Diagram 40] 13 shows a processing example of an image processing system according to the second embodiment. [Diagram 41] An example of excluding areas with no movement. [Diagram 42] An example of narrowing down results using segmentation. [Diagram 43] An example of filtering using contour detection. [Diagram 44] An example of filtering using depth information. [Diagram 45] An example of refinement using towing direction detection. [Figure 46] 13 shows a first detailed configuration example of an evaluation area setting unit. [Figure 47] 4 is a process flow executed by a first detailed configuration example of an evaluation area setting unit. [Figure 48] An example of a movement threshold. [Figure 49] 13 shows a second detailed configuration example of the evaluation area setting unit. [Figure 50] 13 is a process flow executed by a second detailed configuration example of the evaluation area setting unit. [Figure 51] 13 shows a third detailed configuration example of the evaluation area setting unit. [Figure 52] 4 shows a fourth detailed configuration example of the evaluation area setting unit. [Diagram 53] An example of a method for selecting an evaluation area. [Figure 54] 5 shows a fifth detailed configuration example of the evaluation area setting unit. [Figure 55] 6 shows a sixth detailed configuration example of the evaluation area setting unit. [Figure 56] 13 is a process flow executed by a sixth detailed configuration example of the evaluation area setting unit. [Figure 57] 7 shows a seventh detailed configuration example of the evaluation area setting unit. [Figure 58] 13 is a process flow executed by the seventh detailed configuration example of the evaluation area setting unit. [Figure 59] FIG. 13 is a diagram illustrating how the evaluation area is gradually narrowed down over time. [Figure 60] 8 shows an eighth detailed configuration example of the evaluation area setting unit. [Figure 61] 13 is a process flow executed by the eighth detailed configuration example of the evaluation area setting unit. [Figure 62] 9 shows a ninth detailed configuration example of the evaluation area setting unit. [Figure 63] 13 is a process flow executed by a ninth detailed configuration example of the evaluation area setting unit. [Figure 64] An example of threshold shift. [Figure 65] An example of expanding the evaluation area. [Figure 66] 10 shows a detailed configuration example of the evaluation area setting unit. [Figure 67] 13 is a process flow executed by a tenth detailed configuration example of the evaluation area setting unit. [Figure 68] Examples of evaluation area options. [Figure 69] Examples of evaluation area options. [Figure 70] Examples of evaluation area options. [Figure 71] 13 is an example of setting an evaluation area near the tip of a treatment tool. [Figure 72] 13 is a detailed flow of image processing steps in the third embodiment. [Figure 73] Detailed flow of the space analysis steps. [Figure 74] FIG. 13 is an explanatory diagram of a non-interest tissue recognition step. [Figure 75] FIG. 4 is an explanatory diagram of a flow vector calculation step. [Figure 76] FIG. 4 is an explanatory diagram of an evaluation mesh analysis step. [Figure 77] FIG. 4 is an explanatory diagram of an evaluation mesh update step. [Figure 78] FIG. 13 is an explanatory diagram of a modified example of the evaluation mesh analysis step. [Figure 79] FIG. 13 is an explanatory diagram of a modified example of the evaluation mesh analysis step. [Figure 80] FIG. 13 is an explanatory diagram of a modified example of the evaluation mesh updating step. [Figure 81] An explanatory diagram of the basic concept. [Figure 82] An example of a pre-registered pattern. [Figure 83] 13 is a processing flow of an image processing system according to a fourth embodiment. [Figure 84] Example of training data and recognition results for device detection. [Figure 85] Example of training data and recognition results for contact detection. [Figure 86] FIG. 4 is an explanatory diagram of a tissue deformation recognition step. [Figure 87] Example of display on main monitor and sub monitor. [Figure 88] An example of information presented showing tissue laxity. [Figure 89] First flow of loosening presentation. [Figure 90] The second flow of loosening presentation. [Figure 91] An example of information displayed when assisting surgeon traction. [Figure 92] First flow of surgeon traction support. [Figure 93] An example of the relationship between the amount of deformation and the brightness or darkness of a color. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following disclosure, many different embodiments and examples for implementing different features of the presented subject matter are provided. Of course, these are merely examples and are not intended to be limiting. Furthermore, in the present disclosure, reference numbers and / or letters may be repeated in various examples. Such repetition is for the sake of brevity and clarity, and does not necessarily have a relationship between the various embodiments and / or configurations described. In the following disclosure, the terms "unit" and "step" are interchangeable. For example, when a processor is described as executing a "step", the processor may include a "unit" as a hardware or software module for executing the "step". Vice versa.

[0010] 1.Method FIG. 1 shows an example of an endoscopic surgery scene to which the method of the present embodiment can be applied, in which an operator and an assistant work together to perform energy treatment while traction of tissue. As shown in step S1, the assistant grasps the tissue 1 using forceps 11 and 12, and expands the treatment area by traction of the tissue 1. As shown in step S2, after the assistant expands the treatment area of ​​the tissue 1, the operator grasps and traction the tissue using forceps 15. That is, the operator separates the treatment target of the tissue 1 from the surroundings and traction it to ensure safety. As shown in step S3, after the operator has been able to traction the treatment target of the tissue 1, the operator uses the energy treatment tool 16 to treat the tissue 1 within the traction range. As shown in step S4, an example of treatment is incision, in which a new surface 1b of the tissue appears due to the incision, thereby creating a loose surface 1a in the tissue 1.

[0011] In this way, if the tissue 1 becomes loose due to the treatment, the assistant or the surgeon retracts the tissue 1. In the method of the present embodiment described below, information is presented to the user so that the traction state of the tissue 1 in at least one of steps S1 to S4 can be determined. The user is, for example, a surgeon, and the surgeon includes the surgeon and the assistant. The information may be presented to both the surgeon and the assistant, or to either the surgeon or the assistant.

[0012] A basic method for presenting the traction state of tissue in this embodiment will be described with reference to Figs.

[0013] FIG. 2 shows the behavior of the tissue when it is pulled. The forceps 21 for pulling the tissue may be operated by either the surgeon or an assistant. As shown in step S11, before the tissue 2 is pulled, there is slack in the tissue 2 due to the residual stress or its own weight. When the tissue 2 is pulled by the forceps 21 from this state, the slack in the tissue 2 is removed for a while. As shown in step S12, after the slack in the tissue 2 is removed, when the tissue 2 is further pulled by the forceps 21, the tissue 2 begins to stretch or distort. As shown in step S13, when the tissue 2 is further pulled by the forceps 21, a tension region 2a appears where the stretching due to the traction stops and the tissue 2 becomes taut. The tension region 2a is a region where the stress of the tissue 2 rises and a tension suitable for the treatment is applied to the tissue 2, as described in FIG. 3. The tissue 2 around the tension region 2a stretches in response to the traction, and the tissue 2 around it translates in response to the traction. As shown in step S14, when the tissue 2 continues to be pulled by the forceps 21, a region around the tension region 2a in step S13 is newly added to the tension region 2a, and the surrounding tissue 2 becomes a region that stretches in response to the pulling.

[0014] Thus, there is a relationship between the traction of the tissue 2, the deformation of the tissue 2, and the tension applied to the tissue 2. In this embodiment, this relationship is utilized to present traction information to the user without directly detecting the force applied to the tissue 2. That is, by detecting the deformation of the tissue 2 from an image and presenting information related to the deformation to the user, the user can determine the traction state of the tissue 2 or the tension applied to the tissue 2 by looking at the presented information. For example, the surgeon can confirm that the treatment target area is included in the tension area 2a from the presented information related to the deformation, and can treat the treatment target area with the energy treatment tool.

[0015] FIG. 3 shows the relationship between the deformation amount and stress of tissue. The deformation amount may vary as described later, but here, an example is shown in which ΔL / L0 is used as the deformation amount. ΔL / L0 is the ratio of tissue elongation ΔL to traction distance L0. The change Δb2 in ΔL / L0 relative to stress change Δa in the tension region is smaller than the change Δb1 in ΔL / L0 relative to stress change Δa in the non-tension region. In other words, the deformation amount of tissue relative to traction is smaller in the tension region and larger in the non-tension region. This is as explained in FIG. 2, and shows that there is a relationship between the traction of tissue 2, the deformation of tissue 2, and the tension applied to tissue 2.

[0016] FIG. 4 shows a method of detecting tissue deformation in this embodiment. Time t1 to t3 are arbitrary times, t2 is a time after t1, and t3 is a time after t2. For example, times t1 to t3 are fixed intervals such as frames of a video. The image processing system of this embodiment sets an evaluation mesh in an endoscopic image at time t1. Time t1 may be arbitrary timing. The evaluation mesh is also called a lattice or grid. Points to be analyzed that are discretely arranged in the evaluation mesh, that is, each point where the horizontal and vertical lines of the evaluation mesh intersect, is called an analysis point. The analysis point is also called a lattice point or grid point. An area in the evaluation mesh with the nearest four analysis points as vertices, that is, an area surrounded by the horizontal and vertical lines of the evaluation mesh, is called a cell. Note that the direction of the lines defining the evaluation mesh is not limited to vertical and horizontal. The image processing system moves the analysis point by tracking the deformation of the object on the image at times t2 and t3 as if the analysis point was attached to the object. This tracking is achieved by motion detection using image features, such as optical flow detection.

[0017] The deformation amount of the evaluation mesh represents the deformation amount of the tissue due to traction. In this embodiment, the deformation amount of the evaluation mesh is detected from an image without using a sensor for detecting force, and a display according to the deformation amount is performed. As described above, the deformation amount of the tissue is related to the traction state or tension, and the user can determine the traction state or tension of the tissue by looking at the display according to the deformation amount.

[0018] The deformation amount of the evaluation mesh is a quantity defined for each analysis point or each cell, and is not limited to a scalar quantity, but may be a quantity represented by a vector or tensor, for example, displacement, movement, stretch, or strain.

[0019] An example of the deformation amount is the displacement of each analysis point. The displacement here may be only the magnitude of the displacement, or may be a vector including the magnitude and direction of the displacement. The displacement may be a displacement based on the position of the analysis point at a certain time point, or may be a displacement at a predetermined interval such as a frame. The displacement may also be a relative displacement with respect to the positions of surrounding analysis points. Note that the displacement of a point and the movement of a point are synonymous, so in this disclosure, the displacement and movement of a point are used interchangeably, and displacement and movement are interchangeable.

[0020] Another example of the deformation amount is the expansion or contraction of a cell. The expansion or contraction here is a change in the length of a side of the cell, or a change in the distance between two opposing sides of the cell. Alternatively, the expansion or contraction may be expressed by a strain component such as a principal strain. The expansion or contraction may be an expansion or contraction based on the shape of the cell at a certain point in time, or may be an expansion or contraction at a predetermined interval such as a frame.

[0021] Yet another example of the deformation amount is the strain of a cell. The "strain" here may be a tensor amount represented by multiple components, or a part of the multiple components contained in the tensor amount, or a strain in a specific direction such as a principal strain or a minor strain. Alternatively, the deformation amount may be the amount of change in the strain of a cell. The amount of change in the strain may be the amount of change in the strain when the strain of an evaluation mesh at a certain point in time is used as a reference, or may be the amount of change in the strain at a predetermined interval such as a frame.

[0022] The deformation amount does not have to be the above amount itself, but may be an amount calculated using the above various amounts. For example, the deformation amounts of several surrounding analysis points or cells may be averaged, or the deformation amount of the analysis point or cell may be subjected to a time or space filter. Note that the description of "deformation amount" and the descriptions of specific examples such as "displacement," "movement," "stretch," and "strain" used in the following description are mutually interchangeable.

[0023] First to fifth embodiments using the above-mentioned technique will be described below. The contents of the first to fifth embodiments can be combined as appropriate. For example, the contents of one or more of the second to fifth embodiments may be combined with the contents of the first embodiment. Furthermore, even if the description of the configuration, processing, etc. in one embodiment is omitted, the contents of the configuration, processing, etc. described in the other embodiments can be applied.

[0024] 2. First embodiment 5 shows an example of the configuration of a medical system. The medical system includes an endoscope 500, an image processing system 100, and a monitor 700.

[0025] The endoscope 500 is inserted into the patient's body, captures images of the inside of the body, and transmits the image data to the image processing system 100. The endoscope 500 captures images in time series, which will be referred to as time-series images. Each image included in the time-series images will be referred to as an endoscopic image. The time-series images are, for example, endoscopic images of each frame of a video captured by the endoscope 500, or endoscopic images extracted at a predetermined interval from the video. The endoscope 500 may be a rigid endoscope such as a laparoscope or an arthroscope, or a flexible endoscope such as a gastrointestinal endoscope.

[0026] The image processing system 100 detects the amount of deformation of tissue from the endoscopic image, and displays a display corresponding to the amount of deformation on the endoscopic image on the monitor 700. The image processing system 100 may process the endoscopic image captured by the endoscope 500 in real time. Alternatively, the endoscopic image may be recorded in a storage such as a hard disk drive or a non-volatile memory, and the image processing system 100 may process the endoscopic image recorded in the storage. Note that the endoscopic system usually includes the endoscope 500 and an endoscope control device, and the image processing system 100 may be incorporated in the endoscope control device. Alternatively, the image processing system 100 may be a system provided separately from the endoscope control device. In this case, the endoscope control device may generate an endoscopic image from an image signal of the endoscope 500, and output the endoscopic image to the image processing system 100.

[0027] The medical system may include a plurality of monitors, and the image processing system 100 may display different information on each monitor. For example, the medical system may include a first monitor and a second monitor, and the first monitor may display an original endoscopic image, and the second monitor may display an endoscopic image on which information corresponding to the analysis result of the evaluation mesh is superimposed. Alternatively, a so-called picture-in-picture display may be used, in which information corresponding to the analysis result is displayed so as not to overlap with the original endoscopic image in the first monitor. Alternatively, both the first monitor and the second monitor may display an endoscopic image on which information corresponding to the analysis result of the evaluation mesh is superimposed. In this case, the first monitor may display an endoscopic image on which information for the assistant is superimposed, and the second monitor may display an endoscopic image on which information for the operator is superimposed.

[0028] 6 shows an example of the configuration of an image processing system 100. The image processing system 100 includes a processor 110, a memory 120, and a communication unit 160. The image processing system 100 may also include an operation unit 150. The image processing system 100 may be an information processing device such as a PC or a server, or may be a cloud system in which a plurality of information processing devices are connected via a network.

[0029] The memory 120 stores a program 121 in which the contents of the processing executed by the image processing system 100 are described. The processor 110 performs various processing by reading and executing the program 121 from the memory 120. For example, the processor 110 executes the processing of each step described later in FIG. 7. When machine learning is used for the processing, the memory 120 may store a trained model 122 obtained by machine learning. The processor 110 executes the processing using machine learning using the trained model 122. The trained model 122 does not necessarily mean only a model that has been trained specifically to execute the present embodiment, but may be an existing model that has been trained for general purposes. The trained model 122 may include a plurality of machine learning models. The processor 110 and the memory 120 may be configured as follows, for example.

[0030] The processor 110 includes hardware. The processor 110 is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microcomputer, or a digital signal processor (DSP). Alternatively, the processor 110 may be an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor 110 may be one or more of a CPU, a GPU, a microcomputer, a DSP, an ASIC, and an FPGA. The memory 120 is, for example, a semiconductor memory that is a volatile memory or a non-volatile memory. Alternatively, the memory 120 may be a magnetic storage device such as a hard disk device, or an optical storage device such as an optical disk device. The trained model 122 stored in the memory 120 may include, for example, a program in which an AI (Artificial Intelligence) algorithm is written, and data used in the program. For example, the trained model 122 may include a neural network such as a convolutional neural network (CNN). In this case, the trained model 122 includes a program in which the algorithm of the neural network is described, and weight parameters and biases between nodes of the neural network. The neural network includes an input layer to which data is input, an intermediate layer that performs arithmetic processing on the data input through the input layer, and an output layer that outputs a recognition result based on the arithmetic result output from the intermediate layer. The program 121, the trained model 122, or both the program and the trained model 122 may be stored in a non-transitory information storage medium that is a computer-readable medium. The information storage medium is, for example, an optical disk, a memory card, a hard disk drive, or a semiconductor memory. The semiconductor memory is, for example, a ROM or a non-volatile memory. The processor 110 loads the program 121 stored in the information storage medium into the memory 120 and performs various processes based on the program 121.

[0031] The communication unit 160 communicates with the outside of the image processing system 100. The communication unit 160 may include, for example, a connector or interface for connecting the endoscope 500, or may include a connector or interface for connecting the monitor 700, or may include an interface for network connection such as a LAN.

[0032] The operation unit 150 accepts operation input from a user to the image processing system 100. The operation unit 150 is, for example, a button, a switch, a dial, a lever, a keyboard, or a pointing device. Alternatively, the operation unit 150 may be realized by a touch panel provided on the monitor 700.

[0033] The processor 110 may mix machine learning processing and rule-based processing. Examples are given below. However, even if it is described as an example of machine learning processing, it may be realized by rule-based processing, and even if it is described as an example of rule-based processing, it may be realized by machine learning processing.

[0034] Example of processing using machine learning -Detection of the position of the treatment tool within the image. - Region detection of the treatment tool in the image, to ignore the treatment tool that interferes with the analysis point tracking. - Recognizing the type of treatment tool in the image. This is to distinguish whether the treatment tool is being operated by the surgeon or assistant, whether it is being operated by the right hand or the left hand, or whether the treatment tool is a forceps or an energy treatment tool. Detection of the contact state or gripping state between the treatment tool and the biological tissue. Changes in the contact state or gripping state are used as triggers. The triggers are the start of analysis, the end of analysis, the start of presentation, or the end of presentation. -Estimation of evaluation area. This is to estimate the analysis range. The evaluation area is the area where the evaluation mesh is set. - Identification of tissue types within the traction range. This is to determine in advance whether the tissue has a different elasticity.

[0035] Rule-Based Processing Example Feature point tracking. This is to analyze deformation over time using methods such as optical flow. Information obtained by optical flow is analyzed using rule-based processing to track feature points. For the optical flow itself, a classical method may be used, or AI-based RAFT (Recurrent All-Pairs Field Transforms) may be used. Various correction processes. For example, correction of changes in camera angle, translation, enlargement / reduction, rotation, or camera shake. Or correction of drift, which is when the analysis results move on their own due to noise.

[0036] 7 shows a flow of processing executed by the image processing system 100. In step S31, an endoscopic image is input to the processor 110. The processor 110 executes an image processing step in step S32, and outputs the result of the image processing step to the monitor 700 in step S33.

[0037] The image processing step S32 includes a towing scene determination step S21, a towing range estimation step S23, and a presentation information processing step S25. In step S21, the processor 110 determines whether the scene is one in which the target towing range should be estimated from an endoscopic image or the like. That is, the processor 110 determines the timing to start and end the estimation of the towing state. If the processor 110 determines that the scene is one in which the towing range should be estimated (T: True), it estimates the towing range in step S23. If the processor 110 determines that the scene is not one in which the towing range should be estimated (F: False), it transmits information to step S25 that the estimation of the towing range in step S23 will not be performed or the presentation information will not be given. In step S23, the processor 110 calculates the amount of movement of the tissue in the image, makes the analysis point follow the movement of the tissue, and estimates the towing range from the information of the evaluation mesh. In step S25, the processor 110 processes the estimation result of the towing range as information required by the user, and processes the presentation information so that it can be presented on the monitor useful to the user. Between steps S21 and S23, the processor 110 may execute a region setting step of setting a region for estimating the traction state.

[0038] The towing scene determination step S21 will now be described in detail. Fig. 8 shows a table of determination targets and determination methods in the towing scene determination step S21. "UI" in the table is an abbreviation for user interface.

[0039] The output of this determination step may be a label in information processing indicating that it is a towing scene, or may be a specific signal such as an electrical pulse. Furthermore, if it is not a towing scene, the processor 110 may skip the processing of the towing range estimation step without assigning any label in information processing, or may execute the processing of the towing range estimation step in a state where a label in information processing indicating that it is not a towing scene is assigned.

[0040] Each item in the table in Fig. 8 will be explained. Numbers in parentheses, such as (1), indicate embodiments. Items with the same number are corresponding embodiments, for example, (1) in "object to be judged" and (1) in "judgment method" are corresponding embodiments. * indicates a modified example, for example, (1*) is a modified example of (1).

[0041] "Subject to judgment"

[0042] (1) The processor 110 recognizes the grasping of tissue by the forceps of the surgeon as a traction scene.

[0043] (2) The processor 110 recognizes a scene in which at least one assistant forceps grasps tissue as a traction scene.

[0044] (1*)(2*) The processor 110 may recognize the tissue starting to move as a traction scene.

[0045] (3) The processor 110 detects the contact of the energy treatment tool with tissue as a traction scene.

[0046] (3*) The processor 110 detects a signal action from an endoscopic image to determine a traction scene. Examples of signal actions are shown in A1 to A4 in FIG. 9. A1 is a signal action in which the surgeon grasps the tissue 1 with the forceps 15 and the energy treatment tool 16 is within the field of view after the assistant deployment is started. A2 is a signal action in which the surgeon crosses and brings two treatment tools such as the forceps 15 and the energy treatment tool 16 into contact with each other. A3 is a signal action in which the surgeon grasps the space that is not tissue twice with the forceps 15. A4 is a signal action in which the surgeon reciprocates the shafts of two treatment tools such as the forceps 15 and the energy treatment tool 16 twice.

[0047] "Judgment method"

[0048] (1)(2) The processor 110 uses a process for detecting a treatment tool from an endoscopic image and a process for detecting contact between the treatment tool and tissue or grasping of tissue by the treatment tool to recognize the “object to be determined” (1)(2).

[0049] (3) The processor 110 uses a process for detecting a treatment tool from an endoscopic image and a process for detecting contact between the treatment tool and tissue or a process for detecting gripping of tissue by the treatment tool to recognize the "object of determination" (3). Specifically, the processor 110 detects an energy treatment tool from an endoscopic image and detects contact of the energy treatment tool with tissue from the endoscopic image.

[0050] (3*) The processor 110 uses a process for detecting a signal action from an endoscopic image to recognize the "object of judgment" (3*). Since the assistant pulls the tissue with forceps in both hands, there is a problem that the assistant cannot convey to the system that the assistant forceps have completed the opening of the surgical field by expressing the forceps movement in the image. Therefore, the processor 110 can make a judgment such as (3) or (3*) to practically determine that the assistant forceps have completed the opening of the surgical field.

[0051] (1*)(2*) The processor 110 uses a process of detecting a treatment tool from an endoscopic image, a process of detecting the gripping of tissue by a treatment tool from an endoscopic image, and a process of analyzing the deformation amount of tissue from an endoscopic image to recognize the "object of judgment" (1*)(2*). The analysis of the deformation amount is performed as follows. The processor 110 constantly analyzes the deformation amount using an AI-based optical flow method such as RAFT, and recognizes that the tissue has started to move when the deformation amount exceeds a threshold indicating that the tissue slack is removed. For example, the displacement of the analysis point and the strain of the cell are used as the deformation amount. As shown in B1 of FIG. 10, when the shape of a cell connecting four analysis points of the deformation analysis is distorted in one direction like cell 41b compared with cell 41a of the past frame, it is considered to be a deformation that removes slack. As shown in B2, in order to detect the deformation, for example, when the principal strain with the pulling direction 30 as the main component exceeds a threshold, it can be determined that it is a traction scene. In addition, since it is sufficient to detect the movement of the tissue, the relative strain or displacement amount may be compared with a threshold value. The deformation analysis method is not limited to AI-based optical flow, and various other methods such as classical optical flow may be used.

[0052] The towing range estimation step S23 will be described in detail. FIG. 11 shows the towing range estimation and the determination method in the towing range estimation step S23 in a table. Numbers in parentheses such as (1) indicate embodiments. Items with the same numbers in FIG. 8 and FIG. 11 correspond to each other. For example, (1) in FIG. 8 and (1) in FIG. 11 correspond to each other. * indicates a modified example. For example, (1) in FIG. 8 may correspond to (1*) in FIG. 11, and (1*) in FIG. 8 may correspond to (1) in FIG. 11. * may be further followed by an alphabet or the like, but this is for convenience in distinguishing items, and for example, (1*a) is still a modified example of (1). For example, (1) and (1*) in FIG. 8 may correspond to all modified examples with "1*" such as (1*a) as well as (1) and (1*) in FIG. 11.

[0053] "Estimated towing range"

[0054] (2) The processor 110 estimates the range in which the traction has been applied to the tissue, i.e., the range in which the tissue has been deformed, as the traction range.

[0055] (1) The processor 110 estimates the range in which the tissue is taut, i.e., the range in which the tissue is under tension, as the traction range.

[0056] (3) The processor 110 estimates the range where the traction has been relaxed, that is, the range where the deformation, etc. of the tissue that was deformed, etc. by the traction has been relaxed, as the traction range.

[0057] (1*b) When tissues having non-uniform responses to traction are present, the processor 110 estimates the range in which tension is applied to the tissue as the traction range using a determination method described later.

[0058] (1*c) The processor 110 estimates the area where the tissue is under excessive tension as the retraction range.

[0059] "Judgment method"

[0060] (1) In (1) of the "traction estimation range", the processor 110 estimates the range in which the given principal strain decreases below the threshold value after the maximum principal strain representing the strain in the traction direction exceeds the threshold value due to traction by the forceps of the left hand of the surgeon as the region in which the tissue is under tension. The given principal strain is the minimum principal strain if the analysis space is two-dimensional. Alternatively, the given principal strain is another principal strain in an appropriate two-dimensional plane including the maximum principal strain, or a principal strain in a plane perpendicular to the maximum principal strain, or a second principal strain if the analysis space is three-dimensional. As shown in C1 of FIG. 12, the cell 42a is deformed to become like the cell 42b as the tissue is stretched by traction. The cell 42b shows a state in which the principal strain having the traction direction 30 as the main component has become large. Such deformation occurs until the tissue is expanded by traction. As shown in C2, when the tissue is under tension, the cell 43a becomes like the cell 43b due to traction, but the deformation is small. In this state, the cell extends in the traction direction 30 and shrinks in a direction perpendicular to the traction direction 30. That is, the maximum principal strain representing the strain in the traction direction 30 increases, while the principal strain in a direction perpendicular to the traction direction 30 decreases. The processor 110 judges these by a threshold value. In the judgment, it is desirable to analyze the image excluding disturbances caused by the camera. The disturbances include the enlargement / reduction, rotation, translation of the camera, and perspective distortion when viewing a solid object as a 2D image. In the above judgment, only the maximum principal strain may be used.

[0061] (1*a) The processor 110 may estimate the area where the tissue is under tension in the "traction estimation range" (1) by using the change in the ratio of the elastic modulus in the plane of the tissue. Elastic modulus = stress / strain. That is, as shown in C2 of FIG. 12, the elastic modulus becomes large in the area where the tissue is under tension, so it is sufficient to detect, for example, that the elastic modulus becomes larger than a threshold value.

[0062] (2) The processor 110 detects the range of traction by the assistant's forceps by analyzing the deformation of the tissue in (2) of the "estimated traction range". For example, the same analysis method as that in FIG. 10 can be adopted. That is, when the principal strain having the traction direction 30 as the main component exceeds a threshold, the cell can be determined to be included in the traction range. The relative strain or the amount of displacement can be compared with the threshold. When the forceps are not always reflected in the field of view, that is, when the forceps are out of the frame, a point near the position where the forceps were last reflected in the past and where the displacement is maximum can be redefined as a virtual traction position and used for the analysis of the traction direction, or the forceps can be estimated and placed at a coordinate position outside the screen and used for calculation.

[0063] (3) As described in steps S3 and S4 of FIG. 1, when a new surface appears in the tissue due to the energy incision, if the traction by the assistant forceps is maintained in the same state, the tissue may become loose. For this reason, the range where the traction has loosened is estimated as in (3) of the "estimated traction range". Specifically, the processor 110 uses the results of the deformation analysis in the past few seconds to analyze the difference in the amount of deformation, and estimates the range where the traction has loosened based on the difference. Here, an example in which the amount of deformation is the principal strain is described. The processor 110 estimates the range where the principal strain of the cell of the evaluation mesh has changed in the direction in which the tissue shrinks as the area where the tissue may have loosened. Specifically, the processor 110 calculates the amount of decrease in the principal strain of the cell at each timing thereafter, based on the principal strain of the cell at the timing when the energy treatment tool contacts the tissue. When the amount of decrease in the principal strain falls below a threshold, the processor 110 estimates the range including the cell as the range where the traction has loosened. Note that new surfaces that appear after the start of the analysis due to cutting or the like may be ignored in the displacement analysis. Also, new evaluation mesh may be added to the newly appeared surfaces, and deformation analysis may be performed from that point on.

[0064] (1*c) If the traction strength is too strong, the tissue will break. Although (1) for operator support may also be a target, in robotic surgery without touch, there is a similar concern with assistant forceps or a 3rd arm where it is difficult to pay attention. For this reason, by detecting an area where excessive tension is applied as in (1*c) of the "traction estimation range", the area can be separately displayed before the breakage for safety. Specifically, an object yields to stress before breaking. The processor 110 detects the yield based on the deformation amount of the cell. Here, an example in which the deformation amount is the principal strain will be described. The area where the tension that stretches the tissue is applied is defined by the various methods described above. When the processor 110 detects that the principal strain in the traction direction has increased by a certain amount or more within the area, it estimates that the area is an area where excessive tension is applied. This is because it is estimated that the stress of the object has yielded in the area and is about to break. 13, in the region where the tissue is under tension, the cell 44a does not deform much, but as the traction is further strengthened, the principal strain becomes large at a certain point, as in the cell 44b. The processor 110 judges this by a threshold value.

[0065] (1*b1) Consider the problem when a wide range of the inside of the body is shown in the field of view of the endoscope and the surgeon or assistant continuously traction multiple tissues that are deformed in different ways. Here, we use the example of FIG. 14 and assume that the deformation is stretching. As shown in FIG. 14, a tissue 3a that is difficult to stretch and a tissue 3b that is easy to stretch are connected, and the forceps 21 traction them from the side of the tissue 3b that is easy to stretch. If the slack of both tissues 3a and 3b is simply taken up, no stress is generated anywhere in both tissues 3a and 3b. However, once the stretching of any tissue reaches its limit, the tissue 3b that is easy to stretch bears most of the traction amount, and the tissue 3a that is difficult to stretch is hardly stretched. If we consider it in one dimension for simplicity, the traction force is applied to both tissues 3a and 3b on the axis in the same amount. Therefore, it is estimated that the stress (tension) generated in the traction range is generated largely in the tissue 3a that is difficult to stretch.

[0066] Figure 15 shows the relationship between the amount of deformation and stress for the non-stretchable tissue 3a and the easily stretchable tissue 3b. Here, an example is shown in which ΔL / L0 is used as the amount of deformation. It can be seen that ΔL / L0, or the amount of traction, until the tension region is reached is larger for the easily stretchable tissue 3b than for the non-stretchable tissue 3a.

[0067] In this way, when detecting traction in a range that cannot be approximated as homogeneous tissue, if a uniform deformation threshold is used, only stretchable tissue 3b will be detected by the threshold judgment and that area will be presented to the user. In other words, the user must be careful because their eyes will be drawn only to the stretchable tissue in the presented range. When trying to display tissue that is difficult to stretch and has a fast stress rise in response to deformation, such as thick connective tissue, in the same way as other tissues, it is not advisable to set a uniform deformation threshold.

[0068] Therefore, the processor 110 redoes the deformation analysis in (1*b) of the "traction estimation range", that is, re-divides the region. Specifically, when a region appears in which the change in the component of the principal strain in the traction direction becomes small due to traction, the processor 110 re-divides the region again with an equally spaced evaluation mesh and continues the deformation analysis. In this way, the analysis is resumed at a point where the stress against the strain is about to rise, so within the re-divided range, a small strain means the occurrence of a large stress. Therefore, if a strain occurs within a region after the region is re-divided, the region can be estimated as being under tension.

[0069] (1*b2) Consider a problem in which a wide range of the inside of the body is captured in the field of view of the endoscope, and the tissue is homogenous, but the range in which tension is generated by traction is limited due to the degree of physiological adhesion, etc. Here, the example of Fig. 16 is used. The endoscopic image shows the tissue, the forceps 21 of the surgeon or assistant, and the energy treatment tool 16 of the surgeon. The tissue is being traction-pulled in the traction direction 30 by the forceps 21.

[0070] Biological tissues have a certain buffer area against deformation such as stretching or strain, and are characterized by the fact that stress does not rise immediately even when they are pulled. Therefore, tension occurs in region 51 where no further deformation occurs as a result of traction. Here, the deformation is assumed to be stretching. Surrounding region 51 where tension occurs is region 52 which is simply stretched. Arrows 55 around region 52 indicate the direction of tissue stretching. Tissues outside region 52 translate or do not move in response to traction.

[0071] In the above-mentioned modified example (1*b1), the region is re-divided when the tissue stops stretching, but since the stretch of the tissue after re-division is very small, the stretch may be below the limit of deformation detection. This may result in a poor S / N ratio for the estimation of the traction range. In addition, it is difficult to determine whether the tissue stopped stretching simply because traction was stopped or because tension was applied.

[0072] Therefore, the processor 110 surrounds the region 51 where the tissue stretching has stopped in the (1*b) of the "traction estimation range", and if there is a region 52 that is simply stretched around it, it estimates that the region 51 where the stretching has stopped is a region where tension is applied. This makes it unnecessary to detect minute displacements, and improves the detection S / N ratio. The method of estimating the deformation amount of each cell of the evaluation mesh is as described above. When the deformation amount of a cell becomes equal to or greater than a certain amount and the deformation of the cell stops, the processor 110 determines that the hand has simply stopped if the deformation of the entire evaluation mesh has stopped, and estimates that tension is applied to the region including the cell where the deformation has stopped if the deformation of the cells surrounding the cell where the deformation has stopped continues. Further details will be described using Figures 17 to 20. In the following example, the deformation amount is assumed to be the principal strain.

[0073] As shown in FIG. 17, the processor 110 sets an evaluation mesh 60 in an endoscopic image and calculates the principal strain of each cell at each time t(n, n-1, ...). The processor 110 detects cells in which the principal strain is equal to or greater than a threshold and the change in the principal strain is small, and sets the cells as candidate regions 51a. The threshold is, for example, +25% to +60%, but is not limited thereto. The change in the principal strain is small, for example, ±5% / 0.5 seconds on a time average, but is not limited thereto. The processor 110 expands the region around the candidate region 51a and sets the expanded region as the comparison region 52a. For example, 50 pixels around the candidate region 51a is set as the comparison region 52a.

[0074] The graph in the lower diagram of FIG. 18 shows the relationship between the deformation amount of the tissue and the stress. In this graph, the horizontal axis is ΔL / L0. Since it is considered that ΔL / L0 and the principal strain of each cell are both generated in the stretching direction of the tissue, here, ΔL / L0 and the principal strain are described as being correlated. It is considered that the stretching direction of the tissue is, for example, oriented in the direction of pulling by the forceps in the candidate region, and oriented in the candidate region in the comparison region. As shown in the lower diagram of FIG. 18, the processor 110 determines that the candidate region is under tension when the time change ΔL2 of the principal strain of the comparison region is equal to or greater than a predetermined magnification with respect to the time change ΔL1 of the principal strain of the candidate region, or when the difference between ΔL1 and ΔL2 is equal to or greater than a predetermined difference. The predetermined magnification is, for example, 3 times. When the processor 110 determines that the candidate region is under tension, it presents the candidate region as a tension region 51b. For example, the processor 110 applies a color such as light blue to the tension region 51b, superimposes it on the endoscope image, and displays the endoscope image on the monitor. The processor 110 may further present the extension range by using a dotted line 52b or the like.

[0075] 19, if the time change δL2 of the principal strain in the comparison region is smaller than a predetermined ratio relative to the time change δL1 of the principal strain in the candidate region, or if the difference between δL1 and δL2 is smaller than a predetermined difference, the processor 110 estimates that the tension in the candidate region is insufficient and that traction has simply stopped. In this case, the processor 110 does not present the candidate region as a tension region.

[0076] 20, once processor 110 estimates that tension region 51b exists in an image, it continues to display tension region 51b even if the change over time in the principal strain of tension region 51b is small thereafter. For example, once processor 110 estimates that tension region 51b exists in an image, it continues to display tension region 51b until loosening of tissue is detected.

[0077] The display method of the region may be various. For example, the display of the region may be a border, a fill, or a lighting change. The border may be applied only to the outer rim of the candidate region, or to the outer periphery of the candidate region excluding the grip, or to only the end of the candidate region farthest from the grip. The fill may be semi-transparent, may be a superimposed pattern, or may change in density depending on the pulling strength. The lighting change may be lighting, blinking, gradual decrease in brightness, or disappearance over time. In addition, a display as described in the following presentation information processing step S25 may be performed.

[0078] The details of the presented information processing step S25 will be described. The processor 110 outputs at least the analysis result of the towing range as is, or outputs it after performing post-processing. The output may be image information, coordinate information, or a label for information processing to be displayed.

[0079] Fig. 21 shows a table of the processing and presentation method for presentation in the presentation information processing step S25. Numbers in parentheses such as (1) indicate embodiments, and * indicates modified examples. The correspondence between the embodiments and modified examples in Fig. 8, Fig. 11, and Fig. 21 is as explained in Fig. 11. Each item in Fig. 21 will be explained below.

[0080] (1*) The processor 110 presents all of the analysis points in the towed range. Alternatively, the processor 110 may present the towed range by changing the resolution of the analysis points in the towed range. The change in resolution may be, for example, partially thinning out the analysis points, filling in the lack of analysis points, or creating new lines in the evaluation mesh.

[0081] (1), (2), and (3) The processor 110 presents the entire outline of the retraction area. Alternatively, the processor 110 may arbitrarily select and present a portion of the outline of the retraction area. The portion of the outline may be, for example, a portion of the outline that is proximal or distal to the forceps, or a side edge of the outline.

[0082] The (1*a)(2*a)(3*a) processor 110 presents contour lines or ridge lines, or lines similar to contour lines or ridge lines, of the deformation of the tow area.

[0083] The (1*b)(2*b)(3*b) processor 110 may color the towed area. The color may be applied with a transmittance of, for example, 0 to 90%. The same color and transmittance may be applied to the entire towed area, or different colors or transmittances may be applied to each cell.

[0084] (1*c)(2*c)(3*c) Processor 110 presents, on a screen viewed by the surgeon, information limited to the contents necessary for the surgeon. Processor 110 also presents, on a screen viewed by the assistant, information limited to the contents necessary for the assistant. When the medical system includes two monitors, for example, information for the surgeon may be displayed on the main monitor, and information for the assistant may be displayed on the sub-monitor.

[0085] The (1*d)(2*d)(3*d) processor 110 extracts only the range covered by the traction forceps of the assistant, and displays that range on the screen viewed by the assistant.

[0086] (3*e) The processor 110 presents a history of past towing ranges that gradually fade and disappear. That is, the processor 110 displays the color or line, etc., of the older towing range information.

[0087] 22 to 35 show examples of the above-mentioned presentation methods. Note that, although an example in which a treatment tool such as forceps is shown in an endoscopic image is used below, the treatment tool does not necessarily have to be shown in the image.

[0088] FIG. 22 is a specific example of (1), (2), (3), and (1*b), (2*b), and (3*b). The endoscopic image shows tissue, the forceps 21 of the surgeon or assistant, and the energy treatment tool 16 of the surgeon. The processor 110 outputs the traction range in real time as image information by superimposing the range estimated as the traction range among the ranges in which the evaluation mesh was set when analyzing the traction range on the endoscopic image. The processor 110 emphasizes the edges around the frame, centering on the distal frame of the traction range by the surgeon or assistant, paints the inside of the edges semi-transparently, and outputs it to the monitor to present it to the surgeon or assistant.

[0089] 23 is a specific example of (1), (2), and (3). The processor 110 displays the outer frame of a region 71 in which the amount of deformation exceeds a threshold within an evaluation region in which an evaluation mesh is set. The amount of deformation is maximum at a side 71a proximal to the forceps 21, and is at the threshold at a side 71b distal to the forceps 21.

[0090] FIG. 24 shows specific examples of (1), (2), (3), and (1*a), (2*a), and (3*a). The processor 110 displays a line 72c connecting cells whose deformation amount is a predetermined value within the evaluation region in which the evaluation mesh is set. The line 72c corresponds to a contour line of the deformation amount. FIG. 24 shows an example in which the predetermined value is 20%, but the predetermined value may be any value that is greater than the threshold value and less than the maximum value. The outer frame of the region 72 in which the deformation amount exceeds the threshold value may or may not be displayed. Alternatively, only the side lines of the outer frame of the region 72 may be displayed, and the side 72a whose deformation amount is the maximum value and the side 72b whose deformation amount is the threshold value may not be displayed. Alternatively, the line 72c and the sides 72a and 72b may be displayed in different line types, such as a solid line and a dotted line.

[0091] 25 and 26 are specific examples of (1*b)(2*b)(3*b). The processor 110 applies a gradation color to regions 73 and 74 in which the amount of deformation exceeds a threshold within the evaluation region in which the evaluation mesh is set. The color density of the gradation is determined according to the amount of deformation, and may be defined, for example, by a sigmoid function for the amount of deformation. FIG. 25 shows an example in which the color becomes lighter from the maximum value of the amount of deformation to the threshold in the region 73, that is, the color becomes lighter from the proximal to the distal side of the forceps 21. FIG. 26 shows an example in which the color becomes darker from the maximum value of the amount of deformation to the threshold in the region 74, that is, the color becomes darker from the proximal to the distal side of the forceps 21. In FIG. 26, it is easy to visually confirm how far the traction has reached.

[0092] 27 is a specific example of (1*b)(2*b)(3*b). The processor 110 fills in an area 75 in which the amount of deformation exceeds a threshold within the evaluation area in which the evaluation mesh is set, with a certain transmittance and color.

[0093] FIG. 28 is a specific example of (1*). In FIG. 28, the analysis point where the deformation amount exceeds the threshold is indicated by a black circle 63. If only the black circle 63 is displayed or only the cell whose four corners are black circles 63 is filled and displayed, the resolution of the area display may be insufficient. Therefore, the processor 110 uses an existing method such as the center of gravity to increase the resolution of the area display. The processor 110 changes the resolution of the black circle 63 by a known algorithm, and estimates and displays the expanded towing range 76. The processor 110 may display the outline of the towing range 76, or may display the towing range 76 by filling it.

[0094] FIG. 29 is a specific example of (1*a), (2*a), and (3*a). If the evaluation mesh 62 itself is displayed, there is a possibility that the display will be overloaded with information. Therefore, the processor 110 performs a display similar to contour lines by leaving meridians as in 77a within the range in the evaluation mesh 62 where the deformation amount exceeds the threshold. Alternatively, the processor 110 performs a display similar to ridge lines by leaving latitude lines as in 77b within the range in the evaluation mesh 62 where the deformation amount exceeds the threshold. The processor 110 may display the meridians or latitude lines by approximating them with curves. Such a display reduces the amount of information necessary for the surgeon, and simplifies the inside of the presentation area, so that important anatomy can be displayed without being hidden by the presentation information.

[0095] Fig. 30 is a specific example of (3*e). When the tissue 1 is incised by the energy treatment tool 16 and a new surface 1b appears in the tissue 1, loosening occurs in the tissue 1. At this time, the processor 110 presents, as history information, the indications 79a, 79b, and 79c of the traction range that fade over time. The indication 79c is the newest and the darkest, the indication 79b is the next newest and the second darkest, and the indication 79a is the oldest and the lightest. Such a display makes it possible to always compare with the past, particularly to notice loosening of the tissue.

[0096] FIG. 31 is a specific example of (1*c), (2*c), (3*c), and (1*d), (2*d), and (3*d). In the assistant support function, the processor 110 may present information only on the assistant monitor. For example, the processor 110 may display a range 81 in which the assistant forceps 11 and 12 are pulled on the assistant monitor. In the surgeon support function, the processor 110 may present information only on the surgeon monitor. For example, the processor 110 may display a range 82 in which the surgeon forceps 15 is pulled on the assistant monitor. In addition, when the assistant support function and the surgeon support function are used together, the processor 110 may present information for the assistant on the assistant monitor and present information for the surgeon on the surgeon monitor. In addition, the information for the assistant may be utilized by an experienced surgeon for instructions, and when the assistant has more experience than the surgeon, the assistant may give instructions to the surgeon. Therefore, both pieces of information may be displayed on the same screen in different ways.

[0097] In particular, regarding the support of presenting the loosening of traction, that is, regarding (3*c) and (3*d), the following specific examples are conceivable. As the loosening of traction caused by the energy treatment, there may be a range that can be solved only by the assistant forceps 11, 12, and a range that must be solved by the surgeon himself or herself in cooperation with the surgeon forceps 15. In this case, even if the information on the loosening that can only be solved by the surgeon is presented on the monitor for the assistant, it is meaningless unless it is used for guidance. Therefore, in the support of presenting the loosening of traction, the processor 110 may present information only on the monitor for the assistant when traction by the assistant is required, and only on the monitor for the surgeon when traction by the surgeon is required. That is, only information on the range 81 that can be solved by the assistant forceps 11, 12 may be presented on the monitor for the assistant. Also, only information on the range 82 that can be solved by the surgeon forceps 15 may be presented on the monitor for the surgeon. Also, it may be merely presented so that it is clear which one can be dealt with, by labeling or color-coding each of them. The range can be roughly estimated based on, for example, the positional relationship between the forceps 11 and 12, or the positional relationship between the forceps 15, or the positional relationship between the forceps 11, 12, and 15.

[0098] The method of presenting information is not limited to the methods described with reference to FIGS.

[0099] The processor 110 may surround the outer frame of the analysis range, or may use the analyzed analysis points themselves. The processor 110 may also narrow down the analysis range to an area that may be under a certain level of tension, or may change the presentation depending on the level of tension. Alternatively, the processor 110 may focus only on areas with high tension and present them as a warning.

[0100] From another perspective, the processor 110 may display, as a result of estimating the traction region by analyzing the deformation amount of the tissue, anatomical structures that are characteristic of traction in the traction region, such as grooves, ridges, depressions, veins, or loose connective tissue that is raised, with special emphasis. A machine learning method may be used separately to detect the structures. In addition, when a difference in the tracking ability of the analysis points is observed in the analysis of the deformation amount of the tissue, important anatomical structures may be hidden in invisible deep parts in the traction region. Therefore, the processor 110 may display the regions with low tracking ability of the analysis points with emphasis so that the regions can be seen.

[0101] As shown in FIG. 32, the processor 110 may use the time when the forceps 21a or 21b grasps the tissue as the origin and display, by using a frame 91 or the like, a region in which the tissue has been deformed by traction by a certain amount or more.

[0102] 33, the processor 110 may display the direction and strength of deformation in a region where the tissue is deformed by traction by a certain amount or more, starting from when the forceps 21a or 21b grasps the tissue. For example, the processor 110 may display a color indicating the direction and strength of deformation of each cell of the evaluation mesh 92. For example, red may indicate tension, blue may indicate loosening, and the intensity of each color may indicate the strength of deformation.

[0103] As shown in FIG. 34, the processor 110 may display, using a frame 93 or the like, a region in which a certain tension is applied to the tissue by traction, starting from the point when the forceps 21a or 21b grasps the tissue.

[0104] 35, the processor 110 may display a tension intensity distribution 94 of the tissue caused by traction, starting from when the forceps 21a or 21b grasps the tissue. The intensity distribution 94 may be displayed in three levels of color, for example, large, medium, and small, or in three levels of color intensity.

[0105] In this embodiment, the image processing system 100 includes a processor 110. The processor 110 sequentially acquires time-series images captured by the endoscope 500, and performs deformation analysis processing on the time-series images. In the deformation analysis processing, the processor 110 arranges an evaluation mesh including a plurality of analysis points on an image at an arbitrary timing among the time-series images. The processor 110 deforms the evaluation mesh in each image so that each analysis point in each image of the time-series images tracks a feature point of the subject located at each analysis point in the image at the arbitrary timing on which the evaluation mesh is arranged. The processor 110 calculates the deformation amount of each cell of the evaluation mesh based on the magnitude and direction of the movement amount of each analysis point in each image. The processor 110 presents information on the deformation of the evaluation mesh based on the calculated deformation amount. Note that the movement amount of the analysis point here is a vector amount, and includes a magnitude and a direction.

[0106] According to this embodiment, the deformation of the subject is tracked by the evaluation mesh, and information on the deformation of the evaluation mesh is presented to the user based on the amount of deformation. As described in FIG. 2, FIG. 3, etc., there is a relationship between the deformation of tissue and the traction state. This allows the user to determine the traction state of tissue by visually checking the information on the deformation of the subject presented by the system. In addition, in this embodiment, the deformation is tracked and presented by image recognition, so force sensing by a force sensor or the like is not required. In addition, since it is the deformation that is tracked, rule-based tracking processing is possible, or even when tracking is performed by a machine learning model, force sensing information is not required in the learning stage.

[0107] In this embodiment, the processor 110 may superimpose a display in a manner corresponding to the amount of deformation in each image of the time-series images on each image. For example, the processor 110 may superimpose a display in which each cell is colored in a color corresponding to the amount of deformation of each cell on each image.

[0108] According to this embodiment, since the display according to the amount of deformation is superimposed on the endoscopic image, the user can know the amount of deformation of each part of the tissue by looking at the display. By knowing the amount of deformation of each part of the tissue, the user can judge the traction state of the tissue.

[0109] In this embodiment, the processor 110 may present the time-series images on a first monitor, and the processor 110 may present the time-series images and information related to the deformation of the evaluation mesh on a second monitor.

[0110] According to this embodiment, by displaying an endoscopic image without superimposed towing support information on the first monitor, visibility of the endoscopic image is ensured, and by displaying an endoscopic image with superimposed towing support information on the second monitor, the user's towing can be assisted.

[0111] In the present embodiment, the processor 110 may determine whether or not the scene is a towing scene in which the subject is towed by a treatment tool based on time-series images or a user input. When it is determined that the scene is a towing scene, the processor 110 may analyze the deformation of the subject due to towing by performing a deformation analysis process on the evaluation mesh.

[0112] According to this embodiment, in a scene in which a user requires traction information, a deformation analysis process is performed on an evaluation mesh, and traction information based on the results can be presented to the user.

[0113] Furthermore, in this embodiment, the processor 110 may determine the area of ​​the subject that is under tension due to pulling based on the deformation amount of each cell in each image of the time series images, and may superimpose a display showing the determined area on each image. Note that the "(1) stretched (tension) range" in Fig. 11 corresponds to the area of ​​the subject that is under tension due to pulling. Note that it is sufficient that the area of ​​the subject that is under tension is displayed, and the evaluation mesh itself may or may not be displayed.

[0114] 2 and 3, there is a relationship between the traction of the tissue, the deformation of the tissue 2, and the tension applied to the tissue. According to the present embodiment, this relationship can be utilized to present traction information to the user without directly detecting the force applied to the tissue.

[0115] The present embodiment may also be implemented as an image processing method. The image processing method includes the steps of sequentially acquiring time-series images captured by an endoscope, and arranging an evaluation mesh including a plurality of analysis points for an image at an arbitrary timing among the time-series images. The image processing method includes the steps of deforming the evaluation mesh in each image such that each analysis point in each image of the time-series images tracks a feature point of the subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is arranged. The image processing method includes the steps of calculating the deformation amount of each cell of the evaluation mesh based on the magnitude and direction of the movement amount of each analysis point in each image, and presenting information on the deformation of the evaluation mesh based on the calculated deformation amount. The image processing method may be executed by a computer.

[0116] The present embodiment may also be implemented as a non-transitory information storage medium storing a program. The program causes a computer to execute the steps of sequentially acquiring time-series images captured by an endoscope and arranging an evaluation mesh including a plurality of analysis points for an image at an arbitrary timing among the time-series images. The program causes a computer to execute the steps of deforming the evaluation mesh in each image such that each analysis point in each image of the time-series images tracks a feature point of the subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is arranged. The program causes a computer to execute the steps of calculating the deformation amount of each cell of the evaluation mesh based on the magnitude and direction of the movement amount of each analysis point in each image, and presenting information on the deformation of the evaluation mesh based on the calculated deformation amount.

[0117] 3. Second embodiment In the present embodiment, similarly to the first embodiment, the image processing system 100 sets an evaluation region, recognizes the traction information of the tissue within the evaluation region, and presents the recognized traction information to the user. In recognizing the traction state, the image processing system 100 divides the evaluation region into evaluation meshes for analysis.

[0118] There are problems in recognizing the towing state, as described below. Note that, although various aspects are disclosed in the second embodiment, each aspect may solve part of the following problems.

[0119] (1) In terms of functional constraints, if the evaluation area straddles the traction tissue and the background, it may not be possible to analyze or present the correct tissue deformation.

[0120] (2) From the viewpoint of usability, the display of unnecessary information over a wide area may obstruct the doctor's field of vision.

[0121] (3) From the standpoint of efficiency, when performing recognition processing on a wide range, the amount of calculation required can become enormous, which can result in long processing times.

[0122] The above (1) will be supplemented with an example in FIG. 36. As shown in the upper diagram of FIG. 36, an evaluation mesh 310 is set in the region treated by the forceps 301, 302, 305, and 306, and the region includes a background 321. The background 321 is not the tissue of the treatment target, but mainly the tissue on the back side of the field of view from the treatment target. In other words, the background 321 is a region that is not affected by the traction of the forceps. The lower diagram of FIG. 36 is an enlarged view of a part of the upper diagram. As shown in the lower diagram, the region in which the evaluation mesh 310 is set includes a tissue region 322 that moves due to the traction of the forceps 301, and a background 321 that does not move because the traction does not reach it. In a deformation analysis of the evaluation mesh 310, it is influenced by the background 321 that is not related to the traction, and there is a possibility that the traction state of the tissue region 322 cannot be correctly analyzed. More specifically, a single cell may include a tissue region 322 that moves due to the traction and a background 321 that does not move because the traction does not reach it. In such a case, there may be analysis points that correspond to and follow the movement of the tissue region 322 and analysis points that correspond to the stationary background 321 and do not follow the traction, which may result in inappropriate deformation of the evaluation mesh 310.

[0123] In response to the above problems (1) to (3), the image processing system 100 starts analysis with a wide range as the evaluation region, and gradually narrows the evaluation region to a necessary region. Hereinafter, as shown in FIG. 37, an example will be described in which the image processing system 100 presents an evaluation mesh 310 indicating tissue deformation on an endoscopic image when the function is started from OFF. However, the evaluation mesh 310 may be defined in the background after the function is started, and the evaluation mesh 310 may be presented on the endoscopic image at an appropriate timing. For example, the image processing system 100 sets the region operated by the assistant's forceps 301 and 302 and the surgeon's forceps 305 and 306 as the evaluation region, and sets the evaluation mesh 310 in the entire evaluation region. The surgeon's forceps 305 or forceps 306 may be an energy treatment tool. Next, the image processing system 100 deforms the evaluation mesh 310 following the tissue, and excludes regions that do not require analysis of deformation from the evaluation region. This narrows the presentation range of the evaluation mesh 310.

[0124] In this way, the region in which the tissue deformation should be evaluated can be estimated with high accuracy by utilizing dynamic information associated with traction, and as a result, traction information can be efficiently provided to the region required by the user.

[0125] 38 shows an example of the configuration of a medical system in the second embodiment. The medical system includes an image acquisition unit 510, an image processing system 100, and a monitor 700.

[0126] The image acquiring unit 510 acquires an endoscopic image. The image acquiring unit 510 corresponds to, for example, the endoscope 500 in Fig. 4. Alternatively, the image acquiring unit 510 may read out the endoscopic image from a storage in which the endoscopic image is recorded.

[0127] The image processing system 100 includes an I / O 171, an I / O 172, and a processor 110. Although not shown in the figure, the image processing system 100 may include a memory 120 similar to that in FIG.

[0128] The I / O 171 receives image data of an endoscopic image from the image acquisition unit 510, and inputs the image data of the endoscopic image to the processor 110. The I / O 172 transmits presentation information output by the processor 110 to the monitor 700. The I / O 171 and the I / O 172 correspond to the communication unit 160 in FIG.

[0129] The processor 110 includes a device detection unit 111, a contact detection unit 112, a start / end determination unit 113, an evaluation area setting unit 114, a tissue deformation recognition unit 115, and a towing state presentation unit 116. Note that the correspondence with the first embodiment is as follows. The processes performed by the device detection unit 111, the contact detection unit 112, and the start / end determination unit 113 correspond to the towing scene determination step S21 in FIG. 7. The process performed by the evaluation area setting unit 114 is a newly added process. The process performed by the tissue deformation recognition unit 115 corresponds to the towing range estimation step S23 in FIG. 7. Also, the process performed by the towing state presentation unit 116 corresponds to the presentation information processing step S25 in FIG. 7.

[0130] The processing performed by each unit of the processor 110 will be described below with reference to Fig. 39 and Fig. 40. Note that, except for the processing performed by the newly added evaluation area setting unit 114, the processing will be described merely as an example, and the methods described in the towing scene determination step S21, the towing range estimation step S23, and the presentation information processing step S25 of the first embodiment may be appropriately adopted.

[0131] 39, in step S51, a function for presenting a traction state is turned on. For example, the function may be turned on by a user operation, may be turned on by scene recognition such as image recognition of a specific procedure scene, or may be turned on in response to a device operation such as powering on an energy treatment tool.

[0132] In step S52, a user such as an operator or an assistant manipulates tissue with forceps or a treatment tool. The device detection unit 111 detects the forceps or treatment tool from the endoscopic image. As shown in FIG. 40, an endoscopic image is input to the device detection unit 111, and the device detection unit 111 detects the forceps or treatment tool from the endoscopic image using a machine learning model. The machine learning model is, for example, a model using an object detection or segmentation technique. In the learning stage, the machine learning model is machine-trained to recognize the treatment tool or the jaw of the forceps from the endoscopic image using an endoscopic image on which the jaw of the treatment tool or the forceps is annotated as learning data. The annotation is a boundary box surrounding the jaw of the treatment tool or the forceps in the object detection, and is data that fills in the area of ​​the jaw of the treatment tool or the forceps in the segmentation. The device detection unit 111 may recognize the jaw of the treatment tool or the forceps from the endoscopic image using a rule-based detection algorithm.

[0133] In step S53, the contact detection unit 112 detects the contact state between the forceps or treatment tool and tissue from the endoscopic image. As shown in FIG. 40, the contact detection unit 112 detects that the forceps or treatment tool has come into contact with tissue from the endoscopic image using a machine learning model. In the learning stage, the machine learning model is trained by machine learning so that the presence or absence of contact between the forceps or treatment tool and tissue can be recognized from the endoscopic image using the endoscopic image annotated with the presence or absence of contact between the forceps or treatment tool and tissue as learning data. Note that the contact detection unit 112 may recognize the presence or absence of contact between the forceps or treatment tool and tissue from the endoscopic image using a rule-based detection algorithm.

[0134] In step S54, the start / end judgment unit 113 judges whether to start or end the process of making the evaluation mesh follow the tissue deformation based on the contact state between the forceps or treatment tool and the tissue detected by the contact detection unit 112 or based on a preset rule.

[0135] In step S55, the evaluation area setting unit 114 sets an evaluation area for tissue deformation recognition based on the endoscopic image. That is, the evaluation area setting unit 114 narrows the evaluation area as necessary based on the endoscopic image.

[0136] In step S56, the tissue deformation recognition unit 115 recognizes the deformation of the tissue due to traction or the like from the endoscopic image. The traction state presentation unit 116 presents the result of the recognition by the tissue deformation recognition unit 115 to the surgeon or assistant. In FIG. 40, the image shown within the frame of the tissue deformation recognition unit 115 represents the magnitude of the flow vector of each point detected by the optical flow in terms of brightness, and the direction of the flow vector in terms of hue. The tissue deformation recognition unit 115 recognizes the deformation of the tissue by making the evaluation mesh 310 follow the subject using such optical flow detection results. The traction state presentation unit 116 presents the traction state to the surgeon or assistant by applying a color or the like according to the deformation amount of each cell of the evaluation mesh 310 to the endoscopic image.

[0137] In step S57, processor 110 ends the process if the function has been turned off, and returns to step S52 if the function has not been turned off.

[0138] The evaluation area narrowed in step S55 may be applied to the deformation analysis in step S56a. That is, deformation analysis may be performed on the evaluation mesh 310 of the narrowed evaluation area. Alternatively, the evaluation area narrowed in step S55 may be applied to the information presentation in step S56b. That is, in step S56a, deformation analysis may be performed on the evaluation mesh 310 of the evaluation area that has not been narrowed, and in step S56b, information may be presented based on the deformation of the evaluation mesh 310 in the evaluation area narrowed in step S55, among the evaluation meshes 310 subjected to deformation analysis.

[0139] The methods (1-1) to (1-5) for narrowing down the evaluation region in step S55 will be described below. First, an overview of each method will be described with reference to Figs.

[0140] (1-1) Exclusion of non-moving areas (background)

[0141] An example is shown in FIG. 41. The evaluation area setting unit 114 excludes areas ARAa and ARAb in which the deformation amount of the analysis point is equal to or less than a threshold value during a predetermined period after the function is started, from the evaluation area. The deformation amount of the analysis point is, for example, the magnitude of the movement amount. The evaluation area setting unit 114 may correct at least one of the shift, rotation, and enlargement / reduction of the endoscope by a comparison process between frames. For example, a correction method using a transformation matrix described later in the third embodiment may be adopted. In the example of FIG. 41, the area ARAa excluded by the assistant processing is different from the area ARAb excluded by the operator processing. For example, the assistant's excluded area ARAa is an area that does not move even if the assistant's forceps pulls the tissue, and the operator's excluded area ARAb is an area that does not move even if the operator's forceps pulls the tissue. A common evaluation area may be set without separating the processing for the operator and the assistant.

[0142] (1-2) Narrowing down using segmentation

[0143] An example is shown in FIG. 42. The evaluation area setting unit 114 divides the area of ​​the endoscopic image by a segmentation method, and selects an evaluation area from among the multiple areas obtained by the division. The selection may be rule-based, or may use a machine learning model. In the assistant's processing, the endoscopic image is divided into areas ARBa1 and ARBa2 by segmentation, and for example, the area ARBa1 is set as the evaluation area. In the surgeon's processing, the endoscopic image is divided into areas ARBb1 and ARBb2 by segmentation, and for example, the area ARBb1 is set as the evaluation area. Note that a common evaluation area may be set without separating the processing for the surgeon and the assistant.

[0144] (1-3) Narrowing down using contour detection

[0145] An example is shown in FIG. 43. The evaluation area setting unit 114 detects the contour (edge) of tissue generated by traction by image processing. The contour detection is, for example, a Gabor filter or a Hough transform. The evaluation area setting unit 114 limits the evaluation area to an area closed by the detected contour and its extension. In the assistant's processing, a contour EGDa is detected from the endoscopic image, and an area ARDa closed by the contour EGDa and its extension is set as the evaluation area. In the operator's processing, a contour EGDb is detected from the endoscopic image, and an area ARDb closed by the contour EGDb and its extension is set as the evaluation area. For example, the assistant's contour EGDa is the contour of tissue generated by traction of the assistant's forceps, and the operator's contour EGDb is the contour of tissue generated by traction of the operator's forceps. It should be noted that a common evaluation area may be set without separating the processes for the operator and the assistant.

[0146] (1-4) Narrowing down using depth information

[0147] An example is shown in FIG. 44. In particular, in a scene in which the forceps are deployed by an assistant, the three-dimensional depth differs between the tissue being pulled and the background. The depth refers to the distance from the endoscope to the tissue. The evaluation area setting unit 114 narrows the evaluation area to an area ARE on the near side of the depth, with a line LE at which the depth changes significantly as a boundary. The lower diagram of FIG. 44 shows the relationship between the position on the image and the distance (depth) from the endoscope. Here, the position is one-dimensional, but in reality, depth information is acquired two-dimensionally. With a line LE at which the depth changes significantly as a boundary, an area ARE where the distance from the endoscope is small (on the near side of the depth) is set as the evaluation area. The means for acquiring three-dimensional information is, for example, a method using a 3D endoscope using stereo vision, or a method of estimating depth from a two-dimensional image using a machine learning model.

[0148] (1-5) Narrowing down using pulling direction detection

[0149] An example is shown in FIG. 45. The evaluation area setting unit 114 tracks the movement of the target forceps or treatment tool after the function starts. The evaluation area setting unit 114 narrows down the evaluation area to the opposite side of the pulling direction, based on the gripping part of the forceps or treatment tool. The pulling direction here is not the instantaneous moving direction between frames, but the moving direction from a reference timing such as the function start time. In the assistant processing, the area on the opposite side of the pulling direction DRFa1 from the line LFab1 and the area ARFa on the opposite side of the pulling direction DRFa2 from the line LFab2 are set as the evaluation area. The line LFab1 is a line that passes through the gripping part of the assistant forceps 301 and is perpendicular to the pulling direction DRFa1 of the assistant forceps 301. The line LFab2 is a line that passes through the gripping part of the assistant forceps 302 and is perpendicular to the pulling direction DRFa2 of the assistant forceps 302. In the processing for the surgeon, an area ARFb on the opposite side of the pulling direction DRFb from the line LFb is set as the evaluation area. The line LFb is a line that passes through the gripping portion of the surgeon's forceps 305 and is perpendicular to the pulling direction DRFb of the surgeon's forceps 305. Note that a common evaluation area may be set without separating the processing for the surgeon and the assistant.

[0150] Next, the above techniques (1-1) to (1-5) will be described in detail with reference to FIGS.

[0151] (1-1a) First detailed example of excluding non-moving areas (background)

[0152] Fig. 46 shows a first detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes a time measurement unit 114a, a point tracking unit 114b, and an area determination unit 114c. Fig. 47 shows a process flow executed by the first detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in Fig. 39.

[0153] In step S61, the time measurement unit 114a measures the time that has elapsed since a reference timing such as the start of a function. In step S62, the point tracking unit 114b tracks each point on the endoscopic image in time series to follow the movement of the subject. In step S63, when the time measurement unit 114a measures the passage of a certain period of time, the region determination unit 114c calculates the accumulated movement amount (accumulated movement distance) of each point on the endoscopic image. The accumulated movement amount is the amount of movement of each point at a predetermined interval such as between frames, which is accumulated from a reference timing such as the start of a function. In step S64, the region determination unit 114c excludes from the evaluation region a region in which there are many points whose accumulated movement amount is equal to or less than a threshold. The threshold is, for example, a fixed value determined in advance, or a threshold described later in FIG. 48. The region in which there are many points whose accumulated movement amount is equal to or less than a threshold is, for example, a region in which the density of points whose accumulated movement amount is equal to or less than a threshold is equal to or greater than a preset value.

[0154] The points tracked by the point tracking unit 114b are points on the endoscopic image that are uniquely defined regardless of the evaluation mesh 310. These points are set, for example, at a higher density than the analysis points of the evaluation mesh 310. Alternatively, the points tracked by the point tracking unit 114b may be the analysis points of the evaluation mesh 310 at that time.

[0155] FIG. 48 shows an example of the threshold value of the accumulated movement amount. FIG. 48 is a so-called histogram, which shows the relationship between the accumulated movement amount of each point and its frequency. Processor 110 determines the threshold value based on this histogram. For example, processor 110 determines the accumulated movement amount at which the frequency in the histogram is the minimum value as the threshold value. Alternatively, processor 110 may accumulate the frequency in the histogram from the side with the smallest accumulated movement amount, and determine the accumulated movement amount when the accumulated value reaches a predetermined value as the threshold value.

[0156] (1-1b) Second detailed example of excluding non-moving areas (background)

[0157] Fig. 49 shows a second detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes a point tracking unit 114b, a total movement amount calculation unit 114d, and an area determination unit 114c. Fig. 50 shows a process flow executed by the second detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in Fig. 39.

[0158] In step S71, the point tracking unit 114b tracks each point on the endoscopic image in time series to follow the movement of the subject. In step S72, the total movement amount calculation unit 114d sums up the movement amount of each point between frames or at a predetermined interval in the endoscopic image, and accumulates the total value from the start of the function. This accumulated value is called the total movement amount. In step S73, the area determination unit 114c calculates the accumulated movement amount (accumulated movement distance) of each point on the endoscopic image when the total movement amount exceeds the first threshold. The definition of the accumulated movement amount is as described above. The first threshold is, for example, a fixed value determined in advance. In step S74, the area determination unit 114c excludes an area in which there are many points whose accumulated movement amount is equal to or less than the second threshold from the evaluation area. The second threshold is the same as the threshold described in (1-1a).

[0159] According to the second detailed example, even if the surgeon or assistant temporarily stops the traction after the function starts, the process does not proceed to S73 while the traction is stopped, so the evaluation area is not excluded, and when the traction is started again, the process proceeds to step S73 and the evaluation area is excluded in step S74. This allows the time until the evaluation area is determined to be flexibly extended.

[0160] (1-2a) First detailed example of narrowing down results using segmentation

[0161] 51 shows a third detailed configuration example of the evaluation area setting unit 114. An endoscopic image is input to the evaluation area setting unit 114, and the evaluation area setting unit 114 estimates an evaluation area from the endoscopic image using a segmentation model. In the learning stage, the segmentation model is machine-trained so that an evaluation area can be estimated from an endoscopic image using an endoscopic image with annotated evaluation area as learning data.

[0162] According to this detailed example, the evaluation region can be estimated directly as an output of the segmentation model, and also based on the image, even for scenes with little towing motion.

[0163] (1-2b) Second detailed example of narrowing down results using segmentation

[0164] 52 shows a fourth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 divides an area by a segmentation method and sets an evaluation area on a rule basis. The evaluation area setting unit 114 includes a segmentation unit 114f and an area selection unit 114g.

[0165] The segmentation unit 114f divides the endoscopic image into a plurality of regions by a segmentation method. FIG. 52 shows an example of dividing the endoscopic image into region 1 and region 2, but the number of divided regions is not limited to two. The segmentation method may be a non-AI method such as GrabCut, or an AI method such as SAM (Segment Anything Model). The AI ​​here is a general-purpose AI, and does not require prior learning. That is, although learning as a general-purpose AI is performed, special prior learning assuming application to the image processing system 100 of this embodiment is not required. In addition, the AI ​​here divides the image into regions, but does not estimate what each region is. That is, the AI ​​does not determine whether the divided region is an evaluation region or not.

[0166] The region selection unit 114g selects an evaluation region from among the divided regions on a rule basis. An example is shown in FIG. 53. Assume that the segmentation unit 114f divides the endoscopic image into regions ARG1 and ARG2. The region selection unit 114g selects, as the evaluation region, the region from the regions ARG1 and ARG2 that has the greatest overlap with the judgment region ARH in the center of the field of view. In the example of FIG. 53, the region ARG1 has the greatest overlap with the judgment region ARH, so the region ARG1 is selected as the evaluation region. Alternatively, the region selection unit 114g may select, as the evaluation region, the region from the regions ARG1 and ARG2 that includes one point in the center of the field of view.

[0167] In (1-2a), it is necessary to pre-train a network assuming application to the image processing system 100 of this embodiment, but in (1-2b), it is not necessary to pre-train a network.

[0168] (1-3) Detailed example of narrowing down using contour detection

[0169] 54 shows a fifth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes an edge detection unit 114h and an area setting unit 114p.

[0170] The edge detection unit 114h detects the contour (edge) of tissue generated by traction from the endoscopic image using image processing. Contour detection is a non-AI method, such as a Gabor filter or Hough transform. The region setting unit 114p generates a region closed by the contour line and the extension line by extending the detected contour, and limits the evaluation region to that region. The extension line is obtained, for example, by extending the edge of the contour in a straight line to the edge of the image. An example of a region closed by the contour line and the extension line is as shown in FIG. 43.

[0171] According to this detailed example, since the evaluation area is set using a non-AI algorithm, network learning is not required. In addition, since the amount of calculation is smaller than when using an AI method, it is expected that processing speed will be increased or calculation resources will be reduced. By reducing calculation resources, it is expected that it will be possible to realize it using a low-spec PC.

[0172] (1-4) Detailed example of filtering using depth information

[0173] FIG. 55 shows a sixth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 judges the evaluation area by adding depth information to the above method (1-1a), (1-1b), (1-2a), (1-2b), or (1-3). The evaluation area setting unit 114 includes an area setting unit 114q, a depth judgment unit 114r, and an integrated judgment unit 114s. FIG. 56 shows a process flow executed by the sixth detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in FIG. 39.

[0174] In step S81, the region setting unit 114q determines the evaluation region by any one of the above methods (1-1a), (1-1b), (1-2a), (1-2b), and (1-3). In step S82, the depth judgment unit 114r acquires depth information representing the distance distribution of the target tissue from the tip of the endoscope, and judges a line where the distance changes significantly in the depth information, for example, by threshold judgment. The depth judgment unit 114r divides the region at the boundary of the line, and judges the region on the near side of the depth of the divided region as the evaluation region. For example, the endoscope 500 in FIG. 5 is a 3D endoscope, and the depth judgment unit 114r uses the depth information of the 3D image captured by the 3D endoscope. Alternatively, the endoscope 500 in FIG. 5 is a 2D endoscope, and the depth judgment unit 114r estimates the depth information from the endoscopic image using AI that estimates the depth from the 2D image. In step S83, the integrated judgment unit 114s sets, as an evaluation area, an area that belongs to both the evaluation area determined by the area setting unit 114q and the evaluation area determined by the depth judgment unit 114r.

[0175] In particular, in the scene where the assistant deploys the tissue, the three-dimensional depth differs between the towed tissue and the background. According to this detailed example, by combining depth information, the evaluation area can be set taking into account the three-dimensional depth, improving the accuracy of setting the evaluation area.

[0176] (1-5) Detailed example of filtering using pulling direction detection

[0177] FIG. 57 shows a seventh detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 judges the evaluation area by adding information on the pulling direction to the above method (1-1a), (1-1b), (1-2a), (1-2b), or (1-3). The evaluation area setting unit 114 includes an area setting unit 114q, a pulling direction determining unit 114t, and an integrated determining unit 114s. FIG. 58 shows a process flow executed by the seventh detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in FIG. 39.

[0178] In step S91, the region setting unit 114q determines the evaluation region by any one of the above methods (1-1a), (1-1b), (1-2a), (1-2b), and (1-3). In step S92, the pulling direction determining unit 114t tracks the movement of the target forceps from a reference timing such as a function start time. For example, the pulling direction determining unit 114t detects the target forceps from an endoscopic image, obtains a flow vector from the endoscopic image by optical flow, and tracks the movement of the forceps using the flow vector at the position of the detected forceps. The pulling direction determining unit 114t narrows down the evaluation region to the opposite side of the pulling direction based on the gripping part of the forceps or treatment tool. This example is as shown in FIG. 45. In step S93, the integrated determining unit 114s sets the region that belongs to both the evaluation region determined by the region setting unit 114q and the evaluation region determined by the pulling direction determining unit 114t as the evaluation region.

[0179] The tissue on the side of the treatment tool in the traction direction is considered to be tissue that is not the target of traction, and the tissue on the opposite side of the treatment tool in the traction direction is considered to be tissue that is the target of traction. According to this detailed example, by combining information on the traction direction, the evaluation region can be set in the tissue that is the target of traction, thereby improving the accuracy of setting the evaluation region.

[0180] Step S55 of setting the evaluation area in Fig. 39 is executed repeatedly while the function is on. Therefore, as shown in Fig. 59, the evaluation area in which the evaluation mesh 310 is set is gradually narrowed down over time. When the support function is used for a long time, or when the evaluation area is narrowed down excessively due to an error, the evaluation area may be narrowed down too much. Below, methods (2-1), (2-2), and (3) for dealing with excessive narrowing down are explained.

[0181] First, the techniques (2-1) and (2-2) for suppressing area narrowing will be described.

[0182] (2-1) Limited to a set time

[0183] The evaluation area setting unit 114 limits the narrowing down of the evaluation area to a specified time period after a reference timing such as the start of the function. Fig. 60 shows an eighth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes a time measurement unit 114u, a narrowing down suppression unit 114v, and an area setting unit 114w. Fig. 61 shows a process flow executed by the eighth detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in Fig. 39.

[0184] In step S101, the time measurement unit 114u measures the time that has elapsed since a reference timing such as the start of a function. If the specified time has not elapsed, in step S102, the area setting unit 114w continues narrowing down the evaluation area by any one of the above methods (1-1) to (1-5) or a combination of two or more of them. If the specified time has elapsed, in step S103, the narrowing down suppression unit 114v determines to stop narrowing down the evaluation area. In step S104, the area setting unit 114w stops narrowing down the evaluation area based on the stop determination by the narrowing down suppression unit 114v.

[0185] According to this method, narrowing down of the evaluation area is no longer performed after the specified time has elapsed, so excessive narrowing down of the evaluation area can be suppressed compared to when narrowing down is continued.

[0186] (2-2) Changing the threshold over time

[0187] The evaluation area setting unit 114 increases the threshold value used to determine narrowing down of the evaluation area as time passes after the function is started. As the threshold value increases, it becomes more difficult to narrow down the evaluation area. FIG. 62 shows a ninth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes a time measurement unit 114u, a threshold adjustment unit 114x, and an area setting unit 114w. FIG. 63 shows a process flow executed by the ninth detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in FIG. 39.

[0188] In step S111, the time measurement unit 114u measures the time elapsed from a reference timing such as the start of a function. In step S112, the threshold adjustment unit 114x adjusts the threshold for narrowing down the evaluation area according to the elapsed time. In step S113, the area setting unit 114w continues narrowing down the evaluation area using the adjusted threshold. For example, the area setting unit 114w narrows down the evaluation area using the above method (1-1) or (1-4). When the method (1-1) is used, the threshold adjustment unit 114x adjusts the threshold for the amount of movement, and when the method (1-4) is used, the threshold adjustment unit 114x adjusts the threshold for the distance that determines the line where the depth changes significantly. An example of threshold change is shown in FIG. 64. The threshold adjustment unit 114x adjusts the threshold so that the threshold is a function y=f(t) of elapsed time. The function y=f(t) is a function such that the threshold increases monotonically with time. However, this does not exclude a function that includes a section in which the threshold does not increase, or a function that includes a section in which the threshold temporarily decreases.

[0189] Next, we explain the method (3) for expanding the evaluation area that has been narrowed down.

[0190] (3) Expansion of evaluation scope

[0191] For example, in towing assistance by an assistant, it is expected that the assistant will tow the vehicle for a long period of time and apply the towing status display assistance function. In such a case, if the narrowing down of the above (1-1) to (1-5) is continued, the evaluation area may be narrowed down too much, including narrowing down errors. In addition, after the function is started, a new area that should be the evaluation area may be reflected due to a narrowing of the field of view, etc.

[0192] Therefore, the evaluation area setting unit 114 performs a new evaluation area determination using any one of the above methods (1-1) to (1-5) or a combination of two or more of them at regular intervals after the function starts. If the difference between the evaluation area at that stage and the newly determined evaluation area is large, that is, if the evaluation area is too narrowed, the evaluation area setting unit 114 applies the newly determined evaluation area to expand the evaluation area. This allows the evaluation area that has been narrowed down too much to be expanded.

[0193] An example is shown in FIG. 65. The function is turned on in step S201 at time t1. In steps S202 to S204 at times t2 to t4, the evaluation area setting unit 114 continues to determine the evaluation area from the start of the function. In step S203 at time t3 and step S204 at time t4, the evaluation area is gradually narrowed down. Apart from S202 to S204, the evaluation area setting unit 114 determines a new evaluation area in step S214 at time t4 after a certain period has elapsed since the start of the function. The evaluation area setting unit 114 compares the evaluation area of ​​S204 with the evaluation area of ​​S214. If the evaluation area of ​​S214 is larger and the difference between the evaluation area of ​​S214 and the evaluation area of ​​S204 is large, the evaluation area setting unit 114 adopts the evaluation area of ​​S214 in step S225 at time t5. If the condition is not satisfied, the evaluation area setting unit 114 continues to use the evaluation area of ​​S204. Thereafter, the evaluation area setting unit 114 updates the evaluation area at regular intervals in the same manner as above.

[0194] Fig. 66 shows a tenth detailed configuration example of the evaluation area setting unit 114. The evaluation area setting unit 114 includes a time measurement unit 114u, an area review unit 114y, an area setting unit 114w, and a comparison evaluation unit 114z. Fig. 67 shows a process flow executed by the tenth detailed configuration example of the evaluation area setting unit 114. This flow is a detailed flow of S55 in Fig. 39.

[0195] In step S121, the time measurement unit 114u measures the time that has elapsed since a reference timing such as the start of a function. In step S122, the area setting unit 114w continues to narrow down the evaluation area by any one of the above methods (1-1) to (1-5) or a combination of two or more of them, regardless of the elapsed time. In step S123, the area review unit 114y determines a new evaluation area by any one of the above methods (1-1) to (1-5) or a combination of two or more of them at regular intervals. In step S124, the comparison evaluation unit 114z compares the current evaluation area determined by the area setting unit 114w with the new evaluation area determined by the area review unit 114y. In the example of FIG. 65, this corresponds to the comparison between S204 and S214. If the difference between the current evaluation area and the new evaluation area is large, the comparison evaluation unit 114z updates the evaluation area in the area setting unit 114w with the new evaluation area. When the difference between the current evaluation area and the new evaluation area is small, the comparison evaluation unit 114z does not update the evaluation area in the area setting unit 114w. As a specific example, the comparison evaluation unit 114z compares the area of ​​the current evaluation area determined by the area setting unit 114w with the area of ​​the new evaluation area determined by the area review unit 114y. When the ratio of the area of ​​the current evaluation area to the area of ​​the new evaluation area is equal to or less than a specified value, the comparison evaluation unit 114z updates the evaluation area held by the area setting unit 114w with the new evaluation area. The specified value is, for example, 20%, but is not limited to this.

[0196] Other examples of the method for setting the evaluation region (4) and (5) will be described.

[0197] (4) Manual setting of evaluation area (fixed)

[0198] The user sets the evaluation area before or during surgery. The user may be either the surgeon or an assistant. For example, multiple evaluation area options may be provided, and the user may select the evaluation area from the options. Possible operation units for selection during surgery include a button on a scope, a button on a treatment tool, a foot switch, or voice operation. The evaluation area setting unit 114 sets the evaluation area input by user operation. According to this method, the user can utilize the functions according to his / her preference at the necessary timing.

[0199] 68 to 70 show examples of options for the evaluation area in which the evaluation mesh 310 is placed. As shown in FIG. 68, the option may be the location of the evaluation area in the image. In the example of FIG. 68, the center and edges of the image are location options. As shown in FIG. 69, the option may be the size of the evaluation area. In the example of FIG. 69, whole image, large, and small are size options. As shown in FIG. 70, the option may be the granularity of the evaluation mesh 310. In the example of FIG. 70, coarse and fine are granularity options.

[0200] (5) Setting the evaluation area around the surgeon's forceps

[0201] The evaluation area setting unit 114 sets an evaluation area near the tip of the forceps operated by the user. As shown in the lower diagram of Fig. 71, when the user is a surgeon, the evaluation area is set near the tip of the surgeon's forceps 306. As shown in Fig. 71, the evaluation area setting unit 114 may add the vicinity of the tip of the surgeon's forceps as one of the options for the location of the evaluation area described above.

[0202] The evaluation area setting unit 114 utilizes a network that has been trained in advance to detect the tip of the surgeon's forceps from the endoscopic image and set an evaluation area of ​​a specified size at the tip. For example, the device detection unit 111 in Fig. 40 detects the jaw of a treatment tool or forceps from an endoscopic image using a machine learning model. The evaluation area setting unit 114 sets an evaluation area at the position of the detected treatment tool or jaw of the forceps.

[0203] In this embodiment, the processor 110 may determine an evaluation area, which is an area in which the evaluation mesh 310 is arranged or an area of ​​the evaluation mesh 310 to be reflected in the image display, based on at least one of the movement amount of the subject between time-series images, the image feature amount of the subject, and the depth information of the subject. Note that the determination of the evaluation area based on the image feature amount of the subject corresponds to, for example, the narrowing down using segmentation in Fig. 42, the narrowing down using contour detection in Fig. 43, or the narrowing down using detection of the pulling direction in Fig. 45.

[0204] According to this embodiment, it is possible to present traction information by focusing on a tissue region to be treated or a tissue region for which the user requires traction information. Furthermore, by focusing on the evaluation region, the deformation analysis of the evaluation mesh 310 is less likely to be affected by tissue regions that are not to be treated or tissue regions for which the user does not require traction information.

[0205] Also, in this embodiment, the processor 110 may calculate the accumulated amount of movement between frames or at a specified interval for each point of the subject on the time-series images, accumulating the amount of movement from a reference time until a specified period of time has elapsed, and set the evaluation area by excluding areas where the accumulated amount of movement is equal to or less than a threshold value.

[0206] Regions with small amounts of movement are considered to be tissue regions where no treatment has been performed, such as background regions. According to this embodiment, by setting the evaluation region by excluding regions where the amount of movement in a predetermined period is equal to or less than a threshold, it is possible to set the evaluation region by narrowing it down to the tissue region to be treated.

[0207] In this embodiment, the processor 110 may calculate the amount of movement of each point of the subject on the time-series images, calculate a total value by adding up the amount of movement of each point within the image, and calculate a total amount of movement by accumulating the total value from a reference timing. When the total amount of movement exceeds a first threshold, the processor 110 may calculate an accumulated amount of movement by accumulating the amount of movement of each point between frames or at a predetermined interval from the reference timing to the timing when the total amount of movement exceeds the first threshold, and set an evaluation area by excluding an area where the accumulated amount of movement is equal to or less than a second threshold.

[0208] When the user is not performing traction, the movement in the entire endoscopic image is small, so excluding areas with small movement may result in an incorrect setting of the evaluation area. According to this embodiment, when the total amount of movement in the image becomes greater than a certain amount, that is, when it is determined that tissue in the image has moved due to traction or the like, a determination is made to exclude the evaluation area. As a result, the evaluation area is set to the tissue area that has moved due to traction or the like, and areas that have not moved are excluded from the evaluation area, so that the evaluation area can be set correctly.

[0209] In this embodiment, the processor 110 may input the time-series images to a trained model that estimates an evaluation area from an endoscopic image by segmentation, and set the evaluation area based on the estimation result of the trained model. For example, the trained model 122 in Fig. 6 includes a segmentation model, and the evaluation area is estimated by the segmentation model.

[0210] According to this embodiment, the evaluation region can be directly estimated from the endoscopic image by segmentation. Even if there is no information on the movement of tissue, the evaluation region can be set.

[0211] In this embodiment, the processor 110 may input the time-series image to a trained model that divides an image into regions by segmentation, and set the evaluation region to the region that has the largest overlap with a predetermined region in the image among the multiple regions divided by the trained model. The predetermined region is, for example, the judgment region ARH in FIG. 53 or a point in the center of the field of view. For example, the trained model 122 in FIG. 6 includes a segmentation model, and the time-series image is divided into regions by the segmentation model.

[0212] According to this embodiment, since there is no need for learning to estimate the evaluation region from the endoscopic image, for example, a segmentation model that performs existing region division can be utilized. This reduces the learning cost. In this embodiment, too, it is possible to set the evaluation region even if there is no information on the movement of tissues.

[0213] In this embodiment, the processor 110 may perform edge detection processing on the time-series images and set the closed area by the detected edge as the evaluation area. Note that the closed area by the edge refers to an area in an image where the closed area is formed only by the edge. Alternatively, as shown in FIG. 43, in an image where the closed area is not formed only by the edge, the closed area by the edge refers to a closed area formed by the edge and the extension line of the edge.

[0214] According to this embodiment, since the evaluation area can be set using a non-AI algorithm, network learning is not required. In addition, since the amount of calculation is smaller than when using an AI method, it is expected that the processing speed will be increased or the calculation resources can be reduced. By reducing the calculation resources, it is expected that it can be realized using a low-spec PC.

[0215] In this embodiment, the processor 110 may detect the pulling direction of the treatment tool that pulls the subject from the time-series images, and set the evaluation area on the opposite side of the pulling direction from a predetermined position on the treatment tool. Note that the "treatment tool" may be any treatment tool that can pull tissue, and may be, for example, a non-energy treatment tool such as forceps, or an energy treatment tool having jaws such as a bipolar device. Note that in the example of FIG. 45, the predetermined position on the treatment tool is the position of the jaws of the forceps, but is not limited to this and may be any position on the treatment tool.

[0216] The tissue on the side of the predetermined position on the treatment tool in the traction direction is considered to be tissue that is not the target tissue to be tractioned, and the tissue on the opposite side is considered to be tissue to be tractioned. According to this embodiment, since the evaluation region can be set in the tissue to be tractioned, the setting accuracy of the evaluation region is improved.

[0217] In this embodiment, the processor 110 may acquire a depth distribution from the endoscope to the subject, and set an evaluation area in an area on the smaller depth side, with a line where the depth changes sharply in the depth distribution as a boundary. In the example of Fig. 44, the line LE corresponds to the line where the depth changes sharply in the depth distribution, and the area ARE corresponds to the area on the smaller depth side.

[0218] For example, in a scene where an assistant deploys a tissue, the three-dimensional depth may differ between the towed tissue and the background. According to this embodiment, an evaluation area can be set taking into account the three-dimensional depth, improving the accuracy of setting the evaluation area.

[0219] In this embodiment, the processor 110 may update the evaluation area by determining to exclude a part of the evaluation area or to maintain the evaluation area based on at least one of the movement amount, the image feature amount, and the depth information at each given update timing. In the example of Fig. 65, the time t4 or t5 corresponds to the given update timing.

[0220] According to the present embodiment, it is possible to set an appropriate evaluation area according to the ever-changing situation. Moreover, unnecessary evaluation areas are gradually removed, thereby reducing unnecessary displays for the user.

[0221] 4. Third embodiment The configuration of the medical system is similar to that of the medical system described in Fig. 5 of the first embodiment. Moreover, the configuration of the image processing system 100 is similar to that of the image processing system 100 described in Fig. 6 of the first embodiment.

[0222] Fig. 72 shows a detailed flow of the image processing step S32 in the third embodiment. The towing scene determination step S21, the towing range estimation step S23, and the presented information processing step S25 are the same as those in Fig. 7 of the first embodiment. In these steps, various processes described in Figs. 8 to 35 can be adopted. In the third embodiment, an area analysis step S22 is added. That is, when the processor 110 determines in S21 that the scene is one in which the towing range should be estimated (T: True), the processor 110 calculates the amount of movement of tissue in the endoscopic image in step S22, and analyzes the evaluation area based on the calculation result. In step S23, the processor 110 estimates the towing range using the analyzed evaluation area.

[0223] Specifically, the processor 110 sets the evaluation mesh 410 at the time when it is determined in step S21 that the scene is a towing scene, and sets the evaluation mesh 410 at that time as the initial state. The processor 110 analyzes and updates the evaluation mesh 410 using the flow vector calculation results from time to time in step S22. The analyzed and updated evaluation mesh 410 is used for towing range estimation in the subsequent towing range estimation step S23.

[0224] 73 is a detailed flow of the region analysis step S22. The region analysis step S22 includes a non-interest tissue recognition step S151, a flow vector calculation step S152, an evaluation mesh analysis step S153, and an evaluation mesh update step S154. The processing contents of each step will be described below.

[0225] FIG. 74 is an explanatory diagram of the non-interest tissue recognition step S151. The processor 110 recognizes a non-interest tissue region from the input endoscopic image. The recognition is segmentation. The non-interest tissue region refers to a region other than the interest tissue, such as a treatment tool, gauze, or tissue that is not a treatment target. A machine learning model is used for the segmentation. The machine learning model is trained, for example, with an endoscopic image with annotated interest tissue region as training data. In this case, the processor 110 inputs an endoscopic image to the machine learning model to recognize the interest tissue region, and sets the region other than the recognized interest tissue region as a non-interest tissue region. Alternatively, the machine learning model is trained with an endoscopic image with annotated non-interest tissue region as training data. In this case, the processor 110 inputs an endoscopic image to the machine learning model to recognize the non-interest tissue region. FIG. 74 shows an example of using a non-interest tissue recognition AI. It is assumed that the endoscopic image shows treatment tools 401 to 404. The non-interest tissue recognition AI recognizes non-interest tissue regions NFA1 to NFA4 corresponding to the treatment tools 401 to 404.

[0226] FIG. 75 is an explanatory diagram of the flow vector calculation step S152. The processor 110 calculates the flow vector between the endoscopic image of the current frame and the endoscopic image of the previous frame by the optical flow method. The "previous frame" may be the frame immediately before the current frame, or may be the frame a predetermined interval before. Alternatively, the interval between the "previous frame" and the current frame may vary. The optical flow method may be a classical non-AI method, or may be a method using AI such as RAFT. FIG. 75 shows an image in which the magnitude of the flow vector of each point is expressed by brightness and the direction of the flow vector is expressed by hue, as an example of the flow vector calculation result.

[0227] FIG. 76 is an explanatory diagram of the evaluation mesh analysis step S153. The processor 110 estimates the movement of each analysis point of the evaluation mesh 410 based on the flow vector of the endoscopic image. The processor 110 may directly use the flow vector of the pixel where each analysis point is placed. Alternatively, as shown in FIG. 76, the processor 110 may calculate statistics from the flow vectors of the pixels included in the peripheral area 430 of each analysis point 420, and estimate the movement of the analysis point 420 from the statistics. The statistics are, for example, the average value, the median value, or a weighted average based on the distance. By using the statistics, the robustness of the evaluation mesh analysis is improved against disturbances such as halation, image noise, or mist.

[0228] 77 is an explanatory diagram of evaluation mesh update step S154. The processor 110 updates the evaluation mesh 410 based on the analysis result of evaluation mesh analysis step S153. That is, the processor 110 moves each analysis point of the evaluation mesh 410 by the movement amount of each analysis point calculated in evaluation mesh analysis step S153.

[0229] 78 and 79 are explanatory diagrams of a modified example of the evaluation mesh analysis step S153. The flow vector includes information on not only the movement of the tissue but also the movement of the device. Therefore, as shown in the upper diagram of FIG. 78, the movement of the device such as the treatment tools 405-407 affects the deformation of the evaluation mesh 410. In other words, there is a possibility that the evaluation mesh 410 does not correctly represent the deformation of the tissue. Therefore, as shown in the lower diagram of FIG. 78, when analyzing the evaluation mesh 410 using the flow vector, the processor 110 eliminates the influence of the movement of the device by a correction process. In the non-interest tissue recognition step S151, the processor 110 recognizes the non-interest tissue regions NFA5-NFA7 corresponding to the treatment tools 405-407, and in the evaluation mesh analysis step S153, the correction process is performed using the non-interest tissue regions NFA5-NFA7.

[0230] FIG. 79 is an explanatory diagram of the correction process, taking the non-interest tissue region NFA5 as an example. The correction process targets analysis points P22-P24 and P32-P34 that overlap the non-interest tissue region NFA5. The surrounding analysis points are P11-P14, P21, and P31. The processor 110 performs movement estimation using a flow vector for analysis points that do not overlap the non-interest tissue region NFA5. The processor 110 does not perform movement estimation using a flow vector for analysis points P22-P24 and P32-P34 that overlap the non-interest tissue region NFA5. The processor 110 estimates the movement amounts of analysis points P22-P24 and P32-P34 that overlap the non-interest tissue region NFA5 by propagating the movement amounts of the surrounding analysis points P11-P14, P21, and P31 that have been updated using the flow vector. The propagation method uses statistics of the amount of movement of a plurality of neighboring analysis points. The statistics may be, for example, an average, a median, or a weighted average considering distance. Since it is considered that the more analysis points that have been updated in position in the vicinity of an analysis point, the higher the reliability of the correction is, the processor 110 may sequentially perform correction starting from such an analysis point. For example, the processor 110 sets the statistics of the amount of movement of the analysis points P11 to P13, P21, and P31 that are neighbors of the analysis point P22 as an estimate of the amount of movement of the analysis point P22. Next, the processor 110 sets the statistics of the amount of movement of the analysis points P12 to P14, and P22 that are neighbors of the analysis point P23 as an estimate of the amount of movement of the analysis point P23. Here, the estimate of the amount of movement of P22 calculated earlier is used.

[0231] Fig. 80 is an explanatory diagram of a modified example of the evaluation mesh update step S154. As shown in Fig. 80, when the camera of the endoscope zooms in or out, an apparent movement of the subject occurs on the image, and the amount of deformation of the evaluation mesh 410 increases or decreases depending on the apparent movement. Therefore, even though the subject has not actually deformed, information is presented as if the amount of deformation has increased or decreased. Such a problem can occur not only due to zooming in or out, but also due to various movements of the camera.

[0232] Therefore, the processor 110 generates and stores a deformation amount calculation evaluation mesh separately from the normal evaluation mesh, and calculates the deformation amount of the evaluation mesh using the deformation amount calculation evaluation mesh. The deformation amount calculation evaluation mesh is an evaluation mesh that has been corrected for the amount of camera movement in order to be used in calculating the deformation amount of the evaluation mesh. The processor 110 calculates the deformation amount calculation evaluation mesh in the following three steps.

[0233] In a first step, the processor 110 estimates a transformation matrix indicating the camera movement between the previous frame and the current frame using an image processing technique. In a second step, the processor 110 calculates a transformation matrix indicating the camera movement between the initial frame and the current frame. In a third step, the processor 110 calculates an evaluation mesh for deformation calculation using the transformation matrix indicating the camera movement between the initial frame and the current frame.

[0234] A specific example of the first step will be described. The transformation matrix indicating the camera movement between the previous frame and the current frame can be estimated by known techniques. For example, the OpenCV findHomography function can be used. However, since there is a problem with estimation accuracy in abdominal cavity images due to large noise caused by deformation of the foreground tissue, the following methods may be combined.

[0235] It can be assumed that the tissue in the foreground is deformed by the operation of the surgeon or assistant, while the other areas are almost stationary in space. For this reason, the processor 110 may separate the foreground and background of the endoscopic image, and apply a transformation matrix estimation method to only the background area. This makes it possible to accurately estimate the movement of the camera. The method of separating the foreground and background is, for example, a method using depth information as shown in FIG. 44, a method using segmentation such as GrabCut, or a method using the presence or absence of movement of a flow vector. In the case of abdominal cavity images, since the light in the space is dominated by the endoscope light source, it is possible to use brightness information like depth information based on the inverse square law of illuminance and distance.

[0236] A specific example of the second step will be described. As shown in the following formula (1), the processor 110 calculates a transformation matrix indicating the camera movement from the initial frame to the current frame by accumulating transformation matrices indicating the camera movement between the previous frame and the current frame at each time point. In the following formula (1), j H i is the transformation matrix that represents the camera motion between the frame at time instant i and the frame at time instant j. Let the time instant of the current frame be t and the time instant of the initial frame be 0.

[0237]

number

[0238] A specific example of the third step will be described. As shown in the following formulas (2) and (3), the processor 110 calculates an evaluation mesh for deformation calculation by multiplying each point of the evaluation mesh by the inverse matrix of the transformation matrix indicating the camera movement from the initial frame to the current frame. Multiplying by the inverse matrix is ​​equivalent to projecting onto the z=1 plane. In the following formula (2), ( T x, T y) is the position of the analysis point of the evaluation mesh in the coordinate system of the current frame. In the following equation (3), ( O x, O y) is the position of the analysis point of the evaluation mesh for calculating the deformation amount in the current frame in the coordinate system of the initial frame. By canceling the camera movement, the position of the analysis point of the current frame is indicated in the coordinate system of the initial frame, and an evaluation mesh for calculating the deformation amount that reflects only the movement of the subject is obtained.

[0239]

number

[0240]

number

[0241] In this embodiment, the image processing system 100 includes a memory 120. The memory 120 stores a trained model for identifying a non-interest region from an image. The processor 110 inputs the time-series image to the trained model and acquires the identification result of the non-interest region from the trained model. The processor 110 calculates the deformation amount of each cell of the evaluation mesh 410 based on the magnitude and direction of the movement amount of the analysis point that does not overlap the non-interest region. Note that the "non-interest region" corresponds to the above-mentioned non-interest tissue region, and is, for example, a region of a treatment tool, gauze, or tissue that is not a treatment target in an endoscopic image. Note that, for example, the trained model 122 in FIG. 6 includes a region identification model, and the non-interest region is estimated by the region identification model.

[0242] There is a possibility that the movement of the tissue to be treated will not be tracked correctly if the movement of the non-interest region is affected in the deformation analysis of the evaluation mesh 410. According to this embodiment, the deformation analysis of the evaluation mesh 410 can be performed without being affected by the amount of movement of the analysis point that overlaps the non-interest region, so that the evaluation mesh 410 that correctly tracks the movement of the tissue to be treated can be obtained.

[0243] In this embodiment, the non-interest region may be a region of a treatment tool in the time-series images.

[0244] According to this embodiment, it is possible to obtain an evaluation mesh 410 that correctly tracks the movement of the tissue to be treated without being affected by the movement of the treatment tool.

[0245] In this embodiment, processor 110 may estimate the magnitude and direction of the movement of an analysis point that overlaps a non-interest region from the magnitude and direction of the movement of analysis points that do not overlap the non-interest region around the analysis point. In the example of Figure 79, P22 to P24 and P32 to P34 correspond to analysis points that overlap the non-interest region.

[0246] According to this embodiment, the movement amount of an analysis point overlapping a non-interest region can be estimated from the movement amount of the surroundings, i.e., the movement amount of the surrounding tissue. This makes it possible to obtain an evaluation mesh 410 that is not influenced by the movement of the non-interest region and that correctly estimates the movement of the tissue to be treated.

[0247] 5. Fourth embodiment The configuration of the medical system is similar to that of the medical system described in Fig. 5 of the first embodiment. Moreover, the configuration of the image processing system 100 is similar to that of the image processing system 100 described in Fig. 6 of the first embodiment. Methods (1) to (3) relating to some functions will be described below.

[0248] (1) Method for determining the start or end of a function

[0249] 81, the processor 110 recognizes the tissue deformation caused by traction from the endoscopic image, and presents information about the deformation to the surgeon or assistant. That is, the processor 110 makes each analysis point of the evaluation mesh 610 track the deformation of the tissue, and superimposes color information corresponding to the deformation of the cells of the evaluation mesh 410 on the endoscopic image and displays it on the monitor.

[0250] In order to display the tissue deformation caused by traction, it is necessary to set the start and end points of the display function during a series of procedures, i.e., the reference state of the deformation and the end point of the display. It is possible for the surgeon or assistant surgeon to input the data externally, but during the procedure, the surgeon's hands are basically occupied with operating the device, making external input difficult.

[0251] Therefore, the processor 110 recognizes the operating state of the forceps or treatment tool of the surgeon or assistant from the endoscopic image, and sets the timing of the start or end of the function using the recognition result. This makes it possible to present traction information only to the timing required by the surgeon, without any special additional operations. Since the medical system automatically sets the timing of the start or end of the function without the intervention of the surgeon, additional operations by the surgeon can be reduced. Two examples (1a) and (1b) are shown below.

[0252] (1a) Pre-registration of patterns

[0253] Patterns (conditions) according to procedures and scenes are registered in advance in the image processing system 100. For example, the memory 120 in FIG. 6 stores patterns (conditions) registered in advance. The processor 110 starts a function when the conditions of the pattern are satisfied, and ends the function when the conditions of the pattern are not satisfied. FIG. 82 shows an example of a pattern. In FIG. 82, the assistant forceps 607, the operator forceps 605, and the energy treatment tool 606 are shown, and the condition that the operator grasps tissue with the forceps 605 is set as the pattern of energy treatment. That is, the processor 110 starts a function and sets the evaluation mesh 610 when it recognizes that the forceps 605 has grasped tissue, and ends the function when it recognizes that the forceps 605 is not grasping. The patterns are not limited to this, and may be various depending on the procedure or scene. For example, the condition that the assistant grasps tissue with the forceps of both hands may be set as the pattern of surgical field development. The patterns registered in advance may be updated later. As a method for recognizing a pattern from an image or the like, a method for recognizing the state in which tissue is gripped by forceps or a treatment tool can be considered.

[0254] Fig. 83 shows a processing flow of the image processing system 100 in the fourth embodiment. In steps S210 to S213, the processor 110 detects from the endoscopic image that the forceps or treatment tool has come into contact with (or grasped) the tissue, and starts or ends the function of the tissue deformation recognition step S215 based on a predetermined rule. The processor 110 recognizes the tissue deformation information in the tissue deformation recognition step S215, and presents the information to the surgeon or assistant in the traction state presentation step S216. Each step will be described in detail below.

[0255] Note that S210 to S213 correspond to towing scene determination step S21 in Fig. 7, S215 corresponds to towing range estimation step S23 in Fig. 7, and S216 corresponds to presented information processing step S25 in Fig. 7. The processes described in S215 and S216 are merely examples, and the methods of towing range estimation step S23 and presented information processing step S25 described in the first embodiment and the like may be appropriately adopted.

[0256] In step S210, an endoscope image is input to the processor 110. In device detection step S211, the processor 110 detects forceps or treatment tools in the input endoscope image. A network is used for the detection. The network is machine-trained using learning data in which the positions of the forceps or treatment tools are annotated in the endoscope image. FIG. 84 shows an example of learning data and a recognition result in the case where an object detection model is used. In the example of learning data, forceps 608 are shown in the endoscope image, and a boundary box BA is annotated on the jaw. In the learning stage, the network is trained using a large number of annotated endoscope images. In the inference stage, the processor 110 inputs an endoscope image to the network. Here, forceps 601 to 604 are shown in the endoscope image. The network detects the jaws of the forceps 601 to 604 and outputs boundary boxes BB1 to BB4 indicating the positions of the jaws as an attention area. In addition, segmentation may be used to detect the forceps or treatment tool.

[0257] In the contact detection step S212, the processor 110 detects the contact state of the tissue based on the region of interest extracted in the device detection step S211. A network is used for the detection. The network is machine-trained using learning data in which the contact state of the forceps or treatment tool and the tissue is labeled on the endoscopic image. The contact state may be, for example, contact or non-contact between the jaw and the tissue, or gripping or non-gripping of the tissue by the jaw. In the latter case, contact is detected when the jaw is gripping the tissue. FIG. 85 shows an example of the learning data and the recognition result. The learning data includes an endoscopic image showing a state in which the forceps 608 grips the tissue, and an endoscopic image showing a state in which the forceps 609 does not grip the tissue. The former endoscopic image is labeled as "grasping", and the latter endoscopic image is labeled as "non-grasping". In the learning stage, the network is trained using a large number of labeled endoscopic images. In the inference stage, the processor 110 inputs the endoscopic image to the network. Here, the endoscopic image shows forceps 601 to 604. The network detects the gripping state of the forceps 601 to 604, and outputs a boundary box BC indicating the jaw position of the forceps 601 that is determined to be gripping.

[0258] In the start / end determination step S213, the processor 110 determines whether to start or end the recognition in the tissue deformation recognition step S215 based on the contact state between the forceps or treatment tool and the tissue detected in the contact detection step S212 and the preset conditions. The preset conditions are as described in FIG.

[0259] In tissue deformation recognition step S215, the processor 110 recognizes the deformation of the tissue caused by traction by the forceps or the like from the endoscopic image. As shown in Fig. 86, the processor 110 defines an evaluation mesh 610 for the endoscopic image. The processor 110 performs feature point tracking for each analysis point of the evaluation mesh 610, thereby recognizing the behavior of the tissue caused by traction by the forceps 621 from the endoscopic image. The processor 110 acquires the deformation amount of each analysis point or each cell from the evaluation mesh 610.

[0260] In the traction state presentation step S216, the processor 110 highlights each cell of the evaluation mesh 610 with a color or the like based on the deformation amount of the evaluation mesh 610 obtained in the tissue deformation recognition step S215. This display is displayed on the monitor superimposed on the endoscopic image. In this way, information on the deformation of the tissue is presented to the surgeon or assistant. As shown in FIG. 87, the medical system may include a main monitor and a sub-monitor. The processor 110 may display an endoscopic image on which information on the tissue deformation is not superimposed on the main monitor, and an endoscopic image on which information on the tissue deformation is superimposed on the sub-monitor. FIG. 87 shows an example in which the surgeon, who is a surgeon, and the assistant, sees the monitor, but the assistant may see the monitor. Alternatively, the medical system may include a monitor for the surgeon and a monitor for the assistant. The processor 110 may display information suitable for each monitor. For example, the processor 110 may display an endoscopic image on which information on the tissue deformation suitable for the surgeon is superimposed on the monitor for the surgeon, and display an endoscopic image on which information on the tissue deformation suitable for the assistant is superimposed on the monitor for the assistant.

[0261] According to the above-described method, the traction information can be presented only at the timing required by the surgeon. In addition, since the image processing system 100 automatically sets the timing, the additional operation by the surgeon can be reduced.

[0262] (1b) History storage

[0263] The method is basically the same as method (1a), but processor 110 may recognize whether tissue is being grasped or not grasped by the forceps in contact detection step S212, and store the recognition result in memory 120 of Fig. 6 or the like until the recognition result is switched. For example, when processor 110 recognizes grasping, it stores the recognition result of grasping in memory 120, does not update memory 120 until it recognizes non-grasping, and when it recognizes non-grasping, updates memory 120 with the recognition result of non-grasping.

[0264] There may be cases where the forceps disappear from the field of view of the camera due to camera movement or tissue pulling by the forceps. Such a situation is expected to occur especially with assistant forceps. In the method of recognizing the grasping or non-grasping state from the endoscopic image, if the forceps move out of the field of view of the camera, it is not possible to correctly recognize the grasping or non-grasping state.

[0265] Therefore, the processor 110 stores and uses provisional recognition results in the memory 120 for the grasping or non-grasping state until the recognition results based on the endoscopic image are updated, so that the function can be applied even if the forceps move out of the field of view. Even if the forceps are once out of the field of view, they always come back into the field of view when grasping tissue, so grasping can be detected from the endoscopic image. On the other hand, it is conceivable that the forceps will change from grasping to non-grasping outside the field of view. However, in such a case, it is not a problem because the surgeon himself will notice, and the function will end the next time the forceps in a non-grasping state move into the field of view.

[0266] (2) Assistant assistance to indicate loosening of traction

[0267] The assistant may not be aware of the loosening of the tissue caused by the surgeon's procedure and may not be able to maintain proper traction. The assistant may not know how much to pull the loosened tissue to return it to an appropriate traction state. As a result, the surgeon may have to give instructions to the assistant more frequently.

[0268] Therefore, the processor 110 presents the loosening of the tissue accompanying the treatment by the surgeon on the monitor using color information or the like. An example is shown in FIG. 88. An assistant forceps 631, an operator forceps 635, and an energy treatment tool 636 are shown in an endoscopic image. The processor 110 colors the cells in which deformation corresponding to the loosening of the tissue has occurred based on the deformation analysis of the evaluation mesh 610. In FIG. 88, the colored cells are shown by the hatched area ARL. The assistant retracts the tissue with the forceps 631 so that the color of the area ARL disappears, that is, so that the loosening of the tissue is eliminated. Thus, according to this embodiment, the assistant notices the loosening of the tissue and adjusts the retraction so that the presented color information or the like disappears, so that the assistant can autonomously return to an appropriate retraction state.

[0269] FIG. 89 shows the first flow of loosening presentation. In step S161, the function of presenting the traction state is off. The assistant grasps and traction the tissue with the forceps 631. In step S162, when the processor 110 detects that the assistant has completed the tissue deployment, the processor 110 presents the evaluation mesh 610 showing the tissue deformation. The evaluation mesh 610 here is in a reference state of a regular shape. In step S163, the processor 110 causes the analysis points of the evaluation mesh 610 to follow the tissue deformation, and deforms the evaluation mesh 610. The processor 110 may narrow down the evaluation region as described in the second embodiment. FIG. 89 shows an example of narrowing down. In step S164, the processor 110 determines the region ARL where the tissue is loosened based on the deformation amount of each cell of the evaluation mesh 610, and presents the cells of the region ARL with color information on the monitor. In step S165, the assistant traction the tissue with the forceps 631 so that the color information of the region ARL disappears. As shown in step S165, the processor 110 turns off the presentation of the color information when the cell of the evaluation mesh 610 of the area ARL has returned to or approached the reference state.

[0270] FIG. 90 shows the second flow of loosening presentation. In step S171, the function of presenting the traction state is off. The assistant grasps and traction the tissue with the forceps 631. In step S172, when the processor 110 detects that the assistant has completed the tissue deployment, the processor 110 arranges the evaluation mesh 610 for tracking the tissue deformation. The evaluation mesh 610 may be displayed or hidden, and is used for the calculation of the tissue deformation tracking. The evaluation mesh 610 here is a reference state of a regular shape. In step S173, the processor 110 causes the analysis points of the evaluation mesh 610 to follow the tissue deformation, and deforms the evaluation mesh 610. The processor 110 calculates the deformation amount of each cell. The processor 110 may narrow down the evaluation region as described in the second embodiment. FIG. 90 shows an example of narrowing down. In step S174, the processor 110 determines the region ARL where the tissue has loosened based on the deformation amount of each cell of the evaluation mesh 610, and displays the cells of the region ARL with color information attached to them on the monitor. The processor 110 may display only the color information of the region ARL without displaying the evaluation mesh 610, or may display the color information of the region ARL together with the evaluation mesh 610. If the evaluation mesh 610 is not displayed in step S172, the information is displayed only when loosening is determined in step S174.

[0271] (3) Support for surgeon traction by displaying tissue deformation

[0272] A non-expert surgeon does not realize that the traction by the forceps in the left hand is insufficient. For example, a non-expert surgeon does not notice that the traction is insufficient because it is difficult to pay attention to the forceps in the left hand. A non-expert surgeon does not know at least how much the tissue should be pulled depending on the scene or tissue. As a result, when using a monopolar treatment tool, the traction by the forceps in the left hand becomes weak, and heat is diffused to the surrounding tissue of the tissue treated with the monopolar treatment tool.

[0273] Therefore, the processor 110 presents the range of tissue deformed by the traction of the forceps of the left hand of the surgeon using color information. An example is shown in FIG. 91. An assistant forceps 631, an operator forceps 635, and an energy treatment tool 636 are shown in an endoscopic image. The processor 110 colors cells indicating that the tissue has been deformed by the traction of the operator forceps 635 based on the deformation analysis of the evaluation mesh 610. In FIG. 91, the colored cells are shown by hatched areas ARP. By looking at the color information of the area ARP, the surgeon can easily notice whether the traction by the forceps 635 of the left hand is appropriate. According to this embodiment, the frequency of tissue treatment in a state where the tissue is weakly tractioned is reduced, and heat diffusion due to insufficient traction can be suppressed.

[0274] FIG. 92 shows a first flow of the surgeon traction support. In step S181, the function of presenting the traction state is off. This is a state before the surgeon grasps the tissue with the forceps 635. In step S182, when the processor 110 detects that the surgeon has completed grasping the tissue with the forceps 635, it presents the evaluation mesh 610 showing the tissue deformation. The evaluation mesh 610 here is in a reference state of a regular shape. In step S183, the processor 110 causes the analysis points of the evaluation mesh 610 to follow the tissue deformation, and deforms the evaluation mesh 610. The processor 110 presents the area with a larger amount of deformation of the cell in a darker color. In this example, the state in which the amount of deformation of the cell is small in step S183 and almost no color is applied is shown as an example. As shown in step S184, when the surgeon further pulls the tissue with the forceps 635 and the deformation of the tissue becomes larger, the area where the color is presented expands.

[0275] The second flow of the surgeon traction support will be described. As a flow diagram, FIG. 92, which is the same as the first flow, is used. In step S181, the function of presenting the traction state is turned off. In step S182, when the processor 110 detects that the surgeon has completed grasping the tissue with the forceps 635, the processor 110 arranges the evaluation mesh 610 indicating the tissue deformation. The evaluation mesh 610 may be displayed or hidden, and is used in the calculation of the tissue deformation tracking. In step S183, the processor 110 causes the analysis points of the evaluation mesh 610 to follow the tissue deformation, and deforms the evaluation mesh 610. The processor 110 calculates the deformation amount of each cell, and the greater the deformation amount of the cell, the darker the color is presented. FIG. 93 shows an example of the relationship between the deformation amount and the brightness or darkness of the color. In this example, a sigmoid function is used, and when the deformation amount is x, f(x)=1 / (1+e -ax ), where a is an arbitrary positive coefficient. FIG. 93 shows two sigmoid functions with different a, using a solid line and a dotted line. Step S183 in FIG. 92 shows an example in which the amount of deformation of the cell is small and almost no color is applied. As shown in step S184, when the surgeon further pulls the tissue with the forceps 635 and the deformation of the tissue increases, the area in which the color is presented expands and the presented color becomes darker.

[0276] Although the embodiments to which the present disclosure is applied and their modifications have been described above, the present disclosure is not limited to each embodiment or its modifications as it is, and in the implementation stage, the components can be modified and embodied within the scope of the gist of the disclosure. In addition, various disclosures can be formed by appropriately combining multiple components disclosed in each of the above-mentioned embodiments and modifications. For example, some components may be deleted from all the components described in each embodiment or modification. Furthermore, components described in different embodiments or modifications may be appropriately combined. In this way, various modifications and applications are possible within the scope of the gist of the disclosure. In addition, a term described at least once in the specification or drawings together with a different term having a broader meaning or the same meaning can be replaced with that different term in any part of the specification or drawings. [Explanation of symbols]

[0277] 1...tissue, 11, 12...assistant forceps, 15...operator forceps, 16...energy treatment tool, 100...image processing system, 110...processor, 120...memory, 121...program, 122...trained model, 150...operation unit, 160...communication unit, 500...endoscope, 700...monitor, S21...traction scene determination step, S22...area analysis step, S23...traction range estimation step, S25...presentation information processing step

Claims

1. a processor that sequentially acquires time-series images captured by an endoscope and performs deformation analysis processing on the time-series images, The processor, In the deformation analysis process, An evaluation mesh including a plurality of analysis points is arranged for an image at an arbitrary timing among the time series images; deforming the evaluation mesh in each image such that each analysis point in each image of the time-series images tracks a feature point of a subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is arranged; Calculating a deformation amount of each cell of the evaluation mesh based on the magnitude and direction of the movement of each of the analysis points in each of the images; presenting information regarding the deformation of the evaluation mesh based on the calculated deformation amount; 1. An image processing system comprising:

2. In claim 1, The processor, An image processing system comprising: a display of a mode according to the amount of deformation in each of the images of the time-series images is superimposed on each of the images.

3. In claim 2, The processor, an image processing system comprising: a display in which each of the cells is colored in a color corresponding to the amount of deformation of the each of the cells; and a display in which the cells are colored in a color corresponding to the amount of deformation of the each of the cells.

4. In claim 1, The processor, Presenting the time series images on a first monitor; An image processing system characterized in that the time-series images and the information regarding the deformation of the evaluation mesh are displayed on a second monitor.

5. In claim 1, The processor, An image processing system characterized by determining an evaluation area, which is an area in which the evaluation mesh is to be placed or an evaluation area of ​​the evaluation mesh to be reflected in an image display, based on at least one of the amount of movement of the subject between the time-series images, the image features of the subject, and the depth information of the subject.

6. In claim 5, The processor, Calculating an accumulated movement amount obtained by accumulating an amount of movement between frames or at a predetermined interval for each point of the subject on the time-series images from a reference timing until a predetermined period has elapsed; an image processing system that sets the evaluation area by excluding an area in which the accumulated movement amount is equal to or less than a threshold value;

7. In claim 5, The processor, Calculating the amount of movement of each point of the subject on the time-series images; A total value is calculated by adding up the amount of movement of each of the points within the image; Calculate a total movement amount by accumulating the sum from a reference timing; When the total movement amount exceeds a first threshold, a cumulative movement amount is calculated by accumulating the movement amount of each point between frames or at a predetermined interval from the reference timing to the timing at which the total movement amount exceeds the first threshold, an image processing system that sets the evaluation area by excluding an area in which the accumulated movement amount is equal to or less than a second threshold value;

8. In claim 5, The processor, The time-series images are input to a trained model that estimates the evaluation region from an endoscopic image by segmentation; An image processing system comprising: a first section for determining an evaluation area based on an estimation result of the trained model;

9. In claim 5, The processor, The time-series images are input to a trained model that divides images into regions by segmentation; An image processing system characterized in that the evaluation area is set to the area among the multiple areas divided by the trained model that has the largest overlap with a predetermined area in the image.

10. In claim 5, The processor, performing edge detection processing on the time-series images; An image processing system comprising: a closed area defined by the detected edges set as the evaluation area.

11. In claim 5, The processor, A pulling direction of a treatment tool pulling the subject is detected from the time-series images; an image processing system, wherein the evaluation area is set on an opposite side of the traction direction from a predetermined position on the treatment tool;

12. In claim 5, The processor, acquiring a depth distribution from the endoscope to the subject; An image processing system characterized in that the evaluation area is set in an area on the smaller depth side, with a line where the depth changes sharply in the depth distribution as the boundary.

13. In claim 1, A memory for storing a trained model for identifying a non-interest region from an image, The processor, Input the time-series images to the trained model, and obtain a classification result of the non-interest region from the trained model; An image processing system comprising: a processor that calculates the deformation amount of each cell of the evaluation mesh based on the magnitude and direction of the movement of an analysis point that does not overlap the non-interest region.

14. In claim 13, The image processing system according to claim 1, wherein the non-interest region is a region of a treatment tool in the time-series images.

15. In claim 13, The processor, An image processing system characterized by estimating the magnitude and direction of the movement of an analysis point that overlaps the non-interest area from the magnitude and direction of the movement of analysis points around the analysis point that do not overlap the non-interest area.

16. In claim 1, The processor, determining whether the scene is a traction scene in which the subject is tractioned by a treatment tool based on the time-series images or a user input; an image processing system for analyzing deformation of the subject due to towing by performing the deformation analysis process on the evaluation mesh when the towing scene is determined to be a towing scene.

17. In claim 1, The processor, determining an area of ​​the subject that is under tension due to traction based on the deformation amount of each of the cells in each of the images of the time series images; An image processing system comprising: a display showing the determined area superimposed on each of the images.

18. In claim 5, The processor, An image processing system characterized by updating the evaluation area by making a decision to exclude a portion of the evaluation area or to maintain the evaluation area based on at least one of the movement amount, the image feature amount, and the depth information at each given update timing.

19. Sequentially acquiring time-series images captured by an endoscope; A step of arranging an evaluation mesh including a plurality of analysis points for an image at an arbitrary timing among the time series images; a step of deforming the evaluation mesh in each image such that each analysis point in each image of the time-series images tracks a feature point of a subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is arranged; calculating a deformation amount of each cell of the evaluation mesh based on a magnitude and a direction of a movement amount of each of the analysis points in each of the images; presenting information regarding the deformation of the evaluation mesh based on the calculated deformation amount; 13. An image processing method comprising:

20. Sequentially acquiring time-series images captured by an endoscope; A step of arranging an evaluation mesh including a plurality of analysis points for an image at an arbitrary timing among the time series images; a step of deforming the evaluation mesh in each image such that each analysis point in each image of the time-series images tracks a feature point of a subject located at each analysis point in the image at the arbitrary timing in which the evaluation mesh is arranged; calculating a deformation amount of each cell of the evaluation mesh based on a magnitude and a direction of a movement amount of each of the analysis points in each of the images; presenting information regarding the deformation of the evaluation mesh based on the calculated deformation amount; A non-transitory information storage medium that stores a program that causes a computer to execute the above.

Images