Endoscopic system, control method, and program

Abstract

The present disclosure pertains to an endoscope system, a control method, and a program that make it possible to reduce erroneous assessments when assessing an observation region. Provided is an endoscope system comprising an imaging device which is connected to an insertion part and carries out imaging by using an image sensor to receive light guided via the insertion part, and a control device which controls the imaging device, wherein: the control device has one or more processors and one or more storage devices on which a program is stored; and the processor executes the program so as to assess, two or more times, the size of an observation region, which is a different region from a vignetting region occurring on the image sensor due to the insertion part, and on the basis of comparison results obtained by comparing the assessment results from the two or more times, executes a parameter changing process that is based on the assessment results. The present disclosure can be applied, for example, in an endoscope system that uses a surgical endoscope.

Description

Technical Field

[0001] The present disclosure relates to an endoscope system, a control method, and a program, and particularly to an endoscope system, a control method, and a program that can suppress misjudgment when determining an observation region.

Background Art

[0002] Generally, a surgical endoscope uses a camera head to which a scope is connected, and observes the surgical field by inserting the scope into a patient.

[0003] The scope is removable, and the scope to be used is selected by the surgical staff from among a plurality of types of scopes. At this time, since the mechanical flexibility and properties differ depending on the type of the scope, it is necessary to adjust the subsequent image processing according to the type of the scope. Therefore, a method for discriminating the type of the scope is required.

[0004] As this type of discrimination method, for example, the technique disclosed in Patent Document 1 is known. In Patent Document 1, a plurality of evaluation frames arranged at a predetermined interval are set for an endoscope image captured using a scope, an evaluation value for each of the set plurality of evaluation frames is calculated, and the type of the scope is estimated based on the relationship of the calculated evaluation values.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In medical settings using surgical endoscopes, when high-brightness objects such as forceps or gauze are present in the imaging area, or when the scope is inserted into the patient's body, the amount of reflected light is large, causing the vignetting area to temporarily become brighter, which may lead to misjudgment when determining the observation area corresponding to the type of scope.

[0007] This disclosure is made in light of these circumstances and aims to suppress misjudgments when determining the observation area. [Means for solving the problem]

[0008] An endoscope system according to one aspect of the present disclosure comprises an imaging device to which an insertion section is connected, which receives light guided through the insertion section with an image sensor and takes an image, and a control device that controls the imaging device, wherein the control device has one or more processors and one or more storage devices that store a program, and the processor executes the program to determine at least twice the size of an observation area which is a different area from the vignetting area generated on the image sensor by the insertion section, and performs parameter modification processing based on the determination result based on the comparison result obtained by comparing the two or more determination results.

[0009] One aspect of the present disclosure is a control method in which a control device that controls an imaging device to which an insertion part is connected and which receives light guided through the insertion part with an image sensor to take an image, determines the size of an observation area, which is a different area from the vignetting area generated on the image sensor by the insertion part, at least twice, and performs a parameter change process based on the comparison result obtained by comparing the two or more determination results.

[0010] One aspect of this disclosure is a program that causes a computer to function as a control device that determines at least twice the size of an observation area, which is a region different from the vignetting region generated on the image sensor by the insertion part connected to an imaging device that receives light guided through the insertion part with an image sensor and takes an image, and then performs parameter modification processing based on the comparison result obtained by comparing the two or more determination results.

[0011] In one aspect of the present disclosure, an endoscope system, a control method, and a program are provided. In an imaging device that receives light guided through the insertion unit and captures it with an image sensor, the size of the observation area, which is a region different from the vignetting region that occurs on the image sensor, is determined at least twice by the insertion unit, which is connected to the imaging device. Based on the comparison result obtained by comparing the two or more determination results, a parameter modification process based on the determination result is performed.

[0012] Furthermore, the imaging device and control device included in one aspect of the endoscopic system of this disclosure may be independent devices or internal blocks constituting a single device. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows a schematic example of the configuration of an endoscope system to which the technology described herein is applied. [Figure 2] This is a diagram showing a detailed example of the configuration of an endoscope. [Figure 3] This is a block diagram showing a detailed configuration example of a camera head and control device. [Figure 4] This figure shows an example of the configuration of a control program. [Figure 5] This figure shows the first example of an endoscopic image. [Figure 6] This figure shows a second example of an endoscopic image. [Figure 7] This is a flowchart explaining the first processing flow. [Figure 8] This figure shows examples of multiple evaluation frames set in the first process. [Figure 9] It is a flowchart for explaining the flow of the second process. [Figure 10] It is a diagram showing examples of a plurality of evaluation frames set in the second process. [Figure 11] It is a flowchart for explaining the flow of the third process. [Figure 12] It is a diagram showing examples of a plurality of evaluation frames set in the third process. [Figure 13] It is a diagram showing examples of a plurality of evaluation frames set in the high-precision calculation process. [Figure 14] It is a diagram showing examples of a plurality of evaluation frames set in the high-precision calculation process. [Figure 15] It is a flowchart for explaining the flow of the confirmation process. [Figure 16] It is a diagram showing examples of the recognition result and mask diameter information notified to an external module. [Figure 17] It is a diagram showing an example of display of mask diameter information. [Figure 18] It is a diagram showing an example of display of a virtual frame.

Mode for Carrying Out the Invention

[0014] <System Configuration> The outline of a system to which the technology according to the present disclosure can be applied is explained. FIG. 1 is a diagram showing a schematic configuration example of an endoscope system to which the technology according to the present disclosure is applied.

[0015] In FIG. 1, a state where an operator (doctor) 3 is performing a surgery on a patient 4 on a patient bed 2 using an endoscopic surgery system 1 is shown. In FIG. 1, the endoscopic surgery system 1 includes an endoscope 10, other surgical instruments 20 such as a pneumoperitoneum tube 21, an energy treatment instrument 22, and forceps 23, a support arm device 30 that supports the endoscope 10, and a cart 50 on which various devices for endoscopic surgery are mounted.

[0016] The endoscope 10 consists of a scope 101, the tip of which is inserted into the patient's body for a predetermined length, and a camera head 102 connected to the base end of the scope 101. In the example in Figure 1, the endoscope 10 is shown as a rigid endoscope having a rigid tube, but the endoscope 10 may also be configured as a flexible endoscope having a flexible tube.

[0017] The tip of the scope 101 is provided with an opening into which an objective lens is fitted. The endoscope 10 is connected to a light source device 53, and the light (irradiation light) generated by the light source device 53 is guided to the tip of the scope tube by a light guide extending inside the scope 101, and is irradiated towards the object to be observed inside the patient's body 4 through the objective lens. The endoscope 10 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0018] The camera head 102 has an optical system and an image sensor inside. The optical system focuses the light reflected from the object being observed (observation light) onto the image sensor. The image sensor converts the observation light into an image signal corresponding to the subject image. This image signal is transmitted to the control device 51 as RAW data (RAW image).

[0019] The control device 51 is a camera control unit (CCU) that includes a processor such as a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), and a storage device. The control device 51 comprehensively controls the operation of each device, including the endoscope 10 and the display device 52. The control device 51 also receives an image signal from the camera head 102 and performs various image processing operations on the image signal, such as development processing (demosaic processing), to display an observation image (display image) based on the image signal.

[0020] The display device 52 displays a display image based on an image signal processed by the control device 51, in accordance with the control from the control device 51. The light source device 53 is configured to include a light source such as an LED (Light Emitting Diode). The light source device 53 supplies illumination light to the endoscope 10 when photographing the surgical area, etc. The illumination light includes, for example, normal light such as white light as light in a first wavelength band (first wavelength light) and special light such as IR (Infrared) light as light in a second wavelength band (second wavelength light). The IR light may also include NIR (Near Infrared) light.

[0021] The input device 54 is an input interface for the endoscopic surgical system 1. The user can input various information and instructions to the endoscopic surgical system 1 via the input device 54. For example, the user can input instructions to change the imaging conditions (type of light, magnification, focal length, etc.) of the endoscope 10.

[0022] The treatment instrument control device 55 controls the drive of the energy treatment instrument 22 for purposes such as tissue cauterization, incision, or blood vessel sealing. The insufflation device 56 delivers gas into the patient's body via the insufflation tube 21 to inflate the patient's body cavity for the purpose of securing a field of view for the endoscope 10 and securing the operator's workspace. The recorder 57 is a device capable of recording various information related to the surgery. The printer 58 is a device capable of printing various information related to the surgery in various formats such as text, images, or graphs.

[0023] Figure 2 shows a detailed configuration example of the endoscope 10 shown in Figure 1.

[0024] In Figure 2, the endoscope 10 comprises a scope 101 and a camera head 102. In the endoscope 10, the scope 101 is connected to a light source device 53 via a light guide 121, and the camera head 102 is connected to a control device 51 via a transmission cable 122. The control device 51 is connected to a display device 52 via a transmission cable 123 and to the light source device 53 via a transmission cable 124.

[0025] The scope 101 is configured as a rigid endoscope. That is, the scope 101 is a rigid or at least partially flexible, elongated insertable part (scope tube) that is inserted into the body of the patient 4. Inside the scope 101, there is an optical system that uses one or more lenses to focus the image of the subject.

[0026] One end of the light guide 121 is connected to the light source device 53, and according to the control of the control device 51, it supplies illumination light to one end of the light guide 121 to illuminate the inside of the body. The transmission cable 124 transmits control signals from the control device 51 to the light source device 53. The light guide 121 transmits the illumination light supplied from the light source device 53 from one end to the other and supplies it to the scope 101. The illumination light supplied to the scope 101 is emitted from the tip of the scope 101 and illuminates the inside of the body. The observation light (subject image) that is illuminated inside the body and reflected within the body is focused by the optical system inside the scope 101.

[0027] The camera head 102 is detachably connected to the base end (eyepiece 111) of the scope 101. The camera head 102, following control from the control device 51, captures the observation light (subject image) focused by the scope 101 and outputs the resulting image signal (RAW data). The image signal is, for example, an image signal corresponding to 4K resolution (e.g., 3840 x 2160 pixels). A detailed configuration of the camera head 102 will be described later with reference to Figure 3.

[0028] The transmission cable 122 transmits image signals and the like output from the camera head 102 to the control device 51. The transmission cable 122 also transmits control signals, synchronization signals, and power, etc., output from the control device 51 to the camera head 102. The transmission of image signals and the like via the transmission cable 122 may be done using electrical signals or optical signals. The same applies to the transmission of control signals and synchronization signals via the transmission cable 122. Furthermore, communication between the camera head 102 and the control device 51 is not limited to wired communication using the transmission cable 122; wireless communication conforming to a predetermined communication method may also be used.

[0029] The transmission cable 123 transmits the image signal processed by the control device 51 and the control signal output from the control device 51 to the display device 52. The display device 52 displays an image based on the image signal from the control device 51, in accordance with the control from the control device 51. The display device 52 can output sound (voice) based on the control signal from the control device 51.

[0030] The control device 51 is composed of a processor and storage devices, and comprehensively controls the operation of each device, including the light source device 53, the camera head 102, and the display device 52, by outputting control signals and the like to each device. The control device 51 also processes the image signals output from the camera head 102 and controls the operation of each device based on the processing results. The detailed configuration of the control device 51 will be described later with reference to Figure 3.

[0031] Figure 3 is a block diagram showing a detailed configuration example of the camera head 102 and control device 51 shown in Figures 1 and 2. The camera head 102 is an example of an imaging device to which the technology of this disclosure is applied, and it captures images by receiving light guided through the scope 101 with an image sensor. The control device 51 is an example of a control device to which the technology of this disclosure is applied, and it controls the camera head 102.

[0032] The camera head 102 comprises a lens unit 151, an image sensor 152, a drive circuit 153, a communication circuit 154, and a camera head control circuit 155. The camera head 102 and the control device 51 are connected to each other via a transmission cable 122, enabling communication between them.

[0033] The lens unit 151 is an optical system provided at the connection point with the scope 101. Observation light taken in from the tip of the scope 101 is guided to the camera head 102 and then incident on the lens unit 151. The lens unit 151 is composed of a combination of multiple lenses, including a zoom lens and a focus lens.

[0034] The image sensor 152 is composed of an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor and a signal processing circuit (such as an AD conversion circuit). The image sensor 152 may consist of one image sensor (a so-called single-chip type) or multiple image sensors (a so-called multi-chip type). The image sensor 152 does not necessarily have to be provided in the camera head 102. For example, the image sensor 152 may be provided inside the scope 101, immediately after the objective lens.

[0035] The drive circuit 153 is composed of actuators and the like, and under control from the camera head control circuit 155, moves the zoom lens and focus lens included in the lens unit 151 along the optical axis by a predetermined distance. This adjusts the magnification and focus of the image captured by the image sensor 152 as appropriate.

[0036] The communication circuit 154 is composed of communication devices for sending and receiving various types of information with the control device 51. The communication circuit 154 transmits the image signal obtained from the image sensor 152 as RAW data to the control device 51 via the transmission cable 122. The communication circuit 154 also receives control signals from the control device 51 to control the drive of the camera head 102 and supplies them to the camera head control circuit 155. These control signals include information about imaging conditions, such as the image frame rate, exposure value during imaging, image magnification, and focus.

[0037] The imaging conditions such as frame rate, exposure value, magnification, and focus may be specified as appropriate by the user, or they may be automatically set by the control device 51 (or its processor 161) based on the acquired image signal. In other words, in the latter case, the endoscope 10 is equipped with AE (Auto Exposure), AF (Auto Focus), and AWB (Auto White Balance) functions.

[0038] The camera head control circuit 155 controls the drive of the camera head 102 based on the control signal received from the control device 51 via the communication circuit 154.

[0039] The control device 51 comprises processors 161-1 to 161-i, storage devices 162-1 to 162-j, and a communication circuit 163. i and j are integers greater than or equal to 1, and the control device 51 has one or more processors 161 and one or more storage devices 162. In the following description, processors 161-1 to 161-i will be referred to as processor 161 unless there is a need to distinguish between them. Similarly, storage devices 162-1 to 162-j will be referred to as storage device 162 unless there is a need to distinguish between them.

[0040] The processor 161 is a processor such as a CPU. The processor 161 can realize various functions by reading and executing control programs stored in the storage device 162. The processor 161 performs various controls related to imaging the surgical area, etc. by the endoscope 10, and the display of the endoscopic images (medical images) obtained from imaging the surgical area, etc. For example, the processor 161 generates control signals to control the driving of the camera head 102.

[0041] Furthermore, the processor 161 performs various image processing operations on the image signal transmitted from the camera head 102. Based on the image signal after image processing, the processor 161 displays a display image (endoscopic image) showing the surgical area, etc., on the display device 52. At this time, the processor 161 may recognize various objects within the image using various image recognition technologies.

[0042] For example, the processor 161 can recognize surgical instruments such as forceps, specific biological sites, bleeding, and mist produced when using the energy treatment device 22 by detecting the shape and color of the edges of objects contained in the image. When displaying the image on the display device 52, the processor 161 may use the recognition results to overlay various surgical support information onto the image of the surgical area.

[0043] The storage device 162 is a storage device such as a semiconductor memory device or an HDD (Hard Disk Drive). The storage device 162 stores programs such as control programs executed by the processor 161, and information (data, parameters, etc.) necessary for processing by the processor 161.

[0044] The communication circuit 163 consists of communication devices for sending and receiving various types of information with the camera head 102. The communication circuit 163 receives image signals transmitted from the camera head 102 via the transmission cable 122. The communication circuit 163 also transmits control signals to the camera head 102 to control the driving of the camera head 102. Image signals and control signals can be transmitted by telecommunications, optical communications, etc.

[0045] <Control Program Configuration> Figure 4 shows an example of the configuration of a control program. The control program 171 is stored in the storage device 162 and read and executed by the processor 161.

[0046] The control program 171 includes an evaluation frame setting unit 181, an evaluation value calculation unit 182, an evaluation value determination unit 183, a determination result setting unit 184, a determination result determination unit 185, and a notification target setting unit 186.

[0047] The evaluation frame setting unit 181 sets a plurality of evaluation frames that are arranged at predetermined intervals in the endoscopic image (RAW image) corresponding to the image signal (RAW data). Here, the evaluation frame is an area arbitrarily set in the region (the region of the RAW image) corresponding to the imaging surface of an image sensor having a predetermined array pattern such as a Bayer array, and is a frame for acquiring evaluation values ​​such as brightness information (luminance information).

[0048] The evaluation value calculation unit 182 calculates evaluation values ​​for each of the multiple evaluation frames set by the evaluation frame setting unit 181. These evaluation values ​​include, for example, the brightness information (brightness value) within each evaluation frame.

[0049] The evaluation value determination unit 183 determines the relationships (correlations) between the evaluation values ​​for each of the multiple evaluation frames calculated by the evaluation value calculation unit 182. Here, for example, the relationships between evaluation values ​​corresponding to adjacent or regularly spaced evaluation frames are determined. The determination result setting unit 184 sets the determination results according to the relationships between the evaluation values ​​determined by the evaluation value determination unit 183.

[0050] The determination result determination unit 185 determines whether the determination result set by the determination result setting unit 184 is the same for a predetermined number of consecutive times. The notification target setting unit 186 sets the notification targets to be notified to the external module based on the determination result of the determination unit 185.

[0051] <Summary of the Invention Related to This Disclosure> As mentioned earlier, with the endoscope 10, the scope 101 connected to the camera head 102 is inserted into the patient's body, allowing the surgeon 3 to observe the surgical field.

[0052] For example, Figure 5 shows an example of an endoscopic image 200 based on an image signal obtained when the subject image focused by the scope 101 is captured by the camera head 102. In the endoscopic image 200, the black areas on the left and right represent the vignetting area 200A where mechanical vignetting occurs, and the area of ​​the subject image represents the observation area 200B. The vignetting area 200A can also be called the mask area (black area). The observation area 200B is an effective area different from the vignetting area 200A. The boundary between the vignetting area 200A and the observation area 200B is called the mask edge 220.

[0053] In other words, the endoscope 10 is equipped with a slender scope 101, but the shape of the subject image focused by the scope 101 does not match the shape of the imaging surface of the image sensor 152 of the camera head 102, resulting in mechanical vignetting caused by the scope 101. That is, mechanical vignetting occurs on the image sensor 152 of the camera head 102.

[0054] The endoscopic image 200 is subjected to various image processing by the control device 51 and displayed as a display image by the display device 52. For example, as shown in Figure 6, the operator 3 can perform procedures such as excising the affected area using surgical instruments 20, such as energy treatment instruments 22, while viewing the displayed image in real time.

[0055] In this endoscope 10, the scope 101 is detachable, and the surgical staff selects the scope 101 to be used from among several types of scopes. At this time, since the mechanical vignetting and properties differ depending on the type of scope 101 used, it becomes necessary to adjust the subsequent image processing depending on the type of scope 101. Therefore, a method is needed to determine the observation area 200B corresponding to the type of scope 101.

[0056] As shown in Figures 5 and 6, by determining the size of the observation area 200B, it is possible to determine the type of scope 101, and also the size of the vignetting area 200A. Therefore, it can be said that determining the size of the observation area 200B is equivalent to determining the type of scope 101 and the vignetting area 200A. In other words, determining the type of scope 101, determining the size of the observation area 200B, and determining the vignetting area 200A can be considered conceptually the same thing.

[0057] For example, signal processing related to AE and AF is performed on the observation area 200B, and various problems arise if exposure or focusing is performed including the vignetting area 200A, so it becomes necessary to determine the size of the observation area 200B. For example, the technology disclosed in the aforementioned Patent Document 1 proposes a method for determining the type of scope 101 to be used with less computational effort.

[0058] On the other hand, in medical settings using the endoscopic surgical system 1, when high-brightness objects such as forceps 23 or gauze are present in the imaging area, or when the scope 101 is inserted into the patient's body, the amount of reflected light is large, causing the vignetting area 200A to temporarily become brighter, which may lead to misjudgment when determining the size of the observation area 200B corresponding to the type of scope 101. Therefore, it was necessary to prevent misjudgment when determining the observation area 200B.

[0059] Therefore, the technology disclosed herein proposes a method to suppress misjudgments by continuously performing judgment processing of the observation area 200B corresponding to the type of scope 101 and determining whether the judgment result continued for a predetermined number of consecutive times.

[0060] The following describes the details of the technology related to this disclosure. In the endoscopic surgical system 1 (processor 161 of the control device 51), for example, the first to third processes and a confirmation process are executed to determine the type of scope 101 to be used, and these processes will be described in order. In the following description, the image corresponding to the imaging surface of the image sensor 152 of the camera head 102 will be referred to as the endoscopic image.

[0061] <First Processing Flow> First, referring to the flowchart in Figure 7, we will explain the first processing flow executed by the processor 161 of the control device 51.

[0062] In step S10, the evaluation frame setting unit 181 sets multiple evaluation frames for the image (RAW image) corresponding to the image signal from the camera head 102. In this evaluation frame setting, for example, as shown in Figure 8, evaluation frames 210 are provided at the four corners of the central part and the surrounding part of the endoscopic image 200.

[0063] Specifically, in the endoscopic image 200, a rectangular evaluation frame 210-0 (hereinafter also abbreviated as frame 0) is provided in the center, and rectangular evaluation frames 210-1, 210-2, 210-3, and 210-4 (hereinafter also abbreviated as frame 1, frame 2, frame 3, and frame 4) are provided in the four corners: upper left, lower left, upper right, and lower right. The size of the rectangles in evaluation frames 210-1 to 210-4 is smaller than the size of the rectangle in evaluation frame 210-0.

[0064] Furthermore, in Figure 8, four circles of different diameters are superimposed on the endoscopic image 200, centered approximately on its centroid. These circles correspond to the mask edges 220, which are the boundaries between the vignetting region 200A and the observation region 200B.

[0065] In other words, the diameter of the scope 101 used in the endoscope 10 corresponds to the mask edge 220 in the endoscope image 200, and the position where the mask edge 220 is expected to be detected in the endoscope image 200 is known in advance by design. Therefore, evaluation frames 210-0 to 210-4 are provided here to determine whether or not the vignetting region 200A is included.

[0066] In the following explanation, we assume that there are four types of scope 101 to be used, and that the mask types assigned to each mask edge 220 are as follows: mask edge 220-1 is "TYPE1", mask edge 220-2 is "TYPE2", mask edge 220-3 is "TYPE3", and mask edge 220-4 is "TYPE4".

[0067] Returning to the explanation of Figure 7, in step S11, the evaluation value calculation unit 182 calculates evaluation values ​​corresponding to the central frame 0 and the four surrounding corner frames 1 to 4 shown in Figure 8. For these evaluation values, for example, feature quantities obtained from the endoscopic image 200, such as brightness values, can be used.

[0068] In step S11, for the sake of explanation, the evaluation values ​​for each of the multiple evaluation frames 210 are calculated together. However, it is also possible to calculate the evaluation values ​​for the target evaluation frame 210 sequentially for each of the judgment processes (S12, S14, S15, S16) described later. The same method for calculating these evaluation values ​​is used in the second and third processes described later.

[0069] In step S12, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation value for each frame, whether the evaluation value of at least one of the four corner frames (frames 1 to 4) is greater than the evaluation value of the central frame (frame 0).

[0070] In the judgment process of step S12, if the evaluation values ​​of the four corner frames (frames 1 to 4) are greater than the evaluation value of the central frame (frame 0), that is, if the central part of the endoscopic image 200 is dark and its periphery is bright, the process proceeds to step S13. In step S13, the judgment result setting unit 184 sets the mask type to "no rigid endoscope".

[0071] Furthermore, if the determination process in step S12 determines that the evaluation values ​​of the four corner frames 1 to 4 are smaller than the evaluation value of the central frame 0, the process proceeds to step S14. In step S14, the evaluation value determination unit 183 determines, based on the calculation result of the evaluation value of the target frame, whether the evaluation value of at least one of the four corner frames 1 to 4 exceeds a predetermined threshold.

[0072] In the determination process in step S14, if the evaluation values ​​of one to four frames in the four corners of the periphery exceed a predetermined threshold, that is, if it is determined that the entire image is bright, the process proceeds to step S13, and "no rigid mirror" is set as the mask type.

[0073] Furthermore, if the determination process in step S14 determines that the evaluation values ​​of the four corner frames 1 to 4 are below a predetermined threshold, the process proceeds to step S15. In step S15, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of the target frames, whether the evaluation value of the central frame 0 is below a predetermined threshold.

[0074] In the judgment process of step S15, if the evaluation value of the central 0 frame is determined to be below the threshold, that is, if the endoscopic image 200 is determined to be a completely black image, the process proceeds to step S13, and "No rigid endoscope" is set as the mask type.

[0075] Furthermore, if the determination process in step S15 determines that the evaluation value of the central 0 frame exceeds a predetermined threshold, the process proceeds to step S16. In step S16, the evaluation value determination unit 183 determines, based on the calculation result of the evaluation value of the target frame, whether the brightness difference between the 1 to 4 frames at the four corners of the periphery and the central 0 frame is less than or equal to a predetermined threshold.

[0076] If the determination process in step S16 determines that the brightness difference is below a predetermined threshold, the process proceeds to step S13, and "no rigid mirror" is set as the mask type.

[0077] If the determination process in steps S12, S14, S15, and S16 is determined to be positive ("Yes"), the mask type is set to "No rigid mirror" (S13), and the process proceeds to step S17. In step S17, the determination result setting unit 184 sets the recognition result to "Stopped". After that, the process returns to step S10, and the first process described above is repeated.

[0078] Furthermore, if the determination process in step S16 determines that the brightness difference exceeds a predetermined threshold, the process proceeds to step S18. In step S18, the determination result setting unit 184 sets the recognition result to "recognition successful". Subsequently, the processor 161 of the control device 51 executes the second process following the first process. Details of the second process will be described later with reference to Figure 9, etc.

[0079] <Second Processing Flow> Next, with reference to the flowchart in Figure 9, the second processing flow executed by the processor 161 of the control device 51 will be explained.

[0080] In step S30, the evaluation frame setting unit 181 sets multiple evaluation frames for the image corresponding to the image signal from the camera head 102. In this evaluation frame setting, for example, as shown in Figure 10, evaluation frames 210 are provided in the center of the endoscopic image 200 and in its horizontal direction (X direction).

[0081] Specifically, in the endoscopic image 200, a rectangular evaluation frame 210-4 (hereinafter abbreviated as evaluation frame 4) is provided in the central part, including its approximate centroid. Furthermore, to be approximately symmetrical with respect to the central part (including its approximate centroid) of the endoscopic image 200, rectangular evaluation frames 210-0 to 210-3 (hereinafter also abbreviated as evaluation frames 0, 1, 2, and 3) are provided at predetermined intervals on the left side in the horizontal direction, and rectangular evaluation frames 210-5 to 210-8 (hereinafter also abbreviated as evaluation frames 5, 6, 7, and 8) are provided at predetermined intervals on the right side in the horizontal direction.

[0082] The rectangular sizes of evaluation frames 210-0 to 210-3 and evaluation frames 210-5 to 210-8, which are discretely arranged at predetermined intervals on the left and right sides of the central part, are smaller than the rectangular size of evaluation frame 210-4, which is located in the central part. In this disclosure, "discrete" means that the multiple evaluation frames 210 are not arranged consecutively.

[0083] In Figure 10, evaluation frames 0 to 8 are discretely placed at predetermined intervals in the horizontal direction within the endoscopic image 200 in order to determine which of the mask edges 220-1 to 220-4 corresponds to the diameter of the scope 101 being used.

[0084] Specifically, for example, since the position where mask edge 220-1 is detected is known from the design, evaluation frame 0 and evaluation frame 1, and evaluation frame 7 and evaluation frame 8 are arranged so that the detection position of mask edge 220-1 is between evaluation frame 210. Similarly, the detection position of mask edge 220-2 is positioned between evaluation frame 1 and evaluation frame 2, and between evaluation frame 6 and evaluation frame 7, respectively; the detection position of mask edge 220-3 is positioned between evaluation frame 2 and evaluation frame 3, and between evaluation frame 5 and evaluation frame 6, respectively; and the detection position of mask edge 220-4 is positioned between evaluation frame 3 and evaluation frame 4, and between evaluation frame 4 and evaluation frame 5, respectively.

[0085] Returning to the explanation of Figure 9, in step S31, the evaluation value calculation unit 182 calculates evaluation values ​​corresponding to evaluation frames 0 to 8 shown in Figure 10. For example, feature quantities obtained from the endoscopic image 200, such as brightness values, can be used as these evaluation values.

[0086] In step S32, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 0, 1 and evaluation frames 7, 8, whether there is an edge between evaluation frame 0 and evaluation frame 1, and whether there is an edge between evaluation frame 7 and evaluation frame 8.

[0087] Here, for example, the difference between the luminance value obtained from evaluation frame 0 and the luminance value obtained from evaluation frame 1, and the difference between the luminance value obtained from evaluation frame 7 and the luminance value obtained from evaluation frame 8, are compared with a predetermined threshold, and by determining whether these luminance differences exceed the predetermined threshold, it is possible to determine whether there is an edge (mask edge 220-1) between those evaluation frames 210.

[0088] In other words, by determining whether the ratio of the average brightness value in evaluation frame 0 to the average brightness value in evaluation frame 1 exceeds a predetermined threshold, it is possible to determine whether there is an edge between evaluation frame 0 and evaluation frame 1. Similarly, by determining whether the ratio of the average brightness value in evaluation frame 7 to the average brightness value in evaluation frame 8 exceeds a predetermined threshold, it is possible to determine whether there is an edge between evaluation frame 7 and evaluation frame 8.

[0089] Alternatively, a threshold check may be performed for each integral value of the luminance values ​​within each evaluation frame. This integral value can be obtained by dividing the sum of the luminance values ​​of all pixels within each evaluation frame by the number of pixels. That is, by determining whether the integral value of the luminance values ​​in evaluation frame 0 exceeds the threshold (X) and whether the integral value of the luminance values ​​in evaluation frame 1 exceeds the threshold (X), it is possible to determine whether there is an edge between evaluation frame 0 and evaluation frame 1 based on these determination results. The same applies to evaluation frames 7 and 8. The threshold (X) is managed in a table, and for example, different values ​​are set for vignetting area 200A and observation area 200B.

[0090] For the sake of explanation, this example shows how to determine the presence or absence of an edge by thresholding the integral values ​​of the luminance values ​​in evaluation frame 0 and evaluation frame 1 and checking whether the result satisfies predetermined conditions. However, for example, the presence or absence of an edge could be determined by thresholding the integral values ​​of the luminance values ​​in evaluation frames 0 to 3 and checking whether the result satisfies predetermined conditions. In addition to the mean and integral values, statistical measures such as the maximum value, median, and variance can be used for the luminance values ​​within each evaluation frame.

[0091] If the determination process in step S32 is determined to be positive ("Yes"), the process proceeds to step S33. In step S33, the determination result setting unit 184 sets the mask type to "TYPE1".

[0092] Furthermore, if the determination process in step S32 is determined to be negative ("No"), the process proceeds to step S34. In step S34, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 1, 2 and evaluation frames 6, 7, whether there is an edge between evaluation frame 1 and evaluation frame 2, and whether there is an edge between evaluation frame 6 and evaluation frame 7. If the determination process in step S34 is determined to be positive, the process proceeds to step S35. In step S35, the determination result setting unit 184 sets the mask type to "TYPE2".

[0093] Furthermore, if the determination process in step S34 is determined to be negative, the process proceeds to step S36. In step S36, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 2, 3 and evaluation frames 5, 6, whether there is an edge between evaluation frame 2 and evaluation frame 3, and whether there is an edge between evaluation frame 5 and evaluation frame 6. If the determination process in step S36 is determined to be positive, the process proceeds to step S37. In step S37, the determination result setting unit 184 sets the mask type to "TYPE3".

[0094] Furthermore, if the determination process in step S36 is determined to be negative, the process proceeds to step S38. In step S38, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 3, 4 and evaluation frames 4, 5, whether there is an edge between evaluation frame 3 and evaluation frame 4, and whether there is an edge between evaluation frame 4 and evaluation frame 5. If the determination process in step S38 is determined to be positive, the process proceeds to step S39. In step S39, the determination result setting unit 184 sets the mask type to "TYPE4".

[0095] If the determination process in steps S32, S34, S36, and S38 determines that the result is positive, the mask types "TYPE1", "TYPE2", "TYPE3", and "TYPE4" are set respectively (S33, S35, S37, S39), and the process proceeds to step S40. In step S40, the determination result setting unit 184 sets the recognition result to "recognition successful". Subsequently, the processor 161 of the control device 51 executes the third process following the second process. Details of the third process will be described later with reference to Figure 11, etc.

[0096] Furthermore, if the determination process in step S38 is determined to be negative, the process proceeds to step S41. The determination result setting unit 184 then sets the mask type to "no rigid mirror" (S41) and the recognition result to "stopped" (S42). After that, the process returns to step S10 in Figure 7, and the first process described above is executed.

[0097] <Third Processing Flow> Next, with reference to the flowchart in Figure 11, we will explain the third processing flow executed by the processor 161 of the control device 51.

[0098] In step S50, the determination result setting unit 184 determines whether the mask type was set to "TYPE3" or "TYPE4" in the second process. In other words, in this example, the vertical detection positions of the mask edges 220-1 and 220-2, which are "TYPE1" and "TYPE2", are located outside the endoscopic image 200, so the processing related to mask edges 220-1 and 220-2 is excluded.

[0099] If the determination process in step S50 is determined to be positive, the process proceeds to step S51. In step S51, the evaluation frame setting unit 181 sets multiple evaluation frames for the image corresponding to the image signal from the camera head 102. In this evaluation frame setting, for example, as shown in Figure 12, evaluation frames 210 are provided in the center of the endoscopic image 200 and in its vertical direction (Y direction).

[0100] Specifically, in the endoscopic image 200, a rectangular evaluation frame 210-4 (hereinafter abbreviated as evaluation frame 4) is provided in the central part including its approximate centroid. Furthermore, to be approximately symmetrical with respect to the central part (including the approximate centroid) of the endoscopic image 200, rectangular evaluation frames 210-0 and 210-1 (hereinafter abbreviated as evaluation frames 0 and 1) are provided at predetermined intervals on the vertically upper side, and rectangular evaluation frames 210-2 and 210-3 (hereinafter abbreviated as evaluation frames 2 and 3) are provided at predetermined intervals on the vertically lower side.

[0101] The size of the rectangles of evaluation frames 210-0, 210-1, and evaluation frames 210-2, 210-3, which are discretely arranged at predetermined intervals above and below the central part, is smaller than the size of the rectangle of evaluation frame 210-4, which is located in the central part.

[0102] In Figure 12, evaluation frames 210-0 to 210-3 are discretely placed at predetermined intervals in the vertical direction within the endoscopic image 200 in order to determine which mask edge 220 (mask type) corresponds to the diameter of the scope 101 being used. Specifically, the detection position of mask edge 220-3 is positioned between evaluation frame 0 and evaluation frame 1, and between evaluation frame 2 and evaluation frame 3, respectively, and the detection position of mask edge 220-4 is positioned between evaluation frame 1 and evaluation frame 4, and between evaluation frame 2 and evaluation frame 4, respectively.

[0103] Returning to the explanation of Figure 11, in step S52, the evaluation value calculation unit 182 calculates evaluation values ​​corresponding to evaluation frames 0 to 4 shown in Figure 12. For example, feature quantities obtained from the endoscopic image 200, such as brightness values, can be used as these evaluation values.

[0104] In step S53, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 0, 1 and evaluation frames 2, 3, whether there is an edge between evaluation frame 0 and evaluation frame 1, and whether there is an edge between evaluation frame 2 and evaluation frame 3.

[0105] Here, for example, the difference between the luminance value obtained from evaluation frame 0 and the luminance value obtained from evaluation frame 1, and the difference between the luminance value obtained from evaluation frame 2 and the luminance value obtained from evaluation frame 3, are compared with a predetermined threshold, and by determining whether these luminance differences exceed the predetermined threshold, it is possible to determine whether there is an edge (mask edge 220-3) between those evaluation frames 210.

[0106] In other words, by determining whether the ratio of the average brightness value in evaluation frame 0 to the average brightness value in evaluation frame 1 exceeds a predetermined threshold, it is possible to determine whether there is an edge between evaluation frame 0 and evaluation frame 1. Similarly, by determining whether the ratio of the average brightness value in evaluation frame 2 to the average brightness value in evaluation frame 3 exceeds a predetermined threshold, it is possible to determine whether there is an edge between evaluation frame 2 and evaluation frame 3.

[0107] Alternatively, a threshold check may be performed for each integral value of the luminance value within each evaluation frame. That is, by determining whether the integral value of the luminance value in evaluation frame 0 exceeds the threshold (X) and whether the integral value of the luminance value in evaluation frame 1 also exceeds the threshold (X), it is possible to determine whether there is an edge between evaluation frame 0 and evaluation frame 1 based on these determination results. The same applies to evaluation frames 2 and 3. The threshold (X) is managed in a table.

[0108] If the determination process in step S53 is determined to be positive, the process proceeds to step S54. In step S54, the determination result setting unit 184 sets the mask type determined in the second process to "TYPE3".

[0109] Furthermore, if the determination process in step S53 is determined to be negative, the process proceeds to step S55. In step S55, the evaluation value determination unit 183 determines, based on the calculation results of the evaluation values ​​of evaluation frames 1, 4 and evaluation frames 2, 4, whether there is an edge between evaluation frame 1 and evaluation frame 4, and whether there is an edge between evaluation frame 2 and evaluation frame 4. If the determination process in step S55 is determined to be positive, the process proceeds to step S56. In step S56, the determination result setting unit 184 sets the mask type determined in the second process to "TYPE4".

[0110] When the processing in step S54 or S56 is completed, the process proceeds to step S57. In step S57, the determination result setting unit 184 sets the recognition result to "recognition successful". In step S58, the determination result setting unit 184 determines whether the mask type determined in the second process is "TYPE4" and whether the mask type determined in the third process is set to "TYPE3".

[0111] If the determination process in step S58 is determined to be positive, the process proceeds to step S59. In step S59, the determination result setting unit 184 sets the mask type to "TYPE4". In this case, assuming that a large vertical mask diameter was detected due to light leakage, for example, the mask diameter detected in the horizontal direction by the second process is adopted. By selecting and confirming a narrower mask diameter in this way, it is possible to more reliably ensure that the vignetting region 200A is not included when performing subsequent processing.

[0112] Furthermore, if the determination process in step S58 is determined to be negative, the process proceeds to step S60. In step S60, the evaluation value determination unit 183 determines whether the mask type determined in the second process is "TYPE3" and whether the mask type determined in the third process is set to "TYPE4".

[0113] If the determination process in step S60 is determined to be positive, the process proceeds to step S61. In step S61, the determination result setting unit 184 sets the mask type to "TYPE4". In this case, assuming that, for example, a large horizontal mask diameter was detected due to light leakage, the mask diameter detected in the vertical direction by the third process is adopted to more reliably prevent the vignetting region 200A from being included.

[0114] If the process in step S59 or S61 is completed, or if the determination process in step S60 is determined to be negative, the process proceeds to step S62. Furthermore, if the determination process in step S50 described above is determined to be negative, the process also proceeds to step S62.

[0115] In step S62, it is determined whether the operating mode is set to high-precision calculation mode. Here, the mode in which the first to third processes are executed is called the normal mode, and the mode in which the mask diameter size is calculated with higher precision than the normal mode is called the high-precision calculation mode. In the normal mode, the first to third processes are executed. In the high-precision calculation mode, in addition to the first to third processes, the high-precision calculation process (S63) is performed.

[0116] If it is determined in step S62 that the system is set to high-precision calculation mode, the process proceeds to step S63. In step S63, the processor 161 of the control device 51 performs high-precision calculation processing. Details of the high-precision calculation processing will be described later with reference to Figures 13 and 14.

[0117] If it is determined in step S62 that the system is not set to high-precision calculation mode, i.e., that it is set to normal mode, the process in step S63 is skipped. If the process in step S63 is executed when the system is in high-precision calculation mode, or if the process in step S63 is skipped when the system is in normal mode, the processor 161 of the control device 51 performs the confirmation process described later, referring to Figure 15, etc.

[0118] The flow of the first to third processes has been explained above. In the first process, as shown in Figure 8, the central frame 0 and the four corner frames 1 to 4 are set as evaluation frames 210 for the endoscopic image 200, and it is determined whether the endoscopic image 200 contains a vignetting area 200A (mask area) based on the relationship of evaluation values ​​for each of the evaluation frames 210. If the vignetting area 200A is not included, the first process is repeated, while if the vignetting area 200A is included, the second process is executed.

[0119] In the second process, as shown in Figure 10, evaluation frames 210-0 to 210-8 are discretely placed horizontally at predetermined intervals relative to the endoscopic image 200, and a mask type corresponding to the edge position is set based on the relationship of evaluation values ​​for each evaluation frame 210. If no edge corresponding to the evaluation frame 210 is detected, the first process is repeated, while if an edge corresponding to the evaluation frame 210 is detected, the third process is executed.

[0120] Furthermore, in the third process, as shown in Figure 12, evaluation frames 210-0 to 210-4 are discretely set vertically to the endoscopic image 200 at predetermined intervals, and a mask type corresponding to the edge position is set (re-set) based on the relationship of evaluation values ​​for each evaluation frame 210. If the normal mode is set as the operating mode, the confirmation process described later is executed after the normal mode processing is completed. Also, if the high-precision calculation mode is set, the confirmation process described later is executed after the high-precision calculation mode processing is completed.

[0121] In the above explanation, the second and third processes were described in which evaluation frames 210 are set in both the horizontal and vertical directions so that they are point-symmetric with respect to the approximate centroid of the endoscopic image 200, and evaluation is performed. However, it is also possible to set the evaluation frames 210 in only one of the horizontal or vertical directions by performing either the second or third process and perform evaluation. However, as described above, setting the evaluation frames 210 from both the horizontal and vertical directions by performing the second and third processes allows for more accurate setting of the mask type, for example, by anticipating light leakage.

[0122] Furthermore, while the second and third processes show cases where multiple evaluation frames 210 are discretely arranged at predetermined intervals, some of the evaluation frames 210 may be arranged consecutively. Also, the number of discretely arranged evaluation frames 210 is arbitrary; for example, more evaluation frames 210 may be provided for the detection positions of the mask edges 220. The positions where the multiple evaluation frames 210 are arranged are also arbitrary. Moreover, the shape of the discretely arranged evaluation frames 210 is not limited to rectangles; other shapes are also possible, and not all evaluation frames 210 need to have the same shape. The intervals between the multiple evaluation frames 210 do not have to be fixed intervals.

[0123] Furthermore, in the second and third processes, we showed examples of detecting edges (mask edges 220) by using luminance values ​​as evaluation values ​​and comparing the luminance difference with a predetermined threshold. However, for example, quantitative values ​​representing edges or black regions included within the evaluation frame 210 (e.g., feature quantities such as edge amount or black region amount) may be used as evaluation values ​​to detect edges.

[0124] <Details of high-precision calculation process> Next, referring to Figures 13 and 14, we will explain the details of the high-precision calculation process corresponding to step S63 in Figure 11. Here, we will illustrate the case where "TYPE3" is set as the mask type in the second and third processes described above.

[0125] First, in the high-precision calculation process, processing is performed to calculate detailed mask edges in the horizontal direction (X direction) according to the judgment results in the second and third processes described above. In this example, since "TYPE3" is set as the mask type, multiple evaluation frames 210 are set according to the detection position of the mask edge 220-3, as shown in Figure 13.

[0126] Specifically, evaluation frames 210-0 to 210-4 are continuously provided on the left side of the endoscopic image 200, corresponding to the detection position of the mask edge 220-3, so as to be approximately symmetrical (left-right symmetry with the Y-axis as the axis of symmetry) around the approximate centroid of the endoscopic image 200, and evaluation frames 210-5 to 210-9 are continuously provided on the right side of the endoscopic image 200, corresponding to the detection position of the mask edge 220-3.

[0127] The rectangular sizes of evaluation frames 210-0 to 210-4, which are arranged symmetrically and continuously, and the rectangular sizes of evaluation frames 210-5 to 210-9 are substantially the same in shape and substantially the same in size. Each evaluation frame 210 is provided with a start position and an end position in the horizontal direction (X direction). The start position indicates the position of the left end in the X direction of each evaluation frame 210, and the end position indicates the position of the right end in the X direction of each evaluation frame 210.

[0128] In this way, by arranging evaluation frames 210-0 to 210-4 and evaluation frames 210-5 to 210-9 symmetrically and continuously in the horizontal direction with respect to the endoscopic image 200, the left edge position and the right edge position of the mask diameter can be set based on the relationship of evaluation values ​​for each of the evaluation frames 210.

[0129] Next, in the high-precision calculation process, processing is performed to calculate detailed mask edges in the vertical direction (Y direction) according to the judgment results in the second and third processes described above. In this example, since "TYPE3" is set as the mask type, multiple evaluation frames 210 are provided according to the detection position of the mask edge 220-3, as shown in Figure 14.

[0130] Specifically, evaluation frames 210-0 to 210-4 are continuously provided on the upper vertical side of the endoscopic image 200, corresponding to the detection position of the mask edge 220-3, so as to be approximately symmetrical (up and down symmetrical with the X-axis as the axis of symmetry) around the approximate centroid of the endoscopic image 200, and evaluation frames 210-5 to 210-9 are continuously provided on the lower vertical side, corresponding to the detection position of the mask edge 220-3.

[0131] The rectangles of evaluation frames 210-0 to 210-4, which are arranged symmetrically vertically and continuously, and the rectangles of evaluation frames 210-5 to 210-9 are substantially the same shape and substantially the same size. Each evaluation frame 210 is provided with a start position and an end position in the vertical direction (Y direction). The start position indicates the position of the upper end in the Y direction of each evaluation frame 210, and the end position indicates the position of the lower end in the Y direction of each evaluation frame 210.

[0132] In this way, by continuously arranging evaluation frames 210-0 to 210-4 and evaluation frames 210-5 to 210-9 vertically and symmetrically with respect to the endoscopic image 200, the upper edge position and lower edge position in the mask diameter can be set based on the relationship between the evaluation values ​​for each of the evaluation frames 210.

[0133] As described above, when performing high-precision calculation processing, the mask type of the vignetting region 200A included in the endoscopic image 200 is set based on the relationship between evaluation values ​​corresponding to multiple evaluation frames 210, and the left edge position, right edge position, upper edge position, and lower edge position are set according to the mask diameter corresponding to the mask type.

[0134] <Process for final confirmation> Next, referring to the flowchart in Figure 15, the flow of the confirmation process performed by the processor 161 of the control device 51 will be explained. The confirmation process is performed after the normal mode processing or the high-precision calculation mode processing has been completed.

[0135] In step S111, the determination result determination unit 185 determines whether the recognition result has been "recognition successful" for N consecutive times, and whether the mask type has been the same for N consecutive times.

[0136] The number of consecutive occurrences N is a number such as 10, and by comparing it with a pre-set threshold, it is determined whether the judgment result (recognition result and mask diameter information) has been the same for a predetermined number of consecutive times. For example, when inserting the scope 101 into the body of patient 4, it is done via a trocker, but there is a risk of misjudgment due to the use of the trocker, such as light leakage. Here, the number of consecutive occurrences N, such as 10, is estimated based on the actual time the scope 101 is placed in the trocker, and the threshold is set to withstand disturbances.

[0137] For example, if the time interval between N repetitions of the normal mode processing (the interval between each repetition) is represented by image frames output in a time series, it corresponds to approximately 5 frames. However, by increasing this time interval to 10 frames or more, the number of consecutive repetitions N can be reduced, and by using 60 frames, disturbances can be avoided even with just two repetitions. Here, based on the results of detailed simulations by the inventors of this disclosure, it is considered probabilistically appropriate to set the time interval between processing to 5 frames and to determine whether the judgment result is the same for 10 consecutive times. Furthermore, since the actual time spent putting scope 101 into the trocker is assumed to be around 1 second, it is necessary to set thresholds according to the time interval and number of consecutive repetitions so that parameter change processing based on misjudgment results does not occur before or after this time.

[0138] The threshold used to determine the number of consecutive scans N may be determined not only according to disturbances caused by the use of the trocker, but also according to disturbances caused by the presence of high-luminance objects such as forceps 23 or gauze in the imaging area of ​​the image sensor 152. Furthermore, the threshold used to determine the number of consecutive scans N may be changed according to the surgical mode or observation mode. The surgical mode includes modes corresponding to surgery, such as laparoscopy mode, cystoscopy mode, and otolaryngology mode. For example, since the ear canal is narrow and prone to misjudgment, the threshold may be changed from 10 to 60 scans if it is determined to be otolaryngology mode.

[0139] The observation modes include modes such as normal light observation mode and special light observation mode. The normal light observation mode is a mode in which a white image captured when white light is irradiated from the light source device 53 is displayed on the display device 52. The special light observation mode is a mode in which a special light image captured when special light such as IR light or narrowband light is irradiated from the light source device 53 is displayed on the display device 52.

[0140] If the determination process in step S111 is positive ("Yes"), the process proceeds to step S112. In step S112, the notification target setting unit 186 sets the recognition result and mask diameter information determined in this process as notification targets to be notified to the external module.

[0141] On the other hand, if the determination process in step S111 is determined to be negative ("No"), the process proceeds to step S113. In step S113, the notification target setting unit 186 sets the recognition result and mask diameter information determined in the previous process as notification targets to be notified to the external module.

[0142] Figure 16 shows an example of recognition results and mask diameter information that are notified to an external module.

[0143] As shown in Figure 16, the recognition result is set to a value indicating "Stopped" or "Recognition Successful". The mask diameter information is set to a value indicating the mask type as "TYPE0", "TYPE1", "TYPE2", "TYPE3", "TYPE4". "TYPE0" represents none (no rigid endoscope). "TYPE1", "TYPE2", "TYPE3", "TYPE4" represent the size of the mask diameter, with the mask diameter decreasing in that order. In other words, the largest size is "TYPE1" (large), the next largest is "TYPE2" (medium), the next largest is "TYPE3" (small), and the smallest is "TYPE4" (very small).

[0144] For example, if the threshold used to determine the number of consecutive occurrences N is set to 10, and the normal mode processing is repeated 10 times at predetermined time intervals, and the recognition result obtained from the normal mode processing is "recognition successful" 10 times in a row, and the mask type obtained from the normal mode processing is "TYPE4" 10 times in a row, then the determination process in step S111 is determined to be positive ("Yes"). In this case (S111: "Yes"), the recognition result of "recognition successful" and the mask type of "TYPE4" are set as notification targets (S112).

[0145] Furthermore, if the normal mode processing is repeated 10 times at predetermined time intervals, and the recognition result obtained from the normal mode processing is not "recognition successful" for 10 consecutive times (for example, if it is "stopped" at least once), or if the mask type obtained from the normal mode processing is not "TYPE4" for 10 consecutive times (for example, if it is another mask type such as "TYPE3" at least once), the determination process in step S111 is determined to be negative ("No").

[0146] In this case (S111: "No"), the notification target will not be the recognition result and mask type obtained in the current process (the normal mode process that was the subject of the current judgment), but rather the recognition result and mask type determined in the previous process. In other words, if in the normal mode process that was the subject of the previous judgment the recognition result was "recognition successful" 10 times in a row, and the mask type was "TYPE4" 10 times in a row, the notification target will be the recognition result of "recognition successful" and the mask type of "TYPE4" (S113). Note that it is possible to set a default notification target at the start of processing, for example, the smallest size "TYPE4" (extremely small) can be set as the default mask type.

[0147] When the processing in step S112 or step S113 is completed, the process proceeds to step S114. In step S114, the set recognition result and mask diameter information are notified to the external module.

[0148] External modules include, for example, modules related to AE and AF. That is, when performing AE or AF, it is necessary to set the evaluation frame area within the observation area 200B, which is the effective area in the endoscopic image 200. By notifying the AE or AF module of the recognition result and mask diameter information, it becomes possible to set the evaluation frame within the observation area 200B. As a result, the AE or AF module can perform AE or AF control based on the pixel signals within the evaluation frame set within the observation area 200B.

[0149] In this way, in normal mode processing, mask diameter information, etc., is repeatedly judged at predetermined time intervals, and if the judgment result (mask diameter information, etc.) is the same for a predetermined number of consecutive times (for example, 10 times), a parameter change process (for example, a process to change the evaluation frame area of ​​AE or AF) based on the judgment result (mask diameter information, etc.) is executed. In other words, the parameters corresponding to the judgment result will only be switched if the judgment result is the same for a predetermined number of consecutive times. Note that if the same notification target is repeatedly set for the same external module, it is not necessary to notify the judgment result from the second time onward.

[0150] When the high-precision calculation mode is executed, in addition to the first to third processes, the high-precision calculation process (S63) is performed. As a result, the left edge position, right edge position, top edge position, and bottom edge position of the mask diameter are set along with the recognition result and mask diameter information. Therefore, the diameter of the mask and the center position of the mask can be determined using these edge positions.

[0151] For example, when the acceptable error range is narrow, it is necessary to determine the mask diameter and center position more accurately, in which case the high-precision calculation mode is set as the operating mode. In this case, the high-precision calculation mode process is executed repeatedly, and if, for example, at least one of the mask diameter and the mask center position is the same for a predetermined number of consecutive times (e.g., 10 times), a parameter change process based on the determination result (at least one of the mask diameter and the mask center position) can be executed. As a result, when operating in high-precision calculation mode, the mask diameter and the mask center position are always determined with high precision, allowing for a more accurate setting of the evaluation frame, and thus enabling appropriate AE and AF control.

[0152] Once the process in step S114 is completed, the series of processes ends, and the process returns to step S10 in Figure 7, and the process described above is repeated.

[0153] The confirmation process has been explained above. In this confirmation process, the size of the observation area, which is a region different from the vignetting area generated on the image sensor 152 by the scope 101, is repeatedly determined at predetermined time intervals. If the determination result (mask diameter information, etc.) obtained is the same for a predetermined number of consecutive times, a parameter change process based on the determination result is executed. In other words, the size of the observation area is determined at least twice, and the parameter change process is executed based on the comparison result obtained by comparing the two or more determination results. It can also be said that the type of scope 101 (mask type) is determined in order to determine the size of the observation area.

[0154] In this way, the observation area determination process (normal mode processing) corresponding to the type of scope 101 (mask type) is repeated at predetermined time intervals, and by determining whether the determination result has continued for a predetermined number of consecutive times, it is possible to suppress misdetermination when determining the observation area, even when there is the influence of external disturbances. For example, misdetermination when determining the observation area can be suppressed even when high-brightness objects such as forceps 23 or gauze are present in the imaging area, or when the scope 101 is inserted into the patient's body via a trocker.

[0155] <Variation> The mask determination process is intended to be performed when operating in normal light observation mode, but it may also be performed when operating in special light observation mode. Here, the process that includes at least the normal mode process that is repeatedly performed at predetermined time intervals is referred to as the mask determination process. The mask determination process may also include a confirmation process. Alternatively, when switching from normal light observation mode to special light observation mode, parameters based on the determination results (mask diameter information, etc.) obtained when operating in normal light observation mode can be used. That is, when switching to special light observation mode, it is preferable to stop the mask determination process and use the parameters corresponding to the determination results when operating in normal light observation mode immediately before switching to special light observation mode when operating in special light observation mode.

[0156] Images captured in special light observation mode differ from those captured in normal light observation mode, requiring special processing and increasing system complexity. On the other hand, it is very rare to change the scope 101 while observing in special light observation mode. Scope 101 is usually changed when changing the insertion point or the object being observed, and when switching to special light observation mode, it is often desirable to further process the area that was viewed in normal light observation mode. Therefore, by reusing the settings from normal light observation mode when switching to special light observation mode, it is possible to handle the situation without increasing system complexity.

[0157] Furthermore, when switching from special light observation mode to normal light observation mode and returning to normal light observation mode, the mask determination process should also be restarted. Switching between normal light observation mode and special light observation mode may be performed in response to user operations on the input device 54, or in response to operations on operating means provided by an operating room integrated system (not shown) connected to the endoscopic surgery system 1 via a network.

[0158] While it is reasonable to stop the mask detection process when operating in special light observation mode, as described above, if mask detection is performed, the number of times N may be set to a higher value than in normal light observation mode, for example to 60 times, or the evaluation frame area for AE or AF may be fixed to the center.

[0159] Furthermore, it is preferable to stop the mask determination process when alternating between irradiating with white light and IR light from the light source device 53. While it is possible to control AE using only the white image captured when irradiating with white light, or to extract the white image through image processing and perform mask determination, it is reasonable to stop the mask determination process because, as with the special light observation mode, it is rare to change the scope 101. When performing mask determination, weighting of each RGB signal included in the image signal output from the camera head 102 is also possible.

[0160] As shown in Figure 17, an icon 251 corresponding to the currently connected scope 101 may be displayed along with the endoscope image 200. In the example in Figure 17, the mask type "TYPE4" was set as the mask diameter information by the mask determination process, so the icon 251 indicating "TYPE4" is displayed. For example, the icon 251 can be displayed at predetermined timings, such as when the mask determination process is executed or when the notified mask diameter information changes (when the scope 101 is replaced). As shown in Figure 18, a virtual frame 252 indicating the effective area to be targeted for AE or AF may be displayed for the observation area 200B included in the endoscope image 200.

[0161] If it is detected that a high-brightness object such as forceps 23 or gauze has entered the imaging area of ​​the image sensor 152, the mask determination process can be stopped. The area of ​​forceps 23 or gauze may be identified in the endoscopic image 200 and excluded from the target area. The detection of forceps 23 or gauze may be determined when the determination result of the mask determination process changes significantly, or when the change occurs many times in a short period of time. Alternatively, the presence of forceps 23 or gauze may be determined when it is determined that the mask diameter information has changed without the scope 101 being removed.

[0162] Furthermore, in the case of gauze, it is expected that so-called overexposure will continue to occur when imaging is performed in close contact. As a way to deal with such phenomena, for example, by monitoring over time, the evaluation value of the area determined to be the vignetting region 200A can be measured, and if a change is detected in the outer area while the mask determination process is in progress, the threshold used in the mask determination process (for example, the threshold for the time interval or the number of consecutive occurrences) can be changed.

[0163] The embodiments described herein are not limited to those described above, and various modifications are possible without departing from the spirit of this disclosure. Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may exist.

[0164] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices. In addition, if a single step includes multiple processes, those processes can be performed by a single device, or they can be divided and performed by multiple devices.

[0165] In this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0166] Furthermore, this disclosure can take the following form.

[0167] (1) An imaging device is provided, which has an insertion section connected to it and receives light guided through the insertion section with an image sensor to capture an image. A control device for controlling the imaging device and Equipped with, The control device comprises one or more processors and one or more storage devices that store programs. The processor executes the program, The insertion portion determines the size of the observation area, which is a region different from the vignetting area that occurs on the image sensor, at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. Endoscopic system. (2) The aforementioned processor, The size of the observation area is repeatedly determined at predetermined time intervals. If the judgment result is the same for a predetermined number of consecutive times, the parameter change process is executed. The endoscopic system described in (1) above. (3) The aforementioned processor, If the judgment result is the same for a predetermined number of consecutive times, the parameters based on the judgment result obtained in the current judgment will be used. If the judgment result is not the same for a predetermined number of consecutive times, the parameters based on the judgment result obtained in the previous judgment will be used. The endoscopic system described in (2) above. (4) The parameter modification process includes a process to change the area of ​​the evaluation frame for AE (Auto Exposure) or AF (Auto Focus). The endoscopic system described in any of (1) to (3) above. (5) The processor determines whether the determination result is the same for a predetermined number of consecutive times by comparing the number of consecutive occurrences of the determination result with a predetermined threshold that is determined according to the disturbance. The endoscopic system described in (2) or (3) above. (6) The processor modifies the threshold depending on whether it is a surgical mode or an observation mode. The endoscopic system described in (5) above. (7) The processor, when operating in normal light observation mode or special light observation mode, repeatedly determines the size of the observation area at predetermined time intervals. The endoscopic system described in (2) or (3) above. (8) When the processor switches from the normal light observation mode to the special light observation mode, it ensures that parameters based on the determination results obtained during operation in the normal light observation mode are used. The endoscopic system described in (7) above. (9) The processor controls the display of information regarding the size of the insertion section, which corresponds to the size of the observation area, along with the endoscopic image based on the image signal output from the imaging device. The endoscopic system described in any of (1) to (8) above. (10) The processor controls the display of a virtual frame indicating the effective area to be targeted by AE or AF within the observation area included in the endoscopic image. The endoscopic system described in (9) above. (11) A control device that controls an imaging device, which has an insertion part connected to it and receives light guided through the insertion part with an image sensor to take an image, The insertion portion determines the size of the observation area, which is a region different from the vignetting area that occurs on the image sensor, at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. Control method. (12) Computers, The size of the observation area, which is a region different from the vignetting region that occurs on the image sensor by the insertion part connected to an imaging device that receives light guided through the insertion part and takes an image, is determined at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. A program that functions as a control device. [Explanation of Symbols]

[0168] 1 Endoscopic surgical system, 10 Endoscope, 20 Surgical instruments, 30 Support arm device, 51 Control device, 52 Display device, 53 Light source device, 54 Input device, 55 Treatment instrument control device, 56 Insufflation device, 57 Recorder, 58 Printer, 101 Scope, 102 Camera head, 151 Lens unit, 152 Image sensor, 153 Drive circuit, 154 Communication circuit, 155 Camera head control circuit, 161, 161-1 to 161-i Processor, 162, 162-1 to 162-j Storage device, 163 Communication circuit, 171 Control program, 181 Evaluation frame setting unit, 182 Evaluation value calculation unit, 183 Evaluation value determination unit, 184 Judgment result setting unit, 185 Judgment result determination unit, 186 Notification target settings section

Claims

1. An imaging device is provided, which has an insertion section connected to it and receives light guided through the insertion section with an image sensor to capture an image. A control device for controlling the imaging device and Equipped with, The control device comprises one or more processors and one or more storage devices that store programs. The processor executes the program, The insertion portion determines the size of the observation area, which is a region different from the vignetting area that occurs on the image sensor, at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. Endoscopic system.

2. The aforementioned processor, The size of the observation area is repeatedly determined at predetermined time intervals. If the judgment result is the same for a predetermined number of consecutive times, the parameter change process is executed. The endoscopic system according to claim 1.

3. The aforementioned processor, If the judgment result is the same for a predetermined number of consecutive times, the parameters based on the judgment result obtained in the current judgment will be used. If the judgment result is not the same for a predetermined number of consecutive times, the parameters based on the judgment result obtained in the previous judgment will be used. The endoscopic system according to claim 2.

4. The parameter modification process includes a process to change the area of ​​the evaluation frame for AE (Auto Exposure) or AF (Auto Focus). The endoscopic system according to claim 1.

5. The processor determines whether the determination result is the same for a predetermined number of consecutive times by comparing the number of consecutive occurrences of the determination result with a predetermined threshold that is determined according to the disturbance. The endoscopic system according to claim 2.

6. The processor modifies the threshold depending on whether it is a surgical mode or an observation mode. The endoscopic system according to claim 5.

7. The processor, when operating in normal light observation mode or special light observation mode, repeatedly determines the size of the observation area at predetermined time intervals. The endoscopic system according to claim 2.

8. When the processor switches from the normal light observation mode to the special light observation mode, it ensures that parameters based on the determination results obtained during operation in the normal light observation mode are used. The endoscopic system according to claim 7.

9. The processor controls the display of information regarding the size of the insertion section, which corresponds to the size of the observation area, along with the endoscopic image based on the image signal output from the imaging device. The endoscopic system according to claim 1.

10. The processor controls the display of a virtual frame indicating the effective area to be targeted by AE or AF within the observation area included in the endoscopic image. The endoscopic system according to claim 9.

11. A control device that controls an imaging device, which has an insertion part connected to it and receives light guided through the insertion part with an image sensor to take an image, The insertion portion determines the size of the observation area, which is a region different from the vignetting area that occurs on the image sensor, at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. Control method.

12. Computers, The size of the observation area, which is a region different from the vignetting region that occurs on the image sensor by the insertion part connected to an imaging device that receives light guided through the insertion part and takes an image, is determined at least twice. Based on the comparison results obtained by comparing two or more judgment results, the parameter modification process based on the judgment results is executed. A program that functions as a control device.

Images