Image processing method, image processing device, and program

Abstract

To provide an image processing method, an image processing device, and a program which can reduce the load of a process of determining whether or not an image region is suitable for prescribed processing.SOLUTION: The image processing method includes a setting step, a determination step, and an information display step. A setting unit sets, in the setting step, a first region to a first image and a second region to a second image. A determination unit determines, in the determination step, whether or not each of the first and second regions is suitable for prescribed processing. A display control unit displays on a display, in the information display step, information that indicates whether or not at least a portion of the first region is suitable for the prescribed processing and information that indicates whether or not the second region is suitable for the prescribed processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing method, an image processing device, and a program. [Background technology]

[0002] Industrial endoscope systems are used to inspect for abnormalities (such as scratches and corrosion) inside industrial equipment such as boilers, turbines, engines, and pipes. These endoscope systems are equipped with multiple types of optical adapters to inspect various objects under appropriate observation conditions. Optical adapters are attached to the tip of the endoscope and are interchangeable. Endoscope systems with three-dimensional (3D) measurement capabilities have also been developed. These endoscope systems achieve 3D measurement by using a stereo optical adapter. The stereo optical adapter incorporates a stereo optical system that includes two optical systems with parallax.

[0003] To achieve 3D measurement, a method using means other than a stereo optical system has been proposed. For example, a method using light with a spatial structure (pattern) has been proposed. In this method, an endoscope device irradiates a subject with light and receives the light reflected by the subject. The endoscope device generates an image based on the received light. The endoscope device performs 3D measurement by analyzing the image.

[0004] An endoscopic device with 3D measurement capabilities measures the length or area of an object based on multiple measurement points specified by the user on an image. By referring to the measurement results, the user can quantitatively determine the severity of any abnormalities. This also allows the user to take precise measures to prevent the occurrence of abnormalities.

[0005] Users mainly use the 3D measurement function in the following first or second scenarios. In the first scenario, the user uses the 3D measurement function at the inspection site. In the second scenario, the user records inspection images (still images or videos) at the inspection site. In the second scenario, the user uses the 3D measurement function after returning to the office or relay point (such as a dedicated vehicle).

[0006] If the imaging conditions for acquiring the images (still images or videos) required for 3D measurement are not appropriate, the endoscope device will not be able to obtain reliable measurement results. These imaging conditions include composition and brightness.

[0007] In the first scenario, the user may perform imaging using imaging conditions that are not suitable for measurement. Even in this case, the user can change the imaging conditions and perform imaging again using the changed imaging conditions.

[0008] On the other hand, in the second scenario described above, if it is necessary to recapture an image, the user must return to the examination site. Therefore, the time required to redo the work will be longer than in the first scenario. To avoid redoing the work, it is desirable for the user to be able to check during the examination whether the imaging conditions have become suitable for measurement. For example, the endoscope system may determine whether the image is suitable for measurement and notify the user of the determination result. This allows the endoscope system to reduce the risk of the user having to return to the examination site to recapture an image for measurement.

[0009] Patent Document 1 discloses a measurement system that performs 3D measurement by using measurement light. The measurement system uses the blur or light intensity of the light pattern reflected by the subject as an index. Based on the index, the measurement system calculates the reliability of each region of the image used for the 3D measurement and notifies the user of the reliability.

[0010] Similar to Patent Document 1, Patent Document 2 discloses a measurement device that performs 3D measurement by using measurement light. The measurement device generates information about noise in the 3D measurement. The measurement device uses the information to improve the accuracy of determining the reliability of the measurement. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Japanese Patent Application Publication No. 2019-219208 [Patent Document 2] Patent No. 6238521 Summary of the Invention [Problem to be solved by the invention]

[0012] However, when the methods disclosed in Patent Document 1 or Patent Document 2 are applied to an industrial endoscope device, the following problem occurs: While the calculation amount for the process to calculate the reliability is large, the calculation resources of the industrial endoscope device are limited, which may increase the amount of delay in the process in the endoscope device.

[0013] An object of the present invention is to provide an image processing method, an image processing device, and a program that can reduce the processing load of determining whether an area of an image is suitable for a predetermined process. [Means for solving the problem]

[0014] The present invention provides 1. An image processing method for repeatedly determining whether an image is suitable for a predetermined process, comprising: a first setting step in which a setting unit sets a first region in a first image; a first determination step in which a determination unit determines whether the first region is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of a subject; and a second determination step in which the setting unit sets a second region in a second image acquired after the first image, the second region being smaller than the entire region of the second image. and does not include at least a portion of the first region.This is an image processing method having a second setting step, a second judgment step in which the judgment unit judges whether the second area is suitable for the specified processing, and an information display step in which a display control unit displays on a display information indicating whether at least a portion of the first area is suitable for the specified processing and information indicating whether the second area is suitable for the specified processing.

[0015] The image processing method of the present invention further includes a second information display step in which, before the information display step is executed, the display control unit displays on the display information indicating whether the first area is suitable for the specified processing.

[0016] In the image processing method of the present invention, in the first setting step, the setting unit sets the first area including an area in which a first subject is captured in the first image, and in the second setting step, the setting unit sets the second area including an area in which a second subject different from the first subject is captured in the second image.

[0017] The image processing method of the present invention comprises: In the second image A search unit searches for an object area in which the first object is captured. do The method further includes a searching step, wherein in the second setting step, the setting unit sets the second region including a region different from the subject region in the second image.

[0018] In the image processing method of the present invention, in the first setting step, the setting unit sets the first area including an area in which a first subject is captured in the first image, and in the second setting step, the setting unit sets the second area including at least a portion of the area in which the first subject is captured in the second image.

[0019] The image processing method of the present invention further includes a detection step in which a detection unit detects an amount of change in composition between the first image and the second image before the second determination step is executed.

[0020] In the image processing method of the present invention, the information display step is executed only when the amount of change is smaller than a predetermined amount.

[0021] The image processing method of the present invention further includes a generation step in which a generation unit generates information on a judgment area in each of the first image and the second image, and after the information on the judgment area is generated, the setting unit sets the first area as the judgment area in the first image in the first setting step, and the setting unit sets the second area as the judgment area in the second image in the second setting step.

[0022] In the image processing method of the present invention, each of the first image and the second image is generated based on an optical image generated through a stereo optical system.

[0023] In the image processing method of the present invention, in the first judgment step, the judgment unit uses at least one of the three-dimensional shape obtained by executing a process to calculate the three-dimensional shape of the subject in the first region, the distance between the subject in the first region and the camera, the brightness of the first region, the amount of blur in the first region, and the intensity of the texture in the first region.

[0024] In the image processing method of the present invention, in the second judgment step, the judgment unit uses at least one of the three-dimensional shape obtained by executing a process to calculate the three-dimensional shape of the subject in the second region, the distance between the subject in the second region and the camera, the brightness of the second region, the amount of blur in the second region, and the intensity of the texture in the second region.

[0025] In the image processing method of the present invention, the first image is generated based on an optical image generated through a stereo optical system, and the judgment unit uses at least one of a correlation value obtained by performing stereo matching processing on the first region and a parallax amount in the first region in the first judgment step.

[0026] In the image processing method of the present invention, the second image is generated based on an optical image generated through a stereo optical system, and in the second judgment step, the judgment unit uses at least one of a correlation value obtained by performing stereo matching processing on the second region and a parallax amount in the second region.

[0027] The image processing method of the present invention further includes a calculation step in which the processing unit calculates the three-dimensional shape of the subject by using two or more images including at least one of the first image and the second image, and an image display step in which the display control unit displays an image of the three-dimensional shape on the display.

[0028] In the image processing method of the present invention, the determination section uses two or more images including the first image in the first determination step.

[0029] In the image processing method of the present invention, the determination section uses two or more images including the second image in the second determination step.

[0030] The image processing method of the present invention further includes a first association step in which the processing unit associates first execution information indicating that the first judgment step has been executed with the first image, and a second association step in which the processing unit associates second execution information indicating that the second judgment step has been executed with the second image.

[0031] In the image processing method of the present invention, each of the first image and the second image is one of two or more frames that make up a moving image, and the processing unit adds the first execution information to the moving image in the first association step, and the processing unit adds the second execution information to the moving image in the second association step.

[0032] The image processing method of the present invention further includes a third association step in which a processing unit selects either the first image or the second image based on the results of the first judgment step and the second judgment step, and associates information identifying the selected image with the selected image.

[0033] The image processing method of the present invention further includes a fourth associating step in which the processing unit associates information for associating the first image and the second image with at least one of the first image and the second image.

[0034] In the image processing method of the present invention, each of the first image and the second image is one of two or more frames that make up a moving image, and in the fourth association step, the processing unit adds the information to the moving image for associating the first image and the second image with each other. In the image processing method of the present invention, the first region is smaller than the entire region of the first image.

[0035] The present invention provides a method for setting a first area in a first image and a second area in a second image acquired after the first image, the second area being smaller than the entire area of the second image. and does not include at least a portion of the first region. The image processing device has a setting unit, a judgment unit that judges whether each of the first area and the second area is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of a subject, and a display control unit that displays on a display information indicating whether at least a portion of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process.

[0036] The present invention provides A program for causing a computer to execute image processing for repeatedly determining whether an image is suitable for a predetermined process, a first setting step of setting a first region in a first image; a first determination step of determining whether the first region is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of a subject; and a second setting step of setting a second region in a second image acquired after the first image, the second region being smaller than the entire region of the second image. and does not include at least a portion of the first region. This is a program for causing a computer to execute a second setting step, a second judgment step of judging whether the second area is suitable for the specified processing, and an information display step of displaying on a display information indicating whether at least a portion of the first area is suitable for the specified processing and information indicating whether the second area is suitable for the specified processing. [Effects of the Invention]

[0037] According to the present invention, the image processing method, image processing device, and program can reduce the processing load of determining whether or not an area of an image is suitable for a predetermined process. [Brief explanation of the drawings]

[0038] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment of the present invention. [Figure 2] 4 is a flowchart showing the procedure of image processing in the first embodiment of the present invention. [Figure 3] FIG. 10 is a perspective view showing the overall configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram showing the internal configuration of an endoscope apparatus according to a second embodiment of the present invention. [Figure 5] FIG. 10 is a perspective view of the distal end of the insertion section and a stereo optical adapter in an endoscope apparatus according to a second embodiment of the present invention. [Figure 6] 10 is a cross-sectional view of the distal end of an insertion section and a stereo optical adapter in an endoscope apparatus according to a second embodiment of the present invention. FIG. [Figure 7]FIG. 10 is a diagram illustrating a method for calculating three-dimensional coordinates of a measurement point in the second embodiment of the present invention. [Figure 8] FIG. 10 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing the procedure of image processing in a second embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of an image in the second embodiment of the present invention. [Figure 11] 10 is a flowchart showing the procedure of a first determination process in a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing an example of an image in the second embodiment of the present invention. [Figure 13] FIG. 10 is a diagram showing an example of an image in the second embodiment of the present invention. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a third embodiment of the present invention. [Figure 15] 10 is a flowchart showing the procedure of image processing in a third embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of an image in the third embodiment of the present invention. [Figure 17] FIG. 10 is a diagram showing an example of an image in the third embodiment of the present invention. [Figure 18] FIG. 10 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a fourth embodiment of the present invention. [Figure 19] 10 is a flowchart showing the procedure of image processing in a fourth embodiment of the present invention. [Figure 20] 10 is a flowchart showing the procedure of image processing in a fifth embodiment of the present invention. [Figure 21] 13 is a flowchart showing the procedure of area division processing in the fifth embodiment of the present invention. [Figure 22] FIG. 13 is a diagram showing an example of an image in the fifth embodiment of the present invention. [Figure 23] FIG. 13 is a diagram showing an example of an image in the fifth embodiment of the present invention. [Figure 24] FIG. 13 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a sixth embodiment of the present invention. [Figure 25] 13 is a flowchart showing the procedure of image processing in a sixth embodiment of the present invention. [Figure 26] 13 is a flowchart showing the procedure of image processing in a sixth embodiment of the present invention. [Figure 27] FIG. 13 is a diagram showing a moving image in the sixth embodiment of the present invention. [Figure 28] FIG. 13 is a block diagram showing the configuration of a personal computer (PC) according to a modified example of the sixth embodiment of the present invention. [Figure 29] 13 is a flowchart showing the procedure of a measurement process in a modified example of the sixth embodiment of the present invention. [Figure 30] FIG. 13 is a diagram showing an example of an image in a modified example of the sixth embodiment of the present invention. [Figure 31] FIG. 13 is a block diagram showing the functional configuration of a CPU included in an endoscope apparatus according to a seventh embodiment of the present invention. [Figure 32] 13 is a flowchart showing the procedure of image processing in a seventh embodiment of the present invention. [Figure 33] FIG. 13 is a diagram showing an example of an image in the seventh embodiment of the present invention. [Figure 34] FIG. 13 is a diagram showing an example of an image in the seventh embodiment of the present invention. [Figure 35] FIG. 13 is a diagram showing an example of an image in the seventh embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0039] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0040] (First embodiment) 1 shows the configuration of an image processing device 7 according to a first embodiment of the present invention. The image processing device 7 shown in FIG.

[0041] The setting unit 70 sets the first area in the first image (first setting step). The determination unit 71 determines whether the first area is suitable for a predetermined process (first determination step). The predetermined process is a process of calculating three-dimensional coordinates (3D coordinates) of the subject. The setting unit 70 sets the second area in a second image acquired after the first image (second setting step). The second area is smaller than the entire area of the second image. The determination unit 71 determines whether the second area is suitable for the predetermined process (second determination step). The display control unit 72 displays information indicating whether at least a part of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process on the display (information display step).

[0042] Each unit shown in FIG. 1 may be configured with at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). Each unit shown in FIG. 1 may include one or more processors. Each unit shown in FIG. 1 may include one or more logic circuits.

[0043] The computer of the image processing device 7 may load a program and execute the loaded program. The program includes instructions that define the operations of the setting unit 70, the determination unit 71, and the display control unit 72. In other words, the functions of the setting unit 70, the determination unit 71, and the display control unit 72 may be realized by software.

[0044] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the image processing device 7 via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with a difference program.

[0045] Image processing in the first embodiment will be described with reference to Fig. 2. Fig. 2 shows the procedure of image processing.

[0046] The first image and the second image are used for image processing. An image acquisition device such as a camera may acquire the first image and the second image, and the image processing device 7 may acquire the first image and the second image from the image acquisition device. The image processing device 7 may have a recording medium for storing the first image and the second image. Alternatively, the recording medium may be connected to the image processing device 7.

[0047] The setting unit 70 sets a first region in a first image (step S1). Step S1 corresponds to a first setting step.

[0048] The angle of view of the first region does not have to be smaller than the angle of view of the first image. That is, the first region does not have to be a part of the entire area of the first image. The angle of view of the first region may be the same as the angle of view of the first image. That is, the first region may be the same as the entire area of the first image.

[0049] After step S1, the determination unit 71 determines whether the first area set in the first image is suitable for a predetermined process (step S2). Step S2 corresponds to the first determination step.

[0050] After step S2, the setting unit 70 sets the second region in the second image (step S3). Step S3 corresponds to a second setting step.

[0051] The angle of view of the second region is smaller than the angle of view of the second image, i.e., the second region is smaller than the entire area of the second image, and is a portion of the entire area of the second image.

[0052] After step S3, the determination unit 71 determines whether the second area set in the second image is suitable for predetermined processing (step S4). Step S4 corresponds to the second determination step.

[0053] For example, the predetermined processing is a processing (measurement processing) that calculates the 3D coordinates of the subject and measures the size (dimensions) of the subject. Alternatively, the predetermined processing is a processing (3D display processing) that calculates the 3D coordinates of the subject and displays an image. The image visualizes the three-dimensional shape (3D shape) of the subject.

[0054] When the 3D shape of the object is displayed as an image, the user can determine the condition of the object by checking the 3D shape. For example, the user can determine whether an abnormality such as a scratch exists on the object. A measurement process and a 3D display process may be performed. Before the measurement process is performed, the user can check whether the measurement position is correct based on the 3D shape of the object.

[0055] The predetermined process in step S4 is the same as the predetermined process in step S2. If the predetermined process in step S2 is measurement processing, the predetermined process in step S4 is measurement processing. If the predetermined process in step S2 is 3D display processing, the predetermined process in step S4 is 3D display processing.

[0056] After step S4, the display control unit 72 displays information indicating whether at least a part of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process on the display (step S5). Step S5 corresponds to an information display step.

[0057] The display control unit 72 displays the result of the processing in step S2 and the result of the processing in step S4 during a display frame period on the display. For example, the display control unit 72 displays the result of the processing in step S2 and the result of the processing in step S4 simultaneously on the display.

[0058] Before step S4 is executed, the display control unit 72 does not need to display information indicating whether the first region is suitable for the predetermined process on the display.

[0059] The first image and the second image may be included in two or more frames that make up a video. The first image and the second image do not have to be two consecutive frames. One or more frames may exist between the first image and the second image.

[0060] Since the second region is a part of the entire region of the second image, the processing load in step S4 is reduced, and therefore the image processing device 7 can reduce the processing load of determining whether the region of the second image is suitable for a predetermined process.

[0061] (Second embodiment) A second embodiment of the present invention will be described below, in which an example in which the image processing device is an endoscope device will be described.

[0062] The configuration of the endoscope device 1 in the second embodiment will be described with reference to Figures 3 and 4. Figure 3 shows the external appearance of the endoscope device 1. Figure 4 shows the internal configuration of the endoscope device 1.

[0063] The endoscopic device 1 shown in FIG. 3 has an insertion section 2, a main body section 3, an operation section 4, and a display section 5. The endoscopic device 1 captures an image of a subject and generates an image. The subject is an industrial product. The endoscopic device 1 measures the geometric characteristics of the subject by using the generated image. To observe and measure various subjects, the user can replace the optical adapter attached to the tip of the insertion section 2, select a built-in measurement processing program, and add a measurement processing program. The following describes a case where the endoscopic device 1 performs stereo measurement as an example of measurement.

[0064] The insertion section 2 is inserted into the inside of the subject. The insertion section 2 is a long, thin tube that is bendable from the tip 20 to the base end. The insertion section 2 captures an image of the subject and outputs an image signal to the main body 3. An optical adapter is attached to the tip 20 of the insertion section 2. For example, a stereo optical adapter is attached to the tip 20 of the insertion section 2. A monocular optical adapter may also be attached to the tip 20. The main body 3 is a control device that includes a storage section for storing the insertion section 2. The operation section 4 accepts user operations on the endoscope device 1. The display section 5 has a display screen, and displays images of the subject acquired by the insertion section 2, operation menus, etc. on the display screen.

[0065] The operation unit 4 is a user interface. For example, the operation unit 4 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The operation unit 4 accepts user operations on the endoscope device 1. The user can input various information to the endoscope device 1 by operating the operation unit 4.

[0066] The display unit 5 is a monitor (display) such as an LCD (Liquid Crystal Display). The display unit 5 may be a touch panel. In this case, the operation unit 4 and the display unit 5 are integrated.

[0067] The main body 3 shown in FIG. 4 includes an endoscope unit 8, a CCU (Camera Control Unit) 9, and a control device 10.

[0068] The endoscope unit 8 has a light source device and a bending device, not shown. The light source device supplies illumination light necessary for observation to the tip 20. The bending device bends the bending mechanism built into the insertion portion 2.

[0069] The imaging element 28 is built into the tip 20 of the insertion portion 2. The imaging element 28 is an image sensor. The imaging element 28 photoelectrically converts an optical image of a subject formed by the optical adapter, and generates an imaging signal.

[0070] The CCU 9 drives the imaging element 28. An imaging signal output from the imaging element 28 is input to the CCU 9. The CCU 9 performs preprocessing, including amplification and noise removal, on the imaging signal acquired by the imaging element 28. The CCU 9 converts the preprocessed imaging signal into a video signal such as an NTSC signal.

[0071] The control device 10 includes a video signal processing circuit 12, a ROM (Read Only Memory) 13, a RAM (Random Access Memory) 14, a card interface 15, an external device interface 16, a control interface 17, and a CPU (Central Processing Unit) 18.

[0072] The video signal processing circuit 12 performs predetermined video processing on the video signal output from the CCU 9. For example, the video signal processing circuit 12 performs video processing related to improving visibility. For example, the video processing includes color reproduction, gradation correction, noise suppression, and edge enhancement. For example, the video signal processing circuit 12 combines the video signal output from the CCU 9 with a graphic image signal generated by the CPU 18. The graphic image signal includes an image of an operation screen, etc. The video signal processing circuit 12 outputs the combined video signal to the display unit 5.

[0073] Furthermore, the video signal processing circuit 12 outputs the video signal output from the CCU 9 to the CPU 18 as image data. The image data constitutes an image of the subject. When measurement is performed, a stereo optical adapter is attached to the tip 20 of the insertion portion 2. Therefore, multiple optical images of the subject (subject images) are included in the image. The multiple subject images have different parallaxes from each other.

[0074] The stereo optical adapter has two optical systems (stereo optical systems). In the embodiment of the present invention, an example will be described in which the endoscope device 1 simultaneously acquires two subject images formed by the two optical systems. The method of acquiring the subject images is not limited to this. For example, the endoscope device 1 may acquire multiple subject images with different parallaxes in a time-division manner. For example, one of the two optical paths of the stereo optical system is blocked by a shutter, and the image sensor 28 acquires the subject image formed by light that has passed through the other optical path. The acquired subject image can be changed by changing the position blocked by the shutter.

[0075] The ROM 13 is a non-volatile recording medium that stores a program for the CPU 18 to control the operation of the endoscope device 1. The RAM 14 is a volatile recording medium that temporarily stores information used by the CPU 18 to control the endoscope device 1. The CPU 18 controls the operation of the endoscope device 1 based on the program stored in the ROM 13.

[0076] A memory card 42 is connected to the card interface 15. The memory card 42 is a recording medium that is detachable from the endoscope device 1. The card interface 15 imports the control processing information, image information, etc. stored in the memory card 42 into the control device 10. The card interface 15 also records the control processing information, image information, etc. generated by the endoscope device 1 onto the memory card 42.

[0077] An external device such as a USB device is connected to the external device interface 16. For example, a personal computer (PC) 41 is connected to the external device interface 16. The external device interface 16 transmits information to the PC 41 and receives information from the PC 41. This allows the PC 41 to display information. Furthermore, a user can perform operations related to the control of the endoscope device 1 by inputting instructions to the PC 41.

[0078] The control interface 17 communicates with the operation unit 4, the endoscope unit 8, and the CCU 9 for operational control. The control interface 17 notifies the CPU 18 of information input by the user to the operation unit 4. The control interface 17 outputs control signals to the endoscope unit 8 for controlling the light source device and the bending device. The control interface 17 outputs control signals to the CCU 9 for controlling the image sensor 28.

[0079] The program executed by the CPU 18 may be recorded on a computer-readable recording medium. The program recorded on this recording medium may be read and executed by a computer other than the endoscope device 1. For example, the program may be read and executed by the PC 41. The PC 41 may control the endoscope device 1 by transmitting control information for controlling the endoscope device 1 to the endoscope device 1 in accordance with the program. Alternatively, the PC 41 may acquire a video signal from the endoscope device 1 and process the acquired video signal.

[0080] As described above, the endoscope device 1 has the image sensor 28 and the CPU 18. The image sensor 28 captures an image of a subject and generates an image signal. The image signal includes an image of the subject. Therefore, the image sensor 28 captures an image of the subject and generates the image. The image is a two-dimensional image (2D image). The image captured by the image sensor 28 is input to the CPU 18 via the video signal processing circuit 12. The camera in the second embodiment includes the image sensor 28 and an observation optical system.

[0081] 5 and 6 show the configuration of the tip 20 of the insertion section 2 and the stereo optical adapter 30. FIG. 5 shows the appearance of the tip 20 of the insertion section 2 and the stereo optical adapter 30. FIG. 6 shows a cross section of the tip 20 of the insertion section 2 and the stereo optical adapter 30. A first illumination optical system 51, a second illumination optical system 52, a first objective optical system 53, and a second objective optical system 54 are arranged at the tip of the stereo optical adapter 30. FIG. 6 shows a cross section passing through the first objective optical system 53 and the second objective optical system 54.

[0082] The stereo optical adapter 30 is attached to the tip 20 of the insertion section 2. The stereo optical adapter 30 has a fixing ring 50 on which a female thread 50a is formed. A male thread 20a is formed on the tip 20 of the insertion section 2. The stereo optical adapter 30 is fixed to the tip 20 by being threadedly engaged with the male thread 20a via the female thread 50a.

[0083] The imaging element 28 is disposed within the distal end 20. The first objective optical system 53 and the second objective optical system 54 form two optical images on the imaging element 28. The imaging element 28 converts the two optical images into imaging signals. A signal line 2b is connected to the imaging element 28. The imaging signals are supplied to the CCU 9 via the signal line 2b and the endoscope unit 8. The CCU 9 converts the imaging signals into video signals and supplies the video signals to the video signal processing circuit 12.

[0084] A method for calculating three-dimensional coordinates (3D coordinates) of measurement points in stereo measurement will be described with reference to Fig. 7. The midpoint of the line segment connecting the left optical center (first optical center 63) and the right optical center (second optical center 64) is defined as the origin O. Also, the x-axis, y-axis, and z-axis shown in Fig. 7 are defined.

[0085] An image containing an object image is used. The object image is obtained via the left optical system and the right optical system. As shown in the following equations (1) to (3), the 3D coordinates (X, Y, Z) of the measurement target point 60 are calculated by using the principle of triangulation. The two-dimensional coordinates (2D coordinates) of the measurement point 61 and the corresponding point 62 are respectively (X L ,Y L ), (X R ,Y R ) The measurement point 61 is on the left image plane after distortion correction. The corresponding point 62 is on the right image plane after distortion correction.

[0086] The origin of measurement point 61 is intersection point O L The origin of the corresponding point 62 is the intersection point O R Intersection point O L is the point where the optical axis of the left optical system intersects with the image plane. R is located at the intersection of the optical axis of the right optical system and the image plane. The distance between the first optical center 63 and the second optical center 64 is D. The parameter F indicates the focal length. The parameter t is calculated by the function D / (X R -X L ) X=t×X R +D / 2 (1) Y=-t×Y R (2) Z=t×F (3)

[0087] When the coordinates of the measurement point 61 and the corresponding point 62 are determined as described above, the CPU 18 can calculate the 3D coordinates of the measurement target point 60 by using the parameters D and F. By calculating the 3D coordinates of two or more points, the CPU 18 can realize various measurement functions. For example, the CPU 18 can measure the distance between two points, the distance between a line and a point, the area of an area, the depth of a reference plane, etc. The line connects the two points. The area is surrounded by lines connecting multiple points.

[0088] The user can select a desired measurement function from a variety of measurement functions. The CPU 18 can also calculate the distance from the first optical center 63 or the second optical center 64 to the subject (object distance). Optical data is required to perform the above stereo measurement. The optical data indicates the characteristics of the optical system including the tip 20 of the insertion portion 2 and the stereo optical adapter 30.

[0089] 8 shows the functional configuration of CPU 18. The functions of CPU 18 include control unit 180, image acquisition unit 181, area setting unit 182, determination unit 183, display control unit 184, and information reception unit 185. At least one of the blocks shown in FIG. 8 may be configured by a circuit different from CPU 18.

[0090] Each unit shown in Fig. 8 may be configured with at least one of a processor and a logic circuit. Each unit shown in Fig. 8 may include one or more processors. Each unit shown in Fig. 8 may include one or more logic circuits.

[0091] The control unit 180 controls the processes executed by the units shown in Fig. 8. The image acquisition unit 181 acquires an image (image data) of a subject from the video signal processing circuit 12.

[0092] The region setting unit 182 has the function of the setting unit 70 shown in Fig. 1. The region setting unit 182 sets a region for a determination process in the image acquired by the image acquisition unit 181. The determination process is a process for determining whether or not the region is suitable for a predetermined process. The predetermined process is a measurement process or a 3D display process.

[0093] The determination unit 183 has the function of the determination unit 71 shown in Fig. 1. The determination unit 183 executes a determination process in the area set by the area setting unit 182.

[0094] The display control unit 184 controls the processing executed by the video signal processing circuit 12. The CCU 9 outputs a video signal. The video signal includes color data for each pixel of the image acquired by the imaging element 28. The display control unit 184 causes the video signal processing circuit 12 to output the video signal output from the CCU 9 to the display unit 5. The video signal processing circuit 12 outputs the video signal to the display unit 5. The display unit 5 displays an image based on the video signal output from the video signal processing circuit 12. As a result, the display control unit 184 displays the image acquired by the imaging element 28 on the display unit 5.

[0095] The display control unit 184 displays various types of information on the display unit 5. That is, the display control unit 184 displays various types of information on the image. The various types of information include measurement results and the like. The various types of information may also include a cursor. The cursor is a mark that allows the user to specify a specific point on the image.

[0096] For example, the display control unit 184 generates a graphic image signal of various information. The display control unit 184 outputs the generated graphic image signal to the video signal processing circuit 12. The video signal processing circuit 12 combines the video signal output from the CCU 9 and the graphic image signal output from the CPU 18. As a result, the various information is superimposed on the image. The video signal processing circuit 12 outputs the combined video signal to the display unit 5. The display unit 5 displays the image on which the various information is superimposed.

[0097] The display control unit 184 has the function of the display control unit 72 shown in Fig. 1. The display control unit 184 generates a graphic image signal indicating the result of the determination process executed by the determination unit 183. The display control unit 184 outputs the graphic image signal to the video signal processing circuit 12. The same process as described above is executed, and the display unit 5 displays an image on which the result of the determination process is superimposed. In this way, the display control unit 184 displays the result of the determination process on the display unit 5.

[0098] The user operates the operation unit 4 to input various information to the endoscope device 1. The operation unit 4 outputs the information input by the user. The information is input to the control interface 17, which is an input unit. The information is output from the control interface 17 to the CPU 18. The information receiving unit 185 receives the information input to the endoscope device 1 via the operation unit 4.

[0099] For example, the user operates the operation unit 4 to input cursor position information to the endoscope device 1. If the display unit 5 is configured as a touch panel, the user touches the screen of the display unit 5 to input position information indicating a position on the image to the endoscope device 1. The information receiving unit 185 receives the position information input to the endoscope device 1. The information receiving unit 185 calculates the position on the image based on the position information. The display control unit 184 displays the cursor at the position calculated by the information receiving unit 185.

[0100] Image processing in the second embodiment will be described with reference to Fig. 9. Fig. 9 shows the procedure of image processing.

[0101] The image sensor 28 continuously generates imaging signals. That is, the image sensor 28 generates imaging signals for each frame corresponding to a moving image. A moving image includes two or more frames. Each frame is composed of an image captured by the image sensor 28. The image sensor 28 captures a first image, and then captures a second image.

[0102] The image acquisition unit 181 acquires a first image captured by the image sensor 28 (step S101).

[0103] After step S101, the region setting unit 182 sets a first region in the first image for the first determination process using the first image (step S102). Step S102 corresponds to the first setting step in the first embodiment.

[0104] 10 shows the first image IM11 acquired in step S101. In step S102, the region setting unit 182 sets a first region RG11 in the first image IM11. The first region RG11 may have any size and any shape.

[0105] In the example shown in FIG. 10, the region setting unit 182 sets the upper left region in the first image IM11 as the first region RG11. The position of the first region RG11 is not limited to the position shown in FIG. 10. The region setting unit 182 may set the central region in the first image IM11 as the first region RG11. The region setting unit 182 may set two or more regions in the first image IM11 as the first region RG11. The region setting unit 182 may set the size or shape of the first region RG11 according to the computational resources of the endoscope device 1.

[0106] The region setting unit 182 may set the first region RG11 in the first image IM11 by using information previously set in the endoscope device 1. The information indicates the size or shape of the first region RG11. The region setting unit 182 may set the first region RG11 in the first image IM11 according to the state of the first image IM11. For example, the region setting unit 182 may set the first region RG11 in the first image IM11 according to information about a subject appearing in the first image IM11.

[0107] After step S102, the determination unit 183 executes a first determination process. That is, the determination unit 183 determines whether the first area set in step S102 is suitable for a predetermined process (step S103). Step S103 corresponds to the first determination step in the first embodiment.

[0108] The first determination process in step S103 will be described in detail with reference to FIG. 11. In the following example, the determination unit 183 calculates the distance between the subject in the first area and the camera, and uses this distance. This distance is a three-dimensional distance (3D distance) corresponding to the object distance described above. The index calculated by the determination unit 183 is not limited to distance. As will be described later, the determination unit 183 may calculate other indices.

[0109] The first objective optical system 53 forms a first optical image of the object seen from a first viewpoint. The second objective optical system 54 forms a second optical image of the object seen from a second viewpoint different from the first viewpoint. The imaging element 28 generates a stereo image corresponding to the first optical image and the second optical image. The stereo image includes a pair of two images. That is, the stereo image includes an image of the object seen from the first viewpoint and an image of the object seen from the second viewpoint. In the following example, the determination unit 183 calculates the above-mentioned distance by using the stereo image. Each of the first image and the second image is one of two images included in the stereo image. For example, each of the first image and the second image is an image of the object seen from the first viewpoint.

[0110] The determination unit 183 reads parameters necessary for stereo measurement from the ROM 13 (step S103a). These parameters include internal parameters, external parameters, etc., and also parameters for calculating 3D coordinates. These parameters are written to the ROM 13 at the time of shipment from the factory. The determination unit 183 accesses the data written to the ROM 13 and reads these parameters. The endoscope device 1 may communicate with a cloud system and download the necessary parameters. This allows the endoscope device 1 to read these parameters.

[0111] After step S103a, the determination unit 183 initializes an index i managed inside the endoscope device 1. For example, the determination unit 183 sets the index i to 0 (step S103b). The index i indicates the number of the pixel that is the target of the first determination process.

[0112] After step S103b, the determination unit 183 selects the i-th pixel in the first region (step S103c).

[0113] After step S103c, the determination unit 183 executes stereo matching processing to detect a point (corresponding point) corresponding to the i-th pixel (step S103d). The i-th pixel is located in one of the two images included in the stereo image, and the corresponding point is located in the other of the two images. The i-th pixel corresponds to measurement point 61 in FIG. 7. The corresponding point corresponds to corresponding point 62 in FIG. 7.

[0114] After step S103d, the determination unit 183 performs triangulation and calculates the distance between the subject and the camera at the i-th pixel (step S103e).

[0115] After step S103e, the determination unit 183 reads out the threshold value stored in the ROM 13 (step S103f). The threshold value may be a preset fixed value. The fixed value may be a commonly defined value. The threshold value may be adaptively changed depending on various conditions such as the composition of the image or the pixel position (coordinates).

[0116] After step S103f, the determination unit 183 compares the distance calculated in step S103e with the threshold value read out in step S103f. The determination unit 183 determines whether the distance is smaller than the threshold value (step S103g). If the distance is smaller than the threshold value, the determination unit 183 determines that the i-th pixel is suitable for the predetermined processing (step S103h). If the distance is equal to or greater than the threshold value, the determination unit 183 determines that the i-th pixel is not suitable for the predetermined processing (step S103i). The determination unit 183 associates the result of the determination in step S103h or step S103i with the i-th pixel.

[0117] After step S103h or step S103i, the determination unit 183 increments the index i by 1 (step S103j), thereby changing the pixel to be processed.

[0118] After step S103j, the determination unit 183 determines whether the determination in step S103h or step S103i has been performed for all pixels in the first region (step S103k). If the determination unit 183 determines in step S103k that the determination has been performed for all pixels in the first region, the first determination process ends. If the determination unit 183 determines in step S103k that the determination has not been performed for some pixels in the first region, step S103c is performed.

[0119] In the example shown in FIG. 11, two types of judgment results are obtained. One of the two types of judgment results indicates that the i-th pixel is suitable for the predetermined processing. The other of the two types of judgment results indicates that the i-th pixel is not suitable for the predetermined processing. The judgment results are not limited to this example. The judgment unit 183 may calculate the reliability of the predetermined processing instead of performing steps S103g, S103h, and S103i. For example, the reliability is expressed as a percentage.

[0120] The determination unit 183 may calculate an index other than distance in the first determination process. For example, the determination unit 183 may calculate the brightness (luminance) of the first region, the amount of blur in the first region, the intensity of the texture (pattern) in the first region, or the 3D shape of the subject in the first region. The determination unit 183 may calculate a correlation value by performing stereo matching processing on the first region. The determination unit 183 may calculate the amount of parallax in the first region. The determination unit 183 may use the calculated index to determine whether the first region is suitable for a predetermined process. The determination unit 183 may perform this determination by using a combination of two or more indexes.

[0121] Below, first to fifth examples for calculating the above indexes will be explained.

[0122] A first example will be described. Determination unit 183 calculates the brightness of a first region or the intensity of a texture in the first region. Determination unit 183 compares the brightness or intensity with a threshold to determine whether or not the first region is suitable for a predetermined process.

[0123] A second example will be described. The determination unit 183 calculates the amount of blur in the first region by using two or more images including the first image. For example, the determination unit 183 uses the first image and an image acquired before the first image. The determination unit 183 calculates the amount of blur by calculating the difference between these two images. The determination unit 183 compares the amount of blur with a threshold value to determine whether the first region is suitable for predetermined processing.

[0124] A third example will be described. The determination unit 183 executes 3D reconstruction processing by using two or more images including a first image. As a result, the determination unit 183 calculates the 3D shape of the subject in the first region. At this time, the determination unit 183 uses a pair of two images included in a stereo image. The first image is one of the two images. Alternatively, the determination unit 183 uses the first image and one or more images acquired before the first image.

[0125] The 3D shape of the subject includes 3D coordinates of two or more points on the subject. The determination unit 183 selects one of the two or more points as a point of interest. The determination unit 183 calculates the Euclidean distance (first distance) at the point of interest. The Euclidean distance indicates the 3D distance between the point of interest and points surrounding the point of interest. The determination unit 183 calculates the Euclidean distance (second distance) at points surrounding the point of interest by performing a process similar to the process at the point of interest. The determination unit 183 calculates the difference between the first distance and the second distance. If the difference is large, the determination unit 183 determines that the point of interest is not suitable for the predetermined process.

[0126] A fourth example will be described. The determination unit 183 performs stereo matching processing by using two images including a first image. As a result, the determination unit 183 calculates a correlation value in the first region. At this time, the determination unit 183 uses a pair of two images included in the stereo image. The first image is one of the two images. When ZNCC (Zero-means Normalized Cross-Correlation) is used in the stereo matching processing, the correlation value is greater than or equal to -1 and less than or equal to 1. If the correlation value is small, the determination unit 183 determines that the first region is not suitable for the predetermined processing. In the stereo matching processing, SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference) may be used as an index value. If SAD or SSD is large, the determination unit 183 determines that the first region is not suitable for the predetermined processing.

[0127] A fifth example will be described. The determination unit 183 calculates the amount of parallax in the first region by using two images including the first image. At this time, the determination unit 183 uses a pair of two images included in a stereo image. The first image is one of the two images. The determination unit 183 selects one of two or more pixels in the first region as a pixel of interest. The determination unit 183 detects a pixel corresponding to the pixel of interest (a corresponding pixel). The corresponding pixel is located in the other of the two images included in the stereo image. The determination unit 183 calculates the amount of parallax at the pixel of interest (a first amount of parallax) based on the position of the corresponding pixel. The determination unit 183 performs a process similar to the process at the pixel of interest to calculate the amount of parallax at pixels surrounding the pixel of interest (a second amount of parallax). The determination unit 183 calculates the difference between the first amount of parallax and the second amount of parallax. If the difference is large, the determination unit 183 determines that the pixel of interest is not suitable for the predetermined process.

[0128] Image processing in the second embodiment will be described with reference to Fig. 9 again. After step S103, the display control unit 184 displays the result of the first determination processing in step S103 on the display unit 5 (step S104).

[0129] The display control unit 184 displays the first image IM11 shown in FIG. 10 on the display unit 5, and displays the result of the first determination process on the first image IM11. For example, the first region RG11 shown in FIG. 10 includes an OK region and an NG region. The OK region includes pixels that are suitable for a predetermined process. The NG region includes pixels that are not suitable for a predetermined process. The display control unit 184 may display the OK region in a first color and the NG region in a second color different from the first color. For example, the display control unit 184 may display the OK region in green and the NG region in red. The display control unit 184 may display the OK region in a first pattern and the NG region in a second pattern different from the first pattern. The display control unit 184 may display a first line and a second line. The first line surrounds the OK region. The second line surrounds the NG region.

[0130] The display control unit 184 may display the first image IM11 in a predetermined area on the display screen of the display unit 5, and may also display the result of the first determination process in a sub-window. The sub-window is placed in an area on the display screen of the display unit 5 that is different from the predetermined area.

[0131] The display control unit 184 may display the text information on the display unit 5. For example, if 90% of the first region is suitable for a predetermined process, the text information may indicate that the entire first region is suitable for the predetermined process.

[0132] As long as the result of the first determination processing is notified to the user, the method by which the display control unit 184 displays the result of the first determination processing is not limited to the above example. Step S104 does not need to be executed. The display control unit 184 may display the result of the first determination processing together with the result of the second determination processing, which will be described later, on the display unit 5 in step S108, which will be described later, without executing step S104.

[0133] After step S104, the image acquisition unit 181 acquires a second image captured by the image sensor 28 (step S105).

[0134] After step S105, the region setting unit 182 sets a second region in the second image for the second determination process using the second image (step S106). Step S106 corresponds to the second setting step in the first embodiment.

[0135] The position (image coordinates) of the second region in the second image may be the same as or different from the position (image coordinates) of the first region in the first image. In the second embodiment, it is assumed that the composition of the photograph does not change between the first image and the second image, and the composition of the second image is the same as the composition of the first image. In this state, both the subject and the tip 20 are stationary. In the following example, the position of the second region in the second image is different from the position of the first region in the first image.

[0136] For example, a first region of a first image includes an area in which a first object is captured. Because both the object and tip 20 are stationary, the first object is captured in a first region of a second image at the same position as the first region of the first image. The second region of the second image is set at a different position from the first region of the second image. Therefore, the second region includes an area in which a second object different from the first object is captured.

[0137] An example in which the composition of the photograph changes between the first image and the second image, and the composition of the second image differs from the composition of the first image will be described in the third embodiment.

[0138] FIG. 12 shows the second image IM12 acquired in step S105. The first region RG11a and the second region RG12 are shown in FIG. 12. The position of the first region RG11a in the second image IM12 is the same as the position of the first region RG11 in the first image IM11 shown in FIG. 10. The second region RG12 is different from the first region RG11a corresponding to the first region RG11 for which the first determination process was performed. In step S106, the region setting unit 182 sets the second region RG12 in the second image IM12. Like the first region RG11, the second region RG12 may have any size and any shape. The region setting unit 182 may set the second region RG12 in the second image IM12 depending on the observation conditions, etc.

[0139] After step S106, the determination unit 183 executes a second determination process. That is, the determination unit 183 determines whether the second area set in step S106 is suitable for a predetermined process (step S107). Step S107 corresponds to the second determination step in the first embodiment.

[0140] The second determination process is similar to the first determination process shown in Fig. 11. In the second determination process, data in a second region of a second image is used instead of data in a first region of a first image.

[0141] In the second determination process, the determination unit 183 calculates the distance between the subject in the second region and the camera and uses the calculated distance. In the second determination process, the determination unit 183 may calculate an index other than the distance. For example, the determination unit 183 may calculate the brightness (luminance) of the second region, the amount of blur in the second region, the intensity of the texture (pattern) in the second region, or the 3D shape of the subject in the second region. The determination unit 183 may calculate a correlation value by performing stereo matching processing on the second region. The determination unit 183 may calculate the amount of parallax in the second region. The determination unit 183 may use the calculated index to determine whether the second region is suitable for a predetermined process. The determination unit 183 may perform this determination by using a combination of two or more indexes.

[0142] Below, first to fifth examples for calculating the above indexes will be explained.

[0143] A first example will be described. The determination unit 183 calculates the brightness of the second region or the intensity of the texture in the second region. The determination unit 183 compares the brightness or the intensity with a threshold to determine whether the second region is suitable for a predetermined process.

[0144] A second example will be described. The determination unit 183 calculates the amount of blur in the second region by using two or more images including the second image. For example, the determination unit 183 uses a first image and a second image. The determination unit 183 calculates the amount of blur by calculating the difference between these two images. The determination unit 183 compares the amount of blur with a threshold value to determine whether the second region is suitable for predetermined processing.

[0145] A third example will be described. The determination unit 183 performs 3D reconstruction processing by using two or more images including the second image. As a result, the determination unit 183 calculates the 3D shape of the subject in the second region. At this time, the determination unit 183 uses a pair of two images included in a stereo image. The second image is one of the two images. Alternatively, the determination unit 183 uses the second image and one or more images acquired before the second image. The one or more images may include the first image. The method of determination using the 3D shape is the same as the method in the first determination processing.

[0146] A fourth example will be described. The determination unit 183 performs stereo matching processing by using two images including the second image. As a result, the determination unit 183 calculates a correlation value in the second region. At this time, the determination unit 183 uses a pair of two images included in the stereo image. The second image is one of the two images. When ZNCC is used in the stereo matching processing, the correlation value is greater than or equal to -1 and less than or equal to 1. If the correlation value is small, the determination unit 183 determines that the second region is not suitable for the predetermined processing. The SAD or SSD may be used as an index value in the stereo matching processing. If the SAD or SSD is large, the determination unit 183 determines that the first region is not suitable for the predetermined processing.

[0147] A fifth example will be described. The determination unit 183 calculates the amount of parallax in the second region by using two images including the second image. At this time, the determination unit 183 uses a pair of two images included in a stereo image. The second image is one of the two images. The determination method using the amount of parallax is the same as the method in the first determination process.

[0148] After step S107, the display control unit 184 displays the result of the first determination process in step S103 and the result of the second determination process in step S107 on the display unit 5 (step S108). Step S108 corresponds to the information display step in the first embodiment. When step S108 is executed, the image processing ends.

[0149] FIG. 13 shows the second image IM13 displayed on the display unit 5 in step S108. The display control unit 184 displays the second image IM13 on the display unit 5. The display control unit 184 also displays a region RG13 on the second image IM13. The region RG13 is obtained as the logical sum of the first region RG11a and the second region RG12 shown in FIG. 12. The first region RG11a corresponds to the first region RG11 shown in FIG. 10. The result of the first determination process is associated with each pixel of the first region RG11. The result of the second determination process is associated with each pixel of the second region RG12.

[0150] The display control unit 184 superimposes the result of the first determination process on each pixel of the region RG13 corresponding to each pixel of the first region RG11. The display control unit 184 also superimposes the result of the second determination process on each pixel of the region RG13 corresponding to each pixel of the second region RG12.

[0151] Region RG13 includes region RG13a and region RG13b. Region RG13a includes pixels suitable for predetermined processing. Region RG13b includes pixels unsuitable for predetermined processing. Display control unit 184 may use a method similar to the method shown in step S104 to display region RG13a and region RG13b.

[0152] After acquiring the second image, the image sensor 28 acquires the third image. After the image processing shown in FIG. 9 is completed, the image acquisition unit 181 acquires the third image acquired by the image sensor 28. The area setting unit 182 sets a third area in the third image. The third area is different from the area of the third image corresponding to the first area and different from the area of the third image corresponding to the second area. The judgment unit 183 judges whether the third area is suitable for a predetermined process (third judgment process). The display control unit 184 displays the third image on the display unit 5, and displays the results of the first judgment process, the second judgment process, and the third judgment process on the third image.

[0153] The same process as above is repeated when a fourth image, a fifth image, etc. are acquired, and each time an image is acquired, the area of the subject on which the determination process is performed expands.

[0154] The above method may be applied to techniques other than stereo measurement. The above method may be applied to techniques that use images acquired through a monocular optical system. For example, the above method may be applied to structure from motion (SfM). The above method may be applied to multi-viewpoint stereo.

[0155] The above method may be applied to an active 3D measurement technique, in which light having a predetermined pattern, such as a striped pattern or a random pattern, is used. The above method may be applied to a technique called Time of Flight (ToF).

[0156] The user may select whether or not to enable the function that executes the determination process. For example, if the area to be examined is located deeper than the insertion port of the endoscope and the examination does not need to be performed while the insertion portion 2 is being inserted toward the examination destination, it is convenient for the user to disable the function that executes the determination process.

[0157] Each aspect of the present invention may include the following modifications: Before the information display step (step S108) is executed, the display control unit 184 displays, on the display unit 5 (display), information indicating whether the first region is suitable for a predetermined process in a second information display step (step S104).

[0158] Each aspect of the present invention may include the following modifications: In a first setting step (step S102), region setting unit 182 sets a first region including a region in which a first subject is captured in the first image. In a second setting step (step S106), region setting unit 182 sets a second region including a region in which a second subject different from the first subject is captured in the second image.

[0159] 10 and 12, the region setting unit 182 sets a first region RG11 at a first position in a first image IM11. The region setting unit 182 sets a second region RG12 at a second position in a second image IM12. The second position is different from the position in the second image corresponding to the first position.

[0160] Each aspect of the present invention may include the following modifications: Each of the first image and the second image is generated based on an optical image generated through the stereo optical adapter 30 (stereo optical system).

[0161] Each aspect of the present invention may include the following modifications: In a first determination step (step S103), the determination unit 183 uses at least one of the 3D shape of the subject in the first region acquired by executing a process to calculate the 3D shape, the distance between the subject in the first region and the camera, the brightness of the first region, the amount of blur in the first region, and the intensity of the texture in the first region.

[0162] Each aspect of the present invention may include the following modifications: In a second determination step (step S107), the determination unit 183 uses at least one of the 3D shape of the subject in the second region acquired by executing a process to calculate the 3D shape, the distance between the subject in the second region and the camera, the brightness of the second region, the amount of blur in the second region, and the intensity of the texture in the second region.

[0163] Each aspect of the present invention may include the following modifications: The first image is generated based on an optical image generated through the stereo optical adapter 30 (stereo optical system). In a first determination step (step S103), the determination unit 183 uses at least one of a correlation value acquired by performing stereo matching processing on the first region and the amount of parallax in the first region.

[0164] Each aspect of the present invention may include the following modifications: The second image is generated based on an optical image generated through the stereo optical adapter 30 (stereo optical system). In a second determination step (step S107), the determination unit 183 uses at least one of a correlation value acquired by performing stereo matching processing on the second region and the amount of parallax in the second region.

[0165] Each aspect of the present invention may include the following modifications: In the first determination step (step S103), the determination unit 183 uses two or more images including the first image.

[0166] Each aspect of the present invention may include the following modifications: In the second determination step (step S107), the determination unit 183 uses two or more images including the second image.

[0167] In the second embodiment, the endoscope device 1 executes the first determination process and the second determination process at different times. Because the load of the first determination process and the second determination process is distributed, the endoscope device 1 can reduce the load of the determination process. The endoscope device 1 can execute the determination process over a wide range of the image in a short time.

[0168] The endoscope device 1 applies a region (first region) to be determined in a past image (first image) to a current image (second image), and also applies the result of the determination process (first determination process) on the past image to the current image. The endoscope device 1 displays the result of the determination process (first determination process) on the past image together with the result of the determination process (second determination process) on the current image. Therefore, the endoscope device 1 can reduce the time lag from when the image is acquired to when the result of the determination process is displayed. Even when the computational resources of the endoscope device 1 are limited, the endoscope device 1 can reduce the time lag related to the display of the result of the determination process without increasing the amount of calculation.

[0169] The user can confirm that the desired area is suitable for measurement, etc., and take a still image or record a video. The desired area is the area that shows an abnormality, etc. Even if the user leaves the examination site to perform measurements, the risk of the user having to return to the examination site and take the image again is reduced. This improves the work efficiency of the entire examination.

[0170] 12, the region setting unit 182 sets a second region RG12 different from the first region RG11a in the second image IM12. If the composition of the photographs does not change between the first image IM11 and the second image IM12, the region setting unit 182 can easily set the second region RG12 in the second image IM12.

[0171] (Third embodiment) A third embodiment of the present invention will be described. In the second embodiment described above, it is assumed that the composition of the image captured does not change between the first image and the second image. On the other hand, in the third embodiment, it is assumed that the tip 20 of the insertion portion 2 moves and the composition of the image captured changes between the first image and the second image.

[0172] The CPU 18 shown in Fig. 8 is changed to a CPU 18a shown in Fig. 14. Fig. 14 shows the functional configuration of the CPU 18a. The functions of the CPU 18a include a control unit 180, an image acquisition unit 181, an area setting unit 182, a determination unit 183, a display control unit 184, an information receiving unit 185, and an area searching unit 186. At least one of the blocks shown in Fig. 14 may be configured with a circuit different from that of the CPU 18a. Description of the same configuration as that shown in Fig. 8 will be omitted.

[0173] Each unit shown in Fig. 14 may be configured with at least one of a processor and a logic circuit. Each unit shown in Fig. 14 may include one or more processors. Each unit shown in Fig. 14 may include one or more logic circuits.

[0174] Area search section 186 (search section) searches for an area of the second image in which the subject in the first image appears (search step). Area setting section 182 sets a second area in the second image that is different from that area.

[0175] Image processing in the third embodiment will be described with reference to Fig. 15. Fig. 15 shows the procedure of image processing. Description of the same processing as that shown in Fig. 9 will be omitted.

[0176] After step S105, the region search unit 186 searches for a region of the second image that is the same as the first region set in step S102 (step S111). Step S111 corresponds to a search step.

[0177] 16 shows the first image IM21 acquired in step S101. In step S102, the region setting unit 182 sets a first region RG21 in the first image IM21.

[0178] 17 shows the second image IM22 acquired in step S105. After the image sensor 28 acquires the first image IM21, the tip 20 of the insertion portion 2 moves to the right. Therefore, the subject captured in the second image IM22 is shifted to the left compared to the subject captured in the first image IM21.

[0179] In step S111, the region search unit 186 searches for a region in the second image IM22 that contains the same subject as the subject captured in the first region RG21. That is, the region search unit 186 searches for a region in the second image IM22 that corresponds to the first region RG21 for which the first determination process was executed. In this way, the region search unit 186 detects the first region RG22 in the second image IM22. The region search unit 186 detects the first region RG22 by using a general image processing method such as template matching or feature point matching. As long as the region search unit 186 can detect a region that contains the same subject as the subject captured in the first region RG21, the method in step S111 is not limited to the above example.

[0180] The entire subject captured in the first region is not necessarily captured in the second image IM22. In the example shown in Figure 17, the left part of the subject captured in the first region RG21 is not captured in the second image IM22.

[0181] After step S111, the region setting unit 182 sets a second region in the second image for the second determination process using the second image (step S112). Step S112 corresponds to the second setting step in the first embodiment. After step S112, step S107 is executed.

[0182] In the second embodiment described above, since the composition of the photograph does not change between the first image and the second image, the region setting unit 182 sets a second region RG12 in the second image IM12 that is different from the first region RG11a shown in Fig. 12. The method for setting the second region in the third embodiment is different from this method.

[0183] First, the region setting unit 182 sets a provisional region in the second image. The position of the provisional region in the second image is the same as the position of the first region in the first image. In the example shown in FIG. 17, the region setting unit 182 sets a provisional region RG23 in the second image IM22. The position of the provisional region RG23 in the second image IM22 is the same as the position of the first region RG21 in the first image IM21. The provisional region RG23 includes the first region RG22 and the region RG24. The first region RG22 corresponds to the first region RG21 for which the first determination process has been executed. The second determination process has not yet been executed for the region RG24.

[0184] The region setting unit 182 sets a second region in the second image, obtained by excluding at least a portion of the region detected in step S111 from the provisional region. At this time, the region setting unit 182 excludes from the provisional region the region determined by the determination unit 183 in the first determination process as being suitable for the predetermined process (part of the first region RG22). In the example shown in FIG. 17, the second region includes a region RG24 for which the second determination process has not been performed. The second region also includes a region in the second image corresponding to a region determined by the determination unit 183 in the first determination process as being unsuitable for the predetermined process. In other words, the second region includes at least a portion of the first region RG22. If the determination unit 183 determines in the first determination process that the entire first region RG22 is suitable for the predetermined process, the second region does not include the first region RG22. The determination unit 183 can perform the determination process again on a region in the second image corresponding to a region in the first image that is unsuitable for the predetermined process.

[0185] The second region does not need to include a region of the second image that corresponds to a region in the first image that is not suitable for the predetermined processing. The region setting unit 182 may set a second region in the second image that is different from the region detected by the region searching unit 186.

[0186] In step S108, the display control unit 184 displays the results of the first determination processing and the second determination processing on the display unit 5. In the example shown in FIG. 17, the display control unit 184 displays the results of the first determination processing and the second determination processing in the first region RG22 and the region RG24. In some cases, the determination unit 183 determines in the first determination processing that a portion of the first region RG21 in the first image IM21 is not suitable for the predetermined processing. In that case, the determination unit 183 executes the second determination processing on a portion of the first region RG22 in the second image IM22. In that case, in step S108, the display control unit 184 displays the result of the second determination processing, rather than the result of the first determination processing, in a portion of the first region RG22.

[0187] After acquiring the second image, the image sensor 28 acquires the third image. After the image processing shown in FIG. 15 is completed, the image acquisition unit 181 acquires the third image acquired by the image sensor 28. The region search unit 186 searches for a region in the third image that contains the same subject as the subject appearing in the first region in the first image. That is, the region search unit 186 searches for a region in the third image that corresponds to the first region in which the first determination process was executed. The region search unit 186 also searches for a region in the third image that contains the same subject as the subject appearing in the second region in the second image. That is, the region search unit 186 searches for a region in the third image that corresponds to the second region in which the second determination process was executed.

[0188] The region setting unit 182 sets a provisional region in the third image. The position of the provisional region in the third image is the same as the position of the first region in the first image. The region setting unit 182 sets a third region obtained by excluding at least a part of the region detected by the region searching unit 186 from the provisional region in the third image. At this time, the region setting unit 182 excludes from the provisional region any region that the determination unit 183 has determined in the first determination process to be suitable for the predetermined process. The region setting unit 182 also excludes from the provisional region any region that the determination unit 183 has determined in the second determination process to be suitable for the predetermined process.

[0189] The third region may include a region of the third image corresponding to a first region of the first image that is determined by the determination unit 183 to be unsuitable for the predetermined process. The third region may also include a region of the third image corresponding to a second region of the second image that is determined by the determination unit 183 to be unsuitable for the predetermined process.

[0190] The determination unit 183 determines whether the third region is suitable for a predetermined process (third determination process). The display control unit 184 displays the third image on the display unit 5, and also displays the results of the first determination process, the second determination process, and the third determination process on the third image.

[0191] The same process as above is repeated when a fourth image, a fifth image, etc. are acquired, and each time an image is acquired, the area of the subject on which the determination process is performed expands.

[0192] The region setting unit 182 may set, in the second image, a second region obtained by excluding from the provisional region the entire region determined in the first determination process by the determination unit 183. In the example shown in Fig. 17, the region setting unit 182 may set the region RG24 as the second region in the second image IM22.

[0193] Each aspect of the present invention may include the following modifications. In a search step (step S111), the region search unit 186 searches for a first region RG22 (subject region) in which a first subject is captured. The first region RG22 is included in a second image IM22. In a second setting step (step S112), the region setting unit 182 sets a second region in the second image IM22 that includes a region RG24 different from the first region RG22.

[0194] Each aspect of the present invention may include the following modifications: In a first setting step (step S102), the region setting unit 182 sets a first region RG21 including an area in which a first subject is captured in the first image IM21. In a second setting step (step S112), the region setting unit 182 sets a second region including at least a part of the first region RG22 in which the first subject is captured.

[0195] In the third embodiment, even if the computational resources of the endoscope device 1 are small, the endoscope device 1 can execute the determination process over a wide range of the image in a short time. The endoscope device 1 can reduce the time lag in displaying the results of the determination process without increasing the amount of calculation. As a result, the efficiency of the examination is improved.

[0196] 17, the region setting unit 182 sets, in the second image IM22, a second region including a region RG24 that is different from the first region RG22 detected by the region searching unit 186. When the composition of the photograph changes between the first image IM21 and the second image IM22, the region setting unit 182 can set, in the second image IM22, the region RG24, in which the first determination process has not been performed.

[0197] (Fourth embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, the endoscope device 1 executes one of the processing in the second embodiment and the processing in the third embodiment based on the movement of the tip 20 of the insertion portion 2. Furthermore, if the movement of the tip 20 of the insertion portion 2 is large, the endoscope device 1 acquires the first image again without executing the second determination processing.

[0198] The CPU 18a shown in Fig. 14 is changed to a CPU 18b shown in Fig. 18. Fig. 18 shows the functional configuration of the CPU 18b. The functions of the CPU 18b include a control unit 180, an image acquisition unit 181, a region setting unit 182, a determination unit 183, a display control unit 184, an information reception unit 185, a region search unit 186, and a movement determination unit 187. At least one of the blocks shown in Fig. 18 may be configured with a circuit different from that of the CPU 18b. Description of the same configuration as that shown in Fig. 14 will be omitted.

[0199] Each unit shown in Fig. 18 may be configured with at least one of a processor and a logic circuit. Each unit shown in Fig. 18 may include one or more processors. Each unit shown in Fig. 18 may include one or more logic circuits.

[0200] The motion determination unit 187 (detection unit) detects the amount of motion of the subject between the first image and the second image. As a result, the motion determination unit 187 detects the amount of change in composition between the first image and the second image (detection step). The motion determination unit 187 compares the detected amount of motion with a predetermined amount to make a determination regarding the motion of the subject.

[0201] Image processing in the fourth embodiment will be described with reference to Fig. 19. Fig. 19 shows the procedure of image processing. Description of the same processing as that shown in Fig. 9 or Fig. 15 will be omitted.

[0202] After step S105, the movement determination unit 187 detects the amount of movement of the subject by using the first image acquired in step S101 and the second image acquired in step S105 (step S121). Step S121 corresponds to a detection step.

[0203] After step S121, the motion determination unit 187 determines whether the amount of motion detected in step S121 is greater than a first predetermined amount (step S122). The first predetermined amount is greater than 0. If the motion determination unit 187 determines in step S122 that the amount of motion is equal to or less than the first predetermined amount, step S106 is executed. In this case, the region setting unit 182 sets a second region in the second image by using the method in the second embodiment.

[0204] If the motion determination unit 187 determines in step S122 that the amount of motion is greater than the first predetermined amount, the motion determination unit 187 determines whether the amount of motion is greater than a second predetermined amount (step S123). The second predetermined amount is greater than the first predetermined amount.

[0205] If the motion determination unit 187 determines in step S123 that the amount of motion is equal to or less than the second predetermined amount, step S111 is executed. In this case, the region setting unit 182 sets the second region in the second image by using the method in the third embodiment.

[0206] If the movement determination unit 187 determines in step S123 that the amount of movement is greater than the second predetermined amount, step S101 is executed. In step S101, the image acquisition unit 181 acquires, as the first image, the image acquired by the image sensor 28 after the second image. If the composition of the second image is significantly different from the composition of the first image, the endoscope device 1 restarts image processing from the beginning.

[0207] In this case, the control unit 180 erases the information about the area set in the previous image. That area will not be used in the process of setting an area for the determination process in the new image. The control unit 180 also erases the results of the determination process in the area set in the previous image. Therefore, the results of that determination process are not displayed on the display unit 5. The display control unit 184 may display a predetermined message on the display unit 5. The predetermined message indicates that the results of the determination process in the previous image have been erased and that image processing will be started over from the beginning.

[0208] After step S121 is executed, step S123 may be executed without executing step S122, in which case step S106 is not executed.

[0209] Each aspect of the present invention may include the following modifications: Before the second determination step (step S107) is performed, the movement determination unit 187 detects the amount of change in composition between the first image and the second image in a detection step (step S121).

[0210] Each aspect of the present invention may include the following modifications: Only when the amount of change detected in the detecting step (step S121) is smaller than a predetermined amount, the information displaying step (step S108) is executed.

[0211] In the fourth embodiment, even if the computational resources of the endoscope device 1 are small, the endoscope device 1 can execute the determination process over a wide range of the image in a short time. The endoscope device 1 can reduce the time lag in displaying the results of the determination process without increasing the amount of calculation. As a result, the efficiency of the examination is improved.

[0212] If the amount of change in composition between the first image and the second image is large, it is difficult for the area search unit 186 to search for an area in the second image that is the same as the first area in the first image in step S111. Therefore, it is difficult for the endoscope device 1 to apply the results of the determination process on the first image to the second image. In this case, the first image is acquired again, and image processing is performed again. This improves the reliability of the determination process.

[0213] (Fifth embodiment) A fifth embodiment of the present invention will be described. In the second to fourth embodiments, the area on which the determination process is performed is arbitrary. When the endoscope device 1 processes a third image, a fourth image, etc. in addition to the first and second images, the area of the subject on which the determination process is performed gradually expands. As a result, the endoscope device 1 may obtain the result of the determination process on the entire area of the kth image. The number k is 2 or greater.

[0214] On the other hand, the area where the user desires to perform a predetermined process such as a measurement process is often limited to a specific area such as an abnormal area. Therefore, in the fifth embodiment, the endoscope device 1 limits the determination area where the determination process is performed. Then, the endoscope device 1 performs the determination process in the determination area.

[0215] The endoscope device 1 has a CPU 18a shown in FIG. 14. The region setting unit 182 (generation unit) generates information about the judgment region (generation step). The information indicates at least the size of the judgment region. The information may indicate the position of the judgment region in addition to the size of the judgment region. The angle of view of the judgment region is smaller than the angle of view of each of the first image and the second image. In other words, the judgment region is smaller than the entire region of each of the first image and the second image. The judgment region in the second image is an area that shows the same subject as the subject shown in the judgment region in the first image. The size and shape of the judgment region are common between the first image and the second image.

[0216] Region setting unit 182 sets a first region as a judgment region in the first image. The first region is included in the judgment region and is smaller than the judgment region. Region setting unit 182 also sets a second region as a judgment region in the second image. The second region is included in the judgment region and is smaller than the judgment region.

[0217] Image processing in the fifth embodiment will be described with reference to Fig. 20. Fig. 20 shows the procedure of image processing. The image processing shown in Fig. 20 is a modified example of the image processing shown in Fig. 15. The image processing shown in Fig. 20 may be combined with the image processing shown in Fig. 9 or Fig. 19. Description of the same processing as the processing shown in Fig. 9 or Fig. 15 will be omitted.

[0218] The region setting unit 182 generates information about the determination region and stores the information in the RAM 14 (step S131). Step S131 corresponds to a generation step.

[0219] For example, the user operates the operation unit 4 to input information indicating the size of the determination region into the endoscope device 1. If the display unit 5 is configured as a touch panel, the user inputs the information into the endoscope device 1 by touching the screen of the display unit 5. The information receiving unit 185 receives the information input into the endoscope device 1. The region setting unit 182 generates information about the determination region based on the information.

[0220] Alternatively, the image acquisition unit 181 acquires an image captured by the image sensor 28. The region setting unit 182 executes image processing to detect an abnormal region in the image. The region setting unit 182 generates information about a determination region based on the size of the detected region. The method for generating information about the determination region is not limited to the above example.

[0221] If the determination region is large, there is a possibility that the determination unit 183 will not be able to complete the determination process within a preset period (required time). Therefore, the region setting unit 182 executes a region division process. In the region division process, the region setting unit 182 divides the determination region into two or more partial regions (step S132). The size of the partial regions is set so that the determination unit 183 can complete the determination process before the image acquisition unit 181 acquires the next image.

[0222] The region division process in step S132 will be described in detail with reference to Fig. 21. The region setting unit 182 acquires first information indicating the size (first size) of the determination region from the RAM 14 (step S132a). For example, the first information indicates the number of pixels in the determination region.

[0223] After step S132a, the region setting unit 182 acquires second information from the ROM 13 (step S132b). The second information indicates the size (second size) of the region in which the determination unit 183 can perform determination processing in one image. For example, the second information indicates the maximum number of pixels in the region. The second size is set in advance depending on the computational resources of the endoscope device 1.

[0224] In the example shown in Fig. 21, step S132a is executed, and then step S132b is executed. Alternatively, step S132b may be executed first, and then step S132a may be executed.

[0225] After step S132b, the region setting unit 182 determines whether the first size is greater than the second size (step S132c).

[0226] If the region setting unit 182 determines in step S132c that the first size is equal to or smaller than the second size, the region setting unit 182 sets the determination region as a target region for the determination process (step S132d). When step S132d is executed, the region division process ends.

[0227] If the region setting unit 182 determines in step S132c that the first size is greater than the second size, the region setting unit 182 divides the judgment region into two or more partial regions (step S132e), each of which has a size smaller than the second size.

[0228] The method by which the region setting unit 182 divides the judgment region into two or more partial regions is not limited to a specific method. The region setting unit 182 may divide the judgment region in the vertical direction of the image. The region setting unit 182 may divide the judgment region in the horizontal direction of the image. The region setting unit 182 may divide the judgment region in a grid pattern.

[0229] After step S132e, the region setting unit 182 assigns ranks to two or more partial regions (step S132f). For example, if the region setting unit 182 divides the judgment region into two partial regions, the region setting unit 182 assigns first rank to one of the two partial regions and second rank to the other of the two partial regions. The region setting unit 182 uses the first-ranked partial region in the first judgment process and the second-ranked partial region in the second judgment process. When step S132f is executed, the region division process ends.

[0230] Image processing in the fifth embodiment will be described with reference to Fig. 20 again. After step S132, step S101 is executed.

[0231] In step S102, the region setting unit 182 sets a first region in the first image. At this time, the region setting unit 182 uses information about the determination region to set the determination region in the first image. Furthermore, the region setting unit 182 sets the first region in the entirety or part of the determination region in the first image. Specifically, the region setting unit 182 sets the first region in the target region set in step S132d. Alternatively, the region setting unit 182 sets the first region in the partial region assigned the first place in step S132f.

[0232] After step S105, the region search unit 186 searches for a region in the second image that is the same as the judgment region in the first image (step S133). The first region is the whole or part of the judgment region. Therefore, the region search unit 186 searches for a region in the second image that is the same as the judgment region that includes the first region. In step S133, the region search unit 186 executes the same process as step S111 shown in FIG. 15.

[0233] If the region setting unit 182 detects an abnormal region in the first image in step S131, the region searching unit 186 may detect the abnormal region in the second image in step S133. Any method may be used in step S133 as long as the region searching unit 186 can detect a region that is the same as the judgment region.

[0234] After step S133, step S111 is executed. At this time, region search unit 186 processes the region detected in step S133 and searches for a region of the second image that is the same as the first region set in step S102.

[0235] After step S111, in step S112, the region setting unit 182 sets a second region in the second image. At this time, the region setting unit 182 sets the second region in the whole or part of the region detected in step S133. Specifically, the region setting unit 182 sets the second region in the target region set in step S132d. Alternatively, the region setting unit 182 sets the second region in the partial region assigned the second place in step S132f.

[0236] 22 shows the first image IM31 acquired in step S101. In step S102, the region setting unit 182 sets a judgment region RG31 in the first image IM31, and sets a first region in the judgment region RG31. In step S102, the region setting unit 182 sets a first region RG32 in the judgment region RG31 in the first image IM31. The first region RG32 has a size approximately one-third that of the judgment region RG31.

[0237] FIG. 23 shows the second image IM32 acquired in step S105. In step S133, the region search unit 186 searches for a region that includes the same subject as the subject that appears in the judgment region RG31 shown in FIG. 22. In this way, the region search unit 186 detects a judgment region RG33 in the second image IM32. In step S111, the region search unit 186 searches for a region that includes the same subject as the subject that appears in the first region RG32 shown in FIG. In this way, the region search unit 186 detects a first region RG34 in the second image IM32. In step S112, the region setting unit 182 sets a second region in the judgment region RG33 that corresponds to the judgment region RG31 in the second image IM32.

[0238] The second region includes region RG35 in judgment region RG33. The second judgment process has not yet been performed on region RG35. The second region also includes a region of the second image IM32 corresponding to a region that the judgment unit 183 judged in the first judgment process to be unsuitable for the predetermined process. In other words, the second region includes at least a portion of the first region RG34. If the judgment unit 183 judges in the first judgment process that the entire first region RG34 is suitable for the predetermined process, the second region does not include the first region RG34.

[0239] In step S108, the display control unit 184 displays the results of the first determination processing and the second determination processing on the display unit 5. In the example shown in FIG. 23, the display control unit 184 displays the results of the first determination processing and the second determination processing in the first region RG34 and the region RG35. In some cases, the determination unit 183 determines in the first determination processing that a portion of the first region RG32 in the first image IM31 is not suitable for the predetermined processing. In that case, the determination unit 183 executes the second determination processing on a portion of the first region RG34 in the second image IM32. In that case, in step S108, the display control unit 184 displays the result of the second determination processing, rather than the result of the first determination processing, on a portion of the first region RG34.

[0240] Each aspect of the present invention may include the following modifications. In a generating step (step S131), region setting unit 182 generates information about the judgment region in each of the first image and the second image. After the judgment region information is generated, in a first setting step (step S102), region setting unit 182 sets the first region as the judgment region in the first image. In a second setting step (step S112), region setting unit 182 sets the second region as the judgment region in the second image.

[0241] In the fifth embodiment, the user pre-sets the area where he / she wants to perform a predetermined process, and the endoscope device 1 performs the determination process only in that area. The user can quickly determine whether the area is suitable for the predetermined process, thereby improving the efficiency of the examination.

[0242] (Sixth embodiment) A sixth embodiment of the present invention will be described. In the second to fifth embodiments, the endoscope device 1 executes a determination process during an inspection and displays the results of the determination process. In the sixth embodiment, a user brings a video from the inspection site back to their workplace or the like and causes the endoscope device 1 to execute a measurement process. The endoscope device 1 in the sixth embodiment has a function that enables a user to efficiently search the video for images to be used in the measurement process. The endoscope device 1 records the video during the inspection. After the inspection is completed, the endoscope device 1 executes a measurement process by using the video. Below, an example will be described in which the predetermined process described above is a measurement process.

[0243] The CPU 18 shown in Fig. 8 is changed to a CPU 18c shown in Fig. 24. Fig. 24 shows the functional configuration of the CPU 18c. The functions of the CPU 18c include a control unit 180, an image acquisition unit 181, an area setting unit 182, a determination unit 183, a display control unit 184, an information reception unit 185, an area search unit 186, and a video processing unit 188. At least one of the blocks shown in Fig. 24 may be configured with a circuit different from that of the CPU 18c. Description of the same configuration as that shown in Fig. 8 will be omitted.

[0244] Each unit shown in Fig. 24 may be configured with at least one of a processor and a logic circuit. Each unit shown in Fig. 24 may include one or more processors. Each unit shown in Fig. 24 may include one or more logic circuits.

[0245] The video processing unit 188 (processing unit) processes the video. The video processing unit 188 associates predetermined information with images included in the video. The predetermined information is used by the user to select an image. The selected image is used for measurement processing.

[0246] After the determination unit 183 executes the first determination process, the video processing unit 188 associates the first execution information with the first image (first association step). The first execution information indicates that the first determination process has been executed. For example, the video processing unit 188 adds the first execution information to the video file by embedding the first execution information in the video file.

[0247] After the determination unit 183 executes the second determination process, the video processing unit 188 associates the second execution information with the second image (second association step). The second execution information indicates that the second determination process has been executed. The video processing unit 188 adds the second execution information to the video file by embedding the second execution information in the video file.

[0248] After the determination unit 183 executes the first determination process, the video processing unit 188 associates the first result information with the first image. The first result information indicates the result of the first determination process. For example, the video processing unit 188 adds the first result information to the video file by embedding the first result information in the video file. The video processing unit 188 may embed the first result information in a first frame in the video. The first frame corresponds to the first image.

[0249] After the determination unit 183 executes the second determination process, the video processing unit 188 associates the second result information with the second image. The second result information indicates the result of the second determination process. For example, the video processing unit 188 adds the second result information to the video file by embedding the second result information in the video file. The video processing unit 188 may embed the second result information in a second frame in the video. The second frame corresponds to the second image.

[0250] The video processing unit 188 selects either the first image or the second image based on the results of the first determination process and the second determination process. For example, the video processing unit 188 selects the image that is most suitable for the measurement process. The video processing unit 188 associates information that identifies the selected image with the selected image (third association step). For example, the video processing unit 188 adds the information to the video file by embedding the information in the video file.

[0251] The video processing unit 188 associates link information for associating the first image and the second image with at least one of the first image and the second image (fourth associating step). The video processing unit 188 may associate the link information with only the first image or the second image. Alternatively, the video processing unit 188 may associate the link information with the first image and the second image. For example, the video processing unit 188 adds the link information to the video file by embedding the link information in the video file. The video processing unit 188 may embed the link information in at least one of the first frame and the second frame in the video.

[0252] Image processing in the sixth embodiment will be described with reference to Fig. 25 and Fig. 26. Fig. 25 and Fig. 26 show the procedure of image processing. The image processing shown in Fig. 25 and Fig. 26 is a modified example of the image processing shown in Fig. 15. The image processing shown in Fig. 25 and Fig. 26 may be combined with the image processing shown in Fig. 9, Fig. 19, or Fig. 20. Description of the same processing as the processing shown in Fig. 9 or Fig. 15 will be omitted.

[0253] For example, the user operates the operation unit 4 to input a video recording instruction to the endoscope device 1. If the display unit 5 is configured as a touch panel and a record button is displayed on the display unit 5, the user touches the record button to input a video recording instruction to the endoscope device 1. The information receiving unit 185 receives the video recording instruction input to the endoscope device 1 (step S141). After step S141, step S101 is executed.

[0254] After step S101, the video processing unit 188 starts generating a video file. After step S104, the video processing unit 188 embeds first execution information indicating that the first determination process has been executed into the video file (step S142). The first execution information is associated with a first image. Step S142 corresponds to a first association step.

[0255] After step S142, the video processing unit 188 embeds first result information indicating the result of the first determination process into the video file (step S143). After step S143, step S105 is executed.

[0256] The first result information is associated with the first image. For example, the video processing unit 188 embeds the first result information for each pixel of the first image in a video file. The first result information may indicate the reliability of the measurement process. For example, the reliability is expressed as a percentage.

[0257] After step S108, the video processing unit 188 embeds second execution information, which indicates that the second determination process has been executed, into the video file (step S144). The second execution information is associated with the second image. Step S144 corresponds to a second associating step.

[0258] After step S144, the moving image processing unit 188 embeds second result information indicating the result of the second determination process in the moving image file (step S145).

[0259] The second result information is associated with the second image. For example, the video processing unit 188 embeds the second result information for each pixel of the second image in a video file. The second result information may indicate the reliability of the measurement process. For example, the reliability is expressed as a percentage.

[0260] After step S145, the video processing unit 188 selects the image that is most suitable for measurement processing from the first image and the second image. The video processing unit 188 embeds information indicating the selected image in the video file (step S146). Step S146 corresponds to the third associating step.

[0261] The video processing unit 188 selects the image that is most suitable for the measurement process from the first image and the second image based on the first result information and the second result information. Any method may be used to select the image that is most suitable for the measurement process. For example, the video processing unit 188 may select the image that is determined to be suitable for the measurement process over the widest range. Alternatively, the video processing unit 188 may select the image that has the largest sum of the reliability of the measurement process for all pixels.

[0262] After step S146, the video processing unit 188 embeds link information relating to a predetermined area in the second image into the video file. The predetermined area is the area of the second image that the determination unit 183 determined in the first determination process to be suitable for measurement processing. The link information indicates the area of the first image for which the first determination process was performed (step S147). Step S147 corresponds to the fourth association step. When step S147 is performed, the image processing ends.

[0263] 17, the first region RG22 in the second image IM22 includes the region that the determination unit 183 determined in the first determination process to be suitable for measurement processing. The video processing unit 188 associates link information with the region. The link information indicates the first region RG21 in the first image IM21 shown in FIG.

[0264] By using the link information, the endoscope device 1 can identify an image suitable for measurement processing and a specific area within that image. The first determination process is performed in the specified area, but the second determination process is not performed in the specified area. When the endoscope device 1 performs measurement processing in the specified area, it is desirable for the endoscope device 1 to use the first image rather than the second image. The endoscope device 1 embeds link information in the video file for using the first image instead of the second image in the measurement processing.

[0265] Details of steps S142, S144, S146, and S147 will be described with reference to Fig. 27. A moving image file Vd is shown at the top of Fig. 27. The moving image file Vd includes a first image Ia, a second image Ib, a third image Ic, and a fourth image Id. These images are used to perform the determination process.

[0266] The video file includes information corresponding to each of marks Ta, Tb, Tc, and Td. Marks Ta, Tb, Tc, and Td are associated with a first image Ia, a second image Ib, a third image Ic, and a fourth image Id, respectively. In step S142, the video processing unit 188 associates the first execution information corresponding to mark Ta with the first image Ia. In step S144, the video processing unit 188 associates the second execution information corresponding to mark Tb with the second image Ib. Similarly, the video processing unit 188 associates the third execution information corresponding to mark Tc with the third image Ic, and associates the fourth execution information corresponding to mark Td with the fourth image Id. The user can easily find the image used in the measurement process by referring to each mark.

[0267] The mark Td indicates the fourth image Id that is most suitable for the measurement process among the four images for which the determination process has been performed. The video processing unit 188 associates information corresponding to the mark Td with the fourth image Id in a process similar to that of step S146. The mark Td is displayed in a state different from the states of the marks Ta, Tb, and Tc. For example, the mark Td is displayed in a color different from the colors of the marks Ta, Tb, and Tc.

[0268] The regions in each of the first image Ia, the second image Ib, the third image Ic, and the fourth image Id are shown in the lower part of Fig. 27. The first image Ia includes a first region Raa in which the first determination process was performed.

[0269] The second image Ib includes a second region Rbb and a first region Rba. A second determination process is performed in the second region Rbb. The first region Rba corresponds to the first region Raa of the first image Ia. The first region Rba includes an area that the determination unit 183 determined in the first determination process to be unsuitable for measurement processing. The second determination process is also performed in that area of the first region Rba.

[0270] In step S147, the video processing unit 188 associates the link information with the second image Ib. The link information of the second image Ib associates the first region Raa of the first image Ia with the first region Rba of the second image Ib. Specifically, the link information of the second image Ib associates a part of the first region Raa with a part of the first region Rba. In the first determination process, the determination unit 183 determines that the part of the first region Raa is suitable for measurement processing. The determination unit 183 does not perform the second determination process on the part of the first region Rba.

[0271] The third image Ic includes a third region Rcc, a second region Rcb, and a first region Rca. A third determination process is performed on the third region Rcc. The second region Rcb corresponds to the second region Rbb of the second image Ib. The second region Rcb includes a region that the determination unit 183 determined in the second determination process to be unsuitable for measurement processing. The third determination process is also performed on that region of the second region Rcb. The first region Rca corresponds to each of the first region Raa of the first image Ia and the first region Rba of the second image Ib. The first region Rca includes a region that the determination unit 183 determined in the first determination process and the second determination process to be unsuitable for measurement processing. The third determination process is also performed on that region of the first region Rca.

[0272] The video processing unit 188 associates the link information with the third image Ic. The link information of the third image Ic associates the first region Raa of the first image Ia with the first region Rca of the third image Ic. Specifically, the link information of the third image Ic associates a part of the first region Raa with a part of the first region Rca. In the first determination process, the determination unit 183 determines that the part of the first region Raa is suitable for measurement processing. The determination unit 183 does not perform the third determination process on the part of the first region Rca.

[0273] Furthermore, the link information of the third image Ic associates the second region Rbb of the second image Ib with the second region Rcb of the third image Ic. Specifically, the link information of the third image Ic associates a portion of the second region Rbb with a portion of the second region Rcb. In the second determination process, the determination unit 183 determines that the portion of the second region Rbb is suitable for measurement processing. The determination unit 183 does not perform the third determination process on the portion of the second region Rcb.

[0274] The fourth image Id includes a fourth region Rdd, a third region Rdc, a second region Rdb, and a first region Rda. A fourth determination process is performed on the fourth region Rdd. The third region Rdc corresponds to the third region Rcc of the third image Ic. The third region Rdc includes a region that the determination unit 183 determined in the third determination process to be unsuitable for measurement processing. The fourth determination process is also performed on that region of the third region Rdc. The second region Rdb corresponds to the second region Rbb of the second image Ib. The second region Rdb includes a region that the determination unit 183 determined in the second determination process and the third determination process to be unsuitable for measurement processing. The fourth determination process is also performed on that region of the second region Rdb. The first region Rda corresponds to the first region Raa in the first image Ia, the first region Rba in the second image Ib, and the first region Rca in the third image Ic. The first region Rda includes regions that the determination unit 183 determined in the first determination process, the second determination process, and the third determination process to be unsuitable for measurement processing. The fourth determination process is also performed on the region of the first region Rda.

[0275] The video processing unit 188 associates the link information with the fourth image Id. The link information of the fourth image Id associates the first region Raa of the first image Ia with the first region Rda of the fourth image Id. Specifically, the link information of the fourth image Id associates a part of the first region Raa with a part of the first region Rda. In the first determination process, the determination unit 183 determines that the part of the first region Raa is suitable for measurement processing. The determination unit 183 does not perform the fourth determination process on the part of the first region Rda.

[0276] Furthermore, the link information of the fourth image Id associates the second region Rbb of the second image Ib with the second region Rdb of the fourth image Id. Specifically, the link information of the fourth image Id associates a portion of the second region Rbb with a portion of the second region Rdb. In the second determination process, the determination unit 183 determines that the portion of the second region Rbb is suitable for measurement processing. The determination unit 183 does not perform the fourth determination process on the portion of the second region Rdb.

[0277] Furthermore, the link information of the fourth image Id associates the third region Rcc of the third image Ic with the third region Rdc of the fourth image Id. Specifically, the link information of the fourth image Id associates a part of the third region Rcc with a part of the third region Rdc. In the third determination process, the determination unit 183 determines that the part of the third region Rcc is suitable for measurement processing. The determination unit 183 does not perform the fourth determination process on the part of the third region Rdc.

[0278] In step S142, the video processing unit 188 may record text information indicating the first execution information on a still image corresponding to the first image. In step S144, the video processing unit 188 may record text information indicating the second execution information on a still image corresponding to the second image.

[0279] In step S143, video processing unit 188 may record text information indicating the result of the first determination process on a still image corresponding to the first image. In step S145, video processing unit 188 may record text information indicating the result of the second determination process on a still image corresponding to the second image.

[0280] In step S146, video processing unit 188 may record text information indicating the image most suitable for measurement processing on the still image corresponding to that image. In step S147, video processing unit 188 may record text information indicating link information on the still image corresponding to the first image or the second image.

[0281] The CPU 18c may have a motion determination unit 187 shown in FIG. 18. If the motion determination unit 187 determines that the amount of motion of the subject between the first image and the second image is smaller than a predetermined amount, the image acquisition unit 181 may acquire a new second image. The predetermined amount corresponds to a minute amount of motion. The previously acquired second image is not used in the processing after step S105. If the motion determination unit 187 determines that the amount of motion of the subject between the first image and the second image is equal to or greater than a predetermined amount, the processing after step S105 may be executed. This makes it possible to prevent the image of the link destination indicated by the link information from being different for each pixel.

[0282] Each aspect of the present invention may include the following modifications. In a first associating step (step S142), the video processing unit 188 associates first execution information with a first image. The first execution information indicates that a first determination step (step S103) has been performed. In a second associating step (step S144), the video processing unit 188 associates second execution information with a second image. The second execution information indicates that a second determination step (step S107) has been performed.

[0283] Each aspect of the present invention may include the following modifications. Each of the first image and the second image is one of two or more frames constituting a moving image. In a first associating step (step S142), the moving image processing unit 188 adds first execution information to the moving image. In a second associating step (step S144), the moving image processing unit 188 adds second execution information to the moving image.

[0284] Each aspect of the present invention may include the following modifications. In a third associating step (step S146), the video processing unit 188 selects either the first image or the second image based on the results of the first determination step (step S103) and the second determination step (step S107). In the third associating step (step S146), the video processing unit 188 associates information identifying the selected image with the selected image.

[0285] Each aspect of the present invention may include the following modifications: In a fourth associating step (step S147), the video processing unit 188 associates information for associating the first image and the second image with the second image.

[0286] Each aspect of the present invention may include the following modifications. Each of the first image and the second image is one of two or more frames that make up a moving image. In a fourth associating step (step S147), the moving image processing unit 188 adds information for associating the first image and the second image with each other to the moving image.

[0287] In the sixth embodiment, the endoscope device 1 associates various pieces of information with the first image or the second image. When the endoscope device 1 performs a measurement process, the user can easily find the image used in the measurement process. This improves the efficiency of the user's work in the measurement process.

[0288] (Modification of the sixth embodiment) A modified example of the sixth embodiment of the present invention will be described, in which a procedure for measurement processing using a video file containing link information will be described.

[0289] An example in which the PC 41 shown in Fig. 4 executes the measurement process will be described below. The software may be prepared in a cloud environment, and the measurement process may be executed by that software. The software may be installed in an embedded device, and the measurement process may be executed by that software. The environment in which the measurement process is executed is not limited to the above example.

[0290] FIG. 28 shows the configuration of a PC 41. For example, the PC 41 is a desktop PC. The PC 41 may be a portable laptop PC or a tablet terminal. The PC 41 may be a computer system that operates on the cloud. The PC 41 shown in FIG. 28 includes a CPU 43, an operation unit 44, a display unit 45, a communication unit 46, and a memory 47.

[0291] The operation unit 44 is a user interface. For example, the operation unit 44 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel. The operation unit 44 accepts user operations on the PC 41. The user can input various information to the PC 41 by operating the operation unit 44. The operation unit 44 accepts information input by the user and outputs the information to the CPU 43.

[0292] The display unit 45 has a display screen and displays an image of the object in the inspection object on the display screen. The display unit 45 is a monitor (display) such as an LCD. The display unit 45 may be a touch panel. In this case, the operation unit 44 and the display unit 45 are integrated.

[0293] The communication unit 46 communicates with an external device such as the endoscope device 1. For example, the communication unit 46 is connected to the external device via a cable or wirelessly. The communication between the communication unit 46 and the external device may be performed via a LAN (Local Area Network) or the Internet.

[0294] The memory 47 is a volatile or non-volatile memory. For example, the memory 47 is at least one of a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory. The memory 47 stores images and the like.

[0295] The CPU 43 controls the operation of the PC 41 .

[0296] The measurement process in the modified example of the sixth embodiment will be described with reference to Fig. 29. Fig. 29 shows the procedure of the measurement process.

[0297] Before the measurement process is started, the communication unit 46 receives a moving image file from the endoscope device 1. The memory 47 stores the received moving image file.

[0298] When the measurement process is started, the CPU 43 reads out a moving image file from the memory 47 (step S201).

[0299] The CPU 43 displays the video file on the display unit 45. The user operates the operation unit 44 to input information indicating a specific image (frame) to be used in the measurement process to the PC 41. The CPU 43 receives the information and selects the image indicated by the information (step S202).

[0300] When the video file is displayed on the display unit 45, the video file Vd shown in FIG. 27 is displayed on the display unit 45. Also, marks Ta, Tb, Tc, and Td shown in FIG. 27 are displayed on the display unit 45. The user inputs information indicating the image selected by the user into the PC 41 by clicking one of marks Ta, Tb, Tc, and Td. The following describes an example in which the user clicks mark Td and selects a fourth image Id. Mark Td indicates that the fourth image Id is most suitable for measurement processing. The processing procedure described below remains the same even if the user selects an image other than the fourth image Id.

[0301] After step S202, the CPU 43 displays the image selected by the user on the display unit 45 (step S203).

[0302] FIG. 30 shows an example of an image displayed on the display screen 45a of the display unit 45. The CPU 43 displays a moving image file Vd on the display screen 45a. The moving image file Vd includes a first image Ia, a second image Ib, a third image Ic, and a fourth image Id, similar to the moving image file Vd shown in FIG. 27. The CPU 43 displays a fourth image IM41 on the display screen 45a. The fourth image IM41 is the same as the fourth image Id. The fourth image IM41 is one of two images included in a stereo image acquired by the imaging element 28. The other of the two images included in the stereo image is not displayed on the display screen 45a but is recorded in the memory 47.

[0303] After the image is displayed on the display unit 45 in step S203, the user operates the operation unit 44 to input two or more measurement points into the PC 41. The PC 41 may measure the length between two measurement points. The PC 41 may measure the area of a region defined by three or more measurement points. The PC 41 may measure the distance between a line defined by two measurement points and one measurement point. In the following example, an example will be described in which the PC 41 measures the length between two measurement points. Even when the PC 41 sets three or more measurement points, the procedure until the PC 41 obtains the measurement result is the same as the procedure of the measurement process in which the PC 41 sets two measurement points.

[0304] When the user inputs a first measurement point into the PC 41, the CPU 43 sets the first measurement point on the image selected in step S202 (step S204). When the user inputs a second measurement point into the PC 41, the CPU 43 sets the second measurement point on the image selected in step S202 (step S205). In the example shown in Fig. 30, the CPU 43 displays the first measurement point P41 and the second measurement point P42 on the display screen 45a.

[0305] After step S205, the CPU 43 searches the video file for an image containing two measurement points. If two or more images containing two measurement points are identified, the CPU 43 calculates the reliability of the measurement process at each measurement point in the two or more images. The CPU 43 selects the image with the highest reliability (step S206). This image is most suitable for the measurement process.

[0306] In step S206, the CPU 43 selects one image that includes both of the two measurement points. Step S206 will be described in detail below.

[0307] For example, the CPU 43 identifies an image located at the center of a first search range. The image includes a first measurement point. When the image processing shown in FIGS. 25 and 26 is executed, the determination unit 183 executes a first determination process or a second determination process at the first measurement point of the image, and determines that the image is suitable for measurement processing. Link information for the image is embedded in the video file. Furthermore, the CPU 43 identifies an image located at the center of a second search range. The image includes a second measurement point. When the image processing shown in FIGS. 25 and 26 is executed, the determination unit 183 executes a first determination process or a second determination process at the second measurement point of the image, and determines that the image is suitable for measurement processing. Link information for the image is embedded in the video file.

[0308] The fourth image IM41 shown in FIG. 30 includes a first region Rda, a second region Rdb, a third region Rdc, and a fourth region Rdd. A first measurement point P41 is included in the first region Rda. A second measurement point P42 is included in the third region Rdc. As shown in FIG. 27, the first region Rda is associated with a first region Raa in the first image Ia by link information. The first image Ia is located at the center of the first search range. As shown in FIG. 27, the third region Rdc is associated with a third region Rcc in the third image Ic by link information. The third image Ic is located at the center of the second search range.

[0309] For example, the first search range includes an image at the center of the first search range (center image), a predetermined number of images before the center image, and a predetermined number of images after the center image. For example, the second search range includes an image at the center of the second search range (center image), a predetermined number of images before the center image, and a predetermined number of images after the center image. For example, the predetermined number is 100.

[0310] The image at the center of the second search range may differ from the image at the center of the first search range. In this case, the CPU 43 uses the area common to the first and second search ranges (overlapping area) as the search range. The CPU 43 searches for an image that includes the first measurement point and the second measurement point within the search range.

[0311] The method of searching an image including two measurement points is not limited to the above example. In the above example, the CPU 43 uses a range common to the first search range and the second search range as the search range. The search range is obtained as the logical product of the first search range and the second search range. The CPU 43 may also use a search range that includes at least one of the first measurement point and the second measurement point. The search range is obtained as the logical sum of the first search range and the second search range.

[0312] The CPU 43 may use various indices used in the first determination process and the second determination process as indices for calculating the reliability of the measurement process. The CPU 43 may calculate the average value of the reliability value at the first measurement point and the reliability value at the second measurement point. The CPU 43 may use this average value as the final reliability. The CPU 43 may use various statistical values such as a minimum or maximum value instead of the average value. The CPU 43 calculates the reliability of the measurement process for each image included in the search range. The CPU 43 selects the image that is most suitable for the measurement process based on the reliability.

[0313] The computational resources of the endoscope device 1 are limited, and there is a strong demand for reducing processing time. Therefore, the range in which the endoscope device 1 performs the determination process during an examination is limited. In other words, the endoscope device 1 does not perform the determination process again in an area that has been determined to be suitable for measurement processing. The computational resources of the PC 41 are large, and there is a low demand for reducing processing time. Therefore, the PC 41 can calculate the reliability of the measurement process in an image that includes the first measurement point and the second measurement point. There is no need for the PC 41 to calculate the reliability of the measurement process.

[0314] After step S206, the CPU 43 executes a matching process by using the images selected in step S206 to detect corresponding points for each measurement point (step S207). Each measurement point is located in one of the two images included in the stereo image, and the corresponding point is located in the other of the two images. The CPU 43 detects a first corresponding point corresponding to the first measurement point, and detects a second corresponding point corresponding to the second measurement point.

[0315] After step S207, the CPU 43 performs triangulation using each measurement point and each corresponding point. The CPU 43 calculates 3D coordinates corresponding to each measurement point (step S208). The CPU 43 calculates first 3D coordinates corresponding to the first measurement point and calculates second 3D coordinates corresponding to the second measurement point.

[0316] After step S208, the CPU 43 calculates the length (Euclidean distance) between the point having the first 3D coordinates and the point having the second 3D coordinates, thereby measuring the length of the subject (step S209).

[0317] After step S209, the CPU 43 displays the length calculated in step S209 as the measurement result on the display unit 45 (step S210). When step S210 is executed, the measurement process ends.

[0318] In the modification of the sixth embodiment, the PC 41 can detect images suitable for measurement processing by using link information embedded in a video file, thereby reducing the time required for measurement processing.

[0319] (Seventh embodiment) A seventh embodiment of the present invention will be described. The endoscope device 1 executes a determination process similar to the second embodiment and displays the result of the determination process. The endoscope device 1 also executes a 3D reconstruction process to calculate the 3D shape of the subject. The endoscope device 1 displays an image of the 3D shape.

[0320] The CPU 18 shown in Fig. 8 is changed to a CPU 18d shown in Fig. 31. Fig. 31 shows the functional configuration of the CPU 18d. The functions of the CPU 18d include a control unit 180, an image acquisition unit 181, a region setting unit 182, a determination unit 183, a display control unit 184, an information reception unit 185, a region search unit 186, and a 3D shape calculation unit 189. At least one of the blocks shown in Fig. 31 may be configured with a circuit different from that of the CPU 18d. Description of the same configuration as that shown in Fig. 8 will be omitted.

[0321] Each unit shown in Figure 31 may be configured with at least one of a processor and a logic circuit. Each unit shown in Figure 31 may include one or more processors. Each unit shown in Figure 31 may include one or more logic circuits.

[0322] The 3D shape calculation unit 189 (processing unit) calculates the 3D shape of the subject by using two or more images including at least one of the first image and the second image (calculation step). The display control unit 184 displays an image of the 3D shape on the display unit 5 (image display step).

[0323] Image processing in the seventh embodiment will be described with reference to Fig. 32. Fig. 32 shows the procedure of image processing. The image processing shown in Fig. 32 is a modified example of the image processing shown in Fig. 15. The image processing shown in Fig. 32 may be combined with the image processing shown in Fig. 9, Fig. 19, Fig. 20, Fig. 25, or Fig. 26. Description of the same processing as the processing shown in Fig. 9 or Fig. 15 will be omitted.

[0324] After step S108, the 3D shape calculation unit 189 calculates the 3D shape of the subject (step S151). Step S151 corresponds to a calculation step.

[0325] For example, the 3D shape calculation unit 189 uses a pair of two images included in a stereo image. One of the two images is the first image acquired in step S101. Alternatively, the 3D shape calculation unit 189 uses the first image and one or more images acquired before the first image. The 3D shape calculation unit 189 uses a first region in the first image and a region in another image that corresponds to the first region.

[0326] Alternatively, one of the two images included in the stereo image is the second image acquired in step S105. Alternatively, the 3D shape calculation unit 189 uses the second image and one or more images acquired before the second image. The one or more images may include the first image. The 3D shape calculation unit 189 uses a second region in the second image and a region in another image that corresponds to the second region. The 3D shape calculation unit 189 may use a first region in the second image and a region in another image that corresponds to the first region. The first region in the second image corresponds to the first region in the first image.

[0327] After step S151, the display control unit 184 displays an image of the 3D shape calculated in step S151 on the display unit 5 (step S152). Step S152 corresponds to an image display step. When step S152 is executed, the image processing ends.

[0328] 33 shows the second image IM51 displayed on the display unit 5 in step S152. The display control unit 184 displays the second image IM51 on the display unit 5. The display control unit 184 also displays an area RG51 on the second image IM51. The area RG51 includes an area where the first determination process was performed and an area where the second determination process was performed. For example, the determination unit 183 determines that the area RG51 is suitable for a predetermined process.

[0329] The 3D shape calculation unit 189 calculates the 3D shape of the subject corresponding to the region RG51. The display control unit 184 displays the region RG52 on the second image IM51. The display control unit 184 displays an image of the 3D shape of the subject in the region RG52.

[0330] FIG. 34 shows a third image IM52 displayed on the display unit 5. After the second image IM51 is acquired, the third image IM52 is acquired. The display control unit 184 displays the third image IM52 on the display unit 5. The display control unit 184 also displays regions RG53 and RG54 on the third image IM52. The regions RG53 and RG54 include a region where the first determination process was performed, a region where the second determination process was performed, and a region where the third determination process was performed. For example, the determination unit 183 determines that the region RG53 is suitable for the predetermined process. The determination unit 183 determines that the region RG54 is not suitable for the predetermined process.

[0331] The 3D shape calculation unit 189 calculates the 3D shapes of the subject corresponding to the regions RG53 and RG54. The display control unit 184 displays the region RG55 on the third image IM52. The display control unit 184 displays an image of the 3D shape of the subject in the region RG55.

[0332] FIG. 35 shows a fourth image IM53 displayed on the display unit 5. After the third image IM52 is acquired, the fourth image IM53 is acquired. The display control unit 184 displays the fourth image IM53 on the display unit 5. The display control unit 184 also displays an area RG56 on the fourth image IM53. The area RG56 includes an area where the first determination process was performed, an area where the second determination process was performed, an area where the third determination process was performed, and an area where the fourth determination process was performed. For example, the determination unit 183 determines that the area RG56 is suitable for a predetermined process.

[0333] The 3D shape calculation unit 189 calculates the 3D shape of the subject corresponding to the region RG56. The display control unit 184 displays the region RG57 on the fourth image IM53. The display control unit 184 displays an image of the 3D shape of the subject in the region RG57.

[0334] Each aspect of the present invention may include the following modifications. In a calculation step (step S151), the 3D shape calculation unit 189 calculates the 3D shape of the subject by using two or more images including at least one of a first image and a second image. In an image display step (step S152), the display control unit 184 displays an image of the 3D shape on the display unit 5 (display).

[0335] In the seventh embodiment, the endoscope device 1 calculates the 3D shape of the subject and displays the 3D shape. The user can check the 3D shape of the subject in addition to the result of the determination process. This increases the reliability of the determination result when the user determines whether the image is suitable for measurement.

[0336] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims. [Explanation of symbols]

[0337] 1 Endoscopic device 2 Insertion section 3 Main body 4,44 Control unit 5,45 Display section 7 Image Processing Device 8 Endoscope Unit 9 CCU 10 Control device 12 Video signal processing circuit 13 ROM 14 RAM 15 Card Interface 16 External device interface 17 Control Interface 18, 18a, 18b, 18c, 18d, 43 CPU 20 Tip 28 Image sensor 30 Stereo Optical Adapter 41 PC 42 Memory Card 46 Communications Department 47 memory 50 fixing ring 51 First illumination optical system 52 Second illumination optical system 53 First objective optical system 54 Second Objective Optical System 70 Setting section 71,183 Judgment Department 72,184 Display control unit 180 Control Unit 181 Image acquisition unit 182 Area setting section 185 Information Reception Department 186 Area Search Department 187 Motion Judgment Unit 188 Video Processing Unit 189 3D shape calculation section

Claims

1. An image processing method for repeatedly determining whether an image is suitable for a predetermined process, comprising: a first setting step in which a setting unit sets a first region in a first image; a first determination step in which a determination unit determines whether the first region is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of the subject; a second setting step in which the setting unit sets a second region in a second image acquired after the first image, the second region being smaller than the entire region of the second image and not including at least a part of the first region; a second determination step in which the determination unit determines whether the second region is suitable for the predetermined process; an information display step in which a display control unit displays, on a display, information indicating whether at least a part of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process; An image processing method comprising:

2. The method further includes a second information display step in which the display control unit displays information indicating whether the first area is suitable for the predetermined process on the display before the information display step is executed. The image processing method according to claim 1 .

3. In the first setting step, the setting unit sets the first area including an area in which a first subject is captured in the first image; In the second setting step, the setting unit sets the second area including an area in which a second subject different from the first subject is captured in the second image. The image processing method according to claim 1 .

4. The method further includes a search step in which a search unit searches for a subject area in which the first subject is captured in the second image; In the second setting step, the setting unit sets the second area including an area different from the subject area in the second image. The image processing method according to claim 3 .

5. In the first setting step, the setting unit sets the first area including an area in which a first subject is captured in the first image; In the second setting step, the setting unit sets the second area including at least a part of an area in which the first subject is captured in the second image. The image processing method according to claim 1 .

6. The method further includes a detection step in which a detection unit detects an amount of change in composition between the first image and the second image before the second determination step is executed. The image processing method according to claim 1 .

7. The information display step is executed only when the amount of change is smaller than a predetermined amount. The image processing method according to claim 6.

8. The method further includes a generating step of generating information of a determination region in each of the first image and the second image by a generating unit; After the information on the determination region is generated, the setting unit sets the first region as the determination region in the first image in the first setting step; In the second setting step, the setting unit sets the second region as the determination region in the second image. The image processing method according to claim 1 .

9. Each of the first image and the second image is generated based on an optical image generated through a stereo optical system. The image processing method according to claim 1 .

10. In the first determination step, the determination unit uses at least one of the three-dimensional shape of the object in the first region acquired by executing a process of calculating the three-dimensional shape of the object in the first region, the distance between the object in the first region and the camera, the brightness of the first region, the amount of blur in the first region, and the intensity of the texture in the first region. The image processing method according to claim 1 .

11. In the second determination step, the determination unit uses at least one of the three-dimensional shape of the object in the second region acquired by executing a process for calculating the three-dimensional shape of the object in the second region, the distance between the object in the second region and the camera, the brightness of the second region, the amount of blur in the second region, and the intensity of the texture in the second region. The image processing method according to claim 1 .

12. the first image is generated based on an optical image generated through a stereo optical system; In the first determination step, the determination unit uses at least one of a correlation value acquired by performing stereo matching processing on the first region and a parallax amount in the first region. The image processing method according to claim 1 .

13. the second image is generated based on an optical image generated through a stereo optical system; In the second determination step, the determination unit uses at least one of a correlation value acquired by performing stereo matching processing on the second region and a parallax amount in the second region. The image processing method according to claim 1 .

14. a calculation step in which a processing unit calculates a three-dimensional shape of the subject by using two or more images including at least one of the first image and the second image; an image display step in which the display control unit displays an image of the three-dimensional shape on the display; 2. The image processing method of claim 1, further comprising:

15. The determination unit uses two or more images including the first image in the first determination step. The image processing method according to claim 1 .

16. The determination unit uses two or more images including the second image in the second determination step. The image processing method according to claim 1 .

17. a first associating step in which a processing unit associates first execution information indicating that the first determination step has been executed with the first image; a second associating step in which the processing unit associates second execution information indicating that the second determination step has been executed with the second image; 2. The image processing method of claim 1, further comprising:

18. each of the first image and the second image is one of two or more frames constituting a moving image; In the first associating step, the processing unit adds the first execution information to the video; In the second associating step, the processing unit adds the second execution information to the video. The image processing method according to claim 17.

19. The method further includes a third associating step in which a processing unit selects one of the first image and the second image based on a result of the first determination step and a result of the second determination step, and associates information identifying the selected image with the selected image. The image processing method according to claim 17.

20. The method further includes a fourth associating step in which the processing unit associates information for associating the first image and the second image with at least one of the first image and the second image. The image processing method according to claim 17.

21. each of the first image and the second image is one of two or more frames constituting a moving image; In the fourth associating step, the processing unit adds the information for associating the first image and the second image with each other to the video.

21. The image processing method according to claim 20.

22. The image processing method of claim 1, wherein the first area is smaller than the entire area of the first image.

23. An image processing device that repeatedly determines whether an image is suitable for a predetermined process, comprising: a setting unit that sets a first region in a first image and a second region in a second image acquired after the first image, the second region being smaller than the entire region of the second image and not including at least a portion of the first region; a determination unit that determines whether each of the first area and the second area is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of an object; a display control unit that displays, on a display, information indicating whether at least a part of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process; An image processing device having:

24. A program for causing a computer to execute image processing for repeatedly determining whether an image is suitable for a predetermined process, comprising: a first setting step of setting a first region in a first image; a first determination step of determining whether the first region is suitable for a predetermined process, the predetermined process being a process of calculating three-dimensional coordinates of the subject; a second setting step of setting a second region in a second image acquired after the first image, the second region being smaller than the entire region of the second image and not including at least a portion of the first region; a second determination step of determining whether the second region is suitable for the predetermined process; an information display step of displaying on a display information indicating whether at least a part of the first area is suitable for the predetermined process and information indicating whether the second area is suitable for the predetermined process; A program that causes a computer to execute the following.

Images