Image processing device, image processing method, and program

Abstract

This image processing device comprises a state determination unit, a first image processing unit, and a second image processing unit. The state determination unit determines the state of an endoscope. When the imaging field of view of an imaging unit of the endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region and the endoscope is inserted into the inside of an object under inspection, the first image processing unit executes image processing for making the visibility of an image of the second region lower than the visibility of an image of the first region. When the endoscope is inserted into the inside of the object under inspection, the second image processing unit executes image processing for making detection performance for the object in the image of the second region the same as or higher than detection performance for the object in the image of the first region.

Description

Image processing device, image processing method, and program

[0001] The present invention relates to an image processing device, an image processing method, and a program.

[0002] An endoscope is an examination device equipped with a very small camera at its tip, which is inserted into an object to be examined, such as a lumen or a narrow opening. The user observes the object by looking at images generated by the imaging unit at the tip of the endoscope. The user moves the endoscope in an insertion direction toward the back of the object to be examined, or in a removal direction toward the outside of the object to be examined. When inserting the endoscope, the images output from the imaging unit are used to assist in the insertion of the endoscope. In some cases, such as colonoscopy, a detailed check of the interior of the object to be examined is sometimes performed when the endoscope is removed. This is done to first ensure that the endoscope reaches the deepest position of the object to prevent overlooking the entire area.

[0003] The role of images during endoscope insertion may differ from the role of images during examination or removal. When an endoscope is inserted into an examination target, images are effectively used to assist in the insertion of the endoscope. At this time, the user looks at the center of the image generated by the imaging unit to confirm that the endoscope is aligned with the lumen hole. The user then moves the endoscope in the direction confirmed by the image. In such a situation, if the imaging field of view is wider than necessary, the amount of information in the peripheral areas of the image increases. This can cause stress for users who focus on the center of the image. However, lesions or other abnormalities may be captured in the peripheral areas of the image, and users who focus on the center of the image may miss the lesions or other abnormalities.

[0004] Patent Literature 1 discloses a technology that prevents overlooking of an object by linking multiple cameras installed in a vehicle. In this technology, for example, a first object tracking unit tracks an object in an image, and when tracking by the first object tracking unit becomes impossible or inappropriate, a second object tracking unit tracks the object, thereby switching the range of the image according to the purpose.

[0005] Japanese Patent Application Publication No. 2021-051470

[0006] Although the technology disclosed in Patent Document 1 switches between images, it is difficult to apply this technology to endoscopy, and it is difficult for the user to intuitively recognize the object.

[0007] As mentioned above, the role of the image when inserting the endoscope may differ from the role of the image when performing the examination or when removing the endoscope. When inserting the endoscope, it is desirable to ensure visibility in the center of the image and to detect lesions and the like in the peripheral area of ​​the image with high accuracy.

[0008] The present invention aims to provide an image processing device, an image processing method, and a program that can perform image processing appropriate to the role of each region of an image when a small device is used that can easily observe the inside of a thin lumen or a body cavity inserted through a narrow path.

[0009] According to a first aspect of the present invention, an image processing device includes a state determination unit, a first image processing unit, and a second image processing unit. The state determination unit determines whether the state of an endoscope is a first state in which the endoscope is inserted into an object of examination and is advancing deeper into the object of examination, or a second state other than the first state. The first image processing unit divides the imaging field of view of the imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region, and, if the state of the endoscope is determined to be the first state, performs image processing to reduce the visibility of the image of the second region generated by the imaging unit compared to the visibility of the image of the first region generated by the imaging unit, thereby generating an image for display. If the state of the endoscope is determined to be the first state, the second image processing unit performs image processing to increase the object detection performance in the image of the second region to the same level as or higher than the object detection performance in the image of the first region, thereby generating an image for object detection.

[0010] According to the second aspect of the present invention, in the first aspect, when it is determined that the state of the endoscope is the second state, the first image processing unit may not perform the image processing that reduces the visibility of the image of the second region below the visibility of the image of the first region.

[0011] According to a third aspect of the present invention, in the first aspect, the image processing device may include a detection unit and a display control unit. When it is determined that the state of the endoscope is the first state, the detection unit detects an object from the image of the second region in the image for object detection. When it is determined that the state of the endoscope is the first state, the display control unit causes a display unit to display a detection result indicating that the object has been detected together with the image of the first region on which the image processing has been performed.

[0012] According to a fourth aspect of the present invention, in the third aspect, the display control unit may cause the display unit to display position information indicating the position of the object in the image of the second area as the detection result.

[0013] According to a fifth aspect of the present invention, in the first aspect, the image processing device may include a detection unit and a display control unit. When the state of the endoscope is determined to be the first state, the detection unit detects an object from an image of the second region in the image for object detection, and stores, in a recording medium, first position information indicating the position of the object in the image of the second region and second position information indicating the position of the object in the inspection object. When the state of the endoscope is determined to be the second state and the position of the endoscope matches the position indicated by the second position information, the display control unit displays the first position information together with the image generated by the imaging unit on a display unit.

[0014] According to a sixth aspect of the present invention, in the first aspect, when it is determined that the state of the endoscope is the first state, the first image processing unit may delete the image of the second region.

[0015] According to a seventh aspect of the present invention, in the sixth aspect, the image processing device may have a detection unit that detects an object from an image of the second region in the image for object detection when the state of the endoscope is determined to be the first state. If the object is not detected, the first image processing unit may delete the image of the second region. If the object is detected, the first image processing unit may not delete the image of the second region.

[0016] According to an eighth aspect of the present invention, in the sixth aspect, the image processing device may further include a detection unit that, when the state of the endoscope is determined to be the first state, detects an object from an image of the second region in the image for object detection. When the object approaches the center of the imaging field of view in a plurality of images successively generated by the imaging unit, the first region in the plurality of images may gradually increase in size and the second region in the plurality of images may gradually decrease in size.

[0017] According to a ninth aspect of the present invention, in the first aspect, when it is determined that the state of the endoscope is the first state, the first image processing unit may perform a thinning process on the image of the second region.

[0018] According to a tenth aspect of the present invention, in the first aspect, the image processing device may have a correction unit that corrects the image generated by the imaging unit based on the optical characteristics of the endoscope.

[0019] According to an eleventh aspect of the present invention, an image processing device includes a first image processing unit, a detection unit, and a display control unit. When the imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region, the first image processing unit performs first image processing on an image of the first region generated by the imaging unit. The second image processing unit performs second image processing on an image of the second region generated by the imaging unit. The detection unit detects an object in the image on which the second image processing has been performed. The display control unit displays the detection result of the object on a display unit together with the image of the first region on which the first image processing has been performed.

[0020] According to a twelfth aspect of the present invention, in the eleventh aspect, the boundary between the first region and the second region may be switchable between a first state in which the endoscope is inserted into the object of examination and advanced deeper into the object of examination, and a second state in which the endoscope is removed from the object of examination.

[0021] According to a thirteenth aspect of the present invention, an image processing device includes an acquisition unit and a detection unit. When the imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region, the acquisition unit acquires from a recording medium an inference model obtained through machine learning using, as first training data, first training images generated by the imaging unit and subjected to first image processing, and first annotations indicating an object in the first region of the first training images, and using, as second training data, second training images generated by the imaging unit and subjected to second image processing, and second annotations indicating the object in the second region of the second training images. The detection unit uses the inference model acquired by the acquisition unit to detect an object in the image generated by the imaging unit.

[0022] According to a fourteenth aspect of the present invention, an image processing device includes an acquisition unit and a detection unit. When the imaging field of view of an imaging unit of an endoscope is divided into a first region including a center of the imaging field of view and a second region other than the first region, the acquisition unit acquires from a recording medium a first inference model obtained through first machine learning using, as first training data, training images generated by the imaging unit and first annotations indicating an object in the first region of the training images, and acquires from the recording medium a second inference model obtained through second machine learning using, as second training data, the training images and second annotations indicating an object in the second region of the training images. The detection unit uses the first inference model and the second inference model acquired by the acquisition unit to detect an object in the image generated by the imaging unit.

[0023] According to a fifteenth aspect of the present invention, an image processing method includes: determining whether the state of an endoscope is a first state in which the endoscope is inserted into an object of examination and is proceeding deeper into the object of examination, or a second state other than the first state; dividing the imaging field of view of an imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region; and, if the state of the endoscope is determined to be in the first state, performing image processing to reduce the visibility of an image of the second region generated by the imaging unit below the visibility of an image of the first region generated by the imaging unit, thereby generating an image for display; and, if the state of the endoscope is determined to be in the first state, performing image processing to make object detection performance in the image of the second region equal to or higher than the object detection performance in the image of the first region, thereby generating an image for object detection.

[0024] According to a sixteenth aspect of the present invention, an image processing method includes the following: when an imaging field of view of an imaging unit of an endoscope is divided into a first region including a center of the imaging field of view and a second region other than the first region, performing first image processing on an image of the first region generated by the imaging unit; performing second image processing on an image of the second region generated by the imaging unit; detecting an object in the image on which the second image processing has been performed; and displaying the object detection result on a display unit together with the image of the first region on which the first image processing has been performed.

[0025] According to a seventeenth aspect of the present invention, an image processing method includes the following: when an imaging field of view of an imaging unit of an endoscope is divided into a central region including a center of the imaging field of view and a peripheral region other than the central region, image processing for image display is performed on an image of the central region generated by the imaging unit; an object is detected in the image of the peripheral region generated by the imaging unit; and the object detection result is displayed on a display unit that displays the image of the central region.

[0026] According to an eighteenth aspect of the present invention, in the seventeenth aspect, the image of the central region may be a rectangle. The object detection result may be an icon including at least one of information indicating the presence or absence of the object, information indicating the type of the object, and information indicating a direction from the center of the rectangle to one of the four sides of the rectangle that is closest to the position where the object was detected.

[0027] According to a nineteenth aspect of the present invention, there is provided a program for causing a computer to execute the following: Determine whether the state of an endoscope is a first state in which the endoscope is inserted into an object of examination and is advancing deeper into the object of examination, or a second state other than the first state; Divide the imaging field of view of an imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region; If the state of the endoscope is determined to be the first state, perform image processing to reduce the visibility of an image of the second region generated by the imaging unit below the visibility of an image of the first region generated by the imaging unit, thereby generating an image for display; If the state of the endoscope is determined to be the first state, perform image processing to make object detection performance in the image of the second region equal to or higher than the object detection performance in the image of the first region, thereby generating an image for object detection.

[0028] According to a twentieth aspect of the present invention, there is provided a program for causing a computer to perform the following: when an imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region, performing first image processing on an image of the first region generated by the imaging unit; performing second image processing on an image of the second region generated by the imaging unit; detecting an object in the image on which the second image processing has been performed; and displaying the detection result of the object on a display unit together with the image of the first region on which the first image processing has been performed.

[0029] According to the above aspects, the image processing device, the image processing method, and the program can perform image processing suited to the role of each region of an image.

[0030] FIG. 1 is a block diagram showing an example of the configuration of an endoscopic system according to an embodiment of the present invention. FIG. 1 is a block diagram showing an example of the configuration of an imaging unit included in an endoscopic system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of an imaging unit included in an endoscopic system according to an embodiment of the present invention. FIG. 3 is a diagram showing the principle of image correction in an embodiment of the present invention. FIG. 4 is a diagram showing an example of an insertion state in an embodiment of the present invention. FIG. 5 is a diagram showing an example of a removal state in an embodiment of the present invention. FIG. 6 is a diagram showing an example of an inspection state in an embodiment of the present invention. FIG. 7 is a diagram showing an example of an inspection state in an embodiment of the present invention. FIG. 8 is a flowchart showing an example of a processing procedure performed by an image processing device according to an embodiment of the present invention. FIG. 9 is a flowchart showing an example of a processing procedure performed by an image processing device according to an embodiment of the present invention. FIG. 10 is a diagram showing an example of an image in an embodiment of the present invention. FIG. 11 is a diagram showing an example of an image in an embodiment of the present invention. FIG. 12 is a diagram showing an example of an image in an embodiment of the present invention. FIG. 13 is a diagram showing an example of an image in an embodiment of the present invention. FIG. 14 is a block diagram showing an example of the configuration of an inference model generation unit included in an endoscopic system according to an embodiment of the present invention.

[0031] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. An example of an endoscope system including an image processing device will be described below. In the following description, "plurality of XX" means two or more XXs.

[0032] 1 shows an example of the configuration of an endoscope system 1 according to an embodiment of the present invention. The endoscope system 1 shown in FIG. 1 includes a scope 10, an image processing device 20, a recording medium 30, and a display unit 40.

[0033] The scope 10 has an operation unit 11, an information acquisition unit 12, and an endoscope 13. The endoscope 13 is shaped like a long, thin tube, and the tip of the endoscope 13 is bendable. The operation unit 11 has a knob (lever) that is operated to bend the tip of the endoscope 13. The information acquisition unit 12 outputs a signal to the image processing device 20 in accordance with the operation result of the operation unit 11.

[0034] These levers are provided corresponding to the X and Y directions, respectively, in order to bend the tip of the endoscope 13 in a direction corresponding to the X or Y direction in the image generated by the imaging unit 130 of the endoscope 13. Generally, the X-direction lever can be tilted in the X direction (left and right), and the Y-direction lever can be tilted in the Y direction (up and down). It is difficult to determine which of the X-direction lever and the Y-direction lever to tilt, in which direction to tilt the lever, and how far to tilt the lever. The user proceeds with the insertion operation through these operations, but assistance with this operation is an important issue.

[0035] The endoscope 13 is inserted into an object to be examined. The object to be examined is an organ of the subject. For example, the organ may be the digestive tract, such as the large intestine or the stomach. The endoscope 13 has an imaging unit 130. The imaging unit 130 is disposed at the tip of the endoscope 13. The imaging unit 130 has an image sensor and generates multiple images at multiple positions in the digestive tract of the subject. The imaging unit 130 outputs the multiple images to the image processing device 20. A user, such as a doctor, inserts the endoscope 13 into the digestive tract of the subject and advances the endoscope 13 to a predetermined position. The user then performs the examination by slowly bending and pulling the endoscope 13.

[0036] The imaging unit 130 may be changed to the imaging unit 130a shown in Fig. 2. The imaging unit 130a has a front imaging unit 131, a rear imaging unit 132, and a rear imaging unit 133. The front imaging unit 131 generates an image of an object in front of the endoscope 13. The image generated by the front imaging unit 131 corresponds to an image of the central area described below. The front imaging unit 131 and the rear imaging unit 132 generate images of an object in the rear of the endoscope 13. The images generated by the front imaging unit 131 and the rear imaging unit 132 correspond to images of the peripheral area described below.

[0037] When an endoscope having a rearward imaging field of view in addition to a forward imaging field of view is used, as mentioned above, if the imaging field of view becomes wider than necessary, the amount of information in the peripheral areas of the image increases, and a situation is likely to arise in which the user who is gazing at the center of the image feels stressed.

[0038] The front imaging section 131, the rear imaging section 132, and the rear imaging section 133 may generate images simultaneously. The endoscope system 1 may switch between a mode in which the front imaging section 131 generates an image and a mode in which the rear imaging section 132 and the rear imaging section 133 generate images.

[0039] The image processing device 20 includes an examination information generating unit 21 , a central determining unit 22 , a peripheral determining unit 23 , a processing control unit 24 , an image processing unit 25 , and a display control unit 26 .

[0040] The test information generating unit 21 generates test information indicating test conditions based on information input in response to operation of an operation unit (not shown in FIG. 1 ). For example, the test information includes the gender and age of the subject, and the test target. The test target indicates the type of digestive tract (such as the large intestine or stomach). The test information generated by the test information generating unit 21 is recorded on the recording medium 30 as test information 31.

[0041] The center determination unit 22 determines the center of the image based on boundary information 32 recorded on the recording medium 30. When the imaging field of view of the imaging unit 130 is divided into a central portion (first region) including the center of the imaging field of view and a peripheral portion (second region) other than the central portion, the boundary information 32 indicates the position of the boundary between the image of the central portion and the image of the peripheral portion generated by the imaging unit 130. The peripheral portion is the region outside the central portion and surrounds the central portion. For example, the entire image generated by the imaging unit 130 and the image of the central portion are rectangular. The boundary between the central portion and the peripheral portion is also rectangular. The center determination unit 22 outputs information indicating the position of the central portion to the image processing unit 25.

[0042] The periphery determination unit 23 determines the periphery of the image based on the boundary information 32 recorded on the recording medium 30. The periphery determination unit 23 outputs information indicating the position of the periphery to the image processing unit 25.

[0043] The boundary between the central portion and the peripheral portion in the inserted state (first state) may be the same as the boundary between the central portion and the peripheral portion in the removed state (second state). The inserted state is a state in which the endoscope 13 is inserted into the object of examination and advanced deeper into the object of examination. The removed state is a state in which the endoscope 13 is removed from the object of examination. As will be described later, the boundary between the central portion and the peripheral portion in the inserted state may be different from the boundary between the central portion and the peripheral portion in the removed state. In other words, the boundary between the central portion and the peripheral portion may be switchable between the inserted state and the removed state.

[0044] Although it depends on the design, generally, due to the characteristics of the lens, the influence of aberration becomes noticeable when the angle of view is 60° or more. Therefore, the boundary between the central part and the periphery may be set at a position corresponding to the angle of view of 60°.

[0045] The endoscope system 1 switches control for each area of ​​the image corresponding to the area of ​​the imaging field of view of the imaging unit 130, and executes processing according to the role of the image required in the inserted state and other states.

[0046] The processing control unit 24 controls the examination information generation unit 21, the central determination unit 22, the peripheral determination unit 23, the image processing unit 25, and the display control unit 26. The processing control unit 24 also records the image output from the imaging unit 130 as an image 33 on the recording medium 30.

[0047] The image processing unit 25 processes the image generated by the imaging unit 130. The image processing unit 25 has a central image processing unit 250, a peripheral image processing unit 251, a central object detection unit 252, a peripheral object detection unit 253, a state determination unit 254, an image correction unit 255, and a position determination unit 256.

[0048] The central image processing unit 250 (first image processing unit) performs image processing (central image processing) on ​​the image of the central portion determined by the central determination unit 22. The peripheral image processing unit 251 (second image processing unit) performs image processing (peripheral image processing) on ​​the image of the peripheral portion determined by the peripheral determination unit 23. The peripheral image processing (second image processing) may be the same as the central image processing (first image processing), but the peripheral image processing may differ from the central image processing. The imaging mode set in the imaging unit 130 may be switched between the inserted state and the removed state, and the central image processing and peripheral image processing may be set depending on the imaging mode.

[0049] The following switching of the imaging mode between the inserted state and the removed state is assumed: In the inserted state, the image of the center is important to confirm the insertion direction and lumen direction ahead of the imaging unit 130 and to avoid collisions, etc. If the image quality and visibility are poor, correct and safe insertion is difficult.

[0050] Based on the idea that the information in the central image in the inserted state is important, it can also be thought that the amount of information in the peripheral image in the inserted state can be relatively reduced. Although this is sometimes explained specifically, this is the result of emphasizing the features of the invention, and does not necessarily apply in all situations.

[0051] On the other hand, in the withdrawn state, there is no risk of the tip of the endoscope 13 colliding with the wall of the lumen. Rather, it is important to determine the type of object moving in the opposite direction to the direction in which the endoscope 13 advances in the withdrawn state, that is, from the periphery of the image toward the center of the image. If the endoscope 13 is withdrawn quickly, objects captured on the periphery of the image will instantly move away from the tip of the endoscope 13 and become smaller than the size that can be visually recognized. This may cause the user to lose sight of important areas. In order to take advantage of the characteristic that objects appear larger in the peripheral image, it is necessary to devise ways to improve the visibility and image quality of the peripheral image in the withdrawn state.

[0052] When a user visually inserts the endoscope 13 into an object to be examined, image processing that emphasizes visibility, including correct control of exposure, focus, color reproducibility, etc., is important. On the other hand, when a computer performs automatic insertion or insertion assistance based on an image, image processing that increases the amount of information obtained from the image is also effective. Of course, there are factors other than image processing, such as lens drive or lighting control for focusing. Furthermore, unlike the central portion, it is not easy to improve image quality in the peripheral portion due to constraints on the design of the optical system and image sensor, so ingenuity for improving image quality is also important.

[0053] For example, when the endoscope 13 is in the inserted state, the peripheral image processing unit 251 (first image processing unit) performs image processing to reduce the visibility of the peripheral image in the image generated by the imaging unit 130 compared to the visibility of the central image in the image generated by the imaging unit 130, and generates an image for display. For example, the peripheral image processing unit 251 deletes the peripheral image from the image generated by the imaging unit 130. Alternatively, the peripheral image processing unit 251 performs thinning processing on the image generated by the imaging unit 130, thinning out the pixels in the peripheral image. This reduces the number of pixels in the peripheral area.

[0054] The central image processing unit 250 may perform image processing to improve the visibility of the central image. This processing can be called image processing that takes visibility into account, so that the user can easily see the central image when the image is displayed on the display unit 40. In other words, the above processing is intended to generate an image with natural brightness, color expression, and gradation expression, as well as appropriate dynamic range and contrast. Visibility may be impaired by fluids and air bubbles inside the body, and the central image processing unit 250 may perform correction to remove these. The central image processing unit 250 may perform correction based on the display's performance, such as the aspect ratio or pixel count, or the environmental lighting conditions. Adaptive processing technology may also be used to recognize the characteristics of each region of the image and perform appropriate image correction accordingly. It is also important to display a large, easy-to-view image on the display screen so that users at a distance can view the image.

[0055] When the endoscope 13 is in the inserted state, the peripheral image processing unit 251 (second image processing unit) performs image processing to make the object detection performance in the peripheral image equal to or higher than the object detection performance in the central image, and generates an image for object detection. For example, the object is a lesion. For example, the object detection performance may be the minimum size of a detectable object or the object detection speed.

[0056] For example, the central image processor 250 performs image quality improvement processing on the central image, and the peripheral image processor 251 performs image quality improvement processing on the peripheral image, such as noise reduction, edge enhancement, color adjustment, or high dynamic range (HDR) processing.

[0057] The intensity of the processing performed on the peripheral image is equal to or greater than the intensity of the processing performed on the central image, so the effect of improving the image quality of the peripheral image is equal to or greater than the effect of improving the image quality of the central image, i.e., the object detection performance in the peripheral image is equal to or greater than the object detection performance in the central image.

[0058] The process for generating an image for object detection is image processing that assumes that the user does not need to check the peripheral image when the image is displayed on the display unit 40. This is possible because the peripheral object detection unit 253 is assumed to detect objects in the peripheral image. Therefore, the process for generating an image for display can perform image processing that reduces the visibility of the peripheral area more than the visibility of the central area. This image processing may include reducing brightness, reducing color saturation, suppressing gradation, and moderating the dynamic range and contrast. The central image processing unit 250 and the peripheral image processing unit 251 may perform at least one of these processes or other means to generate an image for display.

[0059] Visibility may be impaired by fluids and air bubbles inside the body, and correction to remove these is important for detecting objects. Therefore, the central image processing unit 250 and the peripheral image processing unit 251 may perform the above-mentioned corrections in the process of generating an image for object detection. The central image processing unit 250 and the peripheral image processing unit 251 may perform corrections depending on the display's performance, such as the aspect ratio or number of pixels, or the environmental brightness conditions. This correction may involve processing to reduce visibility. Alternatively, adaptive processing techniques may be used in combination, such as recognizing the characteristics of each region of the image and performing appropriate image correction accordingly. This correction may involve processing to reduce visibility. Since visibility from the user does not need to be emphasized, the central image processing unit 250 and the peripheral image processing unit 251 may reduce or deform the image.

[0060] In the inserted state, the user focuses on the central image. Because the visibility of the peripheral image in the display image is lower than that of the central image, the user is less likely to feel stressed by the influence of the peripheral image. Furthermore, because the object detection performance in the peripheral image in the object detection image is the same as or higher than that in the central image, objects in the peripheral image are less likely to be overlooked.

[0061] For example, when the endoscope 13 is in the removed state, the peripheral image processing unit 251 does not perform image processing that reduces the visibility of the peripheral image compared to the central image, but performs image processing that makes the visibility of the peripheral image the same as the visibility of the central image. In the removed state, the user observes the central image and the peripheral image. If a lesion is found, the user performs an examination of the lesion. The examination of the lesion includes diagnosing or treating the lesion. Because the visibility of the peripheral image is the same as the visibility of the central image, objects in the peripheral area are less likely to be overlooked.

[0062] The central object detection unit 252 detects a feature region having predetermined characteristics as an object from the central image of the image for object detection. The process for detecting this feature region may employ technology similar to that used by a general camera to detect faces and adjust focus and exposure. Pattern matching or an inference model may also be utilized. When an inference model obtained through learning using images containing objects such as lesions and annotations representing the object's position information as training data is used, an endoscopic image is input to the central object detection unit 252. The central object detection unit 252 outputs information indicating the presence or absence of an object in the image, and, if an object is detected in the image, outputs the object's position information.

[0063] The peripheral object detection unit 253 detects a feature region having predetermined characteristics as an object from the peripheral image of the object detection image. The process for detecting this feature region may employ the technology of a general camera detecting faces and adjusting focus and exposure, as described above. When an inference model is used that inputs an endoscopic image, outputs information indicating the presence or absence of an object in the image, and, if an object is detected in the image, outputs the object's location information, the AI ​​technology effectively monitors the image on behalf of medical personnel. If learning is performed using the type, size, and other features of the object as annotations in addition to location information, such information can also be inferred.

[0064] Furthermore, once the position of the object is known, it is possible to calculate the direction from the center of the rectangular image to the side of the rectangle that is closest to the position where the object was detected. For example, when the position on the image is expressed in coordinates with the center of the image as the origin, the horizontal direction of the image can be treated as the X direction and the vertical direction of the image as the Y direction. In other words, if the coordinate of the object in the X direction exceeds the range of the center in the X direction, the object is located outside the center and to the left or right of the center. Furthermore, if the coordinate of the object in the Y direction exceeds the range of the center in the Y direction, the object is located outside the center and above or below the center.

[0065] The state determination unit 254 determines whether the state of the endoscope 13 is an inserted state, a removed state, or an examination state (observation state). In the inserted state, image features of the lumen wall move radially toward the periphery in multiple images obtained in time series. During insertion of the endoscope 13, the user may point the tip of the endoscope 13 in a direction different from the insertion direction to check the insertion direction, or may become lost or stop. Even in such a situation, if the above-mentioned radial movement can be detected as a motion vector feature, the state determination unit 254 may determine that the state of the endoscope 13 is an inserted state.

[0066] If an acceleration sensor is provided at the tip of the endoscope 13, the state determination unit 254 may determine the direction of acceleration based on a signal output from the acceleration sensor and determine that the state of the endoscope 13 is the inserted state. If a specific procedure is determined in advance, the state determination unit 254 may determine that the state of the endoscope 13 is the inserted state during a scheduled time period. The state determination unit 254 may determine that the state of the endoscope 13 is the inserted state until the innermost object is captured in an image.

[0067] The examination state is a state in which the tip of the endoscope 13 is facing the lesion and the user is examining the lesion. In the examination state, an image feature is obtained in which the tip of the endoscope 13 approaches the lesion, and the state determination unit 254 can determine the examination state based on this image feature. In this case, an image is obtained in which the object is enlarged as it approaches the center of the image, and then multiple similar images are acquired consecutively for observation. Therefore, the state determination unit 254 can determine the examination state based on such features.

[0068] The method for determining the removal state of the endoscope 13 is similar to the method for determining the state of the endoscope 13. In multiple images obtained in time series, image features of the lumen wall move radially toward the center. During the insertion of the endoscope 13, the user may point the tip of the endoscope 13 in a direction different from the removal direction to check the insertion direction, or may become lost or stop. Even in such a situation, if the above-mentioned radial movement can be detected as a feature of the motion vector, the state determination unit 254 may determine that the state of the endoscope 13 is in the removal state.

[0069] If an acceleration sensor or the like is provided at the tip of the endoscope 13, the state determination unit 254 may determine the direction of acceleration based on a signal output from the acceleration sensor and determine that the state of the endoscope 13 is the withdrawal state. If a specific procedure is determined in advance, the state determination unit 254 may determine that the state of the endoscope 13 is the withdrawal state during a scheduled time period. The state determination unit 254 may determine that the state of the endoscope 13 is the withdrawal state from the time the innermost object is captured in an image until the endoscope 13 leaves the examination target.

[0070] Even during an examination while the endoscope 13 is being removed, an image feature is obtained that indicates that the tip of the endoscope 13 is approaching the lesion, so the state determination unit 254 can determine the examination state based on that image feature. In this case, an image is obtained in which the object is enlarged as it approaches the center of the image, and then multiple similar images are acquired consecutively for observation. Therefore, the state determination unit 254 can determine the examination state based on such features.

[0071] The image correcting section 255 corrects the image generated by the imaging section 130 based on the optical characteristics of the endoscope 13. The processing executed by the image correcting section 255 will be described in detail with reference to Figs. 3 and 4.

[0072] 3 shows an example of the configuration of the image capturing unit 130. The image capturing unit 130 has an aperture 134, a lens 135, and an image capturing element 136. Light from an object passes through the aperture 134 and the lens 135 and enters the image capturing element 136. A plurality of pixels are arranged on the image capturing surface of the image capturing element 136.

[0073] An optical image of range R1 is formed at the central pixel on the imaging surface of the image sensor 136. Due to the influence of aberration of the lens 135, an optical image of range R2, which is wider than range R1, is formed at the peripheral pixels on the imaging surface of the image sensor 136. Therefore, more information can be obtained from the peripheral pixels than from the central pixel.

[0074] 4 shows the principle of image correction. An optical image of range R3 within the imaging field of view is formed in range R4 on the imaging surface of the image sensor 136 due to the influence of aberration of the lens 135. The image correction unit 255 corrects the positional deviation that occurs in the image due to the aberration of the lens 135. In other words, the image correction unit 255 corrects the distortion of the image. As a result, the image correction unit 255 obtains an image that is the same as the image that would be obtained if the optical image of range R3 were formed in range R5 on the imaging surface.

[0075] 2 is used, an image is generated by each of the front imaging unit 131, the rear imaging unit 132, and the rear imaging unit 133. The image correction unit 255 may correct positional deviations in the images generated by each imaging unit in accordance with the optical characteristics of each imaging unit.

[0076] The position determination unit 256 determines the current position of the tip of the endoscope 13 based on the multiple images generated by the imaging unit 130, and generates current position information indicating that position. This position indicates the position within the digestive tract that is being imaged by the imaging unit 130.

[0077] The "current position of the tip of the endoscope 13" may be in units of organ name (part name) or a subdivided position within the organ. For example, the organ name may be the esophagus or stomach in the case of the upper digestive tract, and the rectum or transverse colon in the case of the lower digestive tract. For example, a subdivided position may be a position about 3 cm after passing the sigmoid colon and entering the transverse colon. The "current position of the tip of the endoscope 13" may be a position within each lumen, the name of the lumen, or a classification position within the lumen. The "current position of the tip of the endoscope 13" may be a combination of the name of the lumen and the lumen direction position, depth, length, or distance within the lumen. A direction perpendicular to the continuous direction (axial direction) of the lumen may also be used as needed.

[0078] In the case of a pouch-shaped organ such as the stomach, the "current position of the tip of the endoscope 13" may be a site name or part name, such as the cardia, fundus, body, pyloric antrum, or pylorus, from the entrance, or information expressing any location of that site in more detail, such as coordinates. Such location information may be detected from image features, such as tissue or blood vessels, specific to that site. The position may be determined by using changes in image features accompanying changes in image frames acquired continuously during insertion of the endoscope 13, or by using blood vessel or tissue patterns. The image processing device 20 may detect the speed (cm / sec) of the endoscope 13 during insertion and convert the insertion position using time information. However, instead of detecting this speed, the image processing device 20 may use, as a constant, the average insertion speed when a typical physician inserts an endoscope. The image processing device 20 may determine the shape of the lumen using three-dimensional (3D) reconstruction using image information, etc., and estimate the position during examination based on the results of this determination.

[0079] In the following example, the position indicates a specific region within a simplified digestive tract. If a position sensor is disposed at the tip of the endoscope 13, the position determination unit 256 may determine the region currently being examined based on information output from the position sensor. For example, a method of detecting the region using magnetism or the like is available. Alternatively, the region may be detected using other medical equipment such as a CT or MRI. The position determination unit 256 generates region information (current position information) indicating the determined region.

[0080] Furthermore, the expression "position in an organ" may refer to the name of a part within an organ, such as the cardia, fundus, body, pyloric antrum, or pylorus, but "position in an organ" may also be used to mean the difference between organs of different classifications, such as the esophagus and the stomach.

[0081] The display control unit 26 superimposes information to be notified to the user on the image output from the image processing unit 25, and outputs the image to the display unit 40. As a result, the display control unit 26 displays the information together with the image. For example, the display control unit 26 displays on the display unit 40 a detection result indicating that an object has been detected together with the image of the center portion processed by the central image processing unit 250.

[0082] For example, when the endoscope 13 is in the inserted state, the peripheral object detection unit 253 detects an object from the peripheral image in the image for object detection. The display control unit 26 causes the display unit 40 to display position information indicating the position of the object in the peripheral image together with the central image processed by the central image processing unit 250. If the peripheral image processing unit 251 deletes the peripheral image or thins out the pixels of the peripheral image, it is difficult for the user to confirm the object in the peripheral image. However, the user can know that the object is present at the position indicated by the position information. Therefore, the user can avoid missing the object.

[0083] When the endoscope is in the inserted state, the peripheral object detection unit 253 detects an object from the image of the peripheral part in the image for object detection. The peripheral object detection unit 253 also stores, in the recording medium 30, first position information indicating the position of the object in the image of the peripheral part and second position information indicating the position of the object in the inspection object. The second position information is region information generated by the position determination unit 256. The first position information indicates the position of the object in the image generated by the imaging unit 130 at the position indicated by the second position information.

[0084] When the state of the endoscope 13 is the withdrawal state and the position of the endoscope 13 matches the position indicated by the second position information, the display control unit 26 causes the display unit 40 to display the second position information together with the image generated by the imaging unit 130. This notifies the user of the position of the object detected in the insertion state. The user can know that the object is present at the position indicated by the second position information. This allows the user to avoid overlooking the object.

[0085] The image processing device 20 may be configured with a processor such as a CPU (Central Processing Unit).

[0086] A computer may load and execute a program, which includes instructions that define the operation of the image processing device 20. In other words, the functions of the image processing device 20 may be realized by software.

[0087] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope system 1 via a transmission medium or by transmission waves in the transmission medium. The "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The functions described above may be realized by combining a program already recorded on the computer with the difference program.

[0088] The processor and the recording medium do not necessarily have to be included in a single device, but may be configured as a plurality of devices with distributed functions working together. Also, the processor and the recording medium may be located on a cloud (network).

[0089] The display unit 40 is a liquid crystal monitor, etc. The display unit 40 displays the image output from the display control unit 26.

[0090] The recording medium 30 is a memory. Inspection information 31, boundary information 32, and an image 33 are recorded on the recording medium 30. These are read by the image processing device 20 and used in the processing executed by the image processing device 20.

[0091] A lesion detected in the inserted state may not be detected in the removed state. FIG. 5A shows an example of the inserted state, and FIG. 5B shows an example of the removed state. In the inserted state shown in FIG. 5A, the endoscope 13 is inserted into the inspection target IT1. As shown in FIG. 5A, in the inserted state, the lesion LS1 may appear in the image generated by the imaging unit 130. However, as shown in FIG. 5B, in the removed state, the lesion LS1 may be hidden behind a fold and not appear in the image generated by the imaging unit 130. In this case, the user may miss the lesion LS1.

[0092] When the endoscope 13 is in the insertion state, the display control unit 26 causes the display unit 40 to display position information indicating the position of the lesion LS1 in the image of the peripheral area together with the image of the central area processed by the central image processing unit 250. For example, the display unit 40 displays the image of the central area and an arrow indicating the position of the lesion LS1. Even when the display unit 40 does not display the image of the peripheral area, the user can know the position of the lesion LS1 in the image of the peripheral area. Therefore, the user can observe the lesion LS1 by pointing the tip of the endoscope 13 toward the lesion LS1, as shown in FIG. 6A . At this time, the state of the endoscope 13 is in the examination state.

[0093] When the endoscope 13 is in the withdrawal state and the position of the endoscope 13 coincides with the position of the object in the examination subject indicated by the second position information, the display control unit 26 causes the display unit 40 to display the first position information indicating the position of the lesion LS1 in the image of the peripheral area together with the image generated by the imaging unit 130. For example, the display unit 40 displays the image generated by the imaging unit 130 and an arrow indicating the position of the lesion LS1. Even if the lesion LS1 is not visible in the image as shown in FIG. 5B , the user can know the position of the lesion LS1. Therefore, the user can point the tip of the endoscope 13 toward the lesion LS1 and observe the lesion LS1 in the gap between the folds as shown in FIG. 6B . At this time, the endoscope 13 is in the examination state.

[0094] An example of the operation of the image processing device 20 will be described with reference to Figures 7 and 8. Figures 7 and 8 show an example of the procedure of the process executed by the image processing device 20.

[0095] (Step S100) The user inputs various inspection conditions into the image processing device 20. The inspection information generation unit 21 determines whether or not the inspection conditions have been input. If the inspection conditions have not been input, the inspection information generation unit 21 repeats this determination. If the inspection conditions have been input, the inspection information generation unit 21 generates inspection information. Then, step S101 is executed.

[0096] (Step S101) After the inspection conditions are input, the process control unit 24 causes the imaging unit 130 to start imaging.

[0097] (Step S102) The state determination unit 254 uses the method described above to determine whether the state of the endoscope 13 is the inserted state, the removed state, or the examination state. The state determination unit 254 does not need to use the entire image generated by the imaging unit 130, and may determine the state of the endoscope 13 by using a partial region of the image.

[0098] (Step S103) As the endoscope 13 moves, it may rotate around its central axis. The state determination unit 254 analyzes multiple images generated by the imaging unit 130 and determines the rotational state of the endoscope 13. The rotational state includes the direction of rotation and the amount of rotation. If an acceleration sensor is provided at the tip of the endoscope 13, the state determination unit 254 may determine the rotational state of the endoscope 13 based on a signal output from the acceleration sensor. Rotational state information indicating the rotational state is recorded on the recording medium 30.

[0099] (Step S104) The process control unit 24 determines whether the state of the endoscope 13 is the inserted state or the removed state based on the result of the process in step S102. If the state of the endoscope 13 is the inserted state or the removed state, step S110 is executed. If the state of the endoscope 13 is other than the inserted state or the removed state, step S150 is executed.

[0100] (Step S110) The process control unit 24 determines whether the state of the endoscope 13 is the inserted state based on the result of the process in step S102. If the state of the endoscope 13 is the inserted state, step S111 is executed. If the state of the endoscope 13 is the removed state, step S130 is executed.

[0101] (Step S111) The center determination unit 22 determines the center of the image based on the boundary information 32 recorded on the recording medium 30. The center determination unit 22 outputs information indicating the position of the center to the image processing unit 25. The periphery determination unit 23 determines the periphery of the image based on the boundary information 32 recorded on the recording medium 30. The periphery determination unit 23 outputs information indicating the position of the periphery to the image processing unit 25.

[0102] (Step S112) The peripheral image processing unit 251 performs peripheral image processing for display on the image of the peripheral area determined by the peripheral determination unit 23. As described above, the peripheral image processing unit 251 deletes the image of the peripheral area from the image generated by the imaging unit 130. Alternatively, the peripheral image processing unit 251 performs thinning processing on the image generated by the imaging unit 130 to thin out the pixels of the image of the peripheral area. The following describes a case where the peripheral image processing unit 251 deletes the image of the peripheral area. The image correction unit 255 corrects the image processed by the peripheral image processing unit 251. The image processed by the image correction unit 255 is output to the display control unit 26.

[0103] The central image processing unit 250 may perform central image processing for display on the central image determined by the central determination unit 22. For example, the central image processing unit 250 may perform processing to improve the image quality of the central image.

[0104] (Step S113) The display control unit 26 outputs the image processed in step S112 to the display unit 40, and causes the display unit 40 to display the image.

[0105] (Step S114) The central image processing unit 250 and the peripheral image processing unit 251 perform image processing to improve object detection performance and generate an image for object detection. For example, the central image processing unit 250 performs processing to improve image quality on the central image, and the peripheral image processing unit 251 performs processing to improve image quality on the peripheral image. In this case, the intensity of the processing performed on the peripheral image may be the same as or higher than the intensity of the processing performed on the central image. The central image processing unit 250 may not perform image processing, and only the peripheral image processing unit 251 may perform image processing.

[0106] (Step S115) The central object detection unit 252 detects an object from the central image of the image processed in step S114. If an object is detected, first position information indicating the position of the object in the central image is recorded on the recording medium 30.

[0107] (Step S116) The position determination unit 256 determines the current position of the tip of the endoscope 13 by using the method described above, and generates second position information indicating that position. The second position information is recorded on the recording medium 30. The first position information recorded in step S115 and the second position information recorded in step S116 are associated with each other. In addition, status information indicating the insertion status is associated with the first position information and the second position information.

[0108] (Step S117) The process control unit 24 determines whether or not an object has been detected in the central image based on the result of the process in step S115. If an object has been detected in the central image, step S118 is executed. If an object has not been detected in the central image, step S120 is executed.

[0109] (Step S118) The display control unit 26 emphasizes the object in the center of the image displayed on the display unit 40. For example, the display control unit 26 displays a frame around the object. After step S118 is executed, step S102 is executed.

[0110] (Step S120) The peripheral object detection unit 253 detects an object from the peripheral image of the image processed in step S114. If an object is detected, first position information indicating the position of the object in the peripheral image is recorded on the recording medium 30. The first position information recorded in step S115 and the second position information recorded in step S120 are associated with each other. In addition, status information indicating the insertion status is associated with the first position information and the second position information.

[0111] (Step S121) The process control unit 24 determines whether or not an object has been detected in the image of the peripheral area based on the result of the process in step S120. If an object has been detected in the image of the peripheral area, step S122 is executed. If an object has not been detected in the image of the peripheral area, step S102 is executed.

[0112] (Step S122) The display control unit 26 displays a detection result indicating that an object has been detected in the image of the peripheral area on the image displayed on the display unit 40. The display control unit 26 may display information indicating the position of the object in the peripheral area on the image displayed on the display unit 40. For example, the display control unit 26 may display an arrow indicating the position indicated by the first position information recorded on the recording medium 30.

[0113] At this time, the display control unit 26 may display information indicating the type of the object in addition to the information indicating the position of the object on the image displayed on the display unit 40. The information indicating the type of the object may include information indicating the characteristics of the object. After step S122 is executed, step S102 is executed.

[0114] 9A shows image IMG1 generated by imaging unit 130. Image IMG1 corresponds to the entire imaging field of view of imaging unit 130. Object T1 is captured in the peripheral portion of image IMG1. Because peripheral image processing unit 251 deletes the peripheral image from image IMG1 in step S112, only central image IMG1c is displayed on display unit 40.

[0115] 9B shows image IMG2 displayed on the display unit 40. Image IMG1c is enlarged and displayed on the display unit 40 as image IMG2. The display control unit 26 displays a message M1 on image IMG2 indicating that object T1 has been detected. Because object T1 does not appear in image IMG1c, the user cannot see object T1. However, because message M1 is displayed, the user can know that object T1 has been detected.

[0116] When the imaging field of view of the imaging unit 130 of the endoscope 13 is divided into a central region including the center of the imaging field of view and a peripheral region other than the central region, the central image processing unit 250 may perform image processing for display on the image of the central region generated by the imaging unit 130. In this case, the central image processing unit 250 may generate an image that makes it easier for the user operating the endoscope 13 to concentrate on the insertion operation.

[0117] A specific lesion or the like may be captured in the peripheral region of the image generated by the imaging unit 130. To effectively utilize such information, the peripheral object detection unit 253 detects objects from the image of the peripheral region. The display control unit 26 displays the object detection results on the display unit 40, which displays the image of the central region. This allows the user to reliably recognize the information they want to obtain from each region.

[0118] In particular, when a user is performing an insertion operation while determining the depth direction of a narrow, winding lumen, the user's attention is focused on how to proceed rather than on the lesion in the peripheral area of ​​the image. At such times, other image information that could provide clues is easily overlooked. In this case, for example, AI functions are monitoring the peripheral area, allowing the user to concentrate on the operation with peace of mind.

[0119] The image of the central region is a rectangle. The result of this object detection is an icon containing information indicating the direction from the center of the rectangle to the side of the rectangle that is closest to the position where the object was detected. This allows the user to know in which direction to move the tip of the endoscope 13. The display control unit 26 may cause the display unit 40 to display a diagram showing a lever for bending the tip of the endoscope 13, the direction in which the lever is operated, and the direction in which the tip of the endoscope 13 bends.

[0120] If the object is a living organism, there is a high possibility that the object will be deformed. Because the user remotely controls the lever to bend the tip of the endoscope 13, the direction in which the tip of the endoscope 13 bends is prone to error. Therefore, unless the user bends the tip of the endoscope 13 without receiving clear instructions and without redoing the process, the object is likely to be overlooked. As mentioned above, the display unit 40 may display information indicating the presence or absence of an object. Furthermore, if an inference model for determining the type of object is installed, the display unit 40 may display such accompanying information. The display unit 40 may also display characteristics of the object, such as its size or color.

[0121] The display control unit 26 displays a message M2 on the image IMG2 prompting the user to display the entire image. When the user performs a predetermined operation, the display control unit 26 causes the display unit 40 to display the image IMG3 shown in FIG. 9C. The image IMG3 corresponds to the entire imaging field of view of the imaging unit 130. The object T1 is captured in the peripheral portion of the image IMG3. The user can confirm the object T1.

[0122] While the endoscope 13 is in the inserted state, step S112 and the like are repeatedly executed. If the peripheral object detection unit 253 does not detect an object in the image of the peripheral area, the peripheral image processing unit 251 deletes the image of the peripheral area from the entire image of the imaging field of view of the imaging unit 130 in step S112. After the peripheral object detection unit 253 detects an object in the image of the peripheral area, the peripheral image processing unit 251 does not delete the image of the peripheral area from the entire image of the imaging field of view of the imaging unit 130 in step S112. Therefore, the entire image IMG3 generated by the imaging unit 130 is displayed on the display unit 40.

[0123] The display control unit 26 displays a message M3 on the image IMG3 to prompt the user to return to a state in which an image of the same range as the image IMG2 is displayed. When the user performs a predetermined operation, the display control unit 26 causes the display unit 40 to display an image similar to the image IMG2. When an object is detected in the image of the peripheral area, the display control unit 26 may cause the display unit 40 to display the image IMG3 without displaying the image IMG2.

[0124] The display control unit 26 may cause the display unit 40 to display the image IMG4 shown in Fig. 10A instead of the image IMG2. The display control unit 26 may also display an alert AL1 on the image IMG4. The alert AL1 indicates the position indicated by the first position information recorded on the recording medium 30. In other words, the alert AL1 indicates the position of the object. The alert AL1 is an arrow indicating that the object is present in the downward direction in the image IMG4.

[0125] After image IMG4 is displayed, the user may point the tip of the endoscope 13 toward the object. In this case, the display control unit 26 causes, for example, image IMG5 shown in Fig. 10B to be displayed on the display unit 40. A portion of object T1 is captured in image IMG5.

[0126] The display control unit 26 may cause the display unit 40 to display the image IMG6 shown in Fig. 11A instead of the image IMG2. The display control unit 26 may also display the warning AL2 on the image IMG6. The position of the warning AL2 indicates the position indicated by the first position information recorded on the recording medium 30. In other words, the position of the warning AL2 indicates the position of the object. Because the warning AL2 is displayed below the image IMG6, the warning AL2 indicates that the object is present in the downward direction in the image IMG6.

[0127] The display control unit 26 may cause the display unit 40 to display the image IMG7 shown in FIG. 11B instead of the image IMG2. The image IMG7 is generated by deleting the peripheral image from the image generated by the imaging unit 130. The display control unit 26 may cause the display unit 40 to display a frame FR1 indicating the range of the image generated by the imaging unit 130. The display control unit 26 may also display the warning AL3 outside the image IMG7 but within the frame FR1. The position of the warning AL3 indicates the position indicated by the first position information recorded on the recording medium 30. In other words, the position of the warning AL3 indicates the position of the object. Because the warning AL3 is displayed below the image IMG7, the warning AL3 indicates that the object is present in the downward direction in the image IMG7.

[0128] (Step S130) The center determination unit 22 determines the center of the image based on the boundary information 32 recorded on the recording medium 30. The center determination unit 22 outputs information indicating the position of the center to the image processing unit 25. The periphery determination unit 23 determines the periphery of the image based on the boundary information 32 recorded on the recording medium 30. The periphery determination unit 23 outputs information indicating the position of the periphery to the image processing unit 25. The boundary between the center and periphery in step S130 may be different from the boundary between the center and periphery in step S111. For example, because the image of the center is important in the inserted state, the center in step S111 may be larger than the center in step S130.

[0129] (Step S131 ) The central image processing unit 250 executes image processing on the image of the central portion determined by the central determination unit 22 .

[0130] (Step S132) The peripheral image processing unit 251 performs image processing on the image of the peripheral area determined by the peripheral determination unit 23. The image correction unit 255 corrects the image processed by the central image processing unit 250 and the peripheral image processing unit 251. The image processed by the image correction unit 255 is output to the display control unit 26.

[0131] The image processing in step S132 may be the same as the image processing in step S131. For example, the central image processing unit 250 and the peripheral image processing unit 251 may perform processing to improve visibility. The processing to improve visibility may be the same as the processing to improve image quality. The intensity of the processing performed on the peripheral image may be the same as the intensity of the processing performed on the central image. Therefore, the visibility of the peripheral image processed by the peripheral image processing unit 251 may be the same as the visibility of the peripheral image processed by the central image processing unit 250.

[0132] (Step S133) The display control unit 26 outputs the image processed in steps S131 and S132 to the display unit 40, and causes the display unit 40 to display the image.

[0133] (Step S134) The central object detection unit 252 detects an object from the central image processed in step S131. If an object is detected, first position information indicating the position of the object in the central image is recorded on the recording medium 30. The peripheral object detection unit 253 detects an object from the peripheral image processed in step S132. If an object is detected, first position information indicating the position of the object in the peripheral image is recorded on the recording medium 30.

[0134] (Step S135) The position determination unit 256 determines the current position of the tip of the endoscope 13 and generates second position information indicating that position. The second position information is recorded on the recording medium 30. The first position information recorded in step S134 and the second position information recorded in step S135 are associated with each other. In addition, status information indicating the removal status is associated with the first position information and the second position information.

[0135] (Step S136) The process control unit 24 determines whether an object has been detected in at least one of the central image and the peripheral image based on the result of the process in step S134. If an object has been detected in at least one of the central image and the peripheral image, step S137 is executed. If an object has not been detected in either the central image or the peripheral image, step S140 is executed.

[0136] (Step S137) The display control unit 26 emphasizes the object in the image displayed on the display unit 40. For example, the display control unit 26 displays a frame at the position indicated by the first position information recorded on the recording medium 30. As a result, the frame is displayed around the object. After step S137 is executed, step S102 is executed.

[0137] (Step S140) The process control unit 24 determines whether or not second position information indicating the same position as the position determined in step S135 is recorded on the recording medium 30. This allows the process control unit 24 to determine whether or not the endoscope 13 has reached the position where the object was detected in the inserted state. If the object detected in the inserted state is detected again in the removed state, second position information indicating the same position as the position determined in step S135 is recorded on the recording medium 30. In this case, the process control unit 24 determines that the endoscope 13 has reached the position where the object was detected in the inserted state. If second position information indicating the same position as the position determined in step S135 is not recorded on the recording medium 30, the process control unit 24 determines that the endoscope 13 has not reached the position where the object was detected in the inserted state. If the endoscope 13 has reached the position where the object was detected in the inserted state, step S141 is executed. If the endoscope 13 has not reached the position where the object was detected in the inserted state, step S102 is executed.

[0138] (Step S141) The display control unit 26 acquires, from the recording medium 30, first position information associated with second position information indicating the same position as the position determined in step S135. The display control unit 26 displays the position indicated by the first position information on the image displayed on the display unit 40. The rotational state of the endoscope 13 in the inserted state may differ from the rotational state of the endoscope 13 in the removed state. The display control unit 26 may correct the position indicated by the first position information, taking into account changes in the rotational state of the endoscope 13.

[0139] (Step S150) The processing control unit 24 determines whether or not first position information, which is associated with status information indicating the insertion status and indicates a position in the image of the peripheral part, is recorded on the recording medium 30. If the peripheral object detection unit 253 detects an object in the insertion status, the first position information, which is associated with status information indicating the insertion status and indicates a position in the image of the peripheral part, is recorded on the recording medium 30 in step S120. If the first position information, which is associated with status information indicating the insertion status and indicates a position in the image of the peripheral part, is recorded on the recording medium 30, the processing control unit 24 determines whether or not the tip of the endoscope 13 is heading in the direction of the position indicated by the first position information.

[0140] If the tip of the endoscope 13 is pointing in the direction of the position indicated by the first position information, the state of the endoscope 13 is the examination state, and step S151 is executed. If the tip of the endoscope 13 is not pointing in the direction of the position indicated by the first position information, step S160 is executed. To avoid complicating the explanation, the following describes a case where the state of the endoscope 13 is the examination state while the user is inserting the endoscope 13 into the examination object. If first position information that is associated with state information indicating the insertion state and indicates the position in the image of the peripheral area is not recorded on the recording medium 30, step S160 is executed.

[0141] When the above-mentioned step S122 is executed, a detection result indicating that an object has been detected in the image of the peripheral area is displayed on the image. At this time, the image of the peripheral area is not displayed, so the object is not displayed. The user determines that the object has been detected and points the tip of the endoscope 13 toward the object to observe it. In the image generated by the imaging unit 130, the object approaches the center of the imaging field of view. This state is determined in step S150.

[0142] (Step S151) The center determination unit 22 outputs information indicating the position of a central portion that is wider than the central portion determined in step S111 to the image processing unit 25. The periphery determination unit 23 outputs information indicating the position of a peripheral portion that is narrower than the peripheral portion determined in step S111 to the image processing unit 25. When an object approaches the center of the imaging field of view in multiple images continuously generated by the imaging unit 130, the central portion in the multiple images gradually becomes larger, and the peripheral portion in the multiple images gradually becomes smaller.

[0143] (Step S152) The peripheral image processing unit 251 deletes the image of the peripheral part determined by the peripheral determination unit 23. The peripheral part deleted in step S152 is smaller than the peripheral part deleted in step S112. The image correction unit 255 corrects the image processed by the peripheral image processing unit 251. The image processed by the image correction unit 255 is output to the display control unit 26.

[0144] (Step S153) The display control unit 26 outputs the image processed in step S152 to the display unit 40, and causes the display unit 40 to display the image.

[0145] (Step S154) The display control unit 26 displays information indicating the positions of the objects in the peripheral area on the image displayed on the display unit 40.

[0146] FIG. 12 shows image IMG8 generated by the imaging unit 130. Image IMG8 corresponds to the entire imaging field of view of the imaging unit 130. Object T1 is captured in the peripheral portion of image IMG8. Since the peripheral image processing unit 251 deletes the peripheral image in step S152, only image IMG8c in the center is displayed on the display unit 40. The display control unit 26 displays warning AL4 on image IMG8c. The size of image IMG8 is the same as the size of image IMG1 shown in FIG. 9A. Image IMG8c is larger than image IMG1c shown in FIG. 9A.

[0147] (Step S160) The central image processing unit 250 or the peripheral image processing unit 251 performs image processing for display on the entire image generated by the imaging unit 130.

[0148] (Step S161) The display control unit 26 outputs the image processed in step S160 to the display unit 40, and causes the display unit 40 to display the image.

[0149] (Step S162) The central object detection unit 252 or the peripheral object detection unit 253 detects objects from the entire image processed in step S160.

[0150] (Step S163) If an object is detected in step S162, the display control unit 26 displays a detection result indicating that the object has been detected on the image displayed on the display unit 40. For example, the display control unit 26 displays a frame around the object. After step S163 is executed, step S102 is executed.

[0151] The central object detection unit 252 and the peripheral object detection unit 253 may detect objects by using an inference model (AI) obtained by deep learning or the like using feature information appearing in the image as training data. The endoscope system 1 may have an inference model generation unit 50 shown in Fig. 13. Fig. 13 shows the configuration of the inference model generation unit 50. The inference model generation unit 50 may be included in the image processing device 20.

[0152] The inference model generation unit 50 includes a learning unit 51. The learning unit 51 may generate training data by adding annotations to frames (frames of interest) in which a specific image feature (such as a lesion) occurs among video data including a series of frames obtained from the start to the end of the examination. The learning unit 51 may also perform learning so as to output a frame of interest when a series of frames of a video image acquired by the endoscope 13 are input, thereby generating the inference model 52.

[0153] For example, the learning unit 51 generates training data D1 for a central image and training data D2 for a peripheral image. The training data D1 for a central image includes a central image CI1 and a central annotation CA1. The central image CI1 is an image of the central part of an image generated by the imaging unit 130. The central annotation CA1 is added to the central image CI1. The training data D2 for a peripheral image includes a peripheral image SI1 and a peripheral annotation SA1. The peripheral image SI1 is an image of the peripheral part of an image generated by the imaging unit 130. The peripheral annotation SA1 is added to the peripheral image SI1.

[0154] The learning unit 51 generates an inference model 52 by performing machine learning using images generated by the imaging unit 130 for multiple cases and annotations resulting from performing lesion determinations on the images as training data. The inference model 52 is, for example, a neural network and is generated through deep learning. The inference model 52 is not limited to a neural network and may be another machine learning model that can output information for an input image. The inference model 52 is recorded on the recording medium 30.

[0155] For example, the central image training data D1 (first training data) used in the machine learning includes a first training image generated by the imaging unit 130 and subjected to first image processing by the central image processing unit 250, and a central annotation CA1 (first annotation) indicating an object in the central portion of the first training image (central image CI1). Furthermore, the peripheral image training data D2 (second training data) used in the machine learning includes a second training image generated by the imaging unit 130 and subjected to second image processing by the peripheral image processing unit 251, and a peripheral annotation SA1 (second annotation) indicating an object in the peripheral portion of the second training image (peripheral image SI1). The processing control unit 24 (acquisition unit) acquires the inference model obtained through the above-described machine learning from the recording medium 30. The central object detection unit 252 or the peripheral object detection unit 253 uses the inference model acquired by the processing control unit 24 to detect objects in the image generated by the imaging unit 130.

[0156] To generate the inference model used in step S115, the first training image is used, on which image processing for display (step S112) has been performed by the central image processing unit 250. To generate the inference model used in step S120, the second training image is used, on which image processing for display (step S112) has been performed by the peripheral image processing unit 251.

[0157] To generate the inference model used in step S134, a first learning image on which image processing (step S131) ​​has been performed by the central image processing unit 250 and a second learning image on which image processing for display (step S132) has been performed by the peripheral image processing unit 251 are used.

[0158] The learning unit 51 may generate a first inference model through first machine learning using a learning image generated by the imaging unit 130 and a central annotation CA1 (first annotation) indicating an object in a central portion of the learning image (central image CI1) as central image training data D1 (first training data). The learning unit 51 may generate a second inference model through second machine learning using a learning image and a peripheral annotation SA1 (second annotation) indicating an object in a peripheral portion of the learning image (peripheral image SI1) as peripheral image training data D2 (second training data). The processing control unit 24 (acquisition unit) may acquire the first inference model and the second inference model from the recording medium 30. The central object detection unit 252 may use the first inference model acquired by the processing control unit 24 to detect an object from a central image in the image generated by the imaging unit 130. The peripheral object detection unit 253 may use the second inference model acquired by the processing control unit 24 to detect objects from the peripheral image in the image generated by the imaging unit 130.

[0159] Deep learning is a multilayered version of the machine learning process that uses neural networks. A typical example is a forward propagation neural network, which sends information from front to back and makes a decision. The simplest example requires three layers: an input layer consisting of N1 neurons, a hidden layer consisting of N2 neurons determined by parameters, and an output layer consisting of N3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights. Adding bias values ​​to the hidden and output layers facilitates the formation of logic gates. While three layers may be used for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, hidden layers with 9 to 152 layers have become practical in terms of training time, decision accuracy, and energy consumption. Alternatively, a convolutional neural network, which compresses image features and uses minimal processing, is also useful for recognizing motion and patterns. Alternatively, a "recurrent neural network" (fully connected recurrent neural network) can be used, which can handle more complex information and allows information analysis in which the meaning changes depending on the order, with information flowing bidirectionally. Other pattern recognition models that use supervised learning include support vector machines or support vector regression, in which the learning involves calculating the weights, filter coefficients, and offsets of the classifier. Another method uses logistic regression processing.

[0160] To perform learning, a general-purpose arithmetic processing circuit such as a CPU or FPGA (Field Programmable Gate Array) may be used, but because much of the processing in neural networks involves matrix multiplication, circuits specialized for matrix calculations called GPUs (Graphic Processing Units) or Tensor Processing Units (TPUs) may also be used. In recent years, such dedicated artificial intelligence (AI) hardware "neural network processing units (NPUs)" have been designed to be integrated and embeddable with circuits such as CPUs, and may even become part of the processing circuit.

[0161] For example, if an inference model is generated through learning using only images acquired through general photography and aligned horizontally and vertically, and the model is then used, it may not be able to perform correct inference on images with differences in vertical and horizontal orientation. Therefore, accurate judgment is possible by reading the above information from the inference information and performing inference on acquired images taking into account horizontal and vertical information from the orientation sensor. It is effective to incorporate horizontal and vertical information into image judgment before performing inference using the inference model. It is preferable that the endoscope system 1 retains information on these conditions and has a sensor for correcting images. The specifications and performance of the inference engine change depending on whether or not such constraints are included during learning. Therefore, such trial and error may be performed in parallel with the annotation process, and the trial and error results may be displayed.

[0162] Similarly, if learning is performed using only images captured at a specific distance from the object, accurate inference cannot be performed for images captured at distances other than the specific distance. In inference using an inference model generated based on such images, accuracy can be improved by enlarging the image of the distant object and using a pseudo-image of the closer object to compensate for the difference in distance. In this case, a distance sensor or the like is used in combination, and correction is performed during image inference to compensate for the difference between the actual enlargement or reduction state of the image and the state of the training data. The image processing device 20 may be equipped with a memory for storing information indicating the training data used to generate the inference model. When inference using the inference model is performed, the above information may be used to correct the image so that the inference model can correctly perform inference. The user may notice whether such correction is necessary during annotation work. A technique that allows verification of provisional learning during annotation work, as in the present embodiment, is significant.

[0163] It is difficult to train a compact inference engine installed in an information terminal product such as a camera or a mobile device to make highly accurate judgments with a small number of layers. Furthermore, since training takes time, innovations are needed to perform accurate annotation and training. When an inference model is generated, the specifications of the inference model change depending on the images used for training, so efficient training may be performed in conjunction with information at the time of training. Therefore, information indicating the type of training performed may be set during annotation work, and this information may be recorded as part of the inference information in a recording unit of an information acquisition device.

[0164] As explained above, in "supervised learning," the "relationship between input and output" is learned using training data in which the output is defined by annotations, and highly reliable inference is performed under specific conditions. In contrast, the image processing device 20 may acquire an inference model that can handle more complex situations by using an "unsupervised learning" technique that learns the "structure of data."

[0165] The image processing device 20 may also use a technique called "reinforcement learning," which learns "behaviors that maximize value and effectiveness." In this technique, learning is performed to find a law that increases state-action value. The learning is performed by estimating and increasing the value of the next state, not the current one, or by trial and error until a specific reward is obtained. Training data may be used to verify the learning results. In this technique, the correct answer output obtained by annotation is not learned as is, but learning is performed to obtain an even better answer. This makes it possible to adapt to unknown situations.

[0166] These may be used in conjunction with supervised learning, or inference may be performed based on unsupervised learning and then based on supervised learning. The annotation data can also be used as validation data for such "unsupervised learning" and "reinforcement learning."

[0167] When a machine is asked to make a judgment, a human must teach the machine how to make the judgment. Here, we used a method to perform image judgment using machine learning, but a rule-based method in which a human applies rules acquired through experience or heuristics to the judgment may also be used.

[0168] As described above, the image processing device 20 can perform image processing suited to the respective roles of the central and peripheral parts of the image.

[0169] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0170] According to each embodiment of the present invention, the image processing device, image processing method, and program can perform image processing suited to the role of each region of an image.

[0171] REFERENCE SIGNS LIST 1 Endoscope system 10 Scope 11 Operation unit 12 Information acquisition unit 13 Endoscope 20 Image processing device 21 Inspection information generation unit 22 Central determination unit 23 Periphery determination unit 24 Processing control unit 25 Image processing unit 26 Display control unit 30 Recording medium 40 Display unit 50 Inference model generation unit 51 Learning unit 130, 130a Imaging unit 131 Front imaging unit 132, 133 Rear imaging unit 134 Aperture 135 Lens 136 Imaging element 250 Central image processing unit 251 Peripheral image processing unit 252 Central object detection unit 253 Peripheral object detection unit 254 State determination unit 255 Image correction unit 256 Position determination unit

Claims

1. An image processing device comprising: a state determination unit that determines whether the state of an endoscope is a first state in which the endoscope is inserted inside an object of inspection and advanced toward the back of the object of inspection, or a second state other than the first state; a first image processing unit that divides an imaging field of view of an imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region, and, if the state of the endoscope is determined to be in the first state, performs image processing to reduce the visibility of an image of the second region generated by the imaging unit to be lower than the visibility of the image of the first region generated by the imaging unit, and generates an image for display; and a second image processing unit that, if the state of the endoscope is determined to be in the first state, performs image processing to make the object detection performance in the image of the second region the same as or higher than the object detection performance in the image of the first region, and generates an image for object detection.

2. The image processing device according to claim 1, wherein when the state of the endoscope is determined to be the second state, the first image processing unit does not perform the image processing that reduces the visibility of the image of the second region below the visibility of the image of the first region.

3. An image processing device as described in claim 1, comprising: a detection unit which detects an object from an image of the second region in the image for object detection when the state of the endoscope is determined to be the first state; and a display control unit which causes a detection result indicating that the object has been detected to be displayed on a display unit together with the image of the first region on which the image processing was performed when the state of the endoscope is determined to be the first state.

4. The image processing device according to claim 3, wherein the display control unit causes the display unit to display position information indicating the position of the object in the image of the second region as the detection result.

5. An image processing device as described in claim 1, comprising: a detection unit that, when it is determined that the state of the endoscope is the first state, detects an object from an image of the second region in the image for object detection, and stores in a recording medium first position information indicating the position of the object in the image of the second region and second position information indicating the position of the object in the object to be inspected; and a display control unit that, when it is determined that the state of the endoscope is the second state and the position of the endoscope coincides with the position indicated by the second position information, causes the first position information to be displayed on a display unit together with an image generated by the imaging unit.

6. The image processing device according to claim 1, wherein, when the state of the endoscope is determined to be the first state, the first image processing unit deletes the image of the second region.

7. An image processing device as described in claim 6, further comprising a detection unit that detects an object from an image of the second region in the image for object detection when the state of the endoscope is determined to be the first state, and when the object is not detected, the first image processing unit deletes the image of the second region, and when the object is detected, the first image processing unit does not delete the image of the second region.

8. An image processing device as described in claim 6, further comprising a detection unit which detects an object from an image of the second region in the image for object detection when the state of the endoscope is determined to be the first state, and when the object approaches the center of the imaging field of view in a plurality of images continuously generated by the imaging unit, the first region in the plurality of images gradually becomes larger and the second region in the plurality of images gradually becomes smaller.

9. The image processing device according to claim 1, wherein, when the state of the endoscope is determined to be the first state, the first image processing unit performs a thinning process on the image of the second region.

10. The image processing device according to claim 1, further comprising a correction unit that corrects the image generated by the imaging unit based on the optical characteristics of the endoscope.

11. An image processing device comprising: a first image processing unit that performs a first image processing on an image of the first area generated by the imaging unit when the imaging field of view of an endoscope is divided into a first area including the center of the imaging field of view and a second area other than the first area; a second image processing unit that performs a second image processing on an image of the second area generated by the imaging unit; a detection unit that detects an object in the image on which the second image processing has been performed; and a display control unit that displays the detection result of the object on a display unit together with the image of the first area on which the first image processing has been performed.

12. The image processing device described in claim 11, wherein the boundary between the first area and the second area is switchable between a first state in which the endoscope is inserted inside the object of examination and advanced toward the back of the object of examination, and a second state in which the endoscope is removed from the object of examination.

13. An image processing device comprising: an acquisition unit that acquires from a recording medium an inference model obtained through machine learning using, when the imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region, a first learning image generated by the imaging unit and subjected to first image processing, and a first annotation indicating an object in the first region of the first learning image as first teacher data, and a second learning image generated by the imaging unit and subjected to second image processing, and a second annotation indicating an object in the second region of the second learning image as second teacher data; and a detection unit that uses the inference model acquired by the acquisition unit to detect an object in an image generated by the imaging unit.

14. An image processing device comprising: an acquisition unit that, when the imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view and a second region other than the first region, acquires from a recording medium a first inference model obtained through a first machine learning that uses, as first teacher data, a learning image generated by the imaging unit and a first annotation indicating an object in the first region of the learning image, and acquires from the recording medium a second inference model obtained through a second machine learning that uses, as second teacher data, the learning image and a second annotation indicating an object in the second region of the learning image; and a detection unit that uses the first inference model and the second inference model acquired by the acquisition unit to detect an object in an image generated by the imaging unit.

15. An image processing method comprising: determining whether the state of an endoscope is a first state in which the endoscope is inserted into an object to be examined and advanced toward the back of the object to be examined, or a second state other than the first state; dividing an imaging field of view of an imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region; if the state of the endoscope is determined to be in the first state, performing image processing to reduce the visibility of an image of the second region generated by the imaging unit to be lower than the visibility of an image of the first region generated by the imaging unit, and generating an image for display; if the state of the endoscope is determined to be in the first state, performing image processing to make the object detection performance in the image of the second region the same as or higher than the object detection performance in the image of the first region, and generating an image for object detection.

16. An image processing method comprising the steps of: dividing the imaging field of view of an imaging unit of an endoscope into a first region including the center of the imaging field of view, and a second region other than the first region; executing a first image processing on an image of the first region generated by the imaging unit; executing a second image processing on an image of the second region generated by the imaging unit; detecting an object in the image on which the second image processing has been performed; and displaying the detection result of the object on a display unit together with the image of the first region on which the first image processing has been performed.

17. An image processing method, in which when the imaging field of view of an imaging unit of an endoscope is divided into a central region including the center of the imaging field of view and a peripheral region other than the central region, performing image processing for display on an image of the central region generated by the imaging unit, detecting an object in an image of the peripheral region generated by the imaging unit, and displaying the detection result of the object on a display unit that displays the image of the central region.

18. The image processing method according to claim 17, wherein the image of the central region is a rectangle, and the object detection result is an icon including at least one of information indicating the presence or absence of the object, information indicating the type of the object, and information indicating a direction from the center of the rectangle toward one of the four sides of the rectangle that is closest to the position where the object was detected.

19. A program for causing a computer to execute the steps of: determining whether the state of an endoscope is a first state in which the endoscope is inserted inside an object to be examined and advanced toward the back of the object to be examined, or a second state other than the first state; dividing the imaging field of view of an imaging unit of the endoscope into a first region including the center of the imaging field of view and a second region other than the first region, and, if it is determined that the state of the endoscope is in the first state, executing image processing to reduce the visibility of the image of the second region generated by the imaging unit below the visibility of the image of the first region generated by the imaging unit, and generating an image for display; and, if it is determined that the state of the endoscope is in the first state, executing image processing to make the object detection performance in the image of the second region the same as or higher than the object detection performance in the image of the first region, and generating an image for object detection.

20. A program for causing a computer to execute the steps of: when the imaging field of view of an imaging unit of an endoscope is divided into a first region including the center of the imaging field of view, and a second region other than the first region; performing a first image processing on an image of the first region generated by the imaging unit; performing a second image processing on an image of the second region generated by the imaging unit; detecting an object in the image on which the second image processing has been performed; and displaying the result of the detection of the object on a display unit together with the image of the first region on which the first image processing has been performed.

Images