Endoscopic surgery system, image processing device, and image processing method

Abstract

The present disclosure relates to an endoscope operation system, an image processing device, and an image processing method which perform observation more excellently by means of an endoscope. A motion component at the time of image capturing is estimated on the basis of an image captured by an endoscope, the reliability is set which indicates the degree of the motion component being a motion component for the entire image, and a correction amount used when a correction process for correcting a blur is performed for the image is controlled according to the reliability. For example, when the motion component is a single motion component, the reliability is set to 1.0, and when the motion component is multiple motion components, the reliability is set to 0.0. The present technology can be applied to, for example, an endoscope operation system used for a surgical operation using an endoscope.

Description

[Technical field]

[0001] The present disclosure relates to an endoscopic surgery system, an image processing device, and an image processing method, and more particularly to an endoscopic surgery system, an image processing device, and an image processing method that enable better endoscopic observation. [Background technology]

[0002] 2. Description of the Related Art In recent years, in medical settings, endoscopic surgery systems that enable surgical operations to be performed while observing an affected area with an endoscope have come to be used in place of conventional open surgery.

[0003] For example, Patent Document 1 discloses an endoscopic device that performs blur correction processing in accordance with movement information of the imaging field of view in response to an operation to move the imaging field of view of an imaging unit and the movement between multiple images acquired in time series. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2012-239644 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the endoscope device disclosed in the above-mentioned Patent Document 1, for example, when the ratio of the motion component of the imaging unit to the motion component of a medical instrument other than the imaging unit is close, it is difficult to correctly estimate the motion component contained in the image. Therefore, as a result of the blur correction process being performed based on an erroneously estimated motion, there is a concern that distortion will occur in the image after the blur correction process, and it is assumed that such an image will not allow good observation.

[0006] The present disclosure has been made in consideration of the above circumstances, and is intended to enable better observation using an endoscope. [Means for solving the problem]

[0007] According to one aspect of the present disclosure, there is provided an endoscopic surgery system and an image processing device, comprising: a motion estimating unit that estimates a motion component at the time of imaging based on an image captured by an endoscope and sets a reliability indicating a degree to which the motion component is a motion component relative to the entire image; and a correction amount control unit that controls a correction amount used when performing a correction process to correct blurring of the image in accordance with the reliability. a calculation unit that calculates norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculates a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; Equipped with. Then, the motion estimation unit refers to a threshold curve represented by a predetermined threshold, with the norm standard deviation on the vertical axis and the angle standard deviation on the horizontal axis, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region below the threshold curve, estimates that the image contains a single movement component and sets a first value as the reliability, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region equal to or greater than the threshold curve, estimates that the image contains multiple movement components and sets a second value as the reliability. Alternatively, a first threshold curve and a second threshold curve represented by the thresholds are referenced with the norm standard deviation on the vertical axis and the angle standard deviation on the horizontal axis, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region less than the first threshold curve, it is presumed that the image contains a single movement component and a first value is set as the reliability, if the norm standard deviation and the angle standard deviation obtained from the image are in a region greater than the first threshold curve and less than the second threshold curve, it is presumed that the image contains multiple movement components and the second value is set as the reliability.

[0008] An image processing method according to one aspect of the present disclosure includes an image processing device estimating a motion component at the time of imaging based on an image captured by an endoscope, setting a reliability indicating a degree to which the motion component is a motion component relative to the entire image, and controlling a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability. calculating norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculating a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; Includes. Then, with the norm standard deviation on the vertical axis and the angle standard deviation on the horizontal axis, a threshold curve represented by a predetermined threshold is referenced, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region below the threshold curve, it is presumed that the image contains a single movement component and a first value is set as the reliability, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region equal to or greater than the threshold curve, it is presumed that the image contains multiple movement components and a second value is set as the reliability. Alternatively, a first threshold curve and a second threshold curve represented by the thresholds are referenced with the norm standard deviation on the vertical axis and the angle standard deviation on the horizontal axis, and if the norm standard deviation and the angle standard deviation obtained from the image are in a region less than the first threshold curve, it is presumed that the image contains a single movement component and a first value is set as the reliability, if the norm standard deviation and the angle standard deviation obtained from the image are in a region greater than the first threshold curve and less than the second threshold curve, it is presumed that the image contains multiple movement components and the second value is set as the reliability.

[0009] In one aspect of the present disclosure, a motion component at the time of imaging is estimated based on an image captured by an endoscope, a reliability indicating a degree to which the motion component is a motion component relative to the entire image is set, and a correction amount used when performing correction processing on the image is controlled according to the reliability. Based on the local motion vectors indicating local motion of the image, the norm and angle of the local motion vectors are calculated, and the norm standard deviation of the local motion vector group and the angle standard deviation of the local motion vector group are calculated. Then, a threshold curve represented by a predetermined threshold is referenced with the norm standard deviation as the vertical axis and the angle standard deviation as the horizontal axis, and if the norm standard deviation and angle standard deviation calculated from the image are in an area below the threshold curve, it is estimated that the image contains a single motion component, and a first value is set as the reliability, and if the norm standard deviation and angle standard deviation calculated from the image are in an area equal to or greater than the threshold curve, it is estimated that the image contains multiple motion components, and a second value is set as the reliability. Alternatively, a first threshold curve and a second threshold curve represented by thresholds are referenced with the norm standard deviation on the vertical axis and the angle standard deviation on the horizontal axis, and if the norm standard deviation and angle standard deviation obtained from the image are in a region less than the first threshold curve, it is presumed that the image contains a single movement component and a first value is set as the reliability, if the norm standard deviation and angle standard deviation obtained from the image are in a region greater than the first threshold curve and less than the second threshold curve, a value between the first value and a second value is set as the reliability, and if the norm standard deviation and angle standard deviation obtained from the image are in a region greater than the second threshold curve, it is presumed that the image contains multiple movement components and a second value is set as the reliability. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a configuration example of an embodiment of an endoscopic surgery system to which the present technology is applied; [Diagram 2] FIG. 1 is a block diagram showing an example of the configuration of an image processing device. [Diagram 3] FIG. 13 is a diagram showing a first example of a threshold curve referenced in estimating a motion component. [Figure 4] 11 is a flowchart illustrating a first processing example of image processing. [Diagram 5] FIG. 13 is a diagram showing a second example of a threshold curve referred to in estimating a motion component. [Figure 6]FIG. 13 is a diagram illustrating the reliability obtained in the transition zone. [Figure 7] 11 is a flowchart illustrating a second example of image processing. [Figure 8] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. [Figure 9] 1 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system. [Figure 10] 2 is a block diagram showing an example of a functional configuration of a camera head and a CCU. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.

[0012] <Example of endoscopic surgery system configuration> FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an endoscopic surgery system to which the present technology is applied.

[0013] The endoscopic surgery system 11 shown in FIG. 1 is configured to include an endoscope 12, an energy treatment tool 13, a display device 14, and an equipment unit 15.

[0014] For example, in surgery using the endoscopic surgery system 11, the endoscope 12 and the energy treatment tool 13 are inserted into the patient's body, and the forceps 16 is also inserted into the patient's body. In the endoscopic surgery system 11, an image of an affected area such as a tumor captured by the endoscope 12 is displayed in real time on the display device 14, and a doctor can treat the affected area using the energy treatment tool 13 and the forceps 16 while viewing the image.

[0015] The endoscope 12 is configured by, for example, attaching a cylindrical lens barrel portion incorporating an optical system such as an objective lens to a camera head equipped with an image pickup element and the like.

[0016] The energy treatment device 13 is a medical instrument used in endoscopic surgery, for example, to remove an affected area or seal a blood vessel by using heat generated by a high-frequency current.

[0017] The display device 14 displays an image obtained by image capture by the endoscope 12 and subjected to image processing in the device unit 15 .

[0018] The equipment unit 15 includes an image processing device 21 (FIG. 2) that performs image processing on the image captured by the endoscope 12, as well as various devices required for performing surgery using the endoscopic surgery system 11. For example, the equipment unit 15 incorporates a CCU (Camera Control Unit) that controls the imaging by the endoscope 12, a light source device that supplies light to be irradiated onto the affected area when the endoscope 12 captures images, a treatment tool device that supplies high-frequency current required for treatment of the affected area by the energy treatment tool 13, and the like.

[0019] In the endoscopic surgery system 11 configured in this manner, for example, image processing is performed in the device unit 15 to appropriately correct blurring occurring in an image captured by the endoscope 12, and the image with the blur appropriately corrected is displayed on the display device 14. Therefore, in surgery using the endoscopic surgery system 11, a doctor or the like can perform treatment on the affected area while observing the image with the blur appropriately corrected.

[0020] <Example of image processing device configuration> FIG. 2 is a block diagram showing an example of the configuration of the image processing device 21 included in the device unit 15. As shown in FIG.

[0021] As shown in FIG. 2, the image processing device 21 includes a feature point extraction unit 22, a local motion vector detection unit 23, a global motion amount estimation unit 24, a calculation unit 25, a motion estimation unit 26, a correction amount control unit 27, and a correction processing unit .

[0022] 1 is input to the image processing device 21. In addition to the overall movement (overall blur) that may occur in this image due to the movement of the endoscope 12, when the energy treatment tool 13, the forceps 16, etc. are imaged by the endoscope 12, local movement (local blur) may occur due to the movement of the energy treatment tool 13, the forceps 16, etc.

[0023] The feature point extraction unit 22 extracts feature points that are points that indicate characteristic locations in an image captured by the endoscope 12, and supplies the multiple extracted points extracted from the image to the local motion vector detection unit .

[0024] The local motion vector detection unit 23 detects a local motion vector indicating a motion occurring in a local portion of the image, based on the motion of each of the multiple feature points supplied from the feature point extraction unit 22. Then, the local motion vector detection unit 23 supplies the multiple local motion vectors detected from the image captured by the endoscope 12 of FIG.

[0025] The global motion amount estimating section 24 estimates a global motion amount that indicates the magnitude of the overall motion of the image based on the multiple local motion vectors supplied from the local motion vector detecting section 23 , and supplies the estimated global motion amount to a correction amount control section 27 .

[0026] The calculation unit 25 performs calculations based on the multiple local motion vectors supplied from the local motion vector detection unit 23. For example, the calculation unit 25 performs calculations to obtain the norm and angle of each local motion vector. The calculation unit 25 then calculates a standard deviation indicating the degree of dispersion of the norms of each local motion vector (hereinafter referred to as the norm standard deviation of the local motion vector group), and supplies it to the motion estimation unit 26. The calculation unit 25 also calculates a standard deviation indicating the degree of dispersion of the angles of each local motion vector (hereinafter referred to as the angle standard deviation of the local motion vector group), and supplies it to the motion estimation unit 26.

[0027] The motion estimation unit 26 estimates the motion components at the time of imaging based on the image to be processed in the image processing by referring to the threshold curve th shown in FIG. 3 described later according to the norm standard deviation and angle standard deviation of the group of local motion vectors calculated by the calculation unit 25. That is, the motion estimation unit 26 estimates whether the image contains a single motion component or multiple motion components. For example, if the motion components contained in the image to be processed are dominated by a single magnitude and direction, it is estimated that the image contains a single motion component. On the other hand, if the motion components contained in the image to be processed are dominated by multiple different magnitudes and directions, it is estimated that the image contains multiple motion components. Then, the motion estimation unit 26 sets a reliability indicating the degree to which it is reliable that the motion components contained in the image are due to the movement of the endoscope 12 and the occurrence of motion in the entire image, according to the estimation result of the motion components, and supplies the reliability to the correction amount control unit 27.

[0028] For example, when the image to be processed contains a single motion component according to the motion component estimation result, the motion estimation unit 26 sets the reliability to 1.0, assuming that the motion component contained in the image is highly likely to be caused by motion occurring relative to the entire image. On the other hand, when the image to be processed contains multiple motion components according to the motion component estimation result, the motion estimation unit 26 sets the reliability to 0.0, assuming that the motion component contained in the image is low in likelihood of being caused by motion occurring relative to the entire image, for example, due to motion of the energy treatment tool 13 or the forceps 16. Note that each reliability is not limited to 0.0 or 1.0, and may be any value between 0.0 and 1.0, for example.

[0029] The correction amount control unit 27 controls the global motion amount supplied from the global motion amount estimation unit 24 in accordance with the reliability supplied from the motion estimation unit 26, and determines the correction amount used when performing correction processing for correcting blur on the image to be processed in the correction processing unit 28. For example, when the reliability supplied from the motion estimation unit 26 is 1.0, the correction amount control unit 27 uses the global motion amount as is as the correction amount and supplies it to the correction processing unit 28. On the other hand, when the reliability supplied from the motion estimation unit 26 is 0.0, the correction amount control unit 27 does not use the global motion amount as the correction amount, and supplies the correction amount as 0 to the correction processing unit 28.

[0030] The correction processing unit 28 performs a correction process to correct blurring occurring in the image to be processed input to the image processing device 21 with a correction amount according to the control by the correction amount control unit 27, and outputs the result to the display device 14. For example, when the correction processing unit 28 receives a correction amount obtained by controlling the global motion amount according to the reliability from the correction amount control unit 27, the correction processing unit 28 corrects blurring of the image by smoothing the image in a time direction according to the correction amount. Therefore, for example, when the correction amount supplied from the correction amount control unit 27 is 0, the correction processing unit 28 does not perform a correction process on the image to be processed, and outputs the image as is.

[0031] That is, when the image to be processed contains a single movement component, the correction processing unit 28 performs a correction process to correct blur according to the global movement amount, and when the image to be processed contains multiple movement components, the correction processing is turned off.

[0032] The image processing device 21 configured as above can perform an appropriate correction process depending on whether the motion component contained in the image to be processed is a single motion component or multiple motion components. For example, when the motion component contained in the image to be processed is highly likely to be caused by motion occurring in the entire image, the image processing device 21 can output an image in which the overall blur occurring in the image due to the motion of the endoscope 12 has been corrected.

[0033] Furthermore, the image processing device 21 can avoid performing inappropriate correction processing when the degree of movement components contained in the image to be processed is low and the degree of local blurring in the image is high. For example, when the degree of local movement due to the energy treatment tool 13, the forceps 16, etc. is high, the estimation of the global motion amount is strongly influenced by the local blurring. In this case, the estimated global motion amount does not correctly reflect the overall blurring, and therefore distortion occurs in the correction result not only in the area of ​​local blurring but also overall.

[0034] Therefore, when the motion components contained in the image to be processed are largely due to the occurrence of multiple motions, the image processing device 21 can avoid distortion occurring in the correction result as a whole, not just in the area of ​​local blur. In other words, when an image contains multiple motion components, a correction process for correcting blur according to the global motion amount is not performed, and thus it is possible to avoid the execution of an inappropriate correction process.

[0035] Therefore, the surgeon performing the surgery can make an observation using an image in which the overall blur caused by the movement of the endoscope 12 itself has been corrected, and can avoid making an observation using a distorted image that occurs when correcting local blur caused by the movement of the energy treatment tool 13, forceps 16, etc., and can make a better endoscopic observation with good image quality that does not feel unnatural.

[0036] Now, the estimation of the motion component by the motion estimator 26 will be described with reference to FIG.

[0037] As described above, the norm standard deviation and the angle standard deviation of the group of local motion vectors calculated by the calculation unit 25 are supplied to the motion estimation unit 26. The motion estimation unit 26 is set with a threshold curve th indicating a threshold for estimating whether the image to be processed contains a single motion component or multiple motion components.

[0038] 3, the vertical axis indicates the norm standard deviation of the local motion vector group, the horizontal axis indicates the angle standard deviation of the local motion vector group, and a threshold curve th is set as shown in the figure. For example, the threshold curve th is expressed as a curve in which the angle standard deviation decreases as the norm standard deviation increases, and the angle standard deviation increases as the norm standard deviation decreases, and is convex in the direction in which the norm standard deviation and the angle standard deviation decrease.

[0039] That is, when an image contains a single motion component, the local motion vectors tend to be aligned in a certain magnitude and direction, resulting in small norm and angle standard deviations, whereas when an image contains multiple motion components, the local motion vectors are divided into multiple groups each pointing in a different direction, resulting in large norm and angle standard deviations.

[0040] As a result, the motion estimation unit 26 can estimate that a single motion component is included in an image to be processed when the norm standard deviation and angle standard deviation of a group of local motion vectors calculated from the image to be processed are in a region less than the threshold curve th shown in Fig. 3 (i.e., a region closer to the origin than the threshold curve th).On the other hand, the motion estimation unit 26 can estimate that a multiple motion component is included in an image to be processed when the norm standard deviation and angle standard deviation of a group of local motion vectors calculated from the image to be processed are in a region equal to or greater than the threshold curve th shown in Fig. 3.

[0041] Therefore, when the degree of norm dispersion and the degree of angle dispersion of the multiple local motion vectors obtained from the image to be processed are small, it is estimated that the image to be processed contains a single motion component, and the reliability is set to 1.0. On the other hand, when the degree of norm dispersion and the degree of angle dispersion of the multiple local motion vectors obtained from the image to be processed are large, it is estimated that the image to be processed contains multiple motion components, and the reliability is set to 0.0.

[0042] <First example of image processing> FIG. 4 is a flowchart illustrating a first processing example of the image processing executed in the image processing device 21.

[0043] For example, when an image captured by the endoscope 12 is input to the image processing device 21, image processing is started with that image as the processing target, and in step S11, the feature point extraction unit 22 extracts multiple feature points from the image to be processed.

[0044] In step S12, the local motion vector detection unit 23 detects a plurality of local motion vectors from the image to be processed, in accordance with the plurality of feature points extracted by the feature point extraction unit 22 in step S11.

[0045] In step S13, the global motion amount estimating section 24 estimates a global motion amount from the image to be processed, in accordance with the multiple local motion vectors detected by the local motion vector detecting section 23 in step S12.

[0046] In step S14, the calculation unit 25 calculates the norm and angle of each of the multiple local motion vectors detected by the local motion vector detection unit 23 in step S12.

[0047] In step S15, the calculation unit 25 calculates the norm standard deviation of the local motion vector group based on the norm of each local motion vector, and calculates the angle standard deviation of the local motion vector group based on the angle of each local motion vector.

[0048] In step S16, the motion estimation unit 26 refers to the threshold curve th shown in FIG. 3 and estimates the motion components contained in the image to be processed, in accordance with the norm standard deviation and angle standard deviation of the group of local motion vectors calculated by the calculation unit 25 in step S15.

[0049] In step S17, the motion estimation unit 26 determines whether the motion component estimation result in step S16 is a single motion component or multiple motion components.

[0050] In step S17, if the motion estimation unit 26 determines that the estimation result of the motion component is a single motion component, the process proceeds to step S 18. Then, in step S18, the motion estimation unit 26 sets the reliability to 1.0.

[0051] On the other hand, in step S17, if the motion estimation unit 26 determines that the estimation result of the motion components is a plurality of motion components, the process proceeds to step S19. Then, in step S19, the motion estimation unit 26 sets the reliability to 0.0.

[0052] After processing step S18 or S19, the process proceeds to step S20, where the correction amount control unit 27 controls the correction amount based on the global motion amount estimated by the global motion amount estimation unit 24 in step S13, in accordance with the reliability set by the motion estimation unit 26.

[0053] In step S21, the correction processing unit 28 performs a correction process to correct blur on the image to be processed with the correction amount controlled by the correction amount control unit 27 in step S20. That is, if the reliability set by the motion estimating unit 26 is 1.0, the correction process is performed to correct blur with a correction amount based on the global motion amount, and if the reliability set by the motion estimating unit 26 is 0.0, the correction process is turned off. Then, the correction processing unit 28 outputs the image obtained by performing the correction process or the image with the correction process turned off to the display device 14, and the process ends.

[0054] As described above, the image processing device 21 can appropriately perform (switch on / off) the correction process for correcting image blur by determining whether the image to be processed contains a single motion component or multiple motion components. This allows the image processing device 21 to output an image with blur corrected when the image to be processed contains a single motion component, and to output an image without blur correction (avoiding the execution of inappropriate correction process) when the image to be processed contains multiple motion components.

[0055] <Second example of image processing> A second processing example of the image processing executed in the image processing device 21 will be described with reference to FIGS.

[0056] In the first processing example of the image processing described above, the image processing device 21 sets the reliability to 0.0 or 1.0 according to the threshold curve th shown in Fig. 3, and switches the image processing for correcting blur on and off. In contrast, in the second processing example of the image processing, a transition band that transitions the reliability between 0.0 and 1.0 is provided, making it possible to adjust the strength of blur correction.

[0057] That is, a first threshold curve th1 and a second threshold curve th2 are set as shown in Fig. 5. The first threshold curve th1 indicates a threshold for estimating whether or not a single movement component is included in the image to be processed, and the second threshold curve th2 indicates a threshold for estimating whether or not a multiple movement component is included in the image to be processed.

[0058] Therefore, when the norm standard deviation and angle standard deviation of the local motion vector group calculated from the image to be processed are in a region less than the first threshold curve th1, the motion estimation unit 26 estimates that the image contains a single motion component and sets the reliability to 1.0. On the other hand, when the norm standard deviation and angle standard deviation of the local motion vector group calculated from the image to be processed are in a region greater than the second threshold curve th2, the motion estimation unit 26 estimates that the image contains multiple motion components and sets the reliability to 0.0.

[0059] Then, when the norm standard deviation and angle standard deviation of the group of local motion vectors calculated from the image to be processed are in a region greater than or equal to the first threshold curve th1 and less than or equal to the second threshold curve th2, the motion estimation unit 26 determines that this is a transition zone between a single motion component and multiple motion components, and sets the reliability to a value between 0.0 and 1.0.

[0060] For example, as shown in FIG. 6, the motion estimation unit 26 can calculate the reliability in the transition zone so that the reliability decreases linearly depending on the values ​​of the norm standard deviation and the angle standard deviation of the local motion vector group from a reliability of 1.0 at the first threshold curve th1 to a reliability of 0.0 at the second threshold curve th2.

[0061] Then, the correction amount control unit 27 can control the correction amount according to the reliability set by the motion estimation unit 26. For example, when the reliability is set to 0.0 and 1.0, the correction amount control unit 27 controls the correction amount in the same manner as the first processing example of the image processing described above. When the reliability is set between 0.0 and 1.0, the correction amount control unit 27 uses the reliability to calculate a correction amount that interpolates between when the correction process is on and when it is off. For example, the correction amount control unit 27 may simply calculate a value obtained by multiplying the correction amount by the reliability as the correction amount according to the motion component.

[0062] By using the second processing example of such image processing, the image processing device 21 can output an image that looks more natural.

[0063] For example, as in the first processing example of image processing, when the reliability is set to 1.0 or 0.0 and the correction process is controlled to be switched on and off, the correction process may be switched on and off frequently. That is, when the reliability that changes with the progression of frames is present near the threshold curve th (FIG. 3) and the estimation result changes between a single movement component and multiple movement components for each frame or every few frames, the correction process will be switched on and off frequently. In such a case, it is expected that the image will become jerky over time, causing an unnatural feeling.

[0064] In contrast to this, in the second processing example of image processing, the reliability is set to a value between 0.0 and 1.0 in the transition band, so that it is possible to avoid frequent on / off switching of the correction process, thereby mitigating such adverse effects and reducing the sense of incongruity.

[0065] FIG. 7 is a flowchart illustrating a second processing example of the image processing executed in the image processing device 21.

[0066] In steps S31 to S35, the same processes as in steps S11 to S15 in Fig. 4 are performed. Then, in step S36, the motion estimation unit 26 estimates a motion component contained in the image to be processed by referring to the first threshold curve th1 and the second threshold curve th2 shown in Fig. 5 described above, in accordance with the norm standard deviation and the angle standard deviation of the group of local motion vectors calculated by the calculation unit 25 in step S35.

[0067] In step S37, the motion estimation unit 26 determines whether the motion component estimation result in step S36 is a single motion component, multiple motion components, or a transition band.

[0068] In step S37, if the motion estimation unit 26 determines that the estimation result of the motion component is a single motion component, the process proceeds to step S38. Then, in step S38, the motion estimation unit 26 sets the reliability to 1.0.

[0069] On the other hand, if the motion estimation unit 26 determines in step S37 that the estimation result of the motion components is a plurality of motion components, the process proceeds to step S39. Then, in step S39, the motion estimation unit 26 sets the reliability to 0.0.

[0070] On the other hand, if the motion estimation unit 26 determines in step S37 that the estimated result of the motion component is a transition zone, the process proceeds to step S40. Then, in step S40, the motion estimation unit 26 sets the reliability to a value between 0.0 and 1.0, as shown in FIG. 6 above.

[0071] After processing step S38, step S39, or step S40, the process proceeds to step S41, where the correction amount control unit 27 controls the correction amount based on the global motion amount estimated by the global motion amount estimation unit 24 in step S33, in accordance with the reliability set by the motion estimation unit 26.

[0072] In step S42, the correction processing unit 28 performs a correction process to correct blur on the image to be processed with the correction amount controlled by the correction amount control unit 27 in step S41. That is, when the reliability set by the motion estimation unit 26 is 1.0, the correction process is performed to correct blur with a correction amount based on the global motion amount, and when the reliability set by the motion estimation unit 26 is 0.0, the correction process is turned off. When the reliability set by the motion estimation unit 26 is a value between 1.0 and 0.0, the correction process is performed to correct blur with a correction amount according to the reliability. Then, the correction processing unit 28 outputs the image obtained by performing the correction process or the image with the correction process turned off to the display device 14, and then the process ends.

[0073] As described above, the image processing device 21 can appropriately perform a correction process for correcting image blur by determining whether the image to be processed contains a single motion component or multiple motion components, or is in a transition zone between a single motion component and multiple motion components. Therefore, the image processing device 21 can mitigate the adverse effects caused by the switching between a single motion component and multiple motion components between frames as described above, and output an image with reduced awkwardness.

[0074] In the image processing device 21, the correction amount control unit 27 may accumulate the reliability successively supplied from the motion estimation unit 26 and smooth the reliability in the time direction. Then, the correction amount control unit 27 can control the correction amount so as to change smoothly using the reliability smoothed in the time direction. This enables the image processing device 21 to mitigate the adverse effect caused by the switching between a single movement component and multiple movement components between frames as described above, and output an image with less unnaturalness.

[0075] In the image processing device 21, the threshold curve th shown in Fig. 3 or the first threshold curve th1 or the second threshold curve th2 shown in Fig. 5 may be adjusted so as to appropriately estimate the movement component contained in the image without being limited to the shapes as shown in the figures. For example, the threshold curve can be adjusted according to the characteristics (zoom magnification, aperture, etc.) of the endoscope 12, and can be adjusted according to the type and circumstances of surgery in which the endoscope 12 is used. Furthermore, the image processing device 21 can be used for image processing of images of a surgical system that does not use the endoscope 12, for example, images captured by a surgical microscope.

[0076] As described above, the endoscopic surgery system 11 can display an image on the display device 14 that has been appropriately subjected to a correction process for correcting blur by estimating the movement components at the time of imaging based on the image to be processed. For example, when the movement of the endoscope 12 occupies a large proportion of the screen, the endoscopic surgery system 11 can correct the overall blur. Furthermore, when the movement of the energy treatment tool 13, the forceps 16, gauze, etc., or the movement within the living body such as pulsation and respiratory movement occupies a large proportion of the screen, the endoscopic surgery system 11 can prevent the occurrence of distortion caused by the correction process being performed due to the movement of these.

[0077] Therefore, in the endoscopic surgery system 11, an image with appropriately corrected blur is displayed on the display device 14, allowing the doctor to properly observe the affected area using the image captured by the endoscope 12.

[0078] <Computer configuration example> Next, the above-mentioned series of processes (image processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

[0079] FIG. 8 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.

[0080] The program can be recorded in advance on the hard disk 105 or the ROM 103 as a recording medium built into the computer.

[0081] Alternatively, the program can be stored (recorded) in a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.

[0082] The program can be installed in the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.

[0083] The computer includes a built-in CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .

[0084] When a command is input by a user operating an input unit 107 via an input / output interface 110, the CPU 102 executes a program stored in a ROM (Read Only Memory) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored in a hard disk 105 into a RAM (Random Access Memory) 104 and executes the program.

[0085] As a result, CPU 102 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. CPU 102 then outputs the processing result from output unit 106 via input / output interface 110, transmits the result from communication unit 108, and further records the result on hard disk 105, as necessary.

[0086] The input unit 107 includes a keyboard, a mouse, a microphone, etc. The output unit 106 includes an LCD (Liquid Crystal Display), a speaker, etc.

[0087] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).

[0088] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.

[0089] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0090] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0091] Furthermore, for example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0092] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0093] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0094] In addition, the processing of the steps of the program executed by the computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a necessary timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the processing of the steps of the program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0095] In addition, the present technologies described in this specification can be implemented independently and individually, as long as there is no contradiction. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.

[0096] <Application example to endoscopic surgery system> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

[0097] FIG. 9 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology according to the present disclosure (the present technology) can be applied.

[0098] 9 shows a state in which an operator (doctor) 11131 is performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgery system 11000. As shown in the figure, the endoscopic surgery system 11000 is composed of an endoscope 11100, other surgical tools 11110 such as an insufflation tube 11111 and an energy treatment tool 11112, a support arm device 11120 that supports the endoscope 11100, and a cart 11200 on which various devices for endoscopic surgery are mounted.

[0099] The endoscope 11100 is composed of a lens barrel 11101, a region of a predetermined length from the tip of which is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to the base end of the lens barrel 11101. In the illustrated example, the endoscope 11100 is configured as a so-called rigid scope having a rigid lens barrel 11101, but the endoscope 11100 may be configured as a so-called flexible scope having a flexible lens barrel.

[0100] An opening into which an objective lens is fitted is provided at the tip of the lens barrel 11101. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extending inside the lens barrel 11101, and is irradiated via the objective lens toward an observation target in the body cavity of the patient 11132. The endoscope 11100 may be a direct-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0101] An optical system and an image sensor are provided inside the camera head 11102, and reflected light (observation light) from an observation target is collected on the image sensor by the optical system. The observation light is photoelectrically converted by the image sensor to generate an electrical signal corresponding to the observation light, i.e., an image signal corresponding to an observation image. The image signal is transmitted to a camera control unit (CCU) 11201 as RAW data.

[0102] The CCU 11201 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and performs overall control of the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102, and performs various types of image processing on the image signal, such as development processing (demosaic processing), for displaying an image based on the image signal.

[0103] Under the control of the CCU 11201, the display device 11202 displays an image based on an image signal that has been subjected to image processing by the CCU 11201.

[0104] The light source device 11203 is composed of a light source such as an LED (light emitting diode), and supplies the endoscope 11100 with irradiation light when photographing an operation site or the like.

[0105] The input device 11204 is an input interface for the endoscopic surgery system 11000. A user can input various information and instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging conditions (type of irradiated light, magnification, focal length, etc.) of the endoscope 11100.

[0106] The treatment tool control device 11205 controls the driving of the energy treatment tool 11112 for cauterizing tissue, incising, sealing blood vessels, etc. The insufflation device 11206 sends gas into the body cavity of the patient 11132 via the insufflation tube 11111 to inflate the body cavity for the purpose of securing the field of view of the endoscope 11100 and securing the working space of the surgeon. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, image, or graph.

[0107] The light source device 11203 that supplies irradiation light to the endoscope 11100 when photographing the surgical site can be composed of a white light source composed of, for example, an LED, a laser light source, or a combination of these. When the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to capture images corresponding to each of the RGB colors in a time-division manner by irradiating the observation object with laser light from each of the RGB laser light sources in a time-division manner and controlling the driving of the image sensor of the camera head 11102 in synchronization with the irradiation timing. According to this method, a color image can be obtained without providing a color filter to the image sensor.

[0108] The light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. The driving of the image sensor of the camera head 11102 may be controlled in synchronization with the timing of the change in the light intensity to obtain images in a time-division manner, and the images may be synthesized to generate an image with a high dynamic range that is free of so-called blackout and whiteout.

[0109] The light source device 11203 may be configured to supply light of a predetermined wavelength band corresponding to the special light observation. In the special light observation, for example, by utilizing the wavelength dependency of light absorption in body tissue, a narrow band light is irradiated compared to the irradiated light (i.e., white light) during normal observation, and a predetermined tissue such as blood vessels on the mucous membrane surface is photographed with high contrast, so-called narrow band imaging is performed. Alternatively, in the special light observation, a fluorescent observation may be performed in which an image is obtained by fluorescence generated by irradiating an excitation light. In the fluorescent observation, it is possible to irradiate an excitation light to a body tissue and observe the fluorescence from the body tissue (autofluorescence observation), or to locally inject a reagent such as indocyanine green (ICG) into the body tissue and irradiate the body tissue with excitation light corresponding to the fluorescent wavelength of the reagent to obtain a fluorescent image. The light source device 11203 may be configured to supply narrow band light and / or excitation light corresponding to such special light observation.

[0110] FIG. 10 is a block diagram showing an example of the functional configuration of the camera head 11102 and the CCU 11201 shown in FIG.

[0111] The camera head 11102 has a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 has a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other by a transmission cable 11400 so as to be able to communicate with each other.

[0112] The lens unit 11401 is an optical system provided at the connection portion with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and enters the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

[0113] The imaging unit 11402 may include one imaging element (so-called single-plate type) or multiple imaging elements (so-called multiple-plate type). When the imaging unit 11402 is configured as a multiple-plate type, for example, each imaging element may generate an image signal corresponding to each of RGB, and a color image may be obtained by combining the image signals. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging elements for acquiring image signals for the right eye and the left eye corresponding to a 3D (dimensional) display. By performing a 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical site. Note that when the imaging unit 11402 is configured as a multiple-plate type, the lens unit 11401 may also be provided in multiple systems corresponding to each imaging element.

[0114] Furthermore, the imaging unit 11402 does not necessarily have to be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101, immediately behind the objective lens.

[0115] The driving unit 11403 is configured by an actuator, and moves the zoom lens and focus lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. This allows the magnification and focus of the image captured by the imaging unit 11402 to be appropriately adjusted.

[0116] The communication unit 11404 is configured by a communication device for transmitting and receiving various information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 via the transmission cable 11400 as RAW data.

[0117] Furthermore, the communication unit 11404 receives a control signal for controlling the driving of the camera head 11102 from the CCU 11201, and supplies the control signal to the camera head control unit 11405. The control signal includes information on the imaging conditions, such as information specifying the frame rate of the captured image, information specifying the exposure value at the time of capturing the image, and / or information specifying the magnification and focus of the captured image.

[0118] The image capturing conditions such as the frame rate, exposure value, magnification, and focus may be appropriately specified by a user, or may be automatically set by the control unit 11413 of the CCU 11201 based on an acquired image signal. In the latter case, the endoscope 11100 is equipped with a so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function.

[0119] The camera head control unit 11405 controls the driving of the camera head 11102 based on a control signal received from the CCU 11201 via the communication unit 11404.

[0120] The communication unit 11411 is configured with a communication device for transmitting and receiving various information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

[0121] Furthermore, the communication unit 11411 transmits, to the camera head 11102, a control signal for controlling the driving of the camera head 11102. The image signal and the control signal can be transmitted by electrical communication, optical communication, or the like.

[0122] The image processing unit 11412 performs various types of image processing on the image signal, which is RAW data sent from the camera head 11102 .

[0123] The control unit 11413 performs various controls related to imaging of the surgical site etc. by the endoscope 11100 and display of the captured image obtained by imaging the surgical site etc. For example, the control unit 11413 generates a control signal for controlling the driving of the camera head 11102.

[0124] Further, the control unit 11413 causes the display device 11202 to display the captured image showing the surgical site, etc., based on the image signal that has been image-processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 can recognize surgical tools such as forceps, specific living body parts, bleeding, mist when the energy treatment tool 11112 is used, etc., by detecting the shape and color of the edge of an object included in the captured image. When the control unit 11413 causes the display device 11202 to display the captured image, it may use the recognition result to superimpose various types of surgery support information on the image of the surgical site. By superimposing and presenting the surgery support information to the surgeon 11131, the burden on the surgeon 11131 can be reduced and the surgeon 11131 can proceed with the surgery reliably.

[0125] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable for communication of electrical signals, an optical fiber for optical communication, or a composite cable of these.

[0126] Here, in the illustrated example, communication is performed wired using a transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may also be performed wirelessly.

[0127] An example of an endoscopic surgery system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the image processing unit 11412 of the CCU 11201 among the configurations described above. By applying the technology according to the present disclosure to the image processing unit 11412 of the CCU 11201, an image with appropriate blur correction can be output, so that the surgeon can reliably check the surgical site.

[0128] Although an endoscopic surgery system has been described here as an example, the technology disclosed herein may also be applied to other systems, such as a microsurgery system.

[0129] <Examples of configuration combinations> The present technology can also be configured as follows. (1) a motion estimation unit that estimates a motion component at the time of imaging based on an image captured by an endoscope and sets a reliability indicating a degree to which the motion component is a motion component relative to the entire image; a correction amount control unit that controls a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability; An endoscopic surgery system comprising: (2) The motion estimation unit estimates whether the image contains a single motion component in which a single magnitude and direction dominates the motion component, or a multiple motion component in which a plurality of different magnitudes and directions dominate the motion component. An endoscopic surgery system as described in (1) above. (3) The motion estimation unit sets a first value as the reliability when it estimates that the image includes a single motion component, and sets a second value as the reliability when it estimates that the image includes multiple motion components. An endoscopic surgery system as described in (2) above. (4) the first value is 1.0; The second value is 0.0 An endoscopic surgery system as described in (3) above. (5) a calculation unit that calculates norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculates a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; The motion estimation unit estimates the motion component included in the image based on the norm standard deviation and the angle standard deviation by referring to a predetermined threshold. An endoscopic surgery system as described in (3) above. (6) The motion estimation unit Refer to a threshold curve represented by the threshold value, with the norm standard deviation as the vertical axis and the angle standard deviation as the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the threshold curve, it is estimated that the image includes a single motion component, and the reliability is set to the first value; When the norm standard deviation and the angle standard deviation calculated from the image are in a region equal to or greater than the threshold curve, it is estimated that the image includes a plurality of motion components, and the reliability is set to the second value. An endoscopic surgery system as described in (5) above. (7) The motion estimation unit Refer to a first threshold curve and a second threshold curve represented by the thresholds, with the norm standard deviation as a vertical axis and the angle standard deviation as a horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the threshold curve, it is estimated that the image includes a single motion component, and the reliability is set to the first value; when the norm standard deviation and the angle standard deviation calculated from the image are in a region that is equal to or greater than the first threshold curve and equal to or less than the second threshold curve, a value between the first value and the second value is set as the reliability; If the norm standard deviation and the angle standard deviation calculated from the image are in a region larger than the threshold curve, it is estimated that the image includes multiple motion components, and the second value is set as the reliability. An endoscopic surgery system as described in (5) above. (8) The correction amount control unit accumulates the reliability levels sequentially supplied from the motion estimation unit, and controls the correction amount using the reliability levels smoothed in a time direction. An endoscopic surgery system according to any one of (1) to (7) above. (9) a feature point extraction unit that extracts feature points that indicate characteristic locations in the image; a local motion vector detection unit that detects the local motion vector based on the movements of the plurality of feature points detected by the feature point extraction unit; The endoscopic surgery system according to (5) above, further comprising: (10) a global motion amount estimating unit that estimates a global motion amount indicating an overall motion of the image based on a plurality of the local motion vectors; Further equipped with The correction amount control unit calculates the correction amount based on the global motion amount. The endoscopic surgery system described in (9) above. (11) a correction processing unit that performs the correction processing on the image in accordance with the correction amount controlled by the correction amount control unit; The endoscopic surgery system according to any one of (1) to (10) above, further comprising: (12) a motion estimation unit that estimates a motion component at the time of imaging based on an image captured by an endoscope and sets a reliability indicating a degree to which the motion component is a motion component relative to the entire image; a correction amount control unit that controls a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability; An image processing device comprising: (13) The image processing device estimating a motion component at the time of imaging based on an image captured by an endoscope, and setting a reliability indicating a degree to which the motion component is a motion component relative to the entire image; controlling a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability; An image processing method comprising:

[0130] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present disclosure. In addition, the effects described in this specification are merely examples and are not limiting, and other effects may be obtained. [Explanation of symbols]

[0131] 11 endoscopic surgery system, 12 endoscope, 13 energy treatment tool, 14 display device, 15 device unit, 16 forceps, 21 image processing unit, 22 feature point extraction unit, 23 local motion vector detection unit, 24 global motion amount estimation unit, 25 calculation unit, 26 motion estimation unit, 27 correction amount control unit, 28 correction processing unit

Claims

1. a motion estimation unit that estimates a motion component at the time of imaging based on an image captured by an endoscope and sets a reliability indicating a degree to which the motion component is a motion component relative to the entire image; a correction amount control unit that controls a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability; a calculation unit that calculates norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculates a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; Equipped with The motion estimation unit Refer to a threshold curve represented by a predetermined threshold value, with the norm standard deviation as the vertical axis and the angle standard deviation as the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; If the norm standard deviation and the angle standard deviation calculated from the image are in a region equal to or greater than the threshold curve, it is estimated that the image includes multiple motion components, and a second value is set as the reliability. ,or, Referring to a first threshold curve and a second threshold curve represented by the thresholds, the norm standard deviation being the vertical axis and the angle standard deviation being the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the first threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; when the norm standard deviation and the angle standard deviation calculated from the image are in a region where they are equal to or greater than the first threshold curve and equal to or less than the second threshold curve, a value between the first value and the second value is set as the reliability; When the norm standard deviation and the angle standard deviation calculated from the image are in a region larger than the second threshold curve, it is estimated that the image includes a plurality of motion components, and the reliability is set to the second value. Endoscopic surgery system.

2. The motion estimation unit estimates whether the image contains a single motion component in which a single magnitude and direction dominates the motion component, or a multiple motion component in which a plurality of different magnitudes and directions dominate the motion component. The endoscopic surgery system of claim 1 .

3. The motion estimation unit sets a first value as the reliability when it estimates that the image includes a single motion component, and sets a second value as the reliability when it estimates that the image includes multiple motion components. The endoscopic surgery system of claim 2 .

4. the first value is 1.0; The second value is 0.0 The endoscopic surgery system of claim 3 .

5. The correction amount control unit accumulates the reliability levels sequentially supplied from the motion estimation unit, and controls the correction amount using the reliability levels smoothed in a time direction. The endoscopic surgery system of claim 1 .

6. a feature point extraction unit that extracts feature points that indicate characteristic locations in the image; a local motion vector detection unit that detects the local motion vector based on the movements of the plurality of feature points detected by the feature point extraction unit; The endoscopic surgery system of claim 1 further comprising:

7. a global motion amount estimating unit that estimates a global motion amount indicating an overall motion of the image based on a plurality of the local motion vectors; Further equipped with The correction amount control unit calculates the correction amount based on the global motion amount. The endoscopic surgery system of claim 6.

8. a correction processing unit that performs the correction processing on the image in accordance with the correction amount controlled by the correction amount control unit; The endoscopic surgery system of claim 1 further comprising:

9. a motion estimation unit that estimates a motion component at the time of imaging based on an image captured by an endoscope and sets a reliability indicating a degree to which the motion component is a motion component relative to the entire image; a correction amount control unit that controls a correction amount used when performing a correction process for correcting blur on the image in accordance with the reliability; a calculation unit that calculates norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculates a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; Equipped with The motion estimation unit Refer to a threshold curve represented by a predetermined threshold value, with the norm standard deviation as the vertical axis and the angle standard deviation as the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; If the norm standard deviation and the angle standard deviation calculated from the image are in a region equal to or greater than the threshold curve, it is estimated that the image includes multiple motion components, and a second value is set as the reliability. ,or, Referring to a first threshold curve and a second threshold curve represented by the thresholds, the norm standard deviation being the vertical axis and the angle standard deviation being the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the first threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; when the norm standard deviation and the angle standard deviation calculated from the image are in a region where they are equal to or greater than the first threshold curve and equal to or less than the second threshold curve, a value between the first value and the second value is set as the reliability; When the norm standard deviation and the angle standard deviation calculated from the image are in a region larger than the second threshold curve, it is estimated that the image includes a plurality of motion components, and the reliability is set to the second value. Image processing device.

10. The image processing device estimating a motion component at the time of imaging based on an image captured by an endoscope, and setting a reliability indicating a degree to which the motion component is a motion component relative to the entire image; controlling a correction amount used in performing a correction process for correcting blur on the image in accordance with the reliability; Calculating norms and angles of a plurality of local motion vectors based on local motion vectors indicating local motion of the image, and calculating a norm standard deviation of the group of local motion vectors and an angle standard deviation of the group of local motion vectors; Including, Refer to a threshold curve represented by a predetermined threshold value, with the norm standard deviation as the vertical axis and the angle standard deviation as the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; If the norm standard deviation and the angle standard deviation calculated from the image are in a region equal to or greater than the threshold curve, it is estimated that the image includes multiple motion components, and a second value is set as the reliability. ,or, Referring to a first threshold curve and a second threshold curve represented by the thresholds, the norm standard deviation being the vertical axis and the angle standard deviation being the horizontal axis; if the norm standard deviation and the angle standard deviation calculated from the image are in a region less than the first threshold curve, it is estimated that the image includes a single motion component, and a first value is set as the reliability; when the norm standard deviation and the angle standard deviation calculated from the image are in a region where they are equal to or greater than the first threshold curve and equal to or less than the second threshold curve, a value between the first value and the second value is set as the reliability; When the norm standard deviation and the angle standard deviation calculated from the image are in a region larger than the second threshold curve, it is estimated that the image includes a plurality of motion components, and the reliability is set to the second value. Image processing methods.

Images