Image processing device, imaging device, image processing method and program

By generating image signals with different frequency characteristics in a high-resolution imaging device and performing mixing processing, the problem of difficulty in maintaining resolution and focus positions in the display system is solved, and user-friendly focus position confirmation is achieved.

CN114073067BActive Publication Date: 2024-11-01SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080050037.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-07-17
Filing Date
2020-05-24
Publication Date
2024-11-01
Estimated Expiration
2040-05-24

AI Technical Summary

Technical Problem

In high-resolution imaging devices, it is difficult to maintain the original resolution and focus position in the display system, especially due to cost and miniaturization limitations, which makes it difficult for users to see clearly and operate when the peak signal is superimposed.

Method used

By generating multiple image signals with different frequency characteristics and performing mixing processing based on the mixing ratio determined by the peak signal, an image signal that does not require superposition of the peak signal is generated to confirm the focus position. The specific method includes using an image signal generation unit to generate high-frequency and low-frequency image signals, and performing alpha mixing through the mixing processing unit, combining high-frequency band edge detection and frequency band control, and adjusting the resolution and frequency band characteristics of the image signal.

Benefits of technology

The image display of confirming the focus position without superimposing the peak signal is realized, which improves the naturalness and visibility of the image, reduces the difficulty of users to identify the focus, and enhances the convenience of operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114073067B_ABST
    Figure CN114073067B_ABST
Patent Text Reader

Abstract

An image capable of confirming a focus position is generated without superimposing a peak signal on a captured image. Therefore, an image processing device according to the present technology includes an image signal generating unit that generates a plurality of image signals having frequency characteristics different from each other from one captured image signal, and a mixing processing unit that mixes the plurality of image signals based on a mixing ratio determined based on a peak signal with respect to the one captured image signal.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present technology relates to the technical field of an image processing device, an imaging device, an image processing method, and a program for generating an image signal for monitor output. In particular, the present technology relates to an imaging device, an imaging signal processing device, and an imaging signal processing method for generating an image signal for an image in which the focusing degree can be confirmed. Background Art

[0002] Some imaging devices include a monitor output for checking the brightness, white balance, etc. of the image being captured in real time. For this reason, it is desirable that the image displayed on the monitor can faithfully represent the brightness, white balance, and resolution of the captured image. However, in an imaging device having many high-resolution imaging sensors, it is difficult to have a display system with a number of pixels equal to or greater than the number of pixels of the imaging sensor from the perspective of cost and miniaturization. As a result, in a display system included in such an imaging device, it is difficult to represent the original sense of resolution and focus position.

[0003] Patent Document 1 discloses a technique for expressing a focus position by superimposing a peak signal detected in a high-resolution area on a captured image with reduced resolution.

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2011-176788 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] However, in the case of superimposing and displaying the peak signal on the image to be captured, there is a possibility that it becomes difficult to see depending on the imaging object (subject), and there are cases where the user feels resistance to use.

[0009] Therefore, an object of the present technology is to generate an image capable of confirming a focus position without superimposing a peak signal on an image to be captured.

[0010] Solution to the problem

[0011] An image processing device according to the present technology includes: an image signal generating unit, which generates a plurality of image signals having frequency characteristics different from each other from one captured image signal; and a mixing processing unit, which mixes the plurality of image signals based on a mixing ratio determined based on a peak signal regarding the one captured image signal.

[0012] The image signals having frequency characteristics different from each other are, for example, an image signal including a high-frequency component and an image signal from which the high-frequency component is removed.

[0013] The image signal generating unit in the above-mentioned image processing device may include a first resolution conversion processing unit that performs a resolution conversion process of converting an input image signal having a first resolution into an output signal having a second resolution.

[0014] The first resolution is, for example, a 4K resolution (3840 horizontal pixels and 2160 vertical pixels), and the second resolution is, for example, an HD resolution (1280 horizontal pixels and 720 vertical pixels). Alternatively, the first resolution may be an 8K resolution (7680 horizontal pixels and 4320 vertical pixels), and the second resolution may be a 4K resolution, a 2K resolution (1920 horizontal pixels and 1080 vertical pixels), or an HD resolution.

[0015] The plurality of image signals in the above-mentioned image processing apparatus may include a first image signal and a second image signal, and the first image signal may be an image signal containing more high-frequency components than the second image signal.

[0016] One image signal is an image containing more high frequency components than the other image signal, and by changing the mixing ratio of the image signals for each portion of the displayed image, the amount of high frequency components contained in each portion of the displayed image can be changed.

[0017] The first image signal in the above-mentioned image processing device may be an image signal that has not passed through a band-limiting filter, and the second image signal may be an image signal that has passed through a low-pass filter.

[0018] The first image signal is usually used to display the focus position in the image. In addition, the second image signal is usually used to display the portion other than the focus position in the image.

[0019] The first image signal in the above-mentioned image processing device may be an image signal that has passed through a high-pass filter, and the second image signal may be an image signal that has not passed through a band-limiting filter.

[0020] The first image signal is an image signal in which high-frequency components are emphasized by a high-pass filter.

[0021] In the above-mentioned image processing device, the first image signal may be an image signal that has passed through a high-pass filter, and the second image signal may be an image signal that has passed through a low-pass filter.

[0022] The first image signal is an image signal obtained by passing through a high-pass filter to emphasize high-frequency components, and the second image signal is an image signal obtained by passing through a low-pass filter to emphasize low-frequency components.

[0023] The above-mentioned image processing apparatus may further include a high-frequency band edge detection unit configured to extract an edge detection signal of a high-frequency band from the one captured image signal as the peak signal.

[0024] That is, the image processing device includes a high-frequency band edge detection unit.

[0025] The high-frequency band edge detection unit in the above-mentioned image processing device may include a second resolution conversion processing unit configured to perform resolution conversion processing while maintaining edge information of the high-frequency band.

[0026] As a result, the peak signal used for determining the mixing ratio is a signal that is down-converted without removing the high frequency component.

[0027] The high-frequency band edge detection unit in the above-mentioned image processing apparatus may include a band control unit that controls the high-frequency band based on lens information related to a lens used for imaging.

[0028] The lens information may be, for example, position information of a zoom lens, information such as an F value, or a lens model capable of specifying the information.

[0029] The blending processing unit in the above-mentioned image processing device may perform alpha blending based on the blending ratio.

[0030] As a result, two image signals having different frequency characteristics are synthesized by the mixing processing unit.

[0031] An imaging device according to the present technology includes: an imaging element having a first number of pixels; a display unit having a second number of pixels, the second number of pixels being smaller than the first number of pixels; an image signal generating unit, the image signal generating unit generating a plurality of image signals having frequency characteristics different from each other from a captured image signal output from the imaging element; and a mixing processing unit, the mixing processing unit mixing the plurality of image signals according to a mixing ratio determined based on a peak signal to generate an image signal to be output to the display unit.

[0032] For example, the first pixel number is based on a pixel number of 3840 pixels horizontally and 2160 pixels vertically which is a 4K resolution, and the second pixel number is based on a pixel number of 1280 pixels horizontally and 720 pixels vertically which is an HD resolution.

[0033] The above-mentioned imaging device may also include a switching processing unit, which is configured to switch between a mode in which the output from the mixing processing unit is input to the display unit, and a mode in which the output from the resolution conversion processing unit is input to the display unit without passing through the mixing processing unit.

[0034] An operator for switching may be provided corresponding to the switching processing unit. The operator may be, for example, an operator such as a physical button, or an operator for operating a menu screen by being able to switch from the menu screen may function as a switching operator. In addition, in the case where the display unit is equipped with a touch panel, the display unit may function as a switching operator. That is, by operating an icon or the like arranged on the display unit, the mode may be switched.

[0035] The mixing ratio in the above-mentioned imaging device can be determined according to the peak signal and user setting.

[0036] The mixing ratio may be determined based on the peak signal only, based on the user setting only, or based on both the peak signal and the user setting. In addition, these switches may be configured to be settable by the user.

[0037] The above-mentioned imaging device may further include a notification unit for notifying a user that a display corresponding to the output from the mixing processing unit is performed, when a display corresponding to the output from the mixing processing unit is performed on the display unit.

[0038] For example, the notification unit may perform a process of displaying an icon indicating that an image subjected to the blending process is displayed on the display unit, or may perform a notification process by providing a light emitting unit such as a notification light emitting diode (LED).

[0039] An image processing method according to the present technology is an image processing method performed by an image processing device, the method comprising: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal; and mixing the plurality of image signals according to a mixing ratio determined based on a peak signal.

[0040] A program according to the present technology is for causing an information processing apparatus to execute the following processing: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal; and mixing the plurality of image signals according to a mixing ratio determined based on a peak signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a perspective view of an imaging device according to an embodiment of the present technology.

[0042] Figure 2is a block diagram of an imaging device according to an embodiment.

[0043] Figure 3 is a block diagram illustrating an example of a signal processing unit in the first embodiment.

[0044] Figure 4 is an image representation of a high-resolution image;

[0045] Figure 5 is an image diagram of a general image;

[0046] Figure 6 This is an image diagram of an image obtained by removing high-frequency components from a normal image.

[0047] Figure 7 is an image diagram of a monitor output image output from the mixing unit.

[0048] Figure 8 is a block diagram illustrating an example of a signal processing unit in the second embodiment.

[0049] Fig. 9 is a block diagram illustrating an example of a signal processing unit in the third embodiment.

[0050] Fig.10 is a block diagram illustrating an example of a signal processing unit in the fourth embodiment.

[0051] Fig.11 is a block diagram illustrating an example of a signal processing unit in the first modification.

[0052] Fig.12 is a block diagram of an imaging device in a second modification.

[0053] Fig.13 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system.

[0054] Fig.14 It is a graphic description Fig.13 A block diagram of an example of the functional configuration of a camera and a CCU shown in the diagram. DETAILED DESCRIPTION

[0055] Hereinafter, the embodiments will be described in the following order with reference to the drawings.

[0056] <1. First Embodiment>

[0057] <2. Second Embodiment>

[0058] <3. Third embodiment>

[0059] <4. Fourth embodiment>

[0060] <5. Modifications>

[0061] <6. Conclusion>

[0062] <7. Application Examples>

[0063] <8. This technology>

[0064] <1. First Embodiment>

[0065] Figure 1 The diagram illustrates the appearance of an imaging device 1 according to the present technology.

[0066] The imaging device 1 includes a camera housing 2 having an internal space in which a substrate and the like are arranged, and a lens housing 3 attached to the front of the camera housing 2 and in which a lens group is arranged.

[0067] An electronic viewfinder (EVF) 4 is provided at the upper portion of the camera housing 2, and a rear monitor 5 is attached to the rear portion of the camera housing 2. In the camera housing 2, various operators 6 for imaging a subject and checking a captured image are provided in respective units. Specific examples of the operators include a playback menu activation button, an OK button, a cross key, a cancel button, a zoom button, a slide button, a shutter button (release button), and the like.

[0068] The EVF 4 includes an EVF monitor 7 that can be visually recognized from the rear, and a frame-like surrounding portion 8 that protrudes toward the rear to surround the upper side and left and right sides of the EVF monitor 7. That is, the EVF monitor 7 is provided in the viewfinder.

[0069] The rear monitor 5 can rotate relative to the camera body 2. For example, the lower end of the rear monitor 5 can be rotated so as to move backward with the upper end of the rear monitor 5 as the rotation axis.

[0070] Note that the right end and the left end of the rear monitor 5 can be used as a rotation axis. In addition, the rear monitor 5 can be rotated in a plurality of directions.

[0071] Figure 2 1 is a block diagram of the imaging device 1. The imaging device 1 includes an optical system 50, an imaging unit 51, a signal processing unit 52, a control unit 53, an optical system driver 54, an operation input unit 55, a display unit 56, and the like. The display unit 56 is, for example, a back monitor 5 or an EVF monitor 7. In the following description, an image is output to the EVF monitor 7 as the display unit 56.

[0072] Note that although not in Figure 2 , however, the imaging device 1 may include a memory unit, a sensor, a storage unit, a communication unit, and the like.

[0073] The optical system 50 includes various lenses such as an incident end lens, a zoom lens, a focus lens, and a condenser lens, an aperture mechanism, etc. The aperture mechanism performs exposure control by adjusting the opening amount of the lens or iris (aperture) so that the signal charge is not saturated and sensing is performed within the dynamic range. The light reflected from the subject (subject light) enters the optical system 50.

[0074] The imaging unit 51 includes, for example, a charge coupled device (CCD) type or a complementary metal oxide semiconductor (CMOS) type imaging element 51 a .

[0075] The sensor surface of the imaging element 51 a includes a sensing element in which a plurality of pixels are two-dimensionally arranged.

[0076] The imaging unit 51 performs, for example, a correlated double sampling (CDS) process, an automatic gain control (AGC) process, etc., and also performs an analog / digital (A / D) conversion process on an electrical signal obtained by photoelectric conversion of light received by the imaging element 51a. Subsequently, the captured image as digital data is output to the signal processing unit 52 of the next stage. Note that in the following description, an example will be given in which the imaging element 51a has a 4K image resolution and the EVF monitor 7 has a 2K resolution.

[0077] The signal processing unit 52 includes, for example, a microprocessor such as a digital signal processor (DSP), a microcomputer, or the like dedicated to digital signal processing.

[0078] The signal processing unit 52 includes various units for performing various signal processing on the digital signal (captured image signal) transmitted from the imaging unit 51 .

[0079] Specifically, the signal processing unit 52 includes a correction processing unit 57, a development processing unit 58, a main line system output processing unit 59, a first resolution conversion processing unit 60, a high frequency band edge detection unit 61, a monitor output generation unit 62, etc. The details will be referred to later. Figure 3 illustrate.

[0080] By performing signal processing by these units, an image signal to be output to the EVF monitor 7 as a display unit and an image signal of 4K resolution as a main line output are output from the signal processing unit 52 .

[0081] The control unit 53 performs overall control of the imaging device 1. Furthermore, the control unit 53 instructs the optical system driver 54 to control each lens included in the optical system 50.

[0082] The optical system driver 54 is provided with, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, a motor driver for a motor for driving an aperture mechanism, and the like.

[0083] The control unit 53 can acquire information about each lens included in the optical system 50. The information of the lens includes, for example, information such as the model of the lens, the position of the zoom lens, and the f-number (aperture value).

[0084] The correction processing unit 57 performs processing such as correction processing between R, G and B color channels, white balance correction, aberration correction and shading correction. The image signal output from the correction processing unit 57 is input to the development processing unit 58 and the high-frequency band edge detection unit 61.

[0085] The development processing unit 58 performs YC generation processing of generating (separating) a luminance (Y) signal and a color (C) signal from R, G, and B image data, processing of adjusting luminance and color, knee correction, γ correction, and the like.

[0086] The main line system output processing unit 59 performs resolution conversion processing, codec processing for encoding for recording or communication, etc., thereby converting the image signal into the final output format, and outputs the converted image signal to the outside as a main line output of a 4K image, etc.

[0087] The first resolution conversion processing unit 60 performs resolution conversion processing (down conversion processing) on ​​the image signal output from the development processing unit 58 to generate an image signal of 2K resolution from the image signal of 4K resolution. The image signal of 2K resolution is input to the monitor output generation unit 62 .

[0088] The high-frequency band edge detection unit 61 performs processing for extracting an edge detection signal of 2K resolution from the image signal that has undergone various types of correction processing by the correction processing unit 57. Specifically, the high-frequency band edge detection unit 61 extracts an edge detection signal in the high-frequency band from the image signal of 4K resolution, and performs resolution conversion processing on the edge detection signal to extract an edge detection signal of 2K resolution.

[0089] Note that the high-frequency band edge detection unit 61 includes a band control unit 63 and a second resolution conversion processing unit 64 (see Figure 3 ). That is, the band control unit 63 performs filtering for extracting the edge detection signal, and the second resolution conversion processing unit 64 performs resolution conversion to obtain an edge detection signal with a 2K resolution.

[0090] Therefore, the control unit 53 controls the band control unit 63 based on the lens information so that the band control is performed based on the frequency of the edge detection signal to be extracted. For example, the band control unit 63 is controlled so that the band passing through the band control unit 63 corresponds to the position of the zoom lens. Specifically, in the case of zoom shooting, the band of the captured image is generally lower than that of equal-magnification imaging. Therefore, in the case of imaging at a low magnification, the band control unit 63 is controlled so that the high-frequency component is extracted as the edge detection signal. On the contrary, in the case of imaging at a high magnification (in the case of zoom shooting at a high magnification), it may not be possible to extract the high-frequency component, and thus it is impossible to obtain an appropriate edge detection signal. Therefore, the band control unit 63 is controlled so that a slightly lower frequency component is extracted as the edge detection signal.

[0091] In addition, the control unit 53 controls so that the frequency band passing through the frequency band control unit 63 becomes appropriate according to the F value. If the F value is too large, it is difficult to extract the high-frequency component as the edge detection signal due to the aperture blur. In addition, if the F value is too small, it is also difficult to extract the high-frequency component as the edge detection signal due to the aperture blur.

[0092] Furthermore, the control unit 53 performs control so that the frequency band passing through the frequency band control unit 63 becomes appropriate according to the type of lens. That is, the frequency band control unit 63 is controlled according to the modulation transfer function (MTF) characteristics of each lens.

[0093] The high-frequency band edge detection unit 61 extracts an edge detection signal in the high-frequency band at a 2K resolution from the image signal at a 4K resolution. Therefore, the high-frequency band edge detection unit 61 performs processing in units of 4 pixels including 2 pixels in the horizontal direction and 2 pixels in the vertical direction. Therefore, the band control unit 63 and the second resolution conversion processing unit 64 included in the high-frequency band edge detection unit 61 also perform processing for each 4 pixels. That is, the band control unit 63 may include a band control unit 63a that processes an upper left pixel among the 4 pixels, a band control unit 63b that processes an upper right pixel, a band control unit 63c that processes a lower left pixel, and a band control unit 63d that processes a lower right pixel.

[0094] The band control unit 63 extracts the edge detection signal of the predetermined band according to the instruction from the control unit 53. The band control unit 63 performs processing for detecting the edge in the horizontal direction and processing for detecting the edge in the vertical direction. As the processing for detecting the edge in the horizontal direction and the processing for detecting the edge in the vertical direction, for example, a gain stabilization processing using a gain coefficient for stabilizing the gain, a processing for removing random noise, etc. are performed.

[0095] In addition, the band control unit 63 in this configuration selects and extracts a signal with a large absolute value from the edge detection signal in the horizontal direction and the edge detection signal in the vertical direction, and outputs the selected signal as an edge detection signal for one pixel to be processed. That is, the horizontal edge detection signal or the vertical edge detection signal for each pixel is selected and output from the band control unit 63 as an edge detection signal for each of the four pixels.

[0096] The second resolution conversion processing unit 64 receives the edge detection signal for four pixels output from the band control unit 63, performs down conversion processing, and outputs the signal as an edge detection signal (peak signal) for one pixel.

[0097] The second resolution conversion processing unit 64 in the present configuration is configured to perform down-conversion processing without damaging edge information in the high-frequency band.

[0098] Specifically, the difference of the continuous edge detection signals is calculated, and the group with the largest difference is output. As a result, the edge detection signal of 2K resolution is generated (extracted) from the edge detection signal of 4K resolution so that the edge information in the high frequency band is not damaged.

[0099] Note that the second resolution conversion processing unit 64 not only performs down-conversion processing, but also performs squaring processing to emphasize the signal in the frequency band within a narrower range. In addition, by using a selector that can select a signal that has been squared and a signal that has not been squared, an edge detection signal corresponding to the user's preference can be output.

[0100] Both the peak signal output from the second resolution conversion processing unit 64 and the image signal (hereinafter described as a “normal image signal”) output from the first resolution conversion processing unit 60 are input to the monitor output generation unit 62 .

[0101] Both the peak signal and the normal image signal are 2K resolution signals corresponding to the resolution of the display unit 56. The monitor output generation unit 62 generates an image signal for displaying an image on the display unit 56 using the two signals.

[0102] Monitor output generation unit 62 generates a first image signal and a second image signal from an image signal obtained by branching a normal image signal, and performs a process of mixing the first image signal and the second image signal based on a peak signal.

[0103] The first image signal and the second image signal are image signals having different frequency characteristics. For example, the first image signal is an image signal having relatively more high-frequency components than the second image signal.

[0104] The monitor output generation unit 62 includes a band limiting filter 65 and a mixing unit 66 .

[0105] Several examples of band limiting filter 65 may be considered. Figure 3 The example illustrated in FIG. 6 is an example in which a low-pass filter (LPF) 65 a is used as the band-limiting filter 65 .

[0106] The normal image signal output from the first resolution conversion processing unit 60 is branched into two image signals, one of which is input to the mixing unit 66 as the normal image signal, and the other is input to the LPF 65 a .

[0107] In the LPF 65 a , an image signal (hereinafter described as “LPF image signal”) from which a frequency band of a set predetermined frequency or higher is removed is generated.

[0108] Note that the filter coefficient of the LPF 65a may be assigned a predetermined value determined in advance by the control unit 53, or may be given as a variable.

[0109] In the case where the filter coefficient is variable, the filter coefficient may be determined by the control unit 53 according to the photographing environment or the like, or may be determined by the control unit 53 based on an operation of a photographer, a video engineer, or the like.

[0110] The mixing unit 66 mixes the normal image signal and the LPF image signal. As a mixing method, for example, a fusion method such as α blending is used.

[0111] For example, if the component (ratio) of the LPF image signal is set to 0 and mixed, a simple normal image is displayed on the display unit 56. In addition, if the component (ratio) of the normal image signal is set to 0 and mixed, an image with high-frequency components removed is displayed on the display unit 56.

[0112] The mixing unit 66 generates a region in which high-frequency components are removed and a region in which high-frequency components remain by changing the mixing ratio for each pixel or each region based on the peak signal.

[0113] Specifically, for pixels (or regions) with large peak signals, the ratio of normal image signals is increased and the ratio of LPF image signals is reduced. In addition, for pixels (or regions) with small peak signals, the ratio of normal image signals is reduced and the ratio of LPF image signals is increased.

[0114] For example, the mixing ratio of the normal image signal and the LPF image signal in the pixel (area) with the largest peak signal and the mixing ratio in the pixel (area) with the smallest peak signal can be determined, and for other pixels (areas), the mixing ratio can change linearly from the pixel (area) with the largest peak signal to the pixel (area) with the smallest peak signal.

[0115] Note that a pixel (region) with a large peak signal is a pixel (region) containing a large amount of high-frequency components, such as a focused pixel (region). In addition, a pixel (region) with a small peak signal is a pixel (region) containing few high-frequency components, such as a pixel (region) that is not focused.

[0116] That is, in the monitor output signal output from the mixing unit 66, the focused pixel (area) is relatively emphasized.

[0117] Will refer to Figure 4 , Figure 5 , Figure 6 and Figure 7 Illustrate examples.

[0118] Figure 4 This is an image diagram of a high-resolution image such as a 4K image. The eyes, nose, and mouth of a person are focused, but the outline and shoulders of the face are not focused. That is, the eyes, nose, and mouth contain a large amount of high-frequency components, while the outline and shoulders of the face contain a small amount of high-frequency components.

[0119] Figure 5 It is an image diagram of an image according to a normal image signal, that is, an image which has been subjected to resolution conversion by the first resolution conversion processing unit 60. For the parts of eyes, nose, and mouth, high-frequency components are retained to a certain extent.

[0120] Figure 6 6 is a diagram of an image based on a normal image signal, that is, an LPF image signal, which has passed through LPF 65a. High-frequency components are also removed from the eyes, nose, and mouth. In addition, high-frequency components are also removed from the contour and shoulder parts of the face.

[0121] The high frequency band edge detection unit 61 is Figure 4 In the peak signal extracted from the high-resolution image shown in the diagram, the pixel areas of the eyes, nose, and mouth are enlarged, while the pixel areas of the contour part and shoulders are reduced.

[0122] Thus, when the blending unit 66 performs blending based on the peak signal, priority is given to the pixel areas of the eyes, nose, and mouth. Figure 5 The normal image signal shown in the diagram is given priority for the pixel areas of the contour and shoulder. Figure 6That is, the image displayed on the display unit 56 by the monitor output signal output from the mixing unit 66 is as shown in FIG. Figure 7 As shown in the diagram.

[0123] That is, in the monitor output signal, a pixel area that is not in focus is set as an LPF image signal, so that the image area in focus is relatively emphasized.

[0124] For the mixing unit 66, there are several possible mixing methods.

[0125] For example, mixing using a distance image may be performed. The distance image refers to an image including a distance pixel signal based on a distance detected by detecting the distance of the subject from the imaging device 1 in the depth direction for each pixel.

[0126] The following is a detailed description. When the imaging device 1 includes a unit for measuring the distance to the subject, the imaging device 1 can generate a distance image.

[0127] First, for a pixel region where the peak signal is high, a normal image signal is mixed preferentially.

[0128] In addition, assuming that the distance to the subject in the pixel area with a high peak signal is D1, the distance to the subject in other pixel areas can be expressed by (D1+d1). d1 varies from pixel to pixel, and when d1 is small, the subject is at the same distance as the focused subject.

[0129] In the mixing using the distance image, when d1 is small, the normal pixel signal is mixed with priority, and when d1 is large, the LPF image signal is mixed with priority.

[0130] As a result, even in a pixel area where the peak signal is small due to less high-frequency components although the pixel area is in focus, mixing is performed with priority given to normal pixel signals, thereby emphasizing the focus of the pixel area.

[0131] As another mixing method, mixing can be performed according to the pixel distance on the image. Specifically, in the image displayed on the display unit 56, the pixels closer to the pixel area with a higher peak signal are preferentially mixed with the normal pixel signal, and the pixels farther away from the pixel area are preferentially mixed with the LPF image signal.

[0132] As a result, it is possible to cause the display unit 56 or the like to display an image in which it is easy to grasp which pixel region is in focus.

[0133] <2. Second Embodiment>

[0134] The imaging device 1A according to the second embodiment has a configuration of a high-frequency band edge detection unit and a monitor output generation unit different from those of the first embodiment.

[0135] Will refer to Figure 8 This will be described in detail.

[0136] The imaging device 1A includes a signal processing unit 52A. Figure 8 2 is a block diagram of the signal processing unit 52A. The signal processing unit 52A includes a high-frequency band edge detection unit 61A and a monitor output generation unit 62A.

[0137] The high-frequency band edge detection unit 61A includes a plurality of band control units 63Aa, 63Ab, . . . and a plurality of second resolution conversion processing units 64Aa, 64Ab, .

[0138] The band control unit 63Aa receives the image signal that has been subjected to various correction processes of the correction processing unit 57, and performs filtering processing for extracting an edge detection signal. The output of the band control unit 63Aa is down-converted by the second resolution conversion processing unit 64Aa, and is input as a peak signal Pa to the mixing unit 66A of the monitor output generation unit 62A.

[0139] The band control unit 63Ab receives the image signal that has been subjected to various correction processes of the correction processing unit 57, and performs filtering processing for extracting an edge detection signal. The output of the band control unit 63Ab is down-converted by the second resolution conversion processing unit 64Ab, and is input as a peak signal Pb to the mixing unit 66A of the monitor output generation unit 62A.

[0140] Similarly, the output signal from the correction processing unit 57 is input to the other band control unit 63A, the extracted edge detection signal is down-converted, and is input as the peak signal P to the mixing unit 66A.

[0141] The plurality of peak signals Pa, Pb, . . . are signals obtained by extracting specific components having different frequencies in the high frequency band.

[0142] The monitor output generating unit 62A includes a plurality of band limiting filters 65, 65, ... and a mixing unit 66A.

[0143] The plurality of band limiting filters 65, 65, ... correspond to the LPFs 65a1, 65a2, ... .

[0144] The LPFs 65a1, 65a2, ... are LPFs having different passbands, and output LPF image signals L1, L2, ... to the mixing unit 66A.

[0145] Note that the LPFs 65a1, 65a2, ... may be band pass filters (BPF).

[0146] A normal image signal as a first image signal not passing through any LPF 65a and LPF image signals L1, L2, . . . as second image signals passing through any LPF 65a are input to the mixing unit 66A.

[0147] The mixing unit 66A mixes a plurality of image signals (normal image signal, LPF image signal L1, LPF image signal L2, ...) based on a plurality of input peak signals Pa, Pb, ... The mixing unit 66A may select two image signals from the plurality of image signals for α mixing, or may select three or more image signals for mixing.

[0148] Furthermore, different α mixing may be performed for each pixel region, or three or more image signals may be mixed for a specific pixel region.

[0149] For example, the mixing ratio of the LPF image signal L1 is determined based on the peak signal Pa. Furthermore, the mixing ratio of the LPF image signal L2 is determined based on the peak signal Pb. Finally, based on the mixing ratio of the LPF image signals L1 and L2, the mixing ratio of the normal image signal is determined. In this way, the mixing ratio of a plurality of image signals can be determined.

[0150] Furthermore, a peak signal P for determining a mixing ratio of a normal image signal may be generated in the high-frequency band edge detection unit 61A, and the peak signal P may be input to the mixing unit 66A.

[0151] Note that the photographer or video engineer may be allowed to manipulate the mixing ratio. As a result, adjustments can be easily made so that an image can be captured as intended by the photographer or video engineer.

[0152] Furthermore, instead of the high-frequency band edge detection unit 61A, a specific band component detection unit 67 may be used. That is, the band control units 63Aa, 63Ab, ... included in the specific band component detection unit 67 extract not only the signal of the high-frequency band but also the signal of the specific band, and the second resolution conversion processing units 64Aa, 64Ab, ... perform down-conversion so that the signal of the specific band is not lost.

[0153] As a result, the mixing unit 66A can generate a monitor output signal in which a signal of a specific frequency band is emphasized. That is, it is possible to freely create an image desired by a photographer or a video engineer.

[0154] <3. Third embodiment>

[0155] The imaging device 1B in the third embodiment is an example in which the monitor output generation unit includes a high-pass filter (HPF) 65 b as the band-limiting filter 65 .

[0156] Will refer to Fig. 9 This will be described in detail.

[0157] The imaging device 1B includes a signal processing unit 52B. Fig. 9 is a block diagram of the signal processing unit 52B.

[0158] The signal processing unit 52B includes a monitor output generating unit 62B. The monitor output generating unit 62B includes a HPF 65B, an adder 68, and a mixing unit 66B.

[0159] The peak signal output from the high-frequency band edge detection unit 61 and the normal image signal output from the first resolution conversion processing unit 60 are input to the monitor output generation unit 62B.

[0160] The normal image signal input to the monitor output generating unit 62B is branched into three signals, one of which is input to the mixing unit 66B as a second image signal while being a normal image signal, one is input to the adder 68, and another is input to the adder 68 via the HPF 65b.

[0161] The image signal output from the adder 68 is input to the mixing unit 66B as an HPF image signal.

[0162] That is, the HPF image signal as the first image signal and the normal image signal as the second image signal are input to the mixing unit 66B together with the peak signal.

[0163] Since the HPF image signal is a signal obtained by adding a high-frequency component to a normal image signal, the HPF image signal is a signal in which the high-frequency component is emphasized compared with the normal image signal.

[0164] The mixing unit 66B performs mixing (eg, α mixing) of the HPF image signal and the normal image signal based on the input peak signal.

[0165] Specifically, the HPF image signal is mixed preferentially for the image area that is in focus, and the normal image is mixed preferentially for the image area that is not in focus.

[0166] As a result, the focused image area is emphasized and displayed on the display unit 56 .

[0167] In particular, by setting the HPF image signal to which a high-frequency component is applied as the first image signal and the second image signal as the normal image signal, the subject can be easily visually identified even in a pixel domain where the peak signal is small, and the status of the captured image can be easily confirmed on the display unit 56.

[0168] <4. Fourth embodiment>

[0169] In the imaging device 1C of the fourth embodiment, a monitor output generation unit 62C includes a third resolution conversion processing unit 69 .

[0170] Will refer to Fig.10 This will be described in detail.

[0171] The imaging device 1C includes a signal processing unit 52C. Fig.10 is a block diagram of the signal processing unit 52C.

[0172] The signal stop unit 52C includes a monitor output generation unit 62C. In addition to the peak signal output from the second resolution conversion processing unit and the normal image signal output from the first resolution conversion processing unit 60, the image signal output from the development processing unit 58 is also input to the monitor output generation unit 62C.

[0173] The normal image signal output from the first resolution conversion processing unit 60 is input to the mixing unit 66C as the first image signal.

[0174] The image signal input from the development processing unit 58 to the monitor output generation unit 62C without passing through the first resolution conversion processing unit 60 remains as a high-resolution image signal that is not down-converted, is input to the LPF 65a serving as the band-limiting filter 65, and after the high-frequency component is removed, is input to the third resolution conversion processing unit 69. The third resolution conversion processing unit 69 performs a down-conversion process from the image signal of 4K resolution to the image signal of 2K resolution, thereby generating an LPF image signal that is input to the mixing unit 66C as the second image signal.

[0175] Mixing section 66C performs mixing processing of a normal image signal as a first image signal and an LPF image signal as a second image signal based on the input peak signal.

[0176] Note that, as described in the third embodiment, the HPF image signal as the first image signal generated using the HPF 65b and the normal image signal as the second image signal may be input to the mixing unit 66C.

[0177] Note that, compared with the imaging device 1C in the fourth embodiment, the imaging device 1 in the first embodiment is a low-resolution image signal obtained by down-converting the input of the LPF 65a, thereby reducing the processing load of the LPF 65a and increasing the processing speed. That is, it can contribute to improving the real-time performance of the image displayed on the display unit 56.

[0178] <5. Various Modifications>

[0179] <5-1. First Modification>

[0180] The plurality of image signals described in the above respective examples may be generated using the HPF and the LPF.

[0181] For example, the first image signal may be an HPF image signal generated using the HPF 65b, and the second image signal may be an LPF image signal generated using the LPF 65a.

[0182] The configuration in which the HPF 65b is used to generate the first image signal and the LPF 65a is used to generate the second image signal can be applied not only to the present embodiment but also to the first, second and third embodiments.

[0183] For example, Fig.11 The diagram illustrates the configuration of the first modification example applied to the first embodiment. That is, the imaging device 1D includes a monitor output generation unit 62D. The monitor output generation unit 62D includes an HPF 65b and an LPF 65a as a band limiting filter 65, an adder 68, and a mixing unit 66D.

[0184] The HPF 65b and the adder 68 generate an HPF image signal as a first image signal, and output the HPF image signal to the mixing unit 66D.

[0185] The LPF 65a generates an LPF image signal as a second image signal, and outputs the LPF image signal to the mixing unit 66D.

[0186] The mixing unit 66D performs mixing processing of the HPF image signal and the LPF image signal based on the peak signal.

[0187] By mixing the HPF image signal and the LPF image signal, the difference between a focused pixel region and a non-focused pixel region can be made clearer, making it easier to confirm a captured image.

[0188] <5-2. Second Modification>

[0189] The second modification example has a configuration that enables selection of which of a mixed image obtained by mixing a plurality of image signals by the mixing unit 66 and a normally captured image is displayed on the display unit 56 .

[0190] Will refer to Fig.12 This will be described in detail.

[0191] The imaging device 1E includes a selector 71 in addition to the above-described various configurations.

[0192] The selector 71 receives the output from the monitor output generation unit 62 (mixing unit 66 ) and the output (image signal) from the first resolution conversion processing unit 60 .

[0193] The selector 71 outputs, to the display unit 56 , any image signal selected from the monitor output generation unit 62 and the first resolution conversion processing unit 60 based on the control signal.

[0194] The control signal from the control unit 53 is input to the selector 71. The control unit 53 outputs a control signal to the selector 71 in accordance with the situation or in accordance with the operation of the photographer (or video engineer) using the various operators 6.

[0195] With this structure, for example, the photographer can cause the display unit 56 to display a desired image among a mixed image in which a focused pixel area is emphasized and a normal image.

[0196] In the imaging device 1E, the control unit 53 can perform control for notifying which one of the mixed image and the normal image is displayed on the display unit 56. For example, the display unit 56 functions as a notification unit, and under the control of the control unit 53, an icon image indicating which one of the mixed image and the normal image is displayed is displayed on the screen of the display unit 56, etc.

[0197] Furthermore, in addition to the display unit 56, a light emitting unit including, for example, a light emitting diode (LED) or the like may be provided as a dedicated notification unit.

[0198] <6. Conclusion>

[0199] As described in the above examples, the image processing device (imaging devices 1, 1A, 1B, 1C and 1D) according to the present technology includes: an image signal generating unit, which generates a plurality of image signals having frequency characteristics different from each other from one captured image signal (image signal output from the correction processing unit 57); and a mixing processing unit (mixing units 66, 66A, 66B, 66C and 66D), which mixes the plurality of image signals based on a mixing ratio, and the mixing ratio is determined based on a peak signal regarding the one captured image signal.

[0200] Examples of the image signal generating unit include the image signal generating unit 70 ( Figure 3 ), an image signal generating unit 70A ( Figure 8 ), an image signal generating unit 70B ( Fig. 9 ), an image signal generating unit 70C ( Fig.10 ), and an image signal generating unit 70D ( Fig.11 ).

[0201] The image signals having frequency characteristics different from each other are, for example, an image signal including a high-frequency component and an image signal from which the high-frequency component is removed.

[0202] By generating an image signal by mixing an image signal including a high-frequency component and an image signal from which the high-frequency component has been removed, an image in which the degree of focus (focus position) can be easily confirmed can be generated. In addition, in the case of generating a display image by mixing image signals having different frequency characteristics generated from captured image signals, the image can be made more natural than a display image in which a peak signal is superimposed, and the image can be easily viewed. In addition, for example, in a display image in which a peak signal is superimposed, the superimposed peak signal is difficult to see, and there is a possibility that it becomes difficult to grasp the degree of focus. According to the present configuration, since all the image signals to be mixed are natural images having only different frequency characteristics, it is easy to confirm the degree of focus.

[0203] As described in the above examples, the image signal generating unit 70 (70A, 70B, 70C and 70D) in the image processing device (imaging devices 1, 1A, 1B, 1C and 1D) may include a first resolution conversion processing unit 60, which performs resolution conversion processing to convert an input image signal having a first resolution into an output signal having a second resolution.

[0204] The first resolution is, for example, a 4K resolution (3840 horizontal pixels and 2160 vertical pixels), and the second resolution is, for example, an HD resolution (1280 horizontal pixels and 720 vertical pixels). Alternatively, the first resolution may be an 8K resolution (7680 horizontal pixels and 4320 vertical pixels), and the second resolution may be a 4K resolution, a 2K resolution (1920 horizontal pixels and 1080 vertical pixels), or an HD resolution.

[0205] By including the first resolution conversion processing unit 60 , a low-resolution image for confirming a captured image can be displayed on the display unit 56 .

[0206] Note that, when the second resolution is set lower than the first resolution, it may be difficult to confirm the focus from the image with the second resolution. Even in this case, an image signal to be output to the display unit is generated by mixing a plurality of natural images so that the focus can be easily checked.

[0207] As described in each of the above examples, the plurality of image signals may include a first image signal and a second image signal, and the first image signal may be an image signal containing more high frequency components than the second image signal.

[0208] One of the image signals is an image containing more high-frequency components than other image signals, and for each portion (pixel area) of the displayed image, the mixing ratio of the image signals is changed, thereby changing the amount of high-frequency components contained in each portion of the displayed image.

[0209] As a result, a display image in which the focus degree of each portion is expressed can be generated. Thus, the user can easily grasp which portion of the image the focus position is located, and focus adjustment becomes easy.

[0210] As described in the first, second and fourth embodiments, the first image signal in the image processing device (imaging devices 1, 1A and 1C) can be an image signal that does not pass through the band limiting filter 65, and the second image signal can be an image signal that passes through the low-pass filter (LPF 65a).

[0211] The first image signal is usually used to display the focus position in the image. In addition, the second image signal is usually used to display the portion other than the focus position in the image.

[0212] By using a low-pass filter, a plurality of image signals having different frequency characteristics can be generated.

[0213] As described in the third embodiment, the first image signal in the imaging device 1B as the image processing device may be an image signal that has passed through the high pass filter (HPF 65 b ), and the second image signal may be an image signal that has not passed through the band limiting filter 65 .

[0214] The first image signal is an image signal in which high-frequency components are emphasized by a high-pass filter.

[0215] By using a high-pass filter, a plurality of image signals having different frequency characteristics can be generated.

[0216] As described in the first modification, the first image signal in the imaging device 1D as the image processing device may be an image signal that has passed through the high pass filter (HPF 65b), and the second image signal may be an image signal that has passed through the low pass filter (LPF 65a).

[0217] The first image signal is an image signal obtained by passing through a high-pass filter to emphasize high-frequency components, and the second image signal is an image signal obtained by passing through a low-pass filter to emphasize low-frequency components.

[0218] As a result, the difference in frequency characteristics between the first image signal and the second image signal increases, and an image signal in which the degree of focus can be checked more easily can be generated.

[0219] As described in the respective examples, the imaging device 1 (1A, 1B, 1C and 1D) as an image processing device may include a high-frequency band edge detection unit 61 (61A), which extracts an edge detection signal in a high-frequency band from a captured image signal as the peak signal.

[0220] That is, the image processing device includes a high-frequency band edge detection unit.

[0221] In an image processing device, generation of a peak signal based on input of an image signal and mixing processing based on the peak signal are implemented. By extracting an edge detection signal in a high-frequency band as a peak signal, an image position containing many high-frequency components can be specified. As a result, an image area containing a large number of high-frequency components and an image area containing a small number of high-frequency components can be specified, and mixing corresponding to the number of high-frequency components can be performed.

[0222] As described in the various examples, the high-frequency band edge detection unit 61 (61A) in the imaging device 1 (1A, 1B, 1C and 1D) as an image processing device may include a second resolution transformation processing unit 64 (64Aa, 64Ab, ...), which performs resolution transformation processing while maintaining edge information in the high-frequency band.

[0223] As a result, the peak signal used for determining the mixing ratio is a signal that is down-converted without removing the high frequency component.

[0224] That is, since high-frequency components remain in the signal corresponding to the low pixel number after down-conversion, the mixing ratio can be appropriately determined. As a result, the first image signal having more high-frequency components is mixed with the second image signal at the focus position of the image, thereby generating an image in which the focusing degree can be easily checked.

[0225] As described in the various examples, the high-frequency band edge detection unit 61 (61A) in the imaging device 1 (1A, 1B, 1C and 1D) as an image processing device may include a band control unit 63 (63Aa, 63Ab, ...), which controls the high-frequency band based on lens information related to the lens used for imaging.

[0226] The lens information may be, for example, position information of a zoom lens, information such as an F value, or a lens model capable of specifying the information.

[0227] The passband of the band control unit is controlled based on the lens information so as to correspond to the frequency of the edge detection signal to be extracted by the high-frequency band edge detection unit, thereby stabilizing the gain of the edge detection signal.

[0228] As described in the respective examples, the blending processing units (blending units 66 , 66A, 66B, 66C, and 66D) in the imaging apparatus 1 ( 1A, 1B, 1C, and 1D) as the image processing apparatus can perform α blending based on the blending ratio.

[0229] As a result, two image signals having different frequency characteristics are synthesized by the mixing processing unit.

[0230] That is, by synthesizing a natural image using an image processing device, it is possible to generate an image in which the degree of focus can be confirmed.

[0231] The imaging device 1 (1A, 1B, 1C and 1D) includes: an imaging element 51a having a first number of pixels; a display unit 56 (EVF monitor 7 or back monitor 5) having a second number of pixels, the second number of pixels being smaller than the first number of pixels; an image signal generating unit 70 (70A, 70B, 70C and 70D), which generates a plurality of image signals having frequency characteristics different from each other from a captured image signal output from the imaging element 51a; and a mixing processing unit (mixing units 66, 66A, 66B, 66C and 66D), which mixes the plurality of image signals according to a mixing ratio determined based on a peak signal, thereby generating an image signal to be output to the display unit 56.

[0232] For example, the first pixel number is based on a pixel number of 3840 pixels horizontally and 2160 pixels vertically which is a 4K resolution, and the second pixel number is based on a pixel number of 1280 pixels horizontally and 720 pixels vertically which is an HD resolution.

[0233] In the case where a captured image signal captured at a high resolution is displayed on the display unit 56 having a resolution lower than the high resolution, it is necessary to down-convert the image signal. In the down-conversion process, there is a possibility that high-frequency components are lost. According to this configuration, an image signal is generated by mixing an image signal containing a relatively high-frequency component and an image signal from which the relatively high-frequency component is removed, so that an image in which the focusing degree (focus position) can be easily confirmed can be generated.

[0234] Furthermore, by generating an image signal by combining down-converted natural images, it is easier to grasp the degree of focus than by displaying an image in which a peak signal is superimposed.

[0235] As described in the second variant example, the imaging device 1E may include a switching processing unit (control unit 53) which switches between a mode in which the output from the mixing processing unit (mixing unit 66) is input to the display unit 56, and a mode in which the output from the resolution transformation processing unit (first resolution transformation processing unit 60) is input to the display unit 56 without passing through the mixing processing unit.

[0236] An operator for switching may be provided corresponding to the switching processing unit. The operator may be, for example, an operator such as a physical button, or an operator for operating a menu screen by being able to switch from the menu screen may function as a switching operator. In addition, in the case where the display unit 56 is equipped with a touch panel, the display unit 56 may function as a switching operator. That is, by operating an icon or the like arranged on the display unit, the mode may be switched.

[0237] Since it is possible to switch between a mode of displaying an image for confirming the focus and a mode of displaying an image that has been subjected to a normal down-conversion process without performing a blending process, the display can be switched according to whether the focus is confirmed. As a result, an image display according to a user's request can be performed.

[0238] As described in the second embodiment, the mixing ratio can be determined according to the peak signal and the setting of the user (photographer, video engineer, etc.).

[0239] The mixing ratio may be determined based on the peak signal only, based on the user setting only, or based on both the peak signal and the user setting. In addition, these switches may be configured to be settable by the user.

[0240] The mixing ratio is determined according to the user's setting, so that an image corresponding to the user's preference can be displayed on the display unit 56. That is, image display corresponding to the user's purpose can be performed.

[0241] As described in the second embodiment, in the case where a display corresponding to the output from the mixing processing unit (mixing unit 66) is performed on the display unit 56 of the imaging device 1E, a notification unit (display unit 56 or light-emitting unit) can be provided for notifying the user that a display corresponding to the output from the mixing processing unit is performed.

[0242] For example, the notification unit may perform a process of displaying an icon indicating that an image subjected to the blending process is displayed on the display unit 56, or may perform the notification process by providing a light emitting unit such as a notification light emitting diode (LED).

[0243] As a result, it is possible to eliminate the possibility that imaging is performed in a state where an image to which the blending process is applied is displayed on the display unit 56 and in a state where it is not recognized that the image is different from the captured image.

[0244] An image processing method according to the present technology is an image processing method performed by an image processing device, the method comprising: generating a plurality of image signals (for example, a first image signal and a second image signal) having frequency characteristics different from each other from a captured image signal; and mixing the plurality of image signals according to a mixing ratio determined based on a peak signal.

[0245] The program according to the present technology is a program executed by an information processing device (image processing device), and includes a process of generating multiple image signals having frequency characteristics different from each other from a captured image signal, and a process of mixing the multiple image signals according to a mixing ratio determined based on a peak signal.

[0246] <7. Application Examples>

[0247] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be applied to an endoscopic operating room system.

[0248] Fig.13 is a diagram illustrating an example of a schematic configuration of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. Fig.13 The diagram illustrates a state where a surgeon (doctor) 5067 is using an endoscopic surgery system 5000 to perform surgery on a patient 5071 on a bed 5069. As shown in the figure, the endoscopic surgery system 5000 includes an endoscope 5001, other surgical tools 5017, a support arm device 5027 that supports the endoscope 5001, and a trolley 5037 carrying various devices used for endoscopic surgery.

[0249] In endoscopic surgery, instead of cutting the abdominal wall, multiple cylindrical puncture instruments called trocars 5025a to 5025d are punctured into the abdominal wall. Subsequently, the lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 from the trocars 5025a to 5025d. In the example shown in the diagram, as other surgical tools 5017, a pneumoperitoneum tube 5019, an energy treatment tool 5021 and surgical forceps 5023 are inserted into the body cavity of the patient 5071. In addition, the energy treatment tool 5021 is a treatment tool that uses high-frequency current or ultrasonic vibration to cut and peel off tissues, close blood vessels, etc. However, the illustrated surgical tool 5017 is only an example, and as the surgical tool 5017, various surgical tools commonly used in endoscopic surgery, such as forceps and retractors, can be used.

[0250] An image of a surgical field in a body cavity of a patient 5071 captured by an endoscope 5001 is displayed on a display device 5041. While viewing the image of the surgical field displayed in real time on the display device 5041, a surgeon 5067 performs treatment such as excision of an affected area using an energy treatment tool 5021 and a surgical forceps 5023. Note that, although not shown, the pneumoperitoneum tube 5019, the energy treatment tool 5021, and the surgical forceps 5023 are supported by the surgeon 5067, an assistant, etc. during surgery.

[0251] (Support arm equipment)

[0252] The support arm device 5027 includes an arm unit 5031 extending from a base portion 5029. In the illustrated example, the arm unit 5031 includes joint portions 5033a, 5033b, and 5033c, and links 5035a and 5035b, and is driven under the control of an arm control device 5045. The endoscope 5001 is supported by the arm unit 5031, and the position and posture of the endoscope 5001 are controlled. As a result, stable fixation of the position of the endoscope 5001 can be achieved.

[0253] (Endoscope)

[0254] The endoscope 5001 includes a lens barrel 5003 and a camera head 5005 connected to the proximal end of the lens barrel 5003, and a region of a predetermined length from the distal end of the lens barrel 5003 is inserted into the body cavity of a patient 5071. In the example shown in the diagram, the endoscope 5001 is illustrated as a so-called rigid scope including the rigid lens barrel 5003, however, the endoscope 5001 may also be configured as a so-called flexible scope including the flexible lens barrel 5003.

[0255] An opening portion in which an objective lens is installed is provided at the distal end of the lens barrel 5003. A light source device 5045 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the distal end of the lens barrel by a light pipe extending inside the lens barrel 5003, and is emitted to an observation object in a body cavity of a patient 5071 via the objective lens. Note that the endoscope 5001 may be a direct-view endoscope, an oblique-view endoscope, or a side-view endoscope.

[0256] The optical system and the imaging element are arranged inside the camera head 5005, and the reflected light (observation light) from the observation object is focused on the imaging element by the optical system. The observation light is photoelectrically converted by the imaging element to generate an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal is sent to the camera control unit (CCU) 5039 as RAW data. Note that the camera head 5005 has the function of adjusting the magnification and focal length by appropriately driving the optical system.

[0257] Note that, for example, in order to cope with stereoscopic observation (3D display) etc., a plurality of imaging elements may be provided in the camera head 5005. In this case, a plurality of relay optical systems are provided inside the lens barrel 5003 so as to guide observation light to each of the plurality of imaging elements.

[0258] (Various equipment mounted on a cart)

[0259] The CCU 5039 includes a central processing unit (CPU), a graphics processing unit (GPU), etc., and controls the operation of the endoscope 5001 and the display device 5041 as a whole. Specifically, the CCU 5039 performs various image processing, such as development processing (demosaic processing), for displaying an image based on the image signal received from the camera 5005. The CCU 5039 provides the image signal after image processing to the display device 5041. In addition, the CCU 5039 sends a control signal to the camera 5005 to control the driving of the camera 5005. The control signal may include information related to imaging conditions such as magnification and focal length.

[0260] The display device 5041 displays an image based on an image signal that has been image-processed by the CCU 5039 under the control of the CCU 5039. In the case where the endoscope 5001 is compatible with high-resolution imaging such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels), and / or in the case where the endoscope 5001 is compatible with 3D display, for example, a display device capable of high-resolution display and / or a display device capable of 3D display can be used as the display device 5041 corresponding to each case. In the case where the display device is compatible with high-resolution imaging such as 4K or 8K, a more immersive feeling can be obtained by using a display device of 55 inches or more as the display device 5041. In addition, depending on the application, a plurality of display devices 5041 of different resolutions and sizes can be set.

[0261] The light source device 5043 includes, for example, a light source such as a light emitting diode (LED), and supplies irradiation light for imaging the operation field to the endoscope 5001 .

[0262] The arm control device 5045 includes, for example, a processor such as a CPU, and operates according to a predetermined program to control the driving of the arm unit 5031 supporting the arm device 5027 according to a predetermined control method.

[0263] The input device 5047 is an input interface for the endoscopic surgery system 5000. Via the input device 5047, the user can input various information or instructions to the endoscopic surgery system 5000. For example, via the input device 5047, the user inputs various information related to the surgery, such as the patient's physical information and information about the surgical procedure of the surgery. In addition, for example, via the input device 5047, the user inputs an instruction to drive the arm unit 5031, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5001, an instruction to drive the energy treatment tool 5021, and the like.

[0264] The type of input device 5047 is not limited, and input device 5047 may be various known input devices. As input device 5047, for example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057 and / or a joystick, etc. may be applied. In the case of using a touch panel as input device 5047, the touch panel may be provided on the display surface of display device 5041.

[0265] Alternatively, the input device 5047 is a device worn by the user, such as a glasses-type wearable device, a head-mounted display (HMD), etc., and various inputs are performed according to the gestures or sight lines of the user detected by these devices. In addition, the input device 5047 includes a camera capable of detecting the movement of the user, and various inputs are performed according to the gestures or sight lines of the user detected from the video captured by the camera. In addition, the input device 5047 includes a microphone capable of collecting the user's voice, and various inputs are performed using voice via the microphone. As described above, the input device 5047 is configured to be able to input various information in a non-contact manner, so that in particular, users belonging to the clean area (e.g., surgeons 5067) can operate the equipment belonging to the non-clean area in a non-contact manner. In addition, since the user can operate the equipment without putting down the surgical tools held in his hands, the convenience of the user is improved.

[0266] The treatment tool control device 5049 controls the driving of the energy treatment tool 5021 for cauterization and incision of tissues, closure of blood vessels, etc. In order to ensure the field of view of the endoscope 5001 and to ensure the working space of the surgeon, the pneumoperitoneum device 5051 sends gas into the body cavity of the patient 5071 via the pneumoperitoneum tube 5019 so as to expand the body cavity. The recorder 5053 is a device capable of recording various information related to the operation. The printer 5055 is a device capable of printing various information related to the operation in various forms, such as text, images, or graphics.

[0267] The unique structure of the endoscopic surgery system 5000 will be described in more detail below.

[0268] (Support arm equipment)

[0269] The support arm device 5027 includes a base portion 5029 as a base, and an arm unit 5031 extending from the base portion 5029. In the example shown in the diagram, the arm unit 5031 includes a plurality of joint portions 5033a, 5033b, and 5033c, and a plurality of links 5035a and 5035b connected by the joint portion 5033b, but in Fig.13 , for the sake of simplicity, the configuration of the arm unit 5031 is simplified. In fact, the shape, number and arrangement of the joint parts 5033a to 5033c and the connecting rods 5035a and 5035b, the direction of the rotation axis of the joint parts 5033a to 5033c, etc. can be appropriately set so that the arm unit 5031 has a desired degree of freedom. For example, the arm unit 5031 can be appropriately configured to have 6 or more degrees of freedom. As a result, since the endoscope 5001 can be freely moved within the movable range of the arm unit 5031, the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 from a desired direction.

[0270] An actuator is provided in the joint parts 5033a to 5033c, and the joint parts 5033a to 5033c are configured to be able to rotate around a predetermined rotation axis by driving the actuator. The drive of the actuator is controlled by the arm control device 5045, thereby controlling the rotation angle of each joint part 5033a to 5033c, thereby controlling the drive of the arm unit 5031. As a result, the control of the position and posture of the endoscope 5001 can be achieved. At this time, the arm control device 5045 can control the drive of the arm unit 5031 by various known control methods, such as force control or position control.

[0271] For example, by the surgeon 5067 appropriately performing operation input via the input device 5047 (including the foot switch 5057), the arm control device 5045 can appropriately control the drive of the arm unit 5031 according to the operation input, thereby controlling the position and posture of the endoscope 5001. Through this control, the endoscope 5001 at the far end of the arm unit 5031 can be moved from any position to any position and then fixedly supported at the moved position. Note that the arm unit 5031 can be operated by the so-called master-slave method. In this case, the user can remotely operate the arm unit 5031 via the input device 5047 installed in a place away from the operating room.

[0272] Furthermore, in the case of applying force control, the arm control device 5045 can perform so-called power-assisted control that receives external force from the user and drives the actuators of the respective joint parts 5033a to 5033c so that the arm unit 5031 can move smoothly according to the external force. As a result, when the user moves the arm unit 5031 while directly contacting the arm unit 5031, the arm unit 5031 can be moved with less force. Thus, the endoscope 5001 can be moved more intuitively with simpler operations, thereby improving the convenience of the user.

[0273] Here, in endoscopic surgery, endoscope 5001 is usually supported by a doctor called an endoscopist. On the other hand, by using support arm device 5027, the position of endoscope 5001 can be more reliably fixed without manual operation, so that the image of the surgical field can be stably obtained and the surgery can be performed smoothly.

[0274] Note that the arm control device 5045 is not necessarily provided on the cart 5037. In addition, the arm control device 5045 is not necessarily a single device. For example, the arm control device 5045 may be provided in each joint portion 5033a to 5033c of the arm unit 5031 supporting the arm device 5027, so that the drive control of the arm unit 5031 can be realized by a plurality of arm control devices 5045 cooperating with each other.

[0275] (Light source equipment)

[0276] The light source device 5043 provides the endoscope 5001 with irradiation light for imaging the surgical field. The light source device 5043 includes, for example, an LED, a laser light source, or a white light source including a combination thereof. At this time, in the case where the white light source is composed of a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so that the white balance of the captured image can be adjusted in the light source device 5043. In addition, in this case, by irradiating the observed object with laser light from each of the RGB laser light sources in a time-division manner, and controlling the drive of the imaging element of the camera 5005 synchronously with the irradiation timing, it is also possible to capture images corresponding to each RGB in a time-division manner. According to this method, a color image can be obtained even if a color filter is not provided in the imaging element.

[0277] In addition, the drive of the light source device 5043 can be controlled so as to change the intensity of the light to be output at predetermined intervals. By controlling the drive of the imaging element of the camera 5005 in synchronization with the change in the intensity of the light, images are acquired in a time-division manner and synthesized, and a high dynamic range image without so-called underexposed shadows and overexposed bright light parts can be generated.

[0278] In addition, the light source device 5043 can be configured to supply light of a predetermined wavelength band corresponding to special light observation. In special light observation, for example, so-called narrow-band imaging is performed, in which light with a narrower wavelength band than the irradiation light (i.e., white light) during normal observation is emitted by utilizing the wavelength dependence of light absorption in body tissue, and predetermined tissues such as blood vessels in the surface layer of the mucosa are imaged with high contrast. Alternatively, in special light observation, fluorescence observation can be performed to obtain an image using fluorescence generated by irradiation with excitation light. In fluorescence observation, for example, by irradiating body tissue with excitation light, fluorescence from body tissue can be observed (autofluorescence observation), or by locally injecting a reagent such as indocyanine green (ICG) into body tissue and irradiating the body tissue with excitation light corresponding to the fluorescence wavelength of the reagent, a fluorescence image can be obtained. The light source device 5043 can be configured to supply narrow-band light and / or excitation light corresponding to such special light observation.

[0279] (Camera and CCU)

[0280] Will refer to Fig.14 , describing in more detail the functions of the camera 5005 and CCU5039 of the endoscope 5001. Fig.14 It is a graphic description Fig.13 A block diagram of an example of the functional configuration of the camera 5005 and the CCU 5039 shown in the diagram.

[0281] See also Fig.14 As its function, the camera head 5005 includes a lens unit 5007, an imaging unit 5009, a driving unit 5011, a communication unit 5013, and a camera control unit 5015. In addition, as its function, the CCU 5039 includes a communication unit 5059, an image processing unit 5061, and a control unit 5063. The camera head 5005 and the CCU 5039 are connected to each other via a transmission cable 5065 so as to be communicable in both directions.

[0282] First, the functional structure of the camera 5005 is described. The lens unit 5007 is an optical system provided at the connection portion with the lens barrel 5003. The observation light received from the far end of the lens barrel 5003 is guided to the camera 5005 and incident on the lens unit 5007. The lens unit 5007 is composed by combining a plurality of lenses including a zoom lens and a focusing lens. The optical characteristics of the lens unit 5007 are adjusted so that the observation light is converged on the light receiving surface of the imaging element of the imaging unit 5009. In addition, the zoom lens and the focusing lens are configured to be movable in position on the optical axis so as to adjust the magnification and focus of the captured image.

[0283] The imaging unit 5009 includes an imaging element and is arranged at the next stage of the lens unit 5007. The observation light passing through the lens unit 5007 is focused on the light receiving surface of the imaging element, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the imaging unit 5009 is provided to the communication unit 5013.

[0284] As an imaging element constituting the imaging unit 5009, for example, a complementary metal oxide semiconductor (CMOS) type image sensor having a Bayer array and capable of color capture is used. Note that as an imaging element, for example, an imaging element capable of capturing a high-resolution image of 4K or higher can be used. By obtaining an image of the surgical field at a high resolution, the surgeon 5067 can grasp the state of the surgical field in more detail, thereby being able to perform surgery more smoothly.

[0285] In addition, the imaging element constituting the imaging unit 5009 is configured to include a pair of imaging elements for acquiring a right-eye image signal and a left-eye image signal corresponding to 3D display. By performing 3D display, the surgeon 5067 can more accurately grasp the depth of the living tissue in the surgical field. Note that in the case where the imaging unit 5009 is configured as a multi-plate imaging unit, a plurality of lens units 5007 are provided corresponding to the respective imaging elements.

[0286] In addition, the imaging unit 5009 is not necessarily provided in the camera head 5005. For example, the imaging unit 5009 may be provided in the lens barrel 5003 immediately behind the objective lens.

[0287] The driving unit 5011 includes an actuator, and moves the zoom lens and the focus lens of the lens unit 5007 by a predetermined distance along the optical axis under the control of the camera control unit 5015. As a result, the magnification and focus of the image captured by the imaging unit 5009 can be appropriately adjusted.

[0288] The communication unit 5013 includes a communication device for sending and receiving various information to and from the CCU 5039. The communication unit 5013 sends the image signal obtained from the imaging unit 5009 as RAW data to the CCU 5039 via the transmission cable 5065. At this time, in order to display the captured image of the operating area with low latency, it is preferred to send the image signal through optical communication. This is because during the operation, the surgeon 5067 observes the state of the affected area through the captured image while performing the operation. Therefore, in order to make the operation safer and more reliable, it is required to display the moving image of the operating area as real time as possible. In the case of optical communication, the communication unit 5013 is provided with a photoelectric conversion module that converts electrical signals into optical signals. The image signal is converted into an optical signal by the photoelectric conversion module and then sent to the CCU 5039 via the transmission cable 5065.

[0289] In addition, the communication unit 5013 receives a control signal for controlling the drive of the camera 5005 from the CCU 5039. The control signal includes, for example, information related to imaging conditions, such as information specifying a frame rate for capturing an image, information specifying an exposure value during imaging, and / or information specifying a magnification and focus of the captured image. The communication unit 5013 provides the received control signal to the camera control unit 5015. Note that the control signal from the CCU 5039 may also be sent via optical communication. In this case, the communication unit 5013 is provided with a photoelectric conversion module that converts an optical signal into an electrical signal, and the control signal is converted into an electrical signal by the photoelectric conversion module and then provided to the camera control unit 5015.

[0290] Note that imaging conditions such as the above-mentioned frame rate, exposure value, magnification, and focus are automatically set based on the acquired image signal by the control unit 5063 of the CCU 5039. That is, the endoscope 5001 is equipped with a so-called automatic exposure (AE) function, an automatic focus (AF) function, and an automatic white balance (AWB) function.

[0291] The camera control unit 5015 controls the driving of the camera 5005 based on the control signal received from the CCU 5039 via the communication unit 5013. For example, the camera control unit 5015 controls the driving of the imaging element of the imaging unit 5009 based on the information specifying the frame rate of the captured image and / or the information specifying the exposure at the time of imaging. In addition, for example, the camera control unit 5015 appropriately moves the zoom lens and the focus lens of the lens unit 5007 via the drive unit 5011 based on the information specifying the magnification and focus of the captured image. The camera control unit 5015 may also have a function of saving information for identifying the lens barrel 5003 and the camera 5005.

[0292] Note that by arranging components such as the lens unit 5007 and the imaging unit 5009 in a sealed structure with high airtightness and waterproofness, the camera 5005 can withstand high-pressure sterilization.

[0293] Next, the functional structure of CCU 5039 is described. Communication unit 5059 includes a communication device for sending and receiving various information to and from camera 5005. Communication unit 5059 receives an image signal sent from camera 5005 via transmission cable 5065. At this time, as described above, the image signal can be appropriately sent by optical communication. In this case, in order to perform optical communication, communication unit 5059 is provided with a photoelectric conversion module that converts an optical signal into an electrical signal. Communication unit 5059 provides the image signal converted into an electrical signal to image processing unit 5061.

[0294] In addition, the communication unit 5059 transmits a control signal for controlling the driving of the camera 5005 to the camera 5005. The control signal may also be transmitted through optical communication.

[0295] The image processing unit 5061 performs various image processing on the image signal as RAW data sent from the camera 5005. Examples of image processing include various known signal processing, such as development processing, high image quality enhancement processing (band emphasis processing, super-resolution processing, noise reduction (NR) processing and / or camera shake correction processing, etc.), and / or magnification processing (electronic zoom processing). In addition, the image processing unit 5061 performs detection processing on the image signal to perform AE, AF and AWB.

[0296] The image processing unit 5061 includes a processor such as a CPU or a GPU, and the processor operates according to a predetermined program, so that the above-mentioned image processing and detection processing can be performed. Note that in the case where the image processing unit 5061 includes a plurality of GPUs, the image processing unit 5061 appropriately divides the information related to the image signal and performs image processing in parallel through the plurality of GPUs.

[0297] The control unit 5063 performs various controls related to the imaging of the surgical field using the endoscope 5001 and the display of the captured image. For example, the control unit 5063 generates a control signal for controlling the driving of the camera 5005. At this time, in the case where the user inputs the imaging conditions, the control unit 5063 generates a control signal based on the user's input. Alternatively, in the case where the endoscope 5001 has an AE function, an AF function, and an AWB function, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance according to the result of the detection processing performed by the image processing unit 5061, and generates a control signal.

[0298] In addition, the control unit 5063 causes the display device 5041 to display the image of the surgical field based on the image signal processed by the image processing unit 5061. At this time, the control unit 5063 uses various image recognition technologies to identify various objects in the surgical field image. For example, the control unit 5063 can identify surgical tools such as surgical forceps, specific living parts, bleeding, mist when using the energy treatment tool 5021, etc. by detecting the shape, color, etc. of the edge of the object contained in the surgical field image. When the image of the surgical field is displayed on the display device 5041, the control unit 5063 uses the recognition result to superimpose various surgical support information on the image of the surgical field. The surgical support information is superimposed and presented to the surgeon 5067, so that the operation can be continued more safely and reliably.

[0299] The transmission cable 5065 connecting the camera head 5005 and the CCU 5039 is an electrical signal cable compatible with electrical signal communication, an optical fiber compatible with optical communication, or a composite cable thereof.

[0300] Here, in the example shown in the diagram, communication is performed by wire using the transmission cable 5065, but communication between the camera 5005 and the CCU 5039 may be performed wirelessly. In the case where communication between the camera 5005 and the CCU 5039 is performed wirelessly, it is not necessary to lay the transmission cable 5065 in the operating room, so that the transmission cable 5065 can be eliminated from interfering with the movement of medical staff in the operating room.

[0301] An example of an endoscopic operating room system 5000 to which the technology of the present disclosure can be applied is described above. Note that the endoscopic operating room system 5000 is described as an example, but the system to which the technology of the present disclosure can be applied is not limited to such an example. For example, the technology of the present disclosure can be applied to a flexible endoscope system for inspection or a microsurgery system.

[0302] The technology according to the present disclosure can be appropriately applied to the CCU 5039 including the image processing unit 5061 in the above-mentioned configuration. By applying the technology according to the present disclosure to the CCU 5039, for example, the degree of focus on the surgical field can be easily confirmed. Thus, the surgery can be performed safely and reliably while confirming that the surgical field is clearly displayed.

[0303] <8. This technology>

[0304] The present technology may also adopt the following configurations.

[0305] (1) An image processing device comprising:

[0306] an image signal generating unit that generates a plurality of image signals having frequency characteristics different from each other from one captured image signal; and

[0307] A mixing processing unit mixes the plurality of image signals based on a mixing ratio determined based on a peak signal with respect to the one captured image signal.

[0308] (2) The image processing device according to (1), wherein

[0309] The image signal generating unit includes a first resolution conversion processing unit that performs a resolution conversion process of converting an input image signal having a first resolution into an output signal having a second resolution.

[0310] (3) The image processing device according to (2), wherein

[0311] The plurality of image signals include a first image signal and a second image signal, and

[0312] The first image signal is an image signal containing more high-frequency components than the second image signal.

[0313] (4) The image processing device according to (3), wherein

[0314] The first image signal is an image signal that has not passed through a band-limiting filter,

[0315] The second image signal is an image signal that has passed through a low-pass filter.

[0316] (5) The image processing device according to (3), wherein

[0317] The first image signal is an image signal that has passed through a high-pass filter.

[0318] The second image signal is an image signal that has not passed through a band-limiting filter.

[0319] (6) The image processing device according to (3), wherein

[0320] The first image signal is an image signal that has passed through a high-pass filter.

[0321] The second image signal is an image signal that has passed through a low-pass filter.

[0322] (7) The image processing device according to any one of (1) to (6), further comprising

[0323] A high-frequency band edge detection unit extracts an edge detection signal in a high-frequency band from the one captured image signal as the peak signal.

[0324] (8) The image processing device according to (7), wherein

[0325] The high-frequency band edge detection unit includes a second resolution conversion processing unit that performs resolution conversion processing while maintaining edge information of the high-frequency band.

[0326] (9) The image processing device according to any one of (7) or (8), wherein

[0327] The high-frequency band edge detection unit includes a band control unit that controls the high-frequency band based on lens information about a lens used for imaging.

[0328] (10) The image processing device according to any one of (1) to (9), wherein

[0329] The blending processing unit performs alpha blending based on the blending ratio.

[0330] (11) An imaging device comprising:

[0331] an imaging element having a first number of pixels;

[0332] a display unit having a second number of pixels, the second number of pixels being less than the first number of pixels;

[0333] an image signal generating unit that generates a plurality of image signals having frequency characteristics different from each other from one captured image signal output from the imaging element; and

[0334] A mixing processing unit is configured to mix the plurality of image signals according to a mixing ratio determined based on a peak signal to generate an image signal to be output to the display unit.

[0335] (12) The imaging device according to (11), further comprising

[0336] A switching processing unit configured to switch between a mode in which the output from the mixing processing unit is input to the display unit and a mode in which the output from the resolution conversion processing unit is input to the display unit without passing through the mixing processing unit.

[0337] (13) The imaging device according to any one of (11) or (12), wherein

[0338] The mixing ratio is determined according to the peak signal and user settings.

[0339] (14) The imaging device according to any one of (11) to (13), further comprising:

[0340] A notification unit is configured to notify a user that a display corresponding to the output from the mixing processing unit is being performed when a display corresponding to the output from the mixing processing unit is being performed on the display unit.

[0341] (15) An image processing method performed by an image processing device, the method comprising:

[0342] generating a plurality of image signals having frequency characteristics different from each other from one captured image signal; and

[0343] The plurality of image signals are mixed according to a mixing ratio determined based on the peak signal.

[0344] (16) A program for causing an information processing device to execute the following processing:

[0345] generating a plurality of image signals having frequency characteristics different from each other from one captured image signal; and

[0346] The plurality of image signals are mixed according to a mixing ratio determined based on the peak signal.

[0347] Reference numerals list

[0348] 1,1A,1B,1C,1D,1E Imaging Device

[0349] 51a Imaging element

[0350] 53 Control Unit

[0351] 56 Display Unit

[0352] 60 first resolution conversion processing unit

[0353] 61,61 High frequency band edge detection unit

[0354] 63,63Aa,63Ab Band Control Unit

[0355] 64,64Aa,64Ab Second resolution conversion processing unit

[0356] 65 Band-limiting filter

[0357] 65a,65a1,65a2 LPF

[0358] 65b HPF

[0359] 66,66A,66B,66C,66D Mixed Unit

[0360] 70,70A,70B,70C,70D Image signal generation unit

Claims

1. An image processing device, include: A processing circuit, the processing circuit being configured to: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal, wherein the plurality of image signals include a first image signal and a second image signal obtained by passing the one captured image signal through a high-pass filter or a low-pass filter, and wherein the first image signal has more high-frequency components than the second image signal; extracting an edge detection signal in a high frequency band from the one captured image signal as a peak signal; determining a mixing ratio based at least on the peak signal with respect to the one captured image signal, wherein for pixels having a large peak signal, the ratio of the first image signal is increased and the ratio of the second image signal is decreased, and for pixels having a small peak signal, the ratio of the first image signal is decreased and the ratio of the second image signal is increased; and The plurality of image signals are mixed according to the mixing ratio determined based on the peak signal with respect to the one captured image signal.

2. The image processing apparatus according to claim 1, wherein The processing circuit is further configured to perform a resolution conversion process of converting an input image signal having a first resolution into an output signal having a second resolution.

3. The image processing apparatus according to claim 1, wherein The first image signal is an image signal that has not passed through a band-limiting filter, The second image signal is an image signal that has passed through a low-pass filter.

4. The image processing apparatus according to claim 1, wherein The first image signal is an image signal that has passed through a high-pass filter. The second image signal is an image signal that has not passed through a band-limiting filter.

5. The image processing apparatus according to claim 1, wherein The first image signal is an image signal that has passed through a high-pass filter. The second image signal is an image signal that has passed through a low-pass filter.

6. The image processing apparatus according to claim 1, wherein The processing circuit is further configured to perform resolution conversion processing while maintaining edge information of a high frequency band.

7. The image processing apparatus according to claim 1, wherein The processing circuit is further configured to control the high frequency band based on lens information related to a lens used for imaging.

8. The image processing apparatus according to claim 1, wherein The processing circuit is further configured to perform alpha blending based on the blending ratio.

9. An imaging device, include: an imaging element having a first number of pixels; a display having a second number of pixels, the second number of pixels being less than the first number of pixels; A processing circuit, the processing circuit being configured to: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal output from the imaging element, wherein the plurality of image signals include a first image signal and a second image signal obtained by passing the one captured image signal through a high-pass filter or a low-pass filter, and wherein the first image signal has more high-frequency components than the second image signal; extracting an edge detection signal in a high frequency band from the one captured image signal as a peak signal; determining a mixing ratio based at least on the peak signal with respect to the one captured image signal, wherein for pixels having a large peak signal, the ratio of the first image signal is increased and the ratio of the second image signal is decreased, and for pixels having a small peak signal, the ratio of the first image signal is decreased and the ratio of the second image signal is increased; and The plurality of image signals are mixed according to the mixing ratio determined based on the peak signal to generate an image signal output to the display.

10. The imaging device according to claim 9, wherein The processing circuit is further configured to switch between a mode in which the output from the blending is input to the display and a mode in which the output from the resolution conversion is input to the display without the blending.

11. The imaging device according to claim 9, wherein The processing circuit is further configured to determine the mixing ratio according to a user setting.

12. The imaging device according to claim 9, wherein The processing circuit is further configured to, in a case where a display corresponding to the output from the mix is ​​performed on the display, notify a user that the display corresponding to the output from the mix is ​​performed.

13. An image processing method performed by an image processing device, the method include: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal, wherein the plurality of image signals include a first image signal and a second image signal obtained by passing the one captured image signal through a high-pass filter or a low-pass filter, and wherein the first image signal has more high-frequency components than the second image signal; extracting an edge detection signal in a high frequency band from the one captured image signal as a peak signal; determining a mixing ratio based at least on the peak signal with respect to the one captured image signal, wherein for pixels having a large peak signal, the ratio of the first image signal is increased and the ratio of the second image signal is decreased, and for pixels having a small peak signal, the ratio of the first image signal is decreased and the ratio of the second image signal is increased; and The plurality of image signals are mixed according to the mixing ratio determined based on the peak signal.

14. A program product for causing an information processing device to execute the following processing: generating a plurality of image signals having frequency characteristics different from each other from one captured image signal, wherein the plurality of image signals include a first image signal and a second image signal obtained by passing the one captured image signal through a high-pass filter or a low-pass filter, and wherein the first image signal has more high-frequency components than the second image signal; extracting an edge detection signal in a high frequency band from the one captured image signal as a peak signal; determining a mixing ratio based at least on the peak signal with respect to the one captured image signal, wherein for pixels having a large peak signal, the ratio of the first image signal is increased and the ratio of the second image signal is decreased, and for pixels having a small peak signal, the ratio of the first image signal is decreased and the ratio of the second image signal is increased; and The plurality of image signals are mixed according to the mixing ratio determined based on the peak signal.