Image capturing device, image capturing method and program

Abstract

An imaging device according to an embodiment of the present technology is provided with an image generation unit, an edge detection unit, and a color control unit. The image generation unit generates a captured image obtained by capturing an image of a subject. The edge detection unit detects an edge portion included in the generated captured image. The color control unit controls the color of an emphasis display for emphasizing the edge portion based on the color information of the edge portion in the captured image for each detected edge portion.

Description

Technical Field

[0001] The present technology relates to an image capturing device, an image capturing method, and a program for capturing an image of a subject. Background Art

[0002] Conventionally, a technology has been developed to assist the focusing operation when taking an image of a subject. For example, in the focused portion, the image becomes clear and the outline (edge) of the subject can be detected. This detection result is used to generate an image representing the focused state, etc.

[0003] Patent document 1 discloses a video signal processing device for emphasizing the contour portion of an image using a contour emphasis signal extracted from a video signal. In the video signal processing device, a contour emphasis signal is generated in which the level becomes higher as the correct focus is approached. For example, the contour emphasis signal is added to the color difference signal according to its level. Therefore, as the focus is gradually approached from an out-of-focus state to a correct focus, the color of the contour becomes darker. This makes it possible to visually grasp the change in the focus state. (Paragraphs

[0020] ,

[0045] ,

[0068] ,

[0081] and Figure 1 and Figure 8 wait)

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2010-135865 Summary of the invention

[0007] Technical issues

[0008] A focusing operation may be performed by visually emphasizing and displaying an edge portion that is in focus, and a technology capable of smoothly improving the visibility of such emphasized display is required.

[0009] In view of the above circumstances, an object of the present technology is to provide an image capturing device, an image capturing method, and a program capable of improving the visibility of emphasized display of an edge portion.

[0010] Solution to the problem

[0011] In order to achieve the above-mentioned object, an image capturing device according to an embodiment of the present technology includes an image generating unit, an edge detecting unit and a color controlling unit.

[0012] The image generation unit generates a captured image by capturing an image of a subject.

[0013] The edge detection unit detects an edge portion included in the generated captured image.

[0014] The color control unit controls, for each detected edge portion, a color of an emphasis display for emphasizing the edge portion based on color information of the edge portion in the captured image.

[0015] In the image capturing apparatus, a captured image of a subject is generated, and an edge portion included in the captured image is detected. In addition, for each detected edge portion, by using color information about the edge portion in the captured image, the color of an emphasis display for emphasizing the edge portion is controlled. This makes it possible to improve the visibility of the emphasis display of the edge portion.

[0016] The color control unit controls the color of the emphasized display so that a color difference between an edge color represented by the color information of the edge portion and the color of the emphasized display becomes larger.

[0017] The color control unit controls the emphasized color so that the edge color and the emphasized color are colors that are far apart in the hue plane or the brightness direction.

[0018] The color control unit sets the color of the emphasized display according to a plurality of color candidates.

[0019] The plurality of color candidates include chromatic color candidates. In this case, the color control unit sets a color candidate having a hue farthest from a hue of the edge color among the plurality of color candidates as the emphasized display color.

[0020] The plurality of color candidates may include a first color candidate and a second color candidate having a different hue from the first color candidate.

[0021] The first color candidate and the second color candidate may be set to colors that are complementary to each other.

[0022] The plurality of color candidates may include achromatic color candidates. In this case, the color control unit may set a color candidate having a brightness farthest from the brightness of the edge color among the plurality of color candidates as a color for emphasized display.

[0023] The plurality of color candidates may include black and white. When the brightness of the edge color is greater than a predetermined threshold, the color control unit may set the emphasized color to black, and when the brightness of the edge color is less than a predetermined threshold, the emphasized color may be set to white.

[0024] The plurality of color candidates may be set by a user.

[0025] The plurality of color candidates may be set according to the subject.

[0026] The edge detection unit may detect the pixel position of the edge portion. In this case, the color control unit may set the color of the emphasized display for each detected pixel position.

[0027] The color information of the edge portion may include pixel information of reference pixels included in a predetermined pixel area surrounding a pixel position of the edge portion.

[0028] The color control unit may calculate the hue of the edge color or the brightness of the edge color based on the pixel information of the reference pixel.

[0029] The color control unit may calculate an average hue of the reference pixels as the hue of the edge color.

[0030] The predetermined pixel area may be set according to the object.

[0031] The color control unit may generate a peak image in which an edge portion of the captured image is emphasized with the emphasized display color.

[0032] In the peak image, a color of a portion different from the edge portion may be set to the same color as that of the original image.

[0033] The color control unit may control the color of the emphasized display so that the display of the edge portion of the peak image changes dynamically.

[0034] An image capturing method according to one embodiment of the present technology is an image capturing method executed by a computer system, and includes generating a captured image in which an image of a subject is captured.

[0035] An edge portion included in the generated captured image is detected.

[0036] For each detected edge portion, a color of an emphasis display for emphasizing the edge portion is controlled based on color information of the edge portion in the captured image.

[0037] A program according to an embodiment of the present technology causes a computer system to execute the following steps:

[0038] generating a captured image in which an image of a subject is captured;

[0039] detecting an edge portion included in the generated captured image; and

[0040] For each detected edge portion, a color of an emphasis display for emphasizing the edge portion is controlled based on color information of the edge portion in the captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 : is a schematic diagram showing a configuration example of an image pickup device according to an embodiment of the present technology.

[0042] Figure 2 is an example of a peak image in which each edge portion of each subject is emphasized.

[0043] Figure 3 A diagram for explaining the hue circle.

[0044] Figure 4 is a flowchart illustrating an example of a peaking display using color overlay colors.

[0045] Figure 5 is an image of a hue ring as an example of an original image.

[0046] Figure 6 It shows that from Figure 5 The image shown is an image of the detected edge parts of the original image.

[0047] Figure 7 is a schematic diagram showing an example of pixel arrangement around edge pixels.

[0048] Figure 8 It shows Figure 6 A schematic diagram showing an example of the average tone of an edge portion is shown.

[0049] Fig. 9 are schematic diagrams each showing an example of a peaking display with colors superimposed in color.

[0050] Fig.10 are schematic diagrams each illustrating an example of peaking display as a comparative example.

[0051] Fig.11 is a flowchart showing an example of peaking display using black and white superimposed colors.

[0052] Fig.12 An image of a stripe pattern is shown as an example of an original image.

[0053] Fig.13 It shows that from Fig.12 The image shown is an image of the detected edge parts of the original image.

[0054] Fig.14 It shows Fig.13 Schematic diagram of an example of brightness in an original image of an edge portion 41 shown in FIG.

[0055] Fig.15 are schematic diagrams each showing an example of peaking display using black and white superimposed colors.

[0056] Fig.16 A peak image is shown as a comparative example.

[0057] Fig.17 are schematic diagrams each for explaining an application example of peaking display.

[0058] Fig.18 is a diagram showing an example of a schematic configuration of an endoscopic surgery system.

[0059] Fig.19 It shows Fig.18 A block diagram of an example of the functional configuration of a camera and a CCU is shown in FIG.

[0060] Fig. 20 is a diagram schematically showing the overall configuration of an operating room system.

[0061] Fig.21 : is a figure which shows the display example of the operation screen on the centralized operation panel.

[0062] Fig. 22 : is a diagram showing an example of a state of surgery to which the operating room system is applied.

[0063] Fig.23 It shows Fig. 22 A block diagram of an example of the functional configuration of a camera and a CCU is shown in FIG.

[0064] Fig.24 is a diagram showing an example of a schematic configuration of a microscope surgery system.

[0065] Fig.25 It shows the use of Fig.24 Figure 2 shows a surgical procedure using the microscope surgical system. DETAILED DESCRIPTION

[0066] Embodiments according to the present technology will be described below with reference to the drawings.

[0067] [Configuration of image capturing device]

[0068] Figure 1 1 is a schematic diagram showing a configuration example of an image pickup device according to an embodiment of the present technology. The image pickup device 100 is a camera device (camera system) for picking up an image of a subject. The image pickup device 100 is configured as, for example, a video camera or a digital still camera.

[0069] The image capturing apparatus 100 includes an image capturing optical system 10, an image sensor 11, a camera controller 12, a video processing unit 13, a recording unit 14, a display device 15, and a user operation unit 16. Incidentally, the recording unit 14 and the display device 15 may be provided outside the image capturing apparatus 100.

[0070] The image capturing optical system 10 includes a lens unit 21 and a lens control unit 22. The lens unit 21 is, for example, an optical system including a plurality of lenses arranged in a lens barrel, and focuses light incident on the lens barrel on a light receiving surface of an image sensor 11 to be described later. The specific configuration of the lens unit 21 is not limited, and a lens according to, for example, the type of subject and the application of the image capturing device 100 can be appropriately used.

[0071] The lens control unit 22 is an operating mechanism for adjusting the optical characteristics of the lens unit 21 (focus, magnification, aperture, etc.). The lens control unit 22 is configured to correspond to, for example, manual focus. For example, by operating the lens control unit 22, the positions of the plurality of lenses are changed, and an operation of adjusting the focus (focusing operation) can be performed. The specific configuration of the lens control unit 22 is not limited, and a mechanism for adjusting, for example, the position of the lens, etc., by electronic control, etc., etc. may be used.

[0072] The image sensor 11 is an image capturing element for receiving light incident on a light receiving surface. The image sensor 11 is arranged so that the light receiving surface faces the lens unit 21, and generates image data by receiving the subject light 23 passing through the lens unit 21. For example, the image data is sequentially generated at a predetermined frame rate. As the image sensor 11, for example, an image capturing element such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor is used.

[0073] Image data is data constituting a captured image by capturing an image of a subject. In the present disclosure, generating image data constituting a captured image corresponds to generating a captured image. Image data includes pixel data for representing a plurality of pixels constituting the captured image. Pixel data is transmitted as, for example, a digital signal (pixel signal). Therefore, it can be said that image data is generated as a collection of a plurality of pixel signals (image signal). In the present embodiment, pixel data corresponds to pixel information.

[0074] Each pixel data includes, for example, RGB data indicating the intensity of red light, green light, and blue light included in the subject light 23. Note that the RGB data can be uniquely converted into, for example, YCrCb data representing a color with brightness Y, red difference Cr, and blue difference Cb. In this way, a configuration can be used in which image data representing a color is generated with brightness and color difference. In addition, the format, type, etc. of the image data are not limited.

[0075] The camera controller 12 is an arithmetic unit that controls the operation of each unit of the image pickup apparatus 100, and has a configuration required for a computer, such as a CPU, a memory (RAM, ROM), etc. The camera controller 12 controls the operation of the image sensor 11 and the video processing unit 13 based on, for example, parameters input through a user operation unit 16 (to be described later). In addition, electronic control of the lens control unit 22, operation control of the display device 15, etc. can be performed.

[0076] The video processing unit 13 is an arithmetic unit that performs various types of processing on the image data (video data) generated by the image sensor 11 at a predetermined frame rate. The video processing unit 13 includes, for example, a register (memory) for recording settings for performing various types of arithmetic processing. By rewriting the register and implementing a predetermined program, desired digital signal processing can be performed.

[0077] The video processing unit 13 is configured by using an IC (Integrated Circuit) for video processing, such as an imaging processor. Incidentally, the camera controller 12 may be configured on an imaging processor together with the video processor 13. For example, by installing and executing the program according to the present embodiment in the RAM of the camera controller 12 and the register of the video processing unit 13, the image capturing method according to the present embodiment is performed by each unit of the image capturing apparatus 100.

[0078] like Figure 1 As shown, the video processing unit 13 includes a camera signal processing unit 30, a peaking detection unit 31, an overlay color setting unit 32, an overlay color selection unit 33, and a synthesis unit 34. Each of these modules is configured in an imaging processor, for example. In addition, these modules can be configured using a dedicated device such as a PLD (Programmable Logic Device), such as an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

[0079] The camera signal processing unit 30 performs correction processing, conversion processing, etc. on the image data generated by the image sensor 11. That is, it can be said that the camera signal processing unit 30 generates image data (captured image) to which various types of processing are applied.

[0080] In the camera signal processing unit 30, correction processing is performed, for example, image capturing system correction such as brightness correction (shading correction), gain correction of each pixel data (pixel signal), gamma correction for corresponding to the monitor gamma value, normalization correction (knee correction) for normalizing the signal level changed due to each correction, etc. For example, the correction processing is sequentially performed so that a captured image is appropriately generated. Alternatively, the camera signal processing unit 30 may perform conversion processing, etc., so as to convert RGB data into YCrCb data, etc. In addition, the type of each processing performed on the image data, etc. is not limited.

[0081] Therefore, the image capturing apparatus 100 generates a captured image by capturing an image of a subject by the image sensor 11 and the camera signal processing unit 30. In the present embodiment, the image sensor 11 and the camera signal processing unit 30 function as an image generating unit.

[0082] The image data to which each process is applied by the camera signal processing unit 30 is output to the main line of the image pickup device 100, for example, as data recorded in a recording unit, or as data used in broadcasting or the like. Figure 1 As shown, in the image capturing apparatus 100, processing by other units of the video processing unit 13 is performed based on the image data output by the camera signal processing unit 30. Hereinafter, the image data output from the camera signal processing unit 30 will be referred to as raw image data, and an image composed of the captured image data will be referred to as a raw image.

[0083] The peaking detection unit 31 detects a portion of the original image to be subjected to peaking display. The peaking display is an emphasis display that colors a focused portion in an image. Specifically, the peaking detection unit 31 detects an edge portion included in the original image (captured image) generated by the image sensor 11 and the camera signal processing unit 30. In the present embodiment, the peaking detection unit 31 corresponds to an edge detection unit.

[0084] For example, for each pixel included in the original image, a high-pass filtering process is performed to extract a component in a high-frequency band. More specifically, the original image data (pixel signal) is input to the high-pass filter, and edge detection data representing the position of a pixel including a high-frequency component is generated. The pixel including the high-frequency component is detected as an edge portion. Therefore, the peaking detection unit detects the pixel position of the edge portion. Therefore, each position of the edge portion can be specified with high precision. By the way, the method of detecting the edge portion is not limited, and any edge detection process can be used.

[0085] For each edge portion detected by the peaking detection unit 31, the superimposed color setting unit 32 controls an emphasized color for emphasizing the edge portion based on the color information about the edge portion in the original image. That is, the superimposed color setting unit 32 sets the color of emphasized display of each edge portion (relative to the superimposed color of the original image) in consideration of the original image. Hereinafter, the color for emphasis may be referred to as a superimposed color.

[0086] The color information about the edge portion is information indicating the color of the edge portion in the original image and the color around the edge portion. Information included in the pixel data of the original image is generally used as the color information. For example, information indicating the RGB intensity (RGB data, etc.) of each pixel of the original image or information indicating the brightness and color difference (YCrCb data, etc.) of each pixel of the original image is color information.

[0087] The setting of the superimposed color is performed for each edge portion. Therefore, each edge portion included in the original image does not have to be emphasized with a single color, and the superimposed color may be different for each edge portion. This makes it possible, for example, to fully emphasize each edge portion regardless of the color of each edge portion in the original image.

[0088] In this embodiment, a superimposed color is set for each pixel position detected as an edge portion. For example, a superimposed color is set for each pixel of the edge portion specified by the edge detection data, and edge enhancement data indicating the superimposed color and the pixel position is generated. The edge enhancement data constitutes a superimposed image superimposed on the original image in the peaking display.

[0089] The superimposed color setting unit 32 sets the superimposed color (emphasized color) from a plurality of color candidates. That is, each edge portion is emphasized using any color candidate among the plurality of color candidates. As described above, by presetting a plurality of color candidates, the process of setting the superimposed color is the process of selecting a color candidate. This makes it possible to easily set the superimposed color of each edge portion.

[0090] In this embodiment, a plurality of color candidates are set by the user. For example, a plurality of color candidates are set by the user, and the information of each color candidate is held by the superimposed color selection unit 33 described below. Based on the color candidate set by the user, the superimposed color of each edge portion is set.

[0091] Each color candidate is set by the user operation unit 16. For example, a plurality of colors are displayed on the display device 15, and a candidate color is selected from these colors. For example, by selecting a basic color by the user, another candidate color can be automatically set. In addition, the method of setting a plurality of candidate colors is not limited.

[0092] In this way, since the user himself can set the candidates of the superimposed color, the emphasized edge portion can be easily recognized visually. The method of setting the superimposed color (emphasis color) and the type of candidate colors will be described in detail later.

[0093] The superimposed color selection unit 33 accepts a user's selection of a plurality of color candidates. For example, information on a color candidate set by the user by operating the user operation unit 16 (described later) is input to the superimposed color selection unit 33 via the camera controller 12. The superimposed color selection unit 33 holds the input information of the color candidate and outputs it to the superimposed color setting unit 32 as appropriate. Note that a default candidate may be set as the plurality of color candidates and stored in the superimposed color selection unit 33.

[0094] The synthesis unit 34 synthesizes an image in which the superimposed image (edge ​​enhancement data) generated by the superimposed color setting unit 32 is superimposed on the original image (original image data) generated by the camera signal processing unit 30. Therefore, the image synthesized by the synthesis unit 34 is a peak image in which the focused portion (edge ​​portion) in the original image is displayed in the superimposed color. In this way, the synthesis unit generates a peak image in which the edge portion of the original image is emphasized in the superimposed color.

[0095] For example, image data in which pixel data corresponding to the edge portion of the original image data is replaced with pixel data representing the superimposed color is generated as a peak image. The method of synthesizing the original image and the superimposed image is not limited, and can be appropriately used, for example, a circuit capable of superimposing other displays on an image.

[0096] As described above, the peak image is an image in which the color of the edge portion is set as the superimposed color, and the color of the portion different from the edge portion is set to the same color as the original image. That is, the peaking display is limited to the edge portion, and the normal image (original image) is used as it is except for the edge portion. This allows the user to utilize the peaking display without discomfort.

[0097] The peak image 40 generated by the synthesis unit 34 is output to the display device 15. In the present embodiment, the superimposed color setting unit 32, the superimposed color selecting unit 33 and the synthesis unit 34 function as a color control unit.

[0098] The recording unit 14 is a nonvolatile recording device. The recording unit 14 is connected to the main line of the image capturing apparatus 100, and records the original image (original image data) output by the camera signal processing unit 30. As the recording unit 14, for example, a magnetic tape, an optical disk, an HDD (hard disk drive), an SSD (solid state drive), etc. can be used. The specific configuration of the recording unit 14 is not limited.

[0099] The display device 15 is a display device for displaying a peak image. As the display device 15, for example, a viewfinder (VF) of a camera (image capturing apparatus 100) can be used. In addition, for example, an external display using a liquid crystal, an organic EL, etc. can be used as the display device 15. The external display may be a sub-display of the image capturing apparatus 100 main body, or may be a display provided separately from the image capturing apparatus 100.

[0100] For example, the user is a photographer (videographer), and watches the peak image displayed on the display device 15, operates the lens control unit 22, etc., thereby operating the focus of the image capture apparatus 100. In the peak image, the focused portion is emphasized with a superimposed color. This allows the user to capture a subject while confirming the position of the focus in the screen.

[0101] Note that it is conceivable that the resolution of the display device 15 is different from the resolution of the image sensor 11. For example, the image sensor 11 is a 4k resolution (3840 pixels horizontally, 2160 pixels vertically), and the display device 15 is a high definition (HD) resolution (1920 pixels horizontally, 1080 pixels vertically). Of course, the present technology is also applicable to the case of using other configurations.

[0102] The resolution of a small display device 15 such as a viewfinder may be lower than the resolution of the image sensor 11. In this case, for example, by the camera signal processing unit 30, a raw image (raw image data) or the like that is down-converted according to the resolution of the display device 15 may be generated. The processes of the peaking detection unit 31, the superimposed color setting unit 32, and the synthesis unit 34 are appropriately performed on the down-converted raw image to generate a peak image suitable for the resolution of the display device 15. Alternatively, the peak image output from the synthesis unit 34 may be down-converted.

[0103] The user operation unit 16 accepts user operations on the image pickup apparatus 100. For example, the color candidates of the above-mentioned superimposed color are set through the user operation unit 16. The user operation unit 16 includes, for example, various switches such as buttons and levers, sliders, knobs, etc. In addition, for example, the display device 15 used as a touch screen can be configured as the user operation unit 16. In addition, the specific configuration of the user operation unit 16 is not limited.

[0104] [Basic operation of the image capturing device]

[0105] Figure 2 is an example of a peak image in which each edge portion of each subject is emphasized. Figure 2 The illustrated peak image 40 and the like describe the basic operation of the image capture device 100 .

[0106] Figure 2 A table 1a on which left and right trays 1b and 1c are arranged, and a chair 1d placed behind the table 1a are schematically shown using gray lines as the subject 1. Here, it is assumed that the image capture device 100 focuses on a portion of the left tray 1B and a portion of the right tray 1C.

[0107] The black line in the figure indicates the highlighted display 42 of the edge portion 41. Figure 2 As shown in FIG. 1 , the emphasis display 42 is superimposed and displayed on a portion (edge ​​portion 41) of the subject 1. Figure 2 In the example shown, two types of color candidates are used as the colors of the emphasized display 42. Hereinafter, the two types of color candidates are described as a first color candidate 43a and a second color candidate 43b. The emphasized display 42 in which the first and second color candidates 43a and 43b are set as superimposed colors is indicated by a black solid line and a black dotted line. The peak image 40 is generally a color image.

[0108] For example, light from the subject 1 (subject light 23) enters the image sensor 11 through the lens unit 21, and an image of the subject 1 is captured. Image data generated by the image sensor 11 is processed by the camera signal processing unit 30, and recorded as raw image data (raw image) in the recording unit 14. After receiving the peaking display processing, the raw image data is displayed on the display device 15 as a peak image 40. Therefore, the user can perform the operation of the lens control unit 22 while viewing the peak image 40 displayed on the display device 15.

[0109] In the peaking display process, the edge portion 41 of the original image is detected by the peaking detection unit 31. The edge portion 41 becomes a focus position that is accurately focused. The superimposed color setting unit 32 sets the superimposed color of the edge portion 41 in consideration of the color of the original image. Specifically, the superimposed color is selected from a plurality of color candidates 43 (e.g., first and second color candidates 43a and 43b) set by the user and held in the superimposed color selection unit 33, according to the color in the original image.

[0110] The setting of the superimposed color is performed based on the edge color of the edge portion 41 in the original image. Here, the edge color is a color represented by the color information about the edge portion 41. For example, it is assumed that a certain pixel is detected as the edge portion 41. The color represented by the color information of the pixel as the edge portion 41, that is, the color of the pixel in the original image, is used as the edge color. In this case, the edge color is the color of the edge portion 41 itself.

[0111] In addition, for example, a color represented by color information of peripheral pixels including those surrounding the edge portion 41 may be used as the edge color. For example, based on the color information of the pixels included in the periphery, the average value of the colors of the respective pixels is calculated and used as the edge color. In this case, the edge color is the average color of the peripheral portion including the edge portion.

[0112] In the image capturing device 100, the superimposed color is set so that the edge color and the superimposed color are far away from each other. In the present disclosure, the colors far away from each other are, for example, colors far away from each other on the hue plane, or colors far away from each other in brightness. It can also be said that the color difference of colors far from each other is large. Therefore, the superimposed color is set to a color that is not similar to the edge color (color with a large difference) in hue or brightness. That is, a color candidate that is not similar to the edge color is selected as the superimposed color.

[0113] For example, in Figure 2 In the peak image 40 shown, it is assumed that the first color candidate 43a is set to red, and the second color candidate is set to light blue. At this time, it is assumed that the left plate 1B is blue and the right plate C is orange. In this case, the edge color of the edge portion 41 detected from the left plate 1b is a color with many blue components. The edge color of the edge portion 41 detected from the right plate 1c is a color with many red and yellow components.

[0114] For example, in the emphasized display 42 of the edge portion 41 of the left plate 1b, the first color candidate 43a (red) which is a color far from the color having many blue components is set as the superimposed color. On the other hand, in the emphasized display 42 of the edge portion 41 of the right plate 1c, the second color candidate 43b (light blue) which is a color far from the color having many red components is set as the superimposed color.

[0115] For example, it is assumed that the first color candidate 43a is set to white, and the second color candidate 43b is set to black. For example, if the edge portion 41 of the left plate 1b is a dark color, the first color candidate 43a having a high brightness is set as the superimposed color. If the edge portion 41 of the right plate 1c has a bright color, the second color candidate 43b having a low brightness is set as the superimposed color.

[0116] Note that the above description is merely an example, and the edge color of each edge portion 41 is determined not only by the color or brightness of the subject 1. In any case, the edge color is calculated using color information in the original image, and a color candidate other than the edge color is appropriately selected.

[0117] As described above, in the present embodiment, the superimposed color is controlled so that the difference between the edge color represented by the color information about the edge portion 41 and the superimposed color becomes large. Therefore, the emphasized display 42 in the peak image 40 can be made sufficiently conspicuous, and the position of the edge portion 41 can be clearly indicated. As a result, the visibility of the emphasized display 42 is improved, and the focusing operation can be sufficiently supported.

[0118] [Peaking display achieved by color overlay]

[0119] Hereinafter, a case where a chromatic color having a hue is used as a superimposed color (a plurality of color candidates 43 ) will be described.

[0120] Figure 3 5 is a schematic diagram for explaining a color tone ring. The color tone ring 50 is a circular ring in which the color tones are arranged in a circle. Figure 3 , the hue ring 50 is schematically shown using grayscale, but in reality, the hue ring 50 is a representation of a plurality of colors. The hue of each color can be calculated based on RGB values, color difference values, and the like.

[0121] Figure 3 The color of each position (angle) of the color tone ring 50 is shown. For example, in the color tone ring 50, each of purple (V), blue (B), light blue (LB), green (G), and yellow (y) is arranged at each position rotated 60° in the clockwise direction with reference to red (R) ( Figure 3 ). Hereinafter, a plane formed by arranging hues in a ring will be referred to as a hue plane.

[0122] For example, the color farthest from red R in the hue plane is light blue LB. Red R and light blue LB are complementary colors arranged at diametrically opposite positions in the hue circle 50. Similarly, green G and purple V become complementary colors, and blue B and yellow Y become complementary colors.

[0123] In this embodiment, a colorful (color color) superimposed color may be set. In this case, the plurality of color candidates include a chromatic color candidate. For example, the chromatic color candidate is set from the same hue ring 50 (in which the brightness and intensity of the color are set equal to each other). The plurality of color candidates are set to have different hues.

[0124] Hereinafter, a case where two color candidates are mainly used will be described. In other words, the plurality of candidate colors are set to include a first color candidate 43a and a second color candidate 43b having a different hue from the first color candidate 43a. In this case, the edge portion 41 is displayed using two colors in the peak image 40. This makes it possible to achieve a simple emphasis display with high visibility. In addition, by suppressing the number of colors of the emphasis display 42, the user can perform a focusing operation by focusing only on two colors, and excellent operability can be demonstrated.

[0125] In this embodiment, the first and second color candidates 43a and 43b are set to colors that are complementary to each other. That is, the first and second color candidates 43a and 43b are set to hues that are rotated 180 degrees in the hue 50.

[0126] For example, two colors that complement each other may be set by the user. Also, for example, the color selected by the user may be set as the first color candidate 43a, and the second color candidate 43b used as a complementary color may be automatically set. As a result, since the edge portion 41 is represented (emphasized display 42) using colors (complementary colors) having hues that are farthest from each other, the edge portion 41 can be made sufficiently conspicuous.

[0127] As will be described later, of the two color candidates that are complementary to each other, the color that is far from the edge color 44 in the hue plane is set as the superimposed color of the edge portion 41. That is, the superimposed color setting unit 32 controls the superimposed color so that the edge color 44 and the superimposed color are colors that are far apart in the hue plane. Figure 3 , the hue of the edge color 44 is schematically shown by a white circle.

[0128] For example, it is assumed that the first color candidate 43a is set to red R, and the second color candidate 43b is set to light blue LB. For example, when calculating the edge color 44a between red R and yellow Y close to yellow Y, the second color candidate 43b (light blue) is selected as the superimposed color, which is a color far from the edge color 44a in the hue plane. In addition, for example, when calculating the edge color 44b between blue B and purple V close to blue B, the first color candidate 43a (red) is selected as the superimposed color, which is a color far from the edge color 44b in the hue plane.

[0129] That is, in the hue circle, for edge colors 44 within a range of ±90° from the first candidate color 43a, the second candidate color 43b is set as the superimposed color, and for edge colors 44 within a range of ±90° from the second candidate color 43b, the first candidate color 43a is set as the superimposed color. As a result, since each edge portion 41 is displayed with a superimposed color away from the edge color 44, the visibility of the emphasized display 42 of the edge portion 41 can be greatly improved regardless of the edge color 44 in the original image.

[0130] Note that two color candidates 43 (first and second color candidates 43a and 43b) that do not have a complementary color relationship may be set. Even in this case, the color candidate 43 that is farther from the edge color 44 in the hue ring 50 is set as the superimposed color. Therefore, the user can perform peaking display using any two colors, and can realize emphasized display 42 with a high degree of freedom.

[0131] It is also possible to set two or more color candidates 43. For example, three color candidates 43 may be set. In this case, for example, the three candidate colors 43 are set to have the farthest angle (120°) from each other in the hue ring 50. Similarly, four color candidates 43, five color candidates 43, etc. may be appropriately set. Even if two or more color candidates 43 are set as described above, the superimposed color is selected from the plurality of candidate colors 43 so that the edge color 44 and the superimposed color are colors that are far away in the hue plane. This makes it possible to fully emphasize the visibility of the emphasized display 42 regardless of the hue of the edge portion 41.

[0132] Figure 4 is a flowchart illustrating an example of peaking display using color overlay colors. Figure 5 is an image of a hue ring 50 as an example of an original image. Figure 5 As an example, refer to the original image 25 shown in FIG. Figure 4 Describes the peaking display process using color overlay colors.

[0133] Figure 5 A captured image (original image 25) is shown in which a hue ring 50 is captured as an example of the subject 1. The periphery (background) of the hue ring 50 is black. Figure 5 The color tone ring 50 shown in FIG. Figure 3 The hue ring 50 is arranged at the same angle as described in FIG. Figure 5 In the original image 25 shown, the image capturing device 100 is focused on the entire hue ring 50 .

[0134] Figure 4The processing shown in FIG. 1 is, for example, processing performed on one original image 25. In the image capturing apparatus 100, for example, each time the original image 25 is generated at a predetermined frame rate, processing is performed. Figure 4 The processing shown is performed and each original image 25 is displayed with peaking.

[0135] First, the hue components of the two color candidates 43 (the first and second color candidates 43a and 43b) set by the user are calculated (step 101). Each hue component is a hue value H representing a hue, and can be represented, for example, as an angle on the hue circle 50. For each of the color candidates 43a and 43b, the hue value H is calculated. For example, the hue value H is calculated by the superimposed color selection unit 33.

[0136] For example, when the color candidate is RGB data, the hue value H is calculated based on the respective intensities of RGB. For example, when the color candidate is color difference data, the hue value H is calculated based on the red color difference Cr and the blue color difference Cb. In addition, the hue value H can be calculated appropriately according to the format of the color candidate data.

[0137] Hereinafter, a case where the first color candidate 43a is set to red R, and the second color candidate 43b is set to light blue LB will be described. In this case, as the hue component of the first color candidate 43a, the angle of the hue ring corresponding to the position of red R is calculated as the hue value H. In addition, as the hue component of the second color candidate 43b, the angle on the hue ring 50 corresponding to the position of light blue LB is calculated as the hue value H. Since the respective color candidates 43a and 43b are complementary colors, the hue values ​​H of the color candidates 43a and 43b are offset from each other by 180°.

[0138] When calculating the tone value H, the loop processing shown in steps 102 to 108 is performed. For example, the loop processing is sequentially performed for each pixel included in the original image 25. Alternatively, the processing may be performed in parallel for a plurality of pixels included in one column (horizontal line or vertical line).

[0139] The positions of pixels to be subjected to peaking display are acquired by the superimposition color setting unit 32 (step 102). Specifically, based on the detection result (edge ​​detection data) of the edge portion 41 detected by the peaking detection unit 31 from the original image 25, the pixel positions of the pixels corresponding to the edge portion 41 are appropriately read.

[0140] Figure 6 It is shown from Figure 5 The image shown is an image of the edge portion 41 detected from the original image 25. Figure 6, pixels (edge ​​pixels 45) corresponding to the edge portion 41 of the original image 25 including the hue ring 50 are shown in white. In the original image 25 including the hue ring 50, the outer edge portion of the hue ring 50 is detected as the edge portion 41. Therefore, Figure 6 As shown, the edge pixels 45 of the color tone ring 50 are distributed in a circular shape. In the superimposed color setting unit 32, each pixel position of each edge pixel 45 is sequentially acquired.

[0141] The superimposition color setting unit 32 acquires peripheral pixel data around the pixel position (edge ​​pixel 45) in step 102 (step 103). The peripheral pixel data is pixel data of pixels included in a predetermined pixel area around the edge pixel 45. Hereinafter, the pixels included in the predetermined pixel area will be described as reference pixels.

[0142] Figure 7 is a schematic diagram showing an example of pixel arrangement around the edge pixel 45 . Figure 7 The pixel arrangement shown is a pixel arrangement of the original image 25 around the edge pixel 45 in the center. Each pixel is represented by a square arranged adjacent to the matrix in the vertical and horizontal directions. The black pixels are the edge pixels 45. The predetermined pixel area 51 is shown using a black thick line.

[0143] The predetermined pixel area 51 includes an edge pixel 45, and peripheral pixels 46 arranged around the edge pixel 45. The edge pixel 45 and the peripheral pixel 46 will be the reference pixel 47. Figure 7 In the example shown, a 3×3 pixel area centered on the edge pixel 45 is set as the predetermined pixel area 51. In this case, eight pixels arranged around the edge pixel 45 become peripheral pixels 46.

[0144] The size, shape, position, etc. of the predetermined pixel area 51 are not limited. For example, a 5×5 or 7×7 pixel area centered on the edge pixel 45 can be set. Moreover, a 2×2 or 4×4 pixel area, as well as a 2×3 and 5×4 pixel area can be set. The edge pixel 45 can be offset from the center of the pixel area. In addition, any area including the edge pixel 45 can be set as the predetermined pixel area 51.

[0145] For example, when the pixel position of the edge pixel 45 is read, the pixel positions of other pixels (peripheral pixels 46) included in the predetermined pixel area 51 are calculated. Then, based on each pixel position, the pixel data of the reference pixel 47 (edge ​​pixel 45 and peripheral pixel 46) in the original image 25 are read respectively.

[0146] The pixel data of the reference pixel 47 includes information indicating the color of each pixel in the predetermined pixel area 51. Therefore, the pixel data of the reference pixel 47 includes color information surrounding the edge pixel 45 (edge ​​portion 41). In the color peaking display, the pixel data is used as the color information of the edge portion 41. Therefore, the color information about the edge portion 41 includes the pixel data of the reference pixel 47 included in the predetermined pixel area 51 surrounding the pixel position of the edge portion 41. In addition, the color indicated by the pixel data of the reference pixel 47 is the edge color 44.

[0147] The superimposed color setting unit 32 calculates an average tone from the peripheral pixel data (pixel data of the reference pixel 47) (step 104). That is, the average value of the tone values ​​of the predetermined pixel area 51 as the periphery of the edge portion 41 is calculated.

[0148] The process of calculating a tone value based on the pixel data of the reference pixel 47 corresponds to the process of calculating the tone of the edge color 44 represented by the pixel data of the reference pixel 47. Therefore, in the image capturing apparatus 100, the superimposed color setting unit 32 calculates the tone of the edge color 44 based on the pixel data of the reference pixel 47. This makes it possible to set the superimposed color according to the tendency of the tone around the edge portion and make the superimposed color sufficiently conspicuous.

[0149] As described above, in the present embodiment, the average hue of the reference pixels 47 is calculated as the hue of the edge color 44. For example, the hue values ​​of the reference pixels 47 (edge ​​pixels 45 and peripheral pixels 46) are calculated based on the pixel data read in step 103. The average hue is calculated by averaging these hue values.

[0150] Figure 8 It is shown Figure 6 Schematic diagram of an example of the average tone of the edge portion 41 shown in FIG. Figure 8 For the sake of clarity, a wide annular area is used to schematically illustrate the Figure 6 The area corresponding to the edge portion 41 shown is shown. The average tone calculated at each pixel position of the edge portion 41 (edge ​​pixel 45) is expressed using grayscale.

[0151] like Figure 8 As shown, the average hue calculated based on each edge pixel 45 is a hue that reflects the hue of the hue ring 50 of the original image 25. For example, for the edge pixel 45 of red R, an average hue close to red R is calculated, and for the edge pixel 45 of light blue LB, an average hue close to light blue LB is calculated. In practice, each average hue is calculated for each edge pixel 45 according to the color of the subject or the background.

[0152] The method of calculating the hue (average hue) of the edge color 44 is not limited. For example, filtering processing such as average filtering or Gaussian filtering may be performed on the original image 25, and the hue may be calculated by acquiring pixel data after applying the filtering to each edge pixel 45. For example, filtering processing is performed for each target pixel area 51. Therefore, the influence of noise and the like can be reduced, and the hue value of the edge color representing the color of each edge pixel 45 can be stably calculated.

[0153] Note that the predetermined pixel area 51 may be set according to the subject 1. For example, based on a technique such as image recognition, the type of the subject 1 and the surface state of the subject 1 are detected, and the size and the like of the pixel area 51 are set based on the detection result thereof.

[0154] For example, when an image of the subject 1 or the like having a fine color pattern on its surface is captured, the pixel area 51 is set to be small. Therefore, the influence of colors other than the edge portion 41 is reduced, and the edge color 44 can be accurately calculated. In the case where the color of the subject is uniform, the pixel area 51 is set to be large. As a result, the edge color 44 can be appropriately calculated, and the color of the periphery of the edge portion 41 is also considered in the calculation.

[0155] The superimposed color setting unit 32 determines whether the average hue of the edge pixel 45 is close to the first color candidate 43a (step 105). Whether the average hue is close to the first color candidate 43a is determined by determining whether the average hue is close to the first color candidate 43a (red R) or the second color candidate 43b (light blue LB) in the hue ring 50. Based on this determination, the superimposed color of the edge pixel 45 is set.

[0156] Fig. 9 are schematic diagrams each showing an example of peaking display with colors superimposed in color. Fig. 9 A is an example of a superimposed image 60 in which an edge portion 41 is displayed in a superimposed color. Fig. 9 B is an example of a peak image 40 obtained by synthesizing the original image 25 and the superimposed image 60 .

[0157] For example, when the average hue of the edge pixel 45 is far from the first color candidate 43a and close to the second color candidate 43b (No in step 105), the superimposed color of the edge pixel 45 is set to the first color candidate 43a (step 106).

[0158] For example, when the average hue is calculated to be outside the range of ±90° with respect to the first color candidate 43a (red R), that is, within the range of ±90° with respect to the second color candidate 43b (light blue LB), the superimposed color of the edge pixel 45 is set to the first color candidate 43a (see Figure 3 ). Therefore, if Fig. 9 As shown in A, for edge pixels 45 whose edge colors 44 (average hue) are calculated to be close to light blue LB, emphasis display 42 of red R is set.

[0159] Furthermore, for example, when the average hue of the edge pixel 45 is close to the first color candidate 43a and away from the second color candidate 43b (Yes at step 105), the superimposed color of the edge pixel 45 is set to the second color candidate 43b (step 107).

[0160] For example, when the average hue is calculated to be within the range of ±90° with respect to the first color candidate 43a (red R), that is, outside the range of ±90° with respect to the second color candidate 43b (light blue LB), the superimposed color of the edge pixel 45 is set to the second color candidate 43b (see Figure 3 ). Therefore, if Fig. 9 As shown in B, for edge pixels 45 whose edge colors 44 are calculated to be close to red R (average hue), emphasis display 42 of light blue LB is set.

[0161] As described above, when the hue value of the average hue is close to the hue value of the first color candidate 43a, the superimposed color of the edge pixel 45 is set to the second color candidate 43b having a complementary color relationship, and when the average hue value is far from the first candidate color 43a, it is set to the first candidate color 43a. That is, the superimposed color setting unit 32 sets the color candidate 43 having the hue farthest from the hue of the edge color 44 among the plurality of color candidates as the superimposed color.

[0162] Fig. 9 B schematically shows a peak image 40 in which an edge portion 41 of a hue ring 50 included in the original image 25 is emphasized. For example, at an edge portion 41 ( Fig. 9 B), the superimposed color is red R (first color candidate 43a). Therefore, the light blue portion of the hue ring 50 can be easily distinguished from the emphasized display 42 (red) of the edge portion 41. In addition, for example, in the edge portion 41 (right side) of the red R of the hue ring 50 Fig. 9 In b), the superimposed color is changed to light blue LB (second color candidate 43 b ). This makes it easy to distinguish the red portion of the hue ring 50 from the emphasized display 42 (light blue) of the edge portion 41 .

[0163] When the superimposed color of the edge pixel 45 is set to one of the first and second color candidates 43a and 43b, it is determined whether the processing has been performed on all pixels of the original image 25 (step 108). If there are still pixels for which the processing has not been performed (step 108 is no), the processing at or after step 102 is repeated. If the processing has been performed on all pixels (step 109 is yes), the processing of the original image 25 has been completed. Then, the processing of the peaking display of the next original image 25 is started.

[0164] Fig.10 is a schematic diagram showing an example of peaking display as a comparative example. Fig.10 The left and right figures of A show the superimposed image 60a and the peak image 40a when a single superimposed color (red R) is used. Fig.10 As shown in FIG. 1A , when the superimposed color is represented by only red R, the hue difference between the red-based edge portions 41 in the hue ring 50 is small, making it difficult to distinguish between the portion of the emphasized display 42 (peaking) and the portion of the subject 1 (hue ring 50 ).

[0165] Fig.10 The left and right images of B are a superimposed image 60b and a peak image 40b when a color of a hue close to the edge color 44 among red R and light blue LB is used as a superimposed color. Fig.10 In B, in the hue ring 50, a superimposed color of light blue LB is set in the blue-based edge portion 41, and a superimposed color of red R is set in the red-based edge portion 41. Therefore, in both the red-based and blue-based edge portions 41, the hue difference is reduced, and the visibility of the emphasized display 42 is reduced.

[0166] On the contrary, Fig. 9 In the peak image 40 shown in B, the superimposed color is set so that the hue difference with respect to the edge color 44 becomes large. Therefore, regardless of the hue, the visibility of the peaking display (emphasized display 42) is improved for any edge portion 41. Therefore, it is easy to determine where the focus of the imaging range is, and the focusing operation of the image capturing device 100 can be fully supported.

[0167] [Peaking display with black and white superimposed with color]

[0168] Hereinafter, a case where achromatic colors having no hue (black and white, etc.) are used as superimposed colors (plural color candidates 43) will be described.

[0169] In the present embodiment, a superimposed color of a non-color color may be set. In this case, the plurality of color candidates 43 include a color candidate 43 of a non-color color. The non-color color candidate 43 is set from colors such as white, black, and gray represented by a predetermined brightness. Each color candidate 43 is set to have a different brightness. Hereinafter, a case where black and white are used as the non-color color candidate 43 will be described. That is, the plurality of color candidates 43 include black and white.

[0170] In the peaking display using the black and white superimposed colors (color candidate 43), the color whose brightness (luminance) is far from the edge color 44 among the two color candidates (black and white) is set as the superimposed color of the edge portion 41. That is, the superimposed color is controlled so that the edge color 44 and the superimposed color are colors far from each other in the brightness direction.

[0171] The edge color 44 is a color represented by color information (pixel data, etc.) about the edge portion 41 of the original image 25 and is generally a color. Therefore, in black and white peaking display, the superimposed color is controlled based on the brightness (luminance) of the color edge color 44.

[0172] For example, if the edge color 44 is brighter than a predetermined threshold, black with low brightness is set as the superimposed color of the edge portion 41. Also, for example, if the edge color 44 is darker than a predetermined threshold, white with high brightness is set as the superimposed color of the edge portion. Therefore, since each edge portion 41 is displayed with a superimposed color having a brightness far from the edge color 44, the visibility of the emphasized display 42 of the edge portion 41 can be greatly improved.

[0173] Fig.11 is a flowchart showing an example of peaking display using black and white superimposed colors. Fig.12 An image of a stripe pattern 55 is shown, which is an example of an original image. Fig.12 As an example, refer to the original image 25 shown in FIG. Fig.11 Illustrates a peaking display process using black and white overlaid with color.

[0174] Fig.12 The captured image (original image 25) is shown in which a stripe pattern 55 is captured as an example of the subject 1. The stripe pattern 55 is a rectangular area in the form of stripes. Fig.12 Each rectangular area is filled with grayscale that gradually changes from white to black from the left. That is, the stripe pattern 55 is a grayscale gradient pattern.

[0175] First, the position of a pixel to be subjected to peaking display is acquired by the superimposed color setting unit 32 (step 201). Specifically, the pixel position of the edge pixel 45 corresponding to the edge portion 41 is read.

[0176] Fig.13 It is shown from Fig.12 The image shown is an image of the edge portion 41 detected from the original image 25. Fig.13 , pixels (edge ​​pixels 45) corresponding to edge portions 41 of the original image 25 including the stripe pattern 55 are displayed in white. From the stripe pattern 55, the boundaries of the respective rectangular regions are detected as edge portions 41. Fig.13 As shown, the edge pixels 45 of the stripe pattern 55 are distributed along lines in the vertical direction corresponding to the boundaries of the respective rectangular regions.

[0177] The pixel data of the pixel position obtained in step 201 is obtained (step 202). That is, the pixel data (color information) of the edge pixel 45 is read from the original image 25. Then, the brightness of the pixel data of the edge pixel 45 is calculated by the superimposed color setting unit 32 (step 203). Fig.11 In the illustrated process, the pixel data of the edge pixel 45 becomes the color information on the edge portion 41, and each color of the edge pixel 45 becomes the edge color 44. The brightness of the edge color 44 is calculated appropriately.

[0178] For example, if Fig.12 The original image 25 shown is a grayscale image, and the grayscale value of the pixel data is calculated as the brightness of the edge pixel 45. If the pixel data is RGB data, the brightness is calculated based on each intensity value of RGB.

[0179] Fig.14 It is shown Fig.13 FIG. 4 is a schematic diagram showing an example of brightness in the original image 25 of the edge portion 41 shown. Fig.14 As shown, for example, at the border of the white rectangular area ( Fig.14 The brightness calculated at each boundary (edge ​​portion 41) becomes darker as it approaches the right side. Fig.14 Since the background is displayed as black, although Fig.14 The brightness display of the right edge portion 41 is no longer visible, but actually a brightness close to black is calculated at the position indicated by the arrow.

[0180] exist Fig.14 In the example shown, the brightness of the pixel corresponding to the edge portion 41 (edge ​​pixel 45) is calculated as it is, but it is not limited to this. For example, the reference Figure 7In this case, the pixel data of the edge pixel 45 and the peripheral pixel 46 (reference pixel 47) contained in the pixel area 51 are acquired, and the average brightness in the pixel area 51 is calculated. That is, the brightness of the edge color 44 can be calculated based on the pixel data of the reference pixel 47. This makes it possible to set the superimposed color according to the brightness of the periphery.

[0181] The superimposed color setting unit 32 determines whether the brightness of the edge pixel 45 is close to black (step 204). This is performed by determining whether the brightness of the edge pixel 45 (brightness of the edge color 44) is close to black and white in the brightness direction.

[0182] In this embodiment, when the brightness of the edge color 44 is greater than a predetermined threshold, the superimposed color setting unit 32 sets the superimposed color to black, and when the brightness of the edge color 44 is less than a predetermined threshold, the superimposed color is set to white. Based on this determination, the superimposed color of the edge pixel 45 is set.

[0183] The predetermined threshold is set to, for example, an intermediate value between black and white. For example, when the brightness of each color is expressed in 256 levels (when the brightness of black is 1 and the brightness of white is 256), the predetermined threshold is set to 128. When the brightness of each color is expressed in 128 levels, the predetermined brightness is set to 64. The predetermined threshold is not limited thereto and can be appropriately set according to, for example, the brightness of the screen, etc.

[0184] Fig.15 are schematic diagrams each showing an example of peaking display using black and white superimposed colors. Fig.15 A and Fig.15 B is an example of the superimposed image 60 in which the edge portion 41 is displayed in black and white superimposed colors. Fig.15 C is an example of a peak image 40 obtained by synthesizing the original image 25 and the superimposed image 60 .

[0185] For example, if the brightness of the edge pixel 45 is higher than a predetermined threshold and is closer to white than black (No in step 204), the superimposed color of the edge pixel 45 is set to black (step 205). For example, at the edge pixel 45 on the left side of the stripe pattern 55, the brightness close to white is calculated (see Fig.14 ). Therefore, if Fig.15 As shown in A, at the edge pixel 45, a brightness close to white is calculated (the left edge pixel 45 of the stripe pattern 55), and a black emphasis display 42 is set.

[0186] Furthermore, for example, if the brightness of the edge pixel 45 is lower than a predetermined threshold and is close to black (Yes in step 204), the superimposed color of the edge pixel 45 is set to white (step 206). For example, at the edge pixel 45 on the right side of the stripe pattern 55, a brightness close to black is calculated (see Fig.14 ). Therefore, if Fig.15 As shown in B, at the edge pixel 45, a brightness close to black is calculated (the left edge pixel 45 of the stripe pattern 55), and a white emphasis display 42 is set.

[0187] Both edge pixels 45 close to white and edge pixels 45 close to black are detected in the central boundary portion of the stripe pattern 55. In this case, even at the same boundary, for edge pixels 45 brighter than a predetermined threshold, a superimposed color of black is set, and for dark edge pixels 45, a superimposed color of white is set.

[0188] As described above, if its brightness is close to black, the superimposed color of the edge pixel 45 is set to white, and conversely, if the brightness is close to white, the superimposed color of the edge pixel 45 is set to black. That is, the superimposed color setting unit 32 sets the color candidate 43 whose brightness is farthest from the brightness of the edge color 44 among the plurality of color candidates 43 (black and white) as the superimposed color.

[0189] Fig.15 C schematically shows a peak image 40 in which an edge portion 41 of a stripe pattern 55 included in the original image 25 is emphasized. For example, in the stripe pattern 55, the superimposed color is black at the right boundary (edge ​​portion 41) close to white. At the right boundary (edge ​​portion 41) close to black, the superimposed color is white. Therefore, as Fig.15 As shown in C, the visibility of the peaking display can be substantially improved from the high brightness portion to the low brightness portion.

[0190] When the superimposed color of the edge pixel 45 is set to black or white, it is determined whether the processing has been performed on all pixels of the original image 25 (step 208). If there are still pixels that have not been processed (no in step 208), the processing of step 201 and thereafter is repeated. When the processing has been performed on all pixels (yes in step 208), the processing on the original image 25 is completed. Fig.15 A black and white superimposed pattern is determined for each pixel of the peaking display shown in C. Then, the process of peaking display of the next original image 25 is started.

[0191] Fig.16 A peak image 40c is shown as a comparative example. Fig.16In the peak image 40c shown, a single superimposed color (white) is used. If the superimposed color is only white, the emphasized display 42 can be visually recognized at the edge portion 41 on the right side with low brightness, but it is difficult to distinguish the emphasized display 42 from the subject (strip pattern 55) at the edge portion 41 with high brightness.

[0192] On the contrary, Fig.15 In the peak image 40 shown in C, the superimposed color is set so that the brightness difference with the edge color 44 increases. Therefore, regardless of the brightness of the edge portion 41, the visibility of the peaking display (emphasized display 42) is improved for any edge portion 41. Therefore, the focused position of the imaging range can be easily determined, and the focusing operation of the image capturing device 100 can be sufficiently supported.

[0193] As described above, in the image capturing apparatus 100 according to the present embodiment, a captured image (raw image 25) of the subject 1 is generated, and the edge portion 41 included in the raw image 25 is detected. In addition, for each detected edge portion 41, the color of the emphasis display 42 for emphasizing the edge portion 41 is controlled using the color information about the edge portion 41 in the raw image 25. This makes it possible to improve the visibility of the emphasis display 42 of the edge portion 41.

[0194] As a method of emphasizing the edge of the subject, there are considered a method of darkening the edge color, a method of superimposing a single color, etc. In this case, depending on the color and background around the edge, a part of the emphasized edge may be difficult to recognize.

[0195] In the present embodiment, the color (superimposed color) of the emphasized display 42 for emphasizing the edge portion 41 is set to a color that is far away from the color (edge ​​color 44) of the edge portion 41 in the original image 25. Therefore, the superimposed color can be made conspicuous regardless of the color of the edge portion 41 itself or the color of the periphery of the edge portion 41 in the original image 25. Therefore, the visibility of the emphasized display 42 can be improved.

[0196] In the color peaking display, for example, two complementary colors (first and second color candidates 43a and 43b) are set as superimposed colors. The superimposed color is automatically selected as a color that is visually easy to recognize by the user according to the hue of the edge portion 41 (hue of the edge color 44). As described above, since the peaking display is performed with a color having a large hue difference according to the color around the peaking position, the user can easily recognize the peaking portion with respect to the color of any hue.

[0197] In the black and white peaking display, for example, the black and white superimposed color is automatically selected as a color that is easy for the user to visually recognize based on the brightness of the edge portion 41 (brightness of the edge color 44). This makes it possible to improve the visibility of the peaking display regardless of the brightness of the edge portion 41.

[0198] In addition, the superimposed color for the emphasized display 42 changes in accordance with the change in the color of the peak position (edge ​​portion 41). This makes it possible to maintain the visibility of the emphasized display 42 even when, for example, the composition, subject, etc. of the original image 25 are constantly changing. As a result, the focus position can always be displayed in a color that is easy to visually recognize, and high usability can be expressed.

[0199] In addition, peaking display can be performed by using the superimposed color (plural color candidates 43) selected by the user. In the image capturing apparatus 100, the superimposed color corresponding to the color of the edge portion 41 is automatically set using the color candidate 43 selected by the user. This makes it possible to easily recognize the emphasized display 42 of the edge portion 41.

[0200] In the image capturing apparatus 100, the peak image 40 is generated by modifying only the edge portion 41 of the original image 25. That is, the peaking display is limited to the edge portion 41, and the remaining portion is the same as the normal image. As a result, an imaging state can be achieved in which the user does not feel uncomfortable without reducing the visibility of the peaking display. This makes it possible to perform a natural focusing operation and exhibit excellent operability.

[0201] <Other embodiments>

[0202] The present technology is not limited to the above-described embodiments, and various other embodiments can be implemented.

[0203] In the above description, a method of setting the display (emphasis display 42) of the edge portion 41 of the peak image 40 according to the edge color 44 is described. For example, the emphasis display 42 may be controlled so that the display of the edge portion 41 changes over time. That is, the color of the emphasis display 42 may be controlled so that the display of the edge portion 41 of the peak image 40 changes dynamically.

[0204] For example, the color of the emphasized display 42 may be set to switch at a constant time. For example, when two color candidates 43 are used, the color candidate 43 as each emphasized display 42 is switched to the other color candidate 43 at a preset time interval. As a result, since the color of the emphasized display 42 is constantly switched and displayed, the edge portion 41 can be easily detected.

[0205] For example, the emphasized display 42 may be displayed to flash at a constant time interval. In this case, the state in which the emphasized display 42 is displayed and the state in which the emphasized display 42 is not displayed are repeated alternately. In this way, by flashing the emphasized display 42 of the edge portion 41, that is, by providing motion to the emphasized display 42, the edge portion 41 can be easily detected. Alternatively, any method of dynamically changing the display of the edge portion 41 may be used.

[0206] Fig.17 The following are schematic diagrams for explaining application examples of peaking display. Fig.17 A schematically shows a state of surgery using an image capturing device. Fig.17 B and Fig.17 C are schematic diagrams respectively showing examples of peaking display applied to an image obtained by photographing a surgical site. Fig.17 In FIG. 17B and FIG. 17C , the emphasis display 42 is schematically shown using a dotted line.

[0207] For example, when performing double-chamber surgery, laparotomy, etc., an image capturing device (endoscopic camera, etc.) for imaging an affected part in the body is used. Also, for example, when performing craniotomy, etc., an image capturing device that magnifies and captures an image of the affected part is used. The present technology can be suitably applied to such an image capturing device.

[0208] Fig.17 A shows a state in which the endoscopist 2 who performs manual focusing of the endoscopic camera 200 during surgery performs the focusing operation of the endoscopic camera 200 while watching a monitor 215b different from the monitor 215a watched by the doctor 3 who performs the surgery. In this case, the peaking display using the present technology is performed only on the image displayed on the monitor 215b seen by the endoscopist 2, and the peaking display is not performed on the image (monitor 215a) watched by the doctor 3 who performs the surgery. As a result, the operation of the endoscopist 2 to focus the endoscopic camera 200 on the affected part can be facilitated without affecting the doctor 3 who performs the surgery.

[0209] Fig.17 B is an example of a peak image 240 obtained by applying peaking display to an image (intraoperative image) captured by the endoscope camera 200 . Fig.17 B shows a state where a part of the organ 5a is clamped by the forceps 4 operated by the doctor 3. The organ 5a clamped by the forceps 4 is connected to the organ 5b covered by the fat 6. On the organ 5b covered by the fat 6, another organ 5c exists.

[0210] The superimposed color for peaking display can be set according to the organs 5a to 5c (subject 1). That is, a plurality of color candidates 43 as candidates for superimposed colors are set according to the subject 1. Hereinafter, a method of setting the color candidate 43 (superimposed color) according to the subject 1 will be described in detail.

[0211] For example, the organ 5, fat 6, etc. included in the intraoperative image are detected using image recognition processing or the like. For example, when the periphery of the fat 6 is the surgical target, the fat 6 is set as the target subject. In this case, a color (blue) that is a complementary color to the color (yellow) of the fat 6 is set as one color candidate 43. As a result, when the focus is on the periphery of the fat 6, the edge portion 41 is displayed by the blue emphasis display 42a, and the focus position can be visually recognized.

[0212] In addition, for example, the forceps 4 in the intraoperative image can be detected by using image recognition processing or the like. In this case, the organ (organ 5a) near the forceps 4 is set as the target subject. In this case, a color that becomes a complementary color of the organ 5a as the target subject is set as the color candidate 43. As a result, the outline of the organ 5a to be operated on can be displayed by setting the emphasis display 42b of the complementary color of the color of the organ 5a. Therefore, the focusing operation can be easily performed on the target subject.

[0213] Fig.17 C is an example of a peak image 240 obtained by applying peaking display to an intraoperative image imaging a tumor portion. Fig.17 C schematically shows a portion of a brain 7 and a tumor 8 exposed by, for example, a craniotomy.

[0214] For example, the tumor 8 is detected using image recognition processing or the like. For example, the tumor 8 can be detected using a color difference from the normal brain 7. The color of the emphasized display 42 (color candidate 43) is set to be complementary to the color of the tumor 8. This makes it possible to emphasize the detected edge portion 41 of the tumor 8 with a color that is easily distinguishable. In addition, the method of setting the color candidate 43 according to the subject 1 is not limited.

[0215] <Application Examples>

[0216] The technology according to the present disclosure is applicable to various products. For example, the technology according to the present disclosure can be applied to an endoscopic surgery system.

[0217] Fig.18 is a diagram showing an example of a schematic configuration of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied. Fig.18The figure shows a state where a surgeon (doctor) 5067 using an endoscopic surgery system 5000 performs surgery on a patient 5071 on a bed 5069. As shown in the figure, the endoscopic surgery system 5000 includes an endoscope 5001, other surgical tools 5017, a support arm device 5027 for supporting the endoscope 5001, and a cart 5037 on which various devices for endoscopic surgery are installed.

[0218] In endoscopic surgery, instead of cutting the abdominal wall, multiple cylindrical opening tools called trockers 5025a to 5025d are inserted into the abdominal wall. Then, the lens barrel 5003 of the endoscope 5001 and other surgical tools 5017 are inserted into the body cavity of the patient 5071 from the trockers 5025a to 5025d. In the example shown, as other surgical tools 5017, a pneumoperitoneum tube 5019, an energy treatment tool 5021 and forceps 5023 are inserted into the body cavity of the patient 5071. The energy treatment tool 5021 is a treatment tool for tissue cutting and stripping, blood vessel sealing, etc. by high-frequency current or ultrasonic vibration. However, the surgical tool 5017 shown in the figure is only an example, and various surgical tools commonly used in endoscopic surgery such as forceps and retractors can be used as the surgical tool 5017.

[0219] An image of a surgical portion in a body cavity of a patient 5071 imaged by an endoscope 5001 is displayed on a display device 5041. A surgeon 5067 performs treatment such as excision of an affected portion using an energy treatment tool 5021 and forceps 5023 while viewing the image of the surgical portion displayed on the display device 5041 in real time. Although not shown, the pneumoperitoneum tube 5019, the energy treatment tool 5021, and the forceps 5023 are supported by the surgeon 5067, an assistant, or the like during surgery.

[0220] (Support arm equipment)

[0221] The support arm device 5027 includes an arm portion 5031 extending from a base 5029. In the example shown, the arm portion 5031 includes joint portions 5033a, 5033b, and 5033c, and links 5035a and 5035b, and is driven under the control of the arm control device 5045. The endoscope 5001 is supported by the arm portion 5031, and its position and posture are controlled. Therefore, fixation at a stable position of the endoscope 5001 can be achieved.

[0222] (Endoscope)

[0223] The endoscope 5001 includes a lens barrel 5003, in which a region of a predetermined length from the distal end is inserted into a body cavity of a patient 5071, and a camera 5005 connected to the base end of the lens barrel 5003. In the illustrated example, the endoscope 5001 configured as a so-called rigid scope having a rigid lens barrel 5003 is shown, but the endoscope 5001 may be configured as a so-called flexible scope having a flexible lens barrel 5003.

[0224] At the distal end of the lens barrel 5003, an opening for mounting an objective lens is provided. A light source device 5043 is connected to the endoscope 5001, and light generated by the light source device 5043 is guided to the distal end of the lens barrel by a light guide extending inside the lens barrel 5003, and irradiates the observation object in the body cavity of the patient 5071 through the objective lens. Note that the endoscope 5001 may be a direct-viewing mirror, a perspective mirror, or a side mirror.

[0225] Inside the camera head 5005, an optical system and an image capturing element are provided, and reflected light (observation light) from an observation object is focused on the image capturing element through the optical system. The observation light is photoelectrically converted by the image capturing element, and an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image, is generated. The image signal is sent to a camera control unit (CCU) 5039 as raw data. Incidentally, the camera head 5005 includes a function of adjusting magnification and focal length by appropriately driving the optical system.

[0226] Note that a plurality of image capturing elements may be provided in the camera 5005 in order to cope with, for example, stereoscopic vision (3D display), etc. In this case, a plurality of relay optical systems are provided inside the lens barrel 5003 in order to guide observation light to each of the plurality of image capturing elements.

[0227] (Various equipment mounted on carts)

[0228] The CCU 5039 includes a CPU (central processing unit), a GPU (graphics processing unit), etc., and generally controls the operation of the endoscope 5001 and the display device 5041. Specifically, the CCU 5039 performs various image processing, such as development processing (demosaic processing), on the image signal received from the camera 5005, so as to display an image based on the image signal. The CCU 5039 provides the display device 5041 with the image signal to which the image processing is applied. The CCU 5039 also sends a control signal to the camera 5005 and controls its driving. Such a control signal may include information about shooting conditions such as magnification and focal length.

[0229] The display device 5041 displays an image based on an image signal that has been image-processed by the CCU 5039 in accordance with the control of the CCU 5039. When the endoscope 5001 is compatible with high-resolution imaging such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels) and / or compatible with 3D display, a display device 5041 capable of high-resolution display and / or capable of 3D display corresponding thereto can be used. In the case where the display device is compatible with high-resolution imaging such as 4K or 8K, a more immersive feeling can be obtained by using a display device of 55 inches or more as the display device 5041. In addition, a plurality of display devices 5041 having different resolutions and sizes can be provided according to the application.

[0230] The light source device 5043 is constituted by, for example, a light source such as an LED (Light Emitting Diode), and supplies irradiation light to the endoscope 5001 when imaging a surgical portion.

[0231] The arm control device 5045 is constituted by, for example, a processor such as a CPU, and operates according to a predetermined program to control the driving of the arm portion 5031 of the supporting arm device 5027 according to a predetermined control method.

[0232] The input device 5047 is an input interface of the endoscopic surgery system 5000. The user can input various types of information and instructions to the endoscopic surgery system 5000 via the input device 5047. For example, the user inputs various types of information about the surgery such as the patient's physical information and information about the surgical process through the input device 5047. In addition, for example, the user inputs an instruction to drive the arm portion 5031, an instruction to change the shooting conditions of the endoscope 5001 (type, magnification, focal length, etc.), an instruction to drive the energy treatment tool 5021, etc. through the input device 5047.

[0233] The type of the input device 5047 is not limited, and the input device 5047 may be any of various known input devices. As the input device 5047, for example, a mouse, a keyboard, a touch pad, a switch, a foot switch 5057, a joystick, etc. may be used. When a touch panel is used as the input device 5047, the touch panel may be provided on the display surface of the display device 5041.

[0234] Alternatively, the input device 5047 is a device worn by the user, such as a wearable device or a glasses-type HMD (head mounted display), and various inputs are performed according to the user's gestures or sight lines detected by the device. The input device 5047 also includes a camera capable of detecting the movement of the user, and various inputs are performed according to the user's gestures or sight lines detected from the image captured by the camera. In addition, the input device 5047 includes a microphone capable of picking up the user's voice, and various inputs are performed by means of voice through the microphone. In this way, by configuring the input device 5047 to be able to input various types of information contactlessly, a user (e.g., surgeon 5067) belonging to a particularly clean area can operate a device belonging to a dirty area contactlessly. In addition, the user can operate the device without releasing his hand from the device, thereby improving the convenience of the user.

[0235] The treatment device control device 5049 controls the driving of the energy treatment device 5021 for cauterization of tissues, incision or sealing of blood vessels, etc. The pneumoperitoneum device 5051 delivers gas into the body cavity of the patient 5071 through the pneumoperitoneum tube 5019 to expand the body cavity to ensure the field of view of the endoscope 5001 and to ensure the working space of the surgeon. The recorder 5053 is a device capable of recording various types of information about the operation. The printer 5055 is a device capable of printing various types of information about the operation in various formats, such as text, images, and graphics.

[0236] Hereinafter, a configuration of an endoscopic surgery system 5000 , and in particular, features of the endoscopic surgery system 5000 , will be described in more detail.

[0237] (Support arm equipment)

[0238] The support arm device 5027 includes a base 5029 serving as a base and an arm portion 5031 extending from the base 5029. In the example shown, the arm portion 5031 includes a plurality of joint portions 5033a, 5033b, 5033c and a plurality of links 5035a, 5035b connected by the joint portions 5033b, but in Fig.18 In the figure, for the sake of simplicity, the structure of the arm portion 5031 is shown in a simplified manner. In fact, the shape, number and arrangement of the joint portions 5033a to 5033c and the connecting rods 5035a and 5035b, the direction of the rotation axis of the joint portions 5033a to 5033c, etc. can be appropriately set so that the arm portion 5031 has the required degree of freedom. For example, the arm portion 5031 can be appropriately configured to have six or more degrees of freedom. Therefore, since the endoscope 5001 can move freely within the movable range of the arm portion 5031, the lens barrel 5003 of the endoscope 5001 can be inserted into the body cavity of the patient 5071 from the desired direction.

[0239] The joint parts 5033a to 5033c are provided with actuators, and the joint parts 5033a to 5033c are configured to rotate around a predetermined rotation axis by driving the actuators. The driving of the actuators is controlled by the arm control device 5045, thereby controlling the rotation angles of the joint parts 5033a to 5033c and controlling the driving of the arm part 5031. Therefore, the position and posture of the endoscope 5001 can be controlled. In this case, the arm controller 5045 can control the driving of the arm part 5031 by various known control methods including force control or position control.

[0240] For example, the surgeon 5067 can appropriately perform operation input through the input device 5047 (including the foot switch 5057) to appropriately control the drive of the arm part 5031 through the arm control device 5045 according to the operation input, thereby controlling the position and posture of the endoscope 5001. Through this control, after the endoscope 5001 at the far end of the arm part 5031 is moved from an arbitrary position to an arbitrary position, the endoscope 5001 can be fixedly supported at the position after the movement. The arm part 5031 can be operated in a so-called master-slave manner. In this case, the user can remotely operate the arm part 5031 through the input device 5047 installed at a location away from the operating room.

[0241] When force control is applied, the arm control device 5045 can perform so-called power-assisted control in which the actuator of each joint portion 5033a to 5033c is driven to receive an external force from the user and smoothly move the arm portion 5031 in accordance with the external force. Therefore, when the user moves the arm portion 5031 while directly touching the arm portion 5031, the arm portion 5031 can be moved with a relatively light force. Therefore, the endoscope 5001 can be moved more intuitively with a simpler operation, thereby improving the convenience of the user.

[0242] Here, generally speaking, in endoscopic surgery, the endoscope 5001 is supported by a physician called an endoscopist. In contrast, by using the support arm device 5027, the position of the endoscope 5001 can be more reliably fixed without manual intervention, and therefore, the image of the surgical part can be stably obtained and the surgery can be smoothly performed.

[0243] The arm control device 5045 is not necessarily provided in the cart 5037. The arm controller 5045 may not necessarily be one device. For example, the arm control device 5045 may be provided in each of the joint portions 5033a to 5033c of the arm portion 5031 of the supporting arm device 5027, and the drive control of the arm portion 5031 may be realized by a plurality of arm control devices 5045 cooperating with each other.

[0244] (Light source equipment)

[0245] When imaging the surgical part, the light source device 5043 provides irradiation light to the endoscope 5001. The light source device 5043 is composed of a white light source including, for example, an LED, a laser light source, or a combination thereof. At this time, when the white light source is configured in a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, and the white balance of the image captured in the light source device 5043 can be adjusted. In addition, in this case, by irradiating the laser beam from each RGB laser light source onto the observed object in a time-division manner, and by controlling the drive of the image capturing element of the camera 5005 in synchronization with the irradiation timing, it is also possible to capture images corresponding to each RGB in a time-division manner. According to this method, a color image can be obtained without providing a color filter in the capturing element.

[0246] In addition, the drive of the light source device 5043 can be controlled to change the intensity of the output light at a predetermined time interval. By controlling the drive of the image capturing element of the camera 5005 in synchronization with the timing of the light intensity change, images are acquired in a time-sharing manner, and synthesized, an image with a high dynamic range can be generated without the so-called white jump component and black compression component.

[0247] In addition, the light source device 5043 can be configured to be able to provide light of a predetermined wavelength band corresponding to special light observation. In special light observation, so-called narrow-band light observation (narrow-band imaging) is performed, in which, for example, by irradiating a narrow-band light, predetermined tissues, such as blood vessels in the surface layer of the mucosa, are photographed with high contrast using the wavelength correlation of light absorbed by body tissues compared to the irradiation light (i.e., white light) in normal observation. Alternatively, in special light observation, fluorescence observation can be performed to obtain an image by irradiating fluorescence generated by excitation light. In fluorescence observation, for example, fluorescence observation can be performed by irradiating excitation light to body tissues and observing fluorescence from body tissues (autofluorescence observation), or by locally injecting reagents such as indocyanine green (ICG) into body tissues and irradiating excitation light corresponding to the fluorescence wavelength of the reagents so as to obtain a fluorescence image. The light source device 5043 can be configured to be able to provide narrow-band light and / or excitation light corresponding to special light observation.

[0248] (Camera and CCU)

[0249] refer to Fig.19 , the functions of the camera 5005 and CCU5039 of the endoscope 5001 will be described in more detail. Fig.19 It is shown Fig.18 A block diagram of an example of the functional configuration of the camera 5005 and the CCU 5039 shown in FIG.

[0250] refer to Fig.19The camera 5005 has a lens unit 5007, an image capturing unit 5009, a driving unit 5011, a communication unit 5013, and a camera control unit 5015 according to its functions. The CCU 5039 has a communication unit 5059, an image processing unit 5061, and a control unit 5063 according to its functions. The camera 5005 and the CCU 5039 are connected for bidirectional communication via a transmission cable 5065.

[0251] First, the functional configuration of the camera head 5005 will be described. The lens unit 5007 is an optical system disposed in the connection portion between the lens barrels 5003. The observation light obtained from the far end of the lens barrel 5003 is guided to the camera head 5005 and enters the lens unit 5007. The lens unit 5007 is formed by combining a plurality of lenses including a zoom lens and a focusing lens. The optical characteristics of the lens unit 5007 are adjusted so that the observation light is gathered on the light receiving surface of the light receiving element of the image capturing element of the image capturing unit 5009. In addition, the zoom lens and the focusing lens are configured to be movable in position on the optical axis to adjust the magnification and focus of the captured image.

[0252] The image capturing unit 5009 includes an image capturing element and is arranged at a later stage of the lens unit 5007. The observation light passing through the lens unit 5007 is collected on a light receiving surface of the image capturing element, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the image capturing unit 5009 is provided to the communication unit 5013.

[0253] As the image capturing element of the image capturing unit 5009, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor capable of capturing a color image with a Bayer array is used. As the image capturing element, for example, a device capable of capturing a high-resolution image of 4K or higher can be used. By obtaining an image of the surgical part at a high resolution, the surgeon 5067 can grasp the state of the surgical part in more detail and can perform the operation smoothly.

[0254] In addition, the image capturing element of the image capturing unit 5009 is configured to have a pair of image capturing elements for acquiring respective image signals for the right eye and the left eye corresponding to the 3D display. By displaying 3D, the surgeon 5067 can more accurately grasp the depth of the living tissue in the surgeon's part. By the way, when the image capturing unit 5009 is composed of a multi-plate type corresponding to each image capturing element, a plurality of systems of the lens unit 5007 are also provided.

[0255] In addition, the image capturing unit 5009 may not necessarily be disposed in the camera head 5005. For example, the image capturing unit 5009 may be disposed in the lens barrel 5003 immediately after the objective lens.

[0256] The driving unit 5011 is constituted by an actuator, and moves the zoom lens and the focus lens of the lens unit 5007 by a predetermined distance along the optical axis through the control of the camera control unit 5015. As a result, the magnification and focus of the image captured by the image capturing unit 5009 can be appropriately adjusted.

[0257] The communication unit 5013 is configured by a communication device for sending and receiving various types of data relative to the CCU 5039. The communication unit 5013 sends the image signal obtained from the image capturing unit 5009 as raw data to the CCU 5039 via the transmission cable 5065. At this time, it is preferable to send the image signal by optical communication so as to display the captured image of the surgical part with low latency. This is because, during the operation, the surgeon 5067 performs the operation while observing the state of the affected part through the captured image, so that for a safer and more reliable operation, it is necessary to display the moving image of the surgical part as real time as possible. When performing optical communication, the communication unit 5013 is provided with a photoelectric conversion module for converting an electrical signal into an optical signal. After the image signal is converted into an optical signal by the photoelectric conversion module, the image signal is transmitted to the CCU 5039 through the transmission cable 5065.

[0258] In addition, the communication unit 5013 receives a control signal from the CCU 5039 for controlling the drive of the camera 5005. The control signal includes, for example, information for specifying a frame rate for capturing an image, information for specifying an exposure value when capturing an image, and / or information for specifying a magnification and a focus for capturing an image. The communication unit 5013 provides the received control signal to the camera control unit 5015. Incidentally, the control signal from the CCU 5039 may also be sent via optical communication. In this case, the communication unit 5013 is provided with a photoelectric conversion module for converting an optical signal into an electrical signal, and the control signal is converted into an electrical signal by the photoelectric conversion module and provided to the camera control unit 5015.

[0259] Incidentally, based on the acquired image signal, the above-mentioned shooting conditions such as frame rate, exposure value, magnification and focus are automatically set by the control unit 5063 of the CCU 5039. That is, the endoscope 5001 is equipped with a so-called AE (automatic exposure) function, AF (automatic focus) function and AWB (automatic white balance) function.

[0260] The camera control unit 5015 controls the driving of the camera 5005 based on the control signal from the CCU 5039 received via the communication unit 5013. For example, the camera control unit 5015 controls the driving of the image capturing element of the image capturing unit 5009 based on the information for specifying the frame rate of the captured image and / or the information for specifying the exposure value when the image is captured. In addition, for example, the camera control unit 5015 appropriately moves the zoom lens and the focus lens of the lens unit 5007 through the drive unit 5011 based on the information for specifying the magnification and focus of the captured image. The camera control unit 5015 may also include a function of storing information for identifying the lens barrel 5003 and the camera 5005.

[0261] Note that the camera head 5005 can be made resistant to high-pressure sterilization by arranging the configuration of the lens unit 5007, the image capturing unit 5009, and the like in an airtight sealed structure having high airtightness and waterproofness.

[0262] Next, the functional configuration of the CCU 5039 will be described. The communication unit 5059 is configured by a communication device for sending and receiving various types of information relative to the camera 5005. The communication unit 5059 receives an image signal sent from the camera 5005 via a transmission cable 5065. At this time, as described above, the image signal can be appropriately sent through optical communication. In this case, corresponding to the optical communication, the communication unit 5059 is provided with a photoelectric conversion module for converting the optical signal into an electrical signal. The communication unit 5059 provides the image signal converted into the electrical signal to the image processing unit 5061.

[0263] In addition, the communication unit 5059 transmits a control signal for controlling the driving of the camera 5005 to the camera 5005. The control signal may also be transmitted through optical communication.

[0264] The image processing unit 5061 performs various image processing on the image signal as the raw data sent from the camera 5005. The image processing includes, for example, various known signal processing such as development processing, image quality enhancement processing (band enhancement processing, super-resolution processing, NR (noise reduction) processing and / or camera shake correction processing) and / or magnification processing (electronic zoom processing). In addition, the image processing unit 5061 performs detection processing on the image signal in order to perform AE, AF and AWB.

[0265] The image processing unit 5061 is configured by a processor such as a CPU and a GPU, and when the processor operates according to a predetermined program, the above-mentioned image processing or detection processing can be performed. When the image processing unit 5061 is constructed by a plurality of GPUs, the image processing unit 5061 appropriately divides the information about the image signal, and the image processing is performed in parallel by the plurality of GPUs.

[0266] The control unit 5063 performs various controls related to the image of the surgical part captured by the endoscope 5001 and the display of the captured image. For example, the control unit 5063 generates a control signal for controlling the drive of the camera head 5005. At this time, when the user inputs the shooting conditions, the control unit 5063 generates a control signal based on the user's input. Alternatively, when the endoscope 5001 has an AE function, an AF function, and an AWB function, the control unit 5063 appropriately calculates the optimal exposure value, focal length, and white balance according to the result of the detection processing of the image processing unit 5061, and generates a control signal.

[0267] In addition, the control unit 5063 displays the image of the surgical part on the display device 5041 based on the image signal processed by the image processing unit 5061. At this time, the control unit 5063 uses various image recognition technologies to identify various objects in the image of the surgical part. For example, by detecting the shape, color, etc. of each edge of the object contained in the image of the surgical part, the control unit 5063 can identify surgical tools such as forceps, specific biological parts, bleeding, fog when using the energy treatment tool 5021, etc. When the image of the surgical part is displayed on the display device 5041, the control unit 5063 uses the recognition result to superimpose and display various types of surgical support information about the image of the surgical part. By superimposing and displaying the operation support information and presenting it to the surgeon 5067, surgery can be performed more safely and reliably.

[0268] The transmission cable 5065 for connecting the camera 5005 and the CCU 5039 is an electric signal cable corresponding to communication through electric signals, an optical fiber corresponding to optical communication, or a composite cable thereof.

[0269] Here, in the example shown, communication is performed wirelessly using the transmission cable 5065, but communication between the camera 5005 and the CCU 5039 may be performed wirelessly. When communication between the two is performed wirelessly, there is no need to place the transmission cable 5065 in the operating room, thereby eliminating the situation where the movement of medical staff in the operating room is hindered by the transmission cable 5065.

[0270] An example of an endoscopic surgery system 5000 to which the technology according to the present disclosure can be applied is described above. Note that although the endoscopic surgery system 5000 is described here as an example, the system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the technology of the present disclosure can be applied to test a flexible endoscope system or a microscope surgery system.

[0271] The technology according to the present disclosure is applicable to the endoscope 5001, CCU 5039, display device 5041, etc. in the above configuration. For example, the image sensor mounted on the endoscope 5001 can be used as a reference. Figure 1 The image processing unit 5061 of the CCU 5039 may be used as a reference Figure 1 The video processing unit 13 is described. The display device 5041 is a reference Figure 1 By applying the technology according to the present disclosure to the endoscope 5001, the CCU 5039, the display device 5041, etc., it is possible to improve the visibility of the peaking display displayed on the display device 5041. As a result, focus adjustment during surgery can be easily achieved, and safe and highly reliable surgery can be performed.

[0272] For example, the technology according to the present disclosure may be applied to an operating room system.

[0273] Fig. 20 Schematically shows the overall configuration of an operating room system 5100 to which the technology according to the present disclosure can be applied. Fig. 20 , the surgical room system 5100 is configured such that a group of equipment installed in an operating room is connected to each other in a coordinated manner via an audio-visual controller (AV controller) 5107 and an operating room controller 5109.

[0274] Various equipment can be installed in the operating room. Fig. 20 As examples, various equipment groups 5101 for endoscopic surgery, a ceiling camera 5187 installed on the ceiling of the operating room for capturing images of the surgeon's hands, a surgical field camera 5189 installed on the ceiling of the operating room for capturing images of the entire operating room, multiple display devices 5103A to 5103D, a recorder 5105, a bed 5183 and lighting equipment 5191 are shown.

[0275] Among these devices, device group 5101 belongs to an endoscopic surgery system 5113 described later, and includes an endoscope, a display device for displaying an image taken by the endoscope, and the like. Each device belonging to the endoscopic surgery system 5113 is also referred to as a medical device. On the other hand, display devices 5103A to 5103D, a recorder 5105, a bed 5183, and a lighting device 5191 are, for example, devices provided separately from the endoscopic surgery system 5113 in an operating room. Each device that does not belong to these endoscopic surgery systems 5113 is also referred to as a non-medical device. The audio-visual controller 5107 and / or the operating room controller 5109 cooperatively control the operations of the medical devices and the non-medical devices.

[0276] The audio-visual controller 5107 centrally controls the processing related to the image display in the medical equipment and non-medical equipment. Specifically, among the equipment included in the operating room system 5100, the equipment group 5101, the ceiling camera 5187 and the operation site camera 5189 can be a device (hereinafter also referred to as a source device) having a function of transmitting information to be displayed during the operation (hereinafter also referred to as display information). Each of the display devices 5103A to 5103D can be a device to which display information is output (hereinafter also referred to as an output device). The recorder 5105 can be a device corresponding to both the source device and the output device. The audio-visual controller 5107 has the function of controlling the operation of the source device and the output device, obtaining display information from the source device, and sending the display information to the output device to display or record the display information. Note that the display information is various images taken during the operation, various information about the operation (for example, the patient's physical information, past examination results, information about the surgical operation, etc.), etc.

[0277] Specifically, as display information, the audio-visual controller 5107 may transmit information about an image of a surgical portion in a patient's body cavity captured by an endoscope in the device group 5101. Furthermore, information about an image of a surgeon's hand captured by the ceiling camera 5187 may be transmitted from the ceiling camera 5187 as display information. In addition, information about an image indicating the entire state of the operating room captured by the operating scene camera 5189 may be transmitted from the operating scene camera 5189 as display information. When other devices having a capturing function exist in the operating room system 5100, the audio-visual controller 5107 may acquire information about images captured by the other devices as display information from the other devices.

[0278] Alternatively, for example, the audio-visual controller 5107 records information about these images captured in the past in the recorder 5105. The audio-visual controller 5107 may acquire information about images captured in the past as display information from the recorder 5105. The recorder 5105 may also record various types of information related to surgery in advance.

[0279] The audio-visual controller 5107 displays the acquired display information (i.e., images captured during surgery and various types of information about surgery) on at least one of the display devices 5103A to 5103D as output devices. In the example shown in the figure, the display device 5103A is a display device suspended from the ceiling of the operating room, the display device 5103B is a display device mounted on the wall surface of the operating room, the display device 5103C is a display device mounted on the table in the operating room, and the display device 5103D is a mobile device having a display function (e.g., a tablet computer (personal computer)).

[0280] Although not in Fig. 20 As shown in FIG. 5 , the operating room system 5100 may include devices outside the operating room. The devices outside the operating room may be, for example, a server connected to a network inside and outside the hospital, a PC used by medical staff, a projector installed in a hospital conference room, etc. When the external device is outside the hospital, the audio-visual controller 5107 may also display the display information on display devices in other hospitals through a video conferencing system for telemedicine, etc.

[0281] The operating room control device 5109 centrally controls the processing except the processing related to the image display in the non-medical equipment. For example, the operating room controller 5109 controls the driving of the bed 5183, the ceiling camera 5187, the operation site camera 5189 and the lighting device 5191.

[0282] The operating room system 5100 is provided with a centralized operation panel 5111, and the user can provide instructions for image display to the audio-visual controller 5107 or give instructions for operating non-medical equipment to the operating room control device 5109 through the centralized operation panel 5111. The centralized operation panel 5111 is configured by a touch panel provided on the display surface of the display device.

[0283] Fig.21 51 is a diagram showing a display example of an operation screen on the centralized operation panel 5111. As an example, Fig.21 FIG. 5 shows an operation screen corresponding to a situation in which two display devices are provided as output devices in the operating room system 5100. Fig.21 The operation screen 5193 includes a source selection area 5195 , a preview area 5197 and a control area 5201 .

[0284] In source selection area 5195, source devices provided in operating room system 5100 and thumbnail screens representing display information included in the source devices are displayed in association with each other. The user can select display information to be displayed on the display device from any source device displayed in source selection area 5195.

[0285] The preview area 5197 displays a preview of the screen displayed on the two display devices (monitor 1, monitor 2) as output devices. In the example shown, four images are displayed on one display device in a picture-in-picture (PinP) manner. The four images correspond to the display information sent from the source device selected in the source selection area 5195. Among the four images, one image is displayed relatively large as a main image, and the remaining three images are displayed relatively small as sub-images. The user can exchange the main image and the sub-image by appropriately selecting the area where the four images are displayed. In addition, a status display area 5199 is provided below the area where the four images are displayed, and the status related to the operation (for example, the time elapsed from the operation, the patient's physical information, etc.) can be appropriately displayed in this area.

[0286] The control area 5201 includes a source operation area 5203 that displays GUI (Graphical User Interface) components for operating the source device and an output operation area 5205 that displays GUI components for operating the output device. In the example shown, the source operation area 5203 includes GUI components for performing various operations (panning, tilting, and zooming) on ​​the camera in the source device with a shooting function. The user can manipulate the operation of the camera in the source device by appropriately selecting these GUI components. Although not shown, in the case where the source device selected in the source selection area 5195 is a recorder (i.e., in the case where an image recorded on the recorder in the past is displayed in the preview area 5197), GUI components for performing operations such as reproduction, reproduction stop, rewinding, and fast forwarding of the image can be provided in the source operation area 5203.

[0287] In addition, the output operation area 5205 is provided with GUI components for performing various operations displayed on the display device as the output device (swapping, flipping, color adjustment, contrast adjustment, switching between 2D display and 3D display). By appropriately selecting these GUI components, the user can operate the display on the display device.

[0288] Note that the operation screen displayed on the centralized operation panel 5111 is not limited to the example shown in the figure, and the user can input operations to various devices controlled by the audio-visual controller 5107 and the operating room control device 5109 set in the operating room system 5100 through the centralized operation panel 5111.

[0289] Fig. 2251 is a diagram showing an example of the state of an operation to which the above-mentioned operating room system is applied. A ceiling camera 5187 and an operating site camera 5189 are provided on the ceiling of the operating room, and the ceiling camera 5187 and the operating site camera 5189 are capable of imaging the hands of a surgeon (doctor) 5181 who is treating the affected part of a patient 5185 on a bed 5183 and the state of the entire operating room. The ceiling camera 5187 and the operating site camera 5189 may be provided with a magnification adjustment function, a focus adjustment function, an imaging direction adjustment function, and the like. An illumination device 5191 is provided on the ceiling of the operating room, and illuminates at least the hands of the surgeon 5181. The illumination device 5191 is capable of appropriately adjusting the amount of irradiated light, the wavelength (color) of the irradiated light, the direction of the irradiated light, and the like.

[0290] like Fig. 20 As shown, the endoscopic surgery system 5113, the bed 5183, the ceiling camera 5187, the surgical site camera 5189 and the lighting equipment 5191 are connected via the audio-visual control device 5107 and the operating room controller 5109 ( Fig. 22 The centralized operation panel 5111 is provided in the operating room, and as described above, the user can appropriately operate these devices present in the operating room through the centralized operation panel 5111.

[0291] Hereinafter, the configuration of the endoscopic surgery system 5113 will be described in detail. As shown in the figure, the endoscopic surgery system 5113 includes an endoscope 5115, other surgical tools 5131, a support arm device 5141 for supporting the endoscope 5115, and a cart 5151 on which various devices for endoscopic surgery are installed.

[0292] In endoscopic surgery, instead of cutting the abdominal wall, multiple cylindrical opening tools 5139a to 5139d called trockers are inserted into the abdominal wall. Then, the lens barrel 5117 of the endoscope 5115 and other surgical tools 5131 are inserted into the body cavity of the patient 5185 from the trockers 5139a to 5139d. In the example shown, as other surgical tools 5131, a pneumoperitoneum tube 5133, an energy treatment tool 5135 and forceps 5137 are inserted into the body cavity of the patient 5185. The energy treatment tool 5135 is a treatment tool for tissue cutting and stripping, blood vessel sealing, etc. by high-frequency current or ultrasonic vibration. However, the surgical tool 5131 shown in the figure is only an example, and various surgical tools such as forceps and retractors commonly used in endoscopic surgery can be used as surgical tools 5131.

[0293] An image of the surgical part in the body cavity of the patient 5185 imaged by the endoscope 5115 is displayed on the display device 5155. The surgeon 5181 performs treatment such as excision of the affected part using the energy therapy device 5135 and the forceps 5137 while observing the image of the surgical part displayed on the display device 5155 in real time. Although not shown, the pneumoperitoneum tube 5133, the energy therapy device 5135, and the forceps 5137 are supported by the surgeon 5181, an assistant, etc. during the operation.

[0294] (Support arm equipment)

[0295] The support arm device 5141 includes an arm portion 5145 extending from a base 5143. In the example shown, the arm portion 5145 includes joint portions 5147a, 5147b, and 5147c, and connecting rods 5149a and 5149b, and is driven under the control of the arm control device 5159. The endoscope 5115 is supported by the arm portion 5145, and its position and posture are controlled. Therefore, fixation of a stable position of the endoscope 5115 can be achieved.

[0296] (Endoscope)

[0297] The endoscope 5115 includes a barrel 5117, in which a region of a predetermined length from the distal end is inserted into a body cavity of a patient 5185, and includes a camera 5119 connected to the base end of the barrel 5117. In the illustrated example, the endoscope 5115 configured as a so-called rigid scope having a rigid barrel 5117 is shown, but the endoscope 5115 may be configured as a so-called flexible scope having a flexible barrel 5117.

[0298] The distal end of the lens barrel 5117 is provided with an opening in which an objective lens is mounted. A light source device 5157 is connected to the endoscope 5115, and light generated by the light source device 5157 is guided to the distal end of the lens barrel through a light guide extending in the lens barrel 5117, and irradiated to an observation object in the body cavity of the patient 5185 through the objective lens. Note that the endoscope 5115 may be a straight-view mirror, an oblique-view mirror, or a side-view mirror.

[0299] Inside the camera head 5119, an optical system and an image capturing element are provided, and the reflected light (observation light) from the observation object is focused on the image capturing element through the optical system. The image capturing element performs photoelectric conversion on the observation light, and generates an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image. As raw data, the image signal is transmitted to the camera control unit (CCU) 5153. By the way, the camera head 5119 has a function of adjusting the magnification and focal length by appropriately driving the optical system.

[0300] Note that a plurality of image capturing elements may be provided in the camera 5119 in order to cope with, for example, stereoscopic vision (3D display), etc. In this case, a plurality of relay optical systems are provided inside the lens barrel 5117 in order to guide observation light to each of the plurality of image capturing elements.

[0301] (Various equipment mounted on carts)

[0302] The CCU 5153 includes a CPU (central processing unit), a GPU (graphics processing unit), etc., and centrally controls the operations of the endoscope 5115 and the display device 5155. Specifically, the CCU 5153 performs various image processing for displaying an image based on the image signal, such as development processing (demosaic processing), on the image signal received from the camera 5119. The CCU 5153 provides the display device 5155 with the image signal to which the image processing is applied. The CCU 5153 is connected to the display device 5155. Fig. 20 5153. The CCU 5153 also provides the audio-visual controller 5107 with an image signal to which image processing is applied. The CCU 5153 also sends a control signal to the camera 5119 and controls its driving. Such a control signal may include information about shooting conditions such as magnification and focal length. Information about shooting conditions may be input through the input device 5161, or may be input through the above-mentioned centralized operation panel 5111.

[0303] Under the control of the CCU 5153, the display device 5155 displays an image based on an image signal to which image processing is applied by the CCU 5153. When the endoscope 5115 is compatible with high-resolution imaging such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels) and / or compatible with 3D display, a display device 5155 capable of high-resolution display and / or capable of 3D display corresponding thereto can be used. In the case where the display device is compatible with high-resolution imaging such as 4K or 8K, a more immersive feeling can be obtained by using a display device of 55 inches or more as the display device 5155. In addition, a plurality of display devices 50155 having different resolutions and sizes can be provided according to the application.

[0304] The light source device 5157 is configured of a light source such as an LED (Light Emitting Diode), for example, and supplies irradiation light to the endoscope 5115 when imaging a surgical portion.

[0305] The arm control device 5159 is configured by a processor such as a CPU, for example, and operates according to a predetermined program to control the driving of the arm portion 5145 of the supporting arm device 5141 according to a predetermined control method.

[0306] The input device 5161 is an input interface of the endoscopic surgery system 5113. The user can input various information and instructions to the endoscopic surgery system 5113 through the input device 5161. For example, the user inputs various types of information about the surgery, such as the patient's physical information and information about the surgical process, through the input device 5161. In addition, for example, the user inputs an instruction to drive the arm portion 5145, an instruction to change the shooting conditions (shooting type, magnification, focal length, etc.) of the endoscope 5115, an instruction to drive the energy treatment tool 5021, etc. via the input device 5161.

[0307] The type of input device 5161 is not limited, and the input device 5161 may be any of various known input devices. As the input device 5161, for example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5057, a joystick, etc. may be used. When a touch panel is used as the input device 5155, the touch panel may be provided on the display surface of the display device 5155.

[0308] Alternatively, the input device 5161 is a device worn by the user, such as a wearable device or a glasses-type head-mounted display (HMD), and various inputs are performed according to user gestures or sight lines detected by the device. The input device 5161 also includes a camera capable of detecting user movement, and various inputs are performed according to user gestures or sight lines detected from images captured by the camera. In addition, the input device 5161 includes a microphone capable of picking up the user's voice, and various inputs are performed by voice through the microphone. In this way, by configuring the input device 5161 to be able to input various types of information contactlessly, a user (e.g., surgeon 5181) belonging to a particularly clean area can operate a device belonging to a dirty area contactlessly. In addition, the user can operate the device without releasing his hand from the device, thereby improving the convenience of the user.

[0309] The treatment device control device 5163 controls the driving of the energy treatment device 5135 for cauterization of tissues, incision or sealing of blood vessels, etc. The pneumoperitoneum device 5165 delivers gas into the body cavity of the patient 5185 through the pneumoperitoneum tube 5133 to expand the body cavity to ensure the field of view of the endoscope 5115 and to ensure the working space of the surgeon. The recorder 5167 is a device capable of recording various types of information about the operation. The printer 5055 is a device capable of printing various types of information about the operation in various formats, such as text, images, and graphics.

[0310] Hereinafter, the construction of the features of the endoscopic surgical system 5113 will be described in more detail.

[0311] (Support arm equipment)

[0312] The support arm device 5141 includes a base 5143 serving as a base and an arm portion 5145 extending from the base 5143. In the example shown, the arm portion 5145 includes a plurality of joint portions 5147a, 5147b, 5147c and a plurality of links 5149a, 5149b connected by the joint portions 5147b, but in Fig. 22 , for the sake of simplicity, the construction of the arm portion 5145 is shown in a simplified manner. In fact, the shape, number and arrangement of the joint portions 5147a to 5147c and the connecting rods 5149a and 5149b, the direction of the rotation axis of the joint portions 5147a to 5147c, etc. can be appropriately set so that the arm portion 5145 has the desired degree of freedom. For example, the arm portion 5145 can be appropriately configured to have six or more degrees of freedom. Therefore, since the endoscope 5115 can move freely within the movable range of the arm portion 5145, the lens barrel 5117 of the endoscope 5115 can be inserted into the body cavity of the patient 5185 from the desired direction.

[0313] The joint parts 5147a to 5147c are provided with actuators, and the joint parts 5147a to 5147c are configured to rotate around a predetermined rotation axis by driving the actuators. The driving of the actuators is controlled by the arm control device 5159, thereby controlling the rotation angles of the joint parts 5147a to 5147c and controlling the driving of the arm part 5145. Therefore, the position and posture of the endoscope 5115 can be controlled. In this case, the arm controller 5159 can control the driving of the arm part 5145 by various known control methods including force control or position control.

[0314] For example, the surgeon 5181 can appropriately perform operation input through the input device 5161 (including the foot switch 5171) to appropriately control the drive of the arm part 5145 through the arm control device 5159 according to the operation input, thereby controlling the position and posture of the endoscope 5115. Through this control, after the endoscope 5115 at the far end of the arm part 5145 is moved from an arbitrary position to an arbitrary position, the endoscope 5115 can be fixedly supported at the position after the movement. The arm part 5145 can be operated in a so-called master-slave manner. In this case, the user can remotely operate the arm part 5145 through the input device 5161 installed at a location away from the operating room.

[0315] When force control is applied, the arm control device 5159 can perform so-called power-assisted control in which the actuator of each joint portion 5147a to 5147c is driven to receive an external force from the user and smoothly move the arm portion 5145 in accordance with the external force. Therefore, when the user moves the arm portion 5145 while directly touching the arm portion 5145, the arm portion 5145 can be moved with a relatively light force. Therefore, the endoscope 5115 can be moved more intuitively with a simpler operation, thereby improving the convenience of the user.

[0316] Here, in endoscopic surgery, endoscope 5115 is generally supported by a physician called an endoscopist. In contrast, by using support arm device 5141, the position of endoscope 5115 can be more reliably fixed without manual intervention, and therefore, an image of the surgical part can be stably obtained and the surgery can be performed smoothly.

[0317] The arm control device 5159 does not have to be provided in the cart 5151. The arm controller 5159 may not have to be one device. For example, the arm control device 5159 may be provided in each of the joint portions 5147a to 5147c of the arm portion 5145 supporting the arm device 5141, and the drive control of the arm portion 5145 may be realized by a plurality of arm control devices 5159 cooperating with each other.

[0318] (Light source equipment)

[0319] When imaging the surgical part, the light source device 5157 provides irradiation light to the endoscope 5115. The light source device 5157 is composed of a white light source including, for example, an LED, a laser light source, or a combination thereof. At this time, when the white light source is configured in a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, and the white balance of the image captured in the light source device 5157 can be adjusted. In addition, in this case, by irradiating the laser beam from each RGB laser light source onto the observed object in a time-division manner, and by controlling the drive of the image capturing element of the camera 5119 in synchronization with the irradiation timing, it is also possible to capture images corresponding to each RGB in a time-division manner. According to this method, a color image can be obtained without providing a color filter in the capturing element.

[0320] In addition, the drive of the light source device 5157 can be controlled to change the intensity of the light output at a predetermined time interval. By controlling the drive of the imaging element of the camera 5119 in synchronization with the timing of the light intensity change, images are acquired in a time-division manner, and synthesized, it is possible to generate an image with a high dynamic range without the so-called white jump part and black compression part.

[0321] In addition, the light source device 5157 can be configured to be able to provide light of a predetermined wavelength band corresponding to special light observation. In special light observation, so-called narrow-band light observation (narrow-band imaging) is performed, in which, for example, by irradiating light of a narrower band than the irradiation light (i.e., white light) in normal observation, predetermined tissues, such as blood vessels in the surface layer of the mucosa, are photographed with high contrast using the wavelength correlation of light absorbed by body tissue. Alternatively, in special light observation, fluorescence observation can be performed to obtain an image by fluorescence generated by irradiating excitation light. In fluorescence observation, for example, fluorescence observation can be performed by irradiating excitation light to body tissue and observing fluorescence from body tissue (autofluorescence observation), or by locally injecting reagents such as indocyanine green (ICG) into body tissue and irradiating excitation light corresponding to the fluorescence wavelength of the reagent so as to obtain a fluorescence image. The light source device 5157 can be configured to be able to provide narrow-band light and / or excitation light corresponding to special light observation.

[0322] (Camera and CCU)

[0323] refer to Fig.23 , the functions of the camera 5119 and CCU5153 of the endoscope 5115 will be described in more detail. Fig.23 It is shown Fig. 22 A block diagram of an example of the functional configuration of the camera 5119 and the CCU 5153 shown in FIG.

[0324] refer to Fig.23 The camera 5119 has a lens unit 5121, an image capturing unit 5123, a driving unit 5125, a communication unit 5127, and a camera control unit 5129 according to its functions. The CCU 5153 has a communication unit 5173, an image processing unit 5175, and a control unit 5177 according to its functions. The camera 5119 and the CCU 5153 are connected for bidirectional communication via a transmission cable 5179.

[0325] First, the functional configuration of the camera head 5119 will be described. The lens unit 5121 is an optical system provided in the connection portion between the lens barrels 5117. The observation light obtained from the distal end of the lens barrel 5117 is guided to the camera head 5119 and enters the lens unit 5121. The lens unit 5121 is constituted by combining a plurality of lenses including a zoom lens and a focusing lens. The optical characteristics of the lens unit 5121 are adjusted so as to gather the observation light on the light receiving surface of the image capturing element of the image capturing unit 5123. In addition, the zoom lens and the focusing lens are configured to be movable in position on the optical axis to adjust the magnification and focus of the captured image.

[0326] The image capturing unit 5123 includes an image capturing element and is arranged at a later stage of the lens unit 5121. The observation light passing through the lens unit 5121 is collected on the light receiving surface of the image capturing element, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the image capturing unit 5123 is provided to the communication unit 5127.

[0327] As the image capturing element of the image capturing unit 5123, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor capable of capturing a color image with a Bayer array is used. As the image capturing element, for example, a device capable of capturing a high-resolution image of 4K or higher can be used. By obtaining an image of the surgical part at a high resolution, the surgeon 5181 can grasp the state of the surgical part in more detail and can perform the operation smoothly.

[0328] In addition, the image capturing element of the image capturing unit 5123 is configured to have a pair of image capturing elements for acquiring respective image signals for the right eye and the left eye corresponding to the 3D display. By displaying 3D, the surgeon 5181 can more accurately grasp the depth of the living tissue in the surgeon's operating part. By the way, when the image capturing unit 5123 is composed of a multi-plate type corresponding to each image capturing element, a plurality of systems of the lens unit 5121 are also provided.

[0329] In addition, the image capturing unit 5123 may not necessarily be disposed in the camera head 5119. For example, the image capturing unit 5123 may be disposed in the lens barrel 5117 immediately after the objective lens.

[0330] The driving unit 5011 is composed of an actuator, and moves the zoom lens and the focus lens of the lens unit 5121 along the optical axis by a predetermined distance through the control of the camera control unit 5129. As a result, the magnification and focus of the image captured by the image capturing unit 5123 can be appropriately adjusted.

[0331] The communication unit 5127 is configured by a communication device for sending and receiving various types of data relative to the CCU 5153. The communication unit 5127 sends the image signal obtained from the image capturing unit 5123 as raw data to the CCU 5153 via the transmission cable 5179. At this time, it is preferable to send the image signal by optical communication so as to display the captured image of the surgical part with low latency. This is because, during the operation, the surgeon 5181 performs the operation while observing the state of the affected part through the captured image, so that for a safer and more reliable operation, it is necessary to display the moving image of the surgical part as real time as possible. When performing optical communication, the communication unit 5127 is provided with a photoelectric conversion module for converting an electrical signal into an optical signal. After the image signal is converted into an optical signal by the photoelectric conversion module, the image signal is transmitted to the CCU5153 through the transmission cable 5179.

[0332] In addition, the communication unit 5127 receives a control signal from the CCU 5153 for controlling the drive of the camera 5119. The control signal includes, for example, information for specifying a frame rate for capturing an image, information for specifying an exposure value when capturing an image, and / or information for specifying a magnification and a focus for capturing an image. The communication unit 5127 provides the received control signal to the camera control unit 5129. Incidentally, the control signal from the CCU 5153 may also be sent via optical communication. In this case, the communication unit 5127 is provided with a photoelectric conversion module for converting an optical signal into an electrical signal, and the control signal is converted into an electrical signal by the photoelectric conversion module and provided to the camera control unit 5129.

[0333] Incidentally, based on the acquired image signal, the above-mentioned shooting conditions such as frame rate, exposure value, magnification and focus are automatically set by the control unit 5177 of the CCU 5153. That is, the endoscope 5115 is equipped with a so-called AE (automatic exposure) function, AF (automatic focus) function and AWB (automatic white balance) function.

[0334] The camera control unit 5129 controls the driving of the camera 5119 based on the control signal of the CCU 5153 received via the communication unit 5127. For example, the camera control unit 5129 controls the driving of the image capturing element of the image capturing unit 5123 based on the information for specifying the frame rate of the captured image and / or the information for specifying the exposure value when the image is captured. In addition, for example, the camera control unit 5129 appropriately moves the zoom lens and the focus lens of the lens unit 5121 through the drive unit 5125 based on the information for specifying the magnification and focus of the captured image. The camera control unit 5129 may also include a function of storing information for identifying the lens barrel 5117 and the camera 5119.

[0335] Note that the camera head 5119 can be made resistant to high-pressure sterilization by arranging the configuration of the lens unit 5121, the image capturing unit 5123, and the like in an airtight sealing structure having high airtightness and waterproofness.

[0336] Next, the functional configuration of the CCU 5153 will be described. The communication unit 5153 is configured by a communication device for sending and receiving various types of information relative to the camera 5119. The communication unit 5173 receives an image signal sent from the camera 5119 via a transmission cable 5179. At this time, as described above, the image signal can be appropriately sent through optical communication. In this case, corresponding to the optical communication, the communication unit 5173 is provided with a photoelectric conversion module for converting the optical signal into an electrical signal. The communication unit 5173 provides the image signal converted into the electrical signal to the image processing unit 5175.

[0337] In addition, the communication unit 5173 transmits a control signal for controlling the driving of the camera 5119 to the camera 5119. The control signal may also be transmitted through optical communication.

[0338] The image processing unit 5175 performs various image processing on the image signal which is the raw data sent from the camera 5119. The image processing includes, for example, various known signal processing such as development processing, image quality enhancement processing (band enhancement processing, super-resolution processing, NR (noise reduction) processing and / or camera shake correction processing) and / or enlargement processing (electronic zoom processing). In addition, the image processing unit 5175 performs detection processing on the image signal so as to perform AE, AF and AWB.

[0339] The image processing unit 5175 is configured by a processor such as a CPU and a GPU, and when the processor operates according to a predetermined program, the above-mentioned image processing or detection processing can be performed. When the image processing unit 5175 is constructed by a plurality of GPUs, the image processing unit 5175 appropriately divides the information about the image signal, and the image processing is performed in parallel by the plurality of GPUs.

[0340] The control unit 5177 performs various controls related to capturing an image of a surgical portion through the endoscope 5115 and displaying the captured image. For example, the control unit 5177 generates a control signal for controlling the drive of the camera head 5119. At this time, when the user inputs a capturing condition, the control unit 5177 generates a control signal based on the user's input. Alternatively, when the endoscope 5115 has an AE function, an AF function, and an AWB function, the control unit 5177 appropriately calculates an optimal exposure value, a focal length, and a white balance according to the result of the detection processing of the image processing unit 5175, and generates a control signal.

[0341] In addition, the control unit 5177 displays the image of the surgical part on the display device 5155 based on the image signal that has been image-processed by the image processing unit 5175. At this time, the control unit 5177 uses various image recognition technologies to recognize various objects in the image of the surgical part. For example, by detecting the shape, color, etc. of each edge of the object contained in the image of the surgical part, the control unit 5177 can recognize surgical tools such as forceps, specific biological parts, bleeding, fog when using the energy treatment tool 5135, etc. When the image of the surgical part is displayed on the display device 5155, the control unit 5177 uses the recognition result to superimpose and display various types of surgical support information about the image of the surgical part. By superimposing and displaying the operation support information and presenting it to the surgeon 5181, surgery can be performed more safely and reliably.

[0342] The transmission cable 5179 for connecting the camera 5119 and the CCU 5153 is an electric signal cable corresponding to communication through electric signals, an optical fiber corresponding to optical communication, or a composite cable thereof.

[0343] Here, in the example shown, communication is performed wirelessly using the transmission cable 5179, but communication between the camera 5119 and the CCU 5153 may be performed wirelessly. When communication between the two is performed wirelessly, there is no need to place the transmission cable 5179 in the operating room, thereby eliminating the situation where the movement of medical staff in the operating room is hindered by the transmission cable 5179.

[0344] An example of an endoscopic surgery system 5100 to which the technology according to the present disclosure can be applied is described above. Note that although the endoscopic surgery system 5100 is described here as an example, the system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the technology of the present disclosure can be applied to test a flexible endoscope system or a microscope surgery system.

[0345] The technology according to the present disclosure is applicable to the endoscope 5115, CCU 5153, display device 5155, etc. in the above configuration. For example, the image sensor mounted on the endoscope 5115 can be used as a reference. Figure 1 The image processing unit 5175 of the CCU 5153 may be used as a reference Figure 1 The video processing unit 13 is described. The display device 5155 is a reference Figure 1 By applying the technology according to the present disclosure to the endoscope 5115, the CCU 5153, the display device 5155, etc., it is possible to improve the visibility of the peaking display displayed on the display device 5155. As a result, focus adjustment during surgery can be easily achieved, and safe and highly reliable surgery can be performed.

[0346] For example, the technology according to the present disclosure can be applied to a microscope surgery system for so-called microsurgery performed while observing a minute area of ​​a patient in a magnified manner.

[0347] Fig.24 is a diagram showing an example of a schematic configuration of a microscope surgery system 5300 to which the technology according to the present disclosure can be applied. Fig.24 , the microscope surgery system 5300 includes a microscope device 5301, a control device 5317, and a display device 5319. In the following description of the microscope surgery system 5300, "user" refers to any medical personnel who uses the microscope surgery system 5300, for example, a surgeon and an assistant.

[0348] The microscope apparatus 5301 includes a microscope portion 5303 for magnifying an observation object (a surgical part of a patient), an arm portion 5309 supporting the microscope portion 5303 at the distal end, and a base 5315 supporting the base end of the arm portion 5309 .

[0349] The microscope section 5303 includes a cylindrical section 5305 having a substantially cylindrical shape, an image capturing unit (not shown) provided in the cylindrical section 5305, and an operation unit 5307 provided in a partial area of ​​the outer circumference of the cylindrical section 5305. The microscope section 5303 is an electronic image capturing type microscope section (so-called video type microscope section) which captures an image captured electronically by the image capturing unit.

[0350] A cover glass for protecting the internal image capturing unit is provided on the opening surface of the lower end of the cylindrical portion 5305. Light from the observation object (hereinafter, also referred to as observation light) passes through the cover glass and enters the image capturing unit inside the cylindrical portion 5305. Incidentally, a light source such as an LED (light emitting diode) may be provided inside the cylindrical portion 5305. When capturing an image, light may be irradiated from the light source to the observation object through the cover glass.

[0351] The image capturing unit includes an optical system for gathering observation light, and an image capturing element for receiving the observation light gathered by the optical system. The optical system is configured by combining a plurality of lenses including a zoom lens and a focusing lens, and the optical characteristics are adjusted so that the observation light is imaged on the light receiving surface of the image capturing element. The image capturing element generates a signal corresponding to the observation light, that is, an image signal corresponding to the observation image, by receiving and photoelectrically converting the observation light. As the image capturing element, for example, a device capable of capturing a color image with a Bayer array is used. The image capturing element may be various known image capturing elements, such as a CMOS (complementary metal oxide semiconductor) image sensor and a CCD (charge coupled device) image sensor. The image signal generated by the image capturing element is transmitted as raw data to the controller 5317. Here, the transmission of the image signal may preferably be performed by optical communication. This is because since the surgeon performs the operation while observing the state of the affected part through the captured image at the surgical site, it is required to display the moving image of the surgical part as real-time as possible in order to perform a safer and more reliable operation. By transmitting the image signal by optical communication, the captured image can be displayed with low latency.

[0352] Incidentally, the image capturing unit may have a driving mechanism for moving the zoom lens and the focus lens of the optical system along the optical axis. By appropriately moving the zoom lens and the focus lens by the driving mechanism, the magnification of the captured image and the focal length when the image is captured can be adjusted. In addition, various functions that can be generally provided in the electronic image capturing type microscope part, such as the AE (auto exposure) function and the AF (auto focus) function, can be installed on the image capturing unit.

[0353] In addition, the image capturing unit may be configured as a so-called single-board image capturing unit having a single image capturing element, or may be configured as a so-called multi-board image capturing unit having a plurality of image capturing elements. For example, in the case where the image capturing unit is configured by a multi-board type, an image signal corresponding to each of R, G, and B may be generated by each image capturing element, and a color image may be obtained by synthesizing the image signals. Alternatively, the image capturing unit may be configured to have a pair of image capturing elements for respectively acquiring image signals for the right eye and the left eye corresponding to stereoscopic vision (3D display). By displaying 3D, the surgeon can more accurately grasp the depth of living tissue in the surgical part. By the way, when the image capturing unit is composed of a multi-board type, a plurality of optical systems may also be provided corresponding to each image capturing element.

[0354] The operation unit 5307 is composed of, for example, a cross joystick or a switch, and is an input device for receiving an operation input from a user. For example, the user can input an instruction to change the magnification of the observed image and the focal length to the observed object through the operation unit 5307. The drive mechanism of the image capturing unit can appropriately move the zoom lens and the focus lens according to the instruction, thereby adjusting the magnification and the focal length. In addition, for example, the user can input an instruction via the operation unit 5307 to switch the operation mode (full free mode and fixed mode (described later)) of the arm part 5309. When the user wants to move the microscope part 5303, it is assumed that the user moves the microscope part 5303 in a state where the user holds the cylindrical part 5305. Therefore, the operation unit 5307 is preferably set at a position that is easy to operate with a finger in a state where the user holds the cylindrical part 5305, so that it can be operated while moving the cylindrical part 5305.

[0355] The arm portion 5309 is configured such that a plurality of links (a first link 5313 a to a sixth link 5313 f ) are rotatably connected to each other through a plurality of joint portions (a first joint portion 5311 a to a sixth joint portion 5311 f ).

[0356] The first joint portion 5311a has a substantially cylindrical shape, and rotatably supports the upper end of the cylindrical portion 5305 of the microscope portion 5303 at its distal end (lower end) around a rotation axis (first axis O1) parallel to the central axis of the cylindrical portion 5305. Here, the first joint portion 5311a may be configured so that the first axis O1 coincides with the optical axis of the image capturing unit of the microscope portion 5303. Therefore, by rotating the microscope portion 5303 around the first axis O1, the field of view may be changed, thereby rotating the captured image.

[0357] The first link 5313a fixedly supports the first joint portion 5311a at the distal end. Specifically, the first link 5313a is a rod-shaped member having a substantially L-shaped shape, and is connected to the first joint portion 5311a so that one side of its distal end side extends in a direction orthogonal to the first axis O1, and the end of this side abuts against the upper end of the outer periphery of the first joint portion 5311a. The second joint portion 5311b is connected to the substantially L-shaped base end side on the end of the other side of the first link 5313a.

[0358] The second joint portion 5311b has a substantially cylindrical shape and rotatably supports the base end of the first link 5313a at its distal end about a rotation axis (second axis O2) orthogonal to the first axis O1. The distal end of the second link 5313b is fixedly connected to the base end of the second joint portion 5311b.

[0359] The second link 5313b is a rod-shaped member having a substantially L-shaped shape, and an end portion of one side thereof is fixedly connected to the base end of the second joint portion 5311b, and one side of the distal end side thereof extends in a direction orthogonal to the second axis O2. The third joint portion 5311c is connected to the other side of the second link 5313b at the base end side of the substantially L-shape.

[0360] The third joint portion 5311c has a substantially cylindrical shape and rotatably supports the base end of the second link 5313b at its distal end around a rotation axis O3 orthogonal to the first axis O1 and the second axis O2. The distal end of the third link 5313c is fixedly connected to the base end of the third joint portion 5311c. By rotating the configuration including the distal end side of the microscope portion 5303 around the second axis O2 and the third axis O3, the microscope portion 5303 can be moved to change the position of the microscope portion 5303 in the horizontal plane. That is, by controlling the rotation around the second axis O2 and the third axis O3, the field of view of the captured image can be moved in the plane.

[0361] The third link 5313c is configured to have a substantially cylindrical shape on its distal end side and is fixedly connected to the cylindrical distal end so that the base end of the third joint part 5311c has substantially the same central axis. The base end side of the third link 5313c has a prismatic shape, and the fourth joint part 5311d is connected to its end.

[0362] The fourth joint portion 5311d has a substantially cylindrical shape, and rotatably supports the base end of the third link 5313c at a distal end thereof about a rotation axis (fourth axis O4) orthogonal to the third axis O3.

[0363] The fourth link 5313d is a rod-shaped member extending substantially linearly, and is fixedly connected to the fourth joint portion 5311d so that the end of its distal end abuts on the substantially cylindrical side surface of the fourth joint portion 5311d while extending orthogonally to the fourth axis O4. The fifth joint portion 5311e is connected to the base end of the fourth link 5313d.

[0364] The fifth joint portion 5311e has a substantially cylindrical shape and rotatably supports the base end of the fourth link 5313d at its distal end side around a fifth axis (O5) of rotation axis parallel to the fourth axis O4. The distal end of the fifth link 5313e is fixedly connected to the base end of the fifth joint portion 5311e. The fourth axis O4 and the fifth axis O5 are rotation axes capable of moving the microscope portion 5303 in the vertical direction. The height of the microscope portion 5303, that is, the distance between the microscope portion 5303 and the observation object can be adjusted by rotating the distal side configuration including the microscope portion 5303 around the fourth axis O4 and the fifth axis O5.

[0365] The fifth link 5313e is configured by combining a first member having a substantially L-shape extending in a vertical direction on one side and in a horizontal direction on the other side, and a second member having a rod shape extending in a vertically downward direction from a portion of the first member extending in a horizontal direction. The base end of the fifth joint portion 5311e is fixedly connected to the vicinity of the upper end of the portion of the first member extending in the vertical direction of the fifth link 5313e. The sixth joint portion 5311f is connected to the base end (lower end) of the second member of the fifth link 5313e.

[0366] The sixth joint portion 5311f has a substantially cylindrical shape and rotatably supports the base end of the fifth link 5313e at its distal end side around a rotation axis (sixth axis O6) parallel to the vertical direction. The distal end of the sixth link 5313f is fixedly connected to the base end of the sixth joint portion 5311f.

[0367] The sixth link 5313F is a rod-shaped member extending in the vertical direction, and a base end thereof is fixedly connected to the upper surface of the base 5315 .

[0368] The rotatable range of the first joint portion 5311a to the sixth joint portion 5311f is appropriately set so that the microscope portion 5303 can perform the desired movement. As a result, in the arm portion 5309 having the above-mentioned configuration, a total of six degrees of freedom of movement including three degrees of freedom of translation and three degrees of freedom of rotation can be achieved for the movement of the microscope portion 5303. Therefore, by configuring the arm portion 5309 so as to achieve six degrees of freedom regarding the movement of the microscope portion 5303, the position and posture of the microscope portion 5303 can be freely controlled within the movable range of the arm portion 5309. Therefore, the surgical portion can be observed from various angles, and the surgery can be performed more smoothly.

[0369] Note that the configuration of the arm portion 5309 shown in the figure is only an example, and the number and shape (length) of the connecting rods constituting the arm portion 5309, the number of joint portions, the arrangement position, the direction of the rotation axis, etc. can be appropriately designed so that the desired degree of freedom can be achieved. For example, as described above, in order to freely move the microscope portion 5303, the arm portion 5309 is preferably configured to have six degrees of freedom, but the arm portion 5309 can be configured to have more degrees of freedom, (i.e., redundant degrees of freedom). When there are redundant degrees of freedom, the posture of the arm portion 5309 can be changed in a state where the position and posture of the microscope portion 5303 are fixed. Therefore, for example, the posture of the arm portion 5309 can be controlled so that the arm portion 5309 does not interfere with the field of view of the surgeon who is viewing the display device 5319, thereby achieving highly convenient control for the surgeon.

[0370] Here, the first joint part 5311a to the sixth joint part 5311f may be provided with an actuator, on which a driving mechanism such as a motor, an encoder for detecting the rotation angle of each joint part, etc. are installed. The control device 5317 appropriately controls the driving of the actuators provided in the first joint part 5311a to the sixth joint part 5311f, thereby controlling the posture of the arm part 5309, that is, the position and posture of the microscope part 5303. Specifically, the control device 5317 can grasp the current posture of the arm part 5309 and the current position and posture of the microscope part 5303 according to the information about the rotation angle of each joint part detected by the encoder. The control device 5317 responds to the user's operation input, uses the grasped information to calculate the control value of each joint part (for example, the rotation angle or the torque generated, etc.) to realize the movement of the microscope part 5303, and drives the driving mechanism of each joint part according to the control value. In this case, the control method of the arm portion 5309 by the control device 5317 is not limited, and various known control methods such as force control or position control can be applied.

[0371] For example, the surgeon performs appropriate operation input through an input device (not shown), whereby the control device 5317 can appropriately control the drive of the arm portion 5309 in response to the operation input, and can control the position and posture of the microscope portion 5303. Through this control, the microscope portion 5303 can be moved from any position to any position, and then fixedly supported at the position after the movement. As an input device, in view of the convenience of the surgeon, for example, it is preferred to use a foot switch that can be operated even if the surgeon has a surgical tool in his hand. In addition, the wearable device or camera provided in the operating room can be used to perform the operation input contactlessly based on gesture detection or line of sight detection. As a result, even users belonging to the clean area can operate the equipment belonging to the non-clean area with a higher degree of freedom. Alternatively, the arm portion 5309 can be operated by the so-called master-slave method. In this case, the user can remotely operate the arm portion 5309 by an input device installed in a position away from the operating room.

[0372] If force control is applied, so-called power assist control can be performed, in which the actuators of the first joint portion 5311a to the sixth joint portion 5311f are driven to receive an external force from the user, and the arm portion 5309 is smoothly moved according to the external force. As a result, when the user attempts to directly move the position by holding the microscope portion 5303, the microscope portion 5303 can be moved with a relatively light force. Therefore, the microscope portion 5303 can be moved more intuitively with a simpler operation, and the convenience of the user can be improved.

[0373] The drive of the arm portion 5309 can be controlled so as to perform a pivoting operation. Here, the pivoting operation is an operation of moving the microscope portion 5303 so that the optical axis of the microscope portion 5303 always faces a predetermined point in space (hereinafter referred to as a pivot point). According to the pivoting operation, since the same observation position can be observed from various directions, the affected part can be observed in more detail. When the microscope portion 5303 is configured so that the focal length cannot be adjusted, the pivoting operation is preferably performed in a state where the distance between the microscope portion 5303 and the pivot point is fixed. In this case, the distance between the microscope portion 5303 and the pivot point can be adjusted to a fixed focal length of the microscope portion 5303. Therefore, the microscope portion 5303 is pivoted around the pivot point on a hemispherical surface (at a radius corresponding to the focal length) having a radius corresponding to the focal length. Fig.24 ) is moved upward (schematically shown in FIG. 5 ), so that a clear captured image can be obtained even if the observation direction is changed. On the other hand, when the microscope portion 5303 is configured to have an adjustable focal length, the pivoting operation can be performed in a state where the distance between the microscope portion 5303 and the pivot point is variable. In this case, for example, based on information about the rotation angle of each joint portion detected by the encoder, the control device 5317 can calculate the distance between the microscope portion 5303 and the pivot point, and automatically adjust the focal length of the microscope portion 5303 based on the calculation result. Alternatively, if an AF function is provided in the microscope portion 5303, the focal length can be automatically adjusted by the AF function each time the distance between the microscope portion 5303 and the pivot point is changed by a pivoting operation.

[0374] The first joint part 5311a to the sixth joint part 5311f may be provided with a brake for limiting its rotation. The operation of the brake may be controlled by the controller 5317. For example, if it is desired to fix the position and posture of the microscope part 5303, the control device 5317 operates the brakes of each joint part. Therefore, the posture of the arm part 5309, that is, since the position and posture of the microscope part 5303 can be fixed without driving the actuator, power consumption can be reduced. If it is desired to move the position and posture of the microscope part 5303, the control device 5317 releases the brake of each joint part, which is sufficient to drive the actuator according to a predetermined control method.

[0375] This operation of the brake can be performed in response to the operation input by the user through the above-mentioned operation unit 5307. When the user wants to move the position and posture of the microscope part 5303, the user operates the operation unit 5307 to release the brakes of the various joint parts. Therefore, the operation mode of the arm part 5309 is switched to a mode in which each joint part can freely perform rotation (full free mode). When the user wants to fix the position and posture of the microscope part 5303, the user operates the operation unit 5307 to operate the brakes of the various joint parts. As a result, the operation mode of the arm part 5309 is switched to a mode in which the rotation of each joint part is restricted.

[0376] The control device 5317 controls the operation of the microscope device 5301 and the display device 5319, thereby controlling the overall operation of the microscope surgery system 5300. For example, the control device 5317 controls the driving of the arm portion 5309 by operating the actuators of the first joint portion 5311a to the sixth joint portion 5311f according to a predetermined control scheme. In addition, for example, the control device 5317 controls the operation of the brakes of the first joint portion 5311a to the sixth joint portion 5311f, thereby changing the operation mode of the arm portion 5309. In addition, for example, the control device 5317 performs various signal processing on the image signal acquired by the image capturing unit of the microscope portion 5303 of the microscope device 5301, thereby generating image data for display, and displays the image data on the display device 5319. In the signal processing, various well-known signal processing such as development processing (demosaic processing), image quality enhancement processing (band enhancement processing, super-resolution processing, NR (noise reduction) processing, and / or camera shake correction processing), and / or enlargement processing (i.e., electronic zoom processing) can be performed.

[0377] Incidentally, the communication between the control device 5317 and the microscope part 5303 and the communication between the control device 5317 and the first to sixth joint parts 5311a to 5311f can be wired communication or wireless communication. In the case of wired communication, communication by electric signal or optical communication can be performed. In this case, according to the communication mode, the cable used for transmission used in the wired communication can be configured as an electric signal cable, an optical fiber or a composite cable thereof. On the other hand, in the case of wireless communication, since there is no need to lay a transmission cable in the operating room, the situation in which the transmission cable hinders the movement of medical staff in the operating room can be eliminated.

[0378] The controller 5317 may be a processor such as a CPU (central processing unit), a GPU (graphics processing unit), etc., or a microcomputer or control board on which a processor and a storage element such as a memory are fixed and loaded. When the processor of the control device 5317 operates according to a predetermined program, the above-mentioned various functions can be realized. In the example shown, the control device 5317 is provided as a device separated from the microscope device 5301, but the control device 5317 can be arranged in the base 5315 of the microscope device 5301 and configured integrally with the microscope device 5301. Alternatively, the control device 5317 can be configured by a plurality of devices. For example, a microcomputer, a control board, etc. can be arranged in each of the first joint part 5311a to the sixth joint part 5311f of the microscope part 5303 and the arm part 5309, and they can be connected to each other so as to be able to communicate with each other, thereby realizing the same function as the function of the control device 5317.

[0379] The display device 5319 is set in the operating room, and under the control of the control device 5317, displays an image corresponding to the image data generated by the control device 5317. In other words, the display device 5319 displays the image of the surgical part taken by the microscope part 5303. Note that instead of the image of the surgical part or together with the image of the surgical part, the display device 5319 can display various types of information about the surgery, for example, the patient's physical information, information about the surgical process, etc. In this case, the display of the display device 5319 can be appropriately switched by the user's operation. Alternatively, a plurality of display devices 5319 can be provided, and each of the plurality of display devices 5319 can display the image of the surgical part and various types of information about the surgery. As the display device 5319, various known display devices such as a liquid crystal display device or an EL (electroluminescence) display device can be applied.

[0380] Fig.25 It shows the use of Fig.24 FIG. 53 is a diagram of a surgical procedure using the microscope surgical system 5300. Fig.25 , a surgeon 5321 is schematically shown performing surgery on a patient 5325 on a bed 5323 using a microscope surgery system 5300. Fig.25 , for simplicity, the control device 5317 in the configuration of the microscope surgery system 5300 is omitted, and the microscope device 5301 is shown in a simplified manner.

[0381] like Fig.25As shown, during surgery, the image of the surgical part captured by the microscope device 5301 is magnified and displayed on a display device 5319 installed on the wall of the operating room by using a microscope surgical system 5300. The display device 5319 is installed at a position facing the surgeon 5321, and the surgeon 5321 performs various treatments on the surgical part, such as removing the affected part, while observing the state of the surgical part through the image projected on the display device 5319.

[0382] As described above, an example of a microscope surgical system 5300 to which the technology according to the present disclosure can be applied is described. Note that although the microscope surgical system 5300 is described as an example herein, the system to which the technology according to the present disclosure can be applied is not limited to such an example. For example, the microscope device 5301 can also be used as a support arm device that supports other observation devices and other surgical tools instead of the microscope portion 5303 at its distal end. As other observation devices, for example, an endoscope can be applied. As other surgical tools, forceps, pliers, pneumoperitoneum tubes for pneumoperitoneum, energy treatment tools for cutting tissue or cauterizing and sealing blood vessels, etc. can be applied. By supporting the observation device and the surgical tool with a support arm device, the position can be fixed more stably than when the medical staff manually supports the observation device and the surgical tool, and the burden on the medical staff can be reduced. The technology according to the present disclosure can be applied to a support arm device that supports a configuration other than such a microscope portion.

[0383] The technology according to the present disclosure can be suitably applied to the microscope section 5303, the control device 5317, the display device 5319, etc. in the above configuration. For example, an image sensor mounted on the microscope section 5303 is used as a reference. Figure 1 The image sensor 11 is described. The control device 5317 is used as a reference Figure 1 The video processing unit 13 is described. The display device 5319 is a reference Figure 1 The display device 15 described. By applying the technology according to the present disclosure to the microscope portion 5303, the control device 5317, the display device 5319, etc., the visibility of the peaking display displayed on the display device 5319 can be improved. As a result, focus adjustment during surgery can be easily achieved, and safe and highly reliable surgery can be performed.

[0384] It is also possible to combine at least two features of the present technology described above. In other words, the various features described in the various embodiments can be freely combined without distinguishing the embodiments. In addition, the various effects described above are not restrictive, but merely illustrative, and other effects may be provided.

[0385] In the present disclosure, “same”, “equal”, “orthogonal”, etc. are concepts including “substantially the same”, “substantially equal”, “substantially orthogonal”, etc. For example, a state included in a predetermined range (for example, within the range of ±10%) relative to “completely the same”, “completely equal”, “completely orthogonal”, etc. is also included.

[0386] (1) An image capturing device comprising:

[0387] An image generating unit, wherein the image generating unit generates a captured image by capturing a subject;

[0388] an edge detection unit that detects an edge portion included in the generated captured image; and

[0389] A color control unit controls, for each detected edge portion, a color of an emphasis display for emphasizing the edge portion based on color information of the edge portion in the captured image.

[0390] (2) The image capture device according to (1), wherein

[0391] The color control unit controls the color of the emphasized display so that a color difference between an edge color represented by the color information of the edge portion and the color of the emphasized display becomes larger.

[0392] (3) The image capture device according to (2), wherein

[0393] The color control unit controls the emphasized color so that the edge color and the emphasized color are colors that are far apart in a hue plane or a brightness direction.

[0394] (4) The image capture device according to (2) or (3), wherein

[0395] The color control unit sets a color for the emphasized display based on a plurality of color candidates.

[0396] (5) The image capture device according to (4), wherein

[0397] The plurality of color candidates include chromatic color candidates, and

[0398] The color control unit sets a color candidate having a hue farthest from a hue of the edge color among the plurality of color candidates as the emphasized display color.

[0399] (6) The image capture device according to (5), wherein

[0400] The plurality of color candidates include a first color candidate and a second color candidate having a different hue from the first color candidate.

[0401] (7) The image capture device according to (6), wherein

[0402] The first color candidate and the second color candidate are set to colors that are complementary to each other.

[0403] (8) The image capture device according to any one of (4) to (7), wherein

[0404] The plurality of color candidates include achromatic color candidates, and

[0405] The color control unit sets a color candidate having a brightness farthest from the brightness of the edge color among the plurality of color candidates as the emphasized display color.

[0406] (9) The image capture device according to (8), wherein

[0407] The plurality of color candidates include black and white, and

[0408] When the brightness of the edge color is greater than a predetermined threshold, the color control unit sets the emphasized display color to black, and when the brightness of the edge color is less than a predetermined threshold, sets the emphasized display color to white.

[0409] (10) The image capture device according to any one of (4) to (9), wherein

[0410] The plurality of color candidates are set by a user.

[0411] (11) The image capture device according to any one of (4) to (10), wherein

[0412] The plurality of color candidates are set according to the subject.

[0413] (12) The image capture device according to any one of (2) to (11), wherein

[0414] The edge detection unit detects the pixel position of the edge portion, and

[0415] The color control unit sets the color of the emphasized display for each detected pixel position.

[0416] (13) The image capture device according to (12), wherein

[0417] The color information of the edge portion includes pixel information of reference pixels included in a predetermined pixel area surrounding a pixel position of the edge portion, and

[0418] The color control unit calculates a hue of the edge color or a brightness of the edge color based on pixel information of the reference pixel.

[0419] (14) The image capture device according to (13), wherein

[0420] The color control unit calculates an average hue of the reference pixels as the hue of the edge color.

[0421] (15) The image capture device according to (13) or (14), wherein

[0422] The predetermined pixel area is set according to the object.

[0423] (16) The image capture device according to any one of (1) to (15), wherein

[0424] The color control unit generates a peak image in which an edge portion of the captured image is emphasized with the emphasized display color.

[0425] (17) The image capture device according to (16), wherein

[0426] In the peak image, the color of a portion different from the edge portion is set to the same color as that of the original image.

[0427] (18) The image capture device according to (16) or (17), wherein

[0428] The color control unit controls the color of the emphasized display so that the display of the edge portion of the peak image changes dynamically.

[0429] (19) An image capturing method executed by a computer system, comprising:

[0430] generating a captured image in which an image of a subject is captured;

[0431] detecting an edge portion included in the generated captured image; and

[0432] For each detected edge portion, a color of an emphasis display for emphasizing the edge portion is controlled based on color information of the edge portion in the captured image.

[0433] (20) A program causing a computer system to execute the following steps:

[0434] generating a captured image in which an image of a subject is captured;

[0435] detecting an edge portion included in the generated captured image; and

[0436] For each detected edge portion, a color of an emphasis display for emphasizing the edge portion is controlled based on color information of the edge portion in the captured image.

[0437] 1 Subject

[0438] 11 Image Sensor

[0439] 13 Video Processing Unit

[0440] 25 Original Image

[0441] 30 Camera signal processing unit

[0442] 31 Peak detection unit

[0443] 32 Overlay color setting units

[0444] 33 Overlay color selection units

[0445] 34 Synthesis Units

[0446] 40,240 peak images

[0447] 41 Edge

[0448] 42,42a,42b Enhanced display

[0449] 43 Color Candidates

[0450] 43a First color candidate

[0451] 43b Second color candidate

[0452] 44, 44a, 44b Edge color

[0453] 47 reference pixels

[0454] 51 Predetermined pixel area

[0455] 100 Image capture device

Claims

1. An image capturing device, comprising: An image generating unit, wherein the image generating unit generates a captured image by photographing the subject; an edge detection unit that detects an edge portion included in the generated captured image; and a color control unit configured to control, for each detected edge portion, a color of an emphasis display for emphasizing the edge portion based on color information of the edge portion in the captured image; in The color control unit controls the color of the emphasized display so that the color difference between the edge color represented by the color information of the edge portion and the color of the emphasized display becomes larger, The edge detection unit detects the pixel position of the edge portion, The color control unit sets the emphasized display color for each detected pixel position, The color information of the edge portion includes pixel information of reference pixels included in a predetermined pixel area surrounding a pixel position of the edge portion, The color control unit calculates the hue of the edge color or the brightness of the edge color based on the pixel information of the reference pixel, The color control unit sets the emphasized display color according to a plurality of color candidates, The plurality of color candidates include chromatic color candidates, and The color control unit sets a color candidate having a hue farthest from a hue of the edge color among the plurality of color candidates as the emphasized display color.

2. The image capture device according to claim 1, wherein The color control unit controls the emphasized color so that the edge color and the emphasized color are colors that are far apart in a hue plane or a brightness direction.

3. The image capture device according to claim 1, wherein The plurality of color candidates include a first color candidate and a second color candidate having a different hue from the first color candidate.

4. The image capture device according to claim 3, wherein The first color candidate and the second color candidate are set to colors that are complementary to each other.

5. The image capture device according to claim 1, wherein The plurality of color candidates include achromatic color candidates, and The color control unit sets a color candidate having a brightness farthest from the brightness of the edge color among the plurality of color candidates as the emphasized display color.

6. The image capture device according to claim 5, wherein The plurality of color candidates include black and white, and When the brightness of the edge color is greater than a predetermined threshold, the color control unit sets the emphasized display color to black, and when the brightness of the edge color is less than a predetermined threshold, sets the emphasized display color to white.

7. The image capture device according to claim 1, wherein The plurality of color candidates are set by a user.

8. The image capture device according to claim 1, wherein The plurality of color candidates are set according to the subject.

9. The image capture device according to claim 1, wherein The color control unit calculates an average hue of the reference pixels as the hue of the edge color.

10. The image capture device according to claim 1, wherein The predetermined pixel area is set according to the object.

11. The image capture device according to claim 1, wherein The color control unit generates a peaking image in which an edge portion of the captured image is emphasized with the emphasized display color.

12. The image capture device according to claim 11, wherein In the peaking image, the color of a portion different from the edge portion is set to the same color as that of the original image.

13. The image capture device according to claim 11, wherein The color control unit controls the color of the emphasized display so that the display of the edge portion of the peaking image changes dynamically.

14. An image capturing method executed by a computer system, comprising: generating a captured image in which an image of a subject is captured; detecting an edge portion included in the generated captured image; and for each detected edge portion, controlling a color of an emphasis display for emphasizing the edge portion based on color information of the edge portion in the captured image, in controlling the color of the emphasized display so that the color difference between the edge color represented by the color information of the edge portion and the color of the emphasized display becomes larger, detecting pixel positions of the edge portions, Setting the highlighted color for each detected pixel position, The color information of the edge portion includes pixel information of reference pixels included in a predetermined pixel area surrounding a pixel position of the edge portion, calculating the hue of the edge color or the brightness of the edge color based on the pixel information of the reference pixel, The color of the emphasized display is set according to a plurality of color candidates, The plurality of color candidates include chromatic color candidates, and A color candidate having a hue farthest from a hue of the edge color among the plurality of color candidates is set as the emphasized display color.

15. A storage medium having a program recorded thereon, the program causing a computer system to execute the following steps: generating a captured image in which an image of a subject is captured; detecting an edge portion included in the generated captured image; and for each detected edge portion, controlling a color of an emphasis display for emphasizing the edge portion based on color information of the edge portion in the captured image, in controlling the color of the emphasized display so that the color difference between the edge color represented by the color information of the edge portion and the color of the emphasized display becomes larger, detecting pixel positions of the edge portions, Setting the highlighted color for each detected pixel position, The color information of the edge portion includes pixel information of reference pixels included in a predetermined pixel area surrounding a pixel position of the edge portion, and the hue of the edge color or the brightness of the edge color is calculated based on the pixel information of the reference pixels. The color of the emphasized display is set according to a plurality of color candidates, The plurality of color candidates include chromatic color candidates, and A color candidate having a hue farthest from a hue of the edge color among the plurality of color candidates is set as the emphasized display color.

Images