Imaging device, imaging signal processing device and imaging signal processing method

Abstract

The object of the present invention is to reduce the image display delay caused by the display delay time. For this purpose, the imaging device according to the present technology has: an imaging element; a delay time acquisition unit that acquires the display delay time of the captured image, the display delay time being based on the exposure time of the imaging element and the development processing time required for the development processing; and an image data generation unit that generates modified display image data in which the image display delay caused by the display delay time is reduced by using the amount of change in the position or direction of the main body of the imaging device during the display delay time. The present invention is intended to prevent image discomfort by reducing the delay in image display.

Description

Technical Field

[0001] The present technology relates to the technical field of an imaging device, an imaging signal processing device, and an imaging signal processing method for generating image data for display from acquired captured image data. Specifically, the present technology relates to an imaging device, an imaging signal processing device, and an imaging signal processing method for performing processing according to a delay time from imaging to display. Background Art

[0002] When displaying images captured by an imaging device such as a camera or camcorder, visually induced motion discomfort (video discomfort) may be caused in the viewer due to the shaking of the image in the moving image. In addition, in still images, there is a concern that the image may be blurred due to camera shake when shooting, which may make it unclear.

[0003] PTL 1 discloses a technology for preventing the motion of a camera from being reflected in a display frame image by changing a cropping area when display frame image data is generated by cropping a part of frame image data.

[0004] [Citation list]

[0005] [Patent Document]

[0006] [PTL 1]

[0007] JP 2017-34494A Summary of the invention

[0008] [Technical issues]

[0009] Incidentally, necessary time is required to expose subject light (light reflected from the subject), perform development processing (various imaging signal processing), etc., in order to display and record captured image data captured by a camera or video camera. That is, the captured image data is not displayed (recorded) immediately, but is displayed (recorded) after a display delay time. The display delay time may cause image discomfort to the viewer.

[0010] For example, when the imaging device and the photographer's head shake left and right while the photographer is looking at an electronic viewfinder (EVF) provided in the imaging device, a deviation based on the display delay time occurs between the visual information of the display image displayed on the monitor inside the EVF and the sense of balance caused by the shaking of the head, which may cause image discomfort.

[0011] Therefore, an object of the present technology is to reduce image display delay caused by display delay time.

[0012] [Solution to the problem]

[0013] An imaging device according to the present technology includes: an imaging element; a delay time acquisition unit configured to acquire a display delay time of a captured image based on an exposure time of the imaging element and a development processing time required for a development process; and an image data generation unit configured to generate modified display image data in which an image display delay caused by the display delay time has been shortened using an amount of change in position or posture of a main body of the imaging device during the display delay time.

[0014] The modified display image data in which the image display delay is reduced is display image data obtained by performing translation, enlargement and reduction, or rotational movement in the up-down direction and the left-right direction when performing conversion from captured image data captured by the imaging element to display image data for display on the display unit, so that the movement amount of the image during the delay time is offset. Hereinafter, such translation, enlargement and reduction, and rotation are collectively referred to as shift.

[0015] The above-mentioned imaging device may include a display unit on which the modified display image data is displayed.

[0016] The display unit is, for example, an electronic viewfinder (EVF) monitor provided in the EVF or a rear monitor attached to the back of the imaging device.

[0017] The display unit in the above imaging device may be arranged inside the electronic viewfinder.

[0018] A housing that covers a display unit (EVF monitor) in the electronic viewfinder may be provided around the electronic viewfinder.

[0019] The image data generating unit in the imaging device as described above may perform a modification process for generating the modified display image data so that a display position of the subject on the display unit is shifted compared to a case where the modification process is not performed.

[0020] By generating modified display image data to be shifted relative to the position on the captured image data, a display following the movement of the imaging device can be performed.

[0021] The image data generating unit in the above imaging device may generate non-modified display image data in which image display delay due to display delay time is not shortened, and the display image data sent to the display unit outside the imaging device may be the non-modified display image data.

[0022] The display unit external to the imaging device is, for example, a stationary television receiver, a liquid crystal display device, or a portable smart phone.

[0023] The image data generating unit in the above-mentioned imaging device can generate non-modified display image data in which the image display delay caused by the display delay time is not shortened, and can include a display image data selecting unit, which is configured to select modified display image data or non-modified display image data as the display image data to be displayed on the display unit according to the shooting status.

[0024] The shooting state is based on, for example, a still image capture mode or a motion image capture mode (i.e., the type of shooting mode). Alternatively, the shooting state may be the amount of movement, acceleration, or angular velocity of the imaging device in the up-down direction and the left-right direction. In addition, the shooting state may be an output destination for display image data.

[0025] When a mode for photographing a moving image is selected as a photographing state, the display image data selection unit in the above-mentioned imaging device may select non-modified display image data.

[0026] In the moving image capturing mode, it is conceivable that the photographer suppresses the movement amount or acceleration change per unit time of the imaging device in the up-down direction and the left-right direction in order to capture an easy-to-view moving image.

[0027] When a mode for capturing a still image is selected as a shooting state, the display image data selection unit in the above-mentioned imaging device may select the modified display image data.

[0028] It is conceivable that in the still image capturing mode, the photographer determines the angle of view so that the subject desired to be photographed fits in the composition desired to be photographed, and moves the imaging device so that the subject fits in the angle of view. In addition, in this case, it is conceivable to quickly move the imaging device so as not to miss the best shooting timing considered by the photographer.

[0029] The image data generating unit in the above-mentioned imaging device may perform generation of modified display image data based on information of an incident optical system for the imaging element.

[0030] The information on the incident optical system of the imaging element is, for example, information on the type of lens such as a telephoto lens or a wide-angle lens, or a set value of an aperture.

[0031] The image data generating unit in the above-mentioned imaging device may use pixel data having a specific brightness for a region in the modified display image data which is not imaged by the imaging element.

[0032] The individual pixels of the display image displayed based on the modified display image data may include pixels in which captured image data within the cropping range of the imaging element is displayed, pixels in which captured image data outside the cropping range of the imaging element is displayed, and pixels outside the range of the imaging element, that is, pixels in which an image within the range not captured by the imaging element is to be displayed.

[0033] The image data generation unit in the above-mentioned imaging device may use the captured image data outside the trimming range for the modified display image data.

[0034] The captured image data in the clipping range refers to an area clipped as a range of lens distortion pixels or pixels effective in processing such as image stabilization in the entire pixel area of ​​the imaging element exposed to the subject light. That is, the captured image data outside the clipping range refers to an area other than the area clipped as a range of lens distortion pixels or pixels effective in processing such as image stabilization.

[0035] The above-mentioned imaging device may include a second imaging element configured to generate second captured image data based on incident light incident through a second incident optical system, wherein the image data generation unit may use the second captured image data for an area in the modified display image data that is not imaged by the imaging element.

[0036] For example, the imaging device may include a sub-imaging device including a second incident optical system and a second imaging element.

[0037] The image data generation unit in the above-mentioned imaging device may use the captured image data within the trimming range for the area not imaged by the imaging element in the modified display image data.

[0038] That is, the modified display image data is generated using the captured image data within the trimming range.

[0039] The image data generating unit in the above-mentioned imaging device may determine the superimposed position according to the shift amount of the subject relative to at least a part of the superimposed display image to be superimposed on the captured image data and displayed.

[0040] The shift amount of the subject is the difference between the position on the display area where the subject is displayed when the image display delay caused by the display delay time is not taken into account and the position on the display area where the subject is displayed when the image display delay caused by the display delay time is taken into account (including translation, enlargement and reduction, and rotational movement).

[0041] Furthermore, the superimposed display image is an icon image for indicating a shooting mode, a display icon indicating the remaining number of shots, an image icon indicating the battery level, an icon indicating a focus aiming position, an image icon indicating the face position of a recognized person, etc. Alternatively, for example, a histogram indicating brightness distribution may also be included in the superimposed display image.

[0042] The image data generating unit in the above-mentioned imaging device may determine the superimposed position according to the shift amount relative to the in-focus imaging used as the superimposed display image.

[0043] For example, a focus target is a superimposed display image that represents a specific location on an image.

[0044] In the above-described imaging device, an area on the display area of ​​the display unit where the captured image is displayed may be smaller than the display area.

[0045] For example, a display image corresponding to the captured image data is displayed in a part of the entire display area of ​​the display unit (for example, a part around the area of ​​the center part).

[0046] The image data generating unit in the above-mentioned imaging device may perform generation of display image data on which a guide frame indicating the output area is superimposed.

[0047] The guide frame indicating the output area is, for example, a guide frame indicating the recording area.

[0048] The imaging signal processing device of the present technology includes a delay time acquisition unit, which is configured to acquire the display delay time of a captured image based on the exposure time of an imaging element included in the imaging device and the development processing time required for the development processing; and an image data generation unit, which is configured to generate modified display image data, in which the image display delay caused by the display delay time has been shortened using the amount of change in the position or posture of the imaging device during the display delay time.

[0049] The imaging signal processing method of the present technology includes obtaining a display delay time of a captured image based on an exposure time of an imaging element included in an imaging device and a development processing time required for a development process; and generating modified display image data, in which the image display delay caused by the display delay time has been shortened using the amount of change in the position or posture of the imaging device during the display delay time. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a perspective view of an imaging device according to an embodiment of the present technology.

[0051] Figure 2 is a block diagram of an imaging device.

[0052] Figure 3 is a diagram showing display delay time.

[0053] Figure 4 is a diagram showing a delay in displaying an image.

[0054] Figure 5 is a diagram showing the amount of change due to translation in the X-axis direction.

[0055] Figure 6 is a diagram showing the amount of change due to yaw.

[0056] Figure 7 is a diagram showing the amount of change due to translation in the Y-axis direction.

[0057] Figure 8 It is a diagram showing the amount of change due to pitch.

[0058] Fig. 9 is a diagram showing the amount of change due to translation in the Z-axis direction.

[0059] Fig.10 is a diagram showing the amount of change due to scrolling.

[0060] Fig.11 is a diagram showing an example of a landscape as an imaging object.

[0061] Fig.12 is a diagram showing the current viewing angle of the imaging device.

[0062] Fig.13 is a diagram showing an image currently displayed on the display unit.

[0063] Fig.14 is a diagram showing a first example of modified display image data.

[0064] Fig.15 is a diagram showing a second example of modified display image data.

[0065] Fig.16 is a diagram showing an example in which display image data is displayed small relative to the entire display area.

[0066] Fig.17 is a diagram showing a third example of modified display image data.

[0067] Fig.18 is a diagram showing a fourth example of modified display image data.

[0068] Fig.19 is a diagram showing an example in which the size of the second region varies depending on the type of lens.

[0069] Fig. 20 is a diagram showing an example of the size of the second area varying with the type of lens, and is a diagram showing that the second area is larger than Fig.19 FIG. 4 is an example of a second region in FIG.

[0070] Fig.21 is a perspective view of an imaging device including a sub-imaging unit.

[0071] Fig. 22 is a perspective view of an imaging device including a sub-imaging unit, seen from another angle.

[0072] Fig.23 is a block diagram of an imaging device including a sub-imaging unit.

[0073] Fig.24 is a diagram showing a sixth example of modified display image data.

[0074] Fig.25 It is a diagram showing expanded image data.

[0075] Fig.26 It is a figure which shows the blur expansion image data.

[0076] Fig. 27 is a diagram showing a state in which the image within the trimming range has been shifted.

[0077] Fig.28 It is a diagram showing a part of the blurred expanded image data.

[0078] Fig.29 is a diagram showing a seventh example of modified display image data.

[0079] Fig.30 This is a flowchart of the main processing.

[0080] Fig.31 is a flowchart of the subroutine processing.

[0081] Fig.32 is a diagram showing an example of display image data including a superimposed display image.

[0082] Fig.33 is a diagram showing an example of modified display image data including a superimposed display image.

[0083] Fig.34 It is a graph showing changes in display delay time.

[0084] Fig.35 is a diagram schematically showing the overall configuration of an operating room system.

[0085] Fig.362 is a diagram showing a display example of an operation screen on the centralized operation panel.

[0086] Fig.37 1 is a diagram showing an example of a state of surgery to which the operating room system is applied.

[0087] Fig.38 It is shown Fig.37 A block diagram showing an example of the functional configuration of a camera head and a CCU.

[0088] Fig.39 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0089] Fig.40 2 is an explanatory diagram showing an example of installation positions of the vehicle exterior information detection unit and the imaging unit. DETAILED DESCRIPTION

[0090] Hereinafter, the embodiments will be described in the following order with reference to the accompanying drawings.

[0091] <1. Configuration of imaging equipment>

[0092] <2. Display delay time>

[0093] <3. Generation of Modified Display Image Data>

[0094] <3-1. Calculation of modification amount>

[0095] <3-2. First Example of Modified Display Image Data>

[0096] <3-3. Second Example of Modified Display Image Data>

[0097] <3-4. Third Example of Modified Display Image Data>

[0098] <3-5. Fourth Example of Modified Display Image Data>

[0099] <3-6. Fifth Example of Modified Display Image Data>

[0100] <3-7. Sixth Example of Modified Display Image Data>

[0101] <3-8. Seventh Example of Modified Display Image Data>

[0102] <4. Flowchart>

[0103] <5. Display Position of Superimposed Display Image>

[0104] <6. Selection of Modified Display Image Data and Non-Modified Display Image Data>

[0105] <7. Modification Example>

[0106] <8. Conclusion>

[0107] <9. Application Example 1>

[0108] <10. Application Example 2>

[0109] <11. This technology>

[0110] <1. Configuration of imaging equipment>

[0111] Figure 1 The outer appearance of the imaging apparatus 1 according to the present embodiment is shown.

[0112] The imaging apparatus 1 includes a camera housing 2 including an internal space in which a substrate and the like are arranged, and a lens housing 3 attached to the front of the camera housing 2 and having a lens group arranged inside.

[0113] In the camera housing 2, an electronic viewfinder (EVF) 4 is provided at the top, and a rear monitor 5 is attached to the rear side. In the camera housing 2, various operators 6 for imaging a subject and for confirming a captured image are provided in respective units. Specifically, the operators are, for example, a playback menu start button, an OK button, a cross key, a cancel button, a zoom key, a slide key, and a shutter button (release button).

[0114] The EVF 4 includes an EVF monitor 7 visually recognizable from behind and a frame-shaped housing 8 protruding rearward to surround the upper side and left and right sides of the EVF monitor 7. That is, the EVF monitor 7 is provided inside the viewfinder.

[0115] The rear monitor 5 can rotate relative to the camera housing 2. For example, the rear monitor 5 can be rotated so that the lower end portion of the rear monitor 5 moves rearward and the upper end portion of the rear monitor 5 serves as a rotation axis.

[0116] The right end portion or the left end portion of the rear monitor 5 may be used as a rotation axis. In addition, the rear monitor 5 may be rotatable in a plurality of directions.

[0117] Figure 2 is a block diagram of the imaging device 1. The imaging device 1 includes an optical system 50, an imaging element unit 51, a signal processing unit 52, a control unit 53, a memory 54, a lens driving driver 55, a user interface (UI) 56, a sensor 57, a storage unit 58, a communication unit 59, and the like.

[0118] The optical system 50 includes various lenses (e.g., an incident end lens, a zoom lens, a focusing lens, and a condenser lens) and an aperture mechanism. The aperture mechanism performs exposure control by, for example, adjusting the aperture amount of a lens or a diaphragm (aperture) so that sensing is performed in a state where the signal charge is not saturated and within a dynamic range.

[0119] The imaging element unit 51 includes, for example, a charge coupled device (CCD) type or a complementary metal oxide semiconductor (CMOS) type imaging element 51 a .

[0120] The sensor surface of the imaging element 51 a includes a sensing element in which a plurality of pixels are arranged two-dimensionally.

[0121] The imaging element unit 51 performs, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. on the electrical signal obtained by photoelectrically converting the light received by the imaging element 51a, and further performs analog / digital (A / D) conversion processing. The captured image data as digital data is output to the signal processing unit 52 at the next stage.

[0122] The signal processing unit 52 is constituted by, for example, a microprocessor (eg, a digital signal processor (DSP)) or a microcomputer dedicated to digital signal processing.

[0123] The signal processing unit 52 performs various signal processes on the digital signal (captured image signal) transmitted from the imaging element unit 51 .

[0124] For example, pre-processing, synchronization processing, YC processing, resolution conversion processing, various codec processing, etc. are performed.

[0125] Various codec processes are, for example, encoding processes for recording or communicating resolution-converted image data.

[0126] In the preprocessing, pinching processing for pinching the black levels of R, G, and B to predetermined levels, correction processing between color channels of R, G, and B, and the like are performed on the captured image data from the imaging element unit 51 .

[0127] In the synchronization process, color separation processing is performed so that the image data of each pixel has all color components of R, G, and B. For example, in the case of the imaging element 51a that has used a color filter in the Bayer array, demosaic processing is performed as the color separation processing.

[0128] In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from R, G, and B image data.

[0129] In the resolution conversion process, a process for converting the resolution is performed on the image data subjected to various signal processes.

[0130] The control unit 53 is constituted by a microcomputer (arithmetic processing device) including a central processing unit (CPU), etc. The control unit 53 includes a timer (not shown) capable of measuring time.

[0131] The memory 54 stores information and the like used for processing of the control unit 53. The illustrated memory 54 comprehensively represents various memories such as a random access memory (RAM), a read only memory (ROM), and a flash memory (nonvolatile memory).

[0132] The memory 54 may be a memory area built in a microcomputer chip serving as the control unit 53 , or may be configured by a separate memory chip.

[0133] RAM is used to temporarily store data, programs, etc., and serves as a working area for various data processing of the CPU.

[0134] The ROM or flash memory is used to store application programs, firmware, and the like for various operations, in addition to an operating system (OS) or content files such as image files that allow the CPU to control each unit.

[0135] The control unit 53 comprehensively controls the entire imaging device 1 by executing a program stored in a ROM, a flash memory, or the like.

[0136] For example, the control unit 53 controls the optical system 50 via the lens driving driver 55 to control a zoom operation, a focus operation, an exposure adjustment, etc. In addition, the control unit 53 receives a detection signal from the sensor 57 and implements various operations.

[0137] In addition, the control unit 53 receives instructions for controlling the shutter speed of the imaging element unit 51 and various signal processing in the signal processing unit 52, as well as operation information corresponding to the user's operation via the user interface 56, and controls the operation of various necessary units for imaging operations, recording operations, playback operations of recorded image files, etc.

[0138] The lens driving driver 55 is provided with, for example, a motor driver for a zoom lens driving motor, a motor driver for a focus lens driving motor, and a motor driver for driving a motor of an aperture mechanism.

[0139] The sensor 57 comprehensively represents various sensors included in the imaging device 1 . Figure 2 There are shown some sensors 57 included in the imaging device 1. Examples of the sensor 57 include an acceleration sensor 60, an angular velocity sensor 61, and a proximity sensor 62.

[0140] The acceleration sensor 60 and the angular velocity sensor 61 detect a position change or a posture change of the camera housing 2, and send the detection signal as acceleration data and angular velocity data to the control unit 53. The control unit 53 can determine the posture or posture change of the camera housing 2 based on the detection signal. The acceleration sensor 60 and the angular velocity sensor 61 can be used not only for each process described below but also for image stabilization.

[0141] For example, the acceleration sensor 60 is provided for each of the mutually orthogonal X-axis, Y-axis, and Z-axis. Furthermore, the angular velocity sensor 61 is provided so as to be able to detect rotation of each of pitch, yaw, and roll, for example.

[0142] In the following description, the optical axis direction of the substantially cylindrical lens housing 3 (ie, the axial direction extending from the imaging device 1 to the subject) is the Z axis, the left-right direction is the X axis, and the up-down direction is the Y axis.

[0143] The proximity sensor 62 is provided near the EVF monitor 7, for example, detects that the user's face is approaching the EVF monitor 7, and sends a detection signal to the control unit 53. The control unit 53 performs on / off operations of various display devices based on the detection signal, for example.

[0144] The storage unit 58 is constituted by, for example, a nonvolatile memory, and is used as a storage area that stores image files (content files) such as still image data or moving image data, attribute information of the image files, thumbnails, and the like.

[0145] Image files are stored in formats such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), and Graphics Interchange Format (GIF).

[0146] The actual form of the storage unit 58 can be considered in various ways. For example, the storage unit 58 may be configured as a flash memory built into the imaging device 1, or may be configured as a memory card (e.g., a portable flash memory) that can be attached to and detached from the imaging device 1, and an access unit that performs access to store or read in or from the memory card. In addition, the storage unit 58 may also be implemented as a hard disk drive (HDD) or the like as a form built into the imaging device 1.

[0147] The communication unit 59 performs data communication or network communication with an external device in a wired or wireless manner.

[0148] For example, the communication unit 59 performs transmission and reception of captured image data (still image files or moving image files) to and from an external display device, a recording device, a playback device, etc. In the following description, image data captured by the imaging element 51a is referred to as captured image data regardless of whether the image data is recorded.

[0149] Furthermore, for example, communication can be performed by various networks such as the Internet, a home network, and a local area network (LAN) as a network communication unit, and transmission and reception of various data to and from a server, a terminal, and the like on the network can be performed.

[0150] When display image data output from the signal processing unit 52 is displayed on an external monitor of the imaging device 1 connected by a video cable or the like, the display image data is transmitted via the control unit 53 and the communication unit 59. The display image data may be transmitted from the signal processing unit 52 to the external monitor without passing through the control unit 53.

[0151] In the present embodiment, the control unit 53 calculates a shift amount for reducing the image display delay caused by the display delay time based on the detection signal of the sensor 57, and sends the shift amount to the signal processing unit 52. The signal processing unit 52 generates modified display image data obtained by modifying the display image data using the shift amount, and outputs the modified display image data to the display unit (EVF monitor 7 or rear monitor 5). This will be described in detail below.

[0152] <2. Display delay time>

[0153] In the case of visually recognizing a subject through a display image displayed on a display unit such as the EVF monitor 7, it takes extra time to recognize the subject, unlike the case of visually recognizing the subject by directly looking at the subject. This is caused by the time required for various processes.

[0154] This will refer to Figure 3 Specific description.

[0155] Figure 3 A shows the time required for a person to recognize a subject when looking directly at the subject. The time required for a person to recognize a subject when looking directly at the subject is the sum of the time it takes to generate video information of the subject in the person's visual system (visual system time T1) and the time it takes for the brain to recognize the video information (cognitive time T2).

[0156] Figure 3 B shows the time required to recognize a subject when visually recognizing the subject from an optical viewfinder (OVF) in an imaging device including an OVF. The time it takes for a person to recognize a subject becomes longer with the distance before the subject light (light reflected from the subject) enters the eye, which is extended due to physical reflection from each mirror ( Figure 3 However, since the extended distance is at most a few cm to tens of cm, and the time it takes for light to pass through this distance is extremely close to zero, Figure 3 The time and Figure 3 The times shown in B are essentially the same.

[0157] Therefore, when the photographer has visually recognized the subject through the OVF, the photographer does not feel uncomfortable.

[0158] Figure 3 C shows the time required for the photographer to recognize the subject when visually recognizing the subject through the EVF monitor 7 provided in the EVF 4.

[0159] The subject light received by the imaging element 51a via the optical system 50 reaches the photographer's eyes after the exposure time of the imaging element 51a (exposure time T4), the time for reading out the electric signal obtained by photoelectric conversion (readout time T5), the time for performing various signal processing on the captured image data (digital signal) (development time T6), and the time for generating display image data and displaying the display image data on the display unit (display time T7). After the above-mentioned times T1 and T2 have passed, the photographer recognizes the subject displayed on the EVF monitor 7.

[0160] That is, the time required to recognize the subject through the EVF 4 becomes longer by T4, T5, T6, and T7 than the time of direct gaze.

[0161] This longer time is hereinafter referred to as "display delay time".

[0162] The display delay time is not constant and varies with shooting conditions, etc. Specifically, the exposure time T4 varies with the setting of the aperture mechanism, etc. The readout time T5 or the development time T6 varies with the size (number of pixels) of the imaging element 51a, etc.

[0163] In addition to image discomfort caused by blurring of an image displayed on a display unit due to camera shake or the like, image discomfort is also caused by a display delay time.

[0164] Specifically, a case will be described in which the photographer swings the camera from left to right while maintaining a state in which the photographer looks at the EVF 4 and then returns to the camera from right to left.

[0165] Figure 4 The movement of the imaging device 1, the scenery in front of the imaging device 1, and the corresponding changes over time of the display image displayed on the EVF monitor 7 of the imaging device 1 are schematically shown. Figure 4 The images Ri1, Ri2, Ri3 and Ri4 in FIG. 1 are respectively the scenery in front of the imaging device 1. In addition, Figure 4 The images Mi1, Mi2, Mi3 and Mi4 in FIG. 1 are display images displayed on the EVF monitor 7, respectively.

[0166] First, it is assumed that the imaging device 1 moves from left to right from time t1 to time t2 , and moves from right to left from time t2 to time t3 .

[0167] In this case, the subject appears to move relatively from right to left from time t1 to time t2 in the scenery in front of the imaging device 1. That is, in the image Ri1 (the image immediately after time t1) and the image Ri2 (the image at the time point of time t2) seen by the observer, the subject is displayed on the left side in the image Ri2.

[0168] Image Ri3 shows the scenery in front of the imaging device 1 when a short time has passed from time t2. Since image Ri3 is in a state where the imaging device 1 has moved slightly to the left compared to image Ri2, the subject on the image has moved slightly to the right. Furthermore, in image Ri4 immediately before time t3 has passed, the subject has moved to the right compared to image Ri3.

[0169] The image Ri2 is an image in which the subject is located at the far left in a series of movements of the imaging device 1 .

[0170] On the other hand, in the display image displayed on the EVF monitor 7, when comparing the image Mi2 when the time t2 has passed and the image Mi3 when a short time has passed from the time t2, the subject in the image Mi3 has moved to the left. That is, due to the display delay time, the timing at which the subject is located at the far left arrives with a slight delay. Therefore, in the time when the image Mi2 and the image Mi3 are displayed, the movement of the imaging device 1, that is, the movement of the photographer's head and the movement of the subject visually recognized through the EVF monitor 7 are opposite to each other. Specifically, although the imaging device 1 moves from right to left, the subject in the EVF monitor 7 looks as if the subject is not moving from left to right but moving in the opposite direction (from right to left).

[0171] Since the actual visual information (the subject moves from right to left) deviates from the expected movement of the subject (from left to right) based on the movement of the photographer's face (from right to left), there is a concern that the photographer may feel uneasy and cause visually induced motion discomfort.

[0172] Such image discomfort may occur not only due to the movement of the imaging device 1 in the left-right direction but also due to yaw, movement in the up-down direction, pitch, movement of the imaging device 1 in the Z-axis direction, and roll.

[0173] The imaging device 1 performs various types of processing to reduce or prevent such image discomfort.

[0174] Hereinafter, these various types of processing will be described with reference to the drawings.

[0175] <3. Generation of Modified Display Image Data>

[0176] <3-1. Calculation of modification amount>

[0177] The signal processing unit 52 performs modification processing (signal processing) to reduce image discomfort caused by the display delay time. For example, the modification processing is performed so that the subject is displayed as being shifted compared to the case where the display image has been displayed on the display unit without the modification processing. In this case, the shift amount is defined as the modification amount in the modification processing (including the movement amount due to translation, the movement amount due to enlargement and reduction, and the movement amount due to rotational movement).

[0178] Reference Figure 4 It is described that the position (image Ri2) of the scene (subject) actually directly visually recognized at time t2 deviates from the position (image Mi2) of the subject displayed on the EVF monitor 7 at that point in time, and the position of the subject displayed on the EVF monitor 7 is displayed on the right side compared with the actual scene.

[0179] Therefore, by performing display processing (modification processing) for shifting the position of the subject displayed on the EVF monitor 7 to the left, the deviation between the actual position of the subject and the position of the subject displayed on the EVF monitor 7 can be reduced (or eliminated).

[0180] In order to perform such display processing, the imaging device 1 of the present embodiment calculates the amount of change in the position or posture (orientation) of the imaging device 1 , and calculates the shift amount from the amount of change.

[0181] In order to calculate the change amount of the position or posture of the imaging device 1, it is necessary to determine the length of the display delay time. For example, when the frame rate is 60 Hz and the display delay is 3 frames, the display delay time is 50 milliseconds (=1 second / 60 Hz)×3 frames).

[0182] That is, under this condition, the amount of change in the position or posture of the imaging device 1 during 50 milliseconds is calculated, and the amount of shift of the display image data is calculated based on the amount of change.

[0183] First, calculation of the movement amount and change in the posture of the imaging device 1 in space will be described.

[0184] In the following description, the change amount of the position or posture of the imaging device 1 and the change amount of the position or posture of the imaging element 51a are considered to be the same. It can be considered that the actual change amount of the imaging element 51a is measured by setting various sensors for measuring the change amount near the imaging element 51a.

[0185] Reference Figure 5 A case where the imaging device 1 has been translated by a distance Δx1 in the X-axis direction (left-right direction) will be described. Figure 5 The imaging element 51a and its focal center FC are shown.

[0186] When the imaging device 1 has moved a distance Δx1 in the X-axis direction, the movement amount of the imaging element 51a also becomes Δx1. When the imaging device 1 moves in a predetermined direction in the X-axis direction, Δx1 has a positive value, and when the imaging device 1 moves in a direction opposite to the predetermined direction, Δx1 has a negative value.

[0187] Next, we will refer to Figure 6 Describe the change in the posture of the imaging element 51a when the imaging device 1 rotates (yaws) around the Y axis. The change in the posture of the imaging element 51a due to the yaw of the imaging device 1 can be approximated as a translation in the X-axis direction of the imaging device 1. When the size of the imaging element 51a in the X-axis direction (i.e., the width of the imaging element 51a) is X, the viewing angle of the imaging element 51a in the X-axis direction is θy, and the rotation angle of the yaw is Δθy, the approximate movement amount Δx2 in the X-axis direction in this case can be calculated by x×Δθy / θy.

[0188] The approximate equation has a positive value in the case of rotation in a predetermined direction, and has a negative value in the case of rotation in a direction opposite to the predetermined direction.

[0189] In summary, the translation and yaw of the imaging device 1 in the X-axis direction can be regarded as the translation of the imaging element 51 a in the X-axis direction, and the movement amount thereof is calculated by Δx1+x×Δθy / θy.

[0190] Next, we will refer to Figure 7 The case where the imaging device 1 has been translated by a distance Δy1 in the Y-axis direction (up-down direction) is described. In this case, the movement amount of the imaging element 51a is also Δy1. When the imaging device 1 moves in a predetermined direction in the Y-axis direction, Δy1 has a positive value, and when the imaging device 1 moves in a direction opposite to the predetermined direction, Δy1 has a negative value.

[0191] Next, we will refer to Figure 8Describe the change in posture of the imaging element 51a when the imaging device 1 rotates (pitches) around the X-axis. The change in posture of the imaging element 51a due to the pitch of the imaging device 1 can be approximated as a translation of the imaging device 1 in the Y-axis direction. In this case, the approximate movement amount Δy2 in the Y-axis direction can be calculated by y×Δθx / θx, where y is the size of the imaging element 51a in the Y-axis direction (i.e., the height of the imaging element 51a), θx is the viewing angle of the imaging element 51a in the Y-axis direction, and Δθx is the rotation angle of the pitch.

[0192] The approximate equation has a positive value in the case of rotation in a predetermined direction, and has a negative value in the case of rotation in a direction opposite to the predetermined direction.

[0193] In summary, the translation and pitch of the imaging device 1 in the Y-axis direction can be regarded as the translation of the imaging element 51 a in the Y-axis direction, and the movement amount thereof is calculated by Δy1+y×Δθx / θx.

[0194] As described above, the translation, yaw, and pitch in the X-axis direction and the Y-axis direction in the imaging device 1 can shift the display image data in the X-axis direction and the Y-axis direction, so that the deviation of the display image based on the display delay time can be reduced or eliminated. Calculating the shift amount requires information about the optical system 50. For example, the shift amount can be calculated using information about the focal length, information about the viewing angle, etc.

[0195] Next, we will refer to Fig. 9 A case where the imaging device 1 has been translated in the Z-axis direction, that is, a case where the imaging device 1 has moved toward or away from the subject is described.

[0196] Fig. 9 The imaging element 51a is shown as being moved from the position indicated by the alternate long and short dashed lines to the photographer. That is, when viewed from the imaging element 51a, the subject OB is moving away.

[0197] When the imaging element 51a moves, the focus center FC also moves to the focus center FC'. The change ratio of the size of the area where the subject light on the imaging element 51a is gathered by the movement of the imaging element 51a can be calculated by z / (z+Δz), where z is the distance between the focus center FC and the subject OB before the movement, and Δz is the movement distance of the imaging element 51a in the z-axis direction (the movement distance of the focus center). Fig. 9 , a height H1 of the image of the subject OB formed on the imaging element 51 a before the movement and a height H2 of the image formed on the imaging element 51 a after the movement are shown.

[0198] Therefore, by multiplying the display image data by z / (z+Δz) using the display image data captured before the imaging element 51 a moves, the size of the object OB after the movement can be expressed.

[0199] Therefore, for the translation of the imaging device 1 in the Z-axis direction, the display delay caused by the display delay time can be eliminated by enlarging or reducing the display image data (for example, applying digital zoom again). The enlargement ratio or reduction ratio can be referred to as the modification amount used in the modification process for generating the modified display image data.

[0200] Finally, refer to Fig.10 A description is given of a change in the posture of the imaging element 51 a when the imaging device 1 is rotated about the Z axis.

[0201] The rolling of the imaging device 1 can be regarded as the same rotation of the imaging element 51a. The rolling of the imaging element 51a can be supported by the rotation of the X-axis and the Y-axis. That is, the display delay caused by the display delay time can be eliminated using the following equation.

[0202] [Mathematical formula 1]

[0203]

[0204] The rotation angles of the X-axis and the Y-axis may be referred to as modification amounts used in a modification process for generating modified display image data.

[0205] As described above, changes in the position and attitude of the imaging device 1 are regarded as translations in the X-axis, Y-axis, and Z-axis directions, and yaw, pitch, and roll rotations, making it possible to eliminate display delays due to display delay time.

[0206] <3-2. First Example of Modified Display Image Data>

[0207] Specifically, an example of modified display image data generated by the signal processing unit 52 in order to eliminate a display delay caused by a display delay time when the imaging device 1 is translated in the X-axis direction will be described with reference to the drawings.

[0208] Fig.11 An example of a landscape to be imaged by the imaging device 1 is shown. Fig.11 The dashed rectangular line shown represents the current viewing angle of the imaging device 1 .

[0209] Fig.12 The angle of view in current imaging according to the current angle of view in imaging by the imaging device 1 is shown, and display image data corresponding thereto is displayed on the EVF monitor 7 after, for example, about 50 milliseconds due to a display delay time.

[0210] Fig.13 is the display image data (without modification processing) displayed on the EVF monitor 7 of the imaging device 1, and is an image corresponding to the angle of view at the position of the imaging device 1 50 milliseconds ago. That is, the angle of view is changed from Fig.11 The dotted line in the figure indicates that the current viewing angle is deviated to the right.

[0211] Fig.14 1 shows the modified display image data generated by the processing of the signal processing unit 52 of the imaging device 1, and according to the viewing angle of the imaging device 1 50 milliseconds ago (see Fig.13 )The captured display image data is shifted left.

[0212] Specifically, a state in which the origin position (eg, upper left pixel) in the data within the trimming range of the captured image data does not match the origin position in the display image data is a shifted state.

[0213] In addition, the "cropping range" is the image range that is cropped in the image stabilization process when electronic image stabilization is applied. When electronic image stabilization is not applied (including when optical image stabilization is adopted), the image range is set to a predetermined image range (e.g., an effective pixel range).

[0214] However, since the display image data is shifted and the modified display image data is generated, the area of ​​a part of the modified display image data is defined as an area that cannot be filled with the captured image data within the cropping range in the imaging element 51a. In the following description, the area in which the captured image data within the cropping range of the imaging element 51a is displayed in the display area of ​​the display unit is referred to as the "first area AR1", and the other area is referred to as the "second area AR2".

[0215] exist Fig.14 In the example shown, the second area AR2 is filled with a black image.

[0216] Pixel data with a specific brightness other than the black image (white, gray, etc.) can be used for the second area AR2. In addition, the pixel data with a specific brightness can be, for example, pixel data such as a stripe pattern (in which pixel data with a first specific brightness and pixel data with a second specific brightness are repeated for each specific pixel).

[0217] Since the state (position, size, etc.) of the subject at the time when the imaging device 1 actually captures the subject is different from the state (position, size, etc.) of the subject at the time when the imaging device 1 actually captures the subject Fig.14 The modified display image data shown is displayed so that the state of the subject displayed on the display unit such as the EVF monitor 7 substantially matches, and the photographer is unlikely to cause discomfort with the image.

[0218] <3-3. Second Example of Modified Display Image Data>

[0219] The individual pixels of the imaging element 51a are classified into pixels located within the cropping range and pixels located outside the cropping range. The captured image data of the pixels located outside the cropping range are not used for the display image data to be displayed on the display unit unless the captured image data belongs to the cropping range newly set during the image stabilization processing.

[0220] In the second example of the modified display image data, captured image data generated by using pixels located outside the trimming range is used for a part of the second area AR2.

[0221] Specifically, this Fig.15 As shown in Fig.15 As shown, captured image data acquired from pixels in an area that is not originally used by correcting barrel distortion (ie, captured image data outside the trimming range) is used for the portion of the second area AR2.

[0222] By using the captured image data outside the cropping range for at least a portion of the second area AR2, the area where the pixel data with a specific brightness is used can be reduced. In addition, using the captured image data captured by the imaging element 51a, the display area where the pixel data with a specific brightness is used can be reduced without making the photographer feel uncomfortable.

[0223] <3-4. Third Example of Modified Display Image Data>

[0224] The third example of the modified display image data is an example in which the area in the display unit where the display image data is displayed is smaller than the entire display area of ​​the display unit. That is, the captured image data captured by the pixels within the cropping range of the imaging element 51a is displayed as display image data of a size smaller than the entire display area of ​​the display unit.

[0225] This will refer to Fig.16 Specific description.

[0226] A part of the display area of ​​the EVF monitor 7 as a display unit is a first area AR1, and the remaining part is a second area AR2.

[0227] Even when all captured image data within the trimming range is used as display image data, the first area AR1 is an area smaller than the entire display area of ​​the EVF monitor 7 .

[0228] When a deviation of the display position due to the display delay time occurs due to the movement of the imaging device 1, the signal processing unit 52 generates modified display image data in which the display image data has been shifted by performing the above-mentioned modification processing, etc. In this case, the display area of ​​the captured image data within the trimming range is set to be smaller than the entire display area of ​​the EVF monitor 7, so that even if the modified display image data is shifted to a certain extent relative to the display image data, all the captured image data within the trimming range fits in the display area of ​​the EVF monitor 7 (see Fig.17 ). That is, this can be achieved by changing the display position of the display image data based on the captured image data on the display unit.

[0229] Therefore, since the entire captured image can be visually recognized (the entire viewing angle can be seen), easy shooting can be ensured.

[0230] <3-5. Fourth Example of Modified Display Image Data>

[0231] In the fourth example of the modified display image data, an icon (guide frame) indicating an area to be photographed when the photographer presses a photographing button is displayed on the display unit.

[0232] This will refer to Fig.17 and 18 Specific description.

[0233] Fig.17 The first area AR1 shown in FIG. 1 is an area in which an image resulting from the subject light focused on the imaging element 51a is displayed not long ago (for example, 50 milliseconds ago). Fig.17 When the capture button is pressed or the like at the time of the state shown, the area of ​​the image (or moving image) to be actually recorded is an area different from the first area AR1.

[0234] In this example, the area is indicated by a dotted line. Specifically, the signal processing unit 52 Fig.17 The modified display image data shown is superimposed with a dotted rectangular line indicating the imaging area IA to generate Fig.18 The modified display image data is shown.

[0235] By displaying on the EVF monitor 7 Fig.18 The modified display image data shown above allows the photographer to determine the range in which shooting can be performed based on the current position or posture of the imaging device 1.

[0236] That is, captured image data just captured is shifted from the center portion and displayed to prevent image discomfort, and a shooting range in which shooting is possible can be determined according to the current state of the imaging device 1 .

[0237] <3-6. Fifth Example of Modified Display Image Data>

[0238] The fifth example of the modified display image data is an example of generating the modified display image data based on the information of the optical system 50. For example, information about the focal length of the optical system 50 is acquired, and the future shift amount is calculated based on the focal length. Therefore, for example, when a wide-angle lens is used as the optical system 50, the angle of view is wide, and thus the shift amount becomes small (see Fig.19 ). On the other hand, when a telephoto lens is used as the optical system 50, the angle of view is narrow and the amount of shift becomes large (see Fig. 20 ).

[0239] When the shift amount is too large in the case of using a telephoto lens, there is a concern that the first area AR1 displayed on the EVF monitor 7 becomes a small area too close to one end of the display area, and it is difficult for the photographer to see the first area AR1. Therefore, an upper limit value may be set for the shift amount so that the first area AR1 does not become smaller than a predetermined area. For example, a configuration may be made to determine the shift amount so that the area of ​​the second area AR2 is smaller than the area of ​​the first area AR1.

[0240] This makes it possible to prevent image discomfort while eliminating visual difficulties.

[0241] <3-7. Sixth Example of Modified Display Image Data>

[0242] The sixth example of the modified display image data is an example using the imaging device 1A including the sub-imaging unit 9 .

[0243] Specifically, the configuration of the imaging apparatus 1A will be described.

[0244] The imaging device 1A includes a sub-imaging unit 9 in addition to a camera housing 2, a lens housing 3, an EVF 4, and a rear monitor 5. The camera housing 2 is provided with various operators 6 (see Fig.21 and 22 ).

[0245] The sub-imaging unit 9 includes a lens group 11 , an imaging element, and the like, which are arranged in a housing 10 .

[0246] Fig.23 is a block diagram of the imaging apparatus 1A.

[0247] The imaging device 1A includes, in addition to the optical system 50, the imaging element unit 51, the signal processing unit 52, the control unit 53, the memory 54, the lens drive driver 55, the user interface (UI) 56, the sensor 57, the storage unit 58 and the communication unit 59, a sub-optical system 50A, a sub-imaging element unit 51A and a sub-lens drive driver 55A for implementing the imaging operation of the sub-imaging unit 9.

[0248] The sub-optical system 50A includes various lenses (e.g., an incident end lens, a zoom lens, a focusing lens, and a condenser lens), an aperture mechanism, etc. The aperture mechanism performs exposure control by, for example, adjusting the aperture amount of a lens or a diaphragm (aperture) so that sensing is performed in a state where the signal charge is not saturated and is within a dynamic range.

[0249] The sub-imaging element unit 51A includes a sub-imaging element 51Aa.

[0250] The sub-imaging element unit 51A includes, for example, a CCD type or CMOS type sub-imaging element 51Aa. The sensor surface of the sub-imaging element 51Aa includes a sensing element in which a plurality of pixels are two-dimensionally arranged.

[0251] The sub-imaging element unit 51A performs CDS processing, AGC processing, etc. on the electric signal obtained by photoelectrically converting the light received by the sub-imaging element 51Aa, and performs A / D conversion processing. Then, the captured image data as digital data is output to the signal processing unit 52 in the subsequent stage.

[0252] In the description of this example, imaging using the optical system 50 and the imaging element unit 51 is defined as imaging using the main imaging unit, and imaging using the sub optical system 50A and the sub imaging element unit 51A is defined as imaging using the sub imaging unit 9 .

[0253] The maximum viewing angle of the sub-imaging unit 9 is made larger than the maximum viewing angle of the main imaging unit. In addition, the number of pixels of the sub-imaging element 51Aa may be smaller than the number of pixels of the imaging element 51a.

[0254] The sixth example of the modified display image data is an example in which captured image data captured by the sub-imaging unit 9 is used in the second area AR2.

[0255] Reference Fig.24 Describe this in detail.

[0256] Fig.24 A state in which a display based on modified display image data has been performed is shown, wherein captured image data (first image data) captured by the main imaging unit is used for the first area AR1, and captured image data (second image data) captured by the sub-imaging unit 9 is used for the second area AR2.

[0257] like Fig.24 As shown, the captured image data with a low resolution is used for the second area AR2.

[0258] When the change in the position or posture of the imaging device 1A is equal to or less than a certain level, the second area AR2 is an area located at the edge of the display area of ​​the display unit, so even if the image is slightly rough, the photographer does not feel uncomfortable. In addition, since the pixel data with a certain brightness is a display image closer to the scenery actually captured by the eye than the pixel data for the second area AR2, the photographer can further visually recognize the picture without feeling uncomfortable.

[0259] Specifically, first, the signal processing unit 52 of the imaging device 1A performs processing for determining the range of the first area AR1. Next, the area for the second area AR2 in the second image data is specified in consideration of, for example, the difference between the angle of view of the main imaging unit and the angle of view of the sub-imaging unit 9. In addition, resolution conversion of the image data in the area is performed so as to generate display image data for display in the second area AR2 in the display unit, and modified display image data is generated from the display image data and the first image data to be displayed in the first area AR1.

[0260] Thus, generation of modified display image data for display in a display unit such as the EVF monitor 7 is performed.

[0261] Since the second area AR2 is an area located at the edge of the display area of ​​the display unit, the boundary of the first image data displayed in the first area AR1 and the second image data displayed in the second area AR2 may not be completely aligned. That is, the area for the second area AR2 in the second image data can be roughly specified. Therefore, the amount of calculation can be reduced.

[0262] <3-8. Seventh Example of Modified Display Image Data>

[0263] The seventh example of the modified display image data is an example of generating the modified display image data by using the captured image data for the first area AR1 as the display image data of the second area AR2.

[0264] The details will be described based on the accompanying drawings.

[0265] Fig.25 The captured image data within the trimming range serving as the image data to be displayed in the first area AR1 is shown to be folded back and expanded outward along the upper, lower, left and right edges of the range. Fig.25 The image data shown is referred to as "expanded image data".

[0266] Fig.26 The image data generated by performing blur processing on the expanded image data is shown. This image data is referred to as "blurred expanded image data".

[0267] This example is to Fig.26 The blurred expanded image data shown is an example of an image displayed in the second area AR2 of the modified display image data.

[0268] Fig. 27 The state in which the image within the trimming range in the captured image data has been shifted according to the change over time in the position or posture of the imaging device 1 as in the above-mentioned examples is shown. When the modified display image data is generated and displayed in this state, the second area AR2 in the display image displayed on the display unit such as the EVF monitor 7 is interpolated using pixel data having a specific brightness.

[0269] Fig.28 A portion of blur-expanded image data generated from captured image data is shown. Fig. 27 The position of the first area AR1 in the image data determines the area to be clipped from the fuzzy expanded image data. Specifically, the clipping Fig.26 The area of ​​the part in Fig.26 The position of the folding line in the blurred expanded image data (the folding line when the expanded image data is generated) is Fig. 27 The boundaries of the clipping range overlap.

[0270] Fig.29 Shown by using Fig. 27 The first area AR1 and Fig.28 The modified display image data generated in the second area AR2.

[0271] like Fig.29 As shown, by using the data within the cropping range in the captured image data captured by the imaging device 1 for the second area AR2, the discomfort of the photographer when visually recognizing the modified display image data can be reduced. In addition, since the change in brightness value or color tone becomes gentle when the object is set relative to the folding line, the discomfort of the photographer can be further reduced. In addition, the image data for the second area AR2 is blurred, so that the discomfort of the photographer can be further reduced.

[0272] The image data for the second area AR2 may be unblurred. Fig.25 The image data shown.

[0273] Furthermore, the image data for the second area AR2 may not be a blurred image in which the blurred expanded image data is out of focus, but may be low-resolution image data generated by performing a resolution conversion process on the expanded image data.

[0274] <4. Flowchart>

[0275] The steps of generating modified display image data will be described using the flowchart in the drawings.

[0276] For example, by Fig.30 The main processing and Fig.31 The subroutine processing shown implements a series of processing.

[0277] The signal processing unit 52 starts exposure of each pixel of the imaging element 51 a of the imaging element unit 51 by performing exposure start processing in step S101 .

[0278] In step S102, the signal processing unit 52 sends a movement amount measurement start trigger to the control unit 53. The control unit 53 receives the trigger and executes Fig.31 Each process of the movement amount measurement subroutine shown is described in detail below.

[0279] The signal processing unit 52 ends the exposure of each pixel of the imaging element 51a by performing exposure end processing in step S103. The timing of performing step S103 is determined according to the time elapsed since the processing of step S101 was performed. The elapsed time is determined according to various parameters, such as the imaging mode or the International Organization for Standardization (ISO) sensitivity.

[0280] Figure 3 The exposure time T4 shown in C is the time required from the execution of step S101 to the execution of step S103.

[0281] In step S104, the signal processing unit 52 performs readout of the electric signal. Specifically, this is a process of reading out the electric signal obtained by photoelectric conversion of each pixel of the imaging element 51a.

[0282] Figure 3 The read time T5 shown in C is the time required to execute the process of step S104.

[0283] In step S105, the signal processing unit 52 performs development processing. The development processing is a process of performing various signal processing on the captured image data. Figure 3 The development time T6 shown in C is the time required to execute the process of step S105.

[0284] In step S106, the signal processing unit 52 sends a movement amount measurement end trigger to the control unit 53. The control unit 53 receiving the trigger performs a process of sending the movement amount (change amount) measured in the subroutine process to the signal processing unit 52. The details will be described below.

[0285] In step S107, the signal processing unit 52 performs a process of acquiring the movement amount calculated by the control unit 53. In this example, although the signal processing unit 52 acquires the movement amount when the control unit 53 performs the transmission process, the signal processing unit 52 may acquire the movement amount by transmitting an acquisition request to the control unit 53.

[0286] In step S108, the signal processing unit 52 performs modification processing based on the acquired movement amount. This modification processing is processing for generating the above-mentioned various types of modified display image data. In addition to the modification processing in step S108, the signal processing unit 52 also performs correction processing for correcting distortion of a lens or the like.

[0287] In step S109 , the signal processing unit 52 performs a process of displaying the modified display image data on the display unit.

[0288] Next, we will refer to Fig.31 A subroutine for measuring the movement amount of the imaging device 1 is described.

[0289] The control unit 53 performs a process of determining whether a movement amount measurement start trigger is received. For example, the movement amount measurement start trigger is received from the signal processing unit 52.

[0290] When the movement amount measurement start trigger has not been received, the control unit 53 performs the process of step S201 again. Furthermore, when the control unit 53 determines that the movement amount measurement start trigger has been received, the control unit 53 proceeds to step S202 and subsequent steps.

[0291] although Fig.31 An example is shown in which the process of step S201 is repeatedly executed until a movement amount measurement start trigger is received, the imaging device 1 may be configured such that the subroutine starts when a movement amount measurement start trigger has been received and the corresponding processes of step S202 and subsequent steps are executed.

[0292] In step S202, the control unit 53 performs resetting of the movement amount integrated value. That is, the movement amount (or change in posture) of the imaging device 1 can be measured with this point in time as a reference.

[0293] In step S203 , the control unit 53 acquires outputs of various sensors such as the acceleration sensor 60 and the angular velocity sensor 61 , and performs integration processing to obtain the movement amount in step S204 .

[0294] Furthermore, the control unit 53 confirms (determines) whether a movement amount measurement end trigger has been received from the signal processing unit 52 in step S204 , and executes the readout processing of step S203 and the integration processing of step S204 again when the movement amount measurement end trigger has not been received.

[0295] The processing of steps S203 and S204 is repeatedly performed so that an appropriate movement amount integrated value corresponding to the movement amount or posture change amount of the imaging device 1 is calculated for each frame.

[0296] When the control unit 53 determines in step S205 that the movement amount measurement end trigger has been received from the signal processing unit 52 , in step S206 , the control unit 53 transmits the movement amount integrated value as the movement amount to the signal processing unit 52 .

[0297] <5. Display Position of Superimposed Display Image>

[0298] For display image data (or modified display image data) for displaying an image on each display unit in the imaging device 1, not only captured image data but also data of images of various icons and the like (hereinafter referred to as "superimposed display image") are used.

[0299] For example, Fig.32 The superimposed display image in the displayed image data shown consists of various icons for the current date and time, the settings of the imaging device 1, the remaining battery level, etc. arranged along the upper and lower ends, a face recognition mark indicating that the face of the subject is being recognized, a stripe pattern indicating an area where the brightness value is equal to or greater than a set value ( Fig.32 ), AF mark (focus aiming), target frame, etc.

[0300] Fig.33 There is shown an example of modified display image data generated so that a subject is shifted and displayed in accordance with the position or change of the imaging device 1. The modified display image data has a first area AR1 and a second area AR2.

[0301] The superimposed display images include a first type of superimposed display image SP1 in which the display position has been shifted in accordance with the displaced and displayed subject, and a second type of superimposed display image SP2 in which the display position has been fixed regardless of the shift of the subject.

[0302] Specifically, a superimposed display image related to the position of the subject such as the remaining battery level or the shooting mode is a first type of superimposed display image SP1 , and a superimposed display image not related to the position of the subject is a second type of superimposed display image SP2 .

[0303] according to Fig.32 and 33 As an example, a face recognition mark, a focus mark, a stripe pattern regarding brightness value, etc. are the first type superimposed display image SP1. Also, icons for the current date and time, settings of the imaging device 1, a remaining battery level, etc. are the second type superimposed display image SP2.

[0304] <6. Selection of Modified Display Image Data and Non-Modified Display Image Data>

[0305] It is conceivable that the above-mentioned various modification processes are applied only according to specific circumstances. Specifically, the control unit 53 (or the signal processing unit 52) ​​can select whether to perform output of modified display image data that has undergone modification processing or to perform output of non-modified display image data that has not undergone modification processing according to the shooting state. The control unit 53 (or the signal processing unit 52) ​​functions as a display image data selection unit that selects modified display image data or non-modified display image data.

[0306] For example, the shooting state may be different from the shooting mode, may be the movement amount of the imaging device 1 (movement amount or acceleration in each axial direction, angular velocity in the axial direction, etc.), or may be the setting of the output destination of captured data.

[0307] For example, by performing the above-mentioned various modification processes when monitoring still image capture, the modified display image data can be displayed on the display unit. That is, the modified display image data can be selected as the display image data displayed on the display unit when monitoring still image capture.

[0308] In still image capturing, the photographer can rotate the imaging device 1 to find a shooting composition (angle of view) by moving the imaging device 1 up, down, left, and right while observing the EVF monitor 7 of the imaging device 1. In addition, when the subject is an animal or a person, a motion of quickly moving the imaging device 1 can be conceived so as not to miss the shooting timing.

[0309] In this case, when there occurs a deviation between the movement of his or her face and the movement of the display image displayed on the display unit for visual recognition, image discomfort may occur.

[0310] Therefore, by performing the above-described modification process when monitoring still image capture, image discomfort can be reduced or prevented.

[0311] It is effective to perform the above-mentioned modification processing when the display unit visually recognized by the photographer is the EVF monitor 7 provided in the EVF 4. When the photographer is looking at the EVF monitor 7, the photographer is very likely to move his / her face while visually recognizing only the display image displayed on the EVF monitor 7. In this case, when the above-mentioned deviation occurs, it is very likely to cause image discomfort, and therefore, by displaying the modified display image data so that the photographer visually recognizes the image in which the deviation has been reduced, the image discomfort can be reduced.

[0312] Furthermore, among the display image data to be displayed on the rear monitor 5, even if it can be inferred that it is the time of monitoring still image capture, normal display image data to which the modification process is not applied may be displayed (selected). In the case of visually recognizing the rear monitor 5, it is likely that the photographer directly visually recognizes the surrounding background together with the rear monitor 5. In this case, since image discomfort is unlikely to be caused by the display image data displayed on the rear monitor 5, the modification process may not be performed.

[0313] In monitoring still image capture, signal processing is applied to a wide range in the imaging element 51a to generate captured image data, thereby increasing the possibility that the second area AR2 is not generated even when the subject is shifted and displayed. Therefore, a reduction in visibility due to the second area AR2 can be avoided.

[0314] Furthermore, it may be configured so that the above-described modification processing is not performed when capturing a moving image. That is, non-modified display image data may be selected as display image data displayed on the display unit when capturing a moving image.

[0315] It is conceivable that an attempt is made to capture a moving image without blurring so that the moving image is not difficult to see when the moving image file recorded when the moving image is captured is viewed later. Therefore, there is less possibility of image discomfort due to not displaying the modified display image data, so the modification process may not be performed.

[0316] Furthermore, there is a request to determine the angle of view actually recorded when capturing a moving image, and therefore, when displaying the modified display image data, the angle of view of the moving image actually recorded does not match the angle of view displayed on the display unit, which may be inappropriate.

[0317] If the modified display image data is displayed while tracking a moving subject while the moving subject is kept in the angle of view, there is a concern that the target subject is displayed outside the center of the display unit, making tracking difficult.

[0318] The photographer can select display image data on which the modification process has not been performed or modified display image data on which the modification process has been performed as an image to be displayed on the display unit.

[0319] When an image captured by the imaging device 1 is displayed on a display unit (e.g., a fixed external monitor) provided outside the imaging device 1 through an external output, normal display image data that has not been subjected to modification processing may be displayed (selected). When a viewer views the captured image through a display unit such as an external monitor, the movement of the imaging device 1 is not synchronized with the movement of the viewer's face, and therefore, there is a concern that when the modified display image data is displayed, it is difficult to see the modified display image data. In particular, when the second area AR2 becomes too wide, that is, when the first area AR1 becomes too narrow, there is a high concern that it will be difficult to see it. In this case, it is preferable to display the display image data to which the modification processing is not applied. In addition, the display image data to be displayed on the display unit provided outside the imaging device 1 may be display image data to which various corrections for distortion caused by the optical system or image stabilization have been applied.

[0320] Preferably, the moving image file to be recorded is treated as unmodified data.

[0321] When viewing a recorded moving image file, it is difficult for a face to move in sync with the motion of the imaging device 1 at the time of shooting, so it is preferable to display the normal display image data without modification processing. In addition, in order to make the moving image easier to see, image data that has been subjected to processing such as image stabilization alone may be displayed.

[0322] The configuration for performing the generation of modified display image data that has undergone the modification process as described above can be applied not only to the imaging device 1 but also to a head-mounted display (HDM) having a camera, etc. For example, the configuration can also be applied to augmented reality (AR) in which an image such as a landscape captured by a camera is used. In the HDM, the wearer only views the display image data displayed on the HDM. In this case, when there is a difference between the movement of the wearer's head and the movement of the display image, the possibility of causing image discomfort is high, and therefore, it is useful to display the modified display image data.

[0323] The image to be displayed on the HMD worn by medical staff when performing medical actions through remote operation is also set to modified display image data, thereby reducing the deviation between the position of the treatment target part in the patient and the displayed image, making it possible to prevent treatment errors due to display delays.

[0324] <7. Modification Example>

[0325] Some examples of display delay time will be described.

[0326] exist Figure 3In the description of C, the sum of the exposure time T4, the readout time T5, the development time T6, and the display time T7 is described as the display delay time (Example 1 of the display delay time).

[0327] The light exposed by the imaging element 51a during the exposure time T4 includes the light reflected from the subject captured at the exposure start time point at the angle of view or the light reflected from the subject captured at the angle of view immediately before the end of the exposure. That is, when the subject moves during the exposure time T4, the same subject has an influence on multiple pixels.

[0328] Taking this into consideration, when considering the display delay time of the pixel on which the subject light has an influence at the last moment in the exposure time T4, the readout time T5, the development time T6 and the display time T7 can be considered (Example 2 of the display delay time).

[0329] In addition, when considering the display delay time of the average pixel position of the subject in the exposure time T4, the delay time starting from the time point when half of the exposure time T4 has passed can be considered, and therefore, 50% of the exposure time T4, the readout time T5, the development time T6, and the display time T7 can be considered (Example 3 of the display delay time).

[0330] That is, the display delay time may be the sum of T4, T5, T6 and T7, may be the sum of half of T4, T5, T6 and T7, or may be the sum of T5, T6 and T7 (see Fig.34 ).

[0331] The display delay time may be measured by the imaging device 1 or may be calculated based on the setting of the shooting mode, etc. When the display delay time is calculated according to various settings, the display delay time may be acquired by referring to, for example, a table indicating the correspondence between the settings and the display delay time.

[0332] <8. Conclusion>

[0333] As described in the above examples, the imaging device 1 (imaging device 1A) according to the present technology includes: an imaging element 51a (sub-imaging device 51Aa); a delay time acquisition unit (signal processing unit 52), which acquires the display delay time of the captured image based on the exposure time T4 of the imaging element 51a (sub-imaging element 51Aa) and the development processing time (development time T6) required for the development processing; and an image data generation unit (signal processing unit 52), which generates modified display image data, in which the image display delay caused by the display delay time is reduced using the change amount (movement amount) in the position or posture of the imaging device body during the display delay time.

[0334] For example, when the captured image data captured by the imaging element 51a is converted into display image data for display on the display unit (EVF monitor 7, etc.), the modified display image data in which the image display delay is reduced is display image data shifted in the up-down direction and the left-right direction. Alternatively, this is display image data after enlargement or reduction.

[0335] By generating the modified display image data in consideration of not only the time required for exposure but also the time required for development processing, it is possible to perform image display with little delay (or to offset delayed image display to some extent) with respect to the movement of the imaging device 1. Therefore, the photographer can visually recognize the display image with little discomfort with respect to the movement of the imaging device 1.

[0336] The imaging device 1 may include a display unit (EVF monitor 7 or the like) on which the modified display image data is displayed, as described in the configuration of the imaging device 1 .

[0337] The display unit is, for example, an EVF monitor 7 provided inside an electronic viewfinder (EVF) or a rear monitor 5 attached to the rear surface of the imaging device.

[0338] Through the modified display image based on the modified display image data displayed on the display unit of the imaging device 1, the photographer himself can visually recognize the modified display image following the movement of the photographer holding the imaging device 1. That is, since the modified display image matching the movement of the photographer can be visually recognized, the image discomfort of the photographer can be reduced.

[0339] As described in the configuration of the imaging device 1 , the display unit (EVF monitor 7 ) may be provided in the electronic viewfinder (EVF 4 ).

[0340] A housing 8 that covers the EVF monitor 7 may be provided around the EVF 4 .

[0341] When the photographer shoots while looking at such an EVF monitor 7, the photographer may be in a state in which the photographer can visually recognize only the display image displayed on the EVF monitor 7. That is, the display image displayed on the rear monitor 5 and the directly visible surrounding scenery are visually recognized in the rear monitor 5, while the surrounding scenery cannot be directly visually recognized in the EVF monitor 7, and only the display image is visually recognized.

[0342] In this case, since the movement of the imaging device 1 is substantially synchronized with the movement of the photographer's face, image discomfort is likely to occur when the movement of the imaging device 1 does not match the movement of the display image of the EVF monitor 7. According to this configuration, modified display image data in which the image display delay due to the display delay time is shortened is generated and displayed on the EVF monitor 7, so that the photographer visually recognizes the display image after the facial movement, thereby being able to reduce the possibility of the occurrence of image discomfort.

[0343] As described in each example of generating modified display image data, the image data generating unit (signal processing unit 52) ​​can perform modification processing for generating modified display image data so that the display position of the subject on the display unit is shifted compared to a case where the modification processing is not performed.

[0344] By performing modification processing so that the display position of the subject is shifted compared with a case where the modification processing is not performed, modified display image data can be generated, and display following the movement of the imaging device 1 can be performed.

[0345] Therefore, since the movement of the imaging device 1 and the movement of the display image are easily synchronized, the occurrence of image discomfort can be reduced.

[0346] As described in the example of selecting modified display image data and non-modified display image data, the image data generating unit (signal processing unit 52) ​​can generate non-modified display image data in which the image display delay caused by the display delay time is not reduced, and the display image data sent to the display unit outside the imaging device 1 can be non-modified display image data.

[0347] The display unit external to the imaging device 1 is, for example, a stationary television receiver, a liquid crystal display device, or a portable smartphone.

[0348] That is, this case is, for example, a case where display image data to be visually recognized by a user who is not synchronized with the movement of the imaging device 1 can be generated. When display image data is generated from captured image data captured by the imaging device 1 so that such a user views an image, non-modified display image data is generated, rather than modified display image data in which an image display delay due to a display delay time is reduced. There is no need to generate modified display image data because, for a user who is not synchronized with the movement of the imaging device 1, image discomfort caused by displaying a display image that does not match the movement of the imaging device 1 does not occur.

[0349] The non-modified display image data is display image data on which modification for reducing the image display delay due to the display delay time has not been performed, and does not mean display image data that has not been corrected at all. That is, the non-modified display image data may be display image data on which correction for smoothing up / down / left / right motion for easy viewing (e.g., correction for removing high-frequency components) or other correction for ensuring easy visibility of the display image has been performed. Of course, the distortion caused by the lens may be corrected.

[0350] As described in the example of selection of modified display image data and non-modified display image data, the image data generating unit (signal processing unit 52) ​​may include a display image data selecting unit (control unit 53 or signal processing unit 52) ​​that generates non-modified display image data in which the image display delay caused by the display delay time is not reduced, and selects modified display image data or non-modified display image data as display image data to be displayed on the display unit according to the shooting status.

[0351] The shooting state is, for example, a still image capture mode or a moving image capture mode (i.e., the type of shooting mode). Alternatively, the shooting state may be the amount of movement, acceleration, or angular velocity of the imaging device in the up-down direction and the left-right direction. In addition, the shooting state may be the output destination of the display image data.

[0352] The modified display image data is display image data that has been modified so that even when the photographer who performs shooting while moving the face visually recognizes the display image data, image discomfort is prevented. However, as described above, there is a problem of image size reduction. On the other hand, for example, the non-modified display image data is display image data to which correction is applied so that the user who is not synchronized with the movement of the imaging device 1 visually recognizes the display image data. For the user who is not synchronized with the movement of the imaging device 1, it is easy to visually recognize such non-modified display image data. In addition, the image size does not decrease.

[0353] A display image suitable for the situation can be displayed by the imaging device 1 including the control unit 53 (or the signal processing unit 52 ) serving as a display image data selection unit that selects modified display image data or non-modified display image data.

[0354] As described in the example of selection of modified display image data and non-modified display image data, when a mode for capturing moving images is selected as a capturing state, the display image data selection unit (signal processing unit 52 or control unit 53) can select non-modified display image data.

[0355] In the moving image capturing mode, it is conceivable that the photographer suppresses the movement amount or acceleration change per unit time of the imaging device 1 in the up-down direction and the left-right direction in order to capture an easy-to-view moving image.

[0356] Since the captured image data captured in this case is likely to be stable and image discomfort is unlikely to occur, the non-modified display image data is selected and displayed on the display unit instead of the modified display image data. Since there is a concern that the modified display image data has a narrow display area as described above, the non-modified display image data is selected so that the user can see that a wide range of the display image is captured therein.

[0357] Furthermore, in the moving image capturing mode, since there is a request from the photographer who wishes to confirm the recorded image itself on the display unit, the request can be met by selecting non-modified display image data.

[0358] As described in the example of selection of modified display image data and non-modified display image data, when the mode for capturing a still image is selected as the shooting state, the display image data selection unit (signal processing unit 52 or control unit 53) selects modified display image data.

[0359] In the still image capturing mode, it is conceivable that the photographer determines the angle of view so that the subject desired to be photographed is in the configuration desired to be photographed, and moves the imaging device 1 to fit in the angle of view. In addition, in this case, it is conceivable to quickly move the imaging device 1 so as not to miss the best shooting timing considered by the photographer.

[0360] That is, in this case, for example, the display image data displayed on the EVF monitor 7 is highly likely to cause image discomfort. Since the display image data selection unit selects the modified display image data when the still image capturing mode is selected, the photographer visually recognizes the display image based on the modified display image data in which the image display delay has been reduced, thereby suppressing image discomfort.

[0361] As described in the fifth example of modified display image data, the image data generation unit (signal processing unit 52 ) generates modified display image data based on information on the incident optical system (optical system 50 ) of the imaging element 51 a .

[0362] The information on the incident optical system of the imaging element 51 a is, for example, information on the type of lens such as a telephoto lens or a wide-angle lens, or a set value of an aperture.

[0363] For example, the range (angle of view) imaged by the imaging element 51a is different between the telephoto lens and the wide-angle lens. Therefore, since the amount of movement (difference amount) of the projection position of the subject on the imaging element 51a when the imaging device 1 is moved is also different, the amount of shift used to generate the modified display image data is also different. Since image display matching the movement of the imaging device 1 can be performed by generating the modified display image data using information of the incident optical system, the effect of reducing image discomfort can be further enhanced.

[0364] As described in the first example of the modified display image data, the image data generation unit (signal processing unit 52 ) may use pixel data having a specific brightness for an area of ​​the modified display image data that is not captured by the imaging element 51 a .

[0365] The pixels of the display image displayed based on the modified display image data may include pixels in which captured image data within the cropping range of the imaging element 51a is displayed (the above-mentioned first area AR1), pixels in which captured image data outside the cropping range of the imaging element 51a is displayed, and pixels outside the range of the imaging element 51a, that is, pixels in which images within the range not captured by the imaging element 51a are to be displayed.

[0366] According to the present configuration, pixel data having a specific brightness is used at least for the area not imaged by the imaging element 51a. The specific pixel data is, for example, a black pixel, a white pixel, or a pixel pattern (a stripe pattern, etc.) repeated for each specific pixel.

[0367] Therefore, it is possible to perform image display matching the movement of the imaging device 1 while displaying any image on each pixel in the display unit.

[0368] As described in the second example of the modified display image data, the image data generation unit (signal processing unit 52 ) may use the captured image data outside the trimming range for the modified display image data.

[0369] The captured image data in the clipping range refers to an area clipped as lens distortion pixels or a range of pixels effective in processing such as image stabilization in the entire pixel area of ​​the imaging element 51a exposed to the subject light. That is, the captured image data outside the clipping range refers to an area (unused area) other than the area clipped as lens distortion pixels or a range of pixels effective in processing such as image stabilization.

[0370] According to the present configuration, the area using pixel data having a specific brightness can be reduced, for example, by using captured image data outside the cropping range for a portion of the modified display image data, and the modified display image data can be generated without discomfort. That is, modified display image data in which the modification processing is not noticeable can be generated.

[0371] The captured image data outside the cropping range can be used for all the second areas AR2 generated by the shift of the display image data, so that it is not necessary to use pixels having a specific brightness (eg, black pixels).

[0372] As described in the sixth example of the modified display image data, a second imaging element (sub-imaging element 51Aa) is included that generates second imaging element data based on incident light incident through a second incident optical system (sub-optical system 50A), and the image data generating unit (signal processing unit 52) ​​can use the second captured image data for an area of ​​the modified display image data that is not captured by the imaging element 51a.

[0373] For example, a sub-imaging device (sub-imaging unit 9 ) including a second incident optical system (sub-optical system 50A) and a second imaging element (sub-imaging element 51Aa) is included.

[0374] By using the data captured by the sub-imaging device (sub-imaging unit 9) for the area not imaged by the imaging element 51a in the modified display image data, it is possible to generate modified display image data with extremely low discomfort. Therefore, it is possible to make image discomfort difficult to occur and reduce the display area using pixel data with a specific brightness.

[0375] The sub-imaging device (sub-imaging unit 9) preferably has a wider angle than the range imaged by the main imaging element 51a of the present imaging device 1. Therefore, the area not imaged by the imaging element 51a can be appropriately filled based on the captured image data captured by the second imaging element (sub-imaging element 51Aa) of the sub-imaging element (sub-imaging element 9).

[0376] In addition, since the captured image data captured by the second imaging element (sub-imaging element 51Aa) is used at the edge of the display area of ​​the display unit, the resolution may be low. Even when the captured image data captured by the second imaging element (sub-imaging element 51Aa) has a low resolution, it is difficult for the photographer to feel discomfort in the displayed image because the captured image data is used only for a part of the area between several frames.

[0377] As described in the seventh example of modified display image data, the image data generating unit (signal processing unit 52 ) may use the captured image data in the trimming range for the area of ​​the modified display image data that is not captured by the imaging element 51 a .

[0378] That is, the modified display image data is generated using the captured image data within the trimming range.

[0379] For example, the captured image data displayed within the cropping range in the first area AR1 is folded back to the second area AR2 side so as to be line-symmetrical about the boundary line between the first area AR1 and the second area AR2 and displayed. Thus, it is possible to generate modified display image data that is more comfortable than filling the second area AR2 with, for example, a black image.

[0380] As described in the section about the display position of the superimposed display image, the image data generating unit (signal processing unit 52) ​​can determine the superimposed position of at least a portion of the superimposed display image (first type superimposed display image SP1) superimposed and displayed on the captured image data based on the displacement amount of the subject.

[0381] In addition, the superimposed display images (first type superimposed display image SP1 and second type superimposed display image SP2) are icon images for indicating a shooting mode, a display icon indicating the remaining number of shots, an image icon indicating a battery charge amount, an icon indicating a focus position, an image icon indicating the position of a recognized person's face, etc. Alternatively, for example, a histogram indicating brightness distribution is also included in the superimposed display image.

[0382] Since there is no deviation between the display image actually displayed on the display unit and the superimposed display image indicating a specific area (or position) on the image when, for example, a portion of the superimposed display image is shifted in the same manner according to the shift amount calculated in the process of generating the modified display image data and displayed, image display suitable for the user or photographer can be performed.

[0383] As described in the section on the display position of the superimposed display image, the image data generating unit (signal processing unit 52) ​​can adjust the display position of the superimposed display image according to the focus aiming ( Fig.18 The superimposed position is determined by the shift amount of the rectangular dotted line (indicates the imaging area IA).

[0384] For example, In Focus is a superimposed display image that represents a specific location on the image.

[0385] When the position of the subject on the display unit is shifted and displayed, the in-focus position may be similarly shifted and displayed so that the correct in-focus position can be indicated.

[0386] As described in the third example of the modified display image data, the area where the captured image is displayed on the display area of ​​the display unit may be smaller than the display area.

[0387] For example, a display image according to the captured image data is displayed in a part (for example, a part surrounding the center part) of the entire display area of ​​the display unit.

[0388] Therefore, even when the area displaying the captured image data is offset in any direction relative to the central portion in the modified display image data, the area displaying the captured image data is easily fitted in the entire display area of ​​the display unit, and therefore, a display image displaying a sufficient range can be presented to the photographer or user. In addition, since a part of the captured image data is not outside the display area, the photographer can determine the width of the viewing angle of the captured image.

[0389] As described in the fourth example of the modified display image data, the image data generation unit (signal processing unit 52 ) may perform generation of display image data on which a guide frame indicating an output area has been superimposed.

[0390] The guide frame indicating the output area is, for example, a guide frame indicating the recording area.

[0391] For example, when the area displaying captured image data is offset in any direction relative to the center portion, it is unlikely that the photographer can determine which area is currently recorded. Superimposing and displaying a guide frame enables the photographer to determine the area currently recorded and take an appropriate shot.

[0392] The effects described in this specification are merely examples and are not limiting, and other effects may be obtained.

[0393] <9. Application Example 1>

[0394] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be applied to an operating room system.

[0395] Fig.35 Schematically shows the overall configuration of an operating room system 5100 to which the technology according to the present disclosure can be applied. Fig.35 , the operating room system 5100 is configured by connecting a group of devices installed in the operating room so that these devices can cooperate with each other via an audio-visual controller (AV controller) 5107 and an operating room control device 5109.

[0396] A variety of equipment can be installed in the operating room. Fig.35 In the figure, as examples, various equipment groups 5101 for endoscopic surgery, a ceiling camera 5187 set on the ceiling of the operating room to image the surgeon's hands, an operating room camera 5189 set on the ceiling of the operating room to image the state of the entire operating room, multiple display devices 5103A to 5103D, a recorder 5105, a bed 5183 and lighting 5191 are illustrated.

[0397] Here, among these devices, the device group 5101 belongs to the endoscopic surgery system 5113 to be described below, and includes an endoscope, a display device that displays an image captured by the endoscope, and the like. Each device belonging to the endoscopic surgery system 5113 is also referred to as a medical device. On the other hand, the display devices 5103A to 5103D, the recorder 5105, the bed 5183, and the lighting 5191 are devices that are set up in an operating room, for example, separately from the endoscopic surgery system 5113. Each of these devices that do not belong to the endoscopic surgery system 5113 is also referred to as a non-medical device. The audio-visual controller 5107 and / or the operating room control device 5109 control the operations of the medical devices and the non-medical devices in cooperation with each other.

[0398] The audio-visual controller 5107 comprehensively controls the processing of image display in medical devices and non-medical devices. Specifically, the device group 5101, the ceiling camera 5187, and the operating room camera 5189 included in the devices in the operating room system 5100 can be devices (hereinafter also referred to as source devices) having the function of sending information to be displayed during surgery (hereinafter also referred to as display information). In addition, the display devices 5103A to 5103D can be devices that output display information (hereinafter also referred to as output destination devices). In addition, the recorder 5105 can be a device corresponding to both the source device and the output destination device. The audio-visual controller 5107 has the function of controlling the operation of the source device and the output destination device to obtain display information from the source device, and sending the display information to the output destination device for display or recording. The display information includes various images captured during surgery, various types of information about surgery (e.g., patient's physical information, past examination results, information about the surgical process), etc.

[0399] Specifically, information about an image of a surgical site in a patient's body cavity captured by an endoscope may be transmitted from the device group 5101 to the audio-visual controller 5107 as display information. In addition, the ceiling camera 5187 may transmit information about an image at the surgeon's hand captured by the ceiling camera 5187 as display information. In addition, the operating room camera 5189 may transmit information about displaying an image showing the state of the entire operating room captured by the operating room camera 5189 as display information. When the operating room system 5100 includes another device having an imaging function, the audio-visual controller 5107 may acquire information about an image captured by another device as display information from the other device.

[0400] Or, for example, information about these images captured in the past is recorded on the recorder 5105 by the audio-visual controller 5107. The audio-visual controller 5107 may acquire information about images captured in the past as display information from the recorder 5105. Various types of information about the surgery may be recorded in the recorder 5105 in advance.

[0401] The audiovisual controller 5107 causes at least one of the display devices 5103A to 5103D as output destination devices to display the acquired display information (i.e., images captured during surgery or various types of information about surgery). In the example shown, the display device 5103A is a display device suspended and mounted from the ceiling of the operating room, the display device 5103B is a display device mounted on the wall of the operating room, the display device 5103C is a display device mounted on the table of the operating room, and the display device 5103D is a mobile device having a display function (e.g., a tablet personal computer (PC)).

[0402] In addition, despite the Fig.35 Although not shown in the figure, the operating room system 5100 may include devices outside the operating room. The devices outside the operating room may be, for example, a server connected to a network constructed inside and outside the hospital, a PC used by medical staff, and a projector installed in a hospital conference room. When such external devices are outside the hospital, the audio-visual controller 5107 may also display display information on a display device of another hospital through a video conferencing system for telemedicine, etc.

[0403] The operating room control device 5109 comprehensively controls the processing except the processing related to the image display in the non-medical equipment. For example, the operating room control device 5109 controls the driving of the bed 5183, the ceiling camera 5187, the operating room camera 5189 and the lighting 5191.

[0404] The operating room system 5100 has a centralized operation panel 5111 , and the user can issue instructions regarding image display to the audio-visual controller 5107 or issue instructions regarding operation of non-medical equipment to the operating room control device 5109 via the centralized operation panel 5111 .

[0405] Fig.36 51 is a diagram showing a display example of an operation screen in the centralized operation panel 5111. Fig.36 , an operation screen corresponding to a case where the operating room system 5100 has two display devices as output destination devices is shown as an example. Fig.36 , the operation screen 5193 has a source selection area 5195, a preview area 5197 and a control area 5201.

[0406] Source devices included in the operating room system 5100 and thumbnail screens showing display information owned by the source devices are linked and displayed in source selection area 5195. The user can select display information desired to be displayed on the display device from any of the source devices displayed in source selection area 5195.

[0407] A preview of the screen displayed on the two display devices (monitor 1 and monitor 2) as the output destination devices is displayed in the preview area 5197. In the example shown, four images are displayed in PinP on one display device. The four images correspond to the display information sent from the source device selected in the source selection area 5195. One of the four images is displayed relatively large as the main image, and the remaining three images are displayed relatively small as sub-images. The user can switch between the main image and the sub-image by appropriately selecting the area where the four images have been displayed. In addition, a status display area 5199 is set below the area where the four images are displayed, and the status of the operation (for example, the elapsed time of the operation and the patient's physical information) is appropriately displayed in this area.

[0408] The control area 5201 has a source operation area 5203, in which a graphical user interface (GUI) component for performing operations on the source device is displayed, and an output destination operation area 5205, in which a GUI component for performing operations on the output destination device is displayed. In the example shown, the source operation area 5203 is provided with GUI components for performing various operations (panning, tilting and zooming) for the camera in the source device with an imaging function. The user can operate the operation of the camera in the source device by appropriately selecting these GUI components. Although not shown, when the source device selected in the source selection area 5195 is a recorder (i.e., when an image recorded on the recorder in the past is displayed in the preview area 5197), the source operation area 5203 can be provided with GUI components for performing operations such as playing, stopping, rewinding, and fast-forwarding images.

[0409] In addition, the output destination operation area 5205 is equipped with GUI components for performing various operations (swapping, flipping, color adjustment, contrast adjustment, switching between 2D display and 3D display) on the display on the display device as the output destination device. The user can operate the display on the display device by appropriately selecting these GUI components.

[0410] The operation screen displayed on the centralized operation panel 5111 is not limited to the illustrated example, and the user can perform operation input via the centralized operation panel 5111 on each device controllable by the audiovisual controller 5107 and the operating room control device 5109 included in the operating room system 5100 .

[0411] Fig.37 5187 and the operating room camera 5189 are provided on the ceiling of the operating room, and can photograph the hand of the surgeon (doctor) 5181 who treats the affected part of the patient 5185 on the bed 5183 and the state of the entire operating room. The ceiling camera 5187 and the operating room camera 5189 may have a magnification adjustment function, a focus adjustment function, a shooting direction adjustment function, and the like. The lighting 5191 is provided on the ceiling of the operating room and illuminates at least the hand of the surgeon 5181. In the lighting 5191, the amount of irradiated light, the wavelength (color) of the irradiated light, the irradiation direction of the light, and the like may be appropriately adjusted.

[0412] The endoscopic surgery system 5113, the bed 5183, the ceiling camera 5187, the operating room camera 5189, and the lighting 5191 are connected so that they can be controlled via the audio-visual controller 5107 and the operating room control device 5109 ( Fig.37 ) cooperate with each other, such as Fig.35 The centralized operation panel 5111 is provided in the operating room, and as described above, the user can appropriately operate these devices present in the operating room through the centralized operation panel 5111.

[0413] Hereinafter, the configuration of the endoscopic surgery system 5113 will be described in detail. As shown in the figure, the endoscopic surgery system 5113 includes an endoscope 5115, other surgical tools 5131, a support arm device 5141 supporting the endoscope 5115, and a cart 5151 in which various endoscopic surgery devices are installed.

[0414] In endoscopic surgery, multiple tubular laparotomy tools called sleeves 5139a to 5139d are inserted into the abdominal wall instead of cutting the abdominal wall and opening the abdomen. The lens barrel 5117 of the endoscope 5115 or other surgical tools 5131 are inserted into the body cavity of the patient 5185 from the sleeves 5139a to 5139d. In the example shown, the pneumoperitoneum tube 5133, the energy treatment tool 5135 and the forceps 5137 are inserted into the body cavity of the patient 5185 as other surgical tools 5131. In addition, the energy treatment tool 5135 is a treatment tool such as cutting and stripping tissue, sealing blood vessels using high-frequency current or ultrasonic vibration. However, the surgical tool 5131 shown is only an example, and various surgical tools (such as forceps and retractors) commonly used in endoscopic surgery can be used as surgical tools 5131.

[0415] An image of a surgical site in a body cavity of a patient 5185 captured by an endoscope 5115 is displayed on a display device 5155. The surgeon 5181 performs a surgery such as resecting an affected portion using an energy therapy tool 5135 or forceps 5137 while viewing in real time the image of the surgical site displayed on the display device 5155. Although not shown, the pneumoperitoneum tube 5133, the energy therapy tool 5135, and the forceps 5137 are supported by the surgeon 5181 or an assistant during surgery.

[0416] (Support arm device)

[0417] The support arm device 5141 includes an arm 5145 extending from a base 5143. In the example shown, the arm 5145 is composed of connecting parts 5147a, 5147b and 5147c and connecting rods 5149a and 5149b, and is driven by the control of the arm control device 5159. The endoscope 5115 is supported by the arm 5145, and its position and posture are controlled. Thus, stable fixation of the position of the endoscope 5115 can be achieved.

[0418] (Endoscope)

[0419] The endoscope 5115 is composed of a lens barrel 5117 having a predetermined length from the tip thereof to be inserted into the body cavity of the patient 5185, and a camera head 5119 connected to the bottom end of the lens barrel 5117. Although the endoscope 5115 configured as a so-called rigid scope having a rigid lens barrel 5117 is shown in the illustrated example, the endoscope 5115 may be configured as a so-called flexible scope having a flexible lens barrel 5117.

[0420] An opening for mounting an objective lens is provided at the tip of the lens barrel 5117. A light source device 5157 is connected to the endoscope 5115, and light generated by the light source device 5157 is guided to the tip of the lens barrel through a light pipe extending in the lens barrel 5117, and irradiated toward an observation target in the body cavity of the patient 5185 through the objective lens. The endoscope 5115 may be a direct endoscope, a fluoroscopic endoscope, or a lateral endoscope.

[0421] An optical system and an imaging element are provided inside the camera head 5119, and reflected light (observation light) from an observation target is gathered on the imaging element through the optical system. The observation light is photoelectrically converted by the imaging element, and an electrical signal corresponding to the observation light, that is, an image signal corresponding to the observation image, is generated. The image signal is sent to the camera control unit (CCU) 5153 as RAW data. The camera head 5119 has a function of adjusting the magnification and focal length by appropriately driving the optical system.

[0422] For example, the camera head 5119 may be provided with a plurality of imaging elements to support stereoscopic viewing (3D display), etc. In this case, a plurality of relay optical systems are provided within the lens barrel 5117 to guide observation light to the plurality of imaging elements.

[0423] (Various devices installed on carts)

[0424] The CCU 5153 is composed of a central processing unit (CPU), a graphics processing unit (GPU), etc., and comprehensively controls the operation of the endoscope 5115 and the display device 5155. Specifically, the CCU 5153 performs various image processing for displaying images based on image signals, for example, developing processing (demosaicing processing) of image signals received from the camera head 5119. The CCU 5153 provides the image signal after image processing to the display device 5155. In addition, Fig.35 The audio-visual controller 5107 is shown connected to the CCU 5153. The CCU 5153 also provides the audio-visual controller 5107 with an image signal that has undergone image processing. In addition, the CCU 5153 sends a control signal to the camera head 5119 and controls its driving. The control signal may include information about imaging conditions, such as magnification or focal length. The information about imaging conditions may be input via the input device 5161, or may be input via the above-mentioned centralized operation panel 5111.

[0425] The display device 5155 displays an image based on an image signal that has been image-processed by the CCU 5153 under the control of the CCU 5153. When the endoscope 5115 is compatible with high-resolution shooting such as 4K (3840 horizontal pixels × 2160 vertical pixels) or 8K (7680 horizontal pixels × 4320 vertical pixels) and / or compatible with 3D display, a display device capable of high-resolution display and / or a display device capable of 3D display can be used as the display device 5155. When the endoscope is compatible with high-resolution shooting such as 4K or 8K, a display device having a size of 55 inches or more is used as the display device 5155, a further sense of immersion can be obtained. In addition, a plurality of display devices 5155 having different resolutions and sizes can be provided according to the application.

[0426] The light source device 5157 is composed of a light source such as a light emitting diode (LED), for example, and provides the endoscope 5115 with irradiation light for imaging the surgical site.

[0427] The arm control device 5159 is composed of a processor such as a CPU, and operates according to a predetermined program to control the driving of the arm portion 5145 of the support arm device 5141 according to a predetermined control scheme.

[0428] The input device 5161 is an input interface for the endoscopic surgery system 5113. The user can input various types of information or instructions to the endoscopic surgery system 5113 through the input device 5161. For example, the user inputs various types of information about the surgery, for example, the patient's body information or information about the surgical operation via the input device 5161. In addition, for example, the user inputs an instruction to drive the arm portion 5145, an instruction to change the imaging conditions (type of irradiation light, magnification, focal length, etc.) of the endoscope 5115, an instruction to drive the energy treatment tool 5135, etc. via the input device 5161.

[0429] The type of input device 5161 is not limited, and the input device 5161 may be various known input devices. As the input device 5161, for example, a mouse, a keyboard, a touch panel, a switch, a foot switch 5171, and / or a joystick may be applied. When a touch panel is used as the input device 5161, the touch panel may be provided on the display surface of the display device 5155.

[0430] Alternatively, the input device 5161 is a device worn by the user, such as a glasses-type wearable device or a head-mounted display (HMD), and various inputs are performed according to the gestures or sight lines of the user detected by these devices. In addition, the input device 5161 includes a camera capable of detecting the movement of the user, and various inputs are performed according to the gestures or sight lines of the user detected from the video captured by the camera. In addition, the input device 5161 includes a microphone capable of picking up the user's voice, and various inputs are performed by voice through the microphone. When the input device 5161 is configured to be able to input various types of information in a non-contact manner in this way, in particular, a user (e.g., a surgeon 5181) belonging to a clean area can operate a device belonging to a dirty area in a non-contact manner. In addition, the user can operate the device without removing the surgical tool he / she holds from his / her hand, which improves the convenience of the user.

[0431] The treatment tool control device 5163 controls the driving of the energy treatment tool 5135 for cauterization and cutting of tissues, sealing of blood vessels, etc. The pneumoperitoneum device 5165 sends gas to the body cavity of the patient 5185 via the pneumoperitoneum tube 5133 to inflate the body cavity of the patient 5185 so as to fix the visual field and the surgeon's work space using the endoscope 5115. The recorder 5167 is a device capable of recording various types of information about the operation. The printer 5169 is a device capable of printing various types of information about the operation in various formats such as text, images, and graphics.

[0432] Below, the specific feature configuration in the endoscopic surgical system 5113 will be described in more detail.

[0433] (Support arm device)

[0434] The support arm device 5141 includes a base portion 5143 as a base and an arm portion 5145 extending from the base portion 5143. In the illustrated example, the arm portion 5145 is composed of a plurality of joint portions 5147a, 5147b, and 5147c and a plurality of connecting rods 5149a, 5149b connected by the joint portion 5147b, but for simplicity, Fig.37 5145 is shown in a simplified manner. In practice, the shape, number and configuration of the joint portions 5147a to 5147c and the connecting rods 5149a and 5149b, the direction of the rotation axis of the joint portions 5147a to 5147c, etc. are appropriately set so that the arm 5145 has a desired degree of freedom. For example, the arm 5145 can be preferably configured to have at least 6 degrees of freedom. Therefore, since the endoscope 5115 can move freely within the movable range of the arm 5145, the lens barrel 5117 of the endoscope 5115 can be inserted into the body cavity of the patient 5185 from a desired direction.

[0435] The actuator is provided in the joint portions 5147a to 5147c, and the joint portions 5147a to 5147c are configured to be rotatable around a predetermined rotation axis by driving the actuator. By driving the actuator controlled by the arm control device 5159, the rotation angle of the joint portions 5147a to 5147c is controlled, and the driving of the arm 5145 is controlled. Thus, the control of the position and posture of the endoscope 5115 can be achieved. In this case, the arm control device 5159 can control the driving of the arm 5145 using various known control schemes such as force control or position control.

[0436] For example, when the surgeon 5181 appropriately performs an operation input via the input device 5161 (including the foot switch 5171), the arm control device 5159 can appropriately control the drive of the arm 5145 in response to the operation input, and control the position and posture of the endoscope 5115. With this control, the endoscope 5115 at the tip of the arm 5145 can be moved from any position to any position, and then the endoscope 5115 can be fixedly supported at the position after the movement. The arm 5145 can be operated by a so-called master / slave scheme. In this case, the user can remotely control the arm 5145 through the input device 5161 installed in a place away from the operating room.

[0437] In addition, when force control is applied, the arm control device 5159 can perform so-called power-assist control, in which the arm control device 5159 receives an external force from the user and drives the actuators of the respective joint portions 5147a to 5147c so that the arm portion 5145 moves smoothly according to the external force. Therefore, when the user moves the arm portion 5145 while directly contacting the arm portion 5145, the arm portion 5145 can be moved with a relatively light force. Therefore, the endoscope 5115 can be moved more intuitively with a simpler operation, and the convenience of the user is improved.

[0438] Here, generally speaking, in endoscopic surgery, the endoscope 5115 is supported by a doctor called an endoscopist. On the other hand, since the position of the endoscope 5115 can be more reliably fixed without manpower by using the support arm device 5141, an image of the surgical site can be stably obtained, so that the surgery can be performed smoothly.

[0439] The arm control device 5159 is not necessarily provided in the cart 5151. In addition, the arm control device 5159 is not necessarily one device. For example, the arm control device 5159 may be provided at each of the joint portions 5147a to 5147c of the arm portion 5145 supporting the arm device 5141, and the driving and control of the arm portion 5145 may be realized by a plurality of arm control devices 5159 cooperating with each other.

[0440] (Light source device)

[0441] The light source device 5157 provides the endoscope 5115 with illumination light when photographing the surgical site. The light source device 5157 is composed of, for example, an LED, a laser light source, or a white light source composed of a combination thereof. In this case, when the white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, and therefore, adjustment of the white balance of the captured image can be performed in the light source device 5157. In addition, in this case, the observation target is irradiated with lasers from each RGB laser light source in a time-sharing manner, and the drive of the imaging element of the camera head 5119 is controlled synchronously with its irradiation timing, so that images corresponding to each RGB can be captured in a time-sharing manner. According to this method, a color image can be obtained without providing a color filter to the imaging element.

[0442] In addition, the drive of the light source device 5157 can be controlled so that the intensity of the output light is changed at predetermined time intervals. By controlling the drive of the imaging element of the camera head 5119 in synchronization with the timing of the change in light intensity, images are acquired in a time-division manner and combined, so-called high dynamic range images can be generated without blackouts and overexposure.

[0443] In addition, the light source device 5157 can be configured to provide light of a predetermined wavelength band corresponding to special light observation. For example, in special light observation, the dependence of light absorption on wavelength in body tissue is utilized to irradiate light in a narrower band than the irradiation light (i.e., white light) in normal observation, thereby performing so-called narrow-band light observation (narrow-band imaging), in which predetermined tissues, such as blood vessels on the surface layer of a mucosa, are photographed with high contrast. Alternatively, in special light observation, fluorescence observation can be performed in which an image is obtained by irradiating fluorescence generated by excitation light. In fluorescence observation, body tissue is irradiated with excitation light and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into the body tissue, and the body tissue is irradiated with excitation light corresponding to the fluorescence wavelength of the reagent, thereby obtaining a fluorescence image. The light source device 5157 can be configured to provide narrow-band light and / or excitation light corresponding to such special light observation.

[0444] (Camera head and CCU)

[0445] Reference Fig.38 The functions of the camera head 5119 and CCU 5153 of the endoscope 5115 are described in more detail. Fig.38 It is shown Fig.37 A block diagram of an example of the functional configuration of the camera head 5119 and the CCU 5153 shown.

[0446] Reference Fig.38 , the camera head 5119 includes as its functions a lens unit 5121, an imaging unit 5123, a driving unit 5125, a communication unit 5127, and a camera head control unit 5129. In addition, the CCU 5153 includes as its functions a communication unit 5173, an image processing unit 5175, and a control unit 5177. The camera head 5119 and the CCU 5153 are bidirectionally communicatively connected via a transmission cable 5179.

[0447] First, the functional configuration of the camera head 5119 will be described. The lens unit 5121 is an optical system provided at the connection portion with the lens barrel 5117. The observation light taken from the tip of the lens barrel 5117 is guided to the camera head 5119 and incident on the lens unit 5121. The lens unit 5121 is configured as a combination of a plurality of lenses including a zoom lens and a focus lens. The optical characteristics of the lens unit 5121 are adjusted so that the observation light is gathered on the light receiving surface of the imaging element of the imaging unit 5123. In addition, the zoom lens and the focus lens are configured so that their positions on the optical axis can be moved to adjust the magnification and focus of the captured image.

[0448] The imaging unit 5123 is composed of an imaging element and is arranged in a stage after the lens unit 5121. The observation light that has passed through the lens unit 5121 is collected on the light receiving surface of the imaging element, and an image signal corresponding to the observation image is generated by photoelectric conversion. The image signal generated by the imaging unit 5123 is provided to the communication unit 5127.

[0449] For example, a complementary metal oxide semiconductor (CMOS) type image sensor having a Bayer array and capable of color shooting is used as an imaging element constituting the imaging unit 5123. For example, an imaging element capable of supporting shooting of 4K or higher resolution images can be used as an imaging element. When an image of the surgical site is obtained at a high resolution, the surgeon 5181 can determine the state of the surgical site in more detail and perform surgery more smoothly.

[0450] In addition, the imaging element constituting the imaging unit 5123 is configured to have a pair of imaging elements for acquiring respective image signals for the right eye and the left eye corresponding to the 3D display. The 3D display enables the surgeon 5181 to more accurately determine the depth of the living tissue in the surgical site. When the imaging unit 5123 is configured as a multi-plate type, a plurality of lens units 5121 are also provided corresponding to the respective imaging elements.

[0451] In addition, the imaging unit 5123 may not necessarily be provided in the camera head 5119. For example, the imaging unit 5123 may be provided immediately after the objective lens in the lens barrel 5117.

[0452] The driving unit 5125 is composed of an actuator, and moves the zoom lens and the focus lens of the lens unit 5121 by a predetermined distance along the optical axis under the control of the camera head control unit 5129. Therefore, the magnification and focus of the image captured by the imaging unit 5123 can be appropriately adjusted.

[0453] The communication unit 5127 is composed of a communication device for sending and receiving various types of information to and from the CCU 5153. The communication unit 5127 sends the image signal obtained from the imaging unit 5123 to the CCU 5153 via the transmission cable 5179 as RAW data. In this case, it is preferable to send the image signal by optical communication so as to display the captured image of the surgical site with low latency. This is because, since the surgeon 5181 performs the operation while observing the state of the affected part using the image captured during the operation, it is necessary to display the moving image of the surgical site as real time as possible for safer and more reliable surgery. When performing optical communication, the communication unit 5127 has a photoelectric conversion module that converts an electrical signal into an optical signal. The image signal is converted into an optical signal by the photoelectric conversion module and then transmitted to the CCU 5153 via the transmission cable 5179.

[0454] In addition, the communication unit 5127 receives a control signal for controlling the drive of the camera head 5119 from the CCU 5153. The control signal includes, for example, information about imaging conditions, such as information for specifying a frame rate for capturing an image, information for specifying an exposure value during imaging, and / or information for specifying a magnification and a focus of the captured image. The communication unit 5127 provides the received control signal to the camera head control unit 5129. The control signal from the CCU 5153 may also be sent via optical communication. In this case, the communication unit 5127 is provided with a photoelectric conversion module that converts an optical signal into an electrical signal, and the control signal is converted into an electrical signal by the photoelectric conversion module and then provided to the camera head control unit 5129.

[0455] The control unit 5177 of the CCU 5153 automatically sets imaging conditions such as frame rate, exposure value, magnification, and focus based on the acquired image signal. That is, the so-called automatic exposure (AE) function, automatic focus (AF) function, and automatic white balance (AWB) function are installed in the endoscope 5115.

[0456] The camera head control unit 5129 controls the driving of the camera head 5119 based on the control signal from the CCU 5153 received via the communication unit 5127. For example, the camera head control unit 5129 controls the driving of the imaging element of the imaging unit 5123 based on the information for specifying the frame rate of the captured image and / or the information for specifying the exposure at the time of imaging. In addition, for example, the camera head control unit 5129 appropriately moves the zoom lens and the focus lens of the lens unit 5121 via the driving unit 5125 based on the information for specifying the magnification and focus of the captured image. The camera head control unit 5129 may also have a function of storing information for identifying the lens barrel 5117 or the camera head 5119.

[0457] By arranging the configuration of the lens unit 5121, the imaging unit 5123, etc. in a sealed structure with high airtightness and waterproofness, the camera head 5119 can be made resistant to high-pressure sterilization.

[0458] Next, the functional configuration of the CCU 5153 will be described. The communication unit 5173 is configured as a communication device for sending various types of information to the camera head 5119 and receiving various types of information from the camera head 5119. The communication unit 5173 receives an image signal sent from the camera head 5119 through the transmission cable 5179. In this case, as described above, the image signal can be appropriately sent through optical communication. In this case, for optical communication, the communication unit 5173 has a photoelectric conversion module that converts an optical signal into an electrical signal. The communication unit 5173 provides the image processing unit 5175 with the image signal converted into the electrical signal.

[0459] Furthermore, the communication unit 5173 transmits a control signal for controlling the driving of the camera head 5119 to the camera head 5119. The control signal may be transmitted by optical communication.

[0460] The image processing unit 5175 performs various image processing on the image signal as RAW data transmitted from the camera head 5119. The image processing includes, for example, various known signal processing such as development processing, high image quality processing (e.g., band enhancement processing, super-resolution processing, noise reduction (NR) processing, and / or camera shake correction processing), and / or enlargement processing (electronic zoom processing). In addition, the image processing unit 5175 performs detection processing for performing AE, AF, and AWB on the image signal.

[0461] The image processing unit 5175 is composed of a processor such as a CPU or a GPU, and when the processor operates according to a predetermined program, the above-mentioned image processing or detection processing can be performed. When the image processing unit 5175 is composed of a plurality of GPUs, the image processing unit 5175 appropriately divides the information related to the image signal and performs image processing in parallel using the plurality of GPUs.

[0462] The control unit 5177 performs various controls regarding imaging of a surgical site using the endoscope 5115 and display of its captured image. For example, the control unit 5177 generates a control signal for controlling the drive of the camera head 5119. In this case, when the user inputs an imaging condition, the control unit 5177 generates a control signal based on the user's input. Alternatively, when the AE function, the AF function, and the AWB function are installed in the endoscope 5115, the control unit 5177 appropriately calculates the optimal exposure value, focal length, and white balance according to the result of the detection processing in the image processing unit 5175, and generates a control signal.

[0463] In addition, the control unit 5177 causes the display device 5155 to display an image of the surgical site based on the image signal subjected to image processing by the image processing unit 5175. In this case, the control unit 5177 recognizes various objects in the surgical site image using various image recognition technologies. For example, the control unit 5177 can detect the shape, color, etc. of the edge of the object included in the surgical site image in order to recognize surgical tools such as forceps, specific biological parts, bleeding, fog when using the energy treatment tool 5135, etc. When the control unit 5177 causes the display device 5155 to display the image of the surgical site, the control unit 5177 uses the result of the recognition to cause various types of surgical support information to be superimposed on the image of the surgical site and displayed. When the surgical support information is superimposed, displayed, and presented to the surgeon 5181, surgery can be performed more safely and reliably.

[0464] The transmission cable 5179 connecting the camera head 5119 to the CCU 5153 is an electric signal cable supporting electric signal communication, an optical fiber supporting optical communication, or a composite cable thereof.

[0465] Here, in the example shown, communication is performed by wire using the transmission cable 5179, but communication between the camera head 5119 and the CCU 5153 may be performed wirelessly. When communication between the camera head 5119 and the CCU 5153 is performed wirelessly, it is not necessary to install the transmission cable 5179 in the operating room, and thus, the situation in which the transmission cable 5179 hinders the movement of medical staff in the operating room can be solved.

[0466] An example of an operating room system 5100 to which the technology according to the present disclosure can be applied has been described above. Here, a case where the medical system to which the operating room system 5100 is applied is an endoscopic surgery system 5113 is described as an example, but the configuration of the operating room system 5100 is not limited to such an example. For example, the operating room system 5100 may be applied to an inspection flexible endoscope system or a microsurgery system instead of the endoscopic surgery system 5113.

[0467] The technology according to the present disclosure can be appropriately applied to the CCU 5153 in the above configuration. Specifically, the above modification processing is performed on the image signal received from the endoscope 5115 to reduce or eliminate the delay time, thereby generating modified display image data and displaying it on the display device 5155 as a display unit. By applying the technology according to the present disclosure to the CCU 5153, the treatment safety caused by the deviation between the image currently captured by the endoscope 5115 and the image displayed with delay due to the display delay time can be improved.

[0468] <10. Application Example 2>

[0469] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be implemented as a device installed in any type of mobile body such as an automobile, an electric car, a hybrid electric car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, a robot, a construction machine, an agricultural machine (tractor), etc.

[0470] Fig.39 7000 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. Fig.39In the example shown, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting the plurality of control units may be an in-vehicle communication network conforming to any standard such as a controller area network (CAN), a local interconnect network (LIN), a LAN (local area network), or FlexRay (registered trademark).

[0471] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a storage unit that stores programs executed by the microcomputer, parameters for various calculations, etc., and a drive circuit that drives various control target devices. Each control unit includes a network I / F for performing communication with other control units via the communication network 7010, and includes a communication I / F for performing communication with devices, sensors, etc. inside and outside the vehicle through wired communication or wireless communication. Fig.39 , a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690 are shown as a functional configuration of the integrated control unit 7600. Other control units also include a microcomputer, a communication I / F, a storage unit, and the like.

[0472] The drive system control unit 7100 controls the operation of devices related to the vehicle drive system according to various programs. For example, the drive system control unit 7100 is used as a control device for a drive force generating device such as an internal combustion engine or a drive motor that generates a drive force for the vehicle, a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force for the vehicle. The drive system control unit 7100 may have the function of a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), and the like.

[0473] The vehicle state detection unit 7110 is connected to the drive system control unit 7100. The vehicle state detection unit 7110 includes, for example, a gyro sensor for detecting the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor for detecting the acceleration of the vehicle, and at least one of sensors for detecting the amount of operation on the accelerator pedal, the amount of operation on the brake pedal, the steering angle of the steering wheel, the engine speed, the wheel speed, etc. The drive system control unit 7100 performs arithmetic processing using the signal input from the vehicle state detection unit 7110 to control the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0474] The body system control unit 7200 controls the operation of various devices installed in the vehicle body according to various programs. For example, the body system control unit 7200 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a reverse light, a brake light, a flasher, or a fog light. In this case, a radio wave transmitted from a portable device instead of a key or a signal of various switches may be input to the body system control unit 7200. The body system control unit 7200 receives the input of the radio wave or the signal and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0475] The battery control unit 7300 controls the auxiliary battery 7310 as a power source for driving the motor according to various programs. For example, information such as battery temperature, battery output voltage, or battery remaining capacity is input to the battery control unit 7300 from a battery device including the auxiliary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and performs temperature adjustment control on the auxiliary battery 7310, a cooling device included in the battery device, and the like.

[0476] The vehicle external information detection unit 7400 detects information outside the vehicle on which the vehicle control system 7000 is installed. For example, at least one of the imaging unit 7410 and the vehicle external information detection unit 7420 is connected to the vehicle external information detection unit 7400. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The vehicle external information detection unit 7420 includes, for example, an environmental sensor for detecting the current weather or at least one of surrounding information detection sensors for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle on which the vehicle control system 7000 is installed.

[0477] For example, the environmental sensor may be at least one of a raindrop sensor for detecting rainy weather, a fog sensor for detecting fog, a sunlight sensor for detecting sunlight, and a snow sensor for detecting snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (light detection and ranging or laser imaging detection and ranging) device. The imaging unit 7410 and the vehicle external information detection unit 7420 may be included as independent sensors or devices, or may be included as a device in which a plurality of sensors or devices are integrated.

[0478] here, Fig.40An example of the installation position of the imaging unit 7410 and the vehicle exterior information detection unit 7420 is shown. For example, the imaging units 7910, 7912, 7914, 7916, and 7918 are provided at least one of the front nose, the side mirror, the rear bumper, the rear door, and the upper part of the windshield in the cab of the vehicle 7900. The imaging unit 7910 included in the front nose and the imaging unit 7918 included in the upper part of the windshield in the cab mainly acquire images in front of the vehicle 7900. The imaging units 7912 and 7914 included in the side mirror mainly acquire images on the side of the vehicle 7900. The imaging unit 7916 included in the rear bumper or the rear door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 included in the upper part of the windshield in the vehicle cab is mainly used to detect the front vehicle, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0479] exist Fig.40 , examples of shooting ranges of the respective imaging units 7910, 7912, 7914, and 7916 are shown. Imaging range a represents the imaging range of the imaging unit 7910 disposed on the front nose, imaging ranges b and c represent the imaging ranges of the imaging units 7912 and 7914 disposed on the side mirrors, and imaging range d represents the imaging range of the imaging unit 7916 disposed on the rear bumper or the rear door. For example, when the image data captured by the imaging units 7910, 7912, 7914, and 7916 are superimposed, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0480] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided at the front, rear, side, corner, and upper portion of the windshield in the vehicle cab of the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930 provided at the front nose, rear bumper, rear door, and upper portion of the windshield in the vehicle cab of the vehicle 7900 may be, for example, LIDAR devices. These vehicle exterior information detection units 7920 to 7930 are mainly used to detect vehicles, pedestrians, obstacles, etc. in front.

[0481] Will refer again Fig.39The description continues. The vehicle external information detection unit 7400 enables the imaging unit 7410 to capture an image of the outside of the vehicle and receive the captured image data. In addition, the vehicle external information detection unit 7400 receives detection information from the connected vehicle external information detection unit 7420. When the vehicle external information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle external information detection unit 7400 sends ultrasonic waves, electromagnetic waves, etc., and receives information about the received reflected waves. The vehicle external information detection unit 7400 can perform object detection processing or distance detection processing on people, vehicles, obstacles, signs, or characters on the road surface based on the received information. The vehicle external information detection unit 7400 can perform environmental recognition processing for identifying rainfall, fog, road conditions, etc. based on the received information. The vehicle external information detection unit 7400 can calculate the distance to the object outside the vehicle based on the received information.

[0482] In addition, the vehicle exterior information detection unit 7400 can perform image recognition processing or distance detection processing for identifying people, vehicles, obstacles, signs, characters on the road surface, etc. based on the received image data. The vehicle exterior information detection unit 7400 can perform processing such as distortion correction or alignment on the received image data, and combine the image data captured by the different imaging units 7410 to generate a bird's-eye view image or a panoramic image. The vehicle exterior information detection unit 7400 can use the image data captured by the different imaging units 7410 to perform viewpoint conversion processing.

[0483] The in-vehicle information detection unit 7500 detects information inside the vehicle. For example, a driver state detection unit 7510 that detects the driver's state is connected to the in-vehicle information detection unit 7500. The driver state detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, a microphone that collects sound in the vehicle, and the like. The biosensor is set in, for example, a seat surface or a steering wheel, and detects biometric information of a passenger sitting on the seat or a driver holding the steering wheel. The in-vehicle information detection unit 7500 can calculate the driver's fatigue or concentration based on the detection information input from the driver state detection unit 7510, and can determine whether the driver is dozing off. The in-vehicle information detection unit 7500 can perform processing such as noise elimination processing on the collected audio signal.

[0484] The integrated control unit 7600 controls the overall operation in the vehicle control system 7000 according to various programs. The input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is implemented by a device such as a touch panel, a button, a microphone, a switch or a joystick, which can be input by a passenger. Data obtained by recognizing a voice input by a microphone can be input to the integrated control unit 7600. The input unit 7800 can be, for example, a remote control device using infrared or other radio waves, or can be a device that supports external connection with respect to the operation of the vehicle control system 7000, such as a mobile phone or a personal digital assistant (PDA). The input unit 7800 can be, for example, a camera, in which case the passenger can input information by gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by a passenger can be input. In addition, the input unit 7800 can include, for example, an input control circuit that generates an input signal based on information input by a passenger or the like using the input unit 7800, and outputs the input signal to the integrated control unit 7600. A passenger or the like operates the input unit 7800 to input various data to the vehicle control system 7000 or to instruct a processing operation.

[0485] The storage unit 7690 may include a read-only memory (ROM) storing various programs executed by the microcomputer and a random access memory (RAM) storing various parameters, calculation results, sensor values, etc. In addition, the storage unit 7690 may be implemented by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, an optical magnetic storage device, etc.

[0486] The general communication I / F 7620 is a general communication I / F that mediates communication with various devices present in the external environment 7750. The general communication I / F 7620 may include a cellular communication protocol, such as a global system for mobile communications (GSM; registered trademark), WiMAX (registered trademark), long term evolution (LTE; registered trademark) or LTE-A (LTE-Advanced), or other wireless communication protocols, such as a wireless LAN (also known as Wi-Fi (registered trademark)) or Bluetooth (registered trademark) implemented therein. The general communication I / F 7620 may be connected to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or a business-specific network) via a base station or an access point. In addition, the general communication I / F 7620 may be connected to a terminal (e.g., a terminal of a driver, a pedestrian, or a store, or a machine type communication (MTC) terminal) present near the vehicle using, for example, a peer-to-peer (P2P) technology.

[0487] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for a vehicle. For example, the dedicated communication I / F 7630 may be equipped with a standard protocol such as Wireless Access in Vehicular Environments (WAVE), which is a combination of a low-layer IEEE802.11p and a high-layer IEEE1609, dedicated short-range communication (DSRC), or cellular communication protocol. The dedicated communication I / F 7630 generally performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0488] The positioning unit 7640 receives, for example, a GNSS signal from a global navigation satellite system (GNSS) satellite (e.g., a GPS signal from a global positioning system (GPS) satellite), performs positioning, and generates position information including the latitude, longitude, and altitude of the vehicle. The positioning unit 7640 may specify the current position by exchanging signals with a wireless access point, or may acquire position information from a terminal having a positioning function (e.g., a mobile phone, a PHS, or a smart phone).

[0489] The beacon receiving unit 7650 receives radio waves or electromagnetic waves transmitted from a radio station installed on the road, etc., and acquires information such as the current position, traffic congestion, prohibited passage, or required time. The function of the beacon receiving unit 7650 may be included in the above-mentioned dedicated communication I / F 7630.

[0490] The in-vehicle device I / F 7660 is a communication interface that mediates the connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using a wireless communication protocol such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC), or wireless USB (WUSB). In addition, the in-vehicle device I / F 7660 can establish a wired connection, such as a universal serial bus (USB), a high-definition multimedia interface (HDMI; registered trademark), or a mobile high-definition link (MHL) through a connection terminal (not shown) (a cable can be used if necessary). The in-vehicle device 7760 may include, for example, a mobile device or wearable device of a passenger and at least one of an information device carried or attached to the vehicle. In addition, the in-vehicle device 7760 may include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with the in-vehicle device 7760.

[0491] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals and the like according to a predetermined protocol supported by the communication network 7010.

[0492] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the acquired in-vehicle and out-vehicle information, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 can perform cooperative control to realize the functions of the advanced driver assistance system (ADAS), including vehicle collision avoidance or collision mitigation, follow-up driving based on the distance between vehicles, vehicle speed maintenance driving, vehicle collision warning, lane departure warning, etc. In addition, the microcomputer 7610 can perform cooperative control by controlling the driving force generation device, the steering mechanism, the braking device, etc. based on the acquired information about the surroundings of the vehicle, for example, for the purpose of automatic driving, in which the vehicle drives autonomously without relying on the driver's operation.

[0493] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures or people based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680, and create local map information including environmental information of the current position of the vehicle. In addition, the microcomputer 7610 may predict the danger of a vehicle collision, the approach of a pedestrian, etc., or entering a closed road based on the acquired information, and generate a warning signal. For example, the warning signal may be a signal for generating a warning sound or turning on a warning light.

[0494] The audio image output unit 7670 transmits an output signal of at least one of audio and image to an output device capable of notifying information to a passenger of the vehicle or outside the vehicle in a visual or audible manner. Fig.39In the example of, an audio speaker 7710, a display unit 7720, and a dashboard 7730 are illustrated as output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be another device other than these devices, such as a wearable device such as headphones, a glasses-type display worn by a passenger, a projector, or a lamp. When the output device is a display device, the display device visually displays the results obtained by various types of processing performed by the microcomputer 7610 or information received from other control units in various formats such as text, images, tables, and graphics. In addition, when the output device is an audio output device, the audio output device converts an audio signal including reproduced audio data, acoustic data, etc. into an analog signal and outputs the audio signal in an audible manner.

[0495] exist Fig.39 In the example shown, at least two control units connected via the communication network 7010 can be integrated into one control unit. Alternatively, each control unit can be composed of multiple control units. In addition, the vehicle control system 7000 may include other control units (not shown). In addition, in the above description, another control unit may have some or all of the functions of any control unit. In other words, as long as information is sent and received via the communication network 7010, predetermined arithmetic processing can be performed by any control unit. Similarly, a sensor or device connected to any control unit can be connected to another control unit, and multiple control units can send and receive detection information to each other via the communication network 7010.

[0496] To achieve reference Figure 2 The imaging device 1 according to the present embodiment described (or referring to Fig.23 The computer program for each function of the imaging device 1A) according to the present embodiment described can be installed in any control unit or the like. In addition, a computer-readable recording medium storing such a computer program can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a flash memory. In addition, the above-mentioned computer program can be distributed via, for example, a network without using a recording medium.

[0497] In the above-mentioned vehicle control system 7000, referring to Figure 2 The imaging device 1 of the present embodiment can be applied to Fig.39The integrated control unit 7600 of the application example shown. For example, the control unit 53, the signal processing unit 52, the memory 54, and the storage unit 58 of the imaging device 1 correspond to the microcomputer 7610 and the storage unit 7690 of the integrated control unit 7600. For example, when monitoring the video outside the vehicle, the modified display image obtained by the microcomputer 7610 performing the modification processing on the captured image data acquired from the imaging unit 7410 via the in-vehicle network I / F 7680 is visually recognized, so that the image display delay caused by the display delay time can be reduced. Therefore, it is possible to prevent the situation of erroneously recognizing the outside of the vehicle due to the delay in the image display, and improve safety.

[0498] In addition, you can Fig.39 The module for integrating the control unit 7600 shown in FIG. 7600 (eg, an integrated circuit module composed of a die) is implemented in reference Figure 2 At least some components of the imaging device 1 described herein. Alternatively, reference Figure 2 The imaging device 1 described may be composed of Fig.39 The vehicle control system 7000 shown is implemented by multiple control units.

[0499] <11. This technology>

[0500] The present technology can also adopt the following configurations.

[0501] (1) An imaging device comprising:

[0502] Imaging element;

[0503] a delay time acquisition unit configured to acquire a display delay time of a captured image based on an exposure time of the imaging element and a development processing time required for a development process; and

[0504] An image data generating unit is configured to generate modified display image data in which an image display delay due to the display delay time has been shortened using an amount of change in the position or posture of the imaging device body during the display delay time.

[0505] (2) The imaging device according to (1), comprising a display unit on which the modified display image data is displayed.

[0506] (3) The imaging device according to (2), wherein the display unit is provided in an electronic viewfinder.

[0507] (4) An imaging device according to any one of (1) to (3), wherein the image data generating unit performs modification processing for generating the modified display image data so that a display position of the subject on the display unit is shifted compared to a case where the modification processing is not performed.

[0508] (5) The imaging device according to any one of (1) to (4),

[0509] wherein the image data generating unit generates non-modified display image data in which the image display delay caused by the display delay time is not shortened, and

[0510] The display image data transmitted to the display unit outside the imaging device is non-modified display image data.

[0511] (6) The imaging device according to any one of (1) to (5),

[0512] wherein the image data generating unit generates non-modified display image data in which the image display delay caused by the display delay time is not shortened, and

[0513] The imaging device includes a display image data selection unit configured to select modified display image data or non-modified display image data as display image data to be displayed on a display unit according to a shooting state.

[0514] (7) The imaging device according to (6), wherein the display image data selection unit selects the non-modification display image data when a mode for capturing a moving image is selected as the capturing state.

[0515] (8) The imaging device according to (6) or (7), wherein when a mode for capturing a still image is selected as the shooting state, the display image data selection unit selects the modified display image data.

[0516] (9) The imaging device according to any one of (1) to (8), wherein the image data generation unit performs generation of modified display image data based on information of an incident optical system for the imaging element.

[0517] (10) According to any one of (1) to (9), the image data generation unit uses pixel data having a specific brightness for a region in the modified display image data that is not imaged by the imaging element.

[0518] (11) The imaging device according to any one of (1) to (10), wherein the image data generation unit uses captured image data outside a trimming range for the modified display image data.

[0519] (12) The imaging device according to any one of (1) to (11), comprising: a second imaging element configured to generate second captured image data based on incident light incident through a second incident optical system,

[0520] The image data generation unit uses the second captured image data for a region of the modified display image data that is not imaged by the imaging element.

[0521] (13) The imaging device according to any one of (1) to (9), wherein the image data generation unit uses the captured image data within the trimming range for a region in the modified display image data that is not imaged by the imaging element.

[0522] (14) The imaging device according to (4), wherein the image data generation unit determines the superimposition position according to a shift amount of the subject relative to at least a portion of a superimposition display image to be superimposed on captured image data and displayed.

[0523] (15) The imaging device according to (14), wherein the image data generation unit determines the superimposed position according to a shift amount relative to the focus aiming imaging serving as the superimposed display image.

[0524] (16) The imaging device according to any one of (1) to (15), wherein an area in which the captured image is displayed on a display area of ​​the display unit is smaller than the display area.

[0525] (17) The imaging device according to (16), wherein the image data generation unit performs generation of display image data on which a guide frame indicating an image output region is superimposed.

[0526] (18) An imaging signal processing device comprising:

[0527] a delay time acquisition unit configured to acquire a display delay time of a captured image based on an exposure time of an imaging element included in the imaging device and a development processing time required for a development process; and

[0528] An image data generating unit is configured to generate modified display image data in which an image display delay due to the display delay time has been shortened using an amount of change in the position or posture of the imaging device during the display delay time.

[0529] (19) An imaging signal processing method, comprising:

[0530] acquiring a display delay time of a captured image based on an exposure time of an imaging element included in the imaging device and a development processing time required for a development process; and

[0531] Modified display image data is generated in which the image display delay due to the display delay time has been shortened using the amount of change in the position or attitude of the imaging device during the display delay time.

[0532] [List of symbols]

[0533] 1 Imaging equipment

[0534] 1A Imaging Equipment

[0535] 4EVF

[0536] 5. Rear Monitor

[0537] 7EVF Monitor

[0538] 50 Optical System

[0539] 50A sub-optical system

[0540] 51a Imaging element

[0541] 51Aa sub-imaging element

[0542] 52 signal processing units

[0543] 53 Control unit

Claims

1. An imaging device, comprising: Imaging element; a delay time acquisition unit configured to acquire a display delay time of a captured image based on an exposure time of an imaging element and a development processing time required for a development process; and An image data generating unit is configured to generate modified display image data in which an image display delay due to the display delay time has been shortened using an amount of change in the position or posture of the imaging device body during the display delay time. 2 . The imaging device according to claim 1 , comprising a display unit on which the modified display image data is displayed. 3 . The imaging device according to claim 2 , wherein the display unit is provided in an electronic viewfinder.

4. The imaging device according to claim 2, wherein: The image data generation unit performs a modification process for generating the modified display image data so that a display position of a subject on the display unit is shifted compared to a case where the modification process is not performed.

5. The imaging device according to claim 1, in, The image data generating unit generates non-modified display image data in which the image display delay due to the display delay time is not shortened, and The display image data transmitted to the display unit outside the imaging device is non-modified display image data.

6. The imaging device according to claim 1, in, The image data generating unit generates non-modified display image data in which the image display delay due to the display delay time is not shortened, and The imaging device includes a display image data selection unit configured to select modified display image data or non-modified display image data as display image data to be displayed on a display unit according to a shooting state.

7. The imaging device according to claim 6, wherein: When a mode for photographing a moving image is selected as the photographing state, the display image data selection unit selects non-modified display image data.

8. The imaging device according to claim 6, wherein: When a mode for capturing a still image is selected as the photographing state, the display image data selection unit selects the modified display image data. 9 . The imaging device according to claim 1 , wherein the image data generation unit performs generation of modified display image data based on information of an incident optical system for the imaging element.

10. The imaging device according to claim 1, wherein The image data generation unit uses pixel data having a specific brightness for an area that is not imaged by the imaging element in the modified display image data.

11. The imaging device according to claim 1, wherein The image data generation unit uses the captured image data outside the trimming range for the modified display image data.

12. The imaging device according to claim 1, comprising: a second imaging element configured to generate second captured image data based on incident light incident through the second incident optical system, The image data generation unit uses the second captured image data for a region of the modified display image data that is not imaged by the imaging element.

13. The imaging device according to claim 1, wherein The image data generation unit uses the captured image data within the trimming range for an area that is not imaged by the imaging element in the modified display image data.

14. The imaging device according to claim 4, wherein: The image data generation unit determines a superimposition position according to a shift amount of the subject relative to at least a portion of a superimposition display image to be superimposed on captured image data and displayed.

15. The imaging device according to claim 14, wherein The image data generating unit determines a superimposed position according to a shift amount relative to the focus aiming image serving as a superimposed display image.

16. The imaging device according to claim 2, wherein: An area on the display area of ​​the display unit where the captured image is displayed is smaller than the display area.

17. The imaging device according to claim 16, wherein: The image data generation unit performs generation of display image data superimposed with a guide frame indicating an image output area.

18. An imaging signal processing device, comprising: a delay time acquisition unit configured to acquire a display delay time of a captured image based on an exposure time of an imaging element included in the imaging device and a development processing time required for a development process; and An image data generating unit is configured to generate modified display image data in which an image display delay due to the display delay time has been shortened using an amount of change in the position or posture of the imaging device during the display delay time.

19. An imaging signal processing method, comprising: acquiring a display delay time of a captured image based on an exposure time of an imaging element included in an imaging device and a development processing time required for a development process; and Modified display image data is generated in which the image display delay due to the display delay time has been shortened using the amount of change in the position or attitude of the imaging device during the display delay time.

Images