Imaging element, camera device, image data processing method and storage medium

Abstract

An imaging element is provided with a processing circuit and a memory. The memory stores video image data obtained by photographing a subject at a first frame rate. The processing circuit performs processing based on the video image data stored in the memory. The output circuit outputs output image data based on the video image data to the outside of the imaging element at a second frame rate. The first frame rate is a frame rate higher than the second frame rate and is determined according to a strobe generation cycle, and the processing circuit detects a strobe influence avoidance moment for avoiding the influence of strobe on the photographing of the imaging element based on a plurality of frames of video image data.

Description

Technical Field

[0001] The technology of the present invention relates to an imaging element, a camera device, an image data processing method and a storage medium. Background Art

[0002] An image processing device is disclosed in Japanese Patent Gazette No. 2017-188760, which includes: a storage unit that stores pixel signals output from an imaging element; a signal processing unit that performs signal processing on the pixel signals stored in the storage unit; and a detection unit that completes detection processing on pixel signals in the same frame before the signal processing is completed by the signal processing unit.

[0003] A signal processing device is disclosed in Japanese Patent Gazette No. 2018-007210, which includes: a brightness information calculation unit, which calculates the brightness information of a camera image; a brightness reference calculation unit, which calculates a brightness reference value representing a reference brightness based on the brightness information at multiple moments; and a correction parameter calculation unit, which calculates a correction parameter for correcting the brightness of the camera image based on the brightness information and the brightness reference value.

[0004] An image processing device is disclosed in the WO2015 / 163145 publication, which comprises: an intensity ratio calculation unit, which calculates a ratio related to the intensity of a signal value at a specified position in an image captured under different exposure conditions; and a contribution calculation unit, which calculates a contribution indicating whether the intensity ratio calculated by the intensity ratio calculation unit originates from a periodic noise component or from movement.

[0005] An image sensor is disclosed in Japanese Patent Gazette No. 2017-501627, which comprises: a camera area including multiple pixels; and one or more stroboscopic detection areas, each including one or more pixels, wherein during the period when at least one pixel in the camera area captures an image, at least one of the one or more pixels in at least one stroboscopic detection area is used multiple times.

[0006] As an imaging device that applies the technology described in Japanese Patent Publication No. 2017-188760, Japanese Patent Publication No. 2018-007210, WO2015 / 163145, and Japanese Patent Table No. 2017-501627, for example, an imaging device that uses an electronic shutter and a mechanical shutter to shoot in a rolling shutter manner can be cited. In such an imaging device, for example, in a photoelectric conversion element, exposure is started in sequence for each row, and a charge signal corresponding to the exposure amount is read. Then, image data is generated based on the read charge signal, and an image represented by the generated image data is displayed on a display. Summary of the invention

[0007] Technical issues to be solved by the invention

[0008] However, when shooting with a rolling shutter method is performed, rolling deviation of the electronic shutter and rolling deviation of the mechanical shutter occur. Here, the rolling deviation of the electronic shutter refers to, for example, the time difference from the start of exposure of the first row in the photoelectric conversion element to the start of exposure of the last row. And the rolling deviation of the mechanical shutter refers to, for example, the time difference from the front curtain of the mechanical shutter passing through the first row in the photoelectric conversion element to the last row.

[0009] As an example, Fig.28 As shown in FIG. 1 , the rolling deviation of the electronic shutter of the imaging element is longer than that of the mechanical shutter. Fig.28 In the example shown, the rolling deviation of the mechanical shutter is shorter than that of the electronic shutter.

[0010] When the frequency of commercial power is 50 Hz, a light source such as a fluorescent lamp that flickers when powered by commercial power flickers at a frequency of 100 Hz. In this environment, if a rolling shutter method is used to shoot, as an example, Fig.30 As shown, in the captured image P1 obtained by shooting, line strobes appear in the vertical direction at intervals equivalent to 10 ms. In addition, when the live preview image is displayed, a plurality of line strobes appear in a flowing manner in the vertical direction.

[0011] For example, if a full-size or medium-size photoelectric conversion element is used Fig.28 As an example, the rolling shutter shooting method shown in FIG. Fig.30 As shown in FIG. 1 , four lines of strobe light appear in the captured image P1. As an example, if a photoelectric conversion element of a size smaller than full size or medium size, i.e., APS-C (Advanced Photo System type-C) size is used, Fig.28 The rolling shutter method shown in FIG. Fig.30 As shown, three lines of strobing appear in the camera image P1.

[0012] On the other hand, as an example Fig.29 As shown, in recent years, the rolling deviation time of the electronic shutter has become shorter than before, and the rolling deviation timing of the electronic shutter has become closer to the rolling deviation timing of the mechanical shutter than before.

[0013] That is, as an example Fig.28 Compared with the rolling shutter shooting method shown in Fig.29 In the rolling shutter shooting method shown in FIG. 1 , the rolling deviation time of the electronic shutter is closer to the rolling deviation time of the mechanical shutter.

[0014] Thus, if we take as an example Fig.29As an example, if you use the rolling shutter method shown in Fig.28 Compared with the rolling shutter method shown in the figure, the number of line flickers that appear in the captured image is reduced.

[0015] For example, if a full-size or medium-size photoelectric conversion element is used as an example Fig.29 The rolling shutter method shown in FIG. Fig.31 As shown in FIG. 2 , two lines of strobe light appear in the captured image P2. Fig.29 The rolling shutter method shown in FIG. Fig.31 As shown in FIG. 1 , the interval between the line strobes is longer than the length of the image area corresponding to the APS-C size in the vertical direction. Therefore, only one line strobe appears in the image area corresponding to the APS-C size in the captured image P2.

[0016] When shooting with a rolling shutter method that avoids the influence of line strobe, it is important to determine the shooting time that can avoid the influence of line strobe. However, in the technologies described in Japanese Patent Publication No. 2017-188760, Japanese Patent Publication No. 2018-007210, WO2015 / 163145, and Japanese Patent Table No. 2017-501627, although it is possible to detect the presence of line strobe, it is difficult to determine the shooting time that can avoid the influence of line strobe. And, as an example, Fig.31 As shown in FIG. 1 , when only one line strobe appears in the image area corresponding to the APS-C size imaging area in the captured image P2 , it becomes increasingly difficult to determine an imaging timing that can avoid the influence of the line strobe.

[0017] In addition, the same can be said even when shooting with a global shutter method. When shooting with a global shutter method, surface flicker is detected by a photoelectric conversion element, but in this case, in order to shoot without the influence of surface flicker, it is also important to determine the shooting time that can avoid the influence of surface flicker. When using the techniques described in Japanese Patent Publication No. 2017-188760, Japanese Patent Publication No. 2018-007210, WO2015 / 163145, and Japanese Patent Table No. 2017-501627, the presence or absence of surface flicker can be detected, but it is difficult to determine the shooting time that can avoid the influence of surface flicker.

[0018] One embodiment of the present invention provides an imaging element, an imaging device, an image data processing method, and a program that can perform imaging without the influence of stroboscopic light.

[0019] Means for solving technical problems

[0020] The first method involved in the technology of the present invention is an imaging element, which includes: a storage unit, which stores video image data obtained by photographing a subject at a first frame rate and is built into the imaging element; a processing unit, which performs processing based on the video image data stored in the storage unit and is built into the imaging element; an output unit, which outputs output image data based on the video image data to the outside of the imaging element at a second frame rate and is built into the imaging element, the first frame rate is a frame rate higher than the second frame rate and is determined based on a period of stroboscopic generation, and the processing unit detects a stroboscopic influence avoiding moment to avoid the influence of stroboscopic on the shooting of the imaging element based on multiple frames of video image data.

[0021] Therefore, the imaging element of the first aspect according to the technology of the present invention can capture images while avoiding the influence of stroboscopic light.

[0022] The second method involved in the technology of the present invention is the imaging element described in the first method, wherein the processing includes measurement detection processing, the measurement detection processing measures the brightness difference between frames of the camera image data, and detects the moment to avoid the stroboscopic effect based on the measured brightness difference, and the processing time required in the measurement detection processing is determined based on the generation cycle.

[0023] Therefore, compared with the case where the brightness difference between frames of the captured image data is not used, the imaging element of the second aspect according to the technology of the present invention can appropriately determine the processing time required for the measurement detection process.

[0024] The third mode involved in the technology of the present invention is the imaging element described in the second mode, wherein the camera image data is classified between frames into the first camera image data and the second camera image data obtained by shooting after the first camera image data, the brightness difference is the subtraction result of the brightness of the first camera image data minus the brightness of the second camera image data, and the stroboscopic effect avoiding time is the time when the positive value of the brightness difference changes to a negative value.

[0025] Therefore, the imaging element of the third aspect according to the technology of the present invention can detect the time when the image represented by the captured image data becomes brightest as the stroboscopic effect avoiding time.

[0026] A fourth aspect according to the technology of the present invention is the imaging element according to the third aspect, wherein the processing unit terminates the measurement detection process under the condition that the luminance difference changes from a positive value to a negative value twice.

[0027] Therefore, the imaging element of the fourth aspect according to the technology of the present invention can reduce the processing time required for the measurement and detection process to the necessary minimum time.

[0028] A fifth aspect according to the technology of the present invention is the imaging element according to the third aspect or the fourth aspect, wherein the second captured image data is image data acquired by capturing two or more frames after the first captured image data.

[0029] Therefore, compared with the case of using the brightness difference between adjacent frames, the imaging element of the fifth aspect involved in the technology of the present invention can suppress the influence of stroboscopic light and avoid erroneous detection of timing.

[0030] The sixth mode involved in the technology of the present invention is an imaging element described in any one of the second to fifth modes, wherein, after the measurement detection processing is performed, until a predetermined condition is satisfied, the first frame rate is set to a frame rate lower than the frame rate in the measurement detection processing.

[0031] Therefore, the imaging element of the sixth aspect according to the technique of the present invention can reduce power consumption compared to the case where the frame rate in the detection process is also applied to processes other than the detection process.

[0032] A seventh aspect according to the technology of the present invention is the imaging element according to the sixth aspect, wherein the processing unit performs the measurement detection process again when a predetermined condition is satisfied.

[0033] Therefore, the seventh aspect according to the technology of the present invention can reduce power consumption compared to the case where the measurement detection process is always executed.

[0034] The eighth aspect of the technology of the present invention is the imaging element according to the seventh aspect, wherein when a predetermined condition is satisfied and the brightness difference measured by the processing unit changes from the brightness difference measured within the processing time, the measurement detection process is performed again.

[0035] Therefore, the imaging element of the eighth aspect according to the technology of the present invention can avoid executing unnecessary measurement and detection processing.

[0036] A ninth aspect according to the technology of the present invention is the imaging element according to any one of the second aspect to the eighth aspect, wherein the brightness difference is a brightness difference between local areas corresponding to each other between frames of captured image data.

[0037] Therefore, compared with the case where the brightness difference is measured for all image areas between frames of captured image data, the ninth method involved in the technology of the present invention can measure the brightness difference with less influence of the movement of the subject and / or hand shaking.

[0038] A tenth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the ninth aspect, wherein the stroboscopic influence avoiding timing is a timing at which the brightness of an image represented by the captured image data reaches a peak.

[0039] Therefore, the imaging element of the tenth aspect according to the technology of the present invention can obtain a brighter image compared to the case where a time different from the time when the brightness of the image represented by the captured image data reaches the peak is set as the stroboscopic influence avoiding time.

[0040] The 11th mode involved in the technology of the present invention is an imaging element according to any one of the 1st to 8th modes, wherein the output image data is image data based on camera image data obtained by capturing at a time when the influence of stroboscopic light is avoided.

[0041] Therefore, the imaging element of the eleventh aspect according to the technology of the present invention can output output image data in which the influence of stroboscopic light is avoided.

[0042] A twelfth aspect of the technology of the present invention is an imaging element according to any one of the first to eleventh aspects, wherein the strobe is line strobe, and the subject is photographed within an imaging area selected according to a cropping factor determined based on a strobe influence avoidance timing.

[0043] Therefore, compared with the case where imaging is performed in the entire imaging area, the imaging element of the twelfth aspect according to the technology of the present invention can suppress the strobe from being reflected in the image.

[0044] A 13th aspect of the technology of the present invention is the imaging element according to any one of the 1st to 12th aspects, wherein the generation cycle is preset to be a generation cycle of stroboscopic light caused by a light source flickering by supplying AC power from a commercial power source.

[0045] Therefore, compared with a case where the strobe generation cycle is not determined in advance, the imaging element of the thirteenth aspect according to the technology of the present invention can reduce the man-hours for determining the strobe generation cycle.

[0046] A fourteenth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the thirteenth aspect, wherein the first frame rate increases as the generation cycle decreases.

[0047] Therefore, compared with the case where the first frame rate is fixed, the fourteenth aspect according to the technology of the present invention can improve the detection accuracy of the stroboscopic influence avoidance timing.

[0048] A fifteenth aspect according to the technology of the present invention is the imaging element according to any one of the first aspect to the fourteenth aspect, wherein the captured image data is image data obtained by capturing an image of a subject using a rolling shutter method.

[0049] Therefore, the fifteenth aspect according to the technology of the present invention can avoid the influence of strobe light generated when photographing a subject using a rolling shutter method.

[0050] A sixteenth aspect of the technology of the present invention is the imaging element according to any one of the first aspect to the fifteenth aspect, wherein the imaging element is a stacked-type imaging element having a photoelectric conversion element, and the storage unit is stacked on the photoelectric conversion element.

[0051] Therefore, compared with the case of using an imaging element of the type in which the storage unit is not stacked on the photoelectric conversion element, the imaging element of the sixteenth aspect involved in the technology of the present invention can improve the detection accuracy of the stroboscopic influence avoidance timing.

[0052] The 17th mode involved in the technology of the present invention is a camera device, which includes: an imaging element described in any one of the 1st to 16th modes; and a control unit that controls a display unit to display an image based on the output image data output by the output unit included in the imaging element.

[0053] Therefore, the imaging device according to the seventeenth aspect of the present invention can perform imaging while avoiding the influence of stroboscopic light.

[0054] The 18th method involved in the technology of the present invention is an image data processing method, which is an image data processing method of an imaging element, wherein the imaging element is built-in with: a storage unit, which stores video image data obtained by photographing a subject at a first frame rate; a processing unit, which processes the video image data stored in the storage unit; and an output unit, which outputs output image data based on the video image data to the outside at a second frame rate, wherein the first frame rate is set to a frame rate higher than the second frame rate, and the first frame rate is determined according to a stroboscopic generation cycle, and the processing unit detects a stroboscopic influence avoidance moment to avoid the influence of stroboscopic on the shooting of the imaging element based on a plurality of frames of video image data.

[0055] Therefore, the image data processing method of the eighteenth aspect according to the technology of the present invention can perform imaging while avoiding the influence of stroboscopic light.

[0056] The 19th method involved in the technology of the present invention is a storage medium that stores a program for causing a computer to function as a processing unit and an output unit included in an imaging element, wherein the imaging element is built-in with: a storage unit that stores video image data obtained by photographing a subject at a first frame rate; a processing unit that performs processing based on the video image data stored in the storage unit; and an output unit that outputs output image data based on the video image data to the outside at a second frame rate, wherein the first frame rate is a frame rate higher than the second frame rate and is determined based on a period of stroboscopic generation, and the processing unit detects a stroboscopic influence avoidance moment based on a plurality of frames of video image data to avoid the influence of stroboscopic influence on the shooting of the imaging element.

[0057] Therefore, the storage medium of the nineteenth aspect according to the technology of the present invention can perform imaging while avoiding the influence of stroboscopic light.

[0058] The 20th method involved in the technology of the present invention is an imaging element, which includes: a memory, which stores video image data obtained by photographing a subject at a first frame rate and is built into the imaging element; a processor, which processes the video image data stored in the storage unit and outputs output image data based on the video image data to the outside of the imaging element at a second frame rate, and is built into the imaging element, the first frame rate is a frame rate higher than the second frame rate and is determined based on a period of stroboscopic generation, and the processor detects a stroboscopic influence avoidance moment to avoid the influence of stroboscopic on the shooting of the imaging element based on multiple frames of video image data.

[0059] According to an embodiment of the present invention, it is possible to achieve an effect of photographing while avoiding the influence of stroboscopic light. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a perspective view showing an example of the appearance of an image pickup device which is an interchangeable-lens camera according to the embodiment.

[0061] Figure 2 It is a rear view showing the rear side of the imaging device according to the embodiment.

[0062] Figure 3 This is a block diagram showing an example of the hardware configuration of the imaging device according to the embodiment.

[0063] Figure 4 This is a schematic structural diagram showing an example of the structure of a hybrid viewfinder of the imaging device according to the embodiment.

[0064] Figure 5 This is a schematic configuration diagram showing an example of a schematic configuration of an imaging element included in the imaging device according to the embodiment.

[0065] Figure 6 This is a block diagram showing an example of a main part configuration of an imaging element included in the imaging device according to the embodiment.

[0066] Figure 7 This is a conceptual diagram showing an example of the relationship between the rolling deviation and the strobe period characteristic in the imaging device according to the conventional technology.

[0067] Figure 8 This is a conceptual diagram showing an example of the relationship between the rolling deviation and the strobe period characteristic in the imaging device according to the embodiment.

[0068] Fig. 9This is a graph showing an example of voltage change of a 50 Hz commercial power supply.

[0069] Fig.10 This is a graph showing an example of the flickering cycle characteristics of a flickering light source that flickers at a frequency of 100 Hz.

[0070] Fig.11 This is a conceptual diagram showing an example of a plurality of frames of captured images obtained by capturing an image of a subject using the imaging device according to the embodiment.

[0071] Fig.12 This is an explanatory diagram for explaining an example of a method for detecting a strobe influence avoidance timing according to an embodiment.

[0072] Fig.13 This is a conceptual diagram showing an example of how the brightness difference is sampled when the sampling frequency is 200 Hz.

[0073] Fig.14 This is a conceptual diagram showing an example of how the brightness difference is sampled when the sampling frequency is 400 Hz.

[0074] Fig.15 This is a conceptual diagram showing an example of how the brightness difference is sampled when the sampling frequency is 800 Hz.

[0075] Fig.16 This is a flowchart showing an example of the imaging process flow involved in the embodiment.

[0076] Fig.17 This is a flowchart showing an example of a strobe-avoiding imaging process flow according to the embodiment.

[0077] Fig.18 It is an explanatory diagram for explaining the termination timing of the measurement detection process according to the embodiment.

[0078] Fig.19 The image diagrams show an example of a captured image obtained by shooting using a mechanical shutter and an example of a captured image obtained by shooting using an electronic shutter.

[0079] Fig. 20 This is a conceptual diagram showing an example of the positional relationship between the image area and the line strobe when the cropping area is set.

[0080] Fig.21 This is a conceptual diagram showing an example of a method of outputting a display frame according to the timing of the peak light amount.

[0081] Fig. 22 This is a timing chart showing an example of sequential processing realized by executing the imaging processing and the strobe-avoiding imaging processing according to the embodiment.

[0082] Fig.23 This is a conceptual diagram showing an example of the relationship between the initial flickering cycle characteristics of the flickering light source and the flickering cycle characteristics of the flickering light source several hours later.

[0083] Fig.24 1 is a conceptual diagram showing an example of plotting the luminance difference when the luminance difference calculated throughout one frame is affected by the movement of the subject and / or hand shaking.

[0084] Fig.25 This is a conceptual diagram showing an example of plotting the brightness difference when the brightness difference calculated from the first segmented image (segmented image on the upper side) obtained by vertically dividing each of the two frames of camera images into two is affected by the movement of the subject and / or hand shaking, and an example of plotting the brightness difference when the brightness difference calculated from the second segmented image (segmented image on the lower side) obtained by vertically dividing each of the two frames of camera images into two.

[0085] Fig.26 This is a conceptual diagram showing an example of a method in which the program according to the embodiment is installed from a storage medium storing the program according to the embodiment to an imaging element.

[0086] Fig. 27 This is a block diagram showing an example of a schematic configuration of a smart device incorporating an imaging element according to an embodiment.

[0087] Fig.28 This is a conceptual diagram showing an example of the relationship between the rolling deviation of the electronic shutter and the rolling deviation of the mechanical shutter.

[0088] Fig.29 It means when the rolling deviation of the electronic shutter is Fig.28 A conceptual diagram showing an example of the relationship between the rolling deviation of a short-time electronic shutter and the rolling deviation of a mechanical shutter.

[0089] Fig.30 It means through Fig.28 An image diagram showing an example of an image captured by the rolling shutter method described above.

[0090] Fig.31 It means through Fig.29 An image diagram showing an example of an image captured by the rolling shutter method described above. DETAILED DESCRIPTION

[0091] Hereinafter, an example of an embodiment of an imaging device according to the technology of the present invention will be described with reference to the drawings.

[0092] As an example, Figure 1As shown, the camera 10 is a lens-interchangeable camera. The camera 10 is a digital camera, which includes a camera body 12, an interchangeable lens 14 mounted on the camera body 12 in an interchangeable manner, and omits a reflex mirror. The interchangeable lens 14 includes an imaging lens 18, and the imaging lens 18 has a focusing lens 16 that can be moved along the optical axis direction by manual operation.

[0093] Furthermore, a hybrid viewfinder (registered trademark) 21 is provided on the camera body 12. Here, the so-called hybrid viewfinder 21 refers to a viewfinder that selectively uses an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF"), for example. In addition, OVF is an abbreviation of "optical viewfinder". And EVF is an abbreviation of "electronic viewfinder".

[0094] The interchangeable lens 14 is replaceably mounted on the imaging device body 12. A focus ring 22 used in a manual focus mode is provided on the lens barrel of the interchangeable lens 14. The focus lens 16 moves along the optical axis direction as the focus ring 22 is manually rotated, and the subject light is imaged on the imaging element 20 (see FIG. 2 ) described later at a focus position corresponding to the subject distance. Figure 3 ).

[0095] An OVF viewfinder window 24 included in the hybrid viewfinder 21 is provided on the front surface of the camera body 12. In addition, a viewfinder switching lever (viewfinder switching unit) 23 is provided on the front surface of the camera body 12. When the viewfinder switching lever 23 is rotated in the direction of the arrow SW, the optical image that can be visually recognized by the OVF and the electronic image (live preview image) that can be visually recognized by the EVF are switched. In addition, the live preview image refers to a dynamic image for display obtained by shooting with a photoelectric conversion element.

[0096] The optical axis L2 of the OVF is different from the optical axis L1 of the interchangeable lens 14. Also, a release button 25 and a dial 28 for setting camera system mode, playback system mode, etc. are provided on the upper surface of the imaging device body 12.

[0097] The release button 25 functions as a shooting preparation indication unit and a shooting indication unit, and can detect pressing operations in two stages: a shooting preparation indication state and a shooting indication state. The shooting preparation indication state refers to, for example, a state in which the button is pressed from the standby position to an intermediate position (half-pressed position), and the shooting indication state refers to a state in which the button is pressed to a final pressed position (full-pressed position) beyond the intermediate position. In addition, hereinafter, "a state in which the button is pressed from the standby position to a half-pressed position" is referred to as "a half-pressed state", and "a state in which the button is pressed from the standby position to a fully-pressed position" is referred to as "a fully-pressed state".

[0098] In the camera device 10 involved in this embodiment, as action modes, the camera mode and the playback mode are selectively set according to the user's instructions. In addition, in the camera mode, the manual focus mode and the automatic focus mode are selectively set according to the user's instructions. In the automatic focus mode, the camera conditions are adjusted by setting the release button 25 to a half-pressed state, and then, if it is set to a full-pressed state immediately afterwards, exposure is performed. That is, after the AE (Automatic Exposure) function is activated by setting the release button 25 to a half-pressed state to set the exposure state, the AF (Auto-Focus) function is activated to control the focus, and if the release button 25 is set to a full-pressed state, shooting is performed.

[0099] As an example, Figure 2 As shown, a touch panel / display 30 , a cross key 32 , a menu key 34 , an instruction button 36 , and a viewfinder eyepiece portion 38 are provided on the back surface of the imaging device body 12 .

[0100] The touch panel / display 30 includes a liquid crystal display (hereinafter referred to as a "first display") 40 and a touch panel 42 (see Figure 3 ).

[0101] The first display 40 displays images and character information, etc. The first display 40 is used to display an instant preview image (live view image), which is an example of a continuous frame image obtained by shooting in continuous frames in the camera mode. In addition, the first display 40 is also used to display an example of a single frame image, that is, a still image, which is obtained by shooting in a single frame when an instruction to shoot a still image is given. In addition, the first display 40 is also used to display a playback image in the playback mode and display a menu screen, etc.

[0102] The touch panel 42 is a transmissive touch panel and overlaps the surface of the display area of ​​the first display 40. The touch panel 42 detects contact by a pointer such as a finger or a stylus pen. The touch panel 42 outputs detection result information indicating the detection result (whether the pointer has contacted the touch panel 42) to a predetermined output destination (for example, the CPU 52 described later (see Figure 3 )). When the touch panel 42 detects contact by the indicator, the detection result information includes two-dimensional coordinates (hereinafter referred to as "coordinates") that can determine the contact position by the indicator on the touch panel 42, and does not include coordinates when the touch panel 42 does not detect contact by the indicator.

[0103] The cross key 32 functions as a multi-function key that outputs various command signals such as selection of one or more menus, zooming and / or frame transfer. The menu key 34 is an operation key that has the following functions: as a menu button, it is used to issue a command to display one or more menus on the screen of the first display 40; and as a command button, it is used to issue a command to confirm and execute the selected content. The instruction button 36 is operated to delete a desired object such as a selected item, cancel the specified content, and return to the previous operation state.

[0104] The camera 10 has a still image capture mode and a moving image capture mode as operation modes of the camera system. The still image capture mode is an operation mode in which a still image obtained by the camera 10 capturing a subject is recorded, and the moving image capture mode is an operation mode in which a moving image obtained by the camera 10 capturing a subject is recorded.

[0105] As an example, Figure 3 As shown, the camera device 10 includes a mount 46 (also see Figure 1 ) and a bayonet mount 44 on the interchangeable lens 14 side corresponding to the bayonet mount 46. The interchangeable lens 14 is mounted on the imaging device body 12 in an interchangeable manner by the bayonet mount 44 being keyed to the bayonet mount 46.

[0106] The imaging lens 18 includes an aperture 47 and a motor 49. The aperture 47 is disposed closer to the imaging device body 12 than the focusing lens 16, and is connected to the motor 49. The aperture 47 is operated by the power of the motor 49 to adjust the exposure.

[0107] The imaging lens 18 includes a sliding mechanism 48 and a motor 50. The sliding mechanism 48 moves the focus lens 16 along the optical axis L1 by operating the focus ring 22. The focus lens 16 is slidably mounted on the sliding mechanism 48 along the optical axis L1. The motor 50 is connected to the sliding mechanism 48, and the sliding mechanism 48 receives power from the motor 50 to slide the focus lens 16 along the optical axis L1.

[0108] The motors 49 and 50 are connected to the camera body 12 via the mounts 44 and 46, and are driven by commands from the camera body 12. In the present embodiment, a stepping motor is used as an example of the motors 49 and 50. Thus, the motors 49 and 50 operate in synchronization with the pulse power according to the commands from the camera body 12. Figure 3 In the example shown, the motors 49 and 50 are provided in the imaging lens 18 , but the present invention is not limited thereto, and the motors 49 and 50 may be provided in the imaging device body 12 .

[0109] The camera device 10 is a digital camera that records still images and moving images obtained by photographing a subject. The camera device body 12 includes an operation unit 54, an external interface (I / F) 63, and a subsequent circuit 90. The subsequent circuit 90 is a circuit that receives data sent from the imaging element 20. In this embodiment, an IC "Integrated Circuit" is used as the subsequent circuit 90. As an example of IC, LSI (Large-Scale Integration) can be cited. In addition, the subsequent circuit 90 is an example of a "circuit" involved in the technology of the present invention.

[0110] The post-stage circuit 90 includes a CPU (Central Processing Unit) 52, an I / F 56, a main storage unit 58, an auxiliary storage unit 60, an image processing unit 62, a first display control unit 64, a second display control unit 66, a position detection unit 70, and a device control unit 74. In this embodiment, one CPU is illustrated as the CPU 52, but the technology of the present invention is not limited thereto, and a plurality of CPUs may be used instead of the CPU 52. That is, various processes executed by the CPU 52 may be executed by one processor or a plurality of physically separated processors.

[0111] In addition, in the present embodiment, the image processing unit 62, the first display control unit 64, the second display control unit 66, the position detection unit 70 and the device control unit 74 are respectively implemented by ASIC (Application Specific Integrated Circuit), but the technology of the present invention is not limited to this. For example, at least one of PLD (Programmable Logic device) and FPGA (Field-Programmable Gate array) can be used to replace ASIC. In addition, at least one of ASIC, PLD and FPGA can be used. In addition, a computer including CPU, ROM (Read Only Memory) and RAM (Random Access Memory) can be used. The CPU can be one or more. In addition, at least one of the image processing unit 62, the first display control unit 64, the second display control unit 66, the position detection unit 70, and the device control unit 74 can be implemented by a combination of hardware structure and software structure.

[0112] The CPU 52 , the I / F 56 , the main storage unit 58 , the auxiliary storage unit 60 , the image processing unit 62 , the first display control unit 64 , the second display control unit 66 , the operation unit 54 , the external I / F 63 , and the touch panel 42 are connected to one another via a bus 68 .

[0113] The CPU 52 controls the entire camera device 10. In the camera device 10 involved in this embodiment, when the automatic focus mode is in progress, the CPU 52 performs focus control by driving the control motor 50 so that the contrast value of the image obtained by shooting is maximized. In addition, when the automatic focus mode is in progress, the CPU 52 calculates the physical quantity representing the brightness of the image obtained by shooting, that is, the AE information. When the release button 25 is set to a half-pressed state, the CPU 52 derives the shutter speed and F value corresponding to the brightness of the image represented by the AE information. Then, the relevant parts are controlled so that the shutter speed and F value obtained are obtained, thereby setting the exposure state.

[0114] The main storage unit 58 is a volatile memory, such as a RAM, and the auxiliary storage unit 60 is a nonvolatile memory, such as a flash memory or a HDD (Hard Disk Drive).

[0115] The auxiliary storage unit 60 stores an imaging program 60A. The CPU 52 reads the imaging program 60A from the auxiliary storage unit 60 and expands the read imaging program 60A to the main storage unit 58. The CPU 52 executes imaging processing described later according to the imaging program 60A expanded to the main storage unit 58 (see Fig.16 ).

[0116] The operation unit 54 is a user interface operated by the user when giving various instructions to the subsequent circuit 90. The operation unit 54 includes the release button 25, the dial 28, the viewfinder switching lever 23, the cross key 32, the menu key 34, and the instruction button 36. Various instructions received through the operation unit 54 are output to the CPU 52 as operation signals, and the CPU 52 executes processing corresponding to the operation signal input from the operation unit 54.

[0117] The position detection unit 70 is connected to the CPU 52. The position detection unit 70 is connected to the focus ring 22 via the mounts 44 and 46, detects the rotation angle of the focus ring 22, and outputs rotation angle information indicating the detection result, that is, the rotation angle, to the CPU 52. The CPU 52 performs processing corresponding to the rotation angle information input from the position detection unit 70.

[0118] When the imaging mode is set, image light representing the subject is formed on the light receiving surface of the color imaging element 20 via the imaging lens 18 including the focus lens 16 that can be moved by manual operation and the mechanical shutter 72 .

[0119] The device control section 74 is connected to the CPU 52. Furthermore, the device control section 74 is connected to the imaging element 20 and the mechanical shutter 72. Furthermore, the device control section 74 is connected to the motors 49 and 50 of the imaging lens 18 via the mounts 44 and 46.

[0120] The device control section 74 controls the imaging element 20 , the mechanical shutter 72 , and the motors 49 and 50 under the control of the CPU 52 .

[0121] As an example, Figure 4 As shown, the hybrid viewfinder 21 includes an OVF 76 and an EVF 78. The OVF 76 is a reverse Galilean viewfinder having an objective lens 81 and an eyepiece lens 86, and the EVF 78 has a second display 80, a prism 84, and an eyepiece lens 86.

[0122] Furthermore, a liquid crystal shutter 88 is disposed in front of the objective lens 81 , and when the EVF 78 is used, the liquid crystal shutter 88 shields light so that an optical image is not incident on the mirror 81 .

[0123] The prism 84 reflects the electronic image or various information displayed on the second display 80 and guides them to the eyepiece lens 86 , and synthesizes the optical image with the electronic image and / or various information displayed on the second display 80 .

[0124] Here, if the viewfinder switching lever 23 is turned Figure 1 The optical image can be visually recognized by the OVF 76 and the electronic image can be visually recognized by the EVF 78.

[0125] In the case of the OVF mode, the second display control unit 66 controls the liquid crystal shutter 88 to be in a non-light-shielding state so that the optical image can be visually recognized from the eyepiece portion. In addition, in the case of the EVF mode, the second display control unit 66 controls the liquid crystal shutter 88 to be in a light-shielding state so that only the electronic image displayed on the second display 80 can be visually recognized from the eyepiece portion.

[0126] In addition, for the sake of convenience, in the following, when it is not necessary to distinguish between the first display 40 and the second display 80, they are referred to as "display devices" without being marked with symbols. The display device is an example of the "display unit" involved in the technology of the present invention. In addition, in the following, for the sake of convenience, when it is not necessary to distinguish between the first display control unit 64 and the second display control unit 66, they are referred to as "display control units" without being marked with symbols.

[0127] The imaging element 20 is an example of a "stacked imaging element" according to the present invention. The imaging element 20 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Figure 5 As shown, the imaging element 20 has a built-in photoelectric conversion element 92, a processing circuit 94, and a memory 96. In the imaging element 20, the processing circuit 94 and the memory 96 are stacked on the photoelectric conversion element 92. In addition, the processing circuit 94 is an example of the "processing unit" involved in the technology of the present invention, and the memory 96 is an example of the "storage unit" involved in the technology of the present invention.

[0128] The processing circuit 94 is, for example, an LSI, and the memory 96 is, for example, a RAM. In this embodiment, a DRAM (Dynamic Random Access Memory) is used as an example of the memory 96, but the technology of the present invention is not limited thereto, and an SRAM (Static Random Access Memory) may also be used.

[0129] In this embodiment, the processing circuit 94 is implemented by an ASIC, but the technology of the present invention is not limited thereto. For example, at least one of a PLD and an FPGA may be used instead of the ASIC. Furthermore, at least one of an ASIC, a PLD, and an FPGA may be used. Furthermore, a computer including a CPU, a ROM, and a RAM may be used. The CPU may be one or more. Furthermore, the processing circuit 94 may be implemented by a combination of a hardware structure and a software structure.

[0130] The photoelectric conversion element 92 has a plurality of photoelectric sensors arranged in a matrix. In the present embodiment, photodiodes are used as an example of the photoelectric sensors. As an example of the plurality of photoelectric sensors, photodiodes with "4896×3265" pixels can be cited.

[0131] The photoelectric conversion element 92 is provided with a color filter, and the color filter includes a G filter corresponding to G (green), an R filter corresponding to R (red), and a B filter corresponding to B (blue) which are most helpful for obtaining a brightness signal. In the present embodiment, for the plurality of photodiodes of the photoelectric conversion element 92, the G filter, the R filter, and the B filter are respectively arranged with a prescribed periodicity in the row direction (horizontal direction) and the column direction (vertical direction). Therefore, when the camera device 10 performs synchronization processing of R, G, and B signals, it can perform processing in a repetitive mode. In addition, the synchronization processing refers to the following processing: all color information is calculated for each pixel from a mosaic image corresponding to the color filter arrangement of a single-plate color imaging element. For example, in the case of an imaging element composed of RGB three-color filters, the synchronization processing refers to the following processing: all RGB color information is calculated for each pixel from a mosaic image composed of RGB.

[0132] In addition, although a CMOS image sensor is exemplified as the imaging element 20 here, the technology of the present invention is not limited thereto, and even if the photoelectric conversion element 92 is a CCD (Charge Coupled Device) image sensor, for example, the technology of the present invention is also applicable.

[0133] The imaging element 20 has a so-called electronic shutter function, and controls the charge accumulation time of each photodiode in the photoelectric conversion element 92 by activating the electronic shutter function under the control of the device control section 74. The charge accumulation time is a so-called shutter speed.

[0134] In the imaging device 10, still image shooting and moving image shooting are performed by rolling shutter method. Still image shooting is achieved by activating the electronic shutter function and operating the mechanical shutter 72, and live preview image shooting is achieved by activating the electronic shutter function without operating the mechanical shutter 72.

[0135] The processing circuit 94 is controlled by the device control unit 74. The processing circuit 94 reads the captured image data obtained by photographing the subject by the photoelectric conversion element 92. Here, the so-called "captured image data" refers to image data representing the subject. The captured image data is the signal charge accumulated in the photoelectric conversion element 92. The processing circuit 94 performs A / D (Analog / Digital: analog / digital) conversion on the captured image data read from the photoelectric conversion element 92. The processing circuit 94 stores the captured image data obtained by performing A / D conversion on the captured image data in the memory 96. The processing circuit 94 obtains the captured image data from the memory 96, and outputs the image data based on the acquired captured image data, that is, the output image data, to the I / F56 of the subsequent circuit 90. In addition, hereinafter, for the convenience of explanation, "image data based on the captured image data, that is, the output image data" is referred to as "output image data".

[0136] The processing circuit 94 performs a first process and a second process on the captured image data. The first process refers to a process of reading the captured image data from the photoelectric conversion element 92 and storing the read captured image data in the memory 96. The second process refers to a process of outputting the output image data to the outside of the imaging element 20. Here, the so-called "outside of the imaging element 20" refers to, for example, the I / F 56 of the subsequent circuit 90. In addition, the subsequent circuit 90 is an example of a "circuit" involved in the technology of the present invention.

[0137] In the imaging element 20, the subject is photographed at a first frame rate. The processing circuit 94 performs a first process at the first frame rate and performs a second process at a second frame rate. The first frame rate is a frame rate higher than the second frame rate.

[0138] In the present embodiment, 60 fps (frames per second) is used as an example of the second frame rate, but the technology of the present invention is not limited thereto, and the second frame rate may be changed if the relationship of “second frame rate < first frame rate” is satisfied.

[0139] The first frame rate is set to be a frame rate that can be changed within a range that does not fall below the second frame rate. For example, in this embodiment, the first frame rate is set to be a frame rate that can be switched between a high frame rate and a low frame rate by the processing circuit 94.

[0140] A high frame rate refers to a frame rate higher than a low frame rate. In the present embodiment, 100fps is adopted as an example of a low frame rate. Furthermore, in the present embodiment, the high frame rate is classified into a first high frame rate and a second high frame rate. The first high frame rate is a frame rate adopted when the stroboscopic generation period described later is 10ms, and the second high frame rate is a frame rate adopted when the stroboscopic generation period described later is 8.33ms. In the present embodiment, 200fps is adopted as an example of the first high frame rate, and 240fps is adopted as an example of the second low frame rate. In addition, between the first high frame rate and the second high frame rate, the relationship of "first high frame rate < second high frame rate" holds. Furthermore, the high frame rate may be a frame rate higher than 240fps, such as 400fps, 480fps, 800fps, or 960fps.

[0141] The output image data obtained by photographing the subject by the imaging element 20 may include a flicker. Figure 7 and Figure 8 As shown, a linear strobe is generated, i.e., a line strobe. Figure 7 An example of a state in which a live preview image obtained by photographing a subject using a camera device according to the prior art is displayed on a display is shown in FIG. Figure 8 2 shows an example of a state in which a live preview image obtained by photographing a subject by the imaging device 10 is displayed on the display device.

[0142] Line strobe is a phenomenon caused by the rolling deviation of the camera device 10 and the flickering of a periodically flickering light source (hereinafter referred to as a "flickering light source"). When a live preview image is displayed on a display device, line strobe appears in a vertical direction. Figure 7 In the example shown, 2 lines flicker in the live preview image. Figure 8 In the example, Figure 7 Compared to the example shown, due to the short scrolling deviation, one line of strobing appears in the live preview image.

[0143] As an example of a flickering light source, a fluorescent lamp can be cited. When AC power is supplied to the fluorescent lamp from a commercial power supply, the flickering cycle of the fluorescent lamp is half of the voltage cycle of the commercial power supply. Fig. 9 and Fig.10As shown in FIG. 1 , when the frequency of the commercial power supply is 50 Hz, the flicker frequency of the fluorescent lamp is 100 Hz, and when the frequency of the commercial power supply is 60 Hz, the flicker frequency of the fluorescent lamp is 120 Hz. In addition, here, a fluorescent lamp is exemplified as a flickering light source, but the flickering light source is not limited to a fluorescent lamp. The flickering light source may be, for example, an LED (Light Emitting Diode) used in a display connected to a personal computer or lighting.

[0144] In the imaging device 10, in order to achieve shooting without causing the line strobe to be reflected in the image represented by the output image data, that is, shooting to avoid the influence of the line strobe, the processing circuit 94 determines the first frame rate according to the generation cycle of the line strobe. That is, the processing circuit 94 changes the first frame rate from the low frame rate to the first high frame rate or the second high frame rate according to the generation cycle of the line strobe (hereinafter referred to as the "strobe generation cycle").

[0145] The stroboscopic generation cycle is the flashing cycle of the flashing light source. Therefore, for example, when shooting in an environment illuminated by a fluorescent lamp, the flashing cycle of the fluorescent lamp is used as the stroboscopic generation cycle by the processing circuit 94. The stroboscopic generation cycle used by the processing circuit 94 may be, for example, a variable value that is changed according to an instruction received by the touch panel 42 and / or the operating unit 54, or may be a fixed value.

[0146] As an example, Figure 6 As shown in FIG. 1 , the processing circuit 94 includes a photoelectric conversion element driving circuit 94A, an AD (Analog-to-Digital) conversion circuit 94B, an image processing circuit 94C, an output circuit 94D, and a storage circuit 94E. The processing circuit 94 operates under the control of the CPU 52 via the device control unit 74. In addition, a cropping factor derivation table 98 is stored in the storage circuit 94E. The details will be described later. The cropping factor derivation table 98 is a table used in the stroboscopic avoidance imaging process described later.

[0147] The photoelectric conversion element driving circuit 94A is connected to the photoelectric conversion element 92 and the AD conversion circuit 94B. The memory 96 is connected to the AD conversion circuit 94B and the image processing circuit 94C. The image processing circuit 94C is connected to the output circuit 94D. The output circuit 94D is connected to the I / F 56 of the subsequent circuit 90.

[0148] The photoelectric conversion element driving circuit 94A controls the photoelectric conversion element 92 under the control of the device control unit 74, and reads analog camera image data from the photoelectric conversion element 92. The AD conversion circuit 94B digitizes the camera image data read by the photoelectric conversion element driving circuit 94A, and stores the digitized camera image data in the memory 96. The memory 96 is a memory that can store camera image data of multiple frames. The image processing circuit 94C processes the camera image data.

[0149] In the camera device 10, in order to achieve shooting without causing line strobe to be reflected in the image represented by the camera image data (hereinafter referred to as "camera image"), the processing circuit 94 detects the strobe effect avoidance time. The strobe effect avoidance time refers to the time when the influence of line strobe on the shooting of the imaging element 20 is avoided. In the present embodiment, as an example of the strobe effect avoidance time, the time when the brightness of the camera image reaches a peak is used. The time when the brightness of the camera image reaches a peak refers to, for example, the time when the brightness difference of the camera image between adjacent frames becomes "0". In addition, the strobe effect avoidance time is not necessarily the time when the brightness of the camera image reaches a peak. In this case, for example, the strobe effect avoidance time may be a time that deviates from the peak of the brightness of the camera image within a range where strobe does not appear in the camera image.

[0150] Here, the brightness difference is the difference between the average brightness of all the camera images of one frame and the average brightness of the entire camera image of another frame, but the technology of the present invention is not limited to this. For example, the brightness difference can be the difference between the average brightness of a local area of ​​the camera image of one frame and the average brightness of a local area of ​​the camera image of another frame.

[0151] Here, reference Fig.11 and Fig.12 , an example of a method for detecting the timing for avoiding the influence of strobe light is described. Fig.11 In the example shown, as an example of a plurality of frames of captured images obtained by shooting at a high frame rate, captured images of the first frame to the Nth frame are shown. The first frame is the latest frame, and the Nth frame is the first frame obtained in the past.

[0152] As an example, Fig.12 As shown, the sequence of the method for detecting the moment of avoiding the influence of stroboscopic light is divided into steps S1 to S4. First, in step S1, whenever new camera image data is stored in the memory 96, the brightness difference between the latest camera image data and the camera image data of the previous frame is calculated, and the brightness difference periodic characteristic curve G representing the periodic characteristics of the brightness difference is generated by plotting the calculated brightness difference in time series.

[0153] In addition, Fig.12 In the example shown, point A 1is the brightness difference between the camera image data of the first frame and the camera image data of the second frame, point A 2 It is the brightness difference between the captured image data of the second frame and the captured image data of the third frame.

[0154] In step S2, the time when the brightness difference is within the range from negative to positive and becomes "0" and the time when the brightness difference is within the range from positive to negative and becomes "0" are detected. Fig.12 In the example shown, the brightness difference at the "▲" position and the brightness difference at the "▼" position on the brightness difference period characteristic curve G are both "0". Fig.12 In the example shown, the “▲” position indicates the moment when the luminance difference changes from a negative value to a positive value and the luminance difference becomes “0”. Also, the “▼” position indicates the moment when the luminance difference changes from a positive value to a negative value and the luminance difference becomes “0”.

[0155] In other words, Fig.12 The "▲" position shown is the position where the brightness difference is "0" within the range where the differential coefficient of the brightness difference period characteristic curve G shows a positive value. Fig.12 The "▼" position shown is a position where the brightness difference is "0" within the range where the differential coefficient of the brightness difference period characteristic curve G shows a negative value.

[0156] In step S3, the time interval at which the brightness difference becomes "0" is detected. Fig.12 In the example, the detection time interval B 1 and time interval B 2 . Time interval B 1 is the time interval from the “▼” position to the “▲” position, time interval B 2 It is the time interval from the “▲” position to the “▼” position.

[0157] In step S4, the stroboscopic influence avoiding time is detected according to the detection results in step S2 and the detection results in step S3. Fig.12 In the example shown, the moment when the camera image becomes brightest in the brightness difference period characteristic curve G, that is, the moment when the brightness reaches the peak, is the flicker influence avoidance moment, and the "▼" position on the brightness difference period characteristic curve G is the flicker influence avoidance moment. The flicker influence avoidance moment, that is, the "▼" position on the brightness difference period characteristic curve G is "interval B 1 +Time Interval B 2 "A constant cycle appears.

[0158] Detecting the stroboscopic effect avoiding time is important for realizing shooting to avoid the influence of line stroboscopic. Therefore, in the imaging element 20, the processing circuit 94 performs the measurement detection processing at the first frame rate in the same manner as the first processing. In other words, the processing performed by the processing circuit 94 includes the measurement detection processing.

[0159] The measurement detection process refers to a process of measuring the brightness difference between frames of the captured image data and detecting the flicker influence avoidance time based on the measured brightness difference. Fig.12 A series of processing shown in steps S1 to S3.

[0160] The detection of the flicker influence avoiding time is realized on the premise of creating the brightness difference period characteristic curve G. In order to improve the detection accuracy of the flicker influence avoiding time, it is necessary to improve the accuracy of the brightness difference period characteristic curve G. In order to improve the accuracy of the brightness difference period characteristic curve G, it is necessary to increase the sampling frequency of the brightness difference in step S1 (= the number of brightness difference measurements / 1 second). However, in order to achieve Fig.12 The sampling of the brightness difference shown in FIG. 1 needs to be performed by culling the camera image data, so the accuracy of the brightness difference is reduced by the amount of culling. If the accuracy of the brightness difference is reduced, it is easy to detect the stroboscopic effect avoidance time incorrectly. In contrast, the imaging element 20 can ensure the necessary sampling frequency without culling the camera image data based on the detection of the stroboscopic effect avoidance time.

[0161] In order to ensure the detection accuracy of the flicker influence avoidance timing, it is important to appropriately determine the sampling frequency. Also, from the perspective of reducing power consumption, it is preferable to appropriately determine the sampling frequency.

[0162] The sampling frequency is uniquely determined by the first frame rate. Therefore, in the imaging element 20, the processing circuit 94 determines the first frame rate according to the strobe generation cycle.

[0163] Furthermore, for example, when the processing circuit 94 performs the measurement detection process from the time when the release button 25 is fully pressed to the time when the image is taken, the shorter the processing time required for the measurement detection process (hereinafter referred to as "processing time"), the better. Shortening the processing time means shortening the time lag from the fully pressed state to the time when the image is taken.

[0164] In this embodiment, as an example of the processing time, the shortest time required to detect the position where the brightness difference becomes "0" three times is used. Fig.12 As shown, the shortest time required to detect the position where the brightness difference becomes "0" three times is the shortest time required to detect the "▼" position twice.

[0165] In addition, in this embodiment, the shortest time required to detect the position where the brightness difference becomes "0" three times is used as the processing time, but the technology of the present invention is not limited to this. For example, the processing time may also be the shortest time required to detect the position where the brightness difference becomes "0" twice. The shortest time required to detect the position where the brightness difference becomes "0" twice refers to the detection of the position where the brightness difference becomes "0" as an example. Fig.12 The time interval B shown in step S3 1 The shortest time required. By detecting the time interval B 1 , then, if the brightness difference is considered to be equal to the time interval B 1 When the same time interval becomes "0", the flicker influence avoidance timing is determined in the same manner as when the position where the brightness difference becomes "0" is detected three times.

[0166] The processing circuit 94 determines the processing time according to the stroboscopic generation cycle. For example, when shooting in an environment where a stroboscopic light source with a stroboscopic frequency of 100 Hz is flashing, the processing circuit 94 sets the first high frame rate as the high frame rate of the first frame rate. In this embodiment, 200fps is used as an example of the first high frame rate. In this case, the processing circuit 94 sets 30ms (=10ms (=1 / 100Hz)×3 times) as the processing time. Here, "3 times" refers to the number of times the brightness difference becomes "0".

[0167] Furthermore, for example, when shooting is performed in an environment where a flashing light source with a flashing frequency of 120 Hz is flashing, the processing circuit 94 sets the first frame rate to the second high frame rate. In the present embodiment, 240 fps is used as an example of the second high frame rate. In this case, the processing circuit 94 sets 25 ms (= 8.33 ms (= 1 / 120 Hz) × 3 times) as the processing time. Here, "3 times" refers to the number of times the brightness difference becomes "0".

[0168] The processing time of 30 ms and 25 ms is the shortest processing time, and a longer processing time may be used. This is because, in consideration of the movement of the subject or hand shaking, when detecting the position where the brightness difference becomes "0" three times, the processing time of 30 ms and 25 ms may not be sufficient. In this case, for example, the processing circuit 94 detects the degree of movement of the subject or hand shaking, and according to the detection result, the processing time of 30 ms and 25 ms may be added with the required time.

[0169] Furthermore, in order to improve the accuracy of the brightness difference period characteristic curve G, it is preferred to increase the sampling frequency. For example, when the sampling frequency is 200 Hz, as an example, Fig.13 As shown in FIG. 1 , the brightness difference is measured three times within one cycle of the flashing light source. And, for example, when the sampling frequency is 400 Hz, as an example, Fig.14 As shown in FIG. 1 , the brightness difference is measured five times within one cycle of the flashing light source. In addition, for example, when the sampling frequency is 800 Hz, as an example, Fig.15 As shown, the brightness difference is measured 9 times within one cycle of the flashing light source.

[0170] Next, the operation of the portion of the imaging device 10 that is related to the technique of the present invention will be described.

[0171] When receiving an instruction to start the image capture process from the operation unit 54, the CPU 52 executes the image capture process according to the image capture program 60A. Fig.16 The imaging process executed by the CPU 52 will be described. In addition, for the sake of convenience, the following description will be based on the case where the flash light source is a fluorescent lamp, the imaging device 10 captures a subject only under the illumination of the fluorescent lamp, and a still image showing the subject is acquired. In addition, for the sake of convenience, the following description will be based on the premise that both AF and AE are executed when the release button 25 is half-pressed.

[0172] exist Fig.16 In the imaging process shown, first, in step S100, the CPU 52 controls the imaging element 20 and the display control unit to start displaying the live preview image on the display device, and then transfers to step S102. In this step S100, the CPU 52 starts shooting with the imaging element 20 at the first frame rate, and outputs the live preview image obtained by shooting with the imaging element 20 to the display control unit. The display control unit displays the input live preview image on the display device.

[0173] In step S102, the CPU 52 determines whether the release button 25 is in a half-pressed state. In step S102, if the release button 25 is in a half-pressed state, the determination is "Yes", and the image capture process moves to step S104. In step S102, if the release button 25 is not in a half-pressed state, the determination is "No", and the image capture process moves to step S112.

[0174] In step S104, the CPU 52 determines whether AF and AE are being executed. In step S104, if AF and AE are not being executed, the determination is "No", and the imaging process moves to step S106. In step S104, if AF and AE are being executed, the determination is "Yes", and the imaging process moves to step S108.

[0175] In step S106 , the CPU 52 starts executing AF and AE, and then the image capturing process moves to step S108 .

[0176] In step S108, the CPU 52 determines whether the condition for ending the image capture process, i.e., the image capture process ending condition, is satisfied. As the image capture process ending condition, for example, a condition that an instruction to end the image capture process is received by the touch panel 42 and / or the operation unit 54 can be cited. Also, as the image capture process ending condition, for example, a condition that the time after the image capture process is started and the determination is "No" in step S112 exceeds a predetermined time can be cited. Here, the so-called "predetermined time" refers to, for example, 5 minutes. The predetermined time can be a fixed value or a variable value that can be changed according to an instruction issued by the user.

[0177] In step S108, if the imaging process end condition is satisfied, the determination is "yes", and the imaging process moves to step S110. In step S108, if the imaging process end condition is not satisfied, the determination is "no", and the imaging process moves to step S102.

[0178] In step S112, the CPU 52 determines whether the release button 25 is in a fully pressed state. In step S112, if the release button 25 is in a fully pressed state, the determination is "Yes", and the image capture process moves to step S114. In step S112, if the release button 25 is not in a fully pressed state, that is, if the release button 25 is not pressed, the determination is "No", and the image capture process moves to step S122.

[0179] In step S122, the CPU 52 determines whether AF and AE are being executed. In step S122, if AF and AE are not being executed, the determination is "No", and the imaging process moves to step S108. In step S122, if AF and AE are being executed, the determination is "Yes", and the imaging process moves to step S124.

[0180] In step S124 , the CPU 52 ends execution of AF and AE, and the image capturing process proceeds to step S108 .

[0181] In step S114 , the CPU 52 ends AF and AE, and the image capturing process proceeds to step S116 .

[0182] In step S116 , the CPU 52 instructs the imaging element 20 to start executing the strobe avoidance imaging process, and the imaging process then shifts to step S118 .

[0183] When the CPU 52 executes the process of step S116 to instruct the imaging element 20 to start the strobe avoidance imaging process, the processing circuit 94 of the imaging element 20 executes the strobe avoidance imaging process.

[0184] Here, reference Fig.17The following describes the stroboscopic avoidance imaging process executed by the processing circuit 94. Note that, for the sake of convenience, the following description is based on the premise that the stroboscopic avoidance imaging process is started in a state where a low frame rate is adopted as the first frame rate.

[0185] exist Fig.17 In the stroboscopic avoidance imaging process shown in FIG. 1 , first, in step S150, the processing circuit 94 starts executing Fig.16 After the imaging process shown in FIG. 1 , it is determined whether the stroboscopic avoidance imaging process is executed for the first time. In other words, the processing circuit 94 starts to execute Fig.16 After the image capturing process shown, it is determined whether the stroboscopic avoidance image capturing process is executed for the first time.

[0186] In step S150, when starting to execute Fig.16 If the stroboscopic avoidance imaging process is executed for the first time after the imaging process shown in FIG. 1, the stroboscopic avoidance imaging process is transferred to step S152. Fig.16 If the strobe avoidance imaging process is not executed for the first time after the illustrated imaging process, that is, if it is executed for the second time or later, the determination is "NO", and the strobe avoidance imaging process is transferred to step S180.

[0187] In step S152, the processing circuit 94 acquires the power frequency information, and the stroboscopic avoidance imaging process is transferred to step S154. The power frequency information refers to the information of the frequency of the commercial power supply used as the power supply source for the fluorescent lamp. The power frequency information is received by the touch panel 52 and / or the operation unit 54, for example, and acquired by the processing circuit 94 via the CPU 52 and the device control unit 74.

[0188] In step S154, the processing circuit 94 calculates the stroboscopic generation period based on the power frequency information acquired in step S152, and the stroboscopic avoidance imaging process is transferred to step S156. In step S154, for example, when the frequency indicated by the power frequency information is 50 Hz, 10 ms is calculated as the stroboscopic generation period, and when the frequency indicated by the power frequency information is 60 Hz, 8.33 ms is calculated as the stroboscopic generation period.

[0189] In step S156, the processing circuit 94 determines the high frame rate of the first frame rate based on the strobe generation cycle calculated in step S154. That is, the processing circuit 94 determines any one of the first high frame rate and the second high frame rate as the high frame rate of the first frame rate.

[0190] In step S156, when the stroboscopic generation period calculated in step S154 is 10 ms, the high frame rate of the first frame rate is determined as the first high frame rate between the first high frame rate and the second high frame rate. Furthermore, when the stroboscopic generation period calculated in step S154 is 8.33 ms, the high frame rate of the first frame rate is determined as the second high frame rate between the first high frame rate and the second high frame rate.

[0191] In the next step S160 , the processing circuit 94 changes the first frame rate from the low frame rate to the high frame rate determined in step S156 , and the strobe-avoidance imaging process moves to step S162 .

[0192] In step S162, the processing circuit 94 calculates the above-mentioned processing time, and then the strobe avoidance imaging process is transferred to step S164.

[0193] In step S164, the image processing circuit 94C acquires brightness from the latest captured image data stored in the memory 96, and then moves to step S166. Here, the "latest captured image data" is an example of the "second captured image data" involved in the technique of the present invention.

[0194] In step S166, the image processing circuit 94C obtains the brightness from the image data of the frame before the latest image data among the image data of the plurality of frames stored in the memory 96, and the stroboscopic avoidance image processing is transferred to step S168. Here, the so-called "image data of the frame before the latest image data" is an example of the "first image data" involved in the technology of the present invention.

[0195] In step S168, the image processing circuit 94C calculates the brightness difference, and the stroboscopic avoidance imaging process is transferred to step S170. The brightness difference calculated by executing step S168 is the result of subtracting the brightness acquired in step S164 from the brightness acquired in step S166. Therefore, if "the brightness acquired in step S166> the brightness acquired in step S164", the brightness difference becomes a positive value, and if "the brightness acquired in step S166<the brightness acquired in step S164", the brightness difference becomes a negative value. And, if "the brightness acquired in step S166=the brightness acquired in step S164", the brightness difference becomes "0".

[0196] In step S170, the image processing circuit 94C determines whether the processing time calculated in step S162 has elapsed after the execution of step S162 is completed. In step S170, if the processing time calculated in step S162 has not elapsed, the determination is "No", and the stroboscopic avoidance imaging process is transferred to step S164. In step S170, if the processing time calculated in step S162 has elapsed, the determination is "Yes", and the stroboscopic avoidance imaging process is transferred to step S172.

[0197] In step S172, the image processing circuit 94C terminates the calculation of the brightness difference and the calculation of the light peak period, and then transfers to step S174. The light peak refers to the peak value of the brightness of the captured image. The light peak period refers to the period that specifies the time to avoid the influence of stroboscopic light. Fig.12 In the example shown, the "time interval B" of step S3 1 +Time Interval B 2 The time of " is equivalent to one cycle of the peak light intensity cycle.

[0198] Furthermore, in step S172, terminating the calculation of the brightness difference means terminating the measurement detection process when the brightness difference changes from a positive value to a negative value twice. Fig.18 As shown in FIG. 1 , when three positions where the brightness difference becomes “0” are detected in the brightness difference period characteristic curve G, the calculation of the brightness difference is terminated. Fig.18 The example shown in the figure shows an example of detecting three positions where the brightness difference becomes "0" in the shortest period and an example of detecting three positions where the brightness difference becomes "0" in the longest period. Fig.18 In the example shown, the light amount peak period is specified by the time interval between the positions of two "▼".

[0199] As an example, Fig.21 As shown in FIG. 1 , if the light amount peak period is calculated by executing the process of step S172, then the display frame is output from the output circuit 94D to the subsequent circuit 90 according to the latest light amount peak period calculated in step S172. If the display frame is output from the output circuit 94D to the subsequent circuit 90, the display frame is displayed on the display device as a live preview image under the control of the CPU 52. Here, the so-called "display frame" refers to the output image data for the live preview image obtained by the image processing circuit 94C processing the camera image data read from the memory 96.

[0200] In step S174, the processing circuit 94 determines whether the peak light intensity of the captured image has arrived based on the peak light intensity cycle calculated in step S172. Here, the so-called "peak light intensity of the captured image" is an example of the "stroboscopic effect avoidance timing" involved in the technology of the present invention.

[0201] In step S174, if the image light intensity peak has not yet arrived, the determination is "No", and step S174 is determined again. In step S174, if the image light intensity peak has arrived, the determination is "Yes", and the stroboscopic avoidance imaging process is transferred to step S176.

[0202] As an example, Fig.19 As shown in FIG. 1 , it is generally known that the rolling deviation of the electronic shutter has a greater influence than the rolling deviation of the mechanical shutter. Therefore, even in the imaging element 20, when “the rolling deviation time of the electronic shutter is greater than the rolling deviation time of the mechanical shutter 72”, as an example, Fig. 20 As shown, two lines will flicker on the screen of the display device. The same is true for the case of "flicker generation period ≈ scrolling deviation time".

[0203] Therefore, in step S176, the photoelectric conversion element driving circuit 94A sets the cropping area according to the stroboscopic influence avoiding time. Then, the photoelectric conversion element driving circuit 94A shoots the subject in the set cropping area in the imaging area of ​​the photoelectric conversion element 92, and then transfers to step S178. In addition, in the present embodiment, the output image data based on the camera image data obtained by executing the processing of step S176 is still image data, but the technology of the present invention is not limited to this. For example, of course, the output image data based on the camera image data obtained by executing the processing of step S176 can also be used as dynamic image data for recording.

[0204] The cropping area is determined by the cropping coefficient determined according to the stroboscopic influence avoiding time. In step S176, the cropping coefficient is derived from the cropping coefficient derivation table 98 stored in the storage circuit 94E. The cropping coefficient derivation table 98 is a table in which the light amount peak period and the cropping coefficient are associated with each other. The cropping coefficient in the cropping coefficient derivation table 98 is a coefficient that specifies the imaging area in which the influence of line stroboscopic is avoided in the imaging area of ​​the photoelectric conversion element 92.

[0205] The processing circuit 94 derives the cropping factor corresponding to the light amount peak period calculated in step S172 from the cropping factor derivation table 98. The photoelectric conversion element driving circuit 94A sets a cropping area that avoids the influence of line strobe according to the cropping factor derived from the cropping factor derivation table 98, and captures the subject in the set cropping area. By executing the processing of step S176, as an example, Fig. 20 As shown, in the display device, the image area equivalent to the cropped area is located between the line strobes, which can suppress the line strobes from being reflected in the screen.

[0206] That is, by executing the process of step S176 , the subject is captured at a timing when the occurrence of line flicker is avoided and in a cutout area where line flicker does not appear in the display device.

[0207] In step S178, the processing circuit 94 generates still image data based on the captured image data obtained by capturing in step S176, outputs the generated still image data to the subsequent circuit 90, and the image capturing process is transferred to step S179. That is, in this step S178, the photoelectric conversion element driving circuit 94A outputs the captured image data obtained by capturing the cutout area in the image capturing area of ​​the photoelectric conversion element 92 to the AD conversion circuit 94B. The AD conversion circuit 94B digitizes the input captured image data and stores it in the memory 96. The image processing circuit 94C reads the captured image data from the memory 96, processes the read captured image data, generates still image data, and outputs the generated still image data to the output circuit 94D. The output circuit 94D outputs the output image data to the I / F 56 of the subsequent circuit 90 at the second frame rate. In addition, the still image data is an example of the "output image data" involved in the technology of the present invention.

[0208] In step S179 , the processing circuit 94 changes the first frame rate from the high frame rate to the low frame rate, and then ends the strobe avoidance imaging process.

[0209] Since the image pickup device 10 is affected by temperature changes and the clock is deviated, the actual stroboscopic generation cycle based on the frequency of the commercial power supply may deviate from the light intensity peak cycle calculated in step S172. Fig.23 As shown, the light source flicker characteristic, which is a flicker light source characteristic, sometimes differs after several hours. Although the light source flicker characteristic deviates in this way, if the display frame output from the imaging element 20 is directly displayed as a live preview image on the display device, line flicker may occur.

[0210] Therefore, in step S180, the processing circuit 94 determines whether the brightness difference confirmation time has come. The brightness difference confirmation time is a time predetermined as a time to confirm whether the brightness difference at the current time deviates from the brightness difference period characteristic curve G based on the brightness difference obtained by executing the processing of the previous steps S164 to S170.

[0211] The predetermined time is a time that is derived in advance from an actual machine test and / or computer simulation as a time when the clock of the imaging device 10 is affected by temperature changes and the line flicker appears in the screen of the display device. As the brightness difference confirmation time, for example, the time when the first predetermined time (for example, 30 minutes) has passed since the processing of the previous step S168 was executed, or the time when the second predetermined time (for example, 1 hour) has passed after the power of the imaging device 10 is turned on.

[0212] In step S180, if the brightness difference confirmation time has not arrived, it is determined as "No", and the stroboscopic avoidance imaging process is transferred to step S174. In step S180, if the brightness difference confirmation time has arrived, it is determined as "Yes", and the stroboscopic avoidance imaging process is transferred to step S181. In addition, the case where the brightness difference confirmation time has arrived is an example of "a case where a predetermined condition is satisfied" involved in the technology of the present invention.

[0213] In step S181 , the processing circuit 94 changes the first frame rate from the low frame rate to the high frame rate determined in step S156 , and the strobe-avoidance imaging process proceeds to step S182 .

[0214] In step S182 , the image processing circuit 94C acquires the brightness from the latest captured image data stored in the memory 96 , and then moves to step S184 .

[0215] In step S184 , the image processing circuit 94C acquires the brightness from the captured image data of the frame before the latest captured image data among the captured image data of the plurality of frames stored in the memory 96 , and the strobe avoidance imaging process is transferred to step S186 .

[0216] In step S186, the image processing circuit 94C calculates the brightness difference, and the stroboscopic avoidance imaging process shifts to step S188. The brightness difference calculated by executing the process of step S186 is a subtraction result of subtracting the brightness acquired in step S182 from the brightness acquired in step S184.

[0217] In step S188 , the image processing circuit 94C determines whether the luminance difference calculated in step S186 deviates from the luminance difference period characteristic curve diagram G based on the luminance difference obtained by executing the previous processes of steps S164 to S170 .

[0218] In step S188, if the brightness difference calculated in step S186 does not deviate from the brightness difference periodic characteristic curve graph G based on the brightness difference obtained by executing the previous processing of steps S164 to S170, it is determined as "No", and the stroboscopic avoidance imaging processing is transferred to step S174. That is, if the brightness difference calculated in step S186 is not on the brightness difference periodic characteristic curve graph G based on the brightness difference obtained by executing the previous processing of steps S164 to S170, it is determined as "No".

[0219] In step S188, if the brightness difference calculated in step S186 deviates from the brightness difference periodic characteristic curve graph G based on the brightness difference obtained by executing the previous processing of steps S164 to S170, it is determined as "yes", and the stroboscopic avoidance imaging processing is transferred to step S164. That is, if the brightness difference calculated in step S186 is on the brightness difference periodic characteristic curve graph G based on the brightness difference obtained by executing the previous processing of steps S164 to S170, it is determined as "yes".

[0220] exist Fig.16 In the imaging process shown in FIG. 118, the CPU 52 determines that the Fig.17 In the step S178 of the stroboscopic avoidance imaging process shown in FIG. 1 , the still image data output from the output circuit 94D is input to the I / F56 of the subsequent circuit 90. In step S118, if the still image data is not input to the I / F56 of the subsequent circuit 90, it is determined as "No", and the determination of step S118 is performed again. In step S118, if the still image data is input to the I / F56 of the subsequent circuit 90, it is determined as "Yes", and the imaging process is transferred to step S120.

[0221] In step S120, the CPU 52 acquires the still image data and performs various processing, and then transfers to step S108. The so-called "various processing" here includes, for example, the processing of outputting the still image data to the image processing unit 62. If the still image data is output to the image processing unit 62, for example, the image processing unit 62 performs signal processing on the still image data, and outputs the still image data that has been subjected to signal processing to an external device (not shown) via the external I / F 63. Here, as the so-called "external device", for example, a memory card, an SSD (Solid State Drive) or a USB (Universal Serial Bus) memory, a PC (Personal Computer), and a server can be cited.

[0222] exist Fig.16In step S110 shown, the CPU 52 ends the display of the live preview image and then ends the image capture process.

[0223] If executed Fig.16 The camera processing and Fig.17 The stroboscopic avoidance imaging process shown in the figure is executed as an example. Fig. 22 Process in the order shown.

[0224] Here, Fig. 22 First, in a state where the release button 25 is not pressed, a live preview image is displayed on the display device, and a first process is executed at a low frame rate in the processing circuit 94 of the imaging element 20 .

[0225] If the release button 25 is pressed halfway, AF and AE are performed. Even if the release button 25 is halfway pressed, a live preview image is displayed on the display device, and the first process is performed at a low frame rate in the processing circuit 94 of the imaging element 20.

[0226] If the release button 25 is in the fully pressed state, the processing circuit 94 performs the measurement detection process in addition to the first process. The first process and the measurement detection process are performed at the high frame rate determined in the process of step S156. Furthermore, even if the release button 25 is in the fully pressed state, the screen of the display device is not shielded, and a live preview image is displayed on the screen of the display device.

[0227] When the imaging element 20 completes the measurement detection process, it starts photographing after waiting for the light intensity to reach the peak value. Waiting for the light intensity to reach the peak value means, for example, waiting for Fig.17 The process continues until the determination of step S174 of the stroboscopic avoidance imaging process shown is "YES".

[0228] When the imaging device 20 completes photographing, a live preview image is displayed on the display device, and the first process is executed at a low frame rate in the processing circuit 94 of the imaging device 20 .

[0229] As described above, the processing circuit 94 and the memory 96 are built into the camera device 10. In the camera device 10, the subject is photographed at the first frame rate, and the camera image data obtained by the photographing is stored in the memory 96. Then, the image processing circuit 94C processes the camera image data stored in the memory 96. The output image data based on the camera image data is output to the subsequent circuit 90 by the output circuit 94D at the second frame rate. Then, the first frame rate is determined according to the stroboscopic generation cycle, and the processing circuit 94 detects the stroboscopic influence avoiding time based on the camera image data of a plurality of frames. Therefore, in the camera device 10, the shooting is performed at the stroboscopic influence avoiding time according to the first frame rate determined according to the stroboscopic generation cycle, so that the shooting avoiding the influence of the stroboscopic can be realized.

[0230] In the imaging device 10, the processing time of the measurement detection processing executed by the processing circuit 94 is determined according to the strobe generation cycle. Therefore, the imaging device 10 can appropriately determine the processing time required for the measurement detection processing compared to the case where the brightness difference between frames of the captured image data is not used.

[0231] Furthermore, the imaging device 10 uses the time when the brightness difference changes from a positive value to a negative value as the stroboscopic influence avoiding time. Therefore, the imaging device 10 can detect the time when the captured image is brightest as the stroboscopic influence avoiding time.

[0232] Furthermore, in the imaging device 10, the measurement detection process is terminated when the brightness difference changes from a positive value to a negative value twice. Therefore, the imaging device 10 can reduce the processing time required for the measurement detection process to the minimum necessary time.

[0233] Furthermore, in the imaging device 10, after the measurement detection process is performed, the first frame rate is set to a frame rate lower than the frame rate in the measurement detection process until the determination is "yes" in step S180 included in the strobe avoidance imaging process. Thus, the imaging device 10 can reduce power consumption compared to a case where the frame rate in the measurement detection process is applied to processes other than the measurement detection process.

[0234] Furthermore, in the imaging device 10, when the determination in step S180 included in the strobe avoidance imaging process is "YES", the processing circuit 94 restarts the measurement detection process. Therefore, the imaging device 10 can reduce power consumption compared to the case where the measurement detection process is always executed.

[0235] Then, in the imaging device 10 , if the determination in step S180 included in the strobe avoidance imaging process is “Yes” and the determination in step S188 is “Yes”, the measurement detection process is restarted. Thus, the imaging device 10 can avoid executing unnecessary measurement detection processes.

[0236] Then, the imaging device 10 captures the image when the brightness of the captured image reaches its peak (steps S174 and S176). Thus, the imaging device 10 can obtain a brighter captured image than when the image is captured at a different time than when the brightness of the captured image reaches its peak.

[0237] Then, in the imaging device 10, output image data obtained by capturing at the timing of the light amount peak is output from the output circuit 94D to the subsequent circuit 90. Therefore, the imaging device 10 can output output image data that is free from the influence of stroboscopic light.

[0238] Furthermore, in the imaging device 10, the display control unit performs control to display an image based on the output image data on the display device. Therefore, the imaging device 10 can display an image that avoids the influence of stroboscopic light on the display device.

[0239] Furthermore, the imaging device 10 captures images in the cropping area selected according to the cropping factor determined according to the stroboscopic influence avoidance timing. Thus, compared with the case of capturing images in the entire imaging area of ​​the photoelectric conversion element 92, the imaging device 10 can suppress the line stroboscopic image from being reflected in the image.

[0240] Furthermore, in the imaging device 10, the strobe generation cycle is predetermined as the strobe generation cycle caused by the flickering light source flickering by supplying AC power from the commercial power supply. Therefore, the imaging device 10 can reduce the man-hours for determining the strobe generation cycle compared to the case where the strobe generation cycle is not predetermined.

[0241] Furthermore, in the camera device 10, when the frequency represented by the power frequency information is 50 Hz, 10 ms is calculated as the stroboscopic generation period, and when the frequency represented by the power frequency information is 60 Hz, 8.33 ms is calculated as the stroboscopic generation period. Then, when the stroboscopic generation period is 10 ms, the high frame rate of the first frame rate is determined as the first high frame rate between the first high frame rate and the second high frame rate. When the stroboscopic generation period is 8.33 ms, the high frame rate of the first frame rate is determined as the second high frame rate between the first high frame rate and the second high frame rate. That is, the first frame rate increases as the stroboscopic generation period shortens. Thus, compared with the case where the first frame rate is fixed, the camera device 10 can improve the detection accuracy of the stroboscopic influence avoidance moment.

[0242] Furthermore, in the imaging device 10, the captured image data is image data obtained by capturing the subject in a rolling shutter manner. Therefore, the imaging device 10 can avoid the influence of stroboscopic light generated when capturing the subject in a rolling shutter manner.

[0243] Furthermore, in the imaging device 10, a stacked imaging element in which the processing circuit 94 and the memory 96 are stacked on the photoelectric conversion element 92 is adopted as the imaging element 20. Thus, compared with the case of using an imaging element of a type in which the storage unit is not stacked on the photoelectric conversion element 92, the imaging device 10 can improve the detection accuracy of the stroboscopic influence avoidance timing.

[0244] In addition, in the above-mentioned embodiment, the difference between the average brightness of the entire camera image of one frame and the average brightness of the entire camera image of another frame in adjacent frames is exemplified as the brightness difference, but the technology of the present invention is not limited to this. If the entire camera image of one frame is set as the calculation object of the brightness difference, the calculation accuracy of the brightness difference is affected by the movement of the subject and / or hand shaking, so the brightness of the entire camera image will vary greatly between frames. For example, Fig.24 As shown, the brightness difference C n and brightness difference C n+1 In this case, the accuracy of the brightness difference period characteristic curve G decreases, and accordingly, the detection accuracy of the flicker influence avoidance timing also decreases.

[0245] In order to reduce the influence of the subject's movement and / or hand shaking, the brightness difference of the local areas corresponding to each other between the frames of the camera image data can be calculated. Fig.25 As shown, a camera image of one frame is divided into two along the vertical direction, namely, a first segmented image and a second segmented image, and the brightness difference between the segmented images corresponding to each other in adjacent frames is calculated. Then, the brightness difference between the segmented images that are less affected by the movement of the subject and / or hand shaking is used in the first segmented image and the second segmented image. Fig.25 In the example shown, the brightness difference of the second divided image is less affected by the movement of the subject and / or hand shaking than the brightness difference of the first divided image, so the brightness difference of the second divided image is used.

[0246] Furthermore, the number of segmentation of the camera image may be more than three, the camera image may be segmented in the horizontal direction instead of the vertical direction, or the camera image may be segmented not only in the vertical direction but also in the horizontal direction. In this way, when there are factors that affect the calculation accuracy of the brightness difference in addition to the movement of the subject and / or hand shaking, it is effective to increase the number of segmented images. For example, when shooting in an environment with a very bright light source, it is difficult to calculate the brightness difference between frames due to the saturation of the pixel signal. Therefore, by increasing the number of segmented images, it is possible to ensure that the area to be calculated is within the range of 100%.

[0247] Furthermore, in the above-mentioned embodiment, the case of calculating the brightness difference between adjacent frames has been described, but the technology of the present invention is not limited to this. For example, since the camera image data of a plurality of frames is stored in the memory 96, the brightness difference between the brightness of the camera image data obtained by shooting two or more frames after the latest camera image data and the brightness of the latest camera image data can be calculated. Fig.11 In the example, instead of calculating the brightness difference of the image data between the first frame and the second frame, the brightness difference of the image data between the first frame and the third frame may be calculated. In this case, the image data of the first frame is an example of the “first image data” involved in the technology of the present invention, and the image data of the third frame is an example of the “second image data” involved in the technology of the present invention.

[0248] In addition, when calculating the brightness difference between the brightness of camera image data obtained by shooting more than two frames after the latest camera image data and the brightness of the latest camera image data, as in the above-mentioned embodiment, the processing time of the measurement detection processing is calculated as the time required for the brightness difference to change from a positive value to a negative value at least twice.

[0249] Furthermore, in the above embodiment, the cropping factor derivation table 98 is exemplified, but the technology of the present invention is not limited thereto. For example, the cropping factor may be calculated using a cropping factor derivation equation with the light intensity peak period as an independent variable and the cropping factor as a dependent variable.

[0250] Furthermore, although line strobes are exemplified in the above-mentioned embodiments, the technology of the present invention is not limited thereto, and the technology of the present invention can be applied to surface strobes that flicker in surface units.

[0251] Furthermore, in the above embodiment, a flashing light source that flashes according to the frequency of the commercial power supply is exemplified, but even when the camera device 10 is shooting in an environment where a flashing light source that flashes independently of the frequency of the commercial power supply flashes, the technology of the present invention can be applied. Fig.17The strobe avoidance imaging process shown does not require step S152 , and instead of the process of step S154 , for example, a process of receiving a strobe generation cycle via the touch panel 42 and / or the operation unit 54 may be applied.

[0252] Furthermore, in the above-described embodiment, from the viewpoint of reducing power consumption, the first frame rate is described by way of example in which the first frame rate is changed from a low frame rate to a high frame rate, but only the high frame rate may be set.

[0253] Furthermore, in the above embodiment, the time when the first predetermined time or the second predetermined time has passed is exemplified as the brightness difference confirmation time, but the technology of the present invention is not limited thereto. For example, the brightness difference confirmation time may be the time when the user issues an instruction to confirm the brightness difference via the touch panel 42 and / or the operation unit 54.

[0254] Furthermore, in the above-mentioned embodiment, as an example of "a situation in which a predetermined condition is satisfied" involved in the technology of the present invention, a situation in which a "yes" is determined in step S180 included in the stroboscopic avoidance imaging process, i.e., a situation in which the brightness difference confirmation time has arrived, has been cited, but the technology of the present invention is not limited to this. For example, in a case where a predetermined instruction is received by the touch panel 42 and / or the operating unit 54 as an instruction to start the processing after step S181 of the stroboscopic avoidance imaging process, the processing after step S181 of the stroboscopic avoidance imaging process can be executed. Furthermore, in a state in which the power of the camera device 10 is turned on, in a case in which the still image shooting instruction reaches a predetermined number of times (e.g., 200 times), the processing after step S181 can be executed.

[0255] Furthermore, in the above-mentioned embodiment, an example of a method of re-executing the processing after step S164 under the condition that the processing of steps S180 to S188 has been executed has been cited for explanation, but the technology of the present invention is not limited to this. For example, the processing after step S164 can be re-executed at the moment when the first specified time has passed after the previous processing of step S168 has been executed, or at the moment when the second specified time has passed after the power of the camera device 10 has been turned on. Furthermore, in the case where a predetermined instruction is received by the touch panel 42 and / or the operating unit 54 as an instruction to restart the processing after step S164 of the strobe avoidance camera processing, the processing after step S164 of the strobe avoidance camera processing can be re-executed. In addition, when the power of the camera device 10 is turned on, the processing after step S164 can be executed when the instruction for still image shooting reaches the predetermined number of times.

[0256] Furthermore, in the above embodiment, the processing circuit 94 implemented by ASIC is exemplified, but the above-mentioned strobe avoidance imaging process may be implemented by a software structure of a computer.

[0257] In this case, for example, Fig.26 As shown, a program 600 is stored in a storage medium 700, and the program 600 is used to enable a computer 20A built into an imaging element 20 to perform the above-mentioned stroboscopic avoidance imaging process. The computer 20A includes a CPU 20A1, a ROM 20A2, and a RAM 20A3. Then, the program 600 of the storage medium 700 is installed in the computer 20A, and the CPU 20A1 of the computer 20A performs the above-mentioned stroboscopic avoidance imaging process according to the program 600. Here, one CPU is illustrated as the CPU 20A1, but the technology of the present invention is not limited to this, and a plurality of CPUs may be used instead of the CPU 20A1. That is, the above-mentioned imaging process and / or the stroboscopic avoidance imaging process may be performed by one processor or a plurality of physically separated processors.

[0258] In addition, as an example of the storage medium 700, any portable storage medium such as an SSD or a USB memory can be cited.

[0259] The program 600 may be stored in a storage unit such as another computer or a server device connected to the computer 20A via a communication network (not shown), and the program 600 may be downloaded in response to a request from the imaging device 10. In this case, the downloaded program 600 is executed by the computer 20A.

[0260] Furthermore, the computer 20A may be provided outside the imaging element 20 . In this case, the computer 20A may control the processing circuit 94 according to the program 600 .

[0261] As hardware resources for executing the various processes described in the above embodiments, various processors shown below can be used. Here, as various processes described in the above embodiments, video processing and strobe avoidance video processing can be cited. As a processor, for example, a general-purpose processor, i.e., a CPU, can be cited. As described above, the processor functions as a hardware resource for executing various processes involved in the technology of the present invention by executing software, i.e., a program. Furthermore, as a processor, for example, a dedicated circuit as a processor can be cited. The processor has a circuit structure specially designed for executing specific processes, such as FPGA, PLD, or ASIC. A memory is also built into or connected to any processor, and any processor also executes various processes by using the memory.

[0262] The hardware resources for executing various processing involved in the technology of the present invention may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (e.g., a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware resources for executing various processing involved in the technology of the present invention may be one processor.

[0263] As an example of a system composed of one processor, first, there is the following method: as represented by computers such as clients and servers, a processor is composed of a combination of one or more CPUs and software, and the processor functions as a hardware resource for executing various processes involved in the technology of the present invention. Second, there is the following method: as represented by SoC (System-on-a-chip), a processor that uses one IC chip to implement the functions of the entire system including multiple hardware resources for executing various processes involved in the technology of the present invention. In this way, various processes involved in the technology of the present invention are implemented by using one or more of the above-mentioned various processors as hardware resources.

[0264] Furthermore, as the hardware configuration of these various processors, more specifically, a circuit combining circuit elements such as semiconductor elements can be used.

[0265] Furthermore, in the above-mentioned embodiment, a lens-interchangeable camera is exemplified as the imaging device 10, but the technology of the present invention is not limited thereto. Fig. 27 As an example, Fig. 27 The smart device 900 shown is an example of the camera device involved in the technology of the present invention. The imaging element 20 described in the above embodiment is mounted in the smart device 900. Even the smart device 900 constructed in this way can obtain the same function and effect as the camera device 10 system described in the above embodiment. In addition, it is not limited to the smart device 900, and the technology of the present invention can also be applied to a PC or a wearable terminal device.

[0266] Furthermore, in the above embodiment, the first display 40 and the second display 80 are exemplified as display devices, but the technology of the present invention is not limited thereto. For example, a separate display attached to the camera body 12 may be used as the "display unit" involved in the technology of the present invention.

[0267] Furthermore, the imaging process and the strobe-avoiding imaging process described in the above-mentioned embodiment are merely examples, and therefore, it is of course possible to delete unnecessary steps, add new steps, or switch the processing order without departing from the scope of the subject matter.

[0268] The recorded contents and illustrated contents shown above are detailed descriptions of the parts involved in the technology of the present invention, and are only an example of the technology of the present invention. For example, the descriptions related to the above-mentioned structure, function, action and effect are descriptions related to an example of the structure, function, action and effect of the parts involved in the technology of the present invention. Therefore, without departing from the scope of the main purpose of the technology of the present invention, it is of course possible to delete unnecessary parts of the recorded contents and illustrated contents shown above, or to add new elements, or to replace them. In addition, in order to avoid complication and to make it easy to understand the parts involved in the technology of the present invention, in the recorded contents and illustrated contents shown above, descriptions related to technical common sense, etc. that do not need to be specifically explained in terms of the technology that can implement the present invention are omitted.

[0269] In this specification, "A and / or B" has the same meaning as "at least one of A and B". That is, "A and / or B" means that there may be only A, only B, or a combination of A and B. Furthermore, in this specification, when "and / or" is added to express three or more items, the same concept as "A and / or B" may also be applied.

[0270] All documents, patent applications, and technical standards described in this specification are incorporated herein by reference to the same extent as if each document, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An imaging element, mounted on a camera device, comprising: a storage unit that stores captured image data obtained by capturing an object at a first frame rate and is built into the imaging element; a processing unit that performs processing based on the captured image data stored in the storage unit and is built into the imaging element; an output unit that outputs output image data based on the captured image data to the outside of the imaging element at a second frame rate and is built into the imaging element, The first frame rate is a frame rate higher than the second frame rate and is determined according to a generation cycle of the strobe. The processing unit detects a stroboscopic influence avoiding timing for avoiding the influence of the stroboscopic light on the imaging of the imaging element based on the captured image data of a plurality of frames, The processing includes a measurement detection processing of measuring a brightness difference between frames of the captured image data and detecting the stroboscopic influence avoiding timing based on the measured brightness difference. The processing time required in the measurement detection process is determined according to the generation cycle, The captured image data is classified between the frames into first captured image data and second captured image data obtained by capturing the image after the first captured image data. The brightness difference is a subtraction result of subtracting the brightness of the second camera image data from the brightness of the first camera image data. The stroboscopic influence avoiding time is the time when the brightness difference changes from a positive value to a negative value.

2. The imaging element according to claim 1, wherein The processing unit terminates the measurement detection process under the condition that the brightness difference changes from the positive value to the negative value twice.

3. The imaging element according to claim 1 or 2, wherein: The second captured image data is image data obtained by capturing the image two or more frames after the first captured image data.

4. The imaging element according to claim 1 or 2, wherein: After the measurement and detection process is performed, the first frame rate is set to a frame rate lower than the frame rate in the measurement and detection process until a predetermined condition is satisfied.

5. The imaging element according to claim 4, wherein: The processing unit performs the measurement detection process again when the predetermined condition is satisfied.

6. The imaging element according to claim 5, wherein: When the predetermined condition is satisfied and the luminance difference measured by the processing unit is changed from the luminance difference measured within the processing time, the measurement and detection process is performed again.

7. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The brightness difference is a brightness difference between local areas corresponding to each other between the frames of the captured image data.

8. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The stroboscopic influence avoiding timing is a timing at which the brightness of the image represented by the captured image data reaches a peak.

9. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The output image data is image data based on the captured image data, and the captured image data is obtained by capturing the image at the stroboscopic effect avoiding timing.

10. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The strobe is a line strobe, The subject is photographed within an imaging area selected according to a cropping factor determined based on the stroboscopic effect avoiding timing.

11. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The generation cycle is set in advance as a generation cycle of stroboscopic light caused by the light source flickering due to the supply of alternating current from the commercial power supply.

12. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The first frame rate increases as the generation period decreases.

13. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The captured image data is image data obtained by photographing the subject in a rolling shutter manner.

14. The imaging element according to any one of claims 1, 2, 5, and 6, wherein: The imaging element is a stacked type imaging element having a photoelectric conversion element, and the storage section is stacked on the photoelectric conversion element.

15. A camera device, comprising: The imaging element according to any one of claims 1, 2, 5, and 6; and The control unit controls a display unit to display an image based on the output image data output by the output unit included in the imaging element.

16. An image data processing method, which is an image data processing method for an imaging element mounted on an imaging device, wherein the imaging element is built with: a storage unit that stores camera image data obtained by photographing a subject at a first frame rate; a processing unit that performs processing based on the camera image data stored in the storage unit; and an output unit that outputs output image data based on the camera image data to the outside at a second frame rate, wherein: The first frame rate is set to a frame rate higher than the second frame rate, and the first frame rate is determined according to a generation period of the stroboscopic light. The processing unit detects a stroboscopic influence avoiding timing for avoiding the influence of the stroboscopic light on the imaging of the imaging element based on the captured image data of a plurality of frames, The processing includes a measurement detection processing of measuring a brightness difference between frames of the captured image data and detecting the stroboscopic influence avoiding timing based on the measured brightness difference. The processing time required in the measurement detection process is determined according to the generation cycle, The captured image data is classified between the frames into first captured image data and second captured image data obtained by capturing the image after the first captured image data. The brightness difference is a subtraction result of subtracting the brightness of the second camera image data from the brightness of the first camera image data. The stroboscopic influence avoiding time is the time when the brightness difference changes from a positive value to a negative value.

17. A storage medium storing a program for causing a computer to function as a processing unit and an output unit included in an imaging element mounted on an imaging device, wherein the imaging element has: a storage unit storing camera image data obtained by photographing a subject at a first frame rate; the processing unit performing processing based on the camera image data stored in the storage unit; and the output unit outputting output image data based on the camera image data to the outside at a second frame rate, wherein: The first frame rate is a frame rate higher than the second frame rate and is determined according to a generation cycle of the strobe. The processing unit detects a stroboscopic influence avoiding timing for avoiding the influence of the stroboscopic light on the imaging of the imaging element based on the captured image data of a plurality of frames, The processing includes a measurement detection processing of measuring a brightness difference between frames of the captured image data and detecting the stroboscopic influence avoiding timing based on the measured brightness difference. The processing time required in the measurement detection process is determined according to the generation cycle, The captured image data is classified between the frames into first captured image data and second captured image data obtained by capturing the image after the first captured image data. The brightness difference is a subtraction result of subtracting the brightness of the second camera image data from the brightness of the first camera image data. The stroboscopic influence avoiding time is the time when the brightness difference changes from a positive value to a negative value.

Images