Imaging device and imaging control program

Abstract

Provided is an imaging device and the like that can automatically perform imaging processing while avoiding the occurrence of moire. [Solution] The imaging device includes a camera unit having an imaging element, a discrimination unit that determines whether or not a moiré area exists in an image formed based on the output of the imaging element, and an imaging processing unit that, when the discrimination unit determines that a moiré area exists in the image, switches the optical conditions related to imaging of the camera unit and continues the imaging process.

Description

[Technical field]

[0001] The present invention relates to an imaging device and an imaging control program. [Background technology]

[0002] Imaging devices that generate image data based on image signals output from an imaging element are widely used. The imaging element generally includes a large number of pixels regularly arranged vertically and horizontally, and outputs an image signal corresponding to a subject image formed on a pixel plane by photoelectrically converting the subject image formed on the pixel plane by each pixel. Here, if at least a part of the subject image formed through an optical lens has a spatial frequency equal to or greater than half the Nyquist frequency defined by the pixel pitch of the imaging element, a moire pattern will occur in the corresponding area of ​​an image generated from the output image signal. The moire pattern appears in the generated image as a non-existent stripe pattern or a false color different from the actual color. Therefore, a technology that notifies a user in advance when such a moire pattern is expected to occur in a captured image is being developed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-10881 A Summary of the Invention [Problem to be solved by the invention]

[0004] Camera units are built into smartphones, and smartphones also function as imaging devices. As smartphones become more widespread, users who do not have much knowledge about cameras are increasingly enjoying taking pictures. In this situation, there is a demand for imaging devices (and therefore smartphones) that automatically perform imaging processing to avoid moire, without requiring users to have any particular knowledge about cameras.

[0005] The present invention has been made to solve such problems, and provides an imaging device and the like that can automatically perform imaging processing that avoids the occurrence of moire. [Means for solving the problem]

[0006] An imaging device according to a first aspect of the present invention comprises a camera unit having an imaging element, a discrimination unit that discriminates whether or not a moiré area exists in an image formed based on the output of the imaging element, and an imaging processing unit that, when the discrimination unit determines that a moiré area exists in the image, switches optical conditions related to imaging of the camera unit and continues the imaging process.

[0007] An imaging control program in a second aspect of the present invention causes a computer to execute a determination step of determining whether or not a moiré area exists in an image formed based on the output of an imaging element possessed by a camera unit, and an imaging processing step of switching optical conditions related to imaging of the camera unit and continuing the imaging processing if it is determined in the determination step that a moiré area exists in the image. Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging device and the like that can automatically perform imaging processing while avoiding the occurrence of moire. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the appearance of an imaging device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a main hardware configuration of the imaging apparatus. [Diagram 3] FIG. 1 is a diagram showing an example of a scene in which a moire image occurs. [Figure 4] 1A and 1B are diagrams for explaining a pixel arrangement of an image sensor and a method for determining moire. [Diagram 5] FIG. 11 is a diagram illustrating a first switching process. [Figure 6] FIG. 11 is a diagram illustrating a second switching process. [Figure 7] FIG. 11 is a flowchart showing a processing procedure for moire avoidance processing. [Figure 8] FIG. 11 is a diagram showing a main hardware configuration of an imaging device according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a flowchart showing a processing procedure for moire avoidance processing. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0011] An imaging device according to a first embodiment of the present invention and an imaging device according to a second embodiment of the present invention will be described in order. FIG. 1 is a diagram showing the appearance of an imaging device 100 according to a first embodiment of the present invention. In particular, FIG. 1(A) is a diagram mainly showing the first surface side of the imaging device 100, and FIG. 1(B) is a diagram mainly showing the second surface side opposite to the first surface side. The imaging device 100 according to the present embodiment is a so-called smartphone, in other words, a smartphone that also functions as an imaging device. In the following, the imaging function related to the present invention among the functions of the smartphone will be described, and the description of other functions as a smartphone, such as the use of image data generated by imaging, will be omitted. Note that in this embodiment, the imaging device 100 will be described using a smartphone as an example, but it may of course be an imaging device as a standalone camera, or may be a device incorporated in a tablet terminal or the like and equipped with an imaging function.

[0012] The imaging device 100 includes a first camera unit 110 and a second camera unit 120 arranged in the same direction on the first surface side. The first camera unit 110 is a camera unit for generating a wide-angle image. The second camera unit 120 is a camera unit for generating a telephoto image. When a user wants to obtain a wide-angle image, the user specifies the first camera unit to capture an image, and when a user wants to obtain a telephoto image, the user specifies the second camera unit to capture an image. In the figure, the first camera unit 110 and the second camera unit 120 are arranged parallel to the long side of the imaging device 100, but the arrangement of the two camera units is not limited to this, and they may be arranged, for example, along a straight line obliquely intersecting the long side. In addition, the arrangement of the first camera unit 110 and the second camera unit 120 may be reversed from the positions shown in the figure.

[0013] The imaging device 100 includes a display 130 on the second surface side. The display 130 is a display device that employs, for example, an organic EL (Electro Luminescence) panel, and displays a real-time image of a subject before imaging (live view display) and an image after imaging. Note that a camera unit for self-portraits, independent of the first camera unit 110 and the second camera unit 120, may be provided on the second surface side.

[0014] A shutter button 161 is provided on the side of the imaging device 100. The user can instruct the imaging device 100 to capture an image by pressing the shutter button 161. A touch panel 162 is also provided superimposed on the display 130. Instead of pressing the shutter button 161, the user can also instruct the imaging device 100 to capture an image by tapping the shutter button displayed on the display 130. The user can also designate a certain area including any point on the subject image displayed in live view as the focusing area by tapping the any point on the subject image. In addition, the user can switch between the first camera unit 110 and the second camera unit 120 or select a displayed menu item by performing a contact operation such as tapping.

[0015] 2 is a diagram showing the main hardware configuration of the imaging device 100. In addition to the above-mentioned first camera unit 110, second camera unit 120, and display 130, the imaging device 100 is made up of a system control unit 150 that controls these units, and peripheral elements that cooperate with the system control unit 150.

[0016] The first camera unit 110 is a camera unit for generating a wide-angle image as described above, and mainly includes a first optical system 111, a first driving mechanism 112, a first image sensor 113, and a first analog front end (AFE) 114. The first optical system 111 is an optical system for forming an image of an incident subject light flux on an image sensor surface of the first image sensor 113. Although the first optical system 111 is shown as a single lens in the figure, it is generally composed of multiple lenses, at least some of which are focusing lenses that can advance and retreat along the optical axis direction. The first driving mechanism 112 is a driving mechanism for moving the focusing lens of the first optical system 111 along the optical axis direction, and includes an actuator that operates according to instructions from the system control unit 150.

[0017] The first imaging element 113 is, for example, a CMOS image sensor. The first imaging element 113 has a color filter employing a Bayer array, and each pixel constituting the first imaging element 113 is classified as an R (red) pixel, a G (green) pixel, or a B (blue) pixel. The first imaging element 113 passes an image signal, which is an output signal, to the first AFE 114 in accordance with an instruction from the system control unit 150. The first AFE 114 adjusts the level of the image signal according to a gain instructed by the system control unit 150, A / D converts it to digital data, and passes it to the work memory 141.

[0018] The second camera unit 120 is a camera unit for generating a telephoto image as described above, and mainly includes a second optical system 121, a second driving mechanism 122, a second image sensor 123, and a second analog front end (AFE) 124. The second optical system 121 is an optical system for forming an image of an incident subject light flux on an image sensor of the second image sensor 123. Although it is shown as one lens in the figure, the second optical system 121 is generally composed of multiple lenses, similar to the first optical system 111, at least a part of which is a focusing lens that can be advanced and retreated along the optical axis direction. However, the second optical system 121 has a longer focal length than the first optical system 111 and has a different lens configuration from the first optical system 111. The second driving mechanism 122 is a driving mechanism for moving the focusing lens of the second optical system 121 along the optical axis direction, and includes an actuator that operates according to an instruction from the system control unit 150. In the first embodiment, both the first optical system 111 and the second optical system 121 are single-focus optical systems with fixed focal lengths.

[0019] The second imaging element 123 is, for example, a CMOS image sensor. The second imaging element 123 has a color filter employing a Bayer array like the first imaging element 113, and each pixel constituting the first imaging element 113 is classified into any one of R (red) pixels, G (green) pixels, and B (blue) pixels. The second imaging element 123 passes an image signal, which is an output signal, to the second AFE 124 in accordance with an instruction from the system control unit 150. The second AFE 124 adjusts the level of the image signal according to a gain instructed by the system control unit 150, A / D converts it to digital data, and passes it to the work memory 141. The first imaging element 113 and the second imaging element may employ the same type of imaging element, or may employ different imaging elements having, for example, different pixel pitches.

[0020] The system control unit 150 is a processor (CPU: Central Processing Unit) that directly or indirectly controls each element constituting the imaging device 100. The system control unit 150 plays the role of various function control units according to the control program being executed. For example, when executing focus control of the first camera unit 110 and the second camera unit 120, the system control unit 150 functions as a focus control unit, and when displaying a captured image on the display 130, the system control unit 150 functions as a display control unit. In particular, the system control unit 150 can function as a discrimination unit 151 and an imaging processing unit 152 according to the processing instructed by the imaging control program. The discrimination unit 151 mainly discriminates whether or not a moire region exists in an active image formed based on the output of an imaging element (either the first imaging element 113 or the second imaging element 123) of an active unit selected by a user and in a usable state among the first camera unit 110 and the second camera unit 120. The imaging processing unit 152 mainly switches the active unit from one currently designated among the first camera unit 110 and the second camera unit 120 to the other not designated, and continues the imaging processing. The specific processing will be described later.

[0021] The imaging device 100 mainly includes a work memory 141, an image processing unit 142, an operation unit 160, a recording unit 170, and a communication interface 180 as peripheral elements that cooperate with the system control unit 150. The work memory 141 is a volatile high-speed memory, and is configured, for example, by a static random access memory (SRAM). The work memory 141 receives pixel data that has been converted sequentially from the first AFE 114 and the second AFE 124, and transfers the data to the image processing unit 142 by consolidating the data into frame data for one frame. The work memory 141 is also appropriately used as a temporary storage area in a processing stage in which the image processing unit 142 performs image processing and in a processing stage in which the system control unit 150 performs focusing processing and the like.

[0022] Image processing unit 142 is formed of, for example, an ASIC (Application Specific Integrated Circuit) that exclusively executes image processing, and performs various image processing such as interpolation processing on the received frame data to generate image data in accordance with a predetermined format. If the generated image data is for recording, it is recorded in recording unit 170, and if it is for display, it is displayed on display 130. Furthermore, when discrimination unit 151 determines whether or not a moiré region exists in the active image, image processing unit 142 executes extraction processing for extracting a moiré region from the active image. Specifics of the extraction processing will be described later.

[0023] The operation unit 160 is an input device including a shutter button 161 and a touch panel 162, and is a member that is operated by the user to give instructions to the imaging device 100. If the imaging device 100 accepts voice input, the operation unit 160 may also include a microphone. The recording unit 170 is a non-volatile memory, and is configured, for example, by an SSD (Solid State Drive). In addition to recording image data generated by imaging, the recording unit 170 holds constants, variables, setting values, control programs, and the like required during operation of the imaging device 100. The communication interface 180 may include a communication unit for a 5G line or a wireless LAN. The communication interface 180 is used, for example, when transferring the generated image data to an external device.

[0024] A typical imaging element used in an imaging device includes many pixels regularly arranged vertically and horizontally, similar to the first imaging element 113 and the second imaging element 123 in this embodiment, and outputs an image signal corresponding to a subject image formed on a pixel plane by photoelectrically converting the subject image formed on the pixel plane by each pixel. Here, if at least a part of the subject image formed through an optical lens has a spatial frequency equal to or greater than half the Nyquist frequency defined by the pixel pitch of the imaging element, a moire pattern will be generated in the corresponding region of an image generated from the output image signal. The moire pattern appears in the generated image as a non-existent false pattern or a false color different from the actual color.

[0025] FIG. 3 is a diagram showing an example of a scene in which a moire image occurs when imaging is performed using the imaging device 100. Specifically, the figure shows a state in which a user is imaging a television image 901 displayed on a television display 900 with the imaging device 100, and an active image 301 including a moire region 302 is displayed as a live view image on the display 130. Note that the active image 301 is an image formed from the output of the imaging element of the active unit selected by the user and in use from among the first camera unit 110 and the second camera unit 120 as described above. The active image 301 is displayed on the display 130 as a live view image at, for example, 60 fps. When the shutter button 161, which is a physical switch, is pressed or the shutter button 390 displayed superimposed on the active image 301 is tapped, the imaging processing unit 152 converts the live view image displayed at that timing or the image captured at that timing into image data and stores it in the recording unit 170.

[0026] As shown in the figure, active image 301 includes moiré region 302 in which a distorted lattice-like false pattern not present in television image 901 appears. Television display 900 has pixels that emit light in, for example, one of RGB, regularly arranged, and television image 901 formed by the light emission can cause moiré in active image 301. The occurrence of moiré is influenced by the spatial frequency of the optical image of the subject formed on the imaging element, as well as the pattern and brightness of television image 901. Note that, although an example in which television image 901 is used as the subject will be described in this embodiment, the subject is of course not limited to this, and moiré can occur when imaging a subject that includes a repeating pattern, such as the fibers of clothing or the wall surface of a building.

[0027] The discrimination unit 151 discriminates whether or not a moiré region exists in the active image 301 formed from the image signal output by the imaging element of the active unit. Fig. 4 is a diagram for explaining the pixel arrangement of the first imaging element 113 and a method for discriminating between moiré. Note that the pixel arrangement of the second imaging element 123 is the same as that of the first imaging element 113, and therefore the method for discriminating between moiré for an image formed based on the output of the second imaging element 123 is also the same.

[0028] As described above, the first imaging element 113 is composed of R, G, and B pixels corresponding to the Bayer array. In this embodiment, attention is focused on G pixels that occupy two of the four pixels, two by two, that form the repeating unit in the Bayer array, and each G pixel is assigned a code G00 to G33 as shown in the figure. In this case, if each pixel value is represented as V(Gmn), the evaluation value E of a unit area of ​​four pixels by four pixels that includes eight G pixels is E={|(V(G02)+V(G22))-(V(G11)+V(G13))|+|(V(G11)+V(G31))-(V(G20)+V(G22))|} / 4 If the evaluation value E is greater than a preset threshold, the unit region is determined to be a moiré unit region. The moiré region 302 shown in Fig. 3 is a collection region of moiré unit regions determined in this way.

[0029] When determining that a collection area of ​​moiré unit areas having a size equal to or larger than a preset size exists in active image 301, discrimination unit 151 determines that a moiré area exists in active image 301. Note that the method for discriminating a moiré area is not limited to this, and other known methods may be adopted.

[0030] Next, a description will be given of the processing of imaging processing unit 152 when discrimination unit 151 determines that a moiré region exists in active image 301. First, a description will be given of the processing of imaging processing unit 152 when the active unit is second camera unit 120 and active image 301 is telephoto image 320 formed based on the output of second imaging element 123.

[0031] FIG. 5 is a diagram for explaining a first switching process for switching the active unit from the second camera unit 120 to the first camera unit 110. When the active unit is the second camera unit 120, the active image 301 is a telephoto image 320. When the discrimination unit 151 discriminates that the telephoto image 320 includes a moire region 323, the imaging processing unit 152 switches the active unit from the second camera unit 120 to the first camera unit 110. Here, "switching" may be automatic switching, or switching may be actively triggered by a user. Then, the work memory 141 receives frame data obtained by converting an image signal output by the first imaging element 113 by the first AFE 114, and the image processing unit 142 receives the frame data and generates a wide-angle image 310. Furthermore, the image processing unit 142 cuts out an image area corresponding to the angle of view of the second optical system 121, that is, an image area corresponding to the entire area of ​​the telephoto image 320, from the generated wide-angle image 310 to generate a cut-out image 311. In addition, the cut-out image here may be an image obtained by fusing an image area of ​​a telephoto image generated by a telephoto camera that corresponds to the angle of view of a wide-angle image with an image area other than the above-mentioned image area generated by a wide-angle camera.

[0032] The cut-out image 311 is displayed as the active image 301 on the display 130. That is, when the telephoto image 320 includes a moire region 323, the active unit is automatically switched, and the cut-out image 311 cut out from the wide-angle image 310 is displayed on the display 130 as the active image 301. When the second camera unit 120 is switched to the first camera unit 110, the optical system forming the subject image is switched to the first optical system 111 different from the second optical system 121, that is, the optical conditions related to imaging are switched. Then, even if moire occurs in the relationship between the spatial frequency of the subject image formed on the imaging surface of the second imaging element 123 and the pixel pitch of the second imaging element 123, it can be expected that moire will not occur in the relationship between the spatial frequency of the subject image formed on the imaging surface of the first imaging element 113 and the pixel pitch of the first imaging element 113. In other words, unless the relationship between the spatial frequency of the subject image formed on the imaging surface of first imaging element 113 and the pixel pitch of first imaging element 113 matches the condition for generating moiré, no moiré region will exist in cropped image 311. In other words, in cropped image 311 shown in the figure, moiré does not appear in moiré-corresponding region 313 that corresponds to moiré region 323 in telephoto image 320.

[0033] When the shutter button 161 is pressed or the shutter button 390 is tapped while the cropped image 311 is displayed on the display 130, the image capture processing unit 152 converts the cropped image 311 displayed at that timing, or the cropped image 311 cropped from the wide-angle image 310 captured at that timing, into image data and stores it in the recording unit 170. When such a first switching process is performed automatically, the user can obtain image data of a desired image that does not include a moire region without being particularly aware of it. In this embodiment, the image process is performed to form the wide-angle image 310 and then crop the cropped image 311. However, the first AFE 114 may selectively A / D convert the image signal for forming the cropped image 311 from among the image signals output by the first image sensor 113 and transfer it to the work memory 141.

[0034] Next, a description will be given of the processing of imaging processing unit 152 when the active unit is first camera unit 110 and it is determined that a moiré region exists in wide-angle image 310, which is active image 301. Fig. 6 is a diagram illustrating a second switching processing for switching the active unit from first camera unit 110 to second camera unit 120.

[0035] When the active unit is first camera unit 110, active image 301 is wide-angle image 310. When discrimination unit 151 determines that wide-angle image 310 includes moiré region 314, imaging processing unit 152 switches the active unit from first camera unit 110 to second camera unit 120. Then, work memory 141 receives frame data obtained by converting an image signal output by second imaging element 123 by second AFE 124, and image processing unit 142 receives the frame data and generates telephoto image 320.

[0036] The telephoto image 320 is displayed as the active image 301 on the display 130. At this time, since it is not possible to extract an image area corresponding to the angle of view of the first optical system 111, i.e., an image area corresponding to the entire area of ​​the wide-angle image 310, from the telephoto image 320, the entire area of ​​the generated telephoto image 320 is displayed as is. Then, the active image 301 displayed as a live view image will be switched to an image with a different angle of view at a certain point, which may cause a sense of incongruity to the user. Therefore, the imaging processing unit 152 executes a notice regarding the adjustment of the angle of view to the user. Specifically, a notice display 325 is displayed superimposed on the telephoto image 320. The notice display 325 includes a notice such as "Moire was detected, so the lens was switched to the telephoto lens."

[0037] When a certain time has elapsed since the start of the notification, the imaging processor 152 removes the notification display 325. When the first camera unit 110 is switched to the second camera unit 120 in this way, the optical system forming the subject image is switched to the second optical system 121 different from the first optical system 111, that is, the optical conditions related to imaging are switched. Then, even if moire occurs in the relationship between the spatial frequency of the subject image formed on the imaging surface of the first imaging element 113 and the pixel pitch of the first imaging element 113, it can be expected that moire will not occur in the relationship between the spatial frequency of the subject image formed on the imaging surface of the second imaging element 123 and the pixel pitch of the second imaging element 123. In other words, unless the relationship between the spatial frequency of the subject image formed on the imaging surface of the second imaging element 123 and the pixel pitch of the second imaging element 123 matches the condition that generates moire, no moire area will exist in the telephoto image 320. That is, in the telephoto image 320 shown in the figure, moire does not appear in a moire corresponding region 324 that corresponds to the moire region 314 in the wide-angle image 310 .

[0038] When the shutter button 161 is pressed or the shutter button 390 is tapped while the telephoto image 320 is displayed on the display 130 in this manner, the image capturing processing unit 152 converts the telephoto image 320 displayed at that time, or the telephoto image 320 captured again at that time, into image data and stores it in the recording unit 170. When such a second switching process is performed automatically, the user recognizes that the image has been switched to the telephoto image 320, and can obtain image data of the desired image that does not include a moiré region.

[0039] Next, a procedure for the moiré avoidance process that is periodically executed while a live view image is being displayed will be described. Fig. 7 is a flow diagram showing the procedure for the moiré avoidance process. The flow starts at a timing that matches the set cycle while a live view image is being displayed.

[0040] In step S101, discrimination unit 151 discriminates whether or not a moiré region exists in active image 301. If it is determined that no moiré region exists, the moiré avoidance process for this period ends. If it is determined that a moiré region exists, the process proceeds to step S102.

[0041] In step S102, the imaging processing unit 152 confirms whether the currently active unit is the second camera unit 120 or the first camera unit 110. If it is confirmed that it is the second camera unit 120, the imaging processing unit 152 proceeds to step S103, and if it is confirmed that it is the first camera unit 110, the imaging processing unit 152 proceeds to step S106.

[0042] In step S103, imaging processing unit 152 switches the active unit to first camera unit 110. Once the active unit has been switched to first camera unit 110, the process proceeds to step S104, where work memory 141 receives the frame data converted by first AFE 114, and image processing unit 142 receives the frame data and generates wide-angle image 310. Furthermore, image processing unit 142 generates cropped image 311 by cropping an image area corresponding to the entire area of ​​telephoto image 320 from generated wide-angle image 310.

[0043] The process proceeds to step S105, where imaging processing unit 152 displays cropped image 311 generated by image processing unit 142 on display 130, thereby completing a series of moire avoidance processes. Note that after live view display using cropped image 311 has started, if shutter button 161 is pressed or shutter button 390 is tapped to instruct imaging and recording, and the recording process is completed, the live view display using cropped image 311 may be ended and the active unit may be returned to the original second camera unit 120.

[0044] When the process proceeds from step S102 to step S106, the imaging processing unit 152 switches the active unit to the second camera unit 120. When the active unit is switched to the second camera unit 120, the process proceeds to step S107, where the work memory 141 receives the frame data converted by the second AFE 124, and the image processing unit 142 receives the frame data and generates the telephoto image 320. The imaging processing unit 152 displays the telephoto image 320 generated by the image processing unit 142 on the display 130 with the notification display 325 superimposed thereon. Then, when a predetermined time has elapsed, in step S108, the notification display 325 is removed, and a series of moire avoidance processes is terminated. Note that after the live view display of the telephoto image 320 is started, when the shutter button 161 is pressed or the shutter button 390 is tapped to instruct image recording and the recording process is completed, the live view display of the telephoto image 320 may be terminated, and the active unit may be returned to the original first camera unit 110.

[0045] Next, an imaging device according to a second embodiment of the present embodiment will be described. FIG. 8 is a diagram showing a main hardware configuration of an imaging device 100' according to a second embodiment of the present embodiment. The imaging device 100' according to the second embodiment is also a so-called smartphone like the imaging device 100 according to the first embodiment, but unlike the imaging device 100, it is equipped with a single camera unit 110' having an optical system 111' as a so-called zoom lens whose focal length can be changed. In the following description, elements similar to those of the imaging device 100 of the first embodiment will be assigned the same reference numerals and their description will be omitted unless otherwise specified.

[0046] Camera unit 110' mainly includes optical system 111', drive mechanism 112', first image sensor 113, and first AFE 114. As described above, in this embodiment, imaging device 100' includes a single camera unit 110', but the image sensor and analog front end functions of camera unit 110' are substantially similar to those of first camera unit 110 in the first embodiment, and therefore, for convenience, they will be referred to as first image sensor 113 and first AFE 114 and described here as they are.

[0047] The optical system 111' is an optical system for forming an image of the incident subject light beam on the imaging surface of the first imaging element 113. The optical system 111' includes a focus lens group 111a and a variable magnification lens group 111b. In the figure, the focus lens group 111a and the variable magnification lens group 111b are each shown as a single lens arranged in front of and behind each other for convenience, but each lens group is generally composed of multiple lenses, and the two lens groups are not necessarily arranged in front of and behind each other.

[0048] The driving mechanism 112' includes a focus lens driving mechanism 112a and a variable magnification lens driving mechanism 112b. The focus lens driving mechanism 112a is a driving mechanism for moving at least some of the lenses in the focus lens group 111a along the optical axis direction, and includes an actuator that operates according to an instruction from the system control unit 150. The system control unit 150 adjusts the focus of the subject image that reaches the first image sensor 113 by giving an instruction to the focus lens driving mechanism 112a. The variable magnification lens driving mechanism 112b is a driving mechanism for moving at least some of the lenses in the variable magnification lens group 111b along the optical axis direction, and includes an actuator that operates according to an instruction from the system control unit 150. The system control unit 150 adjusts the angle of view of the subject image that reaches the first image sensor 113 by giving an instruction to the variable magnification lens driving mechanism 112b.

[0049] Next, a description will be given of the characteristic imaging control of the imaging device 100' through the processing procedure of the moiré avoidance processing that is executed periodically while a live view image is being displayed. Fig. 9 is a flow diagram showing the processing procedure of the moiré avoidance processing. The flow is started at a timing that matches the set cycle while a live view image is being displayed.

[0050] In step S201, discrimination unit 151 discriminates whether or not a moiré area exists in active image 301. In the second embodiment, active image 301 is always an image formed based on the output of first imaging element 113, and in that sense, camera unit 110' can be said to always be an active unit. If it is determined that no moiré area exists, moiré avoidance processing for this cycle is terminated. If it is determined that a moiré area exists, the process proceeds to step S202.

[0051] In step S202, the image capture processing unit 152 checks whether the current magnification variable lens group 111b is at the wide-angle end. If it is determined that the current magnification variable lens group 111b is not at the wide-angle end, the image capture processing unit 152 proceeds to step S203, and if it is determined that the current magnification variable lens group 111b is at the wide-angle end, the image capture processing unit 152 proceeds to step S207.

[0052] In step S203, the image capture processing unit 152 drives the variable magnification lens drive mechanism 112b to change the magnification of the variable magnification lens group 111b to the wide-angle side by a preset change in the angle of view. When the magnification change process is completed, in step S204, the discrimination unit 151 discriminates whether or not a moiré area remains in the active image 301 generated after the magnification change process. If it is determined that a moiré area remains, the process returns to step S202 and the process of further changing the magnification to the wide-angle side is repeated. If it is determined that no moiré area remains, the process proceeds to step S205.

[0053] In step S205, the image processing unit 142 cuts out an image area corresponding to the angle of view of the active image 301 at the start of the moiré avoidance processing from the generated active image 301 to generate a cut-out image 311. Next, in step S206, the image capture processing unit 152 displays the cut-out image 311 generated by the image processing unit 142 on the display 130, and ends the series of moiré avoidance processing. Note that after the start of the live view display using the cut-out image 311, when an instruction to capture and record is given by pressing the shutter button 161 or tapping the shutter button 390 and the recording processing is completed, the live view display using the cut-out image 311 may be ended and the variable magnification lens group 111b may be returned to the magnification state at the start of the moiré avoidance processing.

[0054] When the process proceeds from step S202 to step S207, the image capture processing unit 152 drives the variable magnification lens driving mechanism 112b to change the magnification of the variable magnification lens group 111b to the telephoto side by a preset change in the angle of view. When the variable magnification process is completed, in step S208, the discrimination unit 151 discriminates whether or not a moire area remains in the active image 301 generated after the variable magnification process. If it is determined that a moire area remains, the process proceeds to step S211, and it is confirmed whether or not the variable magnification lens group 111b after the variable magnification is in the telephoto end state. If it is confirmed that the zoom is not at the telephoto end, the process returns to step S207, and the process of changing the magnification to the telephoto side is repeated. If it is confirmed that the zoom is at the telephoto end, the variable magnification lens group 111b is returned to the variable magnification state at the start of the moire avoidance process, the moire avoidance is abandoned, and the moire avoidance process in this cycle is terminated.

[0055] If it is determined in step S208 that no moiré region remains, the process proceeds to step S209. In step S209, the image processing unit 142 generates the active image 301, and the image capture processing unit 152 displays the active image 301 generated by the image processing unit 142 on the display 130 with the notification display 325 superimposed thereon. As described above, the notification display 325 includes a notification text notifying that the angle of view has been changed. Then, when a predetermined time has elapsed, in step S210, the notification display 325 is removed, and a series of moiré avoidance processes are terminated. Note that, after the zoom is changed to the telephoto side and live view display using the active image 301 is started, the shutter button 161 is pressed or the shutter button 390 is tapped to instruct image capture and recording, and the recording process is completed, the live view display using the telephoto angle of view may be terminated, and the variable magnification lens group 111b may be returned to the zoom state at the start of the moiré avoidance process.

[0056] In this way, the optical conditions related to imaging are also switched by automatically changing the magnification of the variable magnification lens group 111b. That is, even if moire occurs in the relationship between the spatial frequency of the subject image formed on the imaging surface of the first imaging element 113 at a certain angle of view (certain focal length) and the pixel pitch of the first imaging element 113, it can be expected that moire will not occur in the relationship between the spatial frequency of the subject image formed on the imaging surface of the first imaging element 113 at another angle of view (another focal length) and the pixel pitch of the first imaging element 113. When the magnification change process for the variable magnification lens group is performed automatically in this way, the user can obtain image data of an image that does not include a moire area in many situations.

[0057] Although the present embodiment has been described above through two examples, the first and second examples may be combined. In addition, the imaging device 100 having two camera units has been described in the first example, but the imaging device may have three or more camera units. In this case, when the determination unit 151 determines that a moire region exists, it is preferable that the imaging processing unit 152 switches the active unit to a camera unit having a wider-angle optical system. In addition, although the present application provides an imaging device etc. that can automatically perform imaging processing to avoid the occurrence of moire, it does not exclude a scenario in which a user proactively adjusts it if he or she is aware of the occurrence of moire. In other words, when the discrimination unit detects the presence of moire, the imaging processing unit gives a hint to the user and instructs the user to move the imaging device toward / away from the subject, thereby avoiding the occurrence of moire. [Explanation of symbols]

[0058] 100, 100'...imaging device, 110...first camera unit, 110'...camera unit, 111...first optical system, 111'...optical system, 111a...focus lens group, 111b...variable magnification lens group, 112...first drive mechanism, 112'...drive mechanism, 112a...focus lens drive mechanism, 112b...variable magnification lens drive mechanism, 113...first imaging element, 114...first analog front end (AFE), 120...second camera unit, 121...second optical system, 122...second drive mechanism, 123...second imaging element, 124...second analog front end (AFE), 130...display ray, 150...system control unit, 151...discrimination unit, 152...imaging processing unit, 141...work memory, 142...image processing unit, 160...operation unit, 161...shutter button, 162...touch panel, 170...recording unit, 180...communication interface, 301...active image, 302...moiré area, 390...shutter button, 310...wide-angle image, 311...cut-out image, 313...moiré corresponding area, 314...moiré area, 320...telephoto image, 323...moiré area, 324...moiré corresponding area, 325...notification display, 900...television display, 901...television image

Claims

1. A camera unit having an image sensor; a determination unit that determines whether or not a moire region exists in an image formed based on an output of the imaging element; an imaging processing unit that switches optical conditions related to imaging of the camera unit and continues imaging processing when the determination unit determines that the moire region exists in the image; An imaging device comprising:

2. the camera unit includes at least a first camera unit having a first image sensor as the image sensor and a first optical system that forms a subject image on the first image sensor, and a second camera unit having a second image sensor as the image sensor and a second optical system different from the first optical system that forms a subject image on the second image sensor, the determination unit determines whether or not the moire region exists in an active image formed based on an output of an image sensor of an active unit selected from the first camera unit and the second camera unit; 2. The imaging device according to claim 1, wherein when the discrimination unit determines that the moire region is present in the active image, the imaging processing unit switches the active unit from one of the first camera unit and the second camera unit to the other and continues the imaging process.

3. 3. The imaging device according to claim 2, wherein when a focal length of the first optical system is shorter than a focal length of the second optical system, the imaging processing unit generates a cropped image by cropping an image area corresponding to an angle of view of the second optical system from a first image that may be formed based on an output of the first imaging element when the active unit is switched from the second camera unit to the first camera unit.

4. 3. The imaging device according to claim 2, wherein when a focal length of the first optical system is shorter than a focal length of the second optical system, the imaging processing unit notifies a user regarding an angle of view adjustment when the active unit is switched from the first camera unit to the second camera unit.

5. the camera unit has a variable focal length optical system that is capable of changing a focal length and forms a subject image on the image sensor; The imaging device according to claim 1 , wherein, when the determination unit determines that the moire region exists in the image, the imaging processing unit changes the focal length of the variable focal length optical system and continues the imaging process.

6. 6. The imaging device according to claim 5, wherein when the variable focal length optical system is changed to a focal length shorter than an original focal length, the imaging processing unit generates a cropped image by cropping an image area corresponding to an angle of view of the original focal length from an image formed based on an output of the imaging element.

7. The imaging device according to claim 5 , wherein the imaging processor notifies a user about an angle of view adjustment when the focal length of the variable focal length optical system is changed to a focal length longer than an original focal length.

8. a determining step of determining whether or not a moire region exists in an image formed based on an output of an image pickup element of the camera unit; an imaging processing step of switching optical conditions related to imaging of the camera unit and continuing imaging processing when it is determined in the determination step that the moire region exists in the image; An imaging control program that causes a computer to execute the above.

Images