Imaging equipment, imaging method, program

Abstract

The present technology relates to: an imaging device that makes it possible to align the timing at which a setting is reflected with respect to different imaging elements; an imaging method; and a program. The invention is provided with a control unit that controls a second imaging element which captures an image at a frame rate which is lower than the frame rate of a first imaging element. If a synchronization signal supplied from the first imaging element is received, the control unit determines whether a frame is to be thinned. If the determination is that the frame is not to be thinned, imaging is performed. If the determination is that the frame is to be thinned, imaging drive is halted. The present technology is applicable, for example, to an imaging device having a plurality of imaging elements.

Description

Technical Field

[0001] The present technology relates to a photographing apparatus, a photographing method, and a program. For example, it relates to a photographing apparatus, a photographing method, and a program that enable control to synchronize exposure conditions and timing of mode changes using a plurality of cameras.

Background Art

[0002] For example, it has been proposed to use a plurality of photographing apparatuses capable of photographing independently to perform stereoscopic photographing or to simultaneously photograph images with different angles of view.

[0003] It has been proposed to use a plurality of photographing apparatuses more effectively by synchronizing the photographing of the plurality of photographing apparatuses or by having each photographing apparatus perform photographing independently (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When performing photographing using a plurality of photographing apparatuses, it is desired that changes in exposure conditions and modes be reflected without timing deviations among the plurality of photographing apparatuses.

[0006] The present technology has been made in view of such a situation, and enables changes in exposure conditions and modes to be synchronized among a plurality of photographing apparatuses.

Means for Solving the Problems

[0007] One aspect of this technology is a first imaging device which includes a control unit that controls a second image sensor that captures images at a frame rate lower than the frame rate of a first image sensor. The control unit, upon receiving a synchronization signal supplied from the first image sensor, determines whether or not the frame is subject to downsampling. If it determines that the frame is not subject to downsampling, it performs imaging; if it determines that the frame is subject to downsampling, it stops the imaging drive.

[0008] One aspect of this technology is a first imaging method in which an imaging device, which includes a control unit that controls a second image sensor that captures images at a frame rate lower than the frame rate of a first image sensor, receives a synchronization signal supplied from the first image sensor, determines whether or not the frame is subject to downsampling, and if it is determined that the frame is not subject to downsampling, it performs imaging, and if it is determined that the frame is subject to downsampling, it stops the imaging drive.

[0009] One aspect of this technology is a first program which causes a computer that controls an imaging device equipped with a control unit that controls a second image sensor that captures images at a frame rate lower than the frame rate of a first image sensor to execute a process that includes the steps of: determining whether or not a frame is subject to downsampling when it receives a synchronization signal supplied from the first image sensor; taking a picture if it is determined not to be subject to downsampling; and stopping the shooting drive if it is determined to be subject to downsampling.

[0010] A second imaging device, representing one aspect of this technology, comprises a first control unit that controls imaging by a first image sensor, a second control unit that controls imaging by a second image sensor, and a third control unit that outputs commands to the first and second control units, respectively, instructing them to switch modes. The third control unit transmits the command to the first control unit, including the timing for reflecting the settings made by the command, and the third control unit transmits the command to the second control unit.

[0011] A second aspect of this technology is an imaging method in which the third control unit of an imaging device, comprising a first control unit that controls imaging with a first image sensor, a second control unit that controls imaging with a second image sensor, and a third control unit that outputs commands to the first control unit and the second control unit, respectively, to instruct mode switching, transmits the command to the first control unit, including the timing to reflect the setting by the command, and transmits the command to the second control unit.

[0012] A second program in one aspect of this technology is a program that causes a computer that controls an imaging device comprising a first control unit that controls imaging with a first image sensor, a second control unit that controls imaging with a second image sensor, and a third control unit that outputs commands to the first control unit and the second control unit, respectively, to execute a process that includes the steps of: the third control unit sending the command to the first control unit, including the timing for reflecting the setting by the command; and the third control unit sending the command to the second control unit.

[0013] In the first aspect of this technology, which includes a first imaging device, imaging method, and program, a control unit that controls a second image sensor that captures images at a frame rate lower than the frame rate of the first image sensor, receives a synchronization signal supplied from the first image sensor and determines whether or not it is a frame to be downsampled. If it determines that it is not a frame to be downsampled, it performs imaging; if it determines that it is a frame to be downsampled, it stops the imaging drive.

[0014] In a second aspect of this technology, the imaging device, imaging method, and program include a first control unit that controls imaging with a first image sensor, a second control unit that controls imaging with a second image sensor, and a third control unit that outputs commands to the first and second control units to instruct them to switch modes. The third control unit sends a command to the first control unit that includes the timing for reflecting the settings made by the command, and sends a command to the second control unit.

[0015] Note that the imaging device may be an independent device or an internal block constituting one device.

[0016] Note that the program can be transmitted via a transmission medium or recorded on a recording medium and provided.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram showing the configuration of an embodiment of an imaging device to which this technology is applied. [Figure 2] It is a diagram showing a configuration example of an imaging device. [Figure 3] It is a diagram showing a configuration example of an imaging device. [Figure 4] It is a diagram showing a configuration example of an imaging device. [Figure 5] It is a diagram showing a configuration example of a sensor. [Figure 6] It is a diagram showing a configuration example of a sensor. [Figure 7] )]]It is a diagram showing a configuration example of a control unit. [Figure 8] It is a diagram for explaining the timing of changing exposure conditions. [Figure 9] It is a diagram for explaining the process of matching the timing of changing exposure conditions. [Figure 10] It is a diagram for explaining the power saving mode. [Figure 11] It is a diagram for explaining an additional function for matching the timing of changing exposure conditions. [Figure 12] It is a diagram for explaining the timing of mode switching. [Figure 13] It is a diagram for explaining the process of matching the timing of mode switching. [Figure 14] It is a diagram for explaining the operation during execution of the zoom function. [Figure 15] It is a diagram for explaining the operation of the imaging device. [Figure 16]This diagram illustrates the process involved in sending and receiving large amounts of data in commands. [Figure 17] This diagram illustrates the process involved in sending and receiving commands with small data volumes. [Figure 18] This figure shows an example of a table held by the control unit. [Figure 19] This figure shows an example of a table held by a sensor. [Figure 20] This is a diagram showing an example of the configuration of an electronic device. [Figure 21] This is a diagram showing an example of a personal computer configuration. [Figure 22] This figure shows an example of a schematic configuration of an endoscopic surgical system. [Figure 23] This block diagram shows an example of the functional configuration of a camera head and CCU. [Figure 24] This block diagram shows an example of a schematic configuration of a vehicle control system. [Figure 25] This is an explanatory diagram showing an example of the installation location of the external information detection unit and the imaging unit. [Modes for carrying out the invention]

[0018] The following describes the embodiments for implementing this technology.

[0019] <Configuration of the imaging device> Since this technology can be applied to imaging devices, the following explanation will use the application of this technology to an imaging device as an example.

[0020] Figure 1 shows the configuration of one embodiment of an imaging device to which this technology is applied. The imaging device 10a shown in Figure 1 has a configuration that includes sensors 11a-1, 11a-2, 11a-3, a control unit 12a, and a signal line 13a.

[0021] Sensors 11a-1 to 11a-3 can be imaging elements such as CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide Semiconductor) sensors. The imaging device 10a can be mounted, for example, on a smartphone. In the following description, when it is not necessary to distinguish between sensors 11a-1 to 11a-3 individually, they will simply be referred to as sensor 11a. The same applies to other parts.

[0022] Here, we will use the imaging device 10 as an example, and continue the explanation assuming that the sensor 11 is an image sensor such as a CCD or CMOS sensor. However, this technology can be applied even if the sensor 11 is a sensor other than an image sensor. For example, it may be a pressure sensor, temperature sensor, acceleration sensor, or illuminance sensor.

[0023] As will be described later, this technology allows data acquisition by reflecting the settings for data acquisition on multiple sensors 11 at the desired timing. In the following explanation, we will use the case where sensor 11 is an image sensor as an example and explain how to synchronize the settings for acquiring image data on multiple sensors 11 at the desired timing.

[0024] Sensor 11a-1 can be used as a wide-angle sensor, sensor 11a-2 as the main sensor, and sensor 11a-3 as the telephoto sensor. The wide-angle sensor captures images at the wide-angle end and captures images over a relatively wide area. The telephoto sensor captures images at the telephoto end and captures images over a relatively narrow area. The main sensor captures images in the range between the wide-angle and telephoto ends.

[0025] Here, we will explain using an example where sensors 11a with different focal lengths at the wide, main, and telephoto ends are provided. However, this technology can also be applied when multiple sensors 11a capture images at the same focal length. For example, this technology can be applied when acquiring stereoscopic images by capturing images with multiple sensors 11a. For example, this technology can also be applied when acquiring distance measurement images.

[0026] The control unit 12a processes signals from the sensor 11a by, for example, executing a predetermined application, and outputs them to a subsequent processing unit (not shown) or controls the sensor 11a. The control unit 12a can be an ISP (Image Signal Processor).

[0027] The example shown in Figure 1 (referred to as Configuration Example 1) is a configuration in which each of sensors 11a-1 to 11a-3 operates as a master, and is a configuration sometimes referred to as a standalone system. In Figure 1, the master is indicated by diagonal lines.

[0028] The sensor 11a and the control unit 12a are connected by a signal line 13a that communicates using, for example, the I2C (Inter-Integrated Circuit) method. In the configuration example 1 shown in Figure 1, each sensor 11a receives control commands from the control unit 12a. Synchronization signals and other signals are also generated by each sensor 11a.

[0029] The imaging device 10 can also have the configurations shown in the example configurations 2 to 4 in Figures 2 to 4.

[0030] Figure 2 shows an example configuration of the imaging device 10b in Configuration Example 2. In Configuration Example 2 shown in Figure 2, sensor 11b-2, which is shown with diagonal lines, is the master, and sensors 11b-1 and 11b-3 are set as slaves.

[0031] In a configuration where one of the sensors 11b-1 to 11b-3 is designated as the master and the other sensors 11b are designated as slaves, the synchronization signal generated by the master sensor 11b-2 is supplied to the slave sensors 11b-1 and 11b-3. A signal line 14b for supplying the synchronization signal is provided between the sensors 11b.

[0032] Figure 3 shows an example configuration of the imaging device 10c in Configuration Example 3. In Configuration Example 3 shown in Figure 3, each of the sensors 11c-1 to 11c-3 has the function to switch between master and slave, and operates as either a master or a slave. When one of the sensors 11c-1 to 11c-3 is operating as a master, the other sensors 11c operate as slaves.

[0033] A synchronization signal generated by the master sensor 11c is supplied to the slave sensor 11c via signal line 14c. Commands from the control unit 12c are supplied to the sensor 11c via signal line 13c.

[0034] Figure 4 shows an example of the configuration of the imaging device 10d in Configuration Example 4. In Configuration Example 4 shown in Figure 4, sensors 11d-1 to 11d-3 operate as slaves, and the control unit 12d operates as the master.

[0035] Since the control unit 12d operates as the master, it generates a synchronization signal and supplies it to each sensor 11d via the signal line 14d. Commands from the control unit 12d are supplied to the sensors 11d via the signal line 13d.

[0036] The explanation continues assuming that the imaging device 10 has three sensors 11-1 to 11-3, which are composed of imaging elements. However, this technology can also be applied when there are two or more sensors 11.

[0037] The imaging device 10 can be configured in any of the configuration examples 1 to 4 shown in Figures 1 to 4, and the technology described below can be applied to any of these configuration examples. In the following description, the sensor 11 operating as a master will be referred to as sensor 11M, and the sensor 11 operating as a slave will be referred to as sensor 11S.

[0038] <Sensor Configuration> Figures 5 and 6 show examples of the configuration of the sensor 11.

[0039] Figure 5 shows an example configuration of the sensor 11M operating as a master. The sensor 11M is configured to include an imaging unit 51, a signal processing unit 52, a synchronization signal generation output unit 53, a control unit 54, a command receiving unit 55, and a table holding unit 56.

[0040] The imaging unit 51 includes an image sensor and lens, receives reflected light from the subject, and outputs a signal corresponding to the amount of light received to the signal processing unit 52. The signal processing unit 52 performs processing such as defect correction, noise reduction, and high dynamic range synthesis (HDR) on the input signal and outputs it to the control unit 12 (Figure 1). The function of processing the signal output from the imaging unit 51 may be distributed between the signal processing unit 52 and the control unit 12 (Figure 1), or it may be handled by only one of them.

[0041] The synchronization signal generation output unit 53 generates a synchronization signal and supplies it to the signal processing unit 52. The synchronization signal generated by the synchronization signal generation output unit 53 is also supplied to the slave sensor 11S. The synchronization signal is, for example, a vertical synchronization signal.

[0042] The control unit 54 controls each part of the sensor 11S in a mode set based on an external signal, for example, a control signal from the control unit 12b (Figure 2).

[0043] The command reception unit 55 receives commands from, for example, the control unit 12b (Fig. 2). The table holding unit 56 holds a table in which codes transmitted as commands are associated with settings. As will be described later, commands are transmitted and received as codes of short data sequences, and the table held in the table holding unit 56 on the sensor 11S side is referred to, and parameters in a predetermined mode are set.

[0044] Fig. 6 is a diagram showing a configuration example of the sensor 11S operating as a slave. The sensor 11S is configured to include a photographing unit 71, a signal processing unit 72, a synchronization signal reception unit 73, a control unit 74, a command reception unit 75, and a table holding unit 76.

[0045] The photographing unit 71, signal processing unit 72, control unit 74, command reception unit 75, and table holding unit 76 of the sensor 11S operating in the slave mode perform substantially the same configurations and processes as the photographing unit 51, signal processing unit 52, control unit 54, command reception unit 55, and table holding unit 56 of the sensor 11M operating in the master mode. The synchronization signal reception unit 73 receives a synchronization signal from the master-side sensor 11M, and the signal processing unit 72 performs processing based on the synchronization signal received by the synchronization signal reception unit 73.

[0046] <Configuration of the control unit> Fig. 7 is a diagram showing a configuration example of the control unit 12. The control unit 12 is composed of a command output unit 91, a command holding unit 92, and a control unit 93. The command holding unit 92 holds commands with short data sequences. The command output unit 91 reads out the commands held in the command holding unit 92 based on instructions from the control unit 93 and outputs them to the corresponding sensor 11.

[0047] As in Configuration Example 4 shown in Fig. 4, when the control unit 12 operates as a master, the control unit 12 also has a function of generating and outputting a synchronization signal.

[0048] <Synchronization of AE control> This section describes the synchronization of the AE (Auto Exposure) control performed in the imaging device 10. To clarify the difference between the synchronization of AE control using this technology and the synchronization of conventional AE control, we will first explain the synchronization of conventional AE control with reference to Figure 8.

[0049] The upper diagram of Figure 8 shows the operation of sensor 11M' acting as a master (to distinguish it from sensor 11M to which this technology is applied, the conventional sensor 11M' is described with a dash, and the same applies to others), and the lower diagram of Figure 8 shows the operation of sensor 11S' acting as a slave.

[0050] In the diagram, I2C represents commands supplied from the control unit 12' to the sensor 11' via the signal line 13'. MIPI (Mobile Industry Processor Interface) represents the output of the captured image (frame). In the diagram, one frame is represented by a rectangle, and AE1 and AE2 written inside the rectangle represent exposure conditions such as exposure time and gain, indicating that AE1 and AE2 are different exposure conditions.

[0051] XVS(Output) represents the synchronization signal transmitted from the master sensor 11M' to the slave sensor 11S' via signal line 14', while XVS(Internal) represents the synchronization signal received by the slave sensor 11S' and internally generated using the received synchronization signal as a control command.

[0052] At time t1, sensor 11M' generates a synchronization signal. This synchronization signal is also supplied to the slave sensor 11S'. Here, we will explain using the example of a 2V system where the master sensor 11M' generates 2 frames during the vertical synchronization period (2V), and the slave sensor 11S' generates 1 frame.

[0053] Sensor 11S' generates a synchronization signal at time t4 (2 frames later) based on the synchronization signal received at time t1. Depending on the specifications, it can be set to generate the signal 1 frame later or 3 frames later. For the sake of simplicity, we will assume here that the signal is generated 0 frames later, and that synchronized processing is performed by sensor 11S' generating a synchronization signal at the same time it receives the synchronization signal from sensor 11M'.

[0054] At time t2, sensors 11M' and 11S' receive commands from the control unit 12' (not shown in Figure 8). Commands are sent to sensors 11M' and 11S' from the control unit 12', and more precisely, sensor 11M' receives the command at time t2, and sensor 11S' receives the command a little while after time t2.

[0055] Here, we will explain using the example where both sensor 11M' and sensor 11S' receive a command during the period (within one frame) between a synchronization signal generated at a predetermined time and the next synchronization signal generated. However, as will be explained later, there is a possibility that a situation may occur where sensor 11M' and sensor 11S' cannot receive a command within one frame. In such cases, a mechanism can be provided to reflect the processing by the command, as will be explained later, at a predetermined timing.

[0056] Here, we assume that the commands received by sensor 11M' and sensor 11S' at time t2 are commands related to AE control. In this case, we assume that the command is to change AE1 to AE2.

[0057] At time t3, sensor 11M' generates a synchronization signal. This synchronization signal is also supplied to the slave sensor 11S', but sensor 11S' does not accept it and does not perform the process of internally generating the synchronization signal.

[0058] As described above, the slave sensor 11S' is set to a mode in which it captures one frame while the master sensor 11M' captures two frames. When a difference is created between the frame rate of the master sensor 11M' and the frame rate of the slave sensor 11S', this was achieved by downsampling or multiplying the synchronization signal in the slave sensor 11S'.

[0059] In the following explanation, the mode in which a frame is generated (output) for each synchronization signal will be described as the normal mode, and the mode in which fewer frames are captured than the number of frames captured in the normal mode will be described as the decimation mode.

[0060] Since the slave sensor 11S' is in decimation mode, the synchronization signal generated at time t3 is decimated.

[0061] At time t4, sensor 11M' generates a synchronization signal and supplies it to sensor 11S'. Sensor 11S' receives the supplied synchronization signal as a control command and executes processing based on that control command.

[0062] At time t4, the master sensor 11M' starts processing based on the received command in order to process the second synchronization signal generated at a time later than the time t2 when the command was received. Specifically, it changes the exposure conditions to AE2 and takes a picture based on the parameters set for exposure conditions AE2.

[0063] On the other hand, in sensor 11S', at time t4, the first synchronization signal is processed as a control command from the moment the command is received. In other words, because sensor 11S' processes the first synchronization signal, the control based on the command received at time t2, in this case the control to switch the exposure condition from AE1 to AE2 and take a picture, is not executed. Therefore, at time t4, sensor 11S' performs the shooting process with parameters based on the exposure condition AE1.

[0064] Subsequently, the slave sensor 11S' performs image capture based on the synchronization signal received at time t5, and then performs image capture based on the synchronization signal received at time t6. Since the synchronization signal processed at time t6 is the second synchronization signal received after the command is received, control based on the command begins at time t6. That is, at time t6, the sensor 11S' switches the exposure condition from AE1 to AE2 and performs image capture based on the parameters set in exposure condition A2.

[0065] In this case, at time t4, the sensor 11M' takes pictures using control based on AE2, while the sensor 11S' takes pictures using control based on AE1. In other words, in this case, the sensor M' and sensor S' take pictures based on different exposure conditions. For example, if images taken by the sensor 11M' and sensor 11S' are combined to generate a new image, combining images taken with different settings may result in a flawed combined image.

[0066] In this way, the control mechanism to prevent settings based on received commands from being reflected at different times and to ensure they are reflected simultaneously will be explained with reference to Figure 9.

[0067] <Synchronization of AE control using this technology> Figure 9 is a diagram illustrating the synchronization of AE control using this technology. Parts identical to those in Figure 8 are described in the same way, and their explanations are omitted as appropriate.

[0068] Sensor 11M generates a synchronization signal at each of the following times: t11, t13, t14, t15, and t16, and supplies it to sensor 11S. Sensor 11M generates a synchronization signal using the synchronization signal generation output unit 53 (Figure 5) and supplies it to the signal processing unit 52, as well as to the slave sensor 11S.

[0069] At each of the following times t11, t13, t14, t15, and t16, the sensor 11M captures an image based on the synchronization signal and the set exposure condition AE1, through processing by the imaging unit 51, signal processing unit 52, and control unit 54 of the sensor 11M, and outputs it to a subsequent processing unit (not shown).

[0070] The synchronization signal receiving unit 73 of the slave sensor 11S receives synchronization signals from sensor 11M at times t11, t13, t14, t15, and t16, respectively, and generates an internal synchronization signal. Sensor 11S does not thin out synchronization signals, even if they are synchronization signals that should be thinned out as explained with reference to Figure 8, but instead generates an internal synchronization signal and supplies it to the signal processing unit 72.

[0071] The signal processing unit 72 determines whether the received synchronization signal is one of the synchronization signals to be downsampled. If it determines that the synchronization signal is not one of the synchronization signals to be downsampled, it performs the shooting process and outputs a frame. On the other hand, if the signal processing unit 72 determines that the synchronization signal is one of the synchronization signals to be downsampled, it does not perform the shooting process. In this case, it processes that the synchronization signal has been received, but stops the drive related to the shooting process and stops image output.

[0072] In other words, the signal processing unit 72 determines whether or not a frame should be thinned out. If it is a frame to be thinned out, it stops the shooting drive; if it is a frame that should not be thinned out, it performs shooting.

[0073] At time t12, a command from the control unit 12 is received by both sensor 11M and sensor 11S. The command is an instruction to change the exposure condition from AE1 to AE2. On the sensor 11M side, at time t14 (when the second synchronization signal is generated), two frames later, the signal processing unit 52 switches the exposure condition from AE1 to AE2, and the imaging unit 51 performs the imaging process.

[0074] On the sensor 11S side, the command received at time t2 is reflected two frames later (the second synchronization signal). After receiving the command, the sensor 11S receives (internally generates) the first synchronization signal at time t13, and receives (internally generates) the second synchronization signal at time t14.

[0075] On the sensor 11S side, when the first synchronization signal is received (time t13), the shooting process is stopped as described above, but the synchronization signal itself is processed as having been received. Therefore, when the second synchronization signal is received (time t14), the command settings are reflected, the exposure conditions are switched from AE1 to AE2, and shooting is performed in AE2 mode.

[0076] At time t14, the master sensor 11M takes a picture with exposure condition AE2, and the slave sensor 11S also takes a picture with exposure condition AE2. In this way, even if there is a difference between the frame rate of the master sensor 11M and the frame rate of the slave sensor 11S, simultaneous AE control can be achieved by control commands transmitted from the control unit 12 at approximately the same time.

[0077] In this way, by adjusting the frame rate by stopping the shooting process itself, power consumption can be reduced. This will be explained with reference to Figure 10.

[0078] Between time t21 and time t26, sensor 11M is in 0-skip mode (normal mode), capturing images without downsampling frames, while sensor 11S is in 3-skip mode (downsampling mode), capturing images by downsampling 3 frames.

[0079] When sensor 11S performs an image capture at time t22, the image capture process is stopped for the next three frames. During this period, as explained with reference to Figure 9, if a synchronization signal corresponding to the synchronization signal to be decimated is received, the synchronization signal itself is processed as received (the synchronization signal is generated internally), but the image capture process is stopped.

[0080] During the period when imaging is stopped, the driving of the parts controlled by the standby control signal STBCP of the analog circuit, the parts controlled by the PLL control signal STB VTPLL of the imaging control clock, and the parts controlled by the PLL control signal STB OPPLL of the MIPI control clock can be stopped.

[0081] When these control signals are High, the drive of the corresponding part is stopped. These signals can be stopped for the period during which shooting is stopped, in the example shown in Figure 10, for the duration of the 3 synchronization signal period. However, to prepare for situations such as receiving a command, these signals can be set to Low (On) as needed for a predetermined period of the 3 synchronization signal period.

[0082] In this way, during the period when the shooting drive is stopped, the parts of the analog circuit controlled by the standby control signal and PLL control signal can be turned off, thereby reducing power consumption. If the period during which the shooting drive is stopped is designated as a low-power mode, this technology makes it possible to clearly define the time it takes to transition to this low-power mode, thereby achieving low power consumption.

[0083] At time t24, sensor 11M receives a command from control unit 12 to switch from 0-skip mode to 3-skip mode. From time t26, sensor 11M switches to 3-skip mode (from normal mode to decimation mode). As a result, at times t27, t28, and t29, sensor 11M stops shooting and no frames are output.

[0084] On the master side, the sensor 11M can be set to a low-power mode during periods when the shooting drive is stopped. As described above, by turning off the control signals STBCP, PLL control signals STB VTPLL, and PLL control signals STB OPPLL, the corresponding parts can be stopped from driving, thereby reducing power consumption.

[0085] In other words, this technology can also be applied when the master-side sensor 11M is in decimation mode, and the sensor 11M can be configured to switch to a low-power consumption mode. In this case, the power consumption of the master-side sensor 11M can be reduced, and the power consumption of the imaging device 10 itself can be reduced.

[0086] <Notification of communication completion timing> Referring to Figure 11, we will now explain the operation of the imaging device 10, which has an added function for notifying the timing of communication completion.

[0087] As shown in Figure 11, a communication synchronization function is used to synchronize the timing of communication completion. Figure 11 is a diagram of Figure 9 with the communication synchronization signal of the communication synchronization function added. In the diagram, the communication synchronization signal is denoted as GPH.

[0088] At time t52, sensor 11M receives a command from control unit 12 to switch the exposure condition from AE1 to AE2. If there is no communication synchronization function, the sensor 11M will reflect the setting of this command, in other words, the reflection of the parameter, from time t56 when the second synchronization signal is generated.

[0089] Suppose that sensor 11S receives a command to switch the exposure condition from AE1 to AE2 at time t54, which is later than time t52, for some reason. Time t54 is later than time t53, and time t53 is the time when the first synchronization signal is generated after the command was issued at time t52.

[0090] As explained with reference to Figure 9, the sensor 11S also processes the synchronization signal corresponding to the synchronization signal to be decimated, thereby controlling the timing at which the command is reflected so that it does not deviate from the timing at which it is reflected on the sensor 11M side.

[0091] If there is no communication synchronization function, as shown in Figure 11, if the time t54 when the command is received on the sensor 11S side is after the time t53 when the first synchronization signal is processed, then the synchronization signal at this time t53 will not be counted as a synchronization signal for the timing of reflecting the command.

[0092] In this case, the sensor 11S processes the first synchronization signal at time t56, and the second synchronization signal is processed at time t57, and the command setting is reflected at time t57. In this way, if there is no communication synchronization function, the AE switching occurs at time t56 on the sensor 11M side and at time t57 on the sensor 11S side, resulting in the AE switching occurring at different timings.

[0093] As described above, if there is a significant delay in the timing of command reception, there is a possibility that the settings based on the received commands will be reflected at a different time. To prevent this, a configuration with added communication synchronization functionality can be implemented.

[0094] When the command (or its beginning) is received at time t52 on the sensor 11M side, the communication synchronization signal is set to High. Since the communication synchronization signal is also supplied from sensor 11M to sensor 11S, the communication synchronization signal is also set to High on the sensor 11S side at time t52.

[0095] At time t55, when sensor 11M receives a command from control unit 12 to set the communication synchronization signal to Low, the communication synchronization signal is set to Low. Since the signal that the communication synchronization signal has been set to Low is also supplied to the slave sensor 11S, the communication synchronization signal on sensor 11S is also set to Low at time t55.

[0096] On the sensor 11S side, commands that have been received while the communication synchronization signal is High are treated as having been received when the communication synchronization signal goes Low. This means that even if commands are received at different times, t52 and t54, they can be treated as having been received at t55. In other words, in the example shown in Figure 11, commands that have been received by the sensor 11S side at time t52 and commands that have been received by the sensor 11M side at time t54 are both treated as commands that were received at t55, and are handled accordingly by the sensor 11M and sensor 11S, respectively.

[0097] In this way, by including a process to synchronize the timing of command reception, even if the timing of command reception differs significantly between sensors 11, the settings based on the command can be reflected at the same time.

[0098] In the example shown in Figure 11, the sensor 11M processes the command received at time t52 as if it had been received at time t55, processes the first synchronization signal at time t56, processes the second synchronization signal at time t57, switches the exposure condition from AE1 to AE2 at time t57, and performs the shooting process.

[0099] In sensor 11S, a command that was received at time t54 is treated as having been received at time t55, so the first synchronization signal is processed at time t56, the second synchronization signal is processed at time t57, and at that time t57, the exposure condition is switched from AE1 to AE2 and the shooting process is performed.

[0100] Therefore, at time t57, both sensor 11M and sensor 11S perform shooting processing based on exposure condition AE2. In the case of sensor 11S, the frame captured at time t57 corresponds to a frame to be thinned out, so the shooting drive is stopped. However, even at the timing of such thinned-out frames, this technology allows for switching of exposure conditions and AE switching in synchronization with the other sensors 11.

[0101] It is also possible to configure the system to include such a communication synchronization function, or in other words, a function that ensures that processing based on commands starts at the same time regardless of when the commands are received.

[0102] <Regarding processing when changing modes> This section describes the processing performed when the mode is changed in the imaging device 10. In order to clarify the difference between the mode change processing using this technology and the conventional mode change processing, we will first explain the conventional mode change processing with reference to Figure 12.

[0103] Mode changes include, for example, changing the field of view or switching to HDR (High Dynamic Range) image shooting mode.

[0104] The upper part of Figure 12 shows the operation of sensor 11M' acting as the master, and the lower part of Figure 12 shows the operation of sensor 11S' acting as the slave. At time t101, sensors 11M' and 11S' take images in Mode 1.

[0105] At time t102, the control unit 12' sends a command to the sensor 11M' to switch the mode from Mode1 to Mode2. At time 104, the sensor 11M' reflects the setting based on the command and, in this case, starts shooting in Mode2.

[0106] Similarly, when instructing sensor 11S' to switch from Mode 1 to Mode 2, a Standby transition command is sent from control unit 12' at time t102 (or slightly later), and a Streaming transition command is sent at time t104. Thus, it was necessary to go through SW Standby (software standby) before transitioning modes.

[0107] Therefore, on the slave side, the mode change is reflected at time t106, and shooting in Mode 2 begins. On the master side, the sensor 11M' starts shooting in Mode 2 at time t104, while on the sensor 11S', shooting in Mode 2 begins at time t106, which is later than time t104. Figure 13 will explain how to control the system to prevent such a timing discrepancy in the reflection of the mode change.

[0108] Figure 13 is a diagram illustrating the process related to mode change using this technology. At time t122, the control unit 12 issues a command to sensors 11M and 11S to switch the mode from Mode1 to Mode2.

[0109] The commands sent to sensor 11M include instructions to switch the mode from Mode1 to Mode2 and instructions regarding the timing of the change. In the example shown in Figure 13, the instruction is to switch the mode from the second synchronization signal. Therefore, sensor 11M generates the first synchronization signal at time t123, and when it generates the second synchronization signal at time t124, it switches the mode from Mode1 to Mode2 and takes a picture.

[0110] In the conventional example shown in Figure 12, the parameters of a command received at time t102 were reflected at time t104 when the next synchronization signal was generated. However, in the example where this technology is applied, as shown in Figure 13, the parameters of a command received at time t122 ​​are reflected at time t124 when the synchronization signal two frames later is generated.

[0111] The command sent to the master sensor 11M includes an instruction to arbitrarily delay the timing of parameter reflection. While the delay is set to 2 frames here, any delay of 1 or more frames is acceptable.

[0112] The command sent to the slave sensor 11S is a control command that allows switching the shooting mode during streaming. By providing such a control command, in the example shown in Figure 13, the parameters of the control command received at time t122 ​​are reflected at time t124, and shooting in Mode 2 can be started.

[0113] In this way, the master sensor 11M is equipped with a function to arbitrarily delay the timing of parameter reflection, and the slave sensor 11S is equipped with a control command that allows switching the shooting mode during streaming, so that the mode can be switched at any time between sensor 11M and sensor 11S.

[0114] On the slave side, the sensor 11S does not have SW Standby and the mode can be changed, which reduces the time until the next frame is output, also known as shutter lag.

[0115] Since the modes can be switched at any time while synchronizing the sensors 11M and 11S, as shown in Figure 14, for example, zoom operations can be smoothly performed by switching between the wide-angle sensor 11b-1, the main sensor 11b-2, and the telephoto sensor 11b-3.

[0116] Between time T10 and time T11, the image captured by sensor 11b-1 on the wide end is output, for example, as a preview image. When sensor 11b-1 is capturing in normal mode, sensor 11b-2 captures in decimation mode, and sensor 11b-3 is in standby mode. Because sensor 11b-2 is capturing in decimation mode, as explained with reference to Figure 10, it is in a power-saving mode and is driven in a mode with reduced power consumption.

[0117] At time T11, the image output as a preview image is switched from the image captured by the wide-end sensor 11b-1 to the image captured by the main sensor 11b-2. This switch is performed using the mode switching process described with reference to Figure 13, and the mode switch can be executed at the desired timing, in Figure 14, at time T11. This transition from sensor 11b-1 to sensor 11b-2 can be performed smoothly, and the preview image can be provided to the user without interruption.

[0118] The exposure conditions are also switched as described with reference to Figure 9, and regardless of when the image output as a preview image is switched, the multiple sensors 11 can be switched in a synchronized manner at any timing.

[0119] From time T11 to time T12, the image captured by the main sensor 11b-2 is output as a preview image. During the period from time T11 to time T12, both sensors 11b-1 and 11b-3 are operating in a reduced power consumption mode because they are capturing images in decimation mode.

[0120] At time T12, the image output as a preview image is switched from the image captured by the main sensor 11b-2 to the image captured by the telephoto sensor 11b-3. This switch is performed using the mode switching process described with reference to Figure 13, so the mode switch can be executed smoothly at time T12. This transition from sensor 11b-2 to sensor 11b-3 can be performed smoothly, and the preview image can be provided to the user without interruption.

[0121] From time T12 onward, the image captured by the telephoto sensor 11b-3 is output as a preview image. When sensor 11b-3 is capturing in normal mode, sensor 11b-2 captures in decimation mode, and sensor 11b-1 is in standby mode. Because sensor 11b-2 is capturing in decimation mode, as explained with reference to Figure 10, it is in a power-saving mode and operates in a mode with reduced power consumption.

[0122] In this way, when performing zoom operations such as switching from the wide-angle sensor 11b-1 to the main sensor 11b-2, and then switching from sensor 11b-2 to the telephoto sensor 11b-3, the switching of sensors 11b can be performed smoothly. Therefore, it is possible to provide the user with images that do not have any interruptions and in which the size of the captured image changes smoothly.

[0123] <Regarding the operation of the imaging device> Referring to the flowchart shown in Figure 15, we will now explain the operation of the imaging device 10 that performs the above-described process. Since the above-described process mainly involves the slave-side sensor 11S, we will focus our explanation on the operation of the sensor 11S.

[0124] In step S11, the synchronization signal receiving unit 73 of the sensor 11S determines whether or not it has received a synchronization signal. If it is determined in step S11 that no synchronization signal has been received, the process returns to step S11, and the subsequent processing is repeated.

[0125] On the other hand, if it is determined in step S11 that a synchronization signal has been received, the process proceeds to step S12. In step S12, the AE setting is updated. If a command is received, the process of changing the AE setting, for example, changing the setting from AE1 to AE2 as explained with reference to Figure 9, is executed. If no command is received, the AE setting is reset. The process in step S12 may be executed only if a command is received.

[0126] In step S13, it is determined whether or not a frame is subject to decimation. The determination of whether or not a frame is subject to decimation is performed as follows: First, if "n-skip" is specified by the control unit 12, the sensor 11S is set to output only one frame out of n consecutive frames, and not output the other frames, i.e., to decimate them.

[0127] The control unit 12 can specify which of the n consecutive frames to output, and this specification is included in the command and supplied to the sensor 11S. The period of the n consecutive frames is counted sequentially, for example, with the frame that started streaming being frame 0, and so on as frame 1, frame 2, ... If the number of skips is changed or a mode transition occurs, this n is initialized.

[0128] In step S13, the signal processing unit 72 determines whether or not the frame has been set as a frame to be thinned out by the control unit 12. If it determines that the frame is one to be thinned out, the process returns to step S11, and the subsequent processing is repeated.

[0129] On the other hand, if it is determined in step S13 that the frame is not subject to downsampling, in other words, if it is determined that the frame should be captured without downsampling, the process proceeds to step S14. In step S14, capture (image acquisition) is performed. Then, in step S15, the signal processing unit 72 determines whether or not the mode setting has been changed.

[0130] If it is determined in step S15 that the mode setting has not been changed, the process returns to step S11 and the subsequent processes are repeated. On the other hand, if it is determined in step S15 that the mode setting has been changed, the process proceeds to step S16.

[0131] In step S16, the signal processing unit 72 updates the mode setting. As explained with reference to Figure 13, the process for updating the mode setting involves receiving a control command from the control unit 12 to update the mode setting, and then executing the mode setting based on that command.

[0132] In step S17, the synchronization state is reset. After the synchronization state is reset, the process returns to step S11, and the subsequent processes are repeated.

[0133] As processing is performed on the slave sensor 11S in this manner, changes to exposure conditions and mode settings can be performed in sync with the master sensor 11M, as explained with reference to Figures 9 and 13.

[0134] The flowchart shown in Figure 15 can also be applied when the master sensor 11M operates in decimation mode.

[0135] <Regarding sending and receiving commands with large data volumes> As explained with reference to Figure 13, this technology allows for synchronous mode changes between sensor 11M and sensor 11S. The command transmitted from the control unit 12 and received by sensor 11 for mode changes includes, for example, shooting mode parameters such as Still, Video, and Preview, and the amount of data may exceed, for example, 100 bytes. When sending and receiving commands with such large data volumes, the time it takes to send and receive the commands may cause the synchronization of mode changes to occur later than the desired timing.

[0136] This will be explained with reference to Figure 16. Figure 16 is a diagram illustrating the transmission and reception of conventional commands. At time t201, sensors 11M' and 11S' perform imaging in Mode 1 based on the synchronization signal.

[0137] At time t202, sensor 11M' receives a command from control unit 12'. The command is sent and received between time t202 and time t203. Depending on the specifications of sensor 11, it may be necessary to send and receive mode parameters exceeding 100 bytes, so the time required for transmission and reception will be proportional to the amount of data in the command.

[0138] On the sensor 11S side, commands from the control unit 12' are received between time t203 and time t205. These commands, like the commands received by sensor 11M', involve a large amount of data, which can lead to longer transmission and reception times. For example, as shown in Figure 16, the synchronization signal at time t204 may be received across the time interval.

[0139] For example, if the command setting is to be reflected one frame later (the second synchronization signal), the mode setting on sensor 11M' will be changed at time t206, and the mode setting on sensor 11S' will be changed at time t207. In this case, to synchronize the timing of the mode setting change, the mode change on sensor 11M' needs to be executed at time t207, two frames later.

[0140] Thus, when sending and receiving commands with large amounts of data, there is a possibility that the transmission and reception may not be completed within the intended frame, and in such cases, the timing of mode transitions may be delayed. Therefore, we will explain how to reduce the amount of data in the commands being sent and received, shorten the time it takes to send and receive, and prevent such delays in mode transition timing, referring to Figure 17.

[0141] At time t222, sensor 11M receives a command from control unit 12 regarding mode switching. As explained with reference to Figures 18 and 19, this command is a small data command (a short code in the data sequence). Because it is a small data command, the time required for transmission and reception is short. As shown in Figure 17, at approximately the same time t222, sensor 11S also receives a command from control unit 12 regarding mode switching.

[0142] In this way, sensors 11M and 11S can receive commands with small data volumes at approximately the same time. Therefore, sensors 11M and 11S can reflect settings (parameters) based on commands at the desired timing. For example, in the example shown in Figure 17, at time t224, one frame later, sensors 11M and 11S can switch from Mode 1 to Mode 2 for imaging.

[0143] Figure 18 shows an example of a command transmitted from the control unit 12 and held in the command holding unit 92 (Figure 7). The command holding unit 92 of the control unit 12 holds a table in which, for example, modes (MC-Mode) and commands are associated. Here, we will explain using the example of a case in which three sensors 11b-1 to 11b-3 are provided in the imaging device 10, as shown in Figure 2.

[0144] Refer to Table 92 in Figure 18. The "MC OFF" mode is a mode that turns off the three sensors 11b (stops the shooting drive), and its command is associated with (000).

[0145] The "MC WIDE" mode is associated with a mode in which sensor 11b-1, which captures images at the wide end, operates in normal mode, while sensors 11b-2 and 11b-3 stop their shooting operation, and the command for this mode is (001).

[0146] The "MC WIDE MAIN" mode is a mode in which the wide-end sensor 11b-1 is driven in normal mode and the main sensor 11b-2 is driven in decimation mode, and its command is associated with (010).

[0147] The "MC MAIN WIDE" mode is a mode in which the main sensor 11b-2 is driven in normal mode and the wide-end sensor 11b-1 is driven in decimation mode, and its command is associated with (011).

[0148] The "MC MAIN" mode is a mode in which the main sensor 11b-2 is driven in normal mode, while the other sensors 11b-1 and 11b-3 are stopped from being driven for shooting, and its command is associated with (100).

[0149] The "MC MAIN TELE" mode is a mode in which the main sensor 11b-2 is driven in normal mode and the telephoto sensor 11b-3 is driven in decimation mode, and its command is associated with (101).

[0150] The “MC TELE MAIN” mode is a mode in which the telephoto end sensor 11b-3 is driven in normal mode and the main sensor 11b-2 is driven in decimation mode, and its command is associated with (110).

[0151] The “MC TELE” mode is a mode in which the telephoto end sensor 11b-3 is driven in normal mode, while the other sensors 11b-1 and 11b-2 are stopped from being driven for shooting, and its command is associated with (111).

[0152] When the control unit 12 instructs the sensor 11b to enter a predetermined mode, it sends a command associated with that predetermined mode. For example, if the system wants to set the mode to "MC WIDE MAIN", the command (010) is sent to sensors 11b-1 through 11b-3.

[0153] The sensor 11 maintains a table as shown in Figure 19 and is configured to read and execute the processing corresponding to the received command.

[0154] Tables 56-1 to 56-3 shown in Figure 19 are held by the control unit 12, and when the power of the imaging device 10b is turned on, the table 56 corresponding to each sensor 11b is transferred and held in the table holding part 56 of the sensor 11b. Alternatively, it can be configured so that each sensor 11b has the table held in advance.

[0155] Sensor 11b-1, which captures images at the wide end, holds table 56-1; sensor 11b-2, which is the main sensor, holds table 56-2; and sensor 11b-3, which captures images at the telephoto end, holds table 56-3. Each table 56 is held in the table holding part 56 of each sensor 11b.

[0156] Each table 56 is a table that associates the mode register (Mc Mode Register), LUT Start which specifies the starting position of the LUT, and LUT ctrl which specifies the number of times the LUT loops.

[0157] LUT Start specifies which lookup table to start with, and LUT ctrl specifies how many times to loop through the parameters defined in the starting lookup table. For example, if LUT Start is "LUT A" and LUT ctrl is "1 loop", it means to start with table LUT A and loop once.

[0158] The table specified in the LUT Start field is provided separately (not shown). For example, if "LUT A" is specified in LUT Start, the corresponding portion of the separately provided table is referenced. This separately provided table may reference other tables, which may also be provided separately. The table referenced by LUT Start could, for example, be an AEB Table (Auto Exposure Bracketing Table).

[0159] Each table 56's Mc Mode Register contains the following items, which are also found in the Mc-Mode column of table 92 (Figure 18) held by the control unit 12: “MC OFF”, “MC WIDE”, “MC WIDE MAIN”, “MC MAIN WIDE”, “MC MAIN”, “MC MAIN TELE”, “MC TELE MAIN”, and “MC TELE”.

[0160] Refer to Table 56-1. In Table 56-1, "LUT Start" under "MC OFF" is associated with "OFF", and "LUT ctrl" is left blank. When the command from the control unit 12 is (000), the sensor 11b-1 is set to the off state (low power consumption mode).

[0161] In Table 56-1, "LUT Start" for "MC WIDE" is associated with "LUT A," and "LUT ctrl" is associated with "1loop." In Table 56-1, "LUT Start" for "MC WIDE MAIN" is associated with "LUT A," and "LUT ctrl" is associated with "1loop." When the command from the control unit 12 is (001) or (010), the sensor 11b-1 is set to refer to the table called LUT A and perform one loop of processing with the values ​​defined in LUT A.

[0162] In table 56-1, "LUT Start" under "MC MAIN WIDE" is associated with "LUT B," and "LUT ctrl" is associated with "1loop." When the command from control unit 12 is (011), sensor 11b-1 is set to refer to the table called LUT B and perform one loop of processing with the values ​​defined in LUT B.

[0163] In Table 56-1, "LUT Start" for "MC MAIN" is associated with "OFF," and "LUT ctrl" is left blank. In Table 56-1, "LUT Start" for "MC MAIN TELE" is associated with "OFF," and "LUT ctrl" is left blank. In Table 56-1, "LUT Start" for "MC TELE MAIN" is associated with "OFF," and "LUT ctrl" is left blank. In Table 56-1, "LUT Start" for "MC TELE" is associated with "OFF," and "LUT ctrl" is left blank. If the command from the control unit 12 is (100), (101), (110), or (111), it is set to the OFF state.

[0164] Since sensor 11b-1 is a camera that captures images at the wide-angle end, it holds table 56-1 which indicates that it is driven when capturing images at the wide-angle end.

[0165] Refer to Table 56-2. In Table 56-2, "MC OFF" is associated with "LUT Start" and "LUT ctrl" is left blank. In Table 56-2, "MC WIDE" is associated with "LUT Start" and "LUT ctrl" is left blank. If the command from the control unit 12 is (000) or (001), the sensor 11b-2 is set to the off state.

[0166] In table 56-2, "LUT Start" under "MC WIDE MAIN" is associated with "LUT B," and "LUT ctrl" is associated with "1loop." When the command from control unit 12 is (010), sensor 11b-2 is set to refer to the table called LUT B and perform one loop of processing with the values ​​defined in LUT B.

[0167] In Table 56-2, "LUT Start" under "MC MAIN WIDE" is associated with "LUT A", and "LUT ctrl" is associated with "1loop". In Table 56-2, "LUT Start" under "MC MAIN" is associated with "LUT A", and "LUT ctrl" is associated with "1loop". In Table 56-2, "LUT A" is associated with "MC MAIN TELE", and "LUT ctrl" is associated with "1loop". When the command from the control unit 12 is (011), (100), or (101), the sensor 11b-2 is set to refer to the table called LUT A and perform one loop of processing with the values ​​defined in LUT A.

[0168] In table 56-2, "LUT Start" under "MC TELE MAIN" is associated with "LUT B," and "LUT ctrl" is associated with "1loop." When the command from control unit 12 is (110), sensor 11b-2 is set to refer to the table called LUT B and perform one loop of processing with the values ​​defined in LUT B.

[0169] In Table 56-2, "LUT Start" under "MC TELE" is associated with "OFF," and "LUT ctrl" is left blank. When the command from the control unit 12 is (111), it is set to the OFF state.

[0170] Since sensor 11b-2 is the main camera, it has more operating modes compared to the other sensors 11b, and Table 56-2 shows that it frequently operates together with the wide-angle sensor 11b-1 and with the telephoto sensor 11b-3.

[0171] Refer to Table 56-3. In Table 56-3, "LUT Start" under "MC OFF" is associated with "OFF," and "LUT ctrl" is left blank.

[0172] In Table 56-3, "LUT Start" for "MC WIDE" is associated with "OFF", and "LUT ctrl" is left blank. In Table 56-3, "LUT Start" for "MC WIDE MAIN" is associated with "OFF", and "LUT ctrl" is left blank. In Table 56-3, "LUT Start" for "MC MAIN WIDE" is associated with "OFF", and "LUT ctrl" is left blank. In Table 56-3, "LUT Start" for "MC MAIN" is associated with "OFF", and "LUT ctrl" is left blank. When the command from the control unit 12 is (000), (001), (010), (011), or (100), the sensor 11b-3 is set to the off state.

[0173] In table 56-3, "LUT Start" under "MC MAIN TELE" is associated with "LUT B," and "LUT ctrl" is associated with "1loop." When the command from control unit 12 is (101), sensor 11b-3 is set to refer to the table called LUT B and perform one loop of processing with the values ​​defined in LUT B.

[0174] In Table 56-3, "LUT Start" under "MC TELE MAIN" is associated with "LUT A," and "LUT ctrl" is associated with "1loop." In Table 56-3, "LUT Start" under "MC TELE" is associated with "LUT A," and "LUT ctrl" is associated with "1loop." When the command from the control unit 12 is (110) or (111), the sensor 11b-3 is set to refer to the table called LUT A and perform one loop of processing with the values ​​defined in LUT A.

[0175] Since sensor 11b-3 is a camera that takes pictures on the telephoto side, it holds table 56-3 which indicates that it is driven when taking pictures on the telephoto side.

[0176] In this way, each sensor 11 maintains a different table. The control unit 12 sends the same command to multiple sensors 11, and each sensor 11 refers to the table 56 it maintains to configure itself. This reduces the amount of data for the commands sent and received, and allows the control unit 12 to execute processes such as mode switching at the desired timing without causing frame misalignment.

[0177] By reducing the amount of command data exchanged between the sensor 11 and the control unit 12, the time required for communication between the sensor 11 and the control unit 12 can be shortened, resulting in lower power consumption.

[0178] Here, we have explained using the example of the control unit 12 controlling three sensors 11, but the number of sensors 11 controlled by the control unit 12 is not limited to three; it is also possible to configure it to control many more sensors 11. Even when the number of sensors 11 controlled by the control unit 12 increases, as described above, processing (settings) can be performed at each sensor 11 by sending commands with a small amount of data to multiple sensors 11.

[0179] Therefore, it becomes possible to control multiple sensors 11 without increasing the processing load of the control unit 12, thereby reducing the processing load of the control unit 12 and allowing the control unit 12 to have more time to perform other tasks.

[0180] For example, LUT A and LUT B, as shown in Table 56 in Figure 19, have their parameters listed in a separate table from Table 56. These parameters can be written according to the specifications of the imaging device 10. It is also possible to have multiple tables other than Table 56, which are referenced sequentially.

[0181] <Examples of application to electronic devices> This technology is applicable to all electronic devices that use an image sensor in the image acquisition unit (photoelectric conversion unit), such as imaging devices like digital still cameras and video cameras, portable terminal devices with imaging functions, and photocopiers that use an image sensor in the image reading unit. The image sensor may be formed as a single chip, or it may be in the form of a module with imaging functions in which the imaging unit and signal processing unit or optical system are packaged together.

[0182] Figure 20 is a block diagram showing an example configuration of an imaging device as an electronic device to which this technology is applied.

[0183] The image sensor 1000 in Figure 20 comprises an optical unit 1001 consisting of a lens group, an image sensor (imaging device) 1002, and a DSP (Digital Signal Processor) circuit 1003 which is a camera signal processing circuit. The image sensor 1000 also includes a frame memory 1004, a display unit 1005, a recording unit 1006, an operation unit 1007, and a power supply unit 1008. The DSP circuit 1003, frame memory 1004, display unit 1005, recording unit 1006, operation unit 1007, and power supply unit 1008 are interconnected via a bus line 1009.

[0184] The optical unit 1001 captures incident light (image light) from the subject and forms an image on the imaging surface of the image sensor 1002. The image sensor 1002 converts the amount of light from the incident light formed on the imaging surface by the optical unit 1001 into an electrical signal on a pixel-by-pixel basis and outputs it as a pixel signal.

[0185] The display unit 1005 is composed of a thin display such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays video or still images captured by the image sensor 1002. The recording unit 1006 records the video or still images captured by the image sensor 1002 onto a recording medium such as a hard disk or semiconductor memory.

[0186] The control unit 1007 issues operation commands for various functions of the image sensor 1000 under user operation. The power supply unit 1008 appropriately supplies various power sources to the DSP circuit 1003, frame memory 1004, display unit 1005, recording unit 1006, and control unit 1007.

[0187] The imaging devices 10a to 10d shown in Figures 1 to 4 can be applied to a part of the imaging device shown in Figure 20.

[0188] <About recording media> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0189] Figure 21 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program. In the computer, the CPU (Central Processing Unit) 2001, ROM (Read Only Memory) 2002, and RAM (Random Access Memory) 2003 are interconnected by a bus 2004. An input / output interface 2005 is further connected to the bus 2004. An input / output interface 2005 is connected to an input unit 2006, an output unit 2007, a storage unit 2008, a communication unit 2009, and a drive 2010.

[0190] The input unit 2006 consists of a keyboard, mouse, microphone, etc. The output unit 2007 consists of a display, speakers, etc. The storage unit 2008 consists of a hard disk, non-volatile memory, etc. The communication unit 2009 consists of a network interface, etc. The drive 2010 drives a removable recording medium 2011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0191] In a computer configured as described above, the CPU 2001 loads, for example, a program stored in the memory unit 2008 into the RAM 2003 via the input / output interface 2005 and the bus 2004, and executes it, thereby performing the series of processes described above.

[0192] The program executed by the computer (CPU2001) can be provided by recording it on a removable recording medium 2011, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0193] In a computer, a program can be installed in the storage unit 2008 via the input / output interface 2005 by inserting the removable storage medium 2011 into the drive 2010. Alternatively, the program can be received by the communication unit 2009 via a wired or wireless transmission medium and installed in the storage unit 2008. Furthermore, the program can be pre-installed in the ROM 2002 or the storage unit 2008.

[0194] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0195] <Examples of application to endoscopic surgical systems> The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein may be applied to an endoscopic surgical system.

[0196] Figure 22 is a diagram showing an example of a schematic configuration of an endoscopic surgical system to which the technology described herein (the technology) may be applied.

[0197] Figure 22 illustrates a surgeon (physician) 11131 performing surgery on a patient 11132 on a patient bed 11133 using an endoscopic surgical system 11000. As shown in the figure, the endoscopic surgical system 11000 consists of an endoscope 11100, other surgical instruments 11110 such as an insufflation tube 11111 and an energy treatment device 11112, a support arm device 11120 for supporting the endoscope 11100, and a cart 11200 equipped with various devices for endoscopic surgery.

[0198] The endoscope 11100 consists of a barrel 11101, the tip of which is inserted into the body cavity of the patient 11132 for a predetermined length, and a camera head 11102 connected to the base end of the barrel 11101. In the illustrated example, the endoscope 11100 is shown as a so-called rigid endoscope having a rigid barrel 11101, but the endoscope 11100 may also be configured as a so-called flexible endoscope having a flexible barrel.

[0199] An opening into which an objective lens is fitted is provided at the tip of the endoscope tube 11101. A light source device 11203 is connected to the endoscope 11100, and the light generated by the light source device 11203 is guided to the tip of the endoscope tube by a light guide extending inside the endoscope tube 11101, and is irradiated through the objective lens towards the object to be observed inside the body cavity of the patient 11132. The endoscope 11100 may be a straight-viewing endoscope, an oblique-viewing endoscope, or a side-viewing endoscope.

[0200] The camera head 11102 contains an optical system and an image sensor. Reflected light from the object being observed (observation light) is focused onto the image sensor by the optical system. The image sensor converts the observation light into electrical signals, generating an electrical signal corresponding to the observation light, i.e., an image signal corresponding to the observed image. This image signal is transmitted as RAW data to the camera control unit (CCU) 11201.

[0201] The CCU11201 consists of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and other components, and comprehensively controls the operation of the endoscope 11100 and the display device 11202. Furthermore, the CCU11201 receives image signals from the camera head 11102 and performs various image processing operations on these image signals, such as development processing (demosaic processing), to display the image based on those image signals.

[0202] The display device 11202 displays an image based on an image signal that has been processed by the CCU 11201, under control from the CCU 11201.

[0203] The light source device 11203 consists of a light source such as an LED (light-emitting diode) and supplies illumination light to the endoscope 11100 when photographing the surgical area, etc.

[0204] The input device 11204 is an input interface for the endoscopic surgical system 11000. The user can input various information and instructions to the endoscopic surgical system 11000 via the input device 11204. For example, the user can input instructions to change the imaging conditions (type of light, magnification, focal length, etc.) of the endoscope 11100.

[0205] The treatment instrument control device 11205 controls the drive of the energy treatment instrument 11112 for purposes such as tissue cauterization, incision, or vascular sealing. The insufflation device 11206 delivers gas into the patient's body cavity via the insufflation tube 11111 to inflate the body cavity of the patient 11132 for the purpose of securing a field of view by the endoscope 11100 and securing the operator's working space. The recorder 11207 is a device capable of recording various information related to the surgery. The printer 11208 is a device capable of printing various information related to the surgery in various formats such as text, images, or graphs.

[0206] The light source device 11203, which supplies illumination light to the endoscope 11100 when imaging the surgical area, can be configured as a white light source consisting of, for example, an LED, a laser light source, or a combination thereof. When the white light source is configured as a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision, so the white balance of the captured image can be adjusted in the light source device 11203. In this case, it is also possible to time-resolve the laser beams from each of the RGB laser light sources onto the object of observation and control the drive of the image sensor of the camera head 11102 in synchronization with the irradiation timing, thereby capturing images corresponding to each of the RGB colors in time-resolved order. According to this method, a color image can be obtained without providing a color filter on the image sensor.

[0207] Furthermore, the light source device 11203 may be controlled to change the intensity of the light it outputs at predetermined time intervals. By controlling the drive of the image sensor of the camera head 11102 in synchronization with the timing of the change in light intensity, images can be acquired in time-division order, and these images can be combined to generate high dynamic range images without so-called black crushing and white clipping.

[0208] Furthermore, the light source device 11203 may be configured to supply light in a predetermined wavelength band corresponding to special light observation. In special light observation, for example, by utilizing the wavelength dependence of light absorption in body tissue and irradiating with narrow-band light compared to the irradiation light used during normal observation (i.e., white light), so-called narrow-band imaging is performed to capture images of predetermined tissues such as blood vessels on the surface of mucous membranes with high contrast. Alternatively, in special light observation, fluorescence observation may be performed to obtain an image from fluorescence generated by irradiation with excitation light. In fluorescence observation, excitation light is irradiated onto body tissue and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is injected into body tissue and excitation light corresponding to the fluorescence wavelength of the reagent is irradiated onto the body tissue to obtain a fluorescence image. The light source device 11203 can be configured to supply narrowband light and / or excitation light suitable for such special light observations.

[0209] Figure 23 is a block diagram showing an example of the functional configuration of the camera head 11102 and CCU 11201 shown in Figure 22.

[0210] The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are connected to each other via a transmission cable 11400 so that they can communicate with one another.

[0211] The lens unit 11401 is an optical system provided at the connection point with the lens barrel 11101. Observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and then incident on the lens unit 11401. The lens unit 11401 is composed of multiple lenses, including a zoom lens and a focus lens.

[0212] The imaging unit 11402 may consist of one image sensor (a so-called single-chip system) or multiple image sensors (a so-called multi-chip system). If the imaging unit 11402 is configured as a multi-chip system, for example, each image sensor may generate an image signal corresponding to RGB, and these signals may be combined to obtain a color image. Alternatively, the imaging unit 11402 may be configured to have a pair of image sensors for acquiring image signals for the right eye and left eye, respectively, corresponding to 3D (dimensional) display. By performing 3D display, the surgeon 11131 can more accurately grasp the depth of the biological tissue in the surgical area. Furthermore, if the imaging unit 11402 is configured as a multi-plate type, multiple lens units 11401 may be provided corresponding to each image sensor.

[0213] Furthermore, the imaging unit 11402 does not necessarily have to be located in the camera head 11102. For example, the imaging unit 11402 may be located inside the lens barrel 11101, directly behind the objective lens.

[0214] The drive unit 11403 is composed of actuators and, under control from the camera head control unit 11405, moves the zoom lens and focus lens of the lens unit 11401 along the optical axis by a predetermined distance. This allows the magnification and focus of the image captured by the imaging unit 11402 to be adjusted as appropriate.

[0215] The communication unit 11404 consists of communication devices for sending and receiving various types of information with the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 as RAW data to the CCU 11201 via the transmission cable 11400.

[0216] Furthermore, the communication unit 11404 receives control signals from the CCU 11201 to control the drive of the camera head 11102 and supplies them to the camera head control unit 11405. These control signals include information regarding imaging conditions, such as information to specify the frame rate of the captured image, information to specify the exposure value at the time of imaging, and / or information to specify the magnification and focus of the captured image.

[0217] The imaging conditions such as frame rate, exposure value, magnification, and focus may be specified by the user as appropriate, or they may be automatically set by the control unit 11413 of the CCU11201 based on the acquired image signal. In the latter case, the endoscope 11100 will be equipped with so-called AE (Auto Exposure), AF (Auto Focus), and AWB (Auto White Balance) functions.

[0218] The camera head control unit 11405 controls the drive of the camera head 11102 based on the control signal received from the CCU 11201 via the communication unit 11404.

[0219] The communication unit 11411 consists of a communication device for sending and receiving various types of information with the camera head 11102. The communication unit 11411 receives image signals transmitted from the camera head 11102 via the transmission cable 11400.

[0220] Furthermore, the communication unit 11411 transmits control signals to the camera head 11102 to control the driving of the camera head 11102. Image signals and control signals can be transmitted by telecommunications, optical communications, etc.

[0221] The image processing unit 11412 performs various image processing operations on the image signal, which is RAW data transmitted from the camera head 11102.

[0222] The control unit 11413 performs various controls related to imaging the surgical area, etc., by the endoscope 11100, and the display of the images obtained from imaging the surgical area, etc. For example, the control unit 11413 generates control signals to control the driving of the camera head 11102.

[0223] Furthermore, the control unit 11413 displays the captured image showing the surgical area on the display device 11202 based on the image signal processed by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition technologies. For example, the control unit 11413 can recognize surgical instruments such as forceps, specific biological sites, bleeding, mist when using the energy treatment device 11112, etc., by detecting the shape and color of the edges of objects included in the captured image. When the control unit 11413 displays the captured image on the display device 11202, it may use the recognition results to superimpose various surgical support information onto the image of the surgical area. By superimposing the surgical support information and presenting it to the surgeon 11131, the burden on the surgeon 11131 can be reduced, and the surgeon 11131 can proceed with the surgery with confidence.

[0224] The transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electrical signal cable compatible with electrical signal communication, an optical fiber compatible with optical communication, or a composite cable thereof.

[0225] In the illustrated example, communication was performed via a wired connection using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may be performed wirelessly.

[0226] <Examples of applications to mobile devices> The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein may be implemented as a device mounted on any type of mobile vehicle, such as an automobile, electric vehicle, hybrid electric vehicle, motorcycle, bicycle, personal mobility device, airplane, drone, ship, or robot.

[0227] Figure 24 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile control system to which the technology described herein may be applied.

[0228] The vehicle control system 12000 comprises multiple electronic control units connected via a communication network 12001. In the example shown in Figure 24, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. The functional configuration of the integrated control unit 12050 is also shown, consisting of a microcomputer 12051, an audio / image output unit 12052, and an in-vehicle network interface 12053.

[0229] The drivetrain control unit 12010 controls the operation of devices related to the vehicle's drivetrain according to various programs. For example, the drivetrain control unit 12010 functions as a control device for a drivetrain generating device that generates driving force for the vehicle, such as an internal combustion engine or a drive motor; a drivetrain transmission mechanism that transmits driving force to the wheels; a steering mechanism that adjusts the steering angle of the vehicle; and a braking device that generates braking force for the vehicle.

[0230] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window system, or various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. In this case, the body system control unit 12020 may receive radio waves transmitted from a portable device that replaces a key or signals from various switches. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0231] The external information detection unit 12030 detects information from outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the external information detection unit 12030. The external information detection unit 12030 causes the imaging unit 12031 to capture images of the outside of the vehicle and receives the captured images. Based on the received images, the external information detection unit 12030 may perform object detection processing such as detecting people, cars, obstacles, signs, or characters on the road surface, or distance detection processing.

[0232] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0233] The in-vehicle information detection unit 12040 detects information inside the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver status detection unit 12041 that detects the driver's state. The driver status detection unit 12041 includes, for example, a camera that images the driver, and the in-vehicle information detection unit 12040 may calculate the driver's level of fatigue or concentration, or determine whether the driver is drowsy, based on the detection information input from the driver status detection unit 12041.

[0234] The microcomputer 12051 can calculate control target values ​​for the drive force generator, steering mechanism, or braking system based on information from inside and outside the vehicle acquired by the external information detection unit 12030 or the internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including collision avoidance or impact mitigation, following based on distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning.

[0235] Furthermore, the microcomputer 12051 can perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on information about the vehicle's surroundings acquired by the external information detection unit 12030 or the internal information detection unit 12040.

[0236] Furthermore, the microcomputer 12051 can output control commands to the body system control unit 12030 based on external information acquired by the external information detection unit 12030. For example, the microcomputer 12051 can control the headlights according to the position of a preceding or oncoming vehicle detected by the external information detection unit 12030, and perform coordinated control aimed at reducing glare, such as switching from high beams to low beams.

[0237] The audio-image output unit 12052 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying information to the vehicle's occupants or to those outside the vehicle. In the example in Figure 24, the output devices are exemplified as an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an onboard display and a head-up display.

[0238] Figure 25 shows an example of the installation position of the imaging unit 12031.

[0239] In Figure 25, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0240] The imaging units 12101, 12102, 12103, 12104, and 12105 are installed, for example, on the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the vehicle 12100. The imaging unit 12101 installed on the front nose and the imaging unit 12105 installed on the upper part of the windshield inside the vehicle mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 installed on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 installed on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 installed on the upper part of the windshield inside the vehicle is mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0241] Figure 25 shows an example of the imaging range of imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of imaging unit 12101 located on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of imaging units 12102 and 12103 located on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of imaging unit 12104 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 12101 to 12104, an overhead view image of the vehicle 12100 can be obtained.

[0242] At least one of the imaging units 12101 to 12104 may have a function for acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera consisting of multiple image sensors, or an image sensor having pixels for phase difference detection.

[0243] For example, the microcomputer 12051, based on distance information obtained from imaging units 12101 to 12104, can determine the distance to each object within the imaging range 12111 to 12114 and the temporal change of this distance (relative speed to vehicle 12100). In particular, it can extract the nearest object on the vehicle 12100's path that is traveling in approximately the same direction as vehicle 12100 at a predetermined speed (e.g., 0 km / h or more) as the preceding vehicle. Furthermore, the microcomputer 12051 can set a predetermined distance to be maintained before the preceding vehicle and perform automatic braking control (including follow-and-stop control) and automatic acceleration control (including follow-and-start control), etc. In this way, cooperative control aimed at autonomous driving, where the vehicle drives autonomously without driver intervention, can be performed.

[0244] For example, the microcomputer 12051 can use distance information obtained from imaging units 12101 to 12104 to classify and extract three-dimensional object data related to three-dimensional objects, such as motorcycles, passenger cars, heavy vehicles, pedestrians, utility poles, and other three-dimensional objects, and use this data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines the collision risk, which indicates the degree of risk of collision with each obstacle. If the collision risk is above a set value and there is a possibility of collision, the microcomputer 12051 can provide driving assistance to avoid collisions by outputting a warning to the driver via the audio speaker 12061 or display unit 12062, or by performing forced deceleration or evasive steering via the drive system control unit 12010.

[0245] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can recognize pedestrians by determining whether or not pedestrians are present in the images captured by the imaging units 12101 to 12104. Such pedestrian recognition is performed, for example, by a procedure to extract feature points from the images captured by the imaging units 12101 to 12104 as infrared cameras, and a procedure to perform pattern matching on a series of feature points that indicate the contour of an object to determine whether or not it is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the image captured by the imaging units 12101 to 12104 and recognizes the pedestrian, the audio-image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline for emphasis on the recognized pedestrian. Alternatively, the audio-image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0246] In this specification, "system" refers to an entire apparatus composed of multiple devices.

[0247] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0248] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0249] Furthermore, this technology can also be configured as follows. (1) A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. Equipped with, The control unit, When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. A photographic device. (2) When the aforementioned synchronization signal is received, the AE (Auto Exposure) settings are updated. The imaging device described in (1) above. (3) A second AE setting received by the first image sensor within a predetermined period after receiving the first AE setting will be treated as having been received at the same time as the first AE setting. The imaging device described in (2) above. (4) A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. A photographic device equipped with, When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. Shooting method. (5) A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. A computer that controls the imaging device equipped with the following: When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. A program that executes a process that includes steps. (6) A first control unit that controls imaging by the first image sensor, A second control unit that controls imaging by the second image sensor, A third control unit outputs commands to the first control unit and the second control unit, respectively, to instruct them to switch modes. Equipped with, The third control unit transmits the command to the first control unit, including the timing for reflecting the settings made by the command. The third control unit transmits the command to the second control unit. A photographic device. (7) The aforementioned command is a short command of a data sequence associated with a predetermined mode, Each of the first and second control units maintains a table in which short commands and parameters of the data column are associated, and when it receives a command from the third control unit, it refers to the table to set the parameters. The imaging device described in (6) above. (8) The table is held by the third control unit, When the power is turned on, the third control unit supplies power to the first control unit and the second control unit, respectively, and the power is held by the first control unit and the second control unit. The imaging device described in (7) above. (9) The aforementioned table further references other tables. The imaging apparatus described in (7) or (8) above. (10) The third control unit transmits the same command to the first control unit and the second control unit. The imaging device according to any one of (6) to (9) above. (11) A first control unit that controls imaging by the first image sensor, A second control unit that controls imaging by the second image sensor, A third control unit outputs commands to the first control unit and the second control unit, respectively, to instruct them to switch modes. The third control unit of the imaging device comprising: The first control unit transmits the command including the timing for reflecting the setting by the command, The second control unit transmits the command Imaging method. (12) A first control unit that controls imaging by a first imaging device, A second control unit that controls imaging by a second imaging device, A third control unit that outputs commands for instructing mode switching to the first control unit and the second control unit respectively A computer that controls an imaging apparatus including The third control unit transmits the command including the timing for reflecting the setting by the command to the first control unit, The third control unit transmits the command to the second control unit A program for causing the computer to execute a process including steps.

Explanation of Signs

[0250] 10 Imaging apparatus, 11 Sensor, 1

Claims

1. A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. Equipped with, The control unit, When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. A photographic device.

2. When the aforementioned synchronization signal is received, the AE (Auto Exposure) settings are updated. The imaging apparatus according to claim 1.

3. A second AE setting received by the first image sensor within a predetermined period after receiving the first AE setting will be treated as having been received at the same time as the first AE setting was received. The imaging apparatus according to claim 2.

4. A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. A photographic device equipped with, When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. Shooting method.

5. A control unit controls a second image sensor that captures images at a frame rate lower than that of the first image sensor. A computer that controls the imaging device equipped with the following: When a synchronization signal is received from the first image sensor, it is determined whether or not the frame is subject to downsampling. If it is determined that the frame is not subject to downsampling, the camera takes a picture. If it is determined that the frame is subject to downsampling, the camera stops taking pictures. A program that executes a process that includes steps.

Images