System, apparatus, and control method

Abstract

To provide a system capable of automatically determining imaging settings of a camera according to a motion state of a subject by integrating the motion state of the subject with an analysis result of a camera and a sensing result of a mobile device.SOLUTION: A system includes: an imaging apparatus comprising imaging means configured to capture an image of a subject, subject motion detection means configured to detect motion of the subject using the image captured by the imaging means, and exposure control means configured to control exposure of the imaging means; and a sensor device comprising sensor means for acquiring information on the subject, and capable of being attached to the subject to be imaged, unlike the imaging means having transmission means for transmitting a sensing result of the sensor means to the imaging apparatus wirelessly. The imaging apparatus is configured to perform imaging while controlling exposure of the imaging means using the sensing result of the sensor means and a result of the subject motion detection means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a technique for controlling camera shooting, and particularly to a camera system, apparatus, and control method for shooting using a wearable device.

Background Art

[0002] Techniques for controlling camera shooting based on sensing results of sensors such as acceleration sensors and human presence sensors are known.

[0003] For example, Patent Document 1 discloses controlling auto shooting of a camera located at a distance according to the situation of a portable device (wearable device).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional techniques disclosed in the above-mentioned patent documents, mainly the timing control of auto shooting is performed based on sensing results, and not even considered up to automatically performing camera shooting settings including the movement state of the subject captured by the camera. Therefore, even if the shooting timing can be controlled, there is a possibility that the captured subject will be blurred when the movement of the subject is fast. Thus, an object of the present invention is to provide a system that enables automatic determination of camera shooting settings according to the movement state of a subject by linking the movement state of the subject with the analysis result of the camera and the sensing result by a portable device.

Means for Solving the Problems

[0006] To achieve the above object, the present invention One of the imaging systems related to thisThis system includes an imaging means for capturing an image of a subject, and a means for detecting the movement of the subject using the image captured by the imaging means. Then, calculate the motion vector of the subject. A means for detecting the motion of a subject, A calculation means for calculating the reliability of the accuracy of the motion vector of the subject, a receiving means for receiving sensing results acquired by an external sensor device and transmitted from the sensor device, and during imaging An exposure control means for controlling the exposure, Prepared Imaging device and 、 Sensor means for acquiring information about a subject and, The sensor means The aforementioned The sensing results are transmitted to the imaging device. to Transmission method and, The imaging comprising Device Unlike ru , attached to the subject to be photographed The above From the sensor device An imaging system The imaging device is The transmission means and the reception means Using the sensing results and the results of the subject motion detection means, the imaging means Exposure control is performed, and the exposure control means uses a motion vector with a reliability of a threshold or higher to control the exposure of the imaging means. that It is characterized by. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide camera control that can more preferably take into account the movement of the subject and reduce subject blur. [Brief explanation of the drawing]

[0008] [Figure 1] Figure showing an example of the present invention [Figure 2] A diagram showing an example of the external appearance of the imaging system 100. [Figure 3] Flowchart illustrating the operation of camera 101 in Example 1 [Figure 4] Flowchart illustrating the operation of camera 101 in Example 2 [Figure 5] A flowchart illustrating the operation of wearable device 102. [Figure 6] Diagram explaining block matching [Figure 7] A flowchart explaining the process of determining the shooting conditions for the preparatory images. [Figure 8] A diagram showing the movement of the subject. [Figure 9]Flowchart showing the calculation process of motion vectors [Figure 10] Diagram for correcting the motion vectors of the subject [Figure 11] Diagram showing the relationship between the motion vectors of the subject and the amount of subject motion blur [Figure 12] Diagram explaining the process of correcting the exposure amount by multiplying by digital gain [Figure 13] Flowchart explaining the details of the main exposure process in Example 2 [Figure 14] Diagram explaining the configuration of the electronic front curtain shutter [Figure 15] Diagram explaining the operation of the electronic front curtain shutter [Figure 16] Diagram explaining the method of performing exposure cut-off control [Figure 17] Image diagram when there are multiple subjects [Figure 18] Flowchart explaining the case when there are multiple wearable devices 102 [Figure 19] Flowchart for determining the main wearable device 102 [Figure 20] Example of a setting screen for setting the priority order of the wearable device 102 [Figure 21] Diagram explaining the items for determining the priority of the wearable device 102 [Figure 22] Diagram explaining the method of calculating the priority order of the wearable device 102

Modes for Carrying Out the Invention

[0009] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the following embodiments do not limit the invention claimed. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the accompanying drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted. In this embodiment, a camera is linked to a wearable device such as a smartphone or a wristband-type terminal, enabling automatic exposure control of the camera.

[0010] This enables exposure control to suppress subject movement to a desired level of blur even in use cases such as unmanned shooting or selfies, thereby expanding the range of applications for automatic mode.

[0011] Figure 1 shows an example of the imaging system 100 described in this embodiment. The imaging system 100 in Figure 1 is realized with two devices: a camera 101 and a wearable device 102. Examples of the wearable device 102 include smartphones and wristband-type terminals, and its form is not limited. Figure 2 shows the external appearance of one embodiment of the present invention. Figure 2(a) shows the imaging system 100 with a digital camera 201 as the camera 101 and a smartphone 202 as the wearable device 102 connected to it.

[0012] The control unit 112 is, for example, a CPU, which reads control programs for each block of the camera 101 from the ROM 113 (described later), loads them into the RAM 114 (described later), and executes them. In this way, the control unit 112 controls the operation of each block of the camera 101.

[0013] ROM113 is an electrically erasable and recordable non-volatile memory that stores the operation programs for each block of the camera 101, as well as parameters necessary for the operation of each block.

[0014] RAM114 is a rewritable volatile memory used for deploying programs executed by the control unit 112, etc., and for temporarily storing data generated by the operation of each block in the camera 101.

[0015] The communication unit 115 communicates according to a predetermined wireless communication standard. For example, wireless communication standards include the IEEE 802.11 standard, so-called Wi-Fi, Bluetooth (registered trademark), and NFC, and it is sufficient to support at least one of them.

[0016] The optical system 121 consists of a group of lenses including a zoom lens and a focus lens, and forms an image of the subject on the imaging surface of the image sensor 122, which will be described later.

[0017] The image sensor 122 is composed of, for example, a CCD or a CMOS sensor. Each pixel of the image sensor 122 converts the optical image formed on the imaging surface of the image sensor 122 by the optical system 121 into an analog image signal, and outputs the resulting analog image signal to the A / D conversion unit 123, which will be described later.

[0018] The A / D conversion unit 123 converts the input analog image signal into digital image data, and the digital image data output from the A / D conversion unit 123 is temporarily stored in the RAM 114.

[0019] The image processing unit 124 applies various image processing functions, such as white balance adjustment, color interpolation, and gamma processing, to the image data stored in the RAM 114. The image processing unit 124 also calculates motion vectors between recorded images and detects subjects. Details of these functions will be described later.

[0020] The recording unit 125 is a removable memory card or the like, and records the image data processed by the image processing unit 124 as a recorded image via the RAM 114.

[0021] The pulse generation unit 126 supplies a scanning clock (horizontal drive pulse) and predetermined control pulses to the image sensor 122 when transitioning from a non-shooting state to a shooting state. Of the scanning clocks generated by the pulse generation unit 126, the clock for vertical scanning is input to the vertical drive modulation unit 111, which will be described later.

[0022] The vertical drive modulation unit 111 modulates the clock signal for vertical scanning from the scan clock signal generated by the pulse generation unit 126 to a predetermined clock frequency and inputs it to the image sensor 122. The vertical drive modulation unit 111 determines the scanning pattern for the reset scan performed for each line of the image sensor 122, which consists of multiple pixels. This line-by-line reset scan of the image sensor 122 realizes the function of an electronic front curtain shutter.

[0023] The gyro sensor 119 is a motion detection sensor that detects angular velocity and determines the magnitude of camera shake.

[0024] The mechanical shutter 118 consists of an openable shutter mechanism that provides a light-blocking mechanism to physically block light (hereinafter referred to as the mechanical shutter). The mechanical shutter 118 also functions as a rear curtain composed of multiple light-blocking blades (hereinafter referred to as the mechanical rear curtain). The control unit 112 controls the exposure time (shutter speed) by adjusting the timing at which the mechanical rear curtain starts moving. On the other hand, the function of the electronic front curtain is realized by sequentially reset scanning the pixels of the image sensor 122 line by line at predetermined timings.

[0025] The display unit 127 is a display device such as an LCD, and it displays images stored in the RAM 114 and the recording unit 125, as well as an operation user interface for receiving user instructions. The display unit 127 also displays images captured by the image sensor 122 during preparation shooting for composition adjustment, etc.

[0026] The configuration of camera 101 has been explained above.

[0027] Next, the configuration of the wearable device 102 will be explained using Figure 1. The wearable device 102 is equipped with a touchscreen display 141, and the liquid crystal display 142 displays text, images, and icons. The touchscreen 143 can be operated by detecting gesture operations.

[0028] The front camera 134 includes a lens and an image sensor such as a CCD or CMOS that converts optical images into electrical signals. This front camera 134 is a small camera module equipped with AF (autofocus), aperture, and shutter speed adjustment functions. The front camera 134 then captures images of objects facing the touchscreen display 141.

[0029] The illuminance sensor 145 acquires illuminance information of the subject being captured by the front camera 134 and the rear camera 135, and uses this information to adjust the exposure time and ISO sensitivity during shooting.

[0030] The control unit 138 is, for example, a CPU, which reads control programs for each block of the wearable device 102 from the ROM 151 (described later), loads them into the RAM 152 (described later), and executes them. In this way, the control unit 138 controls the operation of each block of the wearable device 102. The control unit 138 provides camera functions by controlling the touchscreen 143, switch 144, front camera 134, illuminance sensor 145, rear camera 135, and light 136, etc.

[0031] ROM151 is an electrically erasable and recordable non-volatile memory that stores the operating programs for each block of the wearable device 102, as well as parameters necessary for the operation of each block.

[0032] RAM152 is a rewritable volatile memory used for deploying programs executed by the control unit 112, etc., and for temporarily storing data generated by the operation of each block in the wearable device 102.

[0033] When the sound output is turned on by switch 144, speaker 139 outputs the shutter sound and warning sound used during imaging.

[0034] Connector 133 is used to connect the wearable device 102 to an external device. For example, an AC adapter for charging the battery provided in the power module 132 (described later) is connected to connector 133. Connector 133 is also used for inputting and outputting image data and audio data to and from a non-volatile memory connected from an external source. Note that connector 133 may be a specially designed terminal such as a dock connector, or a general-purpose terminal such as USB (Universal Serial Bus).

[0035] The rear camera 135 is a small camera module similar to the front camera 134. The rear camera 135 captures images of the subject on the opposite side of the front camera 134. The light 136 is a light-emitting module that functions as a flash when the rear camera 135 captures images.

[0036] The communication module 131 communicates according to a predetermined wireless communication standard. For example, wireless communication standards include the IEEE 802.11 standard, so-called Wi-Fi, Bluetooth (registered trademark), and NFC, and the communication module 131 only needs to support at least one of them. Specific communications include inputting and outputting image data obtained through imaging and downloading function-adding program modules to the wearable device 102. It is also used when transmitting information from a group of sensors (illuminance sensor 145, acceleration sensor 146, gyro sensor 147, depth sensor 148), which will be described later, to the camera 101.

[0037] The power module 132 has a rechargeable battery and supplies power to the entire wearable device 102. For example, a lithium-ion battery or a nickel-metal hydride battery can be used as the battery in the power module 132.

[0038] The accelerometer 146 detects the direction and magnitude of acceleration acting on the wearable device 102. The accelerometer 146 is capable of detection in three axes: X, Y, and Z.

[0039] The gyro sensor 147 detects the angle and angular velocity of the wearable device 102.

[0040] The depth sensor 148 measures the distance from the camera to the subject being photographed. Methods for measuring distance include measuring the time it takes for infrared light, light, or ultrasound to reflect off an object and bounce back, and arranging multiple cameras or pixels in parallel to obtain depth information of the subject from the parallax image.

[0041] Figure 2 shows an example of the external appearance of the imaging system 100. Figures 2(a) to (c) represent combinations of a camera 101 and a wearable device 102 that work together as the imaging system 100. The camera 101 and the wearable device 102 are connected wirelessly, for example, using Bluetooth (trademark registered), to communicate and work together with the imaging system 100. 201 shows a digital camera as an example of the camera 101. 202 shows a smartphone as an example of the wearable device 102, and this smartphone often has a camera function. Therefore, as with the front camera 134 and rear camera 135 mounted on the smartphone 204 in Figure 2(c), it is possible to use the camera function to function as the camera 101 side rather than the wearable device 102 side. In addition, the wearable device 102 can also be in the form of a wristband type terminal 203 or other forms other than a smartphone.

[0042] The above describes the appearance and system configuration of the imaging system 100.

[0043] (First embodiment) The following describes the processing of the imaging system 100 according to the first embodiment (Example 1) of the present invention, with reference to the flowcharts in Figures 3 and 5. In Example 1, the wearable device 102 is worn by the subject to be photographed, and the camera 101 is configured as an externally positioned remote camera. Then, subject movement information is sensed through the wearable device 102, and the subject movement information acquired by the wearable device 102 is used as auxiliary information to determine the shooting conditions for the camera 101.

[0044] First, the operation of the wearable device 102 will be explained using Figure 5. The following processes are achieved by the control unit 138 controlling each part of the device according to a program stored in the ROM 151.

[0045] In step S501, the user first turns on the power to the wearable device 102. The wearable device 102 then performs a standby operation to receive sensing signals from sensors (accelerometer 146, gyroscope 147, depth sensor 148, etc.) in order to detect motion information of the subject.

[0046] In step S502, the communication unit 115 of the camera 101 acquires the sensing signal obtained in step S501 at regular time intervals. For example, acceleration information of the wearable device 102 is acquired as a sensing signal. In order to acquire acceleration information, the output from the acceleration sensor 146 is acquired periodically at predetermined time intervals. As a result, it is possible to obtain acceleration information of the subject at the part of the body to which the wearable device 102 is attached. This means that even if the acceleration sensor 146 is not used, it is also possible to indirectly obtain acceleration information of the subject at the part of the body to which the wearable device 102 is attached by using another sensor that can detect the movement state of the subject. For example, by acquiring the change in distance from the camera 101 to the wearable device 102 using the depth sensor 148, the movement speed and acceleration information of the subject can be calculated per unit time.

[0047] In step S503, the user first attaches the wearable device 102 to a specific location, such as the wrist, where it will not interfere with movement. The wearable device 102 senses that it is attached to the subject using a tactile sensor (not shown) or the like. The wearable device 102 can also be freely attached to any location where the subject's movement can be observed. In this case, the attachment location of the wearable device 102 may be identified from image data acquired by the camera 101. There are also known methods for pre-setting the attachment location of the wearable device 102 to the subject, or for pre-recording the acceleration and velocity of a specific location within a predetermined time period and identifying the moving subject location based on the actual movement (Patent No. 6325581). For example, in step S502, acceleration information is acquired from the acceleration sensor 146 of the wearable device 102. Therefore, the attachment location of the wearable device 102 can be identified by comparing the acceleration changes recorded per predetermined time for each attachment location with the acquired acceleration information.

[0048] In step S504, the communication module A131 of the wearable device 102 transmits the acceleration information obtained in step S502 and the information of the part of the wearable device 102 that is attached to the camera 101 as subject motion information.

[0049] The above describes the processing of the wearable device 102. Next, the operation of the camera 101 will be explained using the flowchart in Figure 3. The processing shown in the flowchart in Figure 3 is realized by the control unit 112 controlling each part of the camera 101 of the imaging system 100 according to the program stored in the ROM 113.

[0050] In step S301, the communication unit 115 of the camera 101 receives subject motion information transmitted from the wearable device 102.

[0051] In step S302, the user begins preparatory shooting, such as composing the shot, using the camera 101. During this preparatory shooting period, the camera 101 continuously captures images and displays the recorded images on the display unit 127. The user adjusts the composition while viewing the displayed preparatory shooting images. The processes described later in steps S303, S304, and S305 are performed during the preparatory shooting period.

[0052] In step S303, the control unit 112 of the camera 101 determines the shooting conditions for the preparatory image to be captured in order to detect the motion vector of the subject in the composition. As will be described in detail later, the control unit 112 uses the amount of subject movement in the composition when the preparatory image is taken with the initial shooting conditions and the subject movement information transmitted from the communication module A131 of the wearable device 102 to set a shutter speed that minimizes subject blur on the part of the body where the wearable device 102 is attached (the part of the subject that is the focus of attention).

[0053] In step S304, the display unit 127 displays the subject in the composition and the shooting settings (shutter speed, ISO sensitivity, F-number, etc.).

[0054] In step S305, the control unit 112 of the camera 101 determines whether remote release has been activated. Remote release is a control that sends a timing signal to start exposure from the camera 101 via a wearable device 102 connected to the camera by the user. Remote release can be activated by the output of the gyro sensor 147 when the user wearing the wearable device 102 performs a predetermined gesture, or by pressing the release button (not shown) on the wearable device 102. Alternatively, it determines whether the photographer (user) directly pressed the shutter button (not shown) on the camera 101. In that case, the user presses the shutter button on the camera 101 in time with the shutter timing while looking at the subject displayed on the display unit 127. If the shutter button on the camera 101 is pressed, the process proceeds to the main exposure process in step S306. Conversely, if it is not the shutter timing, the process returns to step S301, and the shooting settings can be redone.

[0055] In step S306, the control unit 112 of the camera 101 performs exposure processing with the shooting settings performed in the above steps in order to capture an image, and records the captured image in the ROM 113.

[0056] In this way, during the preparation shooting, the user checks the motion blur notification image displayed on the display unit 127 and repeatedly sets the exposure time for the main shoot until the desired motion blur is achieved, and then presses the shutter button when the opportunity arises.

[0057] In step S305, when the user presses the shutter button on camera 102, camera 101 takes a picture and records the captured image in ROM 113.

[0058] Next, the process of step S303, which is a feature of the present invention, will be explained with reference to the flowchart in Figure 7.

[0059] In step S701 of Figure 7, the control unit 112 of the camera 101 sets the initial shooting conditions, and the camera 101 captures a series of preparatory images. The initial shooting conditions here mainly refer to the frame rate and shutter speed. The highest frame rate and fastest shutter speed are set within a range that does not affect the process of calculating evaluation values ​​used for controlling auto functions such as AE (auto exposure) and AF (autofocus) control, which are common in cameras. Furthermore, even when the shutter speed is set to a high speed, the optical system 121 is controlled in accordance with the shutter speed so that shooting can be done under appropriate exposure conditions. For example, the aperture of the lens of the optical system 121 and the ISO sensitivity setting of the camera 101 are controlled to enable shooting under appropriate exposure conditions (exposure control). The camera 101 captures a series of images in chronological order with the initial shooting conditions set as described above. It is desirable that the captured images have almost no cumulative blur of moving subjects and that the amount of subject movement between captured consecutive images is as small as possible. However, such conditions tend to increase ISO sensitivity, which can lead to drawbacks such as the acquisition of noisy image data at high ISO settings. On the other hand, because the subject's movement is small, it has the advantage of making it easier to capture the movement of fast-moving subjects.

[0060] In step S702, the image processing unit 124 of the camera 101 calculates the motion vector of the subject from the time-series consecutive preparatory images captured in step S701.

[0061] First, we will explain the motion vector calculated from the preparatory images using Figure 8. Figure 8 shows the movement of a subject. Figure 8(a) is an example of a scene where dog 801 is running to the left and dog 802 is standing still. The motion vector represents the amount of horizontal and vertical movement of the subject between the preparatory images. An example of this motion vector is shown in Figure 8(b).

[0062] Figure 8(b) shows an example of motion vectors in the preparatory image taken in Figure 8(a). In the example in Figure 8(b), the running dog 801 is detected as having a motion vector to the left, while the other stationary dog ​​802 and the background fence are detected as having a motion vector of 0, so their motion vectors are not shown.

[0063] Next, the method for calculating motion vectors will be explained in detail with reference to Figures 9 and 6. Figure 9 is a flowchart of the motion vector calculation process. In the following flowchart, the control unit 112 controls each part of the camera 101 according to the program stored in the ROM 113. In this invention, the block matching method is used as an example of a motion vector calculation method, but the motion vector calculation method is not limited to this example, and other methods such as the gradient method may also be used.

[0064] In step S901 of Figure 9, two temporally adjacent pre-captured images are input to the image processing unit 124 of the camera 101. The image processing unit 124 then sets the M-th pre-captured image as the reference frame and the M+1-th pre-captured image as the reference frame.

[0065] In step S902, the image processing unit 124 places an N×N pixel reference block 602 in the reference frame 601, as shown in Figure 6.

[0066] In step S903, the image processing unit 124 sets the search range 605 for the reference frame 603 as (N+n)×(N+n) pixels surrounding the center coordinates 604 of the reference block 602 of the reference frame 601, as shown in Figure 6.

[0067] In step S904, the image processing unit 124 performs a correlation calculation between the reference block 602 of the reference frame 601 and the reference block 606 of N×N pixels at different coordinates within the search range 605 of the reference frame 603, and calculates a correlation value. The correlation value is calculated based on the sum of absolute values ​​of inter-frame differences for the pixels of the reference block 602 and the reference block 606. In other words, the coordinate with the smallest sum of absolute values ​​of inter-frame differences is the coordinate with the highest correlation value. Note that the method for calculating the correlation value is not limited to calculating the sum of absolute values ​​of inter-frame differences; for example, a method of calculating the correlation value based on the sum of squared inter-frame differences or the normal cross-correlation value may also be used. In the example in Figure 6, it is assumed that the reference block 606 has the highest correlation.

[0068] In step S905, the image processing unit 124 calculates a motion vector based on the reference block coordinates showing the highest correlation value obtained in step S904. In the example in Figure 6, within the search range 605 of the reference frame 603, the motion vector is determined based on the same coordinate 604 corresponding to the center coordinate of the reference block 602 of the reference frame 601 and the center coordinate of the reference block 606. In other words, the distance and direction between the same coordinate 604 and the center coordinate of the reference block 606 are determined as the motion vector.

[0069] In step S906, the image processing unit 124 determines whether or not motion vectors have been calculated for all pixels of the reference frame 601. If the image processing unit 124 determines in step S906 that motion vectors have not been calculated for all pixels, it returns to step S902. In step S902, an N×N pixel reference block 602 is placed on the reference frame 601, centered on the pixels for which motion vectors have not been calculated, and the processing from steps S903 to S905 is performed as described above. That is, the image processing unit 124 repeats the processing from steps S902 to S905 while moving the reference block 602 shown in Figure 6 to calculate the motion vectors for all pixels of the reference frame 601. The unit for calculating the motion vectors may be at the pixel level or at each unit obtained by dividing the image into a predetermined number of divisions. The motion vectors are calculated by performing the above processing between preparation images with similar imaging times.

[0070] Next, in step S703, the image processing unit 124 calculates a vector corresponding to the main part of the subject using the subject motion information acquired from the wearable device 102 and the subject motion vector obtained from the camera 101.

[0071] The image processing unit 124 uses the subject motion information obtained by the wearable device 102 to identify vectors corresponding to the main parts of the subject that the user wants to reduce motion blur on, and then performs a process to correct the motion vectors of the corresponding main parts of the subject. This will be explained in detail using Figure 10.

[0072] First, in order to identify the vectors corresponding to the main parts of the subject, the image processing unit 124 finds a group of candidate motion vectors of the subject. This will be explained using Figure 10(a). Figure 10(a) shows a group of motion vectors (1011, 1012, 1013, 1014) of candidate subjects that will be the main parts. The main part information is the information corresponding to the attachment site of the wearable device 102 in the subject motion information transmitted in step S504. The correspondence between the main parts of the subject and the motion vectors of the subject is achieved by the image processing unit 124 selecting the motion vectors of the subject that correspond to the main parts from the preparation images for which the motion vectors of the subject are to be determined. As a method for detecting the main parts from the preparation images, general subject recognition techniques may be used. For example, if the attachment site of the wearable device 102 is the dog's head, the image processing unit 124 detects the dog's head to which the wearable device 102 is attached within the range in the preparation images for which the motion vectors of the subject have been determined. The image processing unit 124 then selects a group of motion vectors (1011, 1012, 1013, 1014) of subjects located within a predetermined distance from the detected dog's head region, and further detects the motion vector (1011) of the subject with the largest amount of movement from among them, and uses this as the motion vector of the main body part.

[0073] Next, the process of correcting the motion vectors of the subject in the relevant main parts will be explained in detail using Figures 10(b) and (c). Correcting the motion vectors of the subject in the main parts is a process that, in contrast to the motion vector calculation process which has a slow output update rate, corrects the motion vectors of the subject by using acceleration information of the attached part transmitted from the wearable device 102 which has a fast output update rate, thereby simulating an improvement in the update rate of the motion vectors of the subject and obtaining motion vectors of the subject that correspond to changes in the subject's movement.

[0074] Figure 10(b) shows the motion vector of the subject and the timing of acquiring the subject's motion information.

[0075] The motion vector of the subject is calculated by using two or more consecutive preparatory images in a time series to determine the amount of movement between the preparatory images used for the motion vector. For example, to obtain the motion vector 1031 of the subject, it is not possible until the camera 101 has acquired at least two frames of preparatory images 1021 and 1022. Similarly, to obtain the next motion vector 1032 of the subject, it is not possible until the preparatory image 1023 has been acquired. If the subject's movement changes suddenly during the blank period 1041 between calculating the motion vector 1031 and the motion vector 1032, the subject's movement at that time cannot be correctly detected because the update rate of the motion vector is slow. On the other hand, acceleration information as subject movement information detected by the wearable device 102 does not depend on preparatory images and can directly detect the movement of the device, so it is generally possible to detect it at high speed (1051).

[0076] Since the preparatory images that can be acquired by a typical digital camera 101 are at a high speed of around 120fps, the motion vector of the subject will have an update rate of 120fps or less. On the other hand, the output update rate of the accelerometer 146 installed in a typical wearable device 102, such as a smartphone, is 10 to 100 times or more the output update rate of the motion vector of the subject.

[0077] Therefore, by using acceleration information detected by the wearable device 102 to correct the motion vector of the subject, the image processing unit 124 can obtain a more accurate motion vector of the subject even during periods when the motion vector of the subject is not updated. Also, since the motion vector of the subject depends on the pre-captured image, it may not be possible to obtain the motion vector of the subject from low-contrast subjects or from images where accumulated blur or out-of-focus occurs. In other words, it may not be possible to acquire subject motion information with the camera 101 alone, which may result in a slower vector update rate. Therefore, it is effective to correct and update the motion vector of the subject using sensor information from the wearable device 102, which has a fast update rate.

[0078] Next, we will explain the correction process for the motion vectors of the main body parts using Figure 10(c). Figure 10(c) shows the motion vector 1061 of the main body part, the motion vector 1062 of the subject corrected when the subject's movement slowed down before updating the subject's motion vector, and the motion vector 1063 of the subject corrected when the subject's movement sped up.

[0079] Furthermore, since the motion vectors of major parts in an image have both angle and magnitude in multiple directions, they are converted to vector magnitudes using Equation 1. Generally, motion vectors used in images have directionality on a two-dimensional coordinate system, so by applying Equation 1, they can be converted to vector magnitudes as scalars.

[0080]

number

[0081] The motion vector of the subject can be corrected by performing gain processing that corresponds to the acceleration changes of the main body parts up to the point in time when the motion vector of the subject is updated. Therefore, if the amount of acceleration change calculated using the acceleration information of the main body parts detected by the wearable device 102 is α (1 if there is no change in acceleration), the correction of the motion vector of the subject can be expressed as shown in Equation 2. Corrected motion vector of the subject = α × motion vector of the subject ... Equation 2

[0082] Using equation 1 above, the motion vectors of the main parts of the subject are corrected. If the acceleration change α is less than 1, the result is 1062, and conversely, if the acceleration change α is greater than 1, the result is 1063, calculated from the motion vector of the subject before correction. By correcting the motion vectors of the subject in this way, the amount of blur of the main parts is determined so that there is as little time difference as possible with the real-time movement of the subject.

[0083] Next, in step S704, under the direction of the control unit 101, the camera 100's blur estimation unit (not shown) estimates the amount of motion blur generated by the subject at the shutter speed set by the user, based on the motion vector of the main part of the subject calculated in the above step. This amount of motion blur is calculated using the following formula, with respect to the imaging frame rate of the preparatory image used to calculate the motion vector of the subject, the shutter speed set by the user on the camera 101, and the motion vector of the subject. Subject motion blur amount = Subject motion vector * (Frame rate (fps) / Shutter speed (s)) ... Equation 3

[0084] Regarding equation 3 above, the relationship between the motion vector of the subject and the amount of motion blur of the subject will be explained using Figure 11. Figure 11 shows the relationship between the motion vector of the subject and the amount of motion blur of the subject. For example, since the motion vector of the subject is calculated using the frames before and after the preparation image, which is updated at a frame rate of 60fps, the update frame rate of the motion vector of the subject 1101 is also 60fps. On the other hand, the amount of motion blur of the subject is the amount of blur caused by the subject moving during exposure, and therefore corresponds to the shutter speed set by the user. For example, if the user sets the shutter speed to 1 / 120 second, the time between the frames before and after the preparation image for which the motion vector of the subject is calculated will be 1 / 60 second. Therefore, if the motion vector of the subject 1101 is 10 pixels, then half of that length, 5 pixels, will be the amount of motion blur of the subject 1102.

[0085] Next, in step S705, camera 101 compares the amount of subject motion blur calculated in step S704 with the allowable motion amount, and changes the shutter speed of the next preparatory image to be taken so that the amount of subject blur is less than or equal to the allowable motion amount, thereby changing the shooting conditions of the preparatory image. The allowable motion amount is the amount of motion blur that is not noticeable when shot at a predetermined shutter speed. The magnitude of the allowable motion amount is determined by the size of the image sensor such as a CCD or CMOS sensor, the number of pixels, and the resolution of the display used for display. For example, if the image sensor is APS-C, the number of pixels is 200,000, and the allowable motion amount for a PC display with Full HD (1920 x 1080 pixels) is 5 pixels or less. In order for camera 101 to capture a preparatory image so that the amount of motion blur is less than or equal to the allowable motion amount, the shutter speed is determined using the following equations 4 and 5. n = Subject motion blur / Allowable motion ... Equation 4

[0086] At this point, if n obtained by Equation 4 is greater than 1, it indicates that there is a high probability of motion blur occurring when shooting at the currently set shutter speed. If it is 1 or less, it means that the shutter speed is one at which motion blur is unlikely to occur. Therefore, the appropriate shutter speed that minimizes motion blur is calculated using Equation 5 below. Update shutter speed (s) ≤ Set shutter speed (s) * 1 / n ... Equation 5

[0087] To explain using specific numerical values, the motion blur amount 1102 in Figure 11 is 5 pixels, and the allowable motion amount is also 5 pixels. Therefore, according to Equation 4, n=1, which shows that the currently set shutter speed has little effect on subject blur. Therefore, according to Equation 5, the shutter speed for the shooting conditions of the preparation image (which will be the actual exposure conditions if not changed) should be set to an exposure time faster than 1 / 120 (in this case, we will not change it from 1 / 120s). Also, if there is sufficient light for shooting, the shutter speed may be set even faster than 1 / 250 depending on the ISO sensitivity and aperture.

[0088] The above explains an example of updating the shutter speed as a shooting condition for the preparatory image. Furthermore, to improve the accuracy of detecting the subject's motion vector, you can increase the frame rate used to capture the preparatory image and speed up the update rate for calculating the subject's motion vector. The conditions for speeding up the update rate are as follows.

[0089] Update frame rate (fps) ≥ Set frame rate (fps) * n ... Equation 6 The frame rate and shutter speed mentioned above are important shooting conditions for motion detection. Furthermore, to capture images at the correct brightness, the aperture value and ISO sensitivity are also changed in accordance with the frame rate and shutter speed, and controlled to maintain a constant exposure value.

[0090] Regarding the detailed processing of step S303, the process of determining the shooting conditions for the preparation image (which will remain the same for the actual shooting unless changed) was explained using the processes of steps S701 to S705 in Figure 7.

[0091] The processing of the imaging system 100 in Example 1 has been described above. Specifically, subject movement information is sensed through the wearable device 102. Then, the subject movement vector is updated using the subject movement information acquired by the wearable device 102 as auxiliary information, and the shooting conditions of the camera 101 are determined. According to the present invention, by coordinating with the wearable device 102, the detection accuracy of the subject movement vector calculated by the camera 101 is improved, and it becomes possible to set a shutter speed that reduces subject blur. As a result, the photographer can set the shutter speed to suppress the subject movement to the desired level of blur without touching the camera, and adjust the exposure of the shot. The present invention makes it possible to expand the usage scenarios of automatic shooting.

[0092] In Example 1, a method for calculating motion blur was described, which involves converting the motion vector of the subject to match the shutter speed set by the user. However, it is not necessary to convert the motion blur to match the shutter speed. For example, the same process can be easily achieved by comparing the motion vector of the subject with a pre-set threshold and changing the shutter speed to a faster value than the current setting if the threshold is exceeded.

[0093] Example 1 describes a method for identifying the main parts of a subject and selecting the motion vector of the subject in those main parts. Alternatively, the motion vector of the subject with the fastest movement may be selected from the motion vectors of the subject obtained from the preparatory captured images.

[0094] In Example 1, a method for identifying the main part of a subject and selecting the motion vector of the subject in the main part was described. Note that if the camera 101 is equipped with an acceleration sensor similar to the acceleration sensor 146 mounted on the wearable device 102, a motion vector of a subject different from the motion of the acceleration sensor mounted on the camera 101 may be selected. Since the acceleration sensor mounted on the camera 101 provides motion information of the main body that moved the camera 101, selecting a different motion vector makes it possible to filter out motion information of subjects other than the main moving subject.

[0095] Example 1 describes a method for identifying the main parts of a subject and selecting the motion vector of the subject in those main parts. Alternatively, from the calculated motion vectors of the subject, it is also possible to select motion vectors of the subject that are in the center of the field of view when the camera 101 is shooting, or motion vectors of the subject near the area that is the target of autofocus in the image.

[0096] Example 1 describes a method for identifying the main part of a subject and selecting the motion vector of the subject in that main part. If the wearable device 102 is visible in the preparatory image, the position of the wearable device 102 may be detected directly from the image.

[0097] Furthermore, before selecting the motion vectors of the main parts of the subject, a selection process may be performed on the motion vectors of the subject obtained from the preparation images. For example, in calculations such as template matching performed in the process of obtaining the motion vectors of the subject, correlation calculations may be performed. In this case, vectors whose correlation value falls below a predetermined threshold are judged to be motion vectors of the subject with low confidence (confidence below the threshold). This makes it possible to extract only motion vectors of the subject with higher accuracy, where the correlation value is above the threshold (confidence above the threshold).

[0098] (Second embodiment) Next, a second embodiment of the present invention (Example 2) will be described in detail with reference to the drawings. In Example 2, exposure control is performed based on the amount of motion blur of the subject during the exposure process, thereby enabling the acquisition of an image with reduced motion blur. Note that the processing on the wearable device 102 side of the imaging system 100 in Example 2 is the same as in Example 1, so the explanation will be omitted. The operations performed by the control unit 112 of the camera 101, which are characteristic of Example 2, will be explained using the flowchart in Figure 13. Note that the following processing is realized by the control unit 112 controlling each part of the device according to the program stored in the ROM 113 for the camera 101 as part of the imaging system 100. Similarly, the wearable device 102 is realized by the control unit 138 controlling each part of the device according to the program recorded in the ROM 151. The same reference numerals are used for the same steps as in Example 1, and detailed explanations will be omitted.

[0099] First, the shooting operation of camera 101 will be explained using Figure 4. The process from steps S301 to S305 in Figure 4 is the same as the process from steps S301 to S305 in Figure 3 of Example 1, so the explanation will be omitted. In Example 2, the main exposure process in step S401 differs from that of Example 1.

[0100] In step S401, the camera 101 interrupts the exposure process based on the amount of motion blur relative to the point of focus of the subject during exposure, thereby suppressing the occurrence of subject blur during shooting.

[0101] Next, the control of the main exposure process in step S401 performed by the control unit 112 of the camera 101, based on the amount of motion blur relative to the area of ​​focus of the subject during exposure, will be explained in detail using the flowchart in Figure 13.

[0102] In step S306 of Figure 13, the camera 101 starts the exposure process, similar to step S306 of Example 1.

[0103] The configuration of the electronic front curtain shutter and the shooting operation during this exposure process will be explained in detail using Figures 14 and 15.

[0104] Figure 14 is a front view showing the image sensor 122 and the mechanical rear curtain 1403 as seen from the lens side along the optical axis. It shows the state when the reset scan performed on the image sensor 122 after shooting has started and when the mechanical rear curtain 1403 is in the middle of its movement. Arrow 1401 indicates the operating direction of the reset scan (the movement direction of the electronic front curtain 1407) and the movement direction of the mechanical rear curtain 1403. The mechanical rear curtain 1403, which is composed of the mechanical shutter 118 in Figure 1, is shown to be blocking light from a part of the image sensor 122. Furthermore, the reset line 1408 is the line of the reset scan performed on the image sensor 122 (reset line), and corresponds to the edge of the electronic front curtain 1407 as an operation to reset the charge accumulation amount of the pixels to zero. The region 1406 formed by the slit between the reset line 1408 and the end 1405 of the mechanical rear curtain 1403 is controlled to move in the direction of arrow 1401 as the electronic front curtain 1407 and the mechanical rear curtain 1403 move. The time from when the reset line 1408 passes, that is, when the pixels are sequentially reset line by line in the direction of arrow 1401, until the mechanical rear curtain 1403 blocks light is the charge accumulation time due to exposure of the pixels. Thus, since the reset line 1408 moves in the direction of arrow 1401 and charge accumulation for each line begins, the timing of the start of charge accumulation differs for each line of the image sensor 122.

[0105] The timing of charge accumulation will be explained in detail using Figure 15. Figure 15(a) is a conceptual diagram of charge reset and readout start, and Figure 15(b) is a diagram explaining the timing of charge reset and readout for each line. Lines 1501 to 1510 in Figure 15(b) represent the timing of charge reset processing for each line, with the end line 1501 being the first to read out charge and then reset. Conversely, the end line 1510 is the last line to be reset. In this way, the reset timing is controlled for each line. Because the reset timing differs for each line, the control unit 112 controls the charge readout time so that the exposure time is the same for each line from line 1511 to line 1520, so that the charge accumulation time is the same for each line.

[0106] In step S1301 of Figure 13, the control unit 112 corrects the motion vector of the subject calculated by the image processing unit 124 of the camera 101 immediately before the main exposure, which was detected in step S303 of Figure 4. The correction of the motion vector of the subject is performed based on the output of the acceleration sensor 146 included in the subject motion information from the wearable device 102 during exposure. Once the main exposure begins, the camera 101 will not be able to capture a preparatory image unless there are multiple image sensors 122. Therefore, the motion vector of the subject cannot be updated during the main exposure. For this reason, estimation is performed on the motion vector of the subject at the area of ​​interest calculated up to immediately before the main exposure. For estimation, acceleration information from the acceleration sensor 146 is used to correct the motion vector of the main area of ​​the subject, which corresponds to the processing in step S703 of Example 1, and then the processing is performed to convert it into motion blur. Through the above processing, it becomes possible to estimate the amount of motion blur even during the main exposure of the camera 101.

[0107] In step S1302, the control unit 112 of the camera 101 determines whether the amount of motion blur estimated in step S1301 exceeds the allowable amount of motion. If it exceeds the allowable amount of motion, the process proceeds to step S1303. Conversely, if it does not exceed the allowable amount of motion blur, the process proceeds to step S1304.

[0108] In step S1304, the control unit 112 of the camera 101 determines whether the shooting conditions required in step S303 in Figure 4 are met. The shooting conditions mainly involve determining whether the accumulated blur during exposure is affected. This can be determined by whether the exposure time corresponding to the shutter speed set before the actual exposure has been met. If the shooting conditions are met, the process proceeds to step S1305 and the exposure is terminated.

[0109] In step S1303, the control unit 112 of the camera 101 determines that if the image sensor 122 is exposed any further, subject blur will occur in the image, and closes the mechanical shutter 118 earlier than the exposure time set in the shooting conditions. This blocks light from entering the image sensor 122. The process then proceeds to step S1305 to end the exposure. The method of controlling the exposure termination in step S1303 will be explained in detail with reference to Figure 16.

[0110] Figure 16 shows a timing chart of the process in which the control unit 112 of the camera 101 controls the mechanical rear curtain and interrupts exposure when the amount of subject motion blur exceeds the allowable motion amount (allowable threshold) during the exposure process. By controlling the mechanical rear curtain and controlling the exposure time when a situation arises in which subject blur is likely to occur in the image during the exposure, subject blur can be reduced.

[0111] Lines 1501 to 1510 in Figure 16 (same shooting conditions as Figure 15(b)) are reset lines. After the reset process on line 1501 begins, ambient light reaches the image sensor 122, and charge accumulation begins. If no accumulation blur occurs in the image due to the movement of the camera body 101 during exposure, charge accumulates up to the next reset process lines 1511 to 1520 (same shooting conditions as Figure 15(b)). This time, we will explain a method for controlling charge accumulation when the amount of subject motion blur during exposure exceeds the allowable amount of motion.

[0112] The control unit 112 executes the reset process at the first reset line 1501 in Figure 16 and starts exposure. At this time, if the amount of motion blur of the subject becomes large and the control unit 112 detects that it exceeds the allowable threshold for the amount of motion blur of the subject at timing 1611 in Figure 16, the control unit 112 drives the mechanical rear curtain and closes the mechanical shutter 118 so that ambient light does not reach the image sensor 122. Due to the operation of the mechanical shutter 118, the actual exposure time is between 1601 and 1610 in Figure 16. In other words, since there is no exposure from 1601 to 1610 to line 1511 to line 1520, no charge is accumulated on the image sensor 122. Therefore, motion blur caused by the movement of the subject can be prevented. Although charge accumulation was stopped by controlling the mechanical shutter 118, for shutters that do not have a mechanical shutter 118, the same process can be performed by generating a reset pulse from the pulse generation unit 126 to interrupt charge accumulation.

[0113] Next, in step S1306 of Figure 13, the control unit 112 determines whether the set exposure time has been reached. If it determines that the exposure time is still shorter than the set time, the process proceeds to step S1307.

[0114] In step S1307, the control unit 112 performs signal processing to the acquired image data, applying the difference in exposure time as a digital gain to compensate for the insufficient exposure time during image data acquisition, so that the image data has the brightness of the original exposure time. The digital gain obtained from the difference in exposure time can be calculated using the following formula.

[0115] Digital gain = Exposure time set at the start of imaging / (Exposure time set at the start of imaging - (Time from the start of exposure to the release of the mechanical shutter) ... Formula 7 By multiplying the image data by the digital gain calculated using Equation 7, the brightness of the image data is corrected to the brightness corresponding to the expected exposure time. For more precise gain correction, the digital gain can be calculated for each horizontal line of the image data, and the image data can be multiplied by the digital gain for each horizontal line.

[0116] On the other hand, when the set exposure time is reached, the control unit 112 reads the charge for each line and performs a charge reset process for the lines from which reading has been completed, thereby ending the exposure of the camera 101 and acquiring image data.

[0117] Next, the correction process for insufficient exposure performed by the image processing unit 124 in step S1307 will be explained in detail using Figure 12. As an example, in Figure 12, image signal 1201 is the image signal when image is captured with the target exposure, while image signal 1202 is the image signal that does not reach the target exposure because the exposure was interrupted midway and the exposure time was insufficient. In Figure 12, the horizontal axis represents the subject brightness, and the vertical axis represents the image signal level at the subject brightness on the horizontal axis. For example, image signal 1202 is the image signal when image is captured with half the exposure time compared to 1201, which is the target exposure. The exposure when the camera 101 captures an image is generally determined by the F-number, ISO sensitivity, and shutter speed (exposure time). Therefore, if exposure is interrupted midway, the exposure will be insufficient compared to the target exposure by the amount by which the exposure time was shortened. If the exposure time is halved, the exposure will be halved. Therefore, by applying a digital gain of 2, image signal 1202 can be adjusted to the same exposure as image signal 1201, and the original target exposure can be obtained. By performing the above processing to correct each line of the image sensor 122, it becomes possible to photograph the subject with the original target exposure brightness even if the exposure is interrupted.

[0118] The method by which the imaging system 100 of Example 2 controls exposure based on the degree of subject blur during exposure has been explained using the flowchart in Figure 13. These processes make it possible to obtain images with reduced subject blur by controlling the exposure time, even in situations where it is difficult to change the shutter speed during exposure.

[0119] (Third embodiment) Next, a third embodiment (Example 3) of the present invention will be described in detail with reference to the drawings. In Example 3, multiple subjects who are to be photographed wear wearable devices 102, and the camera 101 is set to be external. An example of a scene in which multiple subjects are wearing wearable devices 102 is shown in Figure 17. Figure 17 shows an example in which six subjects, Subjects A to F, are wearing wearable devices 102 1701 to 1708. In this example, Subject A is wearing wearable device 1701, Subject B is wearing wearable devices 1702, 1703, and 1704, Subject C is wearing wearable device 1705, Subject D is wearing wearable device 1706, Subject E is wearing wearable device 1707, and Subject F is wearing wearable device 1708. Subject movement information of each subject is sensed through these wearable devices 102, and the acquired subject movement information is used as auxiliary information to determine the shooting conditions of the camera 101. Note that the processing on the wearable device 102 side of the imaging system 100 in Example 3 is the same as in Example 1, so the explanation will be omitted. The operations performed by the control unit 112 of the camera 101, which are characteristic of Example 3, will be explained using the flowchart in Figure 18. Note that the following processing is realized by the control unit 112 controlling each part of the device according to the program stored in the ROM 113 of the imaging system 100. Similarly, the wearable device 102 is realized by the control unit 138 controlling each part of the device according to the program recorded in the ROM 151. The same reference numerals are used for the same steps as in Example 1, and detailed explanations will be omitted.

[0120] The shooting operation of camera 101 is the same as the processing in steps S301 to S306 in Figure 3 of Example 1, so the explanation will be omitted. The processing in step S303, which is a feature of Example 3, will be explained with reference to the flowchart in Figure 18, but the part that is the same as the processing in steps S701 to S705 in Figure 7 of Example 1 will be omitted from the explanation.

[0121] In step S701 of Figure 18, the control unit 112 of the camera 101 reads the initial shooting conditions from the ROM 113 of the camera 101 and continuously captures preparatory images according to the conditions.

[0122] In step S702, the image processing unit 124 of the camera 101 calculates the motion vector of the subject from the time-series consecutive preparatory images captured in step S701.

[0123] In step S1801, the image processing unit 124 of the camera 101 detects the subject from the time-series sequence of preparatory images captured in step S701. Methods for detecting the subject include general subject detection techniques. Examples include face / facial organ detection and head detection. Face / facial organ detection and head detection are methods that detect the region containing a person's face, organs, and head from the captured image using pattern recognition or machine learning-based techniques.

[0124] In step S1802, the image processing unit 124 determines whether the subject detected in step S1801 is one person or multiple people. If it is determined to be one person, the process proceeds to step S1804; if it is determined to be multiple people, the process proceeds to step S1803.

[0125] In step S1803, since multiple subjects were detected in step S1802, the image processing unit 124 of the camera 101 detects the main subject from among the people wearing the wearable device 102. A general method for detecting the main subject may be used. For example, the main subject may be detected as the person who occupies the largest area in the field of view of the preparatory image, or the person closest to the center of the preparatory image. Alternatively, the system may be configured to detect as the main subject a person that the user has previously registered as the main subject in the ROM 113 of the camera 101.

[0126] In step S1804, the control unit 112 of the camera 101 determines whether the person detected as the main subject in step S1803 is wearing one or multiple wearable devices 102. If it is determined to be one, the process proceeds to step S1806; if it is determined to be multiple, the process proceeds to step S1805. As a method for determining the number of wearable devices 102, the user may pre-register the wearable devices 102 worn by each subject in the ROM 113 of the camera 101. Alternatively, as explained in step S503 of Figure 5 in Embodiment 1, the number of wearable devices 102 may be determined by the image processing unit 124 of the camera 101 identifying which part of each subject the wearable device 102 is attached to. Furthermore, the number can also be determined by communication between the communication unit 115 of the camera 101 and the communication module 131 of the wearable device 102.

[0127] In step S1805, since it was determined in step S1804 that there are multiple wearable devices 102 worn by the person detected as the main subject through steps S1802 and S1803, the main wearable device 102 is detected among the multiple wearable devices 102. The method for detecting the main wearable device 102 will be explained in detail using the flowchart in Figure 19.

[0128] In step S1901, the control unit 112 of the camera 101 determines whether the user has previously set a priority for the wearable device 102. If it is determined that a priority has been set, the process proceeds to step S1904; otherwise, the process proceeds to step S1902.

[0129] As an example of a user setting priority in advance, Figure 20 shows an example of a user-configurable priority setting screen for wearable devices 102 on the display unit 127 of the camera 101. The display unit 127 may be configured not only as a display unit but also as a touch panel that allows the user to input setting values. In Figure 20, 2000 is an example of a setting screen for setting the priority of wearable devices 102. 2001 is the subject ID, which represents subjects A to F in the image diagram of Figure 17. 2002 is the wearable device 102 ID, which represents wearable devices 102 1701 to 1708 in the image diagram of Figure 17. 2003 indicates the priority of the wearable devices 102. In this example setting screen, the user has set the priority of wearable device 102 1704 of subject B to be the highest.

[0130] In step S1902, the control unit 112 of the camera 101 detects a wearable device 102 that is located within a predetermined area of ​​the prepared image for a certain period of time. For example, if the field of view of the prepared image is defined as 100% as the predetermined area, the control unit 112 detects a wearable device 102 that is located within a 90% rectangular area centered on the center coordinates of the prepared image for a time or longer determined by an arbitrarily set threshold.

[0131] In step S1903, the control unit 112 of the camera 101 calculates the priority of the wearable device 102 detected in step S1902 using a priority calculation unit (not shown). The priority is calculated according to the user's pre-set priorities. An example of the setting screen is shown in Figure 21. 2100 is the setting screen for setting the priority items for the wearable device 102. 2101 is the content of the priority items, such as "order by acceleration speed," "setting of attachment location (head, torso, hands, feet)," "area of ​​face within the field of view," "area of ​​attachment location within the field of view," "order by distance between the center of the field of view and the wearable device 102," "order by distance between the center of the face and the wearable device 102," and "face detection reliability." "Order by acceleration speed" is an item to set when you want to give higher priority to wearable devices 102 with faster acceleration speeds. "Wearing location" is an item that sets which part of the body (head, torso, hands, or feet) the wearable device 102 worn on will have higher priority. "Face area within the field of view" is an item that sets when you want to prioritize wearable devices 102 worn by people with a large face area ratio within the field of view. "Wearing location area within the field of view" is an item that sets when you want to prioritize wearable devices 102 worn by people with a large head, torso, hands, or feet area ratio within the field of view. "Distance order between field of view center and wearable device 102" is an item that sets when you want to prioritize wearable devices 102 with a short distance between the center coordinates of the field of view and the center coordinates of each worn wearable device 102. "Distance order between face center and wearable device 102" is an item that sets when you want to prioritize wearable devices 102 with a short distance between the face center coordinates and the center coordinates of each worn wearable device 102. "Face detection reliability" is an item that can be set if you want to give higher priority to wearable devices 102 worn by people with high face detection reliability. 2102 is the item used to determine the priority of the main wearable devices 102, and the items selected by the user are checked against the contents of 2101.

[0132] The method for calculating priority will be explained using Figure 22(a). For example, if priority determination items are set as explained in Figure 21, a score is calculated for each set item, and then the total value is calculated by adding up the scores. Wearable devices 102 are prioritized in order from the one with the highest total score, and the priority for each wearable device 102 is determined in the same way as in Figure 20 explained in step S1901.

[0133] The method for calculating the total score will be explained using Figure 22(a).

[0134] In Figure 22(a), 2201 is the subject ID, representing subjects A to F in the image diagram of Figure 17. 2202 is the wearable device 102 ID, representing wearable devices 102 1701 to 1708 in the image diagram of Figure 17. 2203 is the score value in order of "acceleration speed," with higher score values ​​for wearable devices 102 with higher acceleration rankings. 2204 is the "mount of attachment," and since the hand is selected in Figure 21, wearable devices 102 attached to the hand will have higher score values. 2205 is the "distance from the center of the field of view to the wearable device 102," with higher score values ​​for wearable devices 102 that are closer to the center of the field of view. 2206 is the "face detection confidence ranking," with higher score values ​​for each subject with higher face detection confidence. 2207 is the total score value, which is the sum of the score values ​​calculated in 2203 to 2206. For example, consider the case where the main subject is detected as subject B in step S1803. Subject B is wearing three wearable devices 102, 1702, 1703, and 1704. If "attachment location" is set to the hand as a priority setting item, the total score value of wearable device 102 1703 will be the largest, and therefore the highest priority. Figure 22(b) shows the priority result when prioritizing based on the calculated total score value. In Figure 22(b), 2208 is the subject ID, which represents subjects A to F in the image diagram of Figure 17. 2209 is the wearable device 102 ID, which represents wearable devices 102 1701 to 1708 in the image diagram of Figure 17. 2210 indicates the priority of the wearable devices 102.

[0135] If subject B were to wear wearable device 102 on both their right and left hands, the "attachment location" score would be the same at 100 for both hands. Therefore, the hand with the higher score in the other categories would have a higher priority.

[0136] In step S1904, the control unit 112 of the camera 101 provisionally determines the wearable device 102 with the highest priority as the main wearable device 102 (main sensor device), according to the priority calculated in step S1901 or step S1903.

[0137] In step S1905, the control unit 112 of the camera 101 determines whether the wearable device 102, which was provisionally determined in step S1904, is located within a predetermined area in the preparation image for a certain period of time. For example, if the field of view of the preparation image is defined as 100% as the predetermined area, if the device is located within a 90% rectangular area centered on the center coordinates of the preparation image for a time greater than an arbitrarily set threshold, the process proceeds to step S1907. If it is not located within that area, the process proceeds to step S1906.

[0138] In step S1906, the control unit 112 of the camera 101 provisionally determines the wearable device 102 with the next highest priority after the wearable device 102 provisionally determined in step S1904 as the primary wearable device 102. The process then proceeds to step S1905 and repeats this determination until step S1907, in which the primary wearable device 102 is determined. If, after repeating this determination, it is determined that none of the wearable devices 102 with the priority determined in step S1901 or step S1903 are located within the predetermined area of ​​the prepared image for a certain period of time, the camera 101 determines the wearable device 102 that was pre-set as the default as the primary wearable device 102 in the next step S1907.

[0139] In step S1907, the control unit 112 of the camera 101 detects the wearable device 102 that was tentatively determined in step S1904 or step S1906 as the primary wearable device 102.

[0140] Next, in step S1806 of Figure 18, the control unit 112 of the camera 101 determines the primary wearable device 102 based on the detection results of the flow in Figure 19, in order to acquire information about the wearable device 102 in the next step.

[0141] In step S703, similar to step S703 of Example 1, the camera 101 calculates a vector corresponding to the main part of the subject using the subject motion information obtained from the main wearable device 102 determined in step S1806 and the subject motion vector obtained by the camera 101.

[0142] In step S704, similar to step S704 of Embodiment 1, under the direction of the control unit 101, the camera 100's blur amount estimation unit (not shown) estimates the amount of motion blur generated in the subject at the shutter speed set by the user, based on the motion vectors of the main parts of the subject calculated in the above step.

[0143] In step S705, similar to step S705 of Example 1, camera 101 compares the amount of subject motion blur calculated in step S704 with the allowable motion amount, and changes the shutter speed of the next preparatory image to be taken so that the amount of subject blur is less than or equal to the allowable motion amount, thereby changing the shooting conditions of the preparatory image.

[0144] The processing of the imaging system 100 in Example 3 has been described above. Specifically, even in scenes with multiple subjects, the system determines the main wearable device 102 from among the multiple wearable devices 102 being worn, senses subject movement information through the main wearable device 102, updates the subject's movement vector using the subject movement information acquired by the wearable device 102 as auxiliary information, determines the shooting conditions for the camera 101, and acquires an image with reduced subject blur.

[0145] The criteria for prioritizing wearable device 102 are described as "speed of acceleration," "setting of attachment location (head, torso, hands, feet)," "area of ​​face within the field of view," "area of ​​attachment location within the field of view," "distance from the center of the field of view to wearable device 102," "distance from the center of the face to wearable device 102," and "face detection confidence," but these are not the only criteria. For example, a configuration could be implemented that calculates a higher score and prioritizes devices where the "distance between the camera and the subject" or the "distance between the camera and wearable device 102" is closer.

[0146] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0147] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of symbols]

[0148] 100 Imaging Systems 101 Camera 121 Optical system 122 Image sensor 123 A / D Conversion Unit 124 Image Processing Unit 125 Records Section 126 Pulse generation unit 127 Display section 118 Mecha Shutter 119 Gyro sensor 111 Vertical drive modulation section 112 Control Unit 113 ROM 114 RAM 115 Communications Department 102 Wearable Devices 132 Power Modules 131 Communication Module 133 Connector 134 Front Camera 135 Rear Camera 136 Light 137 System Memory 138 Control Unit 139 speakers 141 Touchscreen Display 142 LCD displays 143 Touchscreen 144 switches 145 Illuminance Sensor 147 Gyro Sensor 148 Depth Sensor 151 ROM 152 RAM 146 Accelerometer

Claims

1. An imaging means for capturing images of a subject, A subject motion detection means detects the movement of the subject using the image captured by the imaging means and calculates the motion vector of the subject, A calculation means for calculating the reliability of the accuracy of the motion vector of the subject, A receiving means that receives sensing results acquired by an external sensor device and transmitted from the sensor device, An imaging device comprising: exposure control means for controlling the exposure setting of the imaging means; A sensor means for acquiring information about the subject, The system includes a transmission means for transmitting the sensing result of the sensor means to the imaging device. An imaging system comprising a sensor device attached to the subject to be photographed, which is different from the aforementioned imaging device, The imaging device controls the exposure of the imaging means using the sensing results transmitted by the transmitting means and received by the receiving means, and the results of the subject motion detection means. The imaging system is characterized in that the exposure control means controls the exposure of the imaging means using a motion vector whose reliability is equal to or greater than a threshold.

2. The imaging system according to claim 1, characterized in that the sensor means outputs information regarding the movement of the attached subject as the sensing result.

3. The imaging system according to claim 1 or 2, characterized in that the sensor means senses information regarding the movement of the subject to which the sensor device is attached, based on any of the amount of movement of the subject at the part to which the sensor device is attached, the amount of change in the movement of the subject, or the change in the position of the subject.

4. The imaging system according to any one of claims 1 to 3, characterized in that the sensor means further acquires part information to which the sensor device is attached.

5. The imaging device further comprises correction means for correcting the motion vector based on the sensing result of the sensor means, The imaging system according to any one of claims 1 to 4, characterized in that the exposure control means controls the exposure of the imaging means using the motion vector corrected by the correction means.

6. The imaging device includes a blur estimation means that estimates the amount of blur in the captured image based on the calculation of the motion vector of the subject motion detection means and the exposure time set by the exposure control means. The imaging system according to any one of claims 1 to 5, characterized in that the exposure control means controls the exposure of the imaging means based on the result of the blur amount estimation means.

7. An imaging means for capturing an image of a subject, A subject motion detection means detects the movement of the subject using the image captured by the imaging means and calculates the motion vector of the subject, A receiving means that receives sensing results acquired by an external sensor device and transmitted from the sensor device, Correction means for correcting the motion vector based on the sensing results, An imaging device comprising: exposure control means for controlling the exposure setting of the imaging means; A sensor means for acquiring information about the subject, The system includes a transmission means for transmitting the sensing result of the sensor means to the imaging device. An imaging system comprising a sensor device attached to the subject to be photographed, which is different from the aforementioned imaging device, The imaging system is characterized in that the exposure control means controls the exposure of the imaging means using the motion vector corrected by the correction means.

8. An imaging means for imaging a subject, A subject motion detection means detects the movement of the subject using the image captured by the imaging means and calculates the motion vector of the subject, A receiving means that receives sensing results acquired by an external sensor device and transmitted from the sensor device, An exposure control means for controlling the exposure setting of the imaging means, An imaging device comprising: a motion vector calculation by the subject motion detection means and a blur estimation means that estimates the amount of blur in the captured image based on the exposure time set by the exposure control means; A sensor means for acquiring information about the subject, The system includes a transmission means for transmitting the sensing result of the sensor means to the imaging device. An imaging system comprising a sensor device attached to the subject to be photographed, which is different from the aforementioned imaging device, The imaging system is characterized in that the exposure control means controls the exposure of the imaging means based on the result of the blur amount estimation means.

9. The imaging system according to any one of claims 1 to 8, characterized in that the update of the output of the sensor means is faster than the update of the output of the subject motion detection means.

10. The imaging system according to any one of claims 1 to 9, characterized in that the exposure control means controls the amount of charge accumulated after exposure has been started by the imaging means.

11. The imaging device further includes a determination means for determining which sensor device to acquire the sensing result from by acquiring information regarding the priority of multiple sensor devices when there are multiple sensor devices. The imaging system according to any one of claims 1 to 10, characterized in that the exposure control means controls the exposure of the imaging device using the sensing result of the determined sensor device and the result of the subject motion detection means.

12. The determination means calculates a priority for each of the plurality of sensor devices, The imaging system according to claim 11, characterized in that the priority is calculated to be higher the greater the acceleration of the sensor device, the closer the distance from the center of the image, the closer the distance from the camera, the larger the area of ​​the subject to which the sensor device is attached, and the larger the area of ​​the part to which the sensor device is attached.

13. A method for controlling an imaging system comprising an imaging device and a sensor device, The imaging step involves capturing an image of the subject, A subject motion detection step which detects the movement of the subject using the image captured in the above imaging step and calculates the motion vector of the subject, A calculation step for calculating the reliability of the accuracy of the motion vector of the subject, A receiving step in which an external sensor device acquires and transmits sensing results from the sensor device, and Control step of the imaging device, comprising: exposure control step for controlling the exposure setting of the imaging means provided in the imaging device; A sensing step to acquire information about the subject, The imaging device, which is different from the imaging device, has a transmission step of transmitting the sensing results of the sensor device to the imaging device, and a control step of the sensor device that is attached to the subject to be photographed, In the exposure control step, exposure control of the imaging means is performed using the sensing result of the sensing step and the result of the subject motion detection step. A control method for an imaging system, characterized in that the exposure control step involves using a motion vector whose reliability, calculated in the calculation step, is equal to or greater than a threshold, to perform exposure control of the imaging means.

14. A method for controlling an imaging system comprising an imaging device and a sensor device, The imaging step involves capturing an image of the subject, A subject motion detection step which detects the movement of the subject using the image captured in the above imaging step and calculates the motion vector of the subject, A receiving step in which an external sensor device acquires and transmits sensing results from the sensor device, A correction step of correcting the motion vector based on the sensing results, Control step of the imaging device, comprising: exposure control step for controlling the exposure setting of the imaging means provided in the imaging device; A sensing step to acquire information about the subject, The imaging device, which is different from the imaging device, has a transmission step of transmitting the sensing results of the sensor device to the imaging device, and a control step of the sensor device that is attached to the subject to be photographed, A control method for an imaging system, characterized in that, in the exposure control step, the exposure of the imaging means is controlled using the motion vector corrected in the correction step.

15. A control method for an imaging system comprising an imaging device and a sensor device, The imaging step involves capturing an image of the subject, A subject motion detection step which detects the movement of the subject using the image captured in the above imaging step and calculates the motion vector of the subject, A receiving step in which an external sensor device acquires and transmits sensing results from the sensor device, An exposure control step for controlling the exposure setting of the imaging means provided in the imaging device, Control step of the imaging device comprising: calculating the motion vector of the subject motion detection means and a blur amount estimation step that estimates the amount of blur in the captured image based on the exposure time set in the imaging means; A sensing step to acquire information about the subject, The imaging device, which is different from the imaging device, has a transmission step of transmitting the sensing results of the sensor device to the imaging device, and a control step of the sensor device that is attached to the subject to be photographed, A control method for an imaging system, characterized in that, in the exposure control step, exposure control of the imaging means is performed based on the result of the blur amount estimation step.

16. A program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 12.

Images