Imaging system, device and controlling method

Abstract

To provide an imaging system configured to automatically determine settings for imaging of a camera according to a motion state of a subject by linking the motion state of the subject to an analysis result of the camera and a sensing result of a portable device.SOLUTION: An imaging system includes an image capturing device and a sensor device worn on a subject to be imaged. The image capturing device includes imaging means which images the subject, subject motion detection means which detects motion of the subject using the image captured by the imaging means, receiving means which receives a sensing result transmitted from an external device and exposure control means which controls exposure of the imaging means. The sensor device includes sensor means which acquires information on the subject, and transmission means which transmits a result sensed by the sensor means to the image capturing device. The image capturing device is configured to receive a result sensed by the sensor means and control exposure of the imaging means using the sensing result of the sensor means and a result of detecting the motion of the subject.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for controlling photography by a camera equipped with an imaging means, and in particular to a photography system, apparatus, and control method using a wearable device or the like. [Background technology]

[0002] 2. Description of the Related Art Conventionally, surveillance cameras that are equipped with an acceleration sensor and a human presence sensor and automatically capture images based on the sensing results of the sensors have been known.

[0003] For example, Patent Document 1 discloses controlling automatic photography of a remote camera according to the status of a portable device (wearable device). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-072673 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional technology disclosed in the above-mentioned patent document mainly controls the timing of automatic shooting based on the sensing results, and does not consider automatically setting the shooting settings of the camera, including the movement state of the subject. Therefore, even if the timing of shooting can be controlled, if the subject moves quickly, the subject may be photographed blurred. Therefore, the object of the present invention is to provide a system that enables automatic determination of the shooting settings of the camera according to the movement state of the subject by linking the analysis results of the camera and the sensing results of a portable device. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the photographing system of the present invention comprises an imaging device having an imaging means for photographing a subject, a subject motion detection means for detecting the motion of the subject using an image captured by the imaging means, a receiving means for receiving sensing results transmitted from an external device and an exposure control means for controlling the exposure of the imaging means, a sensor device attached to a subject to be photographed, and having a sensor means for acquiring information about the subject, and a transmitting means for transmitting the sensing results of the sensor means to the imaging device, wherein the imaging device receives the sensing results of the sensor means and controls the exposure of the imaging means using the sensing results of the sensor means and the detection result of the motion of the subject. Effect of the Invention

[0007] According to the present invention, it is possible to provide an image capturing system capable of performing camera control that can reduce subject blurring by more appropriately taking into account the movement of the subject. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 shows a configuration example of the present invention. [Diagram 2] FIG. 1 is a diagram showing an example of the external appearance of an imaging system 100. [Diagram 3] 1 is a flowchart for explaining the operation of the camera 101 according to the first embodiment. [Figure 4] 11 is a flowchart for explaining the operation of the camera 101 according to the second embodiment. [Diagram 5] 1 is a flowchart illustrating the operation of the wearable device 102. [Figure 6] Diagram explaining block matching [Figure 7] A flowchart for explaining an operation for determining the photographing conditions of a preparation photographed image. [Figure 8] Diagram showing the movement of the subject [Figure 9] Flowchart showing motion vector calculation processing [Figure 10] Diagram of correcting the subject's motion vector [Figure 11]A diagram showing the relationship between the subject's motion vector and the amount of subject motion blur. [Figure 12] FIG. 1 is a diagram for explaining a process of correcting the amount of exposure by multiplying a digital gain; [Figure 13] 11 is a flowchart for explaining details of the main exposure process in the second embodiment. [Figure 14] A diagram explaining the configuration of an electronic front curtain shutter. [Figure 15] A diagram explaining the operation of the electronic front curtain shutter. [Figure 16] FIG. 1 is a diagram for explaining a method for performing exposure termination control. [Figure 17] Illustration of multiple subjects [Figure 18] Flowchart for explaining a case where there are multiple wearable devices 102 [Figure 19] Flowchart for determining a primary wearable device 102 [Figure 20] An example of a setting screen for setting the priority of the wearable device 102 [Figure 21] FIG. 1 is a diagram for explaining items for determining priority of a wearable device 102. [Figure 22] FIG. 1 is a diagram for explaining a method for calculating the priority ranking of a wearable device 102. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. This embodiment is a photography system that links a camera with a wearable device such as a smartphone or a wrist-type terminal, and automatically realizes exposure control operation of the camera.

[0010] This enables exposure control to be performed to keep subject movement below a desired level of blur, even in use cases such as unmanned photography or self-portraits (selfies).

[0011] 1 described in this embodiment is realized by two devices, a camera 101 and a wearable device 102. Examples of the wearable device 102 include a smartphone and a wrist-type terminal. FIG. 2 is a diagram showing the appearance of an embodiment of the present invention. FIG. 2(a) shows the imaging system 100 to which two devices, a single-lens reflex camera as the camera 101 and a smartphone as the wearable device 102, are connected.

[0012] The control unit 112 is, for example, a CPU, and reads out a control program for each block included in the camera 101 from a ROM 113 described later, and deploys and executes the program in a RAM 114 described later. In this way, the control unit 112 controls the entire camera 101, and controls the operation of each block included in the camera 101.

[0013] The ROM 113 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block of the camera 101 as well as parameters and the like required for the operation of each block.

[0014] The RAM 114 is a rewritable volatile memory, and is used for developing programs executed by the control unit 112 and the like, and for temporarily storing data generated by the operation of each block of the camera 101, and the like.

[0015] The communication unit 115 performs communication according to a predetermined wireless communication standard. For example, the wireless communication standard may be the so-called Wi-Fi of the IEEE802.11 standard, Bluetooth (registered trademark), and NFC, and it is sufficient that at least one of them is supported.

[0016] The optical system 121 is composed of a group of lenses including optical elements such as a zoom lens and a focus lens, and forms an incident subject image on an imaging surface of an image sensor 122, which will be described later.

[0017] The image sensor 122 is composed of, for example, a CCD or CMOS sensor, etc. During exposure, each pixel of the image sensor 122 photoelectrically converts an optical image formed on the imaging surface of the image sensor 122 by the optical system 121, and outputs the obtained analog image signal to an A / D converter 123 described later.

[0018] The A / D conversion unit 123 converts the input analog image signal into digital image data, and the digital image data output from the A / D conversion unit 123 is temporarily stored in the RAM 114 .

[0019] The image processing unit 124 applies various image processing such as white balance adjustment, color interpolation, gamma processing, etc. to the image data stored in the RAM 114 to generate an image. In addition, a predetermined calculation process is performed using the captured image signal, and based on the obtained calculation result, imaging control such as exposure control by a pulse generating unit 126 and a vertical drive modulation unit 111 described later and distance measurement control is performed. It is also possible to perform processing such as subject recognition on the image data using an existing method. It is also possible to include a motion vector calculation unit (not shown) and obtain a motion vector between images using multiple image data.

[0020] The recording unit 125 is a removable memory card or the like, and records the image data processed by the image processing unit 124 as a recorded image via the RAM 114.

[0021] When transitioning from a non-imaging state to an imaging state, the pulse generating unit 126 supplies a scanning clock (horizontal drive pulse) and a predetermined control pulse to the image sensor 122. Of the scanning clocks generated by the pulse generating unit 126, a clock for vertical scanning is input to a vertical drive modulation unit 111, which will be described later.

[0022] The vertical drive modulation section 111 modulates the clock for vertical scanning, among the scanning clock signals generated by the pulse generation section 126, to a predetermined clock frequency and inputs it to the image sensor 122. The vertical drive modulation section 111 determines a scanning pattern of the reset scanning performed for each line of the image sensor 122, which is made up of a plurality of pixels. The reset scanning for each line of the image sensor 122 realizes a function as an electronic front curtain shutter.

[0023] The gyro sensor 119 is a motion detection sensor that detects angular velocity, and determines the magnitude of shaking of the camera 101 .

[0024] Mechanical shutter 118 is an open / close type shutter mechanism that realizes a light blocking mechanism that physically blocks light incident on image sensor 122. In this embodiment, mechanical shutter 118 functions as a rear curtain (hereinafter, mechanical rear curtain) made up of multiple light blocking blades. Control unit 112 can control the exposure time (shutter speed) by adjusting the timing at which the mechanical rear curtain starts to travel. On the other hand, the function of the electronic front curtain is realized by sequentially resetting and scanning the pixels of image sensor 122 line by line at a predetermined timing.

[0025] The display unit 127 is a display device such as an LCD, and displays images stored in the RAM 114 and images recorded in the recording unit 125, displays an operation user interface for receiving user instructions, etc. The display unit 127 also displays (live view display) an image captured by the image sensor 122 for composition adjustment during preparation shooting, etc.

[0026] The configuration of the camera 101 has been described above.

[0027] Next, a description will be given of the wearable device 102. The wearable device 102 includes a touch screen display 141, and characters, images, icons, etc. are displayed on a liquid crystal display 142. A touch screen 143 is capable of detecting gesture operations.

[0028] The in-camera 134 includes a lens and an imaging element such as a CCD or CMOS that converts an optical image into an electrical signal. The in-camera 134 is a small camera module that includes AF (autofocus), aperture, and shutter speed adjustment functions. The in-camera 134 captures an image of an object facing the touch screen display 141.

[0029] The illuminance sensor 145 detects the illuminance in the shooting environment.

[0030] Control unit 138 is, for example, a CPU, and reads out a control program for each block of wearable device 102 from ROM 151 (described later), and deploys and executes the control program in RAM 152 (described later). In this way, control unit 138 controls the operation of each block of wearable device 102. Control unit 138 provides a camera function by controlling touch screen 143, switch 144, in-camera 134, illuminance sensor 145, out-camera 135, light 136, and the like.

[0031] ROM 151 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block of wearable device 102 as well as parameters and the like required for the operation of each block.

[0032] The RAM 152 is a rewritable volatile memory, and is used for developing programs executed by the control unit 112 and the like, and for temporarily storing data generated by the operation of each block of the wearable device 102, and the like.

[0033] When a switch 144 is set to ON for sound output, the speaker 139 outputs a shutter sound during imaging and a warning sound.

[0034] The connector 133 is used to connect the wearable device 102 to an external device. For example, an AC adapter for charging a battery provided in the power supply module 132, which will be described later, is connected to the connector 133. The connector 133 is also used when inputting and outputting image data and audio data to and from a non-volatile memory connected from the outside. The connector 133 may be a dedicated terminal such as a Dock connector, or a general-purpose terminal such as a Universal Serial Bus (USB). A communication cable with the outside is also connected to the connector 133, and by connecting to, for example, the camera 101 or the like via the communication cable, it is also possible to communicate information from a group of sensors, which will be described later.

[0035] The outer camera 135 is a small camera module similar to the inner camera 134. The outer camera 135 captures an image of a subject on the opposite side to the inner camera 134. The light 136 is a light emitting module, and functions as a flash when the outer camera 135 captures an image.

[0036] The communication module 131 performs communication according to a predetermined wireless communication standard. For example, wireless communication standards include so-called Wi-Fi of the IEEE802.11 standard, Bluetooth (registered trademark), and NFC, and the communication module 131 only needs to support at least one of them. Specific communications are used for inputting and outputting image data obtained by imaging, downloading a program module for adding functions to an image processing device, and transmitting information from a group of sensors (illuminance sensor 145, acceleration sensor 146, gyro sensor 147, and depth sensor 148) to the camera 101, which will be described later.

[0037] The power supply module 132 includes a rechargeable battery and supplies power to the entire image processing device. The battery included in the power supply module 132 may be, for example, a lithium ion battery or a nickel metal hydride battery.

[0038] The acceleration sensor 146 detects the direction and magnitude of acceleration acting on the image processing device. The acceleration sensor 146 is capable of detection in three axes, the XYZ directions.

[0039] The gyro sensor 147 detects the angle and angular velocity of the image processing device.

[0040] The depth sensor 148 measures the distance from the camera to the subject being photographed. Methods for measuring distance include a method for measuring the time it takes for infrared rays, light, ultrasonic waves, etc. to be reflected from an object and bounce back, and a method for arranging multiple cameras and pixels in parallel and acquiring depth information of the subject from a parallax image.

[0041] FIG. 2 shows an example of the appearance of the imaging system 100. (a) to (c) of FIG. 2 each show a combination of a camera 101 and a wearable device 102 that cooperate as the imaging system 100. The camera 101 and the wearable device 102 are connected wirelessly, for example, by Bluetooth or the like, and communicate with each other to cooperate with the imaging system 100. Reference numeral 201 denotes a mirrorless single-lens camera or a single-lens reflex camera as an example of the camera 101. Reference numeral 202 denotes a smartphone as an example of the wearable device 102, and this smartphone is often equipped with a camera function. Therefore, by using the camera function, it is possible to use the smartphone 204 in FIG. 2(c) as the camera 101 side, rather than the wearable device 102 side, as in the case of the in-camera 134 and the out-camera 135 mounted on the smartphone 204. In addition, the wearable device 102 can be in the form of a wrist-type terminal 203 or the like other than a smartphone. Note that there are various types of cameras 101 as examples of the camera 101, as long as they have an imaging function and a communication function.

[0042] The external appearance and system configuration of the imaging system 100 have been described above.

[0043] (First embodiment) Hereinafter, the processing of the imaging system 100 according to the first embodiment of the present invention will be described with reference to the flowcharts of Fig. 5 and Fig. 3. In the first embodiment of the present invention, a subject to be photographed wears the wearable device 102, and the camera 101 is configured as a remote camera disposed externally. Then, subject movement information is sensed through the wearable device 102, and the photographing conditions of the camera 101 are determined using the subject movement information acquired by the wearable device 102 as auxiliary information. Note that the following processing is realized by the camera 101 as the imaging system 100, in which the control unit 112 controls each part of the device according to a program stored in the ROM 113. Also, the wearable device 102 is realized by the control unit 138 controlling each part of the device according to a program recorded in the ROM 151.

[0044] First, the operation of the wearable device 102 will be described with reference to FIG.

[0045] In step S501, first, the wearable device 102 is powered on. When the power is turned on, the sensors (acceleration sensor 146, gyro sensor 147, depth sensor 148, etc.) start to operate and start to obtain sensing signals in order to detect subject motion information of the wearable device 102. The wearable device 102 performs a standby operation to receive sensing signals from each sensor.

[0046] In step S502, the sensing signals acquired by each sensor in step S501 are acquired at regular time intervals. For example, acceleration information of the wearable device 102 is acquired by periodically acquiring the output from the acceleration sensor 146 at predetermined times. In this manner, it is possible to acquire acceleration information of the subject at the part where the wearable device 102 is worn. It is also possible to indirectly acquire acceleration information of the subject at the part where the wearable device 102 is worn by using a sensor capable of detecting the motion state of another subject, instead of the acceleration sensor 146. As an example, the motion speed and acceleration information of the subject per unit time can be calculated by acquiring the change in distance from the camera 101 to the wearable device 102 by the depth sensor 148. In other words, the wearable device 102 serves as a sensor that acquires various information of the subject.

[0047] In step S503, the camera 101 identifies which part of the subject to be photographed is wearing the wearable device 102. For example, a method of identifying the wearable device 102 is to set the part of the subject to which the wearable device 102 is to be worn in advance. In addition, a technique is known in which the acceleration and speed magnitude of a specific part within a predetermined time are recorded in advance, and the part of the subject that is moving is identified based on the actual movement (Patent No. 6325581). For example, in step S502, acceleration information is acquired from the acceleration sensor 146 of the wearable device 102. Thus, the part of the subject to which the wearable device 102 is worn can be identified by comparing the acceleration change recorded in advance per predetermined time for each part to the acquired acceleration information. Note that the part of the subject to which the wearable device 102 is worn can also be identified from the captured image by the image processing unit 124 in the camera 101, for example, and it is not necessary to configure the wearable device 102 to identify the part of the subject to be worn.

[0048] In step S504, the acceleration information acquired in steps S502 and S503 and information about the part of the body where wearable device 102 is attached are transmitted to camera 101 as "subject movement information."

[0049] The above is the processing of the wearable device 102. Next, the operation of the camera 101 will be described in detail with reference to the flowchart of FIG.

[0050] In step S301, the user turns on the power of the camera 101. Then, the camera 101 receives the subject movement information transmitted from the wearable device 102 via the communication unit 115 etc.

[0051] In step S302, the user starts preparatory shooting such as adjusting the composition with the camera 101. During this preparatory shooting period, the camera 101 continuously captures images and displays them on the display unit 127 (live view display). The user adjusts the composition while viewing the displayed images (preparatory shot images) during preparatory shooting. Note that the processes of steps S304, S305, and S303 described below are performed during the preparatory shooting period, and will be described as being performed before shooting for recording (main shooting).

[0052] In step S303, control unit 112 determines the shooting conditions (shooting parameters) of the preparatory shooting image to be captured in order to detect the motion vector of the subject within the composition. Using the amount of subject movement within the composition when preparatory shooting is performed under the initial shooting conditions and the subject movement information transmitted from wearable device 102, a shutter speed is set that reduces subject blurring of the part where wearable device 102 is worn (part of interest of the subject). Details will be described later.

[0053] In step S304, the subject within the composition and the shooting conditions (shutter speed, ISO sensitivity, F-number, etc.) are displayed on the display unit 127.

[0054] In step S305, the control unit 112 judges whether a remote release has been activated. Here, the term "remote release" refers to a shooting (main shooting) instruction by a cable release or a radio-controlled operating device. For example, in the case of a radio-controlled operating device, a shooting instruction signal is transmitted wirelessly based on the user's operation. The transmitted signal is acquired by, for example, the communication unit 115, and the control unit 112 judges the acquired signal to control each unit. Alternatively, the camera 101 judges whether the photographer (user) directly pressed the shutter button. In that case, the user presses the shutter button in accordance with the shutter timing while looking at the subject displayed on the display unit 127. If the remote release has been activated or the shutter button has been pressed, the process proceeds to the main exposure process in step S306. On the other hand, if it is not the shutter timing, it is possible to redo the shooting settings by returning to step S301.

[0055] In step S306, the camera 101 performs exposure processing using the shooting settings made in the processing of the above steps, and records the captured image in the ROM 113.

[0056] In this way, during the preparatory photographing, the user repeatedly sets the exposure time for the actual photographing until the desired motion blur is achieved, and when the photo opportunity arises, the user issues a command for a remote release or presses the shutter button.

[0057] Next, the process of step S303, which is a feature of the present invention, will be described with reference to the flowchart of FIG.

[0058] In step S701, the camera 101 sets initial shooting conditions and captures images continuously. The initial shooting conditions here mainly refer to a frame rate and a shutter speed. The camera 101 sets the highest frame rate and the highest shutter speed within a range that does not affect the process of calculating an evaluation value used to control an auto function such as AE (auto exposure) or AF (auto focus) control performed in a general camera. Even when the shutter speed is set to a high speed, the optical system 121 is controlled to control the lens aperture and the ISO sensitivity setting of the camera 101 in accordance with the shutter speed so that shooting can be performed under appropriate exposure conditions (exposure control). Images that are consecutive in time series are captured under these initial shooting conditions. It is desirable that the captured images are captured under conditions where there is almost no accumulated blur of a moving subject and the amount of movement of the subject between captured consecutive images is as small as possible. Under such shooting conditions, there are disadvantages such as the ISO value being easily increased and image data with a lot of noise being obtained. On the other hand, since the amount of movement of the subject being photographed can be suppressed, it is possible to capture the movement of even a fast subject.

[0059] In step S702, a motion vector calculation unit (not shown) of the image processing unit 124 calculates a motion vector of the subject from the chronologically consecutive preparatory shot images captured in step S701. The process of calculating the motion vector of the subject will be described in detail with reference to Figs. 6, 8, and 9.

[0060] Fig. 8 shows the movement of a subject. Fig. 8(a) is an example of a scene in which a dog 801 running to the left and a dog 802 standing still are being shot. A motion vector represents the amount of horizontal and vertical movement of the subject between the preparatory shot images. An example of this motion vector is shown in Fig. 8(b).

[0061] Fig. 8(b) is a diagram showing an example of the motion vector of the preparation-captured image of Fig. 8(a). In the example of Fig. 8(b), the image processing unit 124 detects the running dog 801 as having a motion vector in the left direction. On the other hand, the motion vector of the other dog, the stationary dog ​​802 and the fence in the background, is detected as having a motion vector of 0, and therefore the motion vector is not shown.

[0062] The motion vector calculation method will be described in detail with reference to Fig. 9 and Fig. 6. Fig. 9 is a flowchart showing the motion vector calculation process. Note that in the present invention, the block matching method is taken as an example of the motion vector calculation method, but the motion vector calculation method is not limited to this example and may be, for example, a gradient method.

[0063] In step S901 of Figure 9, the motion vector calculation unit of the image processing unit 124 receives two preparatory captured images that are adjacent in time, sets the preparatory captured image of the Mth frame as the base frame, and sets the preparatory captured image of the M+1th frame as the reference frame.

[0064] In step S902, the motion vector calculation unit arranges a base block 602 of N×N pixels in a base frame 601 as shown in FIG.

[0065] In step S903, the motion vector calculation unit sets, as a search range 605 for the reference frame 603, the (N+n)×(N+n) pixels surrounding the central coordinates 604 of a base block 602 in a base frame 601 as shown in FIG.

[0066] In step S904, the motion vector calculation unit performs correlation calculation between the base block 602 of the base frame 601 and the reference block 606 of N×N pixels at different coordinates in the search range 605 of the reference frame 603 to calculate a correlation value. The correlation value is calculated based on the inter-frame absolute difference sum for the pixels of the base block 602 and the reference block 606. In other words, the coordinate with the smallest inter-frame absolute difference sum value is the coordinate with the highest correlation value. Note that the method of calculating the correlation value is not limited to the method of calculating the inter-frame absolute difference sum, and may be, for example, a method of calculating a correlation value based on the inter-frame difference square sum or a normal cross-correlation value. In the example of FIG. 6, it is assumed that the reference block 606 shows the highest correlation.

[0067] In step S905, the motion vector calculation unit calculates a motion vector based on the reference block coordinates showing the highest correlation value calculated in step S904. In the example of Fig. 6, the motion vector is calculated based on coordinates 604 corresponding to the center coordinates of the standard block 602 in the standard frame 601 and the center coordinates of the reference block 606 within a search range 605 in the reference frame 603. In other words, the coordinate distance and direction from the coordinates 604 to the center coordinates of the reference block 606 are calculated as the motion vector.

[0068] In step S906, the motion vector calculation unit determines whether or not the motion vectors have been calculated for all pixels in the reference frame 601. If the motion vector calculation unit determines in step S906 that the motion vectors for all pixels have not been calculated, the process returns to step S902. Then, in step S902, the motion vector calculation unit arranges a reference block 602 of N×N pixels in the above-mentioned reference frame 601 with a pixel for which the motion vector has not been calculated as the center, and the processes from step S903 to step S905 are performed in the same manner as described above. That is, the motion vector calculation unit repeats the processes from step S902 to step S905 while moving the reference block 602 in FIG. 6, thereby calculating the motion vectors for all pixels in the reference frame 601. The motion vector may be calculated in units of pixels or in units obtained by dividing an image into a predetermined number of divisions. The above processes are performed between preparatory captured images captured close to each other to calculate the motion vector.

[0069] Next, in step S703, image processing unit 124 calculates a vector corresponding to a main part of the subject using the subject motion information acquired from wearable device 102 and the motion vector of the subject calculated in step S702. Then, image processing unit 124 uses the subject motion information acquired by wearable device 102 to identify a vector corresponding to a main part of the subject for which the user wishes to reduce subject blur, and then performs processing to correct the motion vector of the subject of the corresponding main part. This correction processing will be specifically described with reference to FIG. 10.

[0070] First, a method for identifying a vector corresponding to a main part of a subject will be described with reference to FIG. 10(a). FIG. 10(a) is a diagram showing a group of motion vectors (1011, 1012, 1013, 1014) of subjects that are candidates for a main part. The main part information is information corresponding to the part where the wearable device 102 is attached, among the subject motion information transmitted in step S504. The correspondence between the main part of the subject and the motion vector of the subject is realized by selecting the motion vector of the subject corresponding to the main part from the preparatory captured image for obtaining the motion vector of the subject. A method such as a general subject recognition technology may be used as a method for detecting the main part from the preparatory captured image. For example, consider a case where the part where the wearable device 102 is attached is the head of a dog. In this case, the image processing unit 124 detects the head area of ​​the dog, which is the part where the wearable device 102 is attached, within the range in which the motion vector of the subject is obtained in the preparatory captured image for obtaining the motion vector of the subject. The image processing unit 124 selects a group of motion vectors (1011, 1012, 1013, 1014) of subjects that exist within a predetermined distance from the detected head region of the dog. Then, from among these, the image processing unit 124 detects the motion vector (1011) of the subject with the largest amount of movement, and sets this as the motion vector of the subject of the main part.

[0071] Next, the process of correcting the motion vector of the subject in the relevant main part will be described in detail with reference to (b) and (c) of Fig. 10. Correcting the motion vector of the subject in the main part is a process of correcting the motion vector of the subject using acceleration information of the attachment part transmitted from wearable device 102, which has a fast output update rate, in comparison with the motion vector calculation process, which has a slow output update rate. This makes it possible to artificially improve the update rate of the motion vector of the subject and obtain the motion vector of the subject corresponding to the change in the motion of the subject.

[0072] FIG. 10(b) is a diagram showing the timing of acquiring the subject motion vector and subject motion information.

[0073] The motion vector of the subject is calculated by using two or more frames of preparatory photographed images that are consecutive in time series, and the amount of movement between the preparatory photographed images used for the motion vector of the subject is calculated. For example, the motion vector 1031 of the subject cannot be calculated until the camera 101 acquires at least two frames of preparatory photographed images 1021 and 1022. In addition, the next motion vector 1032 of the subject cannot be calculated until the preparatory photographed image 1023 is acquired. If the motion of the subject suddenly changes during the blank period 1041 from the motion vector 1031 of the subject to the calculation of the motion vector 1032 of the subject, the update rate of the motion vector of the subject is slow, so the motion of the subject at that timing cannot be detected correctly. On the other hand, the acceleration information as the motion information of the subject detected by the wearable device 102 does not depend on the preparatory photographed images, and the motion of the device can be directly detected, so that high-speed detection is generally possible (1051).

[0074] Since preparatory shot images that can be acquired by a digital single-lens reflex camera, which is a typical camera 101, can be captured at a high speed of about 120 fps, the subject's motion vector has an update rate of 120 fps or less. On the other hand, the output update rate of an acceleration sensor 146 mounted on a smartphone, which is a typical wearable device 102, is 100 times or more faster than the subject's motion vector.

[0075] Therefore, by having the image processing unit 124 correct the motion vector of the subject using the acceleration information detected by the wearable device 102, it is possible to obtain a more accurate motion vector of the subject even during periods when the motion vector of the subject is not updated. Furthermore, since the motion vector of the subject depends on the preparatory captured image, the motion vector of the subject may not be obtained from an image of a low-contrast subject or one in which accumulated blur or out-of-focus occurs, and the update rate of the vector may be slower. Therefore, it is effective to correct and update the motion vector of the subject using sensor information from the wearable device 102, which has a fast update rate.

[0076] Next, the correction process of the motion vector of the main part will be described with reference to Fig. 10(c). Fig. 10(c) shows the motion vector 1061 of the subject of the main part, the motion vector 1062 of the subject corrected when the motion of the subject slows down before updating the motion vector of the subject, and the motion vector 1063 of the subject corrected when the motion of the subject speeds up.

[0077] In addition, since the motion vector of the main part relative to the image has an angle and magnitude in multiple directions, it is converted into the vector magnitude using formula 1. Generally, the motion vector used in an image has a direction in two-dimensional coordinates, so by applying formula 1, it can be converted into the vector magnitude as a scalar.

[0078]

number

[0079] The subject's motion vector can be corrected by performing gain processing corresponding to the acceleration change of the main part up to the point when the subject's motion vector is updated. Therefore, if the acceleration change amount calculated using the acceleration information of the main part detected by the wearable device 102 is α (1 if the acceleration does not change), the correction of the subject's motion vector can be expressed as in Equation 2. Corrected subject motion vector = α × subject motion vector Equation 2

[0080] Using the above formula 1, if the acceleration change amount α is smaller than 1, the motion vector of the subject in the main part before correction is corrected as shown by 1062, and conversely, if the acceleration change amount α is greater than 1, the motion vector of the subject in the main part before correction is corrected as shown by 1063. By correcting the motion vector of the subject in the main part in this way, the amount of blur of the main part can be found with as little difference as possible from the real-time movement of the subject.

[0081] Next, in step S704, the image processing unit 124 estimates the amount of motion blur that occurs in the subject at the motion vector of the main part calculated in the processing of the above steps and the shutter speed set by the user (subject blur amount estimation). This subject motion blur amount is calculated by the following formula using the imaging frame rate of the preparatory captured image for calculating the subject motion vector, the shutter speed set by the user to the camera 101, and the motion vector of the subject. Amount of subject motion blur = Subject motion vector * (frame rate (fps) / shutter speed (s)) ··· Equation 3

[0082] The relationship between the motion vector of the subject and the amount of motion blur of the subject in relation to the above formula 3 will be described with reference to FIG. 11. FIG. 11 is a diagram showing the relationship between the motion vector of the subject and the amount of motion blur of the subject. For example, the motion vector of the subject is calculated using frames before and after the preparatory captured image updated at a frame rate of 60 fps, so the update frame rate of the motion vector of the subject 1101 is also 60 fps. On the other hand, the amount of motion blur of the subject corresponds to the shutter speed set by the user since it is the amount of blur caused by the subject moving during exposure. For example, when the user sets the shutter speed to 1 / 120 seconds, the amount of motion blur of the subject 1102 is 1 / 60 seconds between the frames before and after the preparatory captured image for calculating the motion vector of the subject. That is, if the size of the motion vector of the subject 1101 is 10 pixels in terms of the number of pixels, the amount of motion blur of the subject 1102 is half the length, or 5 pixels.

[0083] Next, in step S705, the control unit 112 compares the amount of subject motion blur calculated in step S704 with the allowable amount of motion, changes the shutter speed of the preparatory image to be captured next so that the amount of subject blur is equal to or less than the allowable amount of motion, and changes the shooting conditions of the preparatory image. The allowable amount of motion is the amount of motion blur that is not noticeable when photographed at a predetermined shutter speed. The size of the allowable amount of motion is determined by the size and number of pixels of an image sensor such as a CCD or CMOS sensor, and the resolution of a display to display the image. For example, the allowable amount of motion for a PC display with an image sensor of APS-C, 200,000 pixels, and full HD (1920×1080 pixels) is set to 5 pixels or less. In order for the camera 101 to capture the preparatory image so that the amount of motion blur is equal to or less than the allowable amount of motion, the shutter speed is determined using the following formulas 4 and 5. n = Amount of subject motion blur / Allowable amount of motion Equation 4

[0084] In this case, if n calculated by Equation 4 is greater than 1, it indicates that there is a high possibility of subject blur occurring when shooting at the currently set shutter speed, and if n is 1 or less, it indicates that the shutter speed is unlikely to cause subject blur. Therefore, the appropriate shutter speed that reduces the occurrence of subject blur is calculated using the following Equation 5. Updated shutter speed (s) ≦ Set shutter speed * (1 / n) Formula 5

[0085] To explain this using specific numerical values, the amount of motion blur 1102 in Fig. 11 is 5 pix, and the allowable amount of motion is also 5 pix. Therefore, according to formula 4, n = 1, and it is understood that the currently set shutter speed has little effect on subject blur. Therefore, according to formula 5, the shutter speed as the shooting condition for the preparation shooting image (which will be the shooting condition for actual exposure if not changed) should be set to an exposure time faster than 1 / 120 (in this case, it is left unchanged from 1 / 120s). Also, if there is sufficient light for shooting, the shutter speed may be set even faster than 1 / 250, taking into account the ISO sensitivity and aperture.

[0086] The above describes an example in which the shutter speed is updated as a shooting condition for the preparation-shot images. Note that in order to improve the accuracy of detecting the motion vector of the subject, the frame rate for shooting the preparation-shot images may be increased to increase the update rate for calculating the motion vector of the subject. Therefore, the update rate is increased using Equation 6. Update frame rate (fps) ≧ Set frame rate (fps) * n Equation 6

[0087] The frame rate and shutter speed are important shooting conditions for motion detection. In order to capture an image with appropriate brightness, the aperture value and ISO sensitivity are also changed along with the change in frame rate and shutter speed to control the exposure value so that it does not change.

[0088] As for the detailed processing of step S303, the processing of determining the shooting conditions for the preparation image (if not changed, the actual shooting will also remain the same) using the processing of steps S701 to S705 in FIG. 7 has been described.

[0089] The processing of the imaging system 100 in the first embodiment has been described above. Specifically, subject motion information is sensed through the wearable device 102, and the subject motion information acquired by the wearable device 102 is used as auxiliary information to update the motion vector of the subject and determine the shooting conditions of the camera 101. According to the present invention, in cooperation with the wearable device 102, it is possible to improve the detection accuracy of the subject motion vector calculated by the camera 101 and set a shutter speed that reduces subject blur. This allows the photographer to set a shutter speed so as to suppress the subject motion desired by the photographer to a desired blur or less without touching the camera, and to adjust the exposure for shooting. The present invention makes it possible to expand the use scenes of auto shooting.

[0090] In the first embodiment, a method for calculating the amount of motion blur by the image processing unit 124 has been described, in which the motion vector of the subject is converted to match the shutter speed set by the user. Note that the conversion of the amount of motion blur does not necessarily have to match the set shutter speed. In that case, the same process can be simply realized by comparing the motion vector of the subject with a preset threshold, and changing the shutter speed to a faster shutter speed than the currently set value when the threshold is exceeded.

[0091] In addition, in the first embodiment, a method for identifying a main part of a subject and selecting a motion vector of the subject in the main part has been described. However, it is also possible to select a motion vector of a subject with the fastest movement from among the motion vectors of the subject obtained from the preparation shot images.

[0092] In the first embodiment, a method for identifying a main part of a subject and selecting a motion vector of the subject in the main part has been described. If the camera 101 is equipped with an acceleration sensor similar to the acceleration sensor 146 mounted on the wearable device 102, a motion vector of the subject other than the motion of the acceleration sensor mounted on the camera 101 may be selected.

[0093] In the first embodiment, a method for identifying a main part of a subject and selecting a motion vector of the subject in the main part has been described. Note that, among the calculated motion vectors of the subject, the motion vector of the subject may be selected from a range that is captured in the center of the angle of view when photographing with the camera 101 or a range near the target of autofocus.

[0094] In the first embodiment, a method has been described in which the image processing unit 124 identifies a main part of a subject and selects a motion vector of the subject in the main part. Here, if the wearable device 102 is captured in the preparation captured image, the image processing unit 124 may detect the position of the wearable device 102 from the image and identify the main part based on the detection result.

[0095] Furthermore, before selecting the motion vector of the subject of the main part, the image processing unit 124 may perform a selection process on the motion vector of the subject calculated from the preparation image. For example, in a calculation such as template matching performed in the process of calculating the motion vector of the subject, a correlation value calculation is performed. At that time, a vector with a low correlation value is determined to be a motion vector of the subject with low reliability. By performing a selection process based on the reliability calculation result in this way, it is possible to extract only the motion vector of the subject with higher accuracy.

[0096] Second embodiment Hereinafter, a second embodiment of the present invention will be described in detail with reference to the drawings. In the second embodiment, exposure control is performed based on the amount of motion blur of the subject during the main exposure process, thereby making it possible to obtain an image with reduced motion blur of the subject. The process of the wearable device 102 of the image capture system 100 in the second embodiment is the same as that in the first embodiment, and therefore will not be described. The operation performed by the control unit 112 of the camera 101, which is a feature of the second embodiment, will be described with reference to the flowchart of FIG. 13. The following process is realized by the camera 101 as the image capture system 100, in that the control unit 112 controls each unit of the device and executes each function according to a program stored in the ROM 113. The wearable device 102 is realized by the control unit 138 controlling each unit of the device and executing each function according to a program recorded in the ROM 151. The same reference numerals are given to the processes of the same steps as those in the first embodiment, and detailed descriptions thereof will be omitted.

[0097] First, the photographing operation of the camera 101 will be described with reference to Fig. 4. The processes from step S301 to step S305 in Fig. 4 are the same as those from step S301 to step S305 in Fig. 3 of the first embodiment, and therefore the description will be omitted. In the second embodiment, the main exposure process in step S401 differs from that in the first embodiment. In step S401, the camera 101 performs exposure interruption process based on the amount of motion blur of a target part of the subject during exposure, and photographs the subject with reduced subject blur.

[0098] Next, the control of the main exposure process in step S401 performed by the control unit 112 of the camera 101 based on the amount of motion blur relative to the target part of the subject during exposure will be described in detail with reference to the flowchart of FIG.

[0099] 13, the camera 101 starts the main exposure process in the same manner as in step S306 in embodiment 1. The configuration of the electronic front curtain shutter and the shooting operation when performing the main exposure process will be described in detail with reference to FIGS.

[0100] 14 is a front view showing the image sensor 122 and the mechanical rear curtain 1403 as viewed from the lens side along the optical axis, and shows the reset scan performed by the image sensor 122 after shooting has started, and the state in the middle of the travel of the mechanical rear curtain 1403. An arrow 1401 indicates the operation direction of the reset scan (the travel direction of the electronic front curtain 1407) and the travel direction of the mechanical rear curtain 1403. FIG. 14 shows a state in which the mechanical rear curtain 1403 formed by the mechanical shutter 118 in FIG. 1 blocks light from a part of the image sensor 122. Furthermore, a reset line 1408 is a line (reset line) of the reset scan performed by the image sensor 122, and corresponds to the end of the electronic front curtain 1407 as an operation for resetting the amount of accumulated charge of a pixel to zero. An area 1406 formed by a slit between a reset line 1408 and an end 1405 of a mechanical rear curtain 1403 is controlled to move in the direction of an arrow 1401 according to the movement of an electronic front curtain 1407 and a mechanical rear curtain 1403. The charge accumulation time due to exposure of the pixel is the time from when the reset line 1408 passes, that is, when the pixels are sequentially reset line by line in the direction of the arrow 1401, until the light is blocked by the mechanical rear curtain 1403. In this way, the reset line 1408 moves in the direction of the arrow 1401 and charge accumulation for each line is started, so the start timing of charge accumulation differs for each line of the image sensor 122. The timing of charge accumulation will be described in detail with reference to FIG. 15. FIG. 15(a) is a conceptual diagram of the resetting of charge and the start of reading, and FIG. 15(b) is an explanatory diagram of the timing of the resetting and reading of charge for each line. Lines 1501 to 1510 in FIG. 15(b) show the timing at which the charge is reset for each line, with end line 1501 reading out the charge first and then resetting the charge. Conversely, end line 1510 is the last line to be reset. In this way, the reset timing is controlled for each line. Because the reset timing differs for each line, control unit 112 controls the charge readout time so that the charge accumulation time is the same for each line, so that the exposure time for each line is the same for lines 1511 to 1520.

[0101] In step S1301 of FIG. 13, the control unit 112 corrects the motion vector of the subject calculated by the camera 101 immediately before the actual exposure detected in step S303 of FIG. 4 based on the output of the acceleration sensor 146 included in the subject motion information from the wearable device 102 during exposure. Once the actual exposure starts, the camera 101 cannot capture the preparatory shot image unless there are multiple image sensors 122, so the motion vector of the subject cannot be updated during the actual exposure. Therefore, the motion vector of the subject in the main part corresponding to the processing of step S703 in the first embodiment is corrected using the acceleration information of the acceleration sensor 146 for the motion vector of the subject in the part of interest calculated immediately before the actual exposure. Then, a process of converting the corrected motion vector into the amount of motion blur is performed. The above process makes it possible to estimate the amount of motion blur even during the actual exposure of the camera 101.

[0102] In step S1302, control unit 112 determines whether the amount of motion blur estimated in step S1301 exceeds the allowable amount of motion. If the amount of motion exceeds the allowable amount of motion, the process proceeds to step S1303, and if the amount of motion blur does not exceed the allowable amount of motion, the process proceeds to step S1304.

[0103] In step S1304, the control unit 112 determines whether the shooting conditions determined in step S303 in Fig. 4 are satisfied. The shooting conditions are determined mainly based on whether they will affect accumulated blur during exposure. This can be determined by whether the exposure time corresponding to the shutter speed set before the main exposure is satisfied. If they are satisfied, the process proceeds to step S1305 and the exposure ends.

[0104] In step S1303, control unit 112 determines that continued exposure of image sensor 122 will result in accumulation blur in the image. Control unit 112 closes mechanical shutter 118 earlier than the exposure time set in the shooting conditions to block light entering image sensor 122 so that external light does not enter image sensor 122, and controls camera 101 to proceed to processing in step S1305 and end exposure. The method of controlling exposure termination in step S1303 will be described in detail with reference to FIG.

[0105] 16 shows a timing chart of the process in which the control unit 112 of the camera 101 controls the mechanical rear curtain to interrupt exposure when the allowable amount of motion is exceeded due to a change over time in the amount of subject motion blur during main exposure processing. By controlling the mechanical rear curtain and controlling the exposure time when a situation arises in which subject motion blur is likely to occur in the image during main exposure, subject motion blur can be reduced.

[0106] Lines 1501 to 1510 in FIG. 16 (the same shooting conditions as in FIG. 15(b)) are reset lines. After the reset process of line 1501 begins, external light reaches the image sensor 122, and accumulation of electric charge begins. If the movement of the camera 101 body does not cause accumulated blur in the image captured during exposure, electric charge is accumulated up to lines 1511 to 1520 of the next reset process (the same shooting conditions as in FIG. 15(b)). This time, a method for controlling the accumulation of electric charge when the amount of subject motion blur during exposure exceeds the allowable amount of motion will be described.

[0107] The control unit 112 executes the reset process of the first reset line 1501 in FIG. 16 to start exposure. At this time, if the amount of subject motion blur increases and is detected to exceed the allowable vibration threshold at the timing of 1611 in FIG. 16, the control unit 112 drives the mechanical rear curtain and closes the mechanical shutter 118 so that external light does not reach the image sensor 122. Due to the action of the mechanical shutter 118, the period between 1601 and 1610 in FIG. 16 becomes the actual exposure time. In other words, since the exposure time is not between 1601 and 1610 and the period between lines 1511 and 1520, no charge is accumulated in the image sensor 122, and thus it is possible to prevent the accumulation blur of the image caused by the influence of the shake of the camera 101 body. In this description, the accumulation of charge is stopped by controlling the mechanical shutter 118, but for a shutter not equipped with the mechanical shutter 118, the same process is possible by generating a reset pulse from the pulse generating unit 126 to interrupt the accumulation of charge.

[0108] Next, in step S1306 in Fig. 13, the control unit 112 determines whether the exposure time has reached the set exposure time. If it is determined that the exposure time is shorter than the set exposure time, the process proceeds to step S1307.

[0109] In step S1307, the control unit 112 performs signal processing to multiply the acquired image by a digital gain equivalent to the difference in exposure time so that the brightness is equal to the original exposure time, in response to the insufficient exposure time of the image data. The digital gain calculated from the difference in exposure time is calculated using the following formula.

[0110] Digital gain = exposure time set at the start of imaging / (exposure time set at the start of imaging - time from start of exposure to when mechanical shutter is turned off)...Equation 7 The brightness of the image data is corrected to the brightness equivalent to the expected exposure time by uniformly multiplying the image by the digital gain calculated using Equation 7. To perform more precise gain correction, a digital gain may be calculated for each horizontal line of the image data and the image data may be multiplied by the digital gain.

[0111] On the other hand, when the set exposure time has elapsed, the control unit 112 reads out the electric charges for each line, and performs a charge reset process from the line for which reading has been completed, thereby ending the exposure of the camera 101 and acquiring an image.

[0112] The process of correcting the exposure amount by applying a digital gain to the insufficient exposure time (exposure amount) performed in step S1307 so that the brightness is the original exposure time will be described in detail with reference to FIG. 12. As an example, image signal 1201 in FIG. 12 shows an image signal when captured with a target exposure amount, and image signal 1202 shows an image signal in which the exposure time is insufficient and the target exposure amount is not reached because the exposure was interrupted midway. In FIG. 12, the horizontal axis shows the subject brightness, and the vertical axis shows the image signal level at the subject brightness on the horizontal axis. For example, image signal 1202 is an image signal when captured with half the exposure time compared to 1201 which is the target exposure amount. The exposure amount when the camera 101 captures an image is generally determined by the F value, ISO sensitivity, and shutter speed (exposure time). Therefore, by interrupting the exposure midway, the exposure amount falls short of the target exposure amount by the amount of the shortened exposure time. When the exposure time is halved, the exposure amount is also halved. Therefore, by applying a 2x digital gain, the image signal 1202 can be adjusted to the same exposure as the image signal 1201, and the original target exposure can be obtained. By correcting the above processing on a line-by-line basis of the image sensor 122, it becomes possible to photograph the subject at the brightness of the original target exposure even if the exposure is interrupted.

[0113] The method of controlling exposure based on the magnitude of subject blur during exposure by the imaging system 100 of the second embodiment has been described above with reference to the flowchart in Fig. 13. With these processes, even in a situation where it is difficult to change the shutter speed during exposure, it is possible to obtain an image with reduced subject blur by controlling the exposure time.

[0114] (Third embodiment) A preferred embodiment of the present invention, Example 3, will be described in detail below with reference to the drawings. Example 3 has a configuration in which a plurality of subjects to be photographed wear wearable devices 102, and a camera 101 is set outside the plurality of subjects.

[0115] An example of a scene in which multiple subjects are wearing wearable devices 102 is shown in Fig. 17. Fig. 17 shows an example in which six subjects, subject A to subject F, are wearing wearable devices 102 denoted by 1701 to 1708. In this example, subject A is wearing wearable device 102 denoted by 1701, subject B is wearing wearable device 102 denoted by 1702, subject C is wearing wearable device 102 denoted by 1705, subject D is wearing wearable device 102 denoted by 1706, subject E is wearing wearable device 1707, and subject F is wearing wearable device 102 denoted by 1708. Subject movement information of each subject is sensed through these multiple wearable devices 102, and the shooting conditions of camera 101 are determined using the obtained subject movement information as auxiliary information.

[0116] The processing on the wearable device 102 side of the imaging system 100 in the third embodiment of the present invention is similar to that in the first embodiment of the present invention, and therefore a description thereof will be omitted. The operation performed by the control unit 112 of the camera 101, which is a feature of the third embodiment, will be described with reference to the flowchart of FIG. 18. In the following processing, the control unit 112 of the camera 101 as the imaging system 100 controls each unit of the device according to a program stored in ROM 113. Also, the wearable device 102 is realized by the control unit 138 controlling each unit of the device according to a program recorded in ROM 151. The same reference numerals are given to the processing of the same steps as in the first embodiment, and detailed description thereof will be omitted.

[0117] The photographing operation of the camera 101 is the same as the processing of steps S301 to S306 in Fig. 3 of the first embodiment, and therefore the description will be omitted. Details of the processing of step S303, which is a feature of the third embodiment, will be described with reference to the flowchart of Fig. 18. Note that the description of the part of the flowchart of Fig. 18 that performs the same processing as the processing of steps S701 to S705 in Fig. 7 of the first embodiment will be omitted.

[0118] In step S701 in FIG. 18, the control unit 112 sets initial shooting conditions in the camera 101, and captures preparatory images successively.

[0119] In step S702, the motion vector calculation section of the image processing section 124 calculates the motion vector of the subject from the time-series consecutive preparatory captured images captured in step S701.

[0120] In step S1801, the image processing unit 124 detects a subject from the chronologically consecutive preparatory captured images captured in step S701. A method of detecting a subject uses a general subject detection technique or the like. For example, face / face organ detection and head detection are included. Face / face organ detection is a method of detecting a face and face organ area in which a person's face and organs exist from a captured image, and head detection is a method of detecting a head area in which a person's head exists from a captured image by a technique based on pattern recognition or machine learning.

[0121] In step S1802, the image processing unit 124 determines whether the subject detected in step S1801 is one person or multiple people. If it is determined that there is one person, the process proceeds to step S1804, and if it is determined that there are multiple people, the process proceeds to step S1803.

[0122] In step S1803, since there are multiple subjects detected in step S1802, image processing unit 124 detects a main subject from among the people wearing wearable device 102. A general main subject detection method may be used to detect the main subject. For example, the person who occupies the largest area of ​​the subject in the angle of view in the preparation-captured image or the person closest to the center of the preparation-captured image is detected as the main subject. Alternatively, a person who has been registered as the main subject in advance by the user may be detected as the main subject.

[0123] In step S1804, it is determined whether the person detected as the main subject in step S1803 is wearing one or multiple wearable devices 102. If it is determined that there is one, the process proceeds to step S1806, and if it is determined that there are multiple wearable devices, the process proceeds to step S1805. In order to determine the number of wearable devices 102, the user may register the wearable devices 102 worn by each subject in advance. Alternatively, it may be possible to determine the number of wearable devices 102 by specifying which part of each subject detected in step S1801 is wearing a wearable device 102. The same process as that described in step S503 of FIG. 5 in the first embodiment is performed.

[0124] In step S1805, since it was determined in step S1804 that the person detected as the main subject is wearing multiple wearable devices 102, a main wearable device 102 is detected from among the multiple wearable devices 102. A method for detecting the main wearable device 102 (main sensor) will be described in detail with reference to the flowchart in FIG.

[0125] In step S1901, the control unit 112 determines whether or not a priority has been set in advance by the user for the wearable device 102. If it is determined that a priority has been set, the process proceeds to step S1904, and if it is determined that a priority has not been set, the process proceeds to step S1902.

[0126] As an example of a priority level set by the user in advance, Fig. 20 shows an example of a setting screen for the priority levels of wearable devices 102 that can be set by the user on camera 101. In Fig. 20, 2000 is a setting screen for setting the priority levels of wearable devices 102, which is displayed on display unit 127 of camera 101 based on an instruction from control unit 112, for example. 2001 is a subject ID, which indicates subjects A to F in the image diagram of Fig. 17. 2002 is a wearable device 102 ID, which indicates wearable devices 102 1701 to 1708 in the image diagram of Fig. 17. 2003 indicates the priority levels of wearable devices 102. In this setting screen example, the user has set the priority level of wearable device 102 1704 of subject B to be the highest.

[0127] In step S1902, the image processing unit 124 detects the wearable device 102 that is located within a predetermined area in the preparatory-captured image for a certain period of time. For example, when the angle of view of the preparatory-captured image is set to 100% as the predetermined area, the image processing unit 124 detects the wearable device 102 that is present within a 90% rectangular range centered on the central coordinates of the preparatory-captured image for a period of time equal to or longer than an arbitrarily set threshold value.

[0128] In step S1903, the control unit 112 calculates the priority of the wearable device 102 detected in step S1902 (priority calculation). The items to be prioritized are set in advance by the user, and the priority is calculated according to the set contents. An example of a setting screen is shown in FIG. 21. Reference numeral 2100 denotes a setting screen for setting items to be prioritized for the wearable device 102. Reference numeral 2101 denotes the contents of the items to be prioritized, and includes items such as "in order of acceleration speed", "setting of the wearing part (head, torso, hands, feet)", "area of ​​the face within the angle of view", "area of ​​the wearing part within the angle of view", "in order of distance between the center of the angle of view and the wearable device 102", "in order of distance between the center of the face and the wearable device 102", and "face detection reliability". Reference numeral 2102 denotes items to be used as the basis for prioritizing the main wearable device 102, and the items selected by the user are checked for the contents of the items in 2101.

[0129] A method for calculating the priority will be described with reference to Fig. 22. For example, when items for determining the priority are set as described with reference to Fig. 21, the control unit 112 calculates a score for each set item, and then calculates a total value by adding up each score. The wearable devices 102 are prioritized in descending order of the total score, and the priority order for each wearable device 102 is determined in the same manner as in Fig. 20 described in step S1901. A method for calculating the total score will be described with reference to Fig. 22.

[0130] In FIG. 22, 2201 is a subject ID and indicates subjects A to F in the image diagram of FIG. 17. 2202 is a wearable device 102 ID and indicates wearable devices 102 1701 to 1708 in the image diagram of FIG. 17. 2203 is a score value in "order of acceleration speed", and the higher the rank of the wearable device 102 in speed, the higher the score value. 2204 is a "wearing part", and since the hand is selected in FIG. 21, the score value of the wearable device 102 worn on the hand is higher. 2205 is a "distance order between the center of the angle of view and the wearable device 102", and the closer the distance between the center of the angle of view and the wearable device 102, the higher the score value. 2206 is a "face detection reliability order", and the higher the reliability of the face detection of each subject, the higher the score value. 2207 is a total score value, which is the sum of the scores calculated in 2203 to 2206. For example, in step S1803, when subject B is detected as the main subject, subject B is wearing three wearable devices 102, 1702, 1703, and 1704. Here, when the "wearing part" is set to the hand as an item for determining the priority, the total score value of the wearable device 102 in 1703 is the largest, and the priority order is the highest.

[0131] Figure 22(b) shows the result of prioritization when prioritization is based on the calculated total score value. In Figure 22(b), 2208 is a subject ID and indicates subjects A to F in the image diagram of Figure 17. 2209 is a wearable device 102 ID and indicates wearable devices 102 1701 to 1708 in the image diagram of Figure 17. 2210 shows the priority order of wearable devices 102.

[0132] If subject B is wearing wearable devices 102 on both his right and left hands, the score values ​​for "wearing part" will both be the same 100, so the higher the score values ​​for the other items, the higher the priority.

[0133] In step S1904, the control unit 112 provisionally determines the wearable device 102 with the highest priority as the main wearable device 102 in accordance with the priority calculated in step S1901 or step S1903.

[0134] In step S1905, the control unit 112 determines whether the wearable device 102 provisionally determined in step S1904 is located within a predetermined area in the preparation-captured image for a certain period of time. For example, assuming that the angle of view of the preparation-captured image is 100% as the predetermined area, if the wearable device 102 is within a 90% rectangular range centered on the central coordinates of the preparation-captured image for an arbitrarily set threshold time or longer, the process proceeds to step S1907. If the wearable device 102 is not located, the process proceeds to step S1906.

[0135] In step S1906, the control unit 112 provisionally determines the wearable device 102 with the next highest priority after the wearable device 102 provisionally determined in step S1904 as the primary wearable device 102. Thereafter, the process proceeds to step S1905, and repeats this determination until the process proceeds to step S1907 where the primary wearable device 102 is determined. Assume that after repeating this determination, it is determined that none of the wearable devices 102 with the priority determined in step S1901 or step S1903 has been located within a predetermined area of ​​the preparation captured image for a certain period of time. In that case, the wearable device 102 that was previously set as the default is determined as the primary wearable device 102 in the next step S1907.

[0136] In step S1907, the image processing unit 124 detects the wearable device 102 tentatively determined in step S1904 or step S1906 as the main wearable device 102. This completes the flow up to the determination of the main sensor.

[0137] Next, in step S1806 in FIG. 18, a main wearable device 102 is determined based on the detection result in the flow in FIG. 19 in order to obtain information about the wearable device 102 in the next step.

[0138] In step S703, image processing unit 124 calculates a vector corresponding to the main part of the subject using the subject motion information acquired from main wearable device 102 determined in step S1806 and the motion vector of the subject determined by camera 101.

[0139] In step S704, the image processing unit 124 estimates the motion vector of the subject in the main part calculated in the processing of the above steps, and the amount of motion blur that will occur in the subject at the shutter speed set by the user.

[0140] In step S705, the control unit 112 compares the amount of subject motion blur calculated in step S704 with the allowable amount of motion, and changes the shutter speed of the preparatory image to be captured next so that the amount of subject blur is less than the allowable amount of motion, and changes the shooting conditions of the preparatory image.

[0141] The processing of the imaging system 100 of the third embodiment has been described above. Specifically, even in a scene in which the subject is more than one person, a main wearable device 102 is determined from the multiple wearable devices 102 worn by the subjects. Then, subject motion information is sensed through the main wearable device 102, and the subject motion information acquired by the wearable device 102 is used as auxiliary information to update the motion vector of the subject. Using the result, the shooting conditions of the camera 101 are determined, making it possible to acquire an image with reduced subject blur.

[0142] Items for determining the priority of the wearable device 102 include, but are not limited to, those introduced in step S1903. For example, the closer the "distance between the camera and the subject" or the "distance between the camera and the wearable device 102", the higher the score calculated, and the higher the priority order may be.

[0143] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0144] 100 Imaging System 101 Camera 121 Optical system 122 Image sensor 123 A / D conversion section 124 Image Processing Unit 125 Recording Department 126 Pulse Generator 127 Display section 118 Mechanical Shutter 119 Gyro sensor 111 Vertical drive modulation section 112 Control section 113 ROM 114 RAM 102 Wearable Devices 132 Power Supply Module 131 Communication Module 133 Connector 134 In-camera 135 Rear Camera 136 Light 137 System Memory 138 Control Unit 139 Speakers 141 Touch Screen Display 142 Liquid crystal display 143 Touchscreen 144 Switch 145 Illuminance sensor 147 Gyro Sensor 148 Depth Sensor 151 ROM 152 RAM 146 Acceleration Sensor

Claims

1. an imaging means for imaging a subject; a subject movement detection means for detecting a movement of a subject using the image captured by the imaging means; a receiving means for receiving a sensing result transmitted from an external device; an exposure control unit for controlling the exposure of the imaging unit; and a sensor means for acquiring information about a subject; a transmitting means for transmitting a sensing result of the sensor means to the imaging device; a sensor device attached to a subject to be photographed, the sensor means senses at least one of an amount of movement, an amount of change in movement, and a change in position corresponding to a part of the subject on which the sensor means is worn; The exposure control means receives the sensing result of the sensor means, and controls the exposure of the imaging means using the sensing result of the sensor means and the detection result of the movement of the subject.

2. the subject motion detection means calculates a motion vector of the subject using the image captured by the imaging means; 2. The photographing system according to claim 1, wherein said exposure control means controls exposure of said image capturing means using the motion vector calculated by said subject motion detection means.

3. the subject motion detection means calculates a motion vector of the subject using the image captured by the imaging means; 2. The photographing system according to claim 1, wherein said exposure control means controls the exposure of said image capturing means by using said motion vector selected by said subject motion detection means using the sensing result of said sensor means.

4. Further, a reliability calculation means is provided for calculating the reliability of the motion vector of the object, 4. The photographing system according to claim 2, wherein the exposure control means selects a motion vector based on the reliability, and controls the exposure of the image pickup means using the selected motion vector.

5. 4. The photographing system according to claim 2, wherein the exposure control means performs a process of correcting the selected motion vector based on the sensing result of the sensor means with respect to the motion vector calculated by the subject motion detection means, and controls the exposure of the imaging means using the corrected motion vector.

6. a subject blur amount estimation means for estimating the amount of blur in an image when photographed based on the motion vector calculated by the subject motion detection means and the exposure time set by the exposure control means, 4. The photographing system according to claim 2, wherein the exposure control means controls the exposure of the image pickup means based on the amount of blur of the image.

7. 2. The photographing system according to claim 1, wherein the interval at which the data output from said sensor means is updated is shorter than the interval at which the data output from said subject movement detection means is updated.

8. 2. The photographing system according to claim 1, wherein the exposure control means controls an exposure time that is set before the image is captured by the image capturing means.

9. 2. The photographing system according to claim 1, wherein said exposure control means controls the amount of charge accumulated in said image pickup means.

10. 2. The photographing system according to claim 1, wherein the exposure control means performs a correction process by applying a digital gain when the exposure is insufficient for a target exposure set before the image is captured.

11. the imaging device has a main sensor determination means for determining a main sensor device when there are a plurality of sensor devices; 2. The photographing system according to claim 1, wherein said exposure control means controls the exposure of said image capturing means using the sensing results of said main sensor device and the detection results of said subject movement detection means.

12. the primary sensor determination means includes a priority calculation means for calculating a priority as a primary sensor device; 12. The photographing system according to claim 11, wherein the priority calculation means calculates a higher priority depending on the acceleration of the sensor device, the distance from the center of the image, the distance from the camera, the area of ​​the subject wearing the sensor device, and the area of ​​the body part wearing the sensor device.

Images