Information processing device, information processing method, and information processing system

Abstract

An information processing device according to the present invention comprises: an acquisition unit that acquires an image obtained by an image capturing device, and focus information related to the focus when the image was captured; and a tracking processing unit that performs, by using the focus information, camera tracking processing on the image.

Description

Information processing device, information processing method, and information processing system

[0001] The present technology relates to an information processing device, an information processing method, and an information processing system, and in particular to a camera tracking technology.

[0002] Conventionally, so-called camera tracking has been performed in which feature points are extracted from a moving image captured by an imaging device and the position and orientation of the camera are estimated based on the extracted feature points (see, for example, Patent Document 1).

[0003] JP 2011-118724 A

[0004] However, in camera tracking, if feature points are extracted from a moving object, it is not possible to distinguish whether the feature points moved due to the movement of the moving object or due to the movement of the imaging device 2, resulting in a decrease in the accuracy of the tracking process. Therefore, it is conceivable to classify each subject in the image as either a moving or a non-moving object by performing image processing before extracting feature points, and then extract feature points from subjects classified as non-moving. However, this method has the problem of requiring a long processing time because image processing is performed to classify moving and non-moving objects.

[0005] The present technology has been made in consideration of the above circumstances, and aims to improve the accuracy of tracking processing while reducing processing time.

[0006] The information processing device according to the present technology includes an acquisition unit that acquires an image obtained by an imaging device and focus information related to the focus when the image was captured, and a tracking processing unit that performs camera tracking processing on the image using the focus information. This enables the information processing device to accurately extract feature points used in the camera tracking processing based on the focus information.

[0007] FIG. 1 is a diagram showing a configuration of an information processing system as an embodiment of the present technology. FIG. 1 is a diagram showing a configuration of an imaging device. FIG. 2 is a block diagram showing the internal configurations of an imaging device and an interchangeable lens. FIG. 3 is a block diagram showing the configuration of a computer. FIG. 4 is a functional block diagram of a control unit. FIG. 5 is a diagram explaining the data structure of an image file obtained by the imaging device. FIG. 6 is a diagram explaining camera tracking processing. FIG. 7 is a diagram explaining extraction of feature points when a person is detected as a moving object. FIG. 8 is a diagram explaining extraction of feature points when an animal is detected as a moving object. FIG. 9 is a diagram explaining extraction of feature points when a subject specified by a user is an animal. FIG. 10 is a diagram explaining movement of the imaging device. FIG. 11 is a flowchart showing the flow of tracking processing. FIG. 12 is a diagram explaining weighting. FIG. 13 is a flowchart showing the flow of tracking processing. FIG. 14 is a functional block diagram of a control unit in a third example.

[0008] Hereinafter, the embodiments will be described in the following order: <1. Information Processing System> <2. Autofocus Processing> <3. Camera Tracking Processing> <4. Modifications> <5. Summary of the Embodiments> <6. Present Technology>

[0009] In this technology, "image" refers to a moving image. Furthermore, "image" does not only refer to a state in which it is displayed on a display unit, but may also refer to image data that is not displayed on a display unit. "Subject" refers not only to an object captured by an imaging device, but also includes the subject image captured in the image. Furthermore, "subject" includes not only people, but also various objects such as animals, birds, insects, cars, trains, roads, and buildings, and further includes parts (portions) of these.

[0010] 1 is a diagram showing the configuration of an information processing system 1 according to an embodiment of the present technology. As shown in FIG. 1, the information processing system 1 includes an imaging device 2 and a computer 4.

[0011] The imaging device 2 is capable of adding meta information to images (moving images) obtained by capturing an incident subject image through the interchangeable lens 3, which includes the focus lens 16 (see FIG. 3 ). Here, "adding" means making the meta information available when performing image editing processing on the image. Therefore, when meta information is added to an image, the image and the meta information may be recorded on a recording medium as a single file or transmitted to another device (computer 4). Furthermore, the image and the meta information may be recorded on separate recording media (or separate recording areas on the same recording medium) or transmitted separately to another device (computer 4).

[0012] The computer 4 is, for example, a personal computer, a mobile terminal device, or a tablet terminal device, and is capable of acquiring images and meta information from the imaging device 2. The computer 4 may also be a server that performs cloud computing, etc. In this case, the computer 4 acquires images and meta information from the imaging device 2 via a network.

[0013] The computer 4 performs camera tracking processing, which will be described in detail later. The camera tracking processing is processing for estimating (calculating) the movement (position and orientation) of the imaging device 2 based on an image captured by the imaging device 2, and estimating the 3D spatial structure within the image.

[0014] 2 is a diagram showing the configuration of the imaging device 2. The imaging device 2 (body) is configured as a digital camera device to which an interchangeable lens 3 is detachably attached. The imaging device 2 may have not only a function for capturing moving images but also a function for capturing still images. Note that although the interchangeable lens 3 is configured to be detachable from the imaging device 2, the interchangeable lens 3 may also be configured not to be detachable from the imaging device 2.

[0015] 2, the imaging device 2 includes an imaging element 55 that captures a subject image incident through the interchangeable lens 3, a display unit 61 that can display the captured image obtained by the imaging element 55 and GUIs such as various operation screens, an operation unit 65 through which the user inputs various operations, etc. The imaging device 2 also includes, for example, a configuration for recording the image captured by the imaging element 55, a configuration for performing image signal processing on the image captured by the imaging element 55, a configuration for communicating with the interchangeable lens 3, etc.

[0016] The interchangeable lens 3 is a lens unit that houses various lenses such as a focus lens, a zoom lens, etc. The interchangeable lens 3 also includes a drive unit that drives these lenses, a control unit that outputs drive signals to the drive unit, and a mount unit that has a connection function and a communication function with the imaging device 2.

[0017] Fig. 3 is a block diagram showing the internal configuration of the imaging device 2 and the interchangeable lens 3. As shown in Fig. 3, the interchangeable lens 3 has a mount section 11 that is detachably attached to a mount section 51 of the imaging device 2. The mount section 11 has a plurality of terminals for electrically connecting to the imaging device 2.

[0018] The interchangeable lens 3 also includes a lens-side control unit 12, a zoom lens 13, an image stabilization lens 14, an aperture 15, a focus lens 16, a detection unit 17, an operation unit 31, a memory 32, and a power supply control unit 33. The interchangeable lens 3 further includes a zoom lens driving unit 21, an image stabilization control unit 22, an aperture control unit 23, and a focus lens driving unit 24.

[0019] The lens side control unit 12 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU reads out a program stored in a predetermined storage device such as the ROM or memory 32 into the RAM and executes it, thereby performing overall control of the interchangeable lens 3.

[0020] For example, the lens-side control unit 12 controls the position of the zoom lens 13 based on instructions from the imaging device 2 supplied via a predetermined communication terminal of the mount unit 11 or user operation received by the operation unit 31. Specifically, the lens-side control unit 12 acquires the current position of the zoom lens 13 detected by the detection unit 17, which is configured, for example, by a magnetic sensor (MR sensor). Then, based on the acquired results, the lens-side control unit 12 determines a drive direction and drive amount for moving the zoom lens 13 to a predetermined position, and outputs the determined drive direction and drive amount together with a movement command to the zoom lens drive unit 21. Based on the movement command supplied from the lens-side control unit 12, the zoom lens drive unit 21 moves the zoom lens 13 in the optical axis direction so as to achieve the instructed drive direction and drive amount.

[0021] Here, the detection unit 17 comprehensively represents the configuration for detecting the state of the interchangeable lens 3, such as the positions of the zoom lens 13, the image stabilization lens 14, and the focus lens 16, and the aperture diameter of the diaphragm 15. In the detection unit 17, the position of the lens can be detected by, for example, a magnetic sensor, a photodiode array, a potentiometer, a reflective encoder, or the like.

[0022] The detection unit 17 also includes a motion sensor 17a. The motion sensor 17a detects the motion of the imaging device 2. Specifically, the motion sensor 17a includes an acceleration sensor that detects acceleration in three mutually orthogonal axial directions, including the optical axis direction, and a gyro sensor that detects angular velocities (pitch, yaw, and roll) around the three axes. The motion sensor 17a detects acceleration and angular velocities, for example, in synchronization with frames constituting moving images obtained by the imaging device 2 (at the same intervals as the frames). The lens-side control unit 12 performs processing to transmit the acceleration and angular velocities detected by the motion sensor 17a to the imaging device 2 as motion information.

[0023] Lens-side control unit 12 controls image stabilization lens 14 to correct camera shake. Specifically, lens-side control unit 12 determines the drive direction and drive amount of image stabilization lens 14 in a direction that cancels out the amount of camera shake, based on the amount of movement (camera shake amount) of imaging device 2 detected by motion sensor 17a, and outputs the determined drive direction and drive amount together with a movement command to image stabilization control unit 22. Based on the movement command supplied from lens-side control unit 12, image stabilization lens 14 moves in the instructed drive direction and drive amount.

[0024] Furthermore, when the power supply is turned off, the lens-side control unit 12 performs control to mechanically lock the image stabilization lens 14. When power is being supplied from the imaging device 2 to the interchangeable lens 3, the image stabilization lens 14 is maintained at a predetermined position through control via the image stabilization control unit 22. On the other hand, when the power supply is turned off, position control by the image stabilization control unit 22 stops, causing the image stabilization lens 14 to fall a predetermined amount in the direction of gravity. Therefore, the lens-side control unit 12 mechanically locks the image stabilization lens 14 via the image stabilization control unit 22 in accordance with the timing when the power supply is turned off, thereby preventing the image stabilization lens 14 from falling. The image stabilization control unit 22 mechanically locks the image stabilization lens 14 based on a locking command supplied from the lens-side control unit 12.

[0025] Furthermore, the lens-side control unit 12 controls the aperture diameter of the diaphragm 15 in accordance with instructions from the imaging device 2 supplied via a predetermined communication terminal of the mount unit 11. Specifically, the lens-side control unit 12 acquires the aperture diameter of the diaphragm 15 detected by the diaphragm detection sensor in the detection unit 17, and issues a command to the diaphragm control unit 23 to drive the diaphragm 15 so as to achieve the F-number instructed by the imaging device 2. The diaphragm control unit 23 drives the diaphragm 15 so as to achieve the aperture diameter instructed by the lens-side control unit 12.

[0026] Furthermore, the lens-side control unit 12 controls the position of the focus lens 16 based on instructions from the imaging device 2 supplied via a predetermined communication terminal of the mount unit 11. Here, for example, in autofocus processing, the imaging device 2 instructs the lens-side control unit 12 of a target focus lens position. The lens-side control unit 12 acquires the current position of the focus lens 16 from the detection unit 17 and determines a drive direction and drive amount for moving the focus lens 16 to the target position based on the acquired current position and the target focus lens position instructed by the imaging device 2. The lens-side control unit 12 then outputs the determined drive direction and drive amount together with a movement command to the focus lens drive unit 24. The focus lens drive unit 24 moves the focus lens 16 in the optical axis direction so as to achieve the instructed drive direction and drive amount.

[0027] Here, the focus lens 16 is configured as a "focus lens group" including one or more optical elements. When the focus lens group includes multiple optical elements, those optical elements are displaced together in association with focus adjustment. This also applies to the zoom lens 13. That is, the zoom lens 13 is configured as a "zoom lens group" including one or more optical elements, and when the zoom lens group includes multiple optical elements, those optical elements are displaced together in association with zoom adjustment.

[0028] In this example, the zoom lens 13 and the focus lens 16 are each configured with one zoom lens group and one focus lens group, respectively, but they may each be configured with a plurality of zoom lens groups and a plurality of focus lens groups.

[0029] The focus lens driving unit 24 may have a lens driving source such as an ultrasonic motor, a DC motor, a linear actuator, a stepping motor, or a piezoelectric element.

[0030] It should be noted that the focus adjustment can also be configured to be performed in response to a user operation received by the operation unit 31 .

[0031] The memory 32 is configured by a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Memory), and can be used to store the operating programs of the lens side control unit 12 and various data.

[0032] The power supply control unit 33 detects the amount of power supplied from the imaging device 2, and based on the detected amount of power, optimally allocates the amount of power to each part within the interchangeable lens 3 (the lens side control unit 12 and various driving parts) and supplies power.

[0033] The imaging device 2, which is the body side, is provided with a mount section 51 to which the interchangeable lens 3 is detachably attached. The mount section 51 has a plurality of terminals for electrically connecting with the mount section 11 of the interchangeable lens 3. When the interchangeable lens 3 is attached to the mount section 51 of the imaging device 2, corresponding terminals are electrically and physically connected between the mount section 51 and the mount section 11 of the interchangeable lens 3. The terminals to be connected include, for example, a terminal for supplying power (power supply terminal), a terminal for transmitting commands and data (communication terminal), and a terminal for transmitting a synchronization signal (synchronization signal terminal).

[0034] The imaging device 2 also includes a body-side control unit 52, a shutter 53, a shutter control unit 54, an imaging element 55, an ADC (Analog to Digital Converter) 56, a frame memory 57, an image signal processing unit 58, a recording unit 59, a recording medium 60, a display unit 61, a memory 62, a power control unit 63, a power supply unit 64, an operation unit 65, and a communication unit 66.

[0035] The power supply control unit 63 supplies power from the power supply unit 64 to each unit of the imaging device 2, including the body-side control unit 52. The power supply control unit 63 also calculates the amount of power that can be supplied to the interchangeable lens 3 based on the operating state of the imaging device 2, and supplies power to the interchangeable lens 3 via the mount unit 51. The power supply unit 64 is configured to include a secondary battery, such as a NiCd battery, a NiMH battery, or a Li battery. The power supply unit 64 can also be configured to receive power from a commercial AC power source via an AC adapter or the like.

[0036] The body side control unit 52 is configured with a microcomputer having a CPU, ROM, RAM, etc., and performs overall control of the imaging device 2 and the interchangeable lens 3 by the CPU reading out a program stored in a predetermined storage device such as the ROM or memory 62 into the RAM and executing it.

[0037] The memory 62 is configured by a non-volatile memory such as an EEPROM, and can be used to store the operating program of the body side control unit 52 and various data.

[0038] The body-side control unit 52 causes the image sensor 55 to perform image capture processing based on an operation signal representing a user operation supplied from the operation unit 65. Furthermore, the body-side control unit 52 transmits a predetermined command to the interchangeable lens 3 side via the mount unit 51 to drive the focus lens 16, zoom lens 13, etc.

[0039] The shutter 53 is disposed in front of the image sensor 55 (on the subject side), and opens and closes under the control of the shutter control unit 54. When the shutter 53 is closed, it blocks light from the subject that has passed through the optical system of the interchangeable lens 3. The shutter control unit 54 detects the open / closed state of the shutter 53, and supplies information indicating the detection result to the body-side control unit 52. The shutter control unit 54 drives the shutter 53 to an open or closed state based on the control of the body-side control unit 52.

[0040] The imaging element 55 is configured as an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and outputs a light reception signal obtained by capturing an image of a subject. If the imaging element 55 is configured as a CCD sensor or a CMOS sensor, an electronic shutter can be used, and therefore the shutter 53 can be omitted. If the shutter 53 is omitted, the shutter control unit 54 used to control it is also omitted.

[0041] The image sensor 55 also has pixels for capturing images (RGB pixels) and pixels for acquiring detection information used in AF processing using the image plane phase difference method, i.e., phase difference detection pixels each having a pair of photoelectric conversion units (diodes) into which pupil-split light is incident and for detecting the phase difference between a pair of images generated by the pair of photoelectric conversion units. In the image sensor 55, the RGB pixels are two-dimensionally arranged in the horizontal and vertical directions in a predetermined array pattern such as a Bayer array. In the image sensor 55, the phase difference detection pixels are discretely arranged in the horizontal and vertical directions on the pixel array surface on which the RGB pixels are two-dimensionally arranged.

[0042] The light reception signals obtained by photoelectric conversion of the RGB pixels in the image sensor 55 are converted into digital signals by the ADC 56, temporarily stored in a frame memory 57, and then input to an image signal processing unit 58. In Fig. 3, the captured image signal obtained by digitally converting the light reception signals of the RGB pixels as described above is referred to as a "captured image signal Si."

[0043] Meanwhile, the light reception signals obtained by photoelectric conversion of the phase difference detection pixels in the image sensor 55 are converted into digital signals by the ADC 56 and supplied to the body-side control unit 52. In Fig. 3, the signals obtained by digitally converting the light reception signals of the phase difference detection pixels in this way are denoted as "phase difference pixel signals Sp."

[0044] The body-side control unit 52 detects the phase difference (phase shift amount) between the pair of images based on the phase difference pixel signal Sp supplied via the ADC 56, and thereby calculates the focus deviation amount, i.e., the defocus amount, for each phase difference detection pixel as phase difference information. Since an existing method can be used to calculate the defocus amount, a detailed description will be omitted. The body-side control unit 52 can perform autofocus processing based on the phase difference information calculated in this manner. The phase difference detection pixel obtains a light reception signal synchronized with, for example, RGB pixels. That is, the body-side control unit 52 calculates the phase difference information synchronized with the frames constituting the moving image obtained by the imaging device 2 (at the same intervals as the frames).

[0045] The image signal processing unit 58 performs predetermined image signal processing on the captured image based on the captured image signal Si input via the frame memory 57. Examples of image signal processing here include demosaic processing, white balance (WB) adjustment, gamma correction, etc. After performing image signal processing on the captured image, the image signal processing unit 58 converts the image into a predetermined file format and records it on the recording medium 60 via the recording unit 59. At this time, the body-side control unit 52 adds meta information to the image recorded on the recording medium 60, as will be described in detail later.

[0046] The image signal processing unit 58 also supplies the image after image signal processing to the display unit 61, causing the display unit 61 to display the captured image.

[0047] The recording medium 60 is configured as a non-volatile memory, and the recording unit 59 is configured to be able to write data to the recording medium 60 and read data recorded on the recording medium 60. Here, the recording medium 60 may be detachable from the imaging device 2.

[0048] The display unit 61 is configured with a display device such as a liquid crystal display or an organic EL display, and is capable of displaying images. The display unit 61 is mounted on the back surface of the imaging device 2, opposite the front surface where the mount unit 51 is disposed, and can display so-called through images, images read from the recording medium 60, GUIs as various operation screens, etc. The display unit 61 may be mounted on a surface other than the back surface.

[0049] The operation unit 65 collectively represents operators with which the user inputs operations to the imaging device 2, such as various hardware keys such as a shutter button, a mode dial, and a zoom button, and a touch panel that is provided so as to be able to detect touch operations on the display screen of the display unit 61. The operation unit 65 accepts operations by the user and supplies an operation signal corresponding to the operation to the body-side control unit 52.

[0050] The communication unit 66 performs wired or wireless communication with an external device.

[0051] [1.3. Computer Configuration] Next, a description will be given of the configuration of the computer 4. Fig. 4 is a block diagram showing the configuration of the computer 4. As shown in Fig. 4, the computer 4 includes a control unit 71, a storage unit 72, a display unit 73, an operation unit 74, a recording unit 75, a recording medium 76, and a communication unit 77.

[0052] The control unit 71 is configured, for example, with a microcomputer having a CPU, ROM, RAM, etc., and performs overall control of the computer 4 by the CPU reading out programs stored in a specified storage device such as the ROM or memory unit 72 into the RAM and executing them.

[0053] The storage unit 72 is configured with a storage medium such as a solid-state memory. Various types of information can be stored in the storage unit 72. The storage unit 72 can also be used to store program data for the control unit 71 to execute various processes.

[0054] The display unit 73 is a liquid crystal display, an organic EL display, or the like, and displays various images.

[0055] The operation unit 74 is an input device used by the user, and includes various types of operators and operation devices, such as a keyboard, a mouse, a button, a dial, a touchpad, a touch panel, etc. When the operation unit 74 detects a user operation, a signal corresponding to the input operation is input to the control unit 71.

[0056] The recording medium 76 is configured as a non-volatile memory, and the recording unit 75 is configured to be able to write data to the recording medium 76 and read data recorded on the recording medium 76. Here, the recording medium 76 may be detachable from the computer 4.

[0057] The communication unit 77 performs wired or wireless communication with an external device.

[0058] Fig. 5 is a functional block diagram of the control unit 71. As shown in Fig. 5, the control unit 71 has functional units serving as a data acquisition unit 81 and a tracking processing unit 82.

[0059] When the recording medium 60 of the imaging device 2 is attached to the computer 4 as the recording medium 76, the data acquisition unit 81 acquires images and meta information from the recording medium 76 via the recording unit 75. The data acquisition unit 81 may also acquire images and meta information by communicating with the imaging device 2 via the communication unit 77.

[0060] The tracking processing unit 82 performs camera tracking processing on the image acquired by the data acquisition unit 81 by referring to meta information. The camera tracking processing performed by the tracking processing unit 82 will be described in detail later.

[0061] 2. Autofocus (AF) Processing Next, the autofocus processing executed by the imaging device 2 will be described.

[0062] The body-side control unit 52 detects a predetermined subject from an image and performs autofocus processing to bring the detected subject into focus. In addition, during the autofocus processing, the body-side control unit 52 can also perform processing to continue focusing by tracking a subject that has been brought into focus once as a tracking target.

[0063] Here, the types of subjects to be detected in the autofocus process are predetermined, and examples include people, living things such as animals, and vehicles such as cars and trains. Furthermore, for each subject to be detected, the parts of the subject that can be detected are predetermined. For example, for a person, these parts include the skeleton, face, and body.

[0064] The body-side control unit 52 detects a subject through a detection process. The detection process is performed using an algorithm learned by deep learning, such as a convolutional neural network (CNN). In the imaging device 2, an algorithm learned by deep learning, for example, is stored in advance in the memory 62. When capturing a moving image, the body-side control unit 52 detects a subject for each frame using the algorithm stored in the memory 62. Then, when a subject is detected, the body-side control unit 52 performs autofocus control to focus on the subject. Note that the detection process can detect multiple subjects for one frame of image.

[0065] The body-side control unit 52 then tracks the subject detected by the detection process between frames to determine whether it is moving. That is, the body-side control unit 52 determines whether the subject detected by the detection process is a moving object. For example, the body-side control unit 52 may determine that the detected subject is a moving object when it is moving in either the up-down direction or the left-right direction between frames. Furthermore, the body-side control unit 52 may determine that the detected subject is a moving object when it is moving in the depth direction between frames based on the amount of focus shift calculated by the phase difference detection pixel corresponding to the position of the subject.

[0066] For each subject determined to be a moving object, the body-side control unit 52 adds focus information indicating the position of the subject to the meta information and assigns it to the image. The focus information indicates the position of the subject (coordinates on the image) as a point or a range. The focus information is information related to focus and is generated for each frame.

[0067] Furthermore, when a predetermined position in the through image displayed on the display unit 61 is selected by the user, the body-side control unit 52 can execute image processing to detect the subject selected by the user and perform autofocus control to bring the subject into focus. Even in this case, the body-side control unit 52 adds focus information indicating the position of the subject selected by the user to the meta information for the image.

[0068] Fig. 6 is a diagram illustrating the data structure of an image file obtained by the imaging device 2. As shown in Fig. 6, when an image of a subject is captured by the imaging device 2, an image file 100 including an image 111 and meta information 112 is generated. The meta information 112 can include focus information 121 and lens information 122 related to the lens of the imaging device 2, such as focal length, focus position, aperture value, and lens distortion information. The image file 100 may also include common audio data.

[0069] <3. Camera Tracking Processing> [3.1. First Example of Camera Tracking Processing] Next, a first example of camera tracking processing performed by the computer 4 will be described. FIG. 7 is a diagram illustrating the camera tracking processing. In FIG. 7, frames constituting an image (moving image) are shown in chronological order from top to bottom. The tracking processing unit 82 performs camera tracking processing on the image 111 using the focus information 121, and detects the movement (position and orientation) of the imaging device 2.

[0070] Specifically, the tracking processing unit 82 extracts a plurality of feature points for each frame constituting an image, as indicated by markers 200 in Fig. 7. The feature points are determined by targeting the same subject portion across multiple frames, for example, by edge processing or the like. In this case, if feature points are extracted from a moving object, it will be impossible to distinguish whether the feature points have moved due to movement of the moving object or movement of the imaging device 2, and the accuracy of the tracking process will decrease.

[0071] Therefore, in the camera tracking process of the first example, the tracking processing unit 82 refers to the focus information 121 and performs a masking process to mask the range including the position of the subject detected as a moving object in the image 111 .

[0072] Then, the tracking processing unit 82 extracts feature points from the area excluding the masked area in the image 111.

[0073] 8 is a diagram illustrating the extraction of feature points when a person is detected as a moving object. As shown in FIG. 8 , assume that a person 300 is detected as a moving object and the position of the person 300 is indicated in the focus information 121. In this case, the tracking processing unit 82 masks a range 301 that includes the position of the person 300 indicated in the focus information. Here, for example, if the position of the person 300 is indicated by multiple pieces of skeletal information, the tracking processing unit 82 sets a range 301 that surrounds the positions indicated by the multiple pieces of skeletal information. Then, the tracking processing unit 82 masks the range 301.

[0074] The tracking processing unit 82 extracts feature points from the area of ​​the image 111 excluding the masked area 301. Therefore, feature points are not extracted from the masked area 301.

[0075] 9 is a diagram illustrating the extraction of feature points when an animal (dog) is detected as a moving object. As shown in FIG. 9, assume that an animal 310 is detected as a moving object and the position of the animal 310 is indicated as a range 311 in the focus information 121. In this case, the tracking processing unit 82 masks the range 311 indicated in the focus information 121.

[0076] The tracking processing unit 82 extracts feature points from the area excluding the masked area 311. Therefore, feature points are not extracted from the masked area 311.

[0077] 10 is a diagram illustrating the extraction of feature points when the subject specified by the user is an animal (bird). As shown in FIG. 10, it is assumed that the animal 320 specified by the user is detected and the position of the animal 320 is indicated as a range 321 in the focus information 121. In this case, the tracking processing unit 82 masks the range 321 indicated in the focus information 121.

[0078] The tracking processing unit 82 extracts feature points from the area excluding the masked area 321. Therefore, feature points are not extracted from the masked area 321.

[0079] Fig. 11 is a diagram illustrating the movement of the imaging device 2. In Fig. 11, the movement trajectory of the imaging device 2 is indicated by an arrow. The tracking processing unit 82 then records the movement in the time direction of feature points of the same subject part extracted for each frame image and performs geometric calculations to estimate the 3D spatial structure within the image. Note that a known method can be used for this calculation method, and therefore a detailed description thereof will be omitted.

[0080] Furthermore, as shown in FIG. 11, the tracking processing unit 82 estimates the movement (XYZ directions) and the attitude (Pan, Tilt, and Roll directions) of the image capturing device 2 within the estimated 3D spatial structure.

[0081] In this way, the computer 4 can reduce the extraction of feature points from moving objects by masking the moving objects based on the focus information 121 obtained by the imaging device 2, thereby enabling the camera tracking process to be performed with high accuracy. Furthermore, the computer 4 does not need to perform processing to distinguish between moving objects and non-moving objects, thereby shortening the processing time for the camera tracking process.

[0082] 12 is a flowchart showing the flow of the tracking process. As shown in FIG. 12, when the tracking process is started, the data acquisition unit 81 acquires the image file 100 from the image capture device 2 in step S1.

[0083] In the next step S2, the tracking processing unit 82 determines whether the acquired image file 100 includes focus information 121. If the image file 100 includes focus information 121 (Yes in step S2), the tracking processing unit 82 masks a range in the image 111 that includes a subject detected as a moving object, based on the focus information 121. If the image file 100 does not include focus information 121 (No in step S2), the tracking processing unit 82 proceeds to step S4 without masking the image 111.

[0084] In step S4, the tracking processing unit 82 extracts feature points from unmasked areas in the image 111. In step S5, the tracking processing unit 82 estimates the 3D spatial structure in the image and the movement of the imaging device 2, and then ends the tracking process.

[0085] [3.2. Second Example of Tracking Processing] In the tracking processing of the first example, the tracking processing unit 82 extracts feature points from areas other than the masked range, so that feature points are not extracted from the masked range.

[0086] In contrast, in the tracking process of the second example, feature points are extracted even in the masked area, but when estimating the 3D spatial structure and the movement of the image capture device 2, the weighting of the feature points in the masked area is reduced to reduce their contribution to these estimations.

[0087] Specifically, similarly to the first example, the tracking processing unit 82 refers to the focus information 121 and performs a masking process to mask the range including the subject detected as a moving object in the image.

[0088] 13 is a diagram illustrating weighting. The tracking processing unit 82 records the movement of the feature points extracted for each frame image in the time direction and performs geometric calculations to estimate the 3D spatial structure within the image and the movement of the image capture device 2.

[0089] At this time, as shown in Fig. 13, for example, a weighting coefficient of less than 1 is assigned to the masked area 301. More specifically, the closer to the center of the masked area 301, the smaller the weighting coefficient value assigned, and the farther from the center, the larger the weighting coefficient value assigned. This is because the closer to the center, the more likely it is to be a moving object. Note that a weighting coefficient of 1 is assigned to areas outside the masked area 301.

[0090] The tracking processing unit 82 then records the time-dependent movement of the feature points extracted for each frame image and, when performing geometric calculations, estimates the 3D spatial structure within the image and the movement of the imaging device 2, taking into account the weighting coefficients assigned to each feature point.

[0091] As described above, in the second example, moving objects are masked based on the focus information 121 obtained by the image capture device 2, and the weighting coefficients of feature points within the masked range 301 are reduced, thereby lowering the contribution of those feature points. This allows the tracking processing unit 82 to perform camera tracking processing with high accuracy. Furthermore, the computer 4 does not need to perform processing to distinguish between moving objects and non-moving objects, thereby shortening the processing time for camera tracking processing.

[0092] 14 is a flowchart showing the flow of the tracking process. As shown in FIG. 14, when the tracking process is started, the data acquisition unit 81 acquires the image file 100 from the image capture device 2 in step S11.

[0093] In the next step S12, the tracking processing unit 82 determines whether the acquired image file 100 includes focus information 121. If the image file 100 includes focus information 121 (Yes in step S12), the tracking processing unit 82 masks an area of ​​the image 111 that includes a subject detected as a moving object, based on the focus information 121. If the image file 100 does not include focus information 121 (No in step S12), the tracking processing unit 82 proceeds to step S14 without masking the image.

[0094] In step S14, the tracking processing unit 82 extracts feature points from the image 111. In step S15, the tracking processing unit 82 determines a weighting coefficient for the masked area in the image 111. At this time, the tracking processing unit 82 determines the weighting coefficient so that the closer to the center the smaller the weighting coefficient.

[0095] In step S16, the tracking processing unit 82 estimates the 3D spatial structure in the image and the movement of the image capturing device 2 using the extracted feature points and their weighting coefficients, and then ends the camera tracking process.

[0096] [3.3. Third Example of Tracking Processing] In the first and second examples, whether the subject is a moving object is determined by the imaging device 2. In contrast, in the third example, whether the subject is a moving object is determined by the computer 4.

[0097] Fig. 15 is a functional block diagram of the control unit 71 in Example 3. As shown in Fig. 17, the control unit 71 functions as a moving object determination unit 83 in addition to a data acquisition unit 81 and a tracking processing unit 82.

[0098] In the third example, the imaging device 2 detects a subject through autofocus processing, but does not determine whether the subject is a moving object. In this case, the body-side control unit 52 includes subject position information indicating the position of the subject in the focus information 121. The body-side control unit 52 also includes phase difference information for each phase difference detection pixel in the imaging element 55 in the focus information 121.

[0099] Then, when the data acquisition unit 81 acquires the image file 100, the moving object determination unit 83 determines whether the subject is a moving object based on the focus information 121, i.e., the subject position information and phase difference information, included in the image file 100. Note that the determination method is the same as that performed by the imaging device 2, and the processing after determining whether the subject is a moving object is the same as in the first or second example, and therefore a description thereof will be omitted.

[0100] As described above, in the third example, the computer 4 determines whether the subject is a moving object based on the focus information 121. In this case, the processing load is smaller than when classifying each subject as a moving object or a non-moving object from the image 111. Therefore, the processing time for the camera tracking process can be reduced in the third example as well.

[0101] 4. Modifications Note that the embodiment is not limited to the specific example described above, and various modified configurations may be adopted. For example, lens information may be used when determining whether a subject is moving. For example, by using the zoom position, it is possible to determine whether the entire image is moving or only a portion of the subject is moving.

[0102] In the above-described embodiment, after a predetermined subject is detected from an image, it is determined whether the subject is a moving object. However, since most subjects detected by autofocus processing can be moving objects, it is not necessary to determine whether a predetermined subject is a moving object after detecting it from an image. This allows all subjects detected from an image to be treated as moving objects. Even in this case, the accuracy of the camera tracking process can be improved because moving objects are not extracted as feature points.

[0103] In the above embodiment, an example has been described in which phase difference information is used as distance information of the subject, but the distance information of the subject is not limited to this as long as it is information from which the distance of the subject can be determined. For example, if the imaging device 2 has a ToF sensor, the measurement value obtained by the ToF sensor may be used as distance information.

[0104] In the above embodiment, the camera tracking process has been described as being divided into first to third examples. The computer 4 may be capable of executing all of the first to third examples of the camera tracking process, or may be capable of executing only some of them.

[0105] 5. Summary of the Embodiment As described above, the information processing device (computer 4) of the embodiment includes an acquisition unit (data acquisition unit 81) that acquires images obtained by the imaging device 2 and focus information 121 related to the focus when the images were captured, and a tracking processing unit 82 that performs camera tracking processing on the image 111 using the focus information 121. This allows the computer 4 to use the focus information not to extract feature points from a range that includes a moving object or to reduce the contribution of feature points in that range. In this case, the computer 4 does not need to classify the subject into a moving object and a torso. This allows the computer 4 to improve the accuracy of the tracking processing while reducing processing time.

[0106] The focus information 121 includes information about the position of the subject detected in the image 111, and the tracking processing unit 82 extracts feature points from an area in the image 111 excluding the area including the position of the subject indicated in the focus information 121, and detects the movement of the imaging device 2 based on the extracted feature points. This allows the computer 4 to use the focus information to avoid extracting feature points from an area including a moving object. In this way, the computer 4 can improve the accuracy of the tracking process while reducing the processing time.

[0107] The focus information 121 includes information about the position of a moving subject detected in the image 111. This eliminates the need for the computer 4 to classify the subject into moving and non-moving subjects, thereby reducing processing time.

[0108] The tracking processing unit 82 detects the movement of the imaging device by weighting the range including the position of the subject indicated in the focus information 121. This allows the computer 4 to reduce the contribution of feature points in the range including the moving object, thereby preventing a decrease in the accuracy of the tracking process. Furthermore, even if the area in the image 111 other than the range including the moving object is small and few feature points can be detected in that area, and the accuracy of the tracking process would decrease if feature points were not extracted in the range including the moving object, the number of extracted feature points is not reduced because feature points are extracted in the range including the moving object. This prevents a decrease in the accuracy of the tracking process.

[0109] The tracking processing unit assigns a lower weight to a location closer to the center in a range including the position of the subject indicated in the focus information 121. This allows the computer 4 to assign a lower weight to a location closer to the center in a range including the position of a moving subject, thereby reducing the degree of contribution, and further suppressing a decrease in the accuracy of the tracking process.

[0110] The focus information 121 includes information about the position of a predetermined type of subject. Most of the predetermined type of subject is a moving object. Therefore, the subject detected by the autofocus process in the imaging device 2 is likely to be a moving object. Therefore, in the camera tracking process, by using information about the position of the predetermined type of subject, it is possible to accurately mask the range including the moving object. Furthermore, since the imaging device 2 often performs autofocus processing and does not need to detect a moving object by a different process, the processing load on the imaging device 2 does not increase significantly.

[0111] The focus information includes information about the position of a subject specified by the user. The subject specified by the user is likely to be a moving object. Therefore, in the camera tracking process, by using information about the position of the subject specified by the user, it is possible to accurately mask the area including the moving object. Furthermore, the processing load on the imaging device 2 does not increase significantly.

[0112] The tracking information includes the position of the subject and distance information (phase difference information) of the subject, and the tracking processing unit 82 determines whether the subject is a moving object based on the distance information of the subject at the position of the subject, thereby further reducing the processing load on the imaging device 2.

[0113] As described above, the information processing method of the embodiment acquires an image obtained by the imaging device 2 and focus information related to the focus when the image was captured, and performs camera tracking processing on the image using the focus information. Also, as described above, the information processing system 1 of the embodiment includes the imaging device 2 and an information processing device, the imaging device 2 includes an imaging unit (image sensor 55) that captures a second image and a first image of lower quality than the second image, and the information processing device includes an acquisition unit that acquires the image obtained by the imaging device 2 and focus information related to the focus when the image was captured, and a tracking processing unit that performs camera tracking processing on the image using the focus information.

[0114] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0115] <6. Present Technology> The present technology may also have the following configuration. (1) An information processing device comprising: an acquisition unit that acquires an image obtained by an imaging device and focus information related to the focus when the image was captured; and a tracking processing unit that performs camera tracking processing on the image using the focus information. (2) The information processing device described in (1), in which the focus information includes information about the position of a subject detected in the image, and the tracking processing unit extracts feature points from an area in the image excluding a range including the position of the subject indicated in the focus information, and detects movement of the imaging device based on the extracted feature points. (3) The information processing device described in (1) or (2), in which the focus information includes information about the position of a subject that is a moving object detected in the image. (4) The information processing device described in (1) or (3), in which the tracking processing unit detects movement of the imaging device by weighting a range including the position of the subject indicated in the focus information. (5) The information processing device described in (4), in which the tracking processing unit assigns a lower weight to a range closer to the center of the range including the position of the subject indicated in the focus information. (6) The information processing device according to any one of (1) to (5), wherein the focus information includes information on the position of a subject of a predetermined type. (7) The information processing device according to any one of (1) to (6), wherein the focus information includes information on the position of a subject specified by a user. (8) The information processing device according to any one of (1) to (7), wherein the focus information includes the position of a subject and distance information of the subject, and wherein the information processing device is equipped with a moving object determination unit that determines whether the subject is a moving object based on the distance information of the subject at the position of the subject. (9) An information processing method, comprising: acquiring an image obtained by an imaging device and focus information related to the focus when the image was captured, and performing camera tracking processing on the image using the focus information.(10) An information processing system comprising an imaging device and an information processing device, wherein the imaging device comprises an imaging unit that captures an image of a subject, and the information processing device comprises: an acquisition unit that acquires an image obtained by the imaging device and focus information related to focus when the image was captured; and a tracking processing unit that performs camera tracking processing on the image using the focus information.

[0116] REFERENCE SIGNS LIST 1 Information processing system 2 Imaging device 4 Computer 17a Motion sensor 55 Imaging element 81 Data acquisition unit 82 Tracking processing unit 83 Feature point extraction unit

Claims

1. An information processing device comprising: an acquisition unit that acquires an image obtained by an imaging device and focus information regarding the focus when the image was captured; and a tracking processing unit that performs camera tracking processing on the image using the focus information.

2. The information processing device of claim 1, wherein the focus information includes information relating to the position of a subject detected in an image, and the tracking processing unit extracts feature points from an area excluding a range in the image that includes the position of the subject indicated in the focus information, and detects movement of the imaging device based on the extracted feature points.

3. The information processing device according to claim 1, wherein the focus information includes information relating to the position of a moving subject detected within an image.

4. The information processing device according to claim 1, wherein the tracking processing section detects the movement of the imaging device by weighting a range including the position of the subject indicated in the focus information.

5. The information processing device according to claim 4, wherein the tracking processing section reduces the weighting of an area closer to the center of a range including the position of the subject indicated in the focus information.

6. The information processing device according to claim 1, wherein the focus information includes information regarding the position of a subject of a predetermined type.

7. The information processing device according to claim 1, wherein the focus information includes information regarding a position of a subject designated by a user.

8. The information processing device according to claim 1, wherein the focus information includes a position of a subject and distance information of the subject, and further comprising a moving object determination unit that determines whether the subject is a moving object based on the distance information of the subject at the position of the subject.

9. An information processing method comprising: acquiring an image obtained by an imaging device and focus information relating to the focus when the image was captured; and performing camera tracking processing on the image using the focus information.

10. An information processing system comprising an imaging device and an information processing device, wherein the imaging device comprises an imaging unit that images a subject, and the information processing device comprises an acquisition unit that acquires an image obtained by the imaging device and focus information related to the focus when the image was captured, and a tracking processing unit that performs camera tracking processing on the image using the focus information.

Images