Imaging device and control method thereof

Abstract

To provide an imaging device and a control method thereof that realize in-camera VFX in a simple way.SOLUTION: An imaging device (1) outputs image data for detecting a position and an orientation of the imaging device to the outside, (2) images an external display device displaying an image for detecting the position and orientation of the imaging device, and (3) captures an image with a display device displaying an image generated according to the position and orientation of the imaging device detected on the basis of the image data obtained by the imaging as a background, in a state in which the display device is a background.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an imaging device and a control method thereof, and more particularly to a technique for estimating the position and orientation of an imaging device. [Background technology]

[0002] VFX (Visual Effects) technology, which creates unrealistic scenes by compositing live-action footage with computer graphics (CG), is attracting attention. In addition, a method is known in which a VFX image is obtained without compositing processing by capturing an image of a subject against a display device or screen that displays CG (in-camera VFX).

[0003] To achieve natural VFX, it is necessary to match the position and orientation of the virtual camera used to render the CG with the position and orientation of the camera capturing the live-action footage. Therefore, it is necessary to detect the position and orientation of the camera and change the CG according to the detected position and orientation.

[0004] Conventionally, the position and orientation of a camera have been detected using a device separate from the camera. For example, a camera tracking system is known that detects the position and orientation of a camera by capturing images of multiple markers placed at known positions with a device attached to the camera or receiving signals from multiple transmitters placed at known positions (Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 3631151 Summary of the Invention [Problem to be solved by the invention]

[0006] Camera tracking systems require the placement of markers or transmitters at known positions or the attachment of devices to the cameras, which increases the time and cost required for installation.

[0007] The present invention has been made in view of the above problems in the conventional technology. In one aspect, the present invention provides an imaging device that realizes in-camera VFX in a simple manner, and a control method thereof. [Means for solving the problem]

[0008] The above object is to provide an imaging device having a control means for controlling the operation of the imaging device, the control means including: (1) a control means for detecting a position and orientation of the imaging device; The (2) Output the data to the outside. Data-driven This is achieved by an imaging device, which is characterized by controlling the imaging device to (3) capture an image of an external display device displaying an image, and (4) capture an image against the background of the display device displaying an image generated in accordance with the position and orientation of the imaging device detected based on image data obtained by imaging. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device that realizes in-camera VFX in a simple manner and a control method thereof. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an imaging device 100 according to an embodiment. [Diagram 2] Schematic diagram of an imaging system according to an embodiment. [Diagram 3] FIG. 1 is a block diagram showing an example of a functional configuration of a display control device 300 according to an embodiment. [Figure 4] 1 is a flowchart showing the operation of the imaging system according to the first embodiment; [Diagram 5] 1 is a timing chart showing the operation of the imaging system according to the first embodiment; [Figure 6]FIG. 1 is a diagram showing an example of pattern data and an image capture result for position detection in an embodiment. [Figure 7] FIG. 1 is a diagram showing an example of a background image and a video imaging result in an embodiment. [Figure 8] FIG. 1 is a schematic diagram showing the movement of an imaging device 100. [Figure 9] FIG. 13 is a diagram showing an example of pattern data and an image capturing result for position detection obtained by the image capturing device 100 after movement in the embodiment. [Figure 10] FIG. 13 is a diagram showing changes in a background image and a video imaging result accompanying the movement of an imaging device; [Figure 11] 11 is a flowchart showing the operation of an imaging system according to a second embodiment. [Figure 12] 11 is a timing chart of the operation of the imaging system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0012] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an imaging device (digital video camera). However, the imaging function is not essential to the present invention, and the present invention can be implemented in any electronic device capable of processing image data. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] 1 is a block diagram showing an example of the configuration of an imaging system including an example of the functional configuration of an imaging device 100 that can embody the present invention. The imaging system 1 includes the imaging device 100, a display control device 300, and a display device 400.

[0014] The image capturing device 100 has a function of detecting its own position and orientation from a captured image, as will be described later. The image capturing device 100 outputs information on the detected position and orientation to the display control device 300.

[0015] The display control device 300 renders CG based on information on the position and orientation of the imaging device 100 obtained from the imaging device 100. The display control device 300 converts the obtained CG into a video signal suitable for the display device 400, and then outputs it to the display device 400. The display control device 300 has an interface capable of communicating with the imaging device 100 and the display device 400, and may be a computer device on which software for rendering CG and converting it into a video signal runs.

[0016] The display device 400 performs display based on a video signal supplied from the display control device 300. The display device 400 may be a flat panel display such as a liquid crystal display (LCD), or may be a combination of a projector and a screen. Also, multiple displays or projectors may be used.

[0017] In this embodiment, in order to synchronize the operations of the image capturing device 100 and the display device 400, the image capturing device 100 and the display device 400 are assumed to be compatible with input and output of a synchronization signal (for example, a GenLock signal).

[0018] The configuration of the imaging device 100 will be described. The lens unit 102 is an optical system that forms an optical image of a subject. The lens unit 102 is composed of multiple lenses including movable lenses and fixed lenses. The movable lenses include at least a focus lens, and may include one or more of a zoom lens and a shift lens.

[0019] The gyro 40 is a motion sensor that outputs a signal that indicates the motion of the imaging device 100. The gyro 40 is a three-axis angular velocity sensor that detects angular velocities around three axes, for example, an axis parallel to the optical axis of the lens unit 102, an axis parallel to the direction of gravity, and an axis perpendicular to these two axes. Note that a sensor that detects acceleration in each axial direction may also be used in combination.

[0020] A system control unit 50, which will be described later, detects the movement of the imaging device 100 from the output of the gyro 40. The system control unit 50 realizes an image blur correction function by controlling the position of the shift lens of the lens unit 102 and the position at which the image processing unit 24 trims the captured image according to the detected movement.

[0021] The aperture of the diaphragm 101 can be adjusted under the control of the system control unit 50. Light passing through the lens unit 102 passes through the opening of the diaphragm 101 and an ND (Neutral Density) filter 102 that is inserted in the optical path as necessary, and forms an optical image of a subject on the imaging plane of the imaging unit 22.

[0022] The imaging unit 22 has an imaging element and peripheral circuits for adjusting the gain of a signal read out from the imaging element and reducing noise. The imaging element may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element has a pixel array in which a plurality of pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from the pixels. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0023] The A / D converter 23 converts the analog image signal output from the imaging unit 22 into a digital image signal (image data). Note that the A / D converter 23 is not necessary when an imaging element having an A / D conversion function or a photon-counting type imaging element is used.

[0024] The image processing unit 24 applies predetermined image processing to the image data output by the A / D converter 23, generates signals and image data according to the application, and acquires and / or generates various information. The image processing unit 24 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 24 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executes software to realize a specific function.

[0025] The image processing applied by the image processing unit 24 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when an image sensor is provided with a color filter, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the imaging optical system 101 (image restoration), correction of the effects of peripheral light falloff of the imaging optical system 101, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. The data processing may include processes such as trimming, synthesis, scaling, encoding and decoding, and header information generation (data file generation). The generation of image data for display and image data for recording is also included in the data processing. In addition, an electronic image stabilization function is realized by moving the trimming area according to the movement of the imaging device 100 detected by the gyro 40. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are merely examples of processes that the image processing unit 24 can apply, and do not limit the processes that the image processing unit 24 can apply.

[0026] The image data output by the A / D converter 23 may be stored in the memory 32. The memory 32 is used, for example, as a buffer for image data or a working memory for the image processing unit 24. In addition, a part of the memory 32 is used as a video memory for the display unit 28.

[0027] The D / A converter 13 converts the image data for display stored in the video memory area of ​​the memory 32 into an analog image signal suitable for display on the display unit 28, and supplies the signal to the display unit 28. The display unit 28 has, for example, an LCD, and performs display according to the analog image signal supplied from the D / A converter 13.

[0028] When the imaging device 100 is in an imaging standby state, the system control unit 50 continuously controls the imaging unit 22 to capture moving images, the image processing unit 24 to generate image data for display and store it in the memory 32, and the D / A converter 13 to generate analog image signals. This allows the display unit 28 to function as an electronic viewfinder (EVF).

[0029] The non-volatile memory 56 is an electrically erasable and recordable memory, such as an EEPROM, and stores programs executed by the system control unit 50, constants, settings of the imaging device 100, GUI (Graphical User Interface) data, and the like.

[0030] The system control unit 50 is, for example, a CPU (also called an MPU or microprocessor). The system control unit 50 loads a program stored in a non-volatile memory 56 into a system memory 53 and executes the program, thereby controlling the operation of each unit of the imaging device 100 and realizing the functions of the imaging device 100.

[0031] The system memory 53 is a RAM, and is used to store the programs read from the non-volatile memory 56 and to temporarily store constants, variables, etc. required for the system control unit 50 to execute the programs. Note that the system memory 53 and the memory 32 may share a single memory space.

[0032] The system timer 56 supplies an operating clock signal to the system control unit 50, the imaging unit 22, the memory 32, the system memory 53, etc. The system timer 56 also has a clock function and provides date and time information. Furthermore, the system timer 56 also provides a timer function for measuring a predetermined time.

[0033] The operation unit 70 is a collective term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to the imaging device 100. The mode change switch 60 and the recording switch 61 also constitute the operation unit 70. The input devices constituting the operation unit 70 have names according to the functions assigned to them. Note that the functions assigned to the same input device may be variable.

[0034] For example, the operation unit 70 includes a menu button, direction keys, an enter key, and the like. When the menu button is operated, the system control unit 50 displays a menu screen on the display unit 28. When the direction keys are operated while the menu screen is displayed, the system control unit 50 changes the selected item or alters the display contents. When the enter key is operated, the system control unit 50 changes the settings or executes an operation according to the item selected at that time.

[0035] When the mode changeover switch 60 is operated, the system control unit 50 switches the operation mode of the imaging device 100 to one of a number of operation modes, such as a recording mode, a playback mode, or an in-camera VFX mode, depending on the operation of the mode changeover switch 60 .

[0036] The recording switch 61 switches between the start of moving image recording and a standby state. When the system control unit 50 detects an operation of the recording switch 61 in the imaging standby state, the system control unit 50 starts the moving image recording operation.

[0037] In addition, when the display unit 28 is a touch display, software keys using GUI images displayed on the display unit 28 also constitute the operation unit 70.

[0038] The power supply control unit 80 supplies each unit of the image capture device 100 with power required for operation based on power supplied from the power supply unit 30, which is, for example, a battery.

[0039] The medium I / F 18 is an interface that allows the system control unit 50 to access a recording medium 200 such as a memory card.

[0040] The external I / F 19 is an interface for outputting a signal to an external device and communicating with the external device. The external I / F 19 is provided with an interface and a connector that comply with standards such as SDI, HDMI (registered trademark), and USB. The external I / F 19 can also output a signal for synchronizing an external device with the imaging device 100, such as a time code or a GenLock signal.

[0041] The system control unit 50 can also control the operation of the imaging device 100 based on a signal or command input from an external device via the external I / F 19. When an external device is connected to the external I / F 19, the system control unit 50 performs predetermined communication with the external device to recognize the type of the external device and confirm the range of mutually controllable operations. The system control unit 50 also acquires information about the display device 400 connected to the display control device 300 via the display control device 300 connected to the external I / F 19.

[0042] The pattern data generating unit 33 generates pattern data to be displayed on the display device 400 in order to detect the position and orientation of the imaging device 100, and supplies the pattern data to the external I / F 19. The pattern data may be data representing the entire image to be displayed, or may be data representing a part of the image (e.g., a repeating unit). In the latter case, the display control device 300 processes the data into data representing the entire image, and then causes the display device 400 to display the data.

[0043] The position and orientation detection unit 34 detects the position and orientation of the imaging device 100 relative to the display device 400 based on an image captured of the display device 400 on which an image based on the pattern data generated by the pattern data generation unit 33 is displayed.

[0044] The display control device 300 is a general-purpose computer device that includes an interface for connecting the image capture device 100 and the display device 400, and executes the operations described below by running a display control application.

[0045] Fig. 2 is a diagram showing a schematic positional relationship between the imaging device 100, the display device 400, and the subject 500 in the imaging system 1. Fig. 2 shows an overhead view of the imaging device 100 and the display device 400 facing each other, with the subject 500 to be imaged being located between the display device 400 and the imaging device 100. The dotted line indicates the imaging range (angle of view) of the imaging device 100 in the horizontal direction.

[0046] 3 is a block diagram showing an example of a functional configuration of the display control device 300. The display control device 300 can be realized using, for example, a computer device. The control unit 301 is, for example, a CPU, and realizes the functions of the display control device 300 by loading a program (display control application program) stored in a ROM 305 into a RAM 306 and executing it.

[0047] The image processing circuit 302 is, for example, a graphics board equipped with a GPU, and is capable of performing image processing such as CG rendering at high speed.

[0048] The first I / F 303 and the second I / F 304 are communication interfaces for connecting external devices, and in this embodiment, the image capture device 100 is connected to the first I / F 303, and the display device 400 is connected to the second I / F 304. In this embodiment, the first I / F 303 and the second I / F 304 are capable of transmitting video signals, synchronization signals, and control signals. For convenience, the display control device 300 and the image capture device 1 1, the display control device 300 and the display device 400 are connected via a single I / F, but they may be connected via different I / Fs depending on the signals to be communicated. The same applies to the connection between the display control device 300 and the display device 400.

[0049] The control unit 301 acquires information about the imaging device 100 and the display device 400 through communication via the first I / F 303 and the second I / F 304, and controls the operations of the imaging device 100 and the display device 400. The display control device 300 may have three or more communication interfaces with external devices.

[0050] The ROM 305 stores programs (firmware, OS, application programs, etc.) executed by the control unit 301, GUI image data such as menu screens, setting values ​​for the display control device 300, and the like.

[0051] The RAM 306 is used as the main memory for the control unit 301 , as well as a working memory for the image processing circuit 125 and a video memory for the display unit 308 .

[0052] The storage unit 307 is a large-capacity storage device such as a hard disk, an SSD, etc. Data required for rendering CG to be displayed on the display device 400 is also stored in the storage unit 307.

[0053] The display unit 308 is, for example, a liquid crystal display device. The display unit 308 may be a touch display. The display unit 308 displays the screen of a display control application, etc.

[0054] The operation unit 309 has a plurality of input devices that can be operated by the user, such as a keyboard, a mouse, a touch pad, etc. When the display unit 308 is a touch display, the operation unit 309 includes a touch panel.

[0055] The clock generation circuit 310 generates a synchronization signal for synchronizing the operations of the imaging device 100 and the display device 400. The synchronization signal may be obtained from an external device.

[0056] Fig. 4 is a flowchart relating to the overall operation of the imaging system 1. Fig. 5 is a timing chart showing the operation timing of each device related to the flowchart of Fig. 4. The operation of the imaging system 1 will be explained below with reference to Figs. 4 and 5. Note that it is assumed that the installation and connections required for the operation have been completed before starting the following operations.

[0057] Specifically, it is assumed that a display control application is running in the display control device 300, and that the display control device 300 has acquired information required for controlling the operations of the imaging device 100 and the display device 400 from the imaging device 100 and the display device 400. For example, it is assumed that the display control device 300 has grasped the amount of delay from when an imaging instruction is given until imaging is performed, the amount of delay from when a display instruction is given until display is performed, and the like.

[0058] In the following description, operations mainly performed by the imaging device 100 are performed by the system control unit 50, and operations mainly performed by the display control device 300 are performed by the control unit 301.

[0059] In S301, the display control device 300 issues a mode change instruction to the in-camera VFX mode to the imaging device 100 and the display device 400 (FIG. 5, time t1). The in-camera VFX mode is an operation mode in which display control and imaging control related to detection of the position and orientation of the imaging device 100, and CG display control and imaging control according to the detected position and orientation are performed.

[0060] In the in-camera VFX mode, the imaging device 100 captures a video including frames for detecting the position and orientation and frames for recording. The imaging device 100 captures each frame in synchronization with a reference clock signal supplied from the display control device 300, in accordance with an instruction from the display control device 300.

[0061] Furthermore, in order to synchronize the operations of the imaging device 100 and the display device 400, the display control device 300 supplies a reference clock signal (GenLock signal) from a clock generation circuit 310 to the imaging device 100 and the display device 400. Furthermore, the display control device 300 confirms synchronization between the imaging device 100 and the display device 400 by receiving, from the imaging device 100 and the display device 400, completion of synchronization with respect to the GenLock signal. Furthermore, the display control device 300 notifies the imaging device 100 and the display device 400 of an operation cycle based on the GenLock signal.

[0062] Thereafter, upon receiving a video signal from the image sensing device 100, the display control device 300 outputs an image based on the video signal to the display device 400, and instructs the image sensing device 100 to generate pattern data for position and orientation detection.

[0063] In S302, the pattern data generating unit 33 of the imaging device 100 generates pattern data for position and orientation detection in response to an instruction from the display control device 300. The pattern data is image data having characteristics that enable identification of which part of the display screen is imaged by the imaging device 100 from a captured image of the display screen displaying an image based on the pattern data.

[0064] Fig. 6(a) shows an example of an image represented by pattern data. Fig. 6(a) shows a state in which pattern data is displayed over the entire display device 400. The pattern data is generated by the pattern data generating section 33 of the imaging device 100 based on information of the display device 400 (e.g., the resolution of the display screen (the number of pixels in the horizontal and vertical directions)).

[0065] For example, the imaging device 100 divides the display screen of the display device 400 into a plurality of blocks, and generates pattern data that displays unique information in each block. From the unique information of the blocks included in an image captured of a portion of the display screen of the display device 400 displaying the pattern data, it is possible to identify which range of the display screen has been captured by the imaging device 100.

[0066] In the example shown in FIG. 6(a), an example of pattern data is shown in which the unique information of each block is displayed as characters "(x, y)" (integers 1≦x≦22, 1≦y≦10) indicating the horizontal and vertical positions of the block. Therefore, by identifying the characters "(x, y)" included in an image capturing a portion of the display screen, for example, by the image processing unit 24, it is possible to identify which portion of the display screen is being captured. Note that the unique information displayed on each block is not limited to characters, and may be a figure, color, or the like, as long as it can be identified by the imaging device 100.

[0067] Meanwhile, information on the three-dimensional positions of multiple points on the display screen of the display device 400 (for example, the three-dimensional positions of at least three of the four vertices of the display screen) is provided from the display control device 300 to the imaging device 100. Based on the information on the three-dimensional positions of the display screen provided from the display control device 300, the imaging device 100 can identify the three-dimensional position of the captured image on the screen displaying the pattern data.

[0068] In S303, the imaging device 100 outputs the pattern data generated in S302 to the display control device 300 via the external I / F 19. The display control device 300 stores the pattern data received via the first I / F 303 in the RAM 306. Then, the image processing circuit 302 generates image data to be output to the display device 400 based on the pattern data. Note that the pattern data may be generated so that it is displayed only in the imaging range of the previous frame.

[0069] The display control device 300 outputs the image data generated by the image processing circuit 302 together with an output instruction to the display device via the second I / F 304. In addition, the display control device 300 outputs an image capture instruction to the image capture device 100.

[0070] In S304, the imaging device 100 and the display device 400 wait until the execution timing (here, time t3) of the imaging process and the display process, respectively. The execution timing is determined by the period notified by the display control device 300 during the synchronization process and the GenLock signal.

[0071] When the execution timing arrives, in S305, the display device 400 performs display based on the image data, and the imaging device 100 performs imaging. Here, it is assumed that the display device 400 displays the image shown in FIG. 6(a), and the imaging device 100 captures the range shown in the frame in FIG. 6(b). In addition to the image based on the pattern data, the captured image includes a person present between the display device 400 and the imaging device 100. The imaging device 100 captures an image by focusing the lens unit 102 on the display screen under predetermined exposure conditions, and generates captured image data by the image processing circuit 125. The captured image data is stored in the memory 32. The exposure time (shutter speed) during imaging is determined so that flicker caused by the display frame rate of the display device 400, the power supply frequency, etc. does not occur in the captured image.

[0072] In S306, the position and orientation detection unit 34 of the imaging device 100 detects the position (spatial coordinates) and orientation of the imaging device 100 from the captured image data and three-dimensional position information of the display screen of the display device 400. First, the position and orientation detection unit 34 specifies which part of the display screen the imaging device 100 has captured (imaging range) based on the captured image data.

[0073] Thereafter, the position and orientation detection unit 34 detects the position and orientation of the imaging device 100 using the three-dimensional positions of six or more points (coordinates) in the captured image. The three-dimensional positions corresponding to the points in the image can be obtained from information on the three-dimensional positions of the display screen provided by the display control device 300. For example, the position and orientation detection unit 34 specifies the three-dimensional positions of six or more points included in an image based on pattern data that appears in the captured image based on information on the three-dimensional positions of the vertices of the display screen, or obtains the positions from the display control device 300. The three-dimensional positions may be, for example, the centers (centers of gravity) of blocks or the three-dimensional positions of the intersections of lattices that separate the blocks.

[0074] For example, if the three-dimensional positions of six points (coordinates) in a captured image can be specified, a known method for solving a P6P (Perspective-6-Point) problem can be used to detect the position and orientation of the image capture device 100. The position and orientation detection unit 34 stores the detected three-dimensional position and orientation (for example, the rotation angle of each axis of a Cartesian coordinate system relative to the direction of gravity) of the image capture device 100 in the memory 32 as position and orientation information.

[0075] In S307, the imaging device 100 medium The position and orientation information is output to the display control device 300 via the I / F 18 .

[0076] In S308, the display control device 300 stores the position and orientation information received through the first I / F 303 in the RAM 306. Then, the image processing circuit 302 uses the viewpoint position and line of sight direction based on the position and orientation information to render the model data stored in advance in the storage unit 307 to generate CG data. This provides CG data (image data for video) corresponding to an image observed at the position of the display screen of the display device 400 when the imaging device 100 is placed in a virtual space. That is, CG data that represents a natural image according to the position and orientation of the imaging device 100 is generated. FIG. 7(a) shows an example of an image represented by CG data rendered according to the current position and orientation of the imaging device 100. This image is used as a background, and is therefore called a background image.

[0077] In S309, the display control device 300 outputs background image data and an output instruction to the display device 400 via the second I / F 304. In addition, the display control device 300 outputs an image capture instruction to the image capture device 100 via the first I / F 303.

[0078] In S310, the imaging device 100 and the display device 400 wait until the timing (here, time t4) to execute the imaging process and the display process, respectively.

[0079] When the execution timing arrives, in S311, the display device 400 performs display based on the image data, and the imaging device 100 performs imaging. Here, it is assumed that the display device 400 displays the image shown in Fig. 7(a), and the imaging device 100 captures the range shown in the frame in Fig. 6(b). In addition to the image based on the background image data, the captured image includes a person present between the display device 400 and the imaging device 100. As a result, the image shown in the frame in Fig. 7(b) is captured.

[0080] In step S312, the imaging device 100 captures the image using the in-camera V The FX image is stored in the recording medium 200 . If the termination condition is not satisfied in S313, the process continues for the next frame by repeatedly executing from S302. The termination condition may be any predetermined condition, such as the imaging device 100 switching to a playback mode or the imaging device 100 being powered off.

[0081] In processing the next frame, it is assumed that the imaging device 100 has moved from the position indicated by the dotted line in Fig. 8 to the position indicated by the solid line. Fig. 9(a) shows a state in which the display device 400 displays an image based on the pattern data in S305. Here, it is assumed that the pattern data generated in S302 is not changed regardless of the movement of the imaging device 100, but the pattern data may be changed depending on the relative positional relationship between the display screen of the display device 400 and the imaging device 100. Fig. 9(b) also shows a schematic diagram of the captured image (within the frame) obtained by the imaging device 100 in S305 and the state of the display screen.

[0082] In S306, the position and orientation detection unit 34 of the image capturing device 100 detects the position and orientation of the image capturing device 100 as described above. In S307, the image sensing device 100 outputs the position and orientation information to the display control device 300. In S308, the display control device 300 generates CG data of a background image based on the position and orientation information. As the imaging direction of the imaging device 100 changes, the background image changes as shown in Fig. 10(a). Then, in S311 (time t7 in Fig. 5), a captured image shown in the frame in Fig. 10(b) is obtained.

[0083] In the above example, the pattern data is displayed and the position and orientation of the image capture device is detected for each frame, but the position and orientation may be detected once for multiple frames. In this case, in frames in which the position and orientation detection is not performed, CG based on the most recently detected position and orientation is repeatedly displayed.

[0084] Furthermore, detection of the position and orientation may be performed at a frequency according to the magnitude of the movement of the image capture device 100 detected by the gyroscope 40. When the image capture device 100 is stationary or the magnitude of the movement is less than a threshold, the frequency of detection of the position and orientation can be lower than when the movement is equal to or greater than the threshold.

[0085] In this way, according to this embodiment, the position and orientation of the imaging device is detected from a captured image obtained with a pattern image for detecting the position and orientation of the imaging device displayed in the background, and CG generated based on the detected position and orientation is displayed in the background. Therefore, it becomes possible to directly obtain a VFX video with a CG background corresponding to the position and orientation of the imaging device by the imaging device without using an external device for detecting the position and orientation of the imaging device.

[0086] <Second embodiment> Next, a second embodiment of the present invention will be described. In this embodiment, the display control device 300 detects the position and orientation of the imaging device 100. Therefore, the imaging device 100 outputs captured image data obtained in a state in which an image based on pattern data is displayed to the display control device 300. The display control device 300 detects the position and orientation of the imaging device 100 in the same manner as the position and orientation detection unit 34 in the first embodiment. Generation of CG based on the position and orientation of the imaging device 100 and timing control of display and imaging are similar to the display control device 300 in the first embodiment.

[0087] Fig. 11 is a flowchart relating to the overall operation of the imaging system 1, similar to Fig. 4. Fig. 12 is a timing chart showing the operation timing of each device relating to the flowchart of Fig. 11, similar to Fig. 5. Hereinafter, this embodiment will be described with reference to Figs. 11 and 12. The configuration of the imaging system 1 may be the same as in the first embodiment, so a description thereof will be omitted.

[0088] Since steps S301 to S305 are the same as those in the first embodiment, the description thereof will be omitted. I The position and orientation detection unit 34 of the image sensing device 100 is not used in this embodiment.

[0089] In step S1007, the display control device 300 3 The image data of the image displaying the pattern data received from the imaging device 100 through the display control unit 301 is stored in the RAM 306. The display control device 300 (control unit 301) then detects the position and orientation of the imaging device 100 in the same manner as the position and orientation detection unit 34 in the first embodiment, and stores the position and orientation information in the RAM 306 (FIG. 12, 1202).

[0090] In S308, the display control device 300 generates a background image (CG) using the position and orientation information calculated in S1007 (FIG. 12, 1203) without receiving the position and orientation information from the imaging device 100. The subsequent processing is similar to that in the first embodiment, and therefore will not be described.

[0091] In this embodiment, the position and orientation of the image capture device 100 is detected by the display control device 300. Therefore, in addition to the same effects as in the first embodiment, it is possible to reduce the processing load on the image capture device 100 and to increase the speed of position and orientation detection.

[0092] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0093] 100: imaging device, 300: display control device, 400: display device, 22: imaging section, 50: system control section, 34: position and orientation detection section

Claims

1. 1. An imaging device, comprising: A control unit for controlling an operation of the imaging device, The control means (1) outputting data for detecting the position and orientation of the imaging device to an external device; (2) capturing an image of an external display device that displays an image based on the data; (3) capturing an image against the background of the display device displaying an image generated in accordance with the position and orientation of the imaging device detected based on image data obtained by the imaging; The imaging device is controlled as follows.

2. The imaging device according to claim 1 , wherein the imaging is performed under the control of an external device different from the display device.

3. The control means further comprises: Detecting a position and orientation of the imaging device based on the image data obtained by capturing an image of an external display device that displays an image based on the data and on three-dimensional positions of a plurality of positions of the display device; outputting the detected position and orientation information to the external device; 3. The imaging device according to claim 2, wherein the imaging device is controlled so as to:

4. The imaging device according to claim 3 , wherein the three-dimensional position is supplied from the external device.

5. The control means further comprises: The imaging device according to claim 2, characterized in that the imaging device is controlled so as to output the image data obtained by capturing an image of an external display device displaying an image based on the data, so that the external device can detect the position and orientation of the imaging device.

6. the imaging device captures an image of a portion of a display screen of the display device; 6. The imaging device according to claim 1, wherein the image based on the data is an image that enables identification of which part of the display screen the imaging device has captured from the image data acquired by capturing an image of the display device.

7. 7. The imaging device according to claim 1, wherein the image generated in accordance with the position and orientation of the imaging device is computer graphics (CG) representing a virtual space observed from the imaging device.

8. 8. The imaging device according to claim 1, wherein the control means controls the imaging device so as to repeatedly execute the operations (1) to (3).

9. 9. The imaging device according to claim 8, wherein the control means reduces the frequency of the operations (1) and (2) when the magnitude of the movement of the imaging device is less than a threshold value, compared to when the magnitude is equal to or greater than the threshold value.

10. A control method executed by an imaging device, comprising: (1) outputting data for detecting a position and orientation of the imaging device to an external device; (2) capturing an image of an external display device displaying an image based on the data; and (3) capturing an image against a background of the display device that displays an image generated in accordance with the position and orientation of the imaging device detected based on image data obtained by the imaging; and 13. A method for controlling an imaging apparatus, comprising:

11. A program for causing a computer included in an imaging apparatus to function as the control unit of the imaging apparatus according to any one of claims 1 to 9.

Images