Imaging system, method for controlling the imaging system, and program

Abstract

This method suppresses positional misalignment between the captured image and the background image when capturing images with an image displayed on a display device as the background, and when generating background images that complement the captured image. [Solution] The imaging system comprises a first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing, and a second generation means for generating a second background image corresponding to the first background image. The imaging device performs imaging at a second timing when the first background image is displayed on a display device, and the second generation means generates a second background image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

Description

[Technical Field]

[0001] The present invention relates to an imaging system, a method for controlling the imaging system, and a program. [Background technology]

[0002] To obtain an image of a subject with a computer graphics (CG) image as the background, there is a technique to acquire an image with a CG image as the background by photographing the subject with the image displayed on a display device as the background, without compositing the background image and the subject image. Patent Document 1 discloses a method for directly acquiring VFX (Visual Effects) video by outputting a background image rendered according to the position and orientation of the imaging device to the display device and then performing imaging with the display device as the background.

[0003] Furthermore, when capturing a subject with an image displayed on a display device as the background, the captured image may need to be interpolated due to factors such as the background image being interrupted because the captured image includes an area outside the display area of ​​the display device. In this case, there are techniques to generate a background image to interpolate the captured image, such as a composite image for the captured image, separate from the captured image. Patent Document 2 discloses the synthesis of a CG image, such as a background image, into the area of ​​the captured image corresponding to the area outside the display area of ​​the display device. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-118468 [Patent Document 2] Japanese Patent Publication No. 2024-35420 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, with conventional technology, there is a risk that positional misalignment between the captured image and the background image may occur due to factors such as changes in the position and orientation of the imaging device over time. The present invention aims to suppress positional misalignment between an image and a background image when capturing an image with an image displayed on a display device as the background, and when generating a background image that complements the captured image. [Means for solving the problem]

[0006] To solve the above problems, the present invention provides an imaging system that performs imaging with an image displayed on a display device as the background, comprising: a first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing; a display control means for causing the display device to display the first background image; and a second generation means for generating a second background image corresponding to the first background image, wherein the imaging device performs imaging at a second timing when the first background image is displayed on the display device, and the second generation means generates the second background image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. [Effects of the Invention]

[0007] According to the present invention, it is possible to suppress positional misalignment between the captured image and the background image when capturing an image with an image displayed on a display device as the background and generating a background image that complements the captured image. [Brief explanation of the drawing]

[0008] [Figure 1] This is an overall diagram of the imaging system. [Figure 2] This figure shows an example of the camera's hardware and functional configuration. [Figure 3] This figure shows an example of the hardware and functional configuration of a system control unit. [Figure 4] This figure shows an example of the hardware and functional configuration of a composite image generation device. [Figure 5] This sequence diagram shows the flow from when the system control unit notifies the start of the processing necessary for capturing in-camera VFX footage, until all the information necessary for generating the background image for compositing is available. [Figure 6] This is a flowchart illustrating the image processing flow. [Figure 7] This figure shows the relationship between the camera, whose position and orientation are as shown in the position and orientation information above and whose focal length is as shown in the camera parameters above, and the image display surface of the display device. [Figure 8] This is a flowchart illustrating the synthesis process. [Figure 9] (A) and (B) are diagrams showing the image display plane of the imaging system in the world coordinate system. [Figure 10] This figure shows the alpha blending process performed on the captured image corresponding to one frame of the in-camera VFX footage, between the captured image and the background image used for compositing. [Figure 11] (A) is a diagram showing the relationship between the captured image and the composite region as a modified example 1, and (B) is a magnified view of the region in the captured image. [Figure 12] This is a flowchart illustrating the synthesis process as a variation 2. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the accompanying drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] Figure 1 is an overall diagram of the imaging system 100. The imaging system 100 of this embodiment is a system that uses a CG image as a background to image a subject as a virtual studio. Specifically, the imaging system 100 displays an image on the display device as a background corresponding to the position and orientation of the camera, and captures the subject located in the foreground of the display device and the display device within the field of view, thereby capturing so-called in-camera VFX images. Furthermore, the imaging system 100 generates an image to complement the captured image, which is an in-camera VFX image, and composites the generated image with the captured image. Examples of images used to complement the captured image include, for example, a background image to complement the background image in the captured image when the captured image includes an area outside the image display area of ​​the display device, resulting in a break in the background image in the captured image.

[0011] The imaging system 100 includes a camera 101, a position and orientation detection device 102, a display device 103, a system control device 104, a background image generation device 105, a display control device 106, and a composite image generation device 107. The camera 101, the position and orientation detection device 102, the display device 103, the system control device 104, the background image generation device 105, the display control device 106, and the composite image generation device 107 are connected via a network 200.

[0012] Camera 101, as an example of an imaging device, generates an image by capturing data. Camera 101 transmits the image captured with the display device 103 as the background to the composite image generation device 107. The position and orientation detection device 102 is attached to the camera 101 and detects the position and orientation of the camera 101. The position and orientation detection device 102 transmits information indicating the detected position and orientation of the camera 101 to the system control device 104. The information indicating the position and orientation of the camera 101 detected by the position and orientation detection device 102 may be referred to as position and orientation information below. Position and orientation information may include, for example, information indicating the amount of rotation and translational movement (6DoF) with respect to the x, y, and z axes of the world coordinate system. However, the position and orientation information may be any information that enables conversion between the world coordinate system and the camera coordinate system. The position and orientation detection device 102 may, for example, have a camera (not shown) and detect the position and orientation of the camera 101 from the positional relationship between the marker and the camera 101 in the captured image obtained by having this camera capture a marker indicated at a fixed position. However, any method may be used by the position and orientation detection device 102 to detect the position and orientation of the camera 101.

[0013] The display device 103 is a large image display device such as an LED wall. The display device 103 may be composed of multiple display panels. The system control unit 104 controls the camera 101, the position and orientation detection device 102, the background image generation device 105, and the display control device 106. The system control unit 104 also transmits a synchronization signal to each of the controlled devices and achieves synchronization of the operation of each device by methods such as genlock (Generator Locking), which controls the operation timing of each device to be in accordance with the reference clock included in the synchronization signal. Examples of synchronization of the operation of each device include the synchronization of the exposure timing of the camera 101 and the image display timing of the display device 103. The system control unit 104 also controls the start and end of image capture of in-camera VFX video. Furthermore, the system control unit 104 transmits position and orientation information to the background image generation device 105.

[0014] As an example of the first generation means, the background image generation device 105 renders a pre-set 3D model of a virtual space according to the position and orientation of the camera 101, and generates a CG image as a background image at a predetermined frame rate. The background image generated by the background image generation device 105 is an image used for display on the display device 103. Therefore, the background image generated by the background image generation device 105 as an image used for display on the display device 103 may be referred to as the display background image below. The camera in the virtual space may be referred to as the virtual camera below. Furthermore, if the imaging direction of the camera 101 is not directly facing the display device 103, the background image generation device 105 applies the coordinate transformation (deformation processing) necessary to display the display background image on the display device 103. The background image generation device 105 also transmits the generated display background image to the display control device 106. The background image shown is an example of the first background image.

[0015] As an example of a display control means, the display control device 106 displays the background image for display, generated by the background image generation device 105, on the display device 103 in accordance with the imaging timing of the camera 101. If the display device 103 is composed of multiple display panels, the display control device 106 divides the background image for display to match each display panel that makes up the display device 103 before displaying it.

[0016] As an example of a second generation means, the composite image generation device 107 generates an image to complement the in-camera VFX video, in other words, a background image to be composited with the in-camera VFX video, and composites the generated background image with the in-camera VFX video. The background image generated by the composite image generation device 107 for compositing with the in-camera VFX video may be referred to as the composite background image below. The composite image generation device 107 generates the composite background image after the capture of the in-camera VFX video is completed. The composite background image is an example of a second background image. The composite background image can also be considered as an image corresponding to the display background image. The composite image generation device 107, like the background image generation device 105, renders a pre-set 3D model of a virtual space according to a predetermined position and orientation. Furthermore, the composite image generation device 107 generates a background image for compositing by applying various image processing, including perspective projection transformation, to the rendered image. In addition, the composite image generation device 107 composites the background image for compositing to a region of the captured image, which is the target of interpolation, as part of the in-camera VFX video. The image obtained by compositing the background image for compositing to the captured image may hereafter be referred to as a composite image.

[0017] The system control unit 104, background image generation device 105, display control device 106, and composite image generation device 107 are, for example, computers. The system control unit 104, background image generation device 105, display control device 106, and composite image generation device 107 may be configured by a single computer or implemented by distributed processing by multiple computers. Furthermore, if the system control unit 104, background image generation device 105, display control device 106, and composite image generation device 107 are implemented by distributed processing by multiple computers, the combination of devices for each computer may be any combination.

[0018] Network 200 can be implemented, for example, by a LAN (Local Area Network), WAN (Wide Area Network), etc., such as the Internet. Network 200 may also be implemented by any or a combination of telephone lines, dedicated digital lines, ATM (Asynchronous Transfer Mode), frame relay lines, cable television lines, wireless lines for data broadcasting, etc.

[0019] Figure 2 shows an example of the hardware and functional configuration of camera 101. The camera 101 includes a control unit 201, ROM 202, RAM 203, optical system 204, imaging unit 205, A / D conversion unit 206, image processing unit 207, recording unit 208, communication unit 209, display unit 210, and instruction input unit 211. The control unit 201, ROM 202, RAM 203, optical system 204, imaging unit 205, A / D conversion unit 206, image processing unit 207, recording unit 208, communication unit 209, display unit 210, and instruction input unit 211 are each connected via a bus.

[0020] The control unit 201 is, for example, a CPU and controls the entire camera 101. More specifically, the control unit 201 controls the operation of each functional part of the camera 101 by reading a control program from the ROM 202, loading it into the RAM 203, and executing it. The control unit 201 also synchronizes the operation of the camera 101 with the operation of an external device by controlling the operation of each functional part of the camera 101 based on a synchronization signal supplied via the communication unit 209. Note that the camera 101 may use a GPU as the control unit 201, in addition to a CPU. ROM202 is a non-volatile memory that can be electrically erased or recorded, and stores the operation programs for each functional unit in the camera 101, as well as parameters necessary for the operation of each functional unit in the camera 101. RAM203 is a rewritable volatile memory used for the deployment of programs executed by the control unit 201 and for the temporary storage of data generated by the operation of each functional unit in the camera 101. The optical system 204 consists of a group of lenses including a zoom lens and a focus lens, and forms an image of the subject on the imaging plane of the imaging unit 205. The imaging unit 205 is an image sensor such as a CCD sensor or a CMOS sensor, and the optical system 204 converts the optical image formed on the imaging surface of the imaging unit 205 into an analog image signal, which is then output to the A / D conversion unit 206. The A / D conversion unit 206 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 206 is temporarily stored in the RAM 203. The image processing unit 207 performs various image processing on the images stored in the RAM 203. Examples of image processing performed by the image processing unit 207 include demosaicing, white balance correction, and gamma correction, which are necessary for image development, display, or recording. The image processing unit 207 also performs processing necessary for improving image quality, such as spatial filtering and noise suppression through the synthesis of multiple images. As an example of a recording means, the recording unit 208 records information such as images on its built-in recording medium. The communication unit 209 connects to external devices via wired or wireless connection to transmit images and receive synchronization signals. The display unit 210 includes a display device such as an LCD and displays images stored in the RAM 203 and images stored in the recording unit 208 on the display device. The display unit 210 also displays a user interface for receiving instructions from the user. The instruction input unit 211 is an interface for receiving instructions from the user. The instruction input unit 211 also includes physical operating elements such as a touch panel and a shutter button.

[0021] Figure 3 shows an example of the hardware and functional configuration of the system control device 104. The system control device 104 includes a control unit 301, a first interface 302, a second interface 303, a third interface 304, a fourth interface 305, a ROM 306, a RAM 307, an instruction input unit 308, and a clock generation unit 309. The control unit 301, the first interface 302, the second interface 303, the third interface 304, the fourth interface 305, the ROM 306, the RAM 307, the instruction input unit 308, and the clock generation unit 309 are each connected via a bus. The control unit 301 is, for example, a CPU and controls the entire system control device 104. More specifically, the control unit 301 controls the operation of each functional part of the system control device 104 by reading a control program from the ROM 306, loading it into the RAM 307, and executing it. The control unit 301 also controls the operating timing of the system control device 104 according to the synchronization signal supplied from the clock generation unit 309.

[0022] The first interface, I / F302, is used for communication with the camera 101. The second interface, I / F303, is an interface used for communication with the position and attitude detection device 102. The third interface, I / F304, is used for communication with the background image generation device 105. The fourth interface, I / F305, is an interface used for communication with the display control device 106. The first I / F302, second I / F303, third I / F304, and fourth I / F305 communicate according to standards corresponding to the external device they are connected to and the type of signals they transmit and receive. Furthermore, multiple interfaces may be used for communication between the system control unit 104 and the external device.

[0023] ROM 306 stores programs executed by the control unit 301 and settings of the system control device 104. Examples of programs executed by the control unit 301 include the BIOS, bootstrap loader, and firmware. RAM307 is a rewritable volatile memory used for purposes such as deploying programs executed by the control unit 301 and temporarily storing information generated by the operation of various functional units in the system control device 104. The instruction input unit 308 is an input device for receiving instructions from the user. Examples of the instruction input unit 308 include a keyboard, mouse, touchpad, etc. The clock generation unit 309 generates a synchronization signal (clock) to synchronize the operation of the system control device 104, the camera 101, the position and orientation detection device 102, the background image generation device 105, and the display control device 106.

[0024] Figure 4 is a block diagram showing an example of the hardware and functional configuration of the composite image generation device 107. The composite image generation device 107 includes a control unit 401, a ROM 402, a RAM 403, an image processing unit 404, a recording unit 405, a communication unit 406, a display unit 407, and an instruction input unit 408. The control unit 401, ROM 402, RAM 403, image processing unit 404, recording unit 405, communication unit 406, display unit 407, and instruction input unit 408 are each connected via a bus. The control unit 401 is, for example, a CPU and controls the entire composite image generation device 107. More specifically, the control unit 401 reads a control program from the ROM 402, loads it into the RAM 403, and executes it, thereby controlling the operation of each functional part of the composite image generation device 107. Note that the composite image generation device 107 may use a GPU as the control unit 401, in addition to a CPU. ROM 402 stores programs executed by the control unit 401, setting values ​​for the composite image generation device 107, and so on. Examples of programs executed by the control unit 401 include the BIOS, bootstrap loader, and firmware. RAM 403 is a rewritable volatile memory used for deploying programs executed by the control unit 401 and for temporarily storing information generated by the operation of each functional unit in the composite image generation device 107. RAM 403 also stores a 3D model of a virtual space. The image processing unit 404 renders the 3D model of the virtual space stored in the RAM 403. The image processing unit 404 also performs various image processing on the image stored in the RAM 403. Examples of image processing performed by the image processing unit 404 include image correction processing for compositing with in-camera VFX footage, such as perspective projection transformation processing. Other examples of image processing performed by the image processing unit 404 include compositing of multiple images. The recording unit 405 records information, including images, on its built-in recording medium. The communications unit 406 connects to external devices via wired or wireless connections to send and receive information such as images. The display unit 407 includes a display device such as an LCD and displays images stored in the RAM 403 and the recording unit 405 on the display device. The display unit 407 also displays a user interface for receiving instructions from the user. The instruction input unit 408 is an input device for receiving instructions from the user. Examples of the instruction input unit 408 include a keyboard, mouse, touchpad, etc.

[0025] Figure 5 is a sequence diagram showing the flow from when the system control device 104 notifies the start of processing necessary for capturing in-camera VFX images, until all the information necessary for generating a background image for compositing is available. The devices to which the system control device 104 notifies the start of processing are the camera 101, the position and orientation detection device 102, the background image generation device 105, and the display control device 106. In addition to notifying the start of processing, the system control device 104 also controls the operating timing of each notified device by sending a synchronization signal to each notified device.

[0026] As shown in Figure 5, at time point 1, the position and orientation detection device 102 detects the position and orientation of the camera 101 and generates position and orientation information according to the detection result. As will be described in detail later, the position and orientation detection device 102 detects the position and orientation of the camera 101 again at a time later than time point 1 and generates position and orientation information again according to the detection result. For this reason, the position and orientation information generated by the detection at time point 1 may be referred to as the previous position and orientation information below. Furthermore, at time point 1, camera 101 detects its parameters. These camera parameters may be referred to as camera parameters below. Examples of camera parameters detected at time point 1 include parameters related to the optical characteristics of camera 101. For example, the focal length of camera 101 is one such parameter. As will be described in detail later, camera 101 detects its parameters again at a later time than time point 1. Therefore, the camera parameters detected at time point 1 may be referred to as the aforementioned camera parameters below. Camera 101 stores these aforementioned camera parameters in RAM 203. Thus, in this embodiment, the timing of the camera's position and orientation detection by the position and orientation detection device 102 and the timing of the camera parameter detection by the camera 101 coincide at time 1. Note that time point 1 is an example of the first timing.

[0027] Next, at time point 2, which is later than time point 1, the control unit 201 of the camera 101 transmits the camera parameters detected at time point 1 to the system control unit 104. The RAM 307 of the system control unit 104 stores the received camera parameters. Next, at time point 3, which is later than time point 2, the position and attitude detection device 102 transmits the position and attitude information generated by the detection at time point 1 to the camera 101 and the system control device 104. The RAM 203 of the camera 101 and the RAM 307 of the system control device 104 store the received position and attitude information. Furthermore, the timing of when the camera transmits the camera parameters and the timing of when the position and attitude detection device 102 transmits the position and attitude information may be reversed or the same.

[0028] Next, at time 4, which is later than time 3, the control unit 301 of the system control device 104 transmits the camera parameters received at time 2 and the position and orientation information received at time 3 to the background image generation device 105. The background image generation device 105 stores the received camera parameters and position and orientation information. Alternatively, camera 101 may transmit the aforementioned camera parameters to background image generation device 105 instead of system control device 104. Also, position and orientation detection device 102 may transmit the aforementioned position and orientation information to background image generation device 105 instead of system control device 104.

[0029] Next, at time point 5, which is later than time point 4, the background image generation device 105 starts generating a background image for display based on the camera parameters and position / orientation information received at time point 4. The method by which the background image generation device 105 generates the background image for display will be described in detail later. Next, at time 6, which is later than time 5, the background image generation device 105 transmits the display background image, which it has generated by starting generation from time 5, to the display control device 106.

[0030] Next, at time 7, which is later than time 6, the display control device 106 outputs the background image for display received from the background image generation device 105 at time 6 to the display device 103, thereby starting the display on the display device 103. Furthermore, at time 7, camera 101 begins to capture images of the subject including the display device 103. In other words, camera 101 begins to capture images of the subject including the display background image shown on the display device 103. Furthermore, at time 7, the position and orientation detection device 102 detects the position and orientation of the camera 101 and generates position and orientation information according to the detection result. The position and orientation detection information generated by the detection at time 7 is position and orientation information generated by a detection at a later timing than the earlier position and orientation information. For this reason, the position and orientation information generated by the detection at time 7 may be referred to as the later position and orientation information below. Furthermore, at time 7, camera 101 detects camera parameters. The camera parameters detected at time 7 are camera parameters detected at a later time than the earlier camera parameters. Therefore, the camera parameters detected at time 7 may be referred to as the later camera parameters below. Also, the earlier and later camera parameters are parameters of the same type. Camera 101 stores the later camera parameters in RAM 203. Thus, in this embodiment, the timing of the start of display of the background image by the display device 103, the timing of the start of image capture by the camera 101, the detection timing by the position and orientation detection device 102, and the detection timing by the camera 101 all coincide at time 7. Note that time point 7 is an example of the second timing.

[0031] Next, at time 8, which is later than time 7, the position and attitude detection device 102 transmits the position and attitude information generated by the detection at time 7 to the camera 101. The RAM 203 of the camera 101 stores the received position and attitude information. Furthermore, the display of the background image by the display device 103 and the imaging by the camera 101, which began at time 7, continue until time 9, which is later than time 8. Furthermore, from time point 9 onward, the processing from time points 1 to 8 is repeated until the system control device 104 instructs the termination of processing. In this case, the display control device 106 causes the newly generated display background image to be displayed on the display device 103 each time a new display background image is generated by the background image generation device 105. This updates the display content of the display device 103 as a display background image captured by the camera 101. In this way, the imaging system 100 captures in-camera VFX video. The camera 101 also acquires the captured image as in-camera VFX video, the previously recorded camera parameters, the previously recorded position and orientation information, the later recorded camera parameters, and the later recorded position and orientation information.

[0032] Note that the relationships between the periods from time point 1 to time point 8 are not limited to the example shown. For example, the period from time 5 to time 7 may be longer than the period from time 1 to time 5. Also, the period from time 7 to time 9 may be longer than the period from time 1 to time 5, and the period from time 5 to time 7.

[0033] Figure 6 is a flowchart showing the flow of the imaging process. The imaging process is the process by which the imaging system 100 captures in-camera VFX images. In this embodiment, the imaging process starts when the user inputs an instruction to start capturing in-camera VFX images to the instruction input unit 308 of the system control device 104. The system control unit 104 notifies the camera 101, position and orientation detection device 102, background image generation device 105, and display control unit 106 to start the processing necessary for capturing in-camera VFX video, and causes them to start the in-camera VFX video capture sequence (S501). At this time, the control unit 301 of the system control unit 104 transmits a synchronization signal as described above. As a result, the camera 101, position and orientation detection device 102, background image generation device 105, and display control unit 106 operate at predetermined timings in the capture sequence.

[0034] Next, the position and orientation detection device 102 detects the position and orientation of the camera 101, and the camera 101 detects the camera parameters mentioned earlier (S502). The detection in step 502 is the detection at time point 1 shown in Figure 5. The position and orientation detection device 102 transmits the position and orientation information generated by the detection in step 502 to the camera 101 and the system control device 104. The camera 101 also transmits the camera parameters and position and orientation information mentioned earlier to the background image generation device 105. Next, the background image generation device 105 generates a display background image based on the camera parameters and position / orientation information (S503). The display background image generated in step 503 is the display background image whose generation begins at time 5 shown in Figure 5. The background image generation device 105 transmits the generated display background image to the display control device 106.

[0035] Next, the display control device 106 displays a background image for display on the display device 103, the camera 101 captures an image, the camera 101 detects subsequent camera parameters, and the position and orientation of the camera 101 is detected by the position and orientation detection device 102 (S504). The processing in step 504 is the processing that is performed at time 7 shown in Figure 5. Therefore, the processing in step 504 generates subsequent position and orientation information. The position and orientation detection device 102 transmits the subsequent position and orientation information to the camera 101. Next, camera 101 records the previously recorded camera parameters, previously recorded position and orientation information, the captured image taken in step 504, the later camera parameters, and the later position and orientation information in the recording unit 208 (S505). More specifically, the recording unit 208 records the captured image to which the previously recorded camera parameters, previously recorded position and orientation information, the later camera parameters, and the later position and orientation information are associated as metadata.

[0036] Next, the control unit 301 of the system control device 104 determines whether or not an instruction has been given to end the capture of the in-camera VFX image (S506). In this embodiment, the determination in step 506 is made based on whether or not the user has input an instruction to end the capture of the in-camera VFX image to the instruction input unit 308 of the system control device 104. If there is no instruction to stop capturing in-camera VFX video (no in S506), the process from step 502 is repeated. More specifically, if there is no instruction to stop capturing in-camera VFX video, the processes from step 502 to step 506 are performed frame by frame based on the synchronization signal supplied from the clock generation unit 309 of the system control device 104. One frame is one frame of the video captured by camera 101 at a predetermined frame rate. Therefore, until the capture of in-camera VFX video is completed, the previous camera parameters, previous position and orientation information, subsequent camera parameters, and subsequent position and orientation information are associated with each frame of the captured image generated by camera 101. In this case, it becomes easier to determine the position and orientation information and camera parameters corresponding to the captured image for each frame.

[0037] Furthermore, if there is an instruction to terminate the acquisition of the in-camera VFX image (yes in S506), the system proceeds to the next step. The system control device 104 instructs the camera 101, position and orientation detection device 102, background image generation device 105, and display control device 106 to terminate the processing necessary for acquiring the in-camera VFX image, and terminates the acquisition sequence of the in-camera VFX image (S507).

[0038] As described above, the imaging process steps 502 to 506 are performed one frame at a time. Therefore, the period from time point 1 to time point 7 shown in Figure 5 is less than one frame. In other words, the period from when the position and attitude detection device 102 detects the position and attitude related to the earlier position and attitude information until when it detects the position and attitude related to the later position and attitude information is less than one frame. Also, the period from when the position and attitude detection device 102 detects the position and attitude related to the earlier position and attitude information until the camera 101 starts capturing the in-camera VFX image is less than one frame.

[0039] Next, we will explain the method by which the background image generation device 105 generates a background image for display. The background image generation device 105 first sets the position and orientation of the virtual camera according to the previously set position and orientation information, and also sets the camera parameters of the virtual camera according to the previously set camera parameters. The camera parameters of the virtual camera set at this time are of the same type as the previously set camera parameters. Then, the background image generation device 105 renders a 3D model of the virtual space according to the set position and orientation and camera parameters of the virtual camera and generates a CG image. The CG image generated by rendering by the background image generation device 105 may be referred to as the background rendering image below. Next, the background image generation device 105 corrects the background rendering image. The correction of the background rendering image is performed to correct distortion and changes in magnification of the display background image in the captured image when the camera 101, at the position and orientation indicated in the position and orientation information above, captures the display background image to be displayed on the display device 103, including the field of view. The background image generation device 105 generates the display background image through this correction.

[0040] Figure 7 shows the relationship between camera 101, whose position and orientation are as shown in the position and orientation information above and whose focal length is as shown in the camera parameters above, and the image display surface on display device 103. Below, we will explain the correction of the background rendering image by the background image generation device 105 using Figure 7. First, let's explain the configuration shown in Figure 7. The image display surface of the display device 103 is the surface facing the camera 101 and is a plane. Also, in the illustrated example, the camera 101 is not directly facing the image display surface of the display device 103. Figure 7 also shows the x and z axes of the world coordinate system, which is represented in three dimensions (x, y, and z axes), and the u axis of the camera coordinate system, which is represented in two dimensions (u and v axes). In the illustrated example, the x axis is parallel to the image display surface of the display device 103 and the ground, and the z axis is perpendicular to the image display surface of the display device 103. Also, in the illustrated example, the u axis is any direction of the image captured by the camera 101, in other words, the horizontal direction of the image captured by the camera 101. Furthermore, the area enclosed by the two dotted lines is the area included in the field of view a of camera 101. Also, plane 701 represents a virtual image display surface that includes the image display surface of the display device 103 which is in a fixed position. In addition, the portion of plane 701 from dividing line 702a to dividing line 702b is the range included in the field of view a of camera 101, in other words, the imaging range of camera 101. Furthermore, the portion of plane 701 from dividing line 703a to dividing line 703b represents the range of the image display surface on the display device 103. Image plane 704 is the image plane of camera 101. Plane 701 also represents a virtual image display surface that includes the image display surface of the display device 103 whose position is known. "The display device 103 whose position is known" means that the positional information indicating the location of the display device 103 is known in the imaging system 100.

[0041] When the background rendering image is captured by the camera 101 so that its entirety is allocated to the entire image plane 704, the position and orientation of the camera 101 and the camera parameters are matched with the image captured in the image to be displayed as the background image for display on the display device 103. Furthermore, when the background rendering image is corrected so that the background image for display is allocated to the portion of the plane 701 from dividing line 702a to dividing line 702b, the camera 101 captures the image so that its entirety is allocated to the entire image plane 704.

[0042] The background image generation device 105 corrects the background rendering image by using the perspective projection transformation formula shown in the following formula (1) based on the previous camera parameters and the previous position and orientation information.

Equation

[0043] Also, R in Equation (1) c1 , T c1 respectively represent the rotation angle and the translational movement amount of the camera 101 at time point 1 shown in FIG. 5 with respect to the x-axis, y-axis, and z-axis in the world coordinate system. R in Equation (1) c1 , T c1 are respectively obtained from the following equations (2) and (3).

Equation

Equation

[0044] Also, A in Equation (1) c1 is the previous camera parameter including the focal length of the camera 101 at time point 1 shown in FIG. 5, and is as shown in the following equation (4).

Equation

[0045] Also, Z in equation (1) c1 This is the coordinate of camera coordinate system C1 in the position and orientation of camera 101 as indicated by the position and orientation information above, and the coordinate Z in the world coordinate system. w This corresponds to the X coordinate system in the world coordinate system. w , Y w , Z w The corresponding X coordinates in camera coordinate system C1 c1 , Y c1 , Z c1 This can be calculated from equation (5) below.

number

[0046] The background rendering image is a 3D model rendered according to the camera parameters and position / orientation of a virtual camera set based on the previously defined camera parameters and position / orientation information. Therefore, the coordinates of the background rendering image are given by u, similar to equation (1). c1 , v c1 It can be understood as being represented by the coordinate X of a predetermined pixel P on the image display surface of the display device 103. wp , Y wp , Z wp Regarding this coordinate, the coordinates u of the background rendering image corresponding to this coordinate c1p , v c1p The coordinate u is calculated from equation (1), c1p , v c1p The pixel value of is obtained. Also, the background image generation device 105 generates the coordinate u c1p , v c1pThe pixel value is set as the pixel value to be displayed at a predetermined pixel P on the image display surface of the display device 103. Furthermore, the background image generation device 105 sets this pixel value for each pixel on the image display surface of the display device 103. As a result, a display background image is generated in which distortion and magnification corrections have been applied so that the entire background rendering image is allocated to the entire image plane 704. Also, when the camera 101 takes an image at time 1, the coordinates of the image reflected in the captured image as the display background image to be displayed on the display device 103 are the same as the background rendering image, u c1 , v c1 It is represented by [this].

[0047] Furthermore, when the camera 101 takes an image at time 1, the coordinates X of a predetermined pixel P on the image display surface of the display device 103 wp , Y wp , Z wp The coordinates u of the background rendering image corresponding to this coordinate c1p , v c1p However, there are cases where the pixel P is not included in the field of view of the camera 101. In this case, the background image generation device 105 does not need to assign a pixel value to the coordinates of the background rendering image corresponding to the coordinates of this pixel P. For example, the pixel P may be set to the pixel value of a still image related to a 3D model of the virtual space used for rendering. In this way, imaging by the imaging system 100 is performed under appropriate lighting conditions. Furthermore, in the following, it is assumed that when camera 101 captures an image at time 1, there are no pixels on the image display surface of the display device 103 that correspond to the coordinates of the background rendering image that are not included in the field of view of camera 101.

[0048] Furthermore, the background image generation device 105 may render a background rendering image covering a wider area than the field of view of the camera 101 when the camera 101 is capturing an image at time 1. In this case, the background image generation device 105 may assign pixel values ​​to the coordinates of the background rendering image covering a wider area than the field of view of the camera 101 at time 1. By doing so, even if the position, orientation, or camera parameters of the camera 101 change from the state at time 1, the capture of an area on the image display surface of the display device 103 where an image different from the display background image is displayed is suppressed.

[0049] In the example described above, it was explained that the generation of the display background image begins at time 5, as shown in Figure 5. Here, time 5 is specifically the time when the generation of the background rendering image begins. Furthermore, the display background image is generated between time 5 and time 6. In other words, the generation of the background rendering image and the display background image takes place during the period from time 5 to time 6.

[0050] Figure 8 is a flowchart showing the flow of the synthesis process. The synthesis process is the process by which the image synthesis generator 107 generates a composite image. In this embodiment, when the user inputs an instruction to generate a composite image into the instruction input unit 408 of the image synthesis generator 107, the synthesis process starts. The control unit 401 of the composite image generation device 107 requests the camera 101 for the captured image, the previous camera parameters, the previous position and orientation information, the subsequent camera parameters, and the subsequent position and orientation information, and obtains the requested information from the camera 101 (S801). More specifically, the control unit 401 of the composite image generation device 107 obtains the captured image to which the previous camera parameters, the previous position and orientation information, the subsequent camera parameters, and the subsequent position and orientation information are associated as metadata. The recording unit 405 of the composite image generation device 107 also records the information obtained in step 801.

[0051] The control unit 401 of the composite image generation device 107 renders the same 3D model of the virtual space that the background image generation device 105 used to generate the background rendering image, according to the position and orientation information, to generate a CG image for synthesis (S802). This section describes the method by which the composite image generation device 107 renders a 3D model to generate a CG image for synthesis. The composite image generation device 107 sets the position and orientation of a virtual camera according to the previously set position and orientation information, and also sets the camera parameters of the virtual camera according to the previously set camera parameters. The camera parameters of the virtual camera set at this time are of the same type as the previously set camera parameters. Then, the composite image generation device 107 renders the 3D model in the virtual space according to the set position and orientation and camera parameters of the virtual camera to generate a CG image for synthesis. The CG image for synthesis generated by rendering by the composite image generation device 107 may be referred to as the composite rendering image below. The composite rendering image is an image generated based on the previously set camera parameters and position and orientation information, similar to the display background image. Therefore, the composite rendering image can also be considered as an image corresponding to the display background image.

[0052] Furthermore, when the background image generation device 105 generates a background rendering image, it renders only the area within the imaging range of the camera 101 that corresponds to the image display surface on the display device 103. On the other hand, when the composite image generation device 107 generates a composite rendering image, it renders the entire imaging range of the virtual camera. In this case, the composite rendering image includes the corresponding CG image outside the image display surface of the display device 103.

[0053] Next, the control unit 401 of the composite image generation device 107 corrects the composite rendering image according to the earlier position and orientation information and the later position and orientation information. More specifically, the control unit 401 generates a composite background image by correcting the rendering image based on the difference between the position and orientation of camera 101 indicated by the earlier position and orientation information and the position and orientation of camera 101 indicated by the later position and orientation information (S803). The method of generating the composite background image by the composite image generation device 107 will be described in detail later. In addition, the RAM 403 or recording unit 405 of the composite image generation device 107 records the generated composite background image.

[0054] The control unit 401 of the composite image generation device 107 determines the region to which the composite background image will be composited onto the captured image acquired in step 801. The control unit 401 determines the region to which the composite background image will be composited based on subsequent position and orientation information and the position of the display device 103. Specifically, the control unit 401 determines the region to which the composite background image will be composited as the region of the captured image in which the display background image displayed on the display device 103 is not visible, among the regions defined as the background display region in the captured image (S804). The region of the captured image in which the display background image displayed on the display device 103 is not visible can also be considered as the region of the captured image corresponding to the region outside the image display region of the display device 103. The region in which the composite background image is composited onto the captured image may be referred to as the composite region below. The method by which the composite image generation device 107 determines the composite region will be described in detail later. In addition, the RAM 403 or recording unit 405 of the composite image generation device 107 records information indicating the composite region.

[0055] The control unit 401 of the composite image generation device 107 composites a composite background image into the composite region of the captured image (S805). More specifically, the control unit 401 replaces the pixel values ​​of pixels in the captured image that correspond to the composite region with the pixel values ​​of pixels in the composite background image that correspond to the region of these pixels. The control unit 401 performs this replacement for each pixel in the captured image that corresponds to the composite region. The control unit 401 does not replace the pixel values ​​of pixels in the captured image that correspond to the composite region. In this way, the control unit 401 generates a composite image in which the areas of the captured image that are not displayed in the display background image within the area designated as the background display region are replaced with the composite background image. The recording unit 405 of the composite image generation device 107 also records the generated composite image.

[0056] The compositing process may be performed for each frame of the in-camera VFX footage. Alternatively, in the compositing process, one step may be performed for all frames of the in-camera VFX footage before proceeding to the next step. If the compositing process is performed for each frame of the in-camera VFX video, after steps 801 to 805 are performed for the first frame, the control unit 401 of the composite image generation device 107 determines whether or not an instruction to end the compositing process has been received. For example, whether or not an instruction to end the compositing process has been received may be determined by whether or not the user has entered an instruction to end the compositing process into the instruction input unit 408 of the composite image generation device 107. If there is no instruction to end the compositing process, the control unit 401 waits until the next frame is transmitted from the camera 101, and when the next frame is transmitted from the camera 101, it performs steps 801 to 805 for the next frame. In this way, steps 801 to 805 are repeated sequentially for each frame of the in-camera VFX video until an instruction to end the compositing process is received.

[0057] Next, we will explain the relationship between the display background image, the composite rendering image, and the composite background image. In this embodiment, it was explained that the composite image generation device 107 generates a composite rendering image based on the aforementioned camera parameters and position / orientation information. It was also explained that the background image generation device 105 generates a display background image based on the aforementioned camera parameters and position / orientation information. Here, when camera 101 captures an image with the position, orientation, and camera parameters shown in Figure 5 at time point 1, the display background image in the captured image will be free from distortion and magnification changes caused by the position, orientation, and camera parameters of camera 101. In this case, there will be no positional shift between the display background image in the captured image and the composite rendering image due to the position, orientation, and camera parameters of camera 101.

[0058] However, due to the time required to generate the background image for display, a period of time occurs between time 1, when the camera parameters and position / orientation information considered for generating the background image for display are detected, and time 7, when camera 101 begins imaging. If the position / orientation or camera parameters of camera 101 change during this period, camera 101 may, at time 7, include the background image for display displayed on the display device 103 in its field of view and image with different position / orientation and camera parameters than at time 1. In this case, due to differences in the position, orientation, and camera parameters of camera 101 between time point 1 and time point 7, the display background image displayed in the captured image will be distorted and its magnification will change due to the position, orientation, and camera parameters of camera 101. In this case, a positional shift will occur between the display background image in the captured image and the composite rendering image. To add to this, the display background image in the captured image is displayed in the area corresponding to the camera parameters and position, orientation, at time point 7. Therefore, a positional shift will occur between the display background image in the captured image and the composite rendering image, corresponding to the differences in camera parameters and the position, orientation, of camera 101 between time point 1 and time point 7.

[0059] Therefore, the composite image generation device 107 generates a composite background image by applying corrections to the composite rendering image that are similar to the distortions and magnification changes that occurred in the display background image in the captured image, thereby suppressing the positional misalignment between the display background image in the captured image and the composite background image.

[0060] This section describes a method by which the composite image generation device 107 corrects the composite rendering image to generate a background image for synthesis. The composite image generation device 107 corrects the composite rendering image using equation (1) above and equation (6) below, which is an equation for perspective projection transformation processing according to the subsequent position and orientation information and subsequent camera parameters.

number

[0061] Also, R in equation (6) c2 , T c2 These are the rotation angle and translational movement of camera 101 at time 7 shown in Figure 5, with respect to the x, y, and z axes in the world coordinate system, respectively. R in equation (6) c2 , T c2 These can be obtained from equations (7) and (8) below, respectively.

number

number

[0062] Also, A in equation (6) c2 This represents the camera parameters, including the focal length of camera 101 at time 7 shown in Figure 5, as shown in equation (9) below.

number

[0063] Also, Z in equation (6) c2 This is the coordinate of camera coordinate system C2 in the position and orientation of camera 101 as indicated by the later position and orientation information, and the coordinate Z in the world coordinate system. w This corresponds to the X coordinate system in the world coordinate system. w , Y w , Z w The corresponding camera coordinate system C2 coordinate X c2 , Y c2 , Z c2 This can be calculated from the following formula (10).

number

[0064] The composite rendering image is a 3D model rendered according to the camera parameters and position / orientation of a virtual camera set based on the previously defined camera parameters and position / orientation information. Therefore, the coordinates of the composite rendering image are given by u, similar to equation (1). c1 , v c1 It can be understood as something represented by this. Furthermore, the composite image generation device 107 converts the coordinates of the composite rendering image to the coordinates of the captured image to generate a background image for synthesis. Therefore, the coordinates of the composite background image are the same as those of the captured image, u c2 , v c2 It can be understood that it is represented by equation (1) and equation (6). c1 , v c1 and the coordinates u of the background image for compositing c2 , v c2The relationship can be determined from equation (11) below.

number

[0065] Figure 9 shows the image display surface of the imaging system 100 in the world coordinate system. Figure 9(A) shows the image display surface of the imaging system 100 as a plane in the world coordinate system. More specifically, in Figure 9(A), the image display surface 901a of the display device 103 is displayed as a plane, and a virtual image display surface 902a is also displayed as a plane. As shown in Figure 9(A), the virtual image display surface 902a is set up so as to include the image display surface 901a of the display device 103. Furthermore, the image display surface 901a of the display device 103 and the portion of the virtual image display surface 902a that is outside the image display surface 901a of the display device 103 are continuous.

[0066] Figure 9(B) shows the image display surface of the imaging system 100 as a curved surface in the world coordinate system. More specifically, Figure 9(B) shows that the image display surface 901b of the display device 103 is displayed as a curved surface, and a virtual image display surface 902b is also displayed as a curved surface. As shown in Figure 9(B), the virtual image display surface 902b is set up so as to include the image display surface 901b of the display device 103. Furthermore, the image display surface 901b of the display device 103 and the portion of the virtual image display surface 902b that is outside the image display surface 901b of the display device 103 are continuous. Furthermore, the virtual image display surface 902a and the virtual image display surface 902b are X w , Y w , Z wIt is expressed by an expression in which one or more of the variables are used. Therefore, it is preferable that the shapes of the image display surface 901a and the image display surface 901b of the display device 103 are shapes that can be expressed by an expression similar to that which represents the virtual image display surface 902a and the virtual image display surface 902b.

[0067] Based on a virtual image display surface that includes the image display surface of the display device 103 whose position is known, Z in equation (11) c1 , Z c2 An example of how to calculate this will be explained. In the following example, it is assumed that the image display surface of the display device 103 is a plane, and that the world coordinate system is set so that the origin of the world coordinate system includes the image display surface of the display device 103. Also, the direction perpendicular to the image display surface of the display device 103 is defined as the Z-axis of the world coordinate system. In this case, the virtual image display surface is represented by the following equation (12).

number

[0068] Also, the X coordinate of the camera coordinate system C2 c2 , Y c2 , Z c2 The corresponding world coordinate X w , Y w , Z w This can be expressed by the following (13), which is a transformation of equation (10).

number

number

[0069] Also, the coordinates u of the background image for compositing c2 , v c2 and the X coordinates of camera coordinate system C2 c2 , Y c2, Z c2 The relationship between them is expressed by the following formula (15).

Number

Number

Number

Number

[0070] Next, based on formula (18), the Z c2q corresponding to the coordinates u c2q and v c2q of a predetermined pixel Q in the background image for synthesis is obtained. Also, by substituting the coordinates u c2q and v c2q and Z c2q into formula (16) and formula (17), the X c2q and Y c2q corresponding to the coordinates u c2q and v c2q are obtained respectively. Also, the coordinates X c2 , Y c2 , and Z c2 in the camera coordinate system C1 corresponding to the coordinates X c1 , Y c1 , and Z c1 in the camera coordinate system C2 are expressed by the following formula (19) based on formula (5) and formula (10).

Number

[0071] As in the example above, based on a virtual image display surface that includes the image display surface of the display device 103 whose position is known, the corrected coordinate u is calculated using equation (11). c2 , v c2 From there, the coordinates u before correction c1 , v c1 The composite image generation device 107 calculates the coordinates u of a predetermined pixel Q in the composite background image according to equation (11). c2q , v c2q The coordinates u of the composite rendering image corresponding to this coordinate c1q , v c1q Find the coordinate u c1q , v c1q The pixel values ​​of the coordinate u are obtained from the composite rendering image. In addition, the composite image generation device 107 obtains the pixel values ​​of the coordinate u c1q , v c1q The pixel value is set as the pixel value of a predetermined pixel Q in the composite background image. The composite image generation device 107 sets this pixel value for each pixel of the composite background image. As a result, a composite background image is generated that takes into account the camera parameters and position / orientation at time 7 shown in Figure 5, in other words, a composite background image in which the positional misalignment with the display background image in the captured image is suppressed. In this way, the composite image generation device 107 generates a composite background image based on the earlier and later position and orientation information, thereby suppressing positional shifts (image shifts, shifts in the translational, rotational, and scaling directions) of the composite background image relative to the display background image in the captured image.

[0072] Note that the corrected coordinate u c2 , v c2 From the coordinates u before correction c1 , v c1 The methods for finding X are not limited to the examples given above. For example, X w , Y w , Z wBy setting a virtual image display surface represented by an expression with variables u, the corrected coordinates u c2 , v c2 From the coordinates u before correction c1 , v c1 This is required. Also, depending on the virtual image display surface, such as when the virtual image display surface is curved, the Z shown in equation (11) is required. c1 , Z c2 Multiple values ​​may be required. In this case, the range of values ​​of the virtual image display surface is appropriately restricted, so that Z is uniquely determined. c1 , Z c2 This is required. Also, the coordinates u of the background image for compositing c2 , v c2 The coordinates u of the composite rendering image corresponding to this coordinate c1 , v c1 However, this may result in coordinates outside the rendered area. To prevent this, the composite image generation device 107 may render a wider area than the imaging range of the virtual camera determined by the position and orientation information and camera parameters when generating a composite rendered image.

[0073] Next, we will explain the method by which the composite image generation device 107 determines the composite region. The composite image generation device 107 detects the area of ​​the display device 103 that is outside the image display surface by setting an equation that represents a virtual image display surface including the image display surface of the display device 103 on the world coordinate system of the virtual space, similar to the method used to generate the background image for synthesis.

[0074] This section describes a method by which the composite image generation device 107 detects an area of ​​the display device 103 that is outside the image display surface. As described above, camera 101 captures images using the position and orientation at time 7 shown in Figure 5 and the camera parameters. Therefore, the coordinates of the captured image are the same as those of the background image for synthesis, using coordinates u in accordance with the camera coordinate system C2. c2 , v c2 It can be understood as being represented by the coordinate u. c2 , v c2 The corresponding world coordinate X w , Yw , Z w This can be expressed by the following equation (20), which is a transformation of equation (6).

number

[0075] Z shown in equation (20) c2 This is similar to the method used to generate the background image for compositing, and is based on an equation representing a virtual image display surface, with coordinates u c2 , v c2 The composite image generation device 107 calculates the coordinates u of a predetermined pixel S in the captured image from equation (20). c2s , v c2s The corresponding coordinate X ws , Y ws , Z ws The composite image generation device 107 calculates the calculated coordinate X. ws , Y ws , Z ws However, it determines whether or not the virtual image display surface is included in the image display surface of the display device 103. Then, the composite image generation device 107 determines the coordinate X ws , Y ws , Z ws However, if it is determined that the virtual image display surface is not included in the image display surface of the display device 103, then coordinate u c2s , v c2s The pixels are set as pixels in the composite region. The composite image generation device 107 determines the composite region by performing this setting for each pixel of the captured image.

[0076] In this embodiment, the composite image generation device 107 composites a composite background image into the composite region of the captured image, but does not composite a composite background image into regions of the captured image other than the composite region. However, the embodiment is not limited to this. The composite image generation device 107 may also composite the background image for compositing to areas of the captured image that are different from the compositing area. More specifically, the composite image generation device 107 may perform alpha blending between the captured image and the background image for compositing in areas of the captured image that are different from the compositing area but are adjacent to the compositing area.

[0077] Figure 10 shows the details of alpha blending between the captured image and the background image for compositing, performed on a captured image 1000 corresponding to one frame of the in-camera VFX video. The captured image 1000 shown in Figure 10 is an captured image as in-camera VFX video captured by camera 101 so that the display background image displayed on the display device 103 is included in the field of view. As shown in Figure 10, the captured image 1000 shows a composite boundary line 1002, a non-composite boundary line 1003, a composite region R1, and a non-composite region R2.

[0078] The composite boundary line 1002 is the boundary line between the composite region R1 and a region different from the composite region R1 within the captured image 1000. In the illustrated example, the region outside the composite boundary line 1002 within the captured image 1000 is the composite region R1. The composite image generation device 107 composites a background image for compositing onto this composite region R1. More specifically, the composite image generation device 107 sets the transparency of the captured image 1000 to 100% and the transparency of the composite image to 0% within the composite region R1. In addition, in the illustrated example, the display background image is not displayed in the composite region R1 of the captured image 1000, but is displayed in a region inside the composite region R1. Therefore, in the illustrated example, the composite boundary line 1002 is the boundary line of the region within the captured image 1000 where the display background image is displayed. Also, in the illustrated example, the composite boundary line 1002 is the boundary line of the image display area of ​​the display device 103 within the captured image 1000.

[0079] The non-compositing boundary line 1003 is the boundary line between the non-compositing region R2 and a region different from the non-compositing region R2 in the captured image 1000. In the illustrated example, the region inside the non-compositing boundary line 1003 in the captured image 1000 is the non-compositing region R2. Furthermore, the non-compositing region R2 is the region where the composite background image is not composited. The composite image generation device 107 sets the transparency of the captured image 1000 to 0% in the non-compositing region R2 and sets the transparency of the composite image to 100%. In addition, in the illustrated example, the composite background image in the captured image 1000 is not displayed in the non-compositing region R2, but is displayed in the region outside the non-compositing region R2. Therefore, the non-compositing boundary line 1003 is the boundary line of the region in the captured image 1000 where the composite background image is displayed.

[0080] Furthermore, the captured image 1000 shows the intermediate region R3 and the step boundary line 1004. The intermediate region R3 is the region between the composite region R1 and the non-composite region R2. The composite image generation device 107 composites a background image for synthesis onto the intermediate region R3. More specifically, the composite image generation device 107 sets the transparency of the background image for synthesis in the intermediate region R3 to be higher than that of the composite region R1 and lower than that of the non-composite region R2. In addition, the composite image generation device 107 sets the transparency of the captured image 1000 in the intermediate region R3 to be lower than that of the composite region R1 and higher than that of the non-composite region R2.

[0081] Furthermore, the intermediate region R3 is shown to consist of a composite region R31 and a non-composite region R32. The composite region R31 is the region of intermediate region R3 that is closer to the composite region R1 than the non-composite region R32. In other words, the composite region R31 is a region that is shorter in distance from the composite region R1 than the non-composite region R32. Note that the distance from the composite region R1 means the shortest distance from the composite region R1. The non-composite region R32 is the region of intermediate region R3 that is closer to the non-composite region R2 than the composite region R31. In other words, the non-composite region R32 is a region that is longer in distance from the composite region R1 than the composite region R31. The step boundary line 1004 is the boundary line between the composite region R31 and the non-composite region R32 of the intermediate region R3 in the captured image 1000. In the illustrated example, the region of the intermediate region R3 in the captured image 1000 that is outside the step boundary line 1004 is the composite region R31, and the region of the intermediate region R3 in the captured image 1000 that is inside the step boundary line 1004 is the non-composite region R32.

[0082] The composite image generation device 107 sets the transparency of the background image for synthesis in the synthesis region R31 to a lower transparency than that of the non-synthesis region R32, and sets the transparency of the captured image 1000 in the synthesis region R31 to a higher transparency than that of the non-synthesis region R32. In other words, the composite image generation device 107 sets the transparency of the background image for synthesis in the non-synthesis region R32 to a higher transparency than that of the synthesis region R31, and sets the transparency of the captured image 1000 in the non-synthesis region R32 to a lower transparency than that of the synthesis region R31. Thus, in the captured image 1000, alpha blending may be performed such that the blending ratio between the captured image 1000 and the composite background image changes in steps in the order of composite region R1, composite-side region R31, non-composite-side region R32, and non-composite region R2. In this case, a composite image is generated in which the boundary between the display background image and the composite background image in the captured image 1000 transitions smoothly. Therefore, the boundary between the display background image and the composite background image becomes less noticeable in the composite image.

[0083] Furthermore, the portion of the captured image 1000 in which transparency is set in the intermediate region R3 is the portion of the captured image 1000 in which the display background image is visible. Therefore, alpha blending in the intermediate region R3 can also be understood as a stepwise change between the transparency of the display background image and the transparency of the composite background image as seen in the captured image 1000.

[0084] In this embodiment, the composite image generation device 107 was described as compositing a background image for compositing onto the captured image, but it is not limited to this. The display control device 106 may, for example, display the captured image and the background image for synthesis on the display device 103 such that the background image for synthesis is superimposed on the synthesis area of ​​the captured image. Even in this case, the user can be shown the captured image and the background image for synthesis with suppressed positional shifts.

[0085] (Variation 1) Next, I will explain some variations. In this embodiment, the composite image generation device 107 has been described as compositing a composite background image onto an area of ​​the captured image that is outside the display background image, but it is not limited to this. The composite image generation device 107 may also composit a composite background image onto the display area of ​​the display background image in the captured image. In other words, the composite image generation device 107 may also composit a composite background image onto the image display area of ​​the display device 103 in the captured image.

[0086] Figure 11(A) shows the relationship between the captured image 1000 and the composite region R1 as a modified example 1. The captured image 1000 shown in Figure 11(A) shows the display region boundary line 1005. The display area boundary line 1005 is the boundary line of the image display area of ​​the display device 103 within the captured image 1000. In the illustrated example, the area inside the display area boundary line within the captured image 1000 is the image display area of ​​the display device 103, and the area outside the display area boundary line within the captured image 1000 is the area outside the image display area of ​​the display device 103. Furthermore, as shown in Figure 11(A), in the captured image 1000, non-composite region R2 is shown both outside and inside the display area boundary line 1005. Also, in the captured image 1000, composite region R1 is shown inside the display area boundary line 1005. More specifically, in the captured image 1000, composite region R1 is shown within region B, which is a part of the area inside the display area boundary line 1005.

[0087] Thus, the composite image generation device 107 may composite a composite background image onto the image display area of ​​the display device 103 within the captured image 1000. In this case, even if a defect occurs in the display area of ​​the background image within the captured image 1000, the composite image is composited onto the area where the defect occurred, making it difficult to recognize the defect. Examples of defects include moiré patterns caused by the pixel arrangement on the image display surface of the display device 103 being reflected in the captured image, and unwanted subjects on the camera 101 side of the display device 103 being reflected in the image. Furthermore, even when the composite image generation device 107 composites a background image for compositing onto the image display area of ​​the display device 103 from the captured image 1000, it may perform alpha blending between the captured image and the composite background image in the area of ​​the captured image adjacent to the compositing area.

[0088] Figure 11(B) is an enlarged view of region B of the captured image 1000 shown in Figure 11(A). More specifically, Figure 11(B) is a diagram showing the contents of alpha blending between the captured image and the background image for synthesis, which is performed on the captured image 1000 as a modified example 1. Within region B of the captured image 1000, a composite region R1, a non-composite region R2, an intermediate region R3, a composite boundary line 1002, a non-composite boundary line 1003, and a step boundary line 1004 are shown.

[0089] In the illustrated example, the area inside the composite boundary line 1002 of the captured image 1000 is the composite region R1. The composite image generation device 107 sets the transparency of the captured image 1000 to 100% and the transparency of the composite image to 0% in the composite region R1. In addition, in the illustrated example, the display background image in the captured image 1000 is not displayed in the composite region R1, but is displayed in the area outside the composite region R1. In the illustrated example, the area of ​​the captured image 1000 outside the non-composite boundary line 1003 is the non-composite region R2. The composite image generation device 107 sets the transparency of the captured image 1000 to 0% and the transparency of the composite image to 100% in the non-composite region R2. In addition, in the illustrated example, the background image for composite is not displayed in the non-composite region R2 of the captured image 1000, but is displayed in the area inside the non-composite region R2.

[0090] In the illustrated example, the area inside the step boundary line 1004 within the intermediate region R3 of the captured image 1000 is the composite region R31, and the area outside the step boundary line 1004 within the intermediate region R3 of the captured image 1000 is the non-composite region R32. The composite image generation device 107 sets the transparency of the background image for compositing in the composite region R31 to a lower transparency than that of the non-composite region R32, and sets the transparency of the captured image 1000 in the composite region R31 to a higher transparency than that of the non-composite region R32. In other words, the composite image generation device 107 sets the transparency of the background image for compositing in the non-composite region R32 to a higher transparency than that of the composite region R31, and sets the transparency of the captured image 1000 in the non-composite region R32 to a lower transparency than that of the composite region R31. Thus, in the modified example 1, alpha blending may be performed such that the transparency of the captured image 1000 decreases and the transparency of the composite background image increases as the region of the captured image 1000 moves further outward from the composite region R1.

[0091] (Modification 2) Next, we will describe a modified example of the synthesis process (see Figure 8). In this embodiment, the composite image generation device 107 suppresses the positional misalignment between the display background image in the captured image and the image synthesized on the captured image by correcting the composite rendering image, but it is not limited to this. The composite image generation device 107 may, for example, suppress the positional misalignment between the display background image in the captured image and the image synthesized on the captured image by correcting the captured image to match the image synthesized on the captured image.

[0092] Figure 12 is a flowchart showing the flow of the synthesis process as a modified example 2. In the synthesis process in modified example 2, the composite image generation device 107 corrects the captured image and then composites the composite rendering image onto the corrected captured image. Note that steps 1201 and 1202 in the synthesis process shown in Figure 12 are the same as steps 801 and 802 in the synthesis process shown in Figure 8. Next, the control unit 401 of the composite image generation device 107 corrects the captured image based on the earlier position and orientation information and the later position and orientation information (S1203). More specifically, the control unit 401 corrects the captured image based on the earlier camera parameters, the earlier position and orientation information, the later camera parameters, and the later position and orientation information. The method by which the composite image generation device 107 corrects the captured image will be described in detail later. In addition, the RAM 403 or recording unit 405 of the composite image generation device 107 records the corrected captured image.

[0093] The composite image generation device 107 determines the region from the corrected captured image to which the composite rendering image will be synthesized (S1204). The method by which the composite image generation device 107 determines the region from the corrected captured image to which the composite rendering image will be synthesized will be described in detail later. The composite image generation device 107 generates a composite image (S1205) by compositing a composite rendering image onto the region determined in step 1205 of the corrected captured image. The method for compositing the composite rendering image in step 1205 of the compositing process shown in Figure 12 is the same method as the method for compositing the background image for compositing in step 805 of the compositing process shown in Figure 8.

[0094] Next, the method by which the composite image generation device 107 corrects the captured image will be described. In the following, the captured image before correction in the synthesis process may be referred to as the uncorrected captured image, and the captured image after correction in the synthesis process may be referred to as the corrected captured image. As mentioned above, the coordinates of the image before correction are u c2 , vc2 It is represented by . Furthermore, the composite image generation device 107 converts the coordinates of the uncorrected captured image to the coordinates of the composite rendering image to obtain the corrected captured image. Therefore, the coordinates of the corrected captured image are u, similar to the coordinates of the composite rendering image. c1 , v c1 It can be understood that it is expressed by the following equation (21), which is a modified version of equation (11).

number

[0095] Z shown in equation (21) c1 , Z c2 Based on the virtual image display surface in the virtual space, the coordinate u is used, similar to the example described above. c1 , v c1 The coordinates u of a predetermined pixel I in the corrected image are determined accordingly. The composite image generation device 107 determines the coordinates u c1i , v c1i Regarding the corresponding coordinates u of the uncorrected image, c2i , v c2i We find the coordinate u using equation (21), c2i , v c2i The pixel values ​​of the coordinate u are obtained from the image captured before correction. In addition, the composite image generation device 107 uses c2i , v c2i The pixel value is set as the pixel value of a predetermined pixel I in the corrected captured image. The composite image generation device 107 performs this setting for each pixel of the corrected captured image, thereby generating a corrected captured image with suppressed positional misalignment with the composite rendering image.

[0096] Next, we will explain the method by which the composite image generation device 107 determines the region in the corrected captured image where the composite rendering image will be composited. In the following explanation, it will be assumed that the composite rendering image will be composited in the region of the corrected captured image that is outside the display background image, in other words, the region that is outside the image display area of ​​the display device 103. Coordinates u of the corrected image c1 , v c1The corresponding world coordinate X w , Y w , Z w This is expressed by the following equation (22), which is a transformation of equation (1).

number

[0097] Z shown in equation (22) c2 Based on the virtual image display surface in the virtual space, the coordinate u is used, similar to the example described above. c1 , v c1 The coordinates u of a predetermined pixel J in the corrected image are determined according to the following. The composite image generation device 107 uses equation (22) to determine the coordinates u of a predetermined pixel J in the corrected image. c1j , v c1j The corresponding world coordinate X wj , Y wj , Z wj The composite image generation device 107 calculates the coordinate X. wj , Y wj , Z wj However, it determines whether or not the virtual image display surface is included in the area of ​​the display device 103's image display surface. The composite image generation device 107 determines the coordinate X wj , Y wj , Z wj If it is determined that the virtual image display surface is not included in the area of ​​the image display surface of the display device 103, then coordinate u c1j , v c1j The pixels are determined to be the region in the corrected captured image where the composite rendering image will be synthesized. The composite image generation device 107 performs this determination for each pixel of the corrected captured image to determine the region in the corrected captured image where the composite rendering image will be synthesized.

[0098] As described above, in this embodiment, the background image generation device 105 generates a display background image corresponding to the position and orientation of the camera 101 at the first timing (see time point 1 in Figure 5). The display control device 106 causes the display background image to be displayed on the display device 103. The composite image generation device 107 generates a composite background image. The camera 101 takes an image at the second timing (see time point 7 in Figure 5) when the display background image is displayed on the display device 103. The composite image generation device 107 then generates a composite background image based on the position and orientation of the camera 101 at the first timing and the position and orientation of the camera 101 at the second timing. In this case, when capturing images with the image displayed on the display device 103 as the background and generating a composite background image to complement the captured image, the positional misalignment between the captured image and the composite background image is suppressed.

[0099] Furthermore, in this embodiment, the composite image generation device 107 corrects the captured image taken by the camera 101. Therefore, the composite image generation device 107 can also be considered as a correction means. The composite image generation device 107 corrects the captured image based on the position and orientation of the camera 101 at a first timing and the position and orientation of the camera 101 at a second timing. In this case, when capturing an image with the image displayed on the display device 103 as the background and generating a composite rendering image that complements the captured image, the positional misalignment between the captured image and the composite rendering image is suppressed. In this case, the composite rendering image is also treated as a second background image.

[0100] Furthermore, in the imaging system 100, the process of generating a display background image corresponding to the position and orientation of the camera 101 at the first timing can also be considered as the first generation process. In addition, in the imaging system 100, the process of displaying the display background image on the display device 103 can also be considered as the display control process. Furthermore, in the imaging system 100, the process of generating a composite background image can also be considered as the second generation process. Furthermore, in the imaging system 100, the process of correcting the image captured by the camera 101 can also be considered as a correction process.

[0101] Furthermore, the composite image generation device 107 determines the position and orientation of a virtual camera located in the virtual space according to the position and orientation of camera 101 at the first timing. Then, the composite image generation device 107 generates a composite rendered image by rendering a 3D model of the virtual space according to the determined position and orientation of the virtual camera. In this case, the misalignment between the display background image generated by the background image generation device 105 and the composite rendering image is suppressed.

[0102] Furthermore, the composite image generation device 107 corrects the composite rendering image based on the difference between the position and orientation of the camera 101 at the first timing and the position and orientation of the camera 101 at the second timing to generate a composite background image. In this case, even if there is a difference between the position and orientation of camera 101 at the first timing and the position and orientation of camera 101 at the second timing, the positional misalignment between the captured image and the background image for synthesis is suppressed.

[0103] Furthermore, the correction of the composite rendering image includes performing a perspective projection transformation process on the composite rendering image according to the known position information of the display device 103. In this case, the composite rendering image is corrected according to the position of the display device 103.

[0104] Furthermore, the composite image generation device 107 combines the captured image taken by the camera 101 at the second timing with the background image for synthesis. Therefore, the composite image generation device 107 can also be considered as a synthesis means. In this case, compared to a configuration where the captured image and the background image for compositing are not combined, the content displayed in the area of ​​the captured image where the background image for compositing is combined becomes less recognizable to the user.

[0105] Furthermore, the composite image generation device 107 determines the composite region in the captured image based on the position and orientation of the camera 101 at the second timing and the known position information of the display device 103, and composites the composite background image onto the determined composite region of the captured image. In this case, the region in the captured image where the background image for synthesis is synthesized is suppressed from shifting according to the relationship between the position and orientation of the camera 101 and the position of the display device 103.

[0106] Furthermore, the composite region is the region of the captured image that corresponds to the area outside the image display area of ​​the display device 103. In this case, it is possible to provide in-camera VFX video generated by imaging that includes an area outside the image display area of ​​the display device 103 in its field of view.

[0107] Furthermore, the composite image generation device 107 combines the captured image and the composite background image in such a way that, in the intermediate region of the captured image, the ratio of the transmittance of the captured image to the transmittance of the composite background image changes in steps according to the distance from the synthesis region (see Figure 10). In this case, it becomes difficult to distinguish the boundary between the display background image and the composite background image in the captured image.

[0108] Furthermore, the composite image generation device 107 determines the composite region based on parameters related to the optical characteristics of the camera 101. Examples of parameters related to the optical characteristics of the camera 101 include the focal length. In this case, the region in the captured image where the background image for synthesis is combined is suppressed from shifting according to the optical characteristics of the camera 101.

[0109] Furthermore, the time difference between the first timing and the second timing is less than one frame of image capture by camera 101. In this case, for each frame of the in-camera VFX footage, a composite background image is generated in which the positional misalignment between the captured image and the display background image is suppressed. Therefore, it is prevented from generating captured images in which the positional misalignment with the composite background image is not suppressed.

[0110] Furthermore, the recording unit 208 of the camera 101 records information indicating the position and orientation of the camera 101 at a first timing and information indicating the position and orientation of the camera 101 at a second timing. In this case, regardless of the timing at which the background image for synthesis is generated, the earlier position and orientation information and the later position and orientation information can be provided to the composite image generation device 107.

[0111] Furthermore, the composite image generation device 107 generates a composite background image based on the focal length of the camera 101 at the first timing and the focal length of the camera 101 at the second timing. In this case, even if the focal length of the camera 101 differs between the first timing and the second timing, the positional misalignment between the display background image and the composite background image in the captured image is suppressed.

[0112] The method for suppressing the misalignment between the captured image and the background image for synthesis is not limited to the examples above. For example, taking into account that the frequencies of areas in the image where fine grayscale changes occur correspond to high frequencies, and the frequencies of areas in the image where smooth grayscale changes occur correspond to low frequencies, the composite image generation device 107 may correct the image according to the high-frequency components of the image. An example of image correction by the composite image generation device 107 in response to high-frequency components of an image is described below. In some cases, subtle variations in grayscale may occur in the display background image and the composite background image within the captured image. In this case, the relationship between the distribution of high-frequency components in the display background image and the composite background image within the captured image allows for the determination of the positional relationship between the display background image and the composite background image within the captured image. Therefore, the composite image generation device 107 detects the distribution of high-frequency components in the display background image within the captured image and the distribution of high-frequency components within the composite image generation device 107. Based on the detection results, the composite image generation device 107 corrects at least one of the captured image and the composite background image so as to suppress positional misalignment between them. In other words, the composite image generation device 107 corrects the relative positions of the captured image and the composite background image according to the detection results. Even in this case, positional misalignment between the display background image and the composite background image within the captured image is suppressed. In this case, the composite image generation device 107 can also be considered as a position correction means.

[0113] Furthermore, although this embodiment describes the imaging system 100 acquiring the preceding and succeeding camera parameters for each frame of the in-camera VFX video, it is not limited to this. For example, the camera parameters may be fixed during the shooting of the in-camera VFX video. In this case, since the camera parameters in the in-camera VFX video are single regardless of the frame, it is sufficient for the imaging system 100 to acquire this single camera parameter.

[0114] Furthermore, although this embodiment describes the boundary between the display background image and the composite background image as the outer edge of the image display area on the display device 103 within the captured image, it is not limited to this. The boundary between the display background image and the composite background image may, for example, be the contour portion of a subject located within the image display area on the display device 103 within the captured image, but in a predetermined area on the camera 101 side of the display device 103. In this case, the background image generation device 105 may reduce the load required for rendering and correction necessary for image generation by limiting the generation of background rendering images and display background images to the image display area of ​​the display device 103 that is designated as the composite area within the captured image.

[0115] Furthermore, although this embodiment has described the camera parameter acquired by the imaging system 100 as the focal length, it is not limited to this. Examples of camera parameters acquired by the imaging system 100 include information on distortion caused by the optical system 204 of the camera 101. Distortion caused by the optical system 204 includes barrel distortion and pincushion distortion, and when the focal length of the camera 101 changes, the barrel distortion and pincushion distortion may also change. In this case, as shown in equation (4), A c1 Or, as shown in equation (9) A c2 By replacing this with an equation that includes information about the distortion caused by the optical system 204, the distortion information from the optical system 204 is used as a camera parameter in image generation. Furthermore, by using the distortion information from the optical system 204 in image generation, even if there is a difference in the distortion caused by the optical system 204 between time point 1 (see Figure 5) and time point 7, the positional shift between the display background image and the composite background image in the captured image is suppressed. In this way, the composite image generation device 107 corrects at least one of the captured image and the composite background image based on parameters relating to the optical characteristics of the camera 101. In this case, even if the optical characteristics of the camera 101 change over time, the positional shift between the display background image and the composite background image in the captured image is suppressed. In this case, the composite image generation device 107 can also be considered as an optical correction means. Parameters relating to the optical characteristics of the camera 101 include the focal length of the camera 101 and distortion caused by the optical system 204 of the camera 101.

[0116] Furthermore, in this embodiment, it has been explained that the time difference between the detection timing of the position and orientation of the camera 101, which serves as the reference for generating the display background image (see time point 1 in Figure 5), and the start timing of image capture (see time point 7 in Figure 5) is less than one frame, but this is not limited to this. The time difference between time point 1 and time point 7 shown in Figure 5 may be one frame or may exceed one frame. Even in this case, the positional misalignment between the display background image and the composite background image in the captured image due to a difference in the position and orientation of the camera 101 between the detection timing of the position and orientation of the camera 101, which serves as the reference for generating the display background image, and the start timing of image capture is suppressed.

[0117] Furthermore, if the time difference between time point 1 and time point 7 shown in Figure 5 is a multiple of one frame, the start timing of image acquisition coincides with the detection timing of the position and orientation of camera 101, which serves as the reference for generating the display background image for the frame following the first frame. In this case, the later position and orientation information used for generating the composite background image for the first frame and the earlier position and orientation information used for generating the display background image for the next frame are the same information, so only one of the pieces of information needs to be managed.

[0118] Furthermore, the composite image generation device 107 may, for example, compare the display background image and the composite background image in the captured image, and correct either the display background image or the composite background image in the captured image according to the result of the comparison. The composite image generation device 107 may, for example, translate the entire composite background image so that the difference between the region of the display background image in the captured image and the corresponding composite background image becomes smaller. In this case, residual misalignments, such as background image misalignment caused by the position and orientation detection accuracy of the camera 101 by the position and orientation detection device 102, are suppressed, and the positional misalignment between the display background image and the composite background image in the captured image is further suppressed.

[0119] Furthermore, although this embodiment describes the camera 101 recording the captured image, the previous position and orientation information, the subsequent position and orientation information, the previous camera parameters, and the subsequent camera parameters, it is not limited to this. The captured image, the previous position and orientation information, the subsequent position and orientation information, the previous camera parameters, and the subsequent camera parameters may be recorded in any device in the imaging system 100. Also, the captured image, the previous position and orientation information, the subsequent position and orientation information, the previous camera parameters, and the subsequent camera parameters may each be recorded in different devices in the imaging system 100.

[0120] Furthermore, although this embodiment describes the composite image generation device 107 generating a background image for compositing and a composite image after the completion of capturing the in-camera VFX video, it is not limited to this. The composite image generation device 107 may also generate a background image for compositing and a composite image in parallel with capturing the in-camera VFX video.

[0121] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of this embodiment to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0122] This embodiment includes the following configuration. (Composition 1) An imaging system that performs imaging with an image displayed on a display device as the background, A first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing, Display control means for causing the display device to display the first background image, A second generation means for generating a second background image corresponding to the first background image, Equipped with, The imaging device captures an image at a second timing when the first background image is displayed on the display device. The imaging system is characterized in that the second generation means generates a second background image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. (Configuration 2) The imaging system according to Configuration 1, characterized in that the second generation means determines the position and orientation of a virtual camera located in the virtual space according to the position and orientation of the imaging device at the first timing, and generates a rendered image by rendering a three-dimensional model of the virtual space according to the determined position and orientation of the virtual camera. (Composition 3) The imaging system according to configuration 2, characterized in that the second generation means generates a second background image by correcting the rendering image based on the difference between the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. (Composition 4) The imaging system according to configuration 3, characterized in that the correction includes performing a perspective projection transformation process on the rendered image according to known position information of the display device. (Composition 5) The imaging system according to any one of configurations 1 to 3, further comprising a combining means for combining an image captured by the imaging device at the second timing with the second background image. (Composition 6) The imaging system according to configuration 5, characterized in that the synthesis means determines a synthesis region in the captured image based on the position and orientation of the imaging device at the second timing and known position information of the display device, and synthesizes the second background image over the determined synthesis region of the captured image. (Composition 7) The imaging system according to configuration 6, characterized in that the composite region is a region of the captured image that corresponds to a region outside the image display region of the display device. (Composition 8) The region of the captured image includes the composite region, the non-composite region where the second background image is not composited, and the intermediate region which is the region between the composite region and the non-composite region. The imaging system according to configuration 6, characterized in that the synthesis means synthesizes the captured image and the second background image in such a way that the ratio of the transmittance of the captured image to the transmittance of the second background image is changed in steps according to the distance from the synthesis region in the intermediate region of the captured image. (Composition 9) The imaging system according to the configuration described above, wherein the synthesis means determines the synthesis region based on parameters relating to the optical characteristics of the imaging device. (Composition 10) The imaging system according to any one of configurations 1 to 1, characterized in that the time difference between the first timing and the second timing is less than one frame of imaging by the imaging device. (Composition 11) An imaging system according to any one of configurations 1 to 10, further comprising recording means for recording information indicating the position and orientation of the imaging device at the first timing and information indicating the position and orientation of the imaging device at the second timing. (Composition 12) The imaging system according to any one of configurations 1 to 11, further comprising optical correction means for correcting at least one of the captured image and the second background image based on parameters relating to the optical characteristics of the imaging device. (Composition 13) The imaging system according to any one of configurations 1 to 12, further comprising position correction means for correcting the relative position between the captured image and the second background image based on the high-frequency components of the captured image and the high-frequency components of the second background image. (Composition 14) The imaging system according to any one of configurations 1 to 13, characterized in that the second generation means generates a second background image based on the focal length of the imaging device at the first timing and the focal length of the imaging device at the second timing. (Composition 15) An imaging system that performs imaging with an image displayed on a display device as the background, A first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing, Display control means for causing the display device to display the first background image, A second generation means for generating a second background image corresponding to the first background image, Correction means for correcting the captured image captured by the imaging device, Equipped with, The imaging device captures the image at a second timing when the first background image is displayed on the display device. The second generation means generates the second background image based on the position and orientation of the imaging device at the first timing, The imaging system is characterized in that the correction means corrects the captured image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. (Composition 16) A control method for an imaging system that performs imaging with an image displayed on a display device as the background, A first generation step of generating a first background image corresponding to the position and orientation of the imaging device at a first timing, A display control step that causes the display device to display the first background image, A second generation step of generating a second background image corresponding to the first background image, It has, The imaging device performs imaging at a second timing when the first background image is displayed on the display device, In the second generation step, the second background image is generated based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. A control method for an imaging system characterized by that. (Configuration 17) A control method for an imaging system that performs imaging with an image displayed on a display device as a background, A first generation step of generating a first background image corresponding to the position and orientation of the imaging device at a first timing, A display control step of displaying the first background image on the display device, A second generation step of generating a second background image corresponding to the first background image, A correction step of correcting the captured image captured by the imaging device, Having, The imaging device captures the captured image at a second timing when the first background image is displayed on the display device, In the second generation step, the second background image is generated based on the position and orientation of the imaging device at the first timing, In the correction step, the captured image is corrected based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing. A control method for an imaging system characterized by that. (Configuration 18) A program for causing a computer to function as the imaging system according to any one of Configurations 1 to 15.

[0123] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and they are not excluded from the scope of the present invention.

Explanation of Signs

[0124] 100...Imaging system, 101...Camera, 102...Position and orientation detection device, 103...Display device, 104...System control device, 105...Background image generation device, 106...Display control device, 107...Composite image generation device

Claims

1. An imaging system that performs imaging with an image displayed on a display device as the background, A first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing, Display control means for causing the display device to display the first background image, A second generation means for generating a second background image corresponding to the first background image, Equipped with, The imaging device captures an image at a second timing when the first background image is displayed on the display device. The imaging system is characterized in that the second generation means generates a second background image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

2. The imaging system according to claim 1, characterized in that the second generation means determines the position and orientation of a virtual camera located in the virtual space according to the position and orientation of the imaging device at the first timing, and generates a rendered image by rendering a three-dimensional model of the virtual space according to the determined position and orientation of the virtual camera.

3. The imaging system according to claim 2, characterized in that the second generation means generates a second background image by correcting the rendering image based on the difference between the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

4. The imaging system according to claim 3, characterized in that the correction includes performing a perspective projection transformation process on the rendered image according to known positional information of the display device.

5. The imaging system according to claim 1, further comprising a combining means for combining an image captured by the imaging device at the second timing with the second background image.

6. The imaging system according to claim 5, characterized in that the synthesis means determines a synthesis region in the captured image based on the position and orientation of the imaging device at the second timing and known position information of the display device, and synthesizes the second background image over the determined synthesis region of the captured image.

7. The imaging system according to claim 6, characterized in that the composite region is a region of the captured image that corresponds to a region outside the image display region of the display device.

8. The region of the captured image includes the composite region, the non-composite region where the second background image is not composited, and the intermediate region which is the region between the composite region and the non-composite region. The imaging system according to claim 6, characterized in that the synthesis means synthesizes the captured image and the second background image in such a way that the ratio of the transmittance of the captured image to the transmittance of the second background image is changed in steps according to the distance from the synthesis region in the intermediate region of the captured image.

9. The imaging system according to claim 6, characterized in that the synthesis means determines the synthesis region based on parameters relating to the optical characteristics of the imaging device.

10. The imaging system according to claim 1, characterized in that the time difference between the first timing and the second timing is less than one frame of imaging by the imaging device.

11. The imaging system according to claim 1, further comprising recording means for recording information indicating the position and orientation of the imaging device at the first timing and information indicating the position and orientation of the imaging device at the second timing.

12. The imaging system according to claim 1, further comprising optical correction means for correcting at least one of the captured image and the second background image based on parameters relating to the optical characteristics of the imaging device.

13. The imaging system according to claim 1, further comprising position correction means for correcting the relative position between the captured image and the second background image based on the high-frequency components of the captured image and the high-frequency components of the second background image.

14. The imaging system according to claim 1, characterized in that the second generation means generates the second background image based on the focal length of the imaging device at the first timing and the focal length of the imaging device at the second timing.

15. An imaging system that performs imaging with an image displayed on a display device as the background, A first generation means for generating a first background image corresponding to the position and orientation of the imaging device at a first timing, Display control means for causing the display device to display the first background image, A second generation means for generating a second background image corresponding to the first background image, Correction means for correcting the captured image captured by the imaging device, Equipped with, The imaging device captures the image at a second timing when the first background image is displayed on the display device. The second generation means generates the second background image based on the position and orientation of the imaging device at the first timing, The imaging system is characterized in that the correction means corrects the captured image based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

16. A control method for an imaging system that performs imaging with an image displayed on a display device as the background, A first generation step of generating a first background image corresponding to the position and orientation of the imaging device at a first timing, A display control step that causes the display device to display the first background image, A second generation step of generating a second background image corresponding to the first background image, It has, The imaging device captures an image at a second timing when the first background image is displayed on the display device. A method for controlling an imaging system, characterized in that, in the second generation step, a second background image is generated based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

17. A control method for an imaging system that performs imaging with an image displayed on a display device as the background, A first generation step of generating a first background image corresponding to the position and orientation of the imaging device at a first timing, A display control step that causes the display device to display the first background image, A second generation step of generating a second background image corresponding to the first background image, A correction step for correcting the captured image captured by the imaging device, It has, The imaging device captures the image at a second timing when the first background image is displayed on the display device. In the second generation step, the second background image is generated based on the position and orientation of the imaging device at the first timing. A method for controlling an imaging system, characterized in that, in the correction step, the captured image is corrected based on the position and orientation of the imaging device at the first timing and the position and orientation of the imaging device at the second timing.

18. A program for causing a computer to function as an imaging system according to any one of claims 1 to 15.

Images