Information processing device, system, information processing method, and program

Abstract

To reduce discomfort for a user in switching between a real camera video and a virtual viewpoint video.SOLUTION: An information processing apparatus according to an embodiment of the present invention is an information processing apparatus having: acquisition means that acquires imaging information including information representing the position and attitude of an imaging apparatus and motion information representing a specific motion of the imaging apparatus; and viewpoint information generation means that, when switching between a picked-up video acquired by the imaging apparatus and a virtual viewpoint video generated on the basis of a plurality of images, generates viewpoint information representing the position of a virtual viewpoint for generating the virtual viewpoint video and a line-of-sight direction from the virtual viewpoint on the basis of, the imaging information and motion information acquired by the acquisition means.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a technology for generating virtual viewpoint images. [Background technology]

[0002] Attention has been focused on a technology in which a plurality of imaging devices (cameras) are installed at different positions, and images are captured synchronously from multiple viewpoints, and a virtual viewpoint video is generated using the multi-viewpoint video obtained by the imaging. As such a technology, Patent Document 1 discloses a technology in which a plurality of cameras are arranged to surround a subject, and a virtual viewpoint video corresponding to a specified viewpoint (virtual viewpoint) is generated using images of the subject captured by the plurality of cameras.

[0003] Furthermore, when an event such as a soccer game captured from multiple viewpoints is also captured by a physical camera such as a broadcasting camera (hereinafter referred to as a real camera), the virtual viewpoint video and the real camera video captured by the real camera can be combined for television broadcasting or Internet distribution. Patent Document 2 discloses a technology that provides a means for matching the position, attitude, and angle of view of the virtual viewpoint video with the real camera, and seamlessly switches between the virtual viewpoint video and the real camera video. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2008-015756 A [Patent Document 2] JP 2020-42665 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 2, for example, the movement of the real camera, such as camera shake, cannot be reproduced in the virtual viewpoint video. In this way, when switching between the real camera video and the virtual viewpoint video, the movement of the real camera is not reflected, and the user is likely to recognize the difference between the real camera video and the virtual viewpoint video.

[0006] In view of the above problem, the present disclosure aims to provide a technique for reducing the sense of discomfort felt by a user when switching between a real camera image and a virtual viewpoint image. [Means for solving the problem]

[0007] One embodiment of the present invention is an information processing device characterized by having: an acquisition means for acquiring imaging information including information representing a position and attitude of an imaging device, and motion information representing camera shake of the imaging device; a first generation means for generating viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint for generating the virtual viewpoint video based on the imaging information and motion information acquired by the acquisition means, when switching between an imaging image acquired by the imaging device and a virtual viewpoint video generated based on a plurality of images; and a second generation means for generating the virtual viewpoint video that reproduces the camera shake of the imaging device based on the viewpoint information. Effect of the Invention

[0008] According to the present disclosure, it is possible to reduce the sense of discomfort felt by the user when switching between real camera footage and virtual viewpoint footage. [Brief description of the drawings]

[0009] [Figure 1] Operator type for operating the system according to the first embodiment [Diagram 2] FIG. 1 is a block diagram showing a configuration of a system according to a first embodiment. [Diagram 3] FIG. 1 is a block diagram showing a hardware configuration of a virtual camera control device 700 according to a first embodiment. [Figure 4]FIG. 1 is a block diagram showing a software configuration of a virtual camera control device 700 according to a first embodiment. [Diagram 5] Breakdown of camera parameters according to the first embodiment [Figure 6] Flow of switching from real camera image to virtual viewpoint image according to the first embodiment [Figure 7] Flow of switching from virtual viewpoint image to real camera image according to the first embodiment [Figure 8] Camera path type according to the first embodiment [Figure 9] 1 is an example of a camera movement schedule when switching from a real camera image to a virtual viewpoint image according to the first embodiment; [Figure 10] 1 is an example of a camera movement schedule when switching from a virtual viewpoint image to a real camera image according to the first embodiment; [Figure 11] Flowchart of camera movement schedule generation processing according to the first embodiment [Figure 12] Flowchart of camera path output processing according to the first embodiment [Figure 13] FIG. 11 is a block diagram showing the configuration of a system according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the concept of the present disclosure is not limited to the configurations shown in the drawings.

[0011] [First embodiment] <System configuration> An information processing system in this embodiment will be described. The information processing system in this embodiment has a function of switching and outputting an image captured by an imaging device (hereinafter referred to as a real camera) for actually capturing images, such as a broadcast camera, (hereinafter referred to as a real camera), and a virtual viewpoint image corresponding to a virtual viewpoint. The virtual viewpoint is a viewpoint specified by a user. In addition, in the following description, for convenience of explanation, a camera (hereinafter referred to as a virtual camera) virtually placed at the position of the virtual viewpoint will be used for explanation. That is, the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint correspond to the position and attitude of the virtual camera, respectively. Also, the field of view (field of view) from the virtual viewpoint corresponds to the angle of view of the virtual camera.

[0012] In addition, the virtual viewpoint video in this embodiment is also called a free viewpoint video, but is not limited to a video corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint video. In addition, in this embodiment, the case where the virtual viewpoint is specified by a user operation will be mainly described, but the virtual viewpoint may be automatically specified based on the result of image analysis, etc. In addition, in this embodiment, the case where the virtual viewpoint video is a moving image will be mainly described. It can be said that the virtual viewpoint video is a video captured by a virtual camera.

[0013] 1 is a diagram showing types of operators on an information processing system according to this embodiment. The operators who operate this system include at least three types of operators: a real camera operator who operates a real camera, a virtual camera operator who operates a virtual camera, and an image switching operator who switches between a real camera image and a virtual viewpoint image. There may be a plurality of operators of each type. Also, one operator may serve as different types of operators. Also, the operators do not need to be people, and may be, for example, programmed software.

[0014] 2 is a block diagram showing the configuration of an information processing system according to this embodiment. The system includes a time source 100, a real camera 200, a delay device 300, a switching device 400, a camera group 500, a virtual viewpoint video generation device 600, and a virtual camera control device 700.

[0015] The time source 100 outputs time information (hereinafter, referred to as "time code") to the real camera 200 and the camera group 500. The real camera 200 and the camera group 500 capture images at the same timing based on this time code, and output the images captured at the same timing to the outside.

[0016] The real camera 200 is a handheld camera of a broadcasting station placed at the side of a competition field or the like, and the position, attitude, and angle of view of the real camera 200 are controlled by a real camera operator. The real camera 200 outputs the captured image (hereinafter, described as "real camera image") and a time code to the delay device 300. The real camera 200 has built-in measuring devices (sensors) such as a GPS sensor having a receiver, a gyro sensor, and an acceleration sensor, or has such measuring devices attached to the outside. The real camera 200 outputs these sensor information, as well as imaging information including information indicating the position, attitude, and angle of view (zoom value) of the real camera (hereinafter, these are collectively referred to as "real camera parameters") to the delay device 300 together with the real camera image. Note that, although FIG. 2 shows an example in which one real camera is used, the system may have multiple real cameras. In addition, the type of real camera is not limited to a handheld camera. For example, the real camera may be a camera controlled by a pan head, a camera controlled while suspended from a wire, or a non-moving camera.

[0017] The delay device 300 stores the real camera video and the real camera parameters output from the real camera 200 in an internal memory or an external memory. The delay device 300 outputs the stored real camera video to the switching device 400 after delaying the stored real camera video by a preset predetermined time. The delay device 300 is used to adjust the timing of switching between the real camera video and the virtual viewpoint video in the switching device 400. Usually, a virtual viewpoint video having the same time code as the real camera video requires time for processing to generate the virtual viewpoint video, so a delay of several frames to several tens of seconds occurs. The delay device 300 has a role of absorbing this processing delay. The delay device 300 also outputs the stored real camera parameters and the time code corresponding to the real camera parameters to the virtual camera control device 700 at a preset predetermined timing.

[0018] The switching device 400 receives the real camera video transmitted by the delay device 300 and the virtual viewpoint video transmitted by the virtual viewpoint video generation device 600. The switching device 400 receives switching control information set by a video switching operator, and outputs either the real camera video or the virtual viewpoint video to the outside based on the received switching control information. Here, the real camera video and the virtual viewpoint video captured with the same time code are input to the switching device 400 at the same timing by the delay device 300. Therefore, the switching device 400 switches between the videos with the same time code simply by switching the video to be output. Note that, although FIG. 2 shows a form in which the delay device 300 and the switching device 400 are separate, the switching device 400 may have the delay device 300 built in. In this case, the switching device 400 receives the virtual viewpoint video with the time code transmitted by the virtual viewpoint video generation device 600, and switches between the videos with the same time code.

[0019] The camera group 500 is composed of a plurality of imaging devices installed to surround a competition field or the like. In this embodiment, the imaging devices (cameras) included in the camera group 500 are cameras different from the real cameras 200. The captured images acquired by the camera group 500 are used to generate a virtual viewpoint video. Each video captured by the camera group 500 is output to a virtual viewpoint video generating device 600. Note that the real cameras 200 may be cameras included in the camera group 500.

[0020] The virtual viewpoint video generating device 600 generates a virtual viewpoint video by creating a three-dimensional model from the multi-view video and mapping a texture according to the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint. Here, it is assumed that information on the position, attitude, and angle of view of each camera of the camera group 500 has been measured and acquired in advance, and the three-dimensional model is generated based on this information and the multi-view video. Then, the virtual viewpoint video generating device 600 outputs the virtual viewpoint video generated based on the three-dimensional model to the switching device 400.

[0021] The virtual camera control device 700 generates a virtual camera path representing the movement path of the virtual camera based on the operation information input by the virtual camera operator. The virtual camera path is information representing not only the movement path of the virtual camera position but also changes in the attitude and angle of view of the virtual camera, and is also called virtual viewpoint information. The virtual camera path in this embodiment is composed of information on the position, attitude, and angle of view of the virtual camera, and a time code. In the following description, the virtual camera path may be simply referred to as a camera path.

[0022] The virtual camera control device 700 outputs the generated virtual camera path to the virtual viewpoint video generation device 600. The virtual camera control device 700 outputs the generated virtual camera path on a frame-by-frame basis, or outputs a plurality of camera paths for several to several thousand frames at once. At this time, the virtual camera control device 700 reflects the movement of the real camera, such as hand shake, based on the real camera parameters and time code transmitted by the delay device 300 and stored in the memory in the virtual camera control device 700, in the virtual camera path. This is a main feature of this embodiment, and details will be described later. The virtual camera control device 700 may directly input the real camera parameters and time code from the real camera 200.

[0023] <Configuration of the virtual camera control device> The configuration of the virtual camera control device 700 according to this embodiment will be described below with reference to Fig. 3 and Fig. 4. Fig. 3 is a block diagram showing the hardware configuration of the virtual camera control device 700.

[0024] The virtual camera control device 700 includes a CPU 7001 , a ROM 7002 , a RAM 7003 , an auxiliary storage device 7004 , a display unit 7005 , an operation unit 7006 , a communication I / F 7007 , and a bus 7008 .

[0025] The CPU 7001 realizes each functional module of the virtual camera control device 700 shown in FIG. 4 by controlling the entire virtual camera control device 700 using computer programs and data stored in the ROM 7002 or the RAM 7003. The virtual camera control device 700 may have one or more dedicated hardware different from the CPU 7001, and at least a part of the processing by the CPU 7001 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor). The ROM 7002 stores programs that do not require modification. The RAM 7003 temporarily stores programs and data supplied from the auxiliary storage device 7004, and data supplied from the outside via the communication I / F 7007. The auxiliary storage device 7004 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

[0026] The display unit 7005 is composed of, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) for the user to operate the virtual camera control device 700. The operation unit 7006 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, and receives operations by the user and inputs various instructions to the CPU 7001. The communication I / F 7007 is used for communication between the virtual camera control device 700 and an external device of the virtual camera control device 700. For example, when the virtual camera control device 700 is connected to an external device by wire, a communication cable is connected to the communication I / F 7007. When the virtual camera control device 700 has a function of wirelessly communicating with an external device, the communication I / F 7007 has an antenna. The bus 7008 connects each part of the virtual camera control device 700 to transmit information.

[0027] In this embodiment, the display unit 7005 and the operation unit 7006 are assumed to exist inside the virtual camera control device 700, but at least one of the display unit 7005 and the operation unit 7006 may exist as a separate device outside the virtual camera control device 700. In this case, the CPU 7001 may operate as a display control unit that controls the display unit 7005 and an operation control unit that controls the operation unit 7006.

[0028] The hardware configurations of the devices other than the virtual camera control device 700 included in the information processing system according to this embodiment are similar to the configuration of the virtual camera control device 700.

[0029] 4 is a block diagram showing the functional configuration of the virtual camera control device 700. The virtual camera control device 700 has a real camera parameter input unit 7011, an operation input unit 7012, a camera movement schedule generation unit 7013, a virtual camera control unit 7014, and a camera path generation unit 7015. The camera path generation unit is also called a viewpoint information generation unit.

[0030] The real camera parameter input unit 7011 outputs the real camera parameters and time code input from outside the virtual camera control device 700 to the camera movement schedule generating unit 7013, the virtual camera control unit 7014, and the camera path generating unit 7015. The operation input unit 7012 outputs operation information of the virtual camera operator input from outside the virtual camera control device 700 to the virtual camera control unit 7014 and the camera path generating unit 7015. The operation input unit 7012 also outputs switching control information of the video switching operator input from outside the virtual camera control device 700 to the camera path generating unit 7015. The camera movement schedule generating unit 7013 generates a camera movement schedule based on the real camera parameters, and outputs the generated camera movement schedule to the camera path generating unit 7015. The camera movement schedule will be described later.

[0031] The virtual camera control unit 7014 generates a camera path for the autopilot mode based on the operation information and the real camera parameters, and outputs the generated camera path to the camera path generation unit 7015. The autopilot mode is a mode used to gradually bring the composition of the virtual viewpoint video closer to the composition of the real camera video. That is, in the autopilot mode, the difference between the position of the virtual camera and the position of the real camera is controlled to be small. Similarly, in the autopilot mode, the difference between the attitude of the virtual camera and the attitude of the real camera is controlled to be small. Similarly, in the autopilot mode, the difference between the angle of view of the virtual camera and the angle of view of the real camera is controlled to be small.

[0032] In this way, in the autopilot mode, the position, attitude, and angle of view of the virtual camera are changed so as to gradually approach the position, attitude, and angle of view of the real camera. This causes the composition of the virtual viewpoint video corresponding to the view from the virtual camera and the composition of the real camera video corresponding to the view from the real camera to gradually approach each other. The camera path generation unit 7015 generates a camera path based on various input information, and outputs the generated camera path to the virtual viewpoint video generation device 600.

[0033] <Example of movement information> 5 shows a breakdown of the real camera parameters. The real camera parameters consist of parameters such as the position, orientation, and zoom value of the real camera, as well as parameters of motion information of the real camera 200. The motion information includes (1) information related to the vibration and rotation of the real camera 200, such as the vibration amount and rotation amount of the real camera 200, (2) information related to the movement of the real camera, such as the movement direction, movement speed, and movement amount of the real camera 200, and (3) information related to zoom movement, such as the zoom value, movement direction, and movement speed.

[0034] The vibration amount of the real camera 200 is the amount of change per given time in each direction (X direction, Y direction, and Z direction) in a three-dimensional orthogonal coordinate system with the origin being a given position on the body of the real camera 200. This information is obtained, for example, from an acceleration sensor of the real camera 200.

[0035] The amount of rotation of the real camera 200 is the amount of change per given time around the X-axis, Y-axis, or Z-axis. This information is obtained from a gyro sensor of the real camera 200, for example.

[0036] The amount of movement of the real camera 200 is the amount of change caused by the movement of the housing of the real camera 200. As this amount of change, a value obtained by adding and smoothing the amount of vibration, amount of rotation, etc. described above within a predetermined time period may be used. Alternatively, the amount of movement of the real camera 200 may be measured using a GPS sensor.

[0037] The zoom value (so-called zoom value) is the focal length value of the real camera 200. The zoom movement direction refers to the wide-angle direction or the telephoto direction. The zoom movement speed refers to the amount of change in focal length over a given period of time. These pieces of information are obtained, for example, from a sensor built into the lens.

[0038] The motion information does not have to include all of the above information, and may include at least any of the information (1) to (3), for example. The motion information may also include information other than the above information.

[0039] As described above, the motion information in this embodiment is information that represents vibration, camera shake, positional deviation, zoom deviation, and the like of the real camera. However, the real camera 200 is a camera that can control the position, attitude, and angle of view, and the motion of the real camera due to this control may differ from the motion represented by the motion information described above. Therefore, the motion of the real camera represented by the motion information is a specific motion that satisfies, for example, the following conditions. That is, the motion represented by the motion information may be the motion of the real camera observed while no control of the position, attitude, zoom, and the like of the real camera is being performed. Furthermore, the motion represented by the motion information may be a motion in which the amount of change in the position, attitude, and zoom of the real camera in a given time is smaller than a threshold value.

[0040] <Video switching flow> 6 and 7 show an overview of the video switching according to this embodiment. First, the flow of switching from a real camera video to a virtual viewpoint video will be described with reference to FIG.

[0041] Switching instruction timing T101 is the timing at which the video switching operator inputs switching control information to the switching device 400. Switching execution timing T102 is the timing at which the switching device 400, having received the switching instruction, actually executes switching from the real camera video to the virtual viewpoint video. Switching from the real camera video to the virtual viewpoint video may be executed immediately upon receiving the switching instruction from the video switching operator, or may be executed after a predetermined time has elapsed after receiving the switching instruction. Note that FIG. 6 shows an example of immediate execution.

[0042] The real camera images F101 to F107 are real camera images input from the delay device 300 to the switching device 400 for each frame.

[0043] The virtual viewpoint images F201 to F207 are input from the virtual viewpoint image generation device 600 to the switching device 400 for each frame.

[0044] The output videos F301 to F307 are videos selectively output from either the real camera or virtual viewpoint video by the switching device 400. In Fig. 6, the real camera videos F101 to F102 are selected as the output videos F301 to F302. Furthermore, after switching execution timing T102, the virtual viewpoint videos F203 to F207 are selected as the output videos F303 to F307 in response to switching execution.

[0045] 6, virtual camera camera paths P101 to P107 indicate the types of camera paths that virtual camera control device 700 outputs to virtual viewpoint video generation device 600. In FIG.

[0046] Here, the types of virtual camera paths will be described. The types of virtual camera paths are shown in Fig. 8. In this embodiment, four camera paths, 1st to 4th, are used. These camera paths are referred to as camera paths 1 to 4 in this specification and Fig. 8, and are represented as (1) to (4) in Figs. 6 and 7.

[0047] Camera path 1 is a virtual camera path that represents operation by a real camera operator. When camera path 1 is adopted, the composition of the virtual viewpoint image is the same as the composition of the real camera image. Camera path 2 is a virtual camera path that represents operation by a virtual camera operator plus operation according to a camera movement schedule. Camera path 3 is a virtual camera path that represents operation by a virtual camera operator. Camera path 4 is a virtual camera path that represents operation by an autopilot mode plus operation according to a camera movement schedule.

[0048] References P101 to P107 in FIG. 6 indicate the transition of the virtual camera path in this example. First, the virtual camera path is switched from camera path 1 (P102) to camera path 2 (P103) upon switching execution. This switching timing is the timing when the output image is switched from the real camera image (output image F302) to the virtual viewpoint image (output image F303), but since the composition of the real camera image and the virtual viewpoint image match, there is no change in appearance. After the switching execution timing T102, the virtual viewpoint image becomes an image based on camera path 2. As described above, camera path 2 represents the operation of the virtual camera operator plus the operation according to the camera movement schedule, so the movement information of the camera vibration, movement, zoom operation, etc. by the real camera operator is taken over. Vibration such as hand shake is gradually alleviated thereafter. The camera movement schedule is a schedule of this alleviation, and the vibration amount, etc. is determined in advance for each frame. When the vibrations converge, that is, when the camera motion schedule ends, the camera path is switched from camera path 2 (P105) to camera path 3 (P106). This means that the camera path is operated only by the virtual camera operator.

[0049] Next, the flow of switching from the virtual viewpoint video to the real camera video will be described with reference to FIG.

[0050] The switching instruction timing T201 is the timing when the video switching operator inputs switching control information to the switching device 400. The switching execution timing T202 is the timing when the switching device 400, which has received the switching instruction, actually executes switching from the virtual viewpoint video to the real camera video. Usually, the composition of the virtual viewpoint video and the composition of the real camera video are different. Therefore, in order to realize a smooth switching, it is necessary to gradually bring the composition of the virtual viewpoint video closer to the composition of the real camera video in the virtual viewpoint video and finally match them. Therefore, the switching instruction timing T201 and the switching execution timing T202 are different. Specifically, the switching execution timing T202 is later than the switching instruction timing T201.

[0051] The real camera images F401 to F407 are real camera images input from the delay device 300 to the switching device 400 for each frame.

[0052] The virtual viewpoint images F501 to F507 are virtual viewpoint images input from the virtual viewpoint image generation device 600 to the switching device 400 for each frame.

[0053] The output videos F601-F607 are videos obtained by selectively outputting either the real camera video or the virtual viewpoint video by the switching device 400. In Fig. 7, the virtual viewpoint videos F501-F505 are selected as the output videos F601-F605. Furthermore, after a certain amount of time has elapsed following switching execution timing T202, which is the trigger for switching execution, the real camera videos F406-F407 are selected as the output videos F606-F607.

[0054] Reference characters P201 to P207 in FIG. 7 indicate the transition of the virtual camera path output by the virtual camera control device 700 to the virtual viewpoint video generation device 600. First, the virtual camera path is switched from camera path 3 (P202) to camera path 4 (P203) in response to a switching instruction at switching instruction timing T201. As described above, camera path 4 represents an operation in the autopilot mode, and this switching causes the composition of the virtual viewpoint video to gradually approach the composition of the real camera video. In addition, operation information represented by a camera movement schedule is also added to camera path 4. Therefore, movement information such as camera shake occurring in the real camera video is gradually added to the information for generating the virtual viewpoint video.

[0055] Next, the virtual camera path is switched from camera path 4 (P205) to camera path 1 (P206) upon switching execution at switching execution timing T202. This switching timing is the timing when the output video is switched from the virtual viewpoint video (output video F605) to the real camera video (output video F606), but since the compositions of the real camera video and the virtual viewpoint video match, there is no change in appearance. Also, at this timing, the motion information added to the information for generating the virtual viewpoint video by the camera motion schedule is the same as the motion information of the real camera video. Therefore, no strangeness in appearance is caused. Thereafter, the composition of the virtual viewpoint video will be the same as the video based on camera path 1, that is, the composition of the real camera video.

[0056] Note that the composition of the virtual viewpoint image and the composition of the real camera image at the time of switching may not be completely the same. For example, the composition of the virtual viewpoint image at the time of switching may be controlled so as to satisfy a predetermined criterion. The predetermined criterion may be, for example, that the difference between the size of an object included in the virtual viewpoint image and the size of the same object included in the real camera image is smaller than a threshold value. In addition, the predetermined criterion may be that the difference in position, attitude, and angle of view between the virtual camera and the real camera is smaller than a threshold value.

[0057] <Example of camera movement schedule> Fig. 9 shows an example of a camera movement schedule used when switching from a real camera image to a virtual viewpoint image. Specifically, Fig. 9(a) shows how the vibration amount (or rotation amount) on the virtual viewpoint image changes (attenuates) in the time direction. Note that Fig. 9 shows only one direction of the XYZ orthogonal coordinates in a simplified manner.

[0058] The camera movement schedule is updated by acquiring real camera parameters at every frame period while the real camera video is being output from the switching device 400 (F301 to F302 in FIG. 6). Therefore, immediately after switching to the virtual viewpoint video in response to switching execution at switching execution timing T102 (F303 in FIG. 6), a value that is approximately the same as the vibration amount of the real camera 200 immediately before and a value that is approximately the same as the rotation amount of the real camera 200 are reproduced on the virtual viewpoint video. After switching to the virtual viewpoint video, the update of the camera movement schedule stops, and thereafter, the operation of the vibration amount and rotation amount of each frame is added to the operation of the virtual camera operator according to the camera movement schedule. Each of these vibration amount and rotation amount is gradually attenuated by a predetermined coefficient (F303 to F305 in FIG. 6). Then, when the camera movement schedule ends, that is, when the vibration and rotation have settled, the operation is switched to only the operation by the virtual camera operator (F306, P106 in FIG. 6). In order to provide a steady state in which a small amount of normal vibration remains in the camera image, a camera movement schedule may be adopted that always leaves a minimum constant amount of vibration.

[0059] Fig. 9(b) shows another example of the camera movement schedule, showing how the viewpoint movement speed (or zoom movement speed) on the virtual viewpoint image changes (decreases) in the time direction. The operation of the camera movement schedule is the same as that of Fig. 9(a).

[0060] Fig. 10 shows an example of a camera movement schedule used when switching from a virtual viewpoint image to a real camera image. Specifically, Fig. 10(a) shows how the vibration amount (or rotation amount) on the virtual viewpoint image changes (amplifies) in the time direction. Note that Fig. 10 shows only one direction of the XYZ orthogonal coordinates in a simplified manner.

[0061] The camera movement schedule is updated by acquiring real camera parameters every frame until a switching instruction is input at switching instruction timing T201 (F601 to F602 in FIG. 7). The waveform shape of this camera movement schedule is the reverse of the shape when switching from the real camera video to the virtual viewpoint video (see FIG. 9).

[0062] Then, at switching execution timing T201 (after F603 in FIG. 7), the vibration and rotation amounts on the virtual viewpoint image are reproduced so as to gradually approach those of the real camera 200 according to the camera movement schedule. Then, when the vibration and rotation amounts become substantially the same as those of the real camera 200, switching is made to the real camera image (F606 in FIG. 7).

[0063] Fig. 10(b) shows another example of the camera movement schedule, showing how the viewpoint movement speed (or zoom movement speed) on the virtual viewpoint image changes (increases) in the time direction. The operation of the camera movement schedule is the same as that of Fig. 10(a).

[0064] <Camera movement schedule generation flow> FIG. 11 is a flowchart showing the flow of the process of generating a camera movement schedule according to this embodiment.

[0065] In step S1101, the camera movement schedule generating unit 7013 acquires the actual camera parameters transmitted by the actual camera parameter input unit 7011. Hereinafter, "step S~" will be abbreviated to "S~".

[0066] In S1102, the camera movement schedule generating unit 7013 generates schedule information regarding the vibration amount and rotation amount of the virtual camera.

[0067] In S1103, the camera movement schedule generating unit 7013 generates schedule information for the moving speed of the virtual camera.

[0068] In S1104, the camera movement schedule generating unit 7013 generates schedule information for the zoom movement speed of the virtual camera.

[0069] The order of generation in S1102 to S1104 may be reversed, and each step may be executed in parallel.

[0070] In S1105, the camera movement schedule generation unit 7013 outputs the schedule information generated in S1102 to S1104 to the camera path generation unit 7015.

[0071] The processes from S1101 to S1105 are performed in one frame cycle. Thereafter, unless the CPU 7001 determines in S1106 that an end instruction has been issued, the processes from S1101 are repeated.

[0072] <Camera path output flow> FIG. 12 is a flowchart showing the flow of the camera path output process according to this embodiment.

[0073] At S1201, the camera path generation unit 7015 acquires operation information. At S1202, the camera path generation unit 7015 acquires actual camera parameters. At S1203, the camera path generation unit 7015 acquires camera movement schedule information. At S1204, the camera path generation unit 7015 acquires a camera path for automatic control. Note that the order of S1201 to S1204 may be reversed, or S1201 to S1204 may be executed in parallel.

[0074] In S1205, the camera path generation unit 7015 determines whether the image currently being output by the switching device 400 is a real camera image. If the determination result in this step is true, the process proceeds to S1206, whereas if the determination result is false (i.e., if the image currently being output by the switching device 400 is a virtual viewpoint image), the process proceeds to S1211.

[0075] In S1206, the camera path generation unit 7015 determines whether switching from the real camera image to the virtual viewpoint image is being performed (see T102 in FIG. 6). If the determination result in this step is true, the process proceeds to S1208, whereas if the determination result is false, the process proceeds to S1207.

[0076] If switching from the real camera video to the virtual viewpoint video has not yet been executed, the camera path generation unit 7015 outputs camera path 1 to the virtual viewpoint video generation device 600 in S1207.

[0077] If switching from the real camera image to the virtual viewpoint image has been completed, in S1208, the camera path generation unit 7015 determines whether the schedule execution according to the camera movement schedule is in progress. If the determination result in this step is true, the process proceeds to S1209, whereas if the determination result is false, the process proceeds to S1210.

[0078] In S1209, the camera path generation unit 7015 outputs the camera path 2 to the virtual viewpoint video generation device 600.

[0079] In S1210, the camera path generation unit 7015 outputs the camera path 3 to the virtual viewpoint video generation device 600.

[0080] In S1211, the camera path generation unit 7015 determines whether an instruction to switch from the virtual viewpoint video to the real camera video has been issued (see T201 in FIG. 7). If the determination result in this step is true, the process proceeds to S1213, whereas if the determination result is false, the process proceeds to S1212.

[0081] In S1212, the camera path generation unit 7015 outputs the camera path 3 to the virtual viewpoint video generation device 600.

[0082] If a switching instruction has been issued (YES in S1211), in S1213, the camera path generation unit 7015 determines whether or not the schedule is being executed according to the camera movement schedule. If the result of the determination in this step is true, the process proceeds to S1214, whereas if the result of the determination is false, the process proceeds to S1215.

[0083] In S1214, the camera path generation unit 7015 outputs camera path 4 to the virtual viewpoint video generation device 600.

[0084] In S1215, the camera path generation unit 7015 outputs the camera path 1 to the virtual viewpoint video generation device 600.

[0085] The camera path generating unit 7015 performs the processes of S1201 to S1215 in one frame cycle unless there is a particular instruction to end the process in S1216.

[0086] Also, immediately after determining in S1206 that switching is to be performed, the camera path generation unit 7015 stops acquiring camera movement schedule information for switching from the real camera image to the virtual viewpoint image in S1203. Alternatively, the camera movement schedule information is acquired but the camera movement schedule is not updated. Similarly, immediately after determining in S1211 that a switching instruction has been issued, the camera path generation unit 7015 stops acquiring camera movement schedule information for switching from the virtual viewpoint image to the real camera image in S1203.

[0087] This embodiment can reduce the sense of incongruity felt after switching between the real camera and the virtual camera, and can provide the viewer with a more realistic video experience. In this embodiment, the real camera 200 and the camera group 500 are real cameras, but they may be replaced with an information processing device (specifically, a computer) that simulates the cameras.

[0088] [Second embodiment] In the first embodiment, the real camera parameters are generated using information from measuring devices such as a gyro sensor and an acceleration sensor that are built into the real camera 200 or provided outside the real camera 200. In contrast, in this embodiment, the information processing system has a means for estimating some of the real camera parameters, separate from the real camera 200.

[0089] <System configuration> 13 is a block diagram showing the configuration of an information processing system according to this embodiment. This system includes a time source 100, a real camera 200, a delay device 300, a switching device 400, a camera group 500, a virtual viewpoint video generating device 600, a virtual camera control device 700, and an image analysis device 800. The components other than the image analysis device 800 are the same as those in the first embodiment, but the real camera 200 of this embodiment does not need to include a measuring device for generating real camera parameters.

[0090] The real camera video transmitted by the delay device 300 is input to the image analysis device 800. The image analysis device 800 derives motion information of the real camera 200 based on the input real camera video, and outputs the derived motion information to the virtual camera control device 700. The image analysis device 800 may be integrated with the virtual camera control device 700.

[0091] The motion information output by the image analysis device 800 is all or part of the information described in the first embodiment. The image analysis device 800 has a buffer that accumulates several frames of real camera video, analyzes the video difference between frames to derive motion information, and outputs the derived motion information to the virtual camera control device 700 in one frame cycle. The video difference between frames is derived based on, for example, the amount of movement and direction of movement of pixel values ​​in the video between frames, or the amount of movement and direction of movement of the position of an object in the video between frames. In addition, when deriving the motion difference, the amount of movement and direction of movement of a feature point in the video are derived using a field line, a structure, or the like, whose position does not change in real space, as a feature point. Note that other methods may be used to derive the motion difference.

[0092] As described above, according to this embodiment, even if the real camera 200 cannot output motion information, it is possible to obtain the motion information based on video analysis, thereby achieving the same effects as in the first embodiment.

[0093] [Third embodiment] In the first embodiment, the real camera 200 outputs real camera parameters, and the output real camera parameters are input to the virtual camera control device 700 via the delay device 300, thereby reproducing the movement of the real camera image on the virtual viewpoint image. In contrast, in this embodiment, the real camera 200 outputs information other than the movement information, and the output information is input to the virtual camera control device 700. The information other than the movement information includes, for example, at least one of the lens distortion value of the real camera 200 and information indicating the focus state of the real camera 200.

[0094] The lens distortion value is uniquely determined by the type and focal length of the lens. Therefore, the real camera parameters output by the real camera 200 according to this embodiment include lens distortion information as well as the zoom value. When outputting a virtual camera path to the virtual viewpoint image generation device 600, the virtual camera control device 700 also outputs the lens distortion information. Then, the virtual viewpoint image generation device 600 reproduces the lens distortion indicated by the lens distortion information on the virtual viewpoint image.

[0095] The focus state is, for example, information that indicates the degree of focus on a real camera image. Specifically, the value of the focus degree is set in advance to be divided into a predetermined number of stages, and the real camera parameters include the set value. For example, the out-of-focus state is set to 0, the in-focus state is set to 10, and the focus state is expressed in stages by integer values ​​1 to 9 between them. When outputting a camera path to the virtual viewpoint image generating device 600, the virtual camera control device 700 also outputs information on the focus state. The virtual viewpoint image generating device 600 reproduces the same degree of blur as the real camera image by performing image processing such as blurring on the virtual viewpoint image based on the information on the focus state.

[0096] Regarding the lens distortion value, the virtual camera control device 700 may have a table in advance in which distortion values ​​for each lens focal length are stored, and the table may be used to determine the distortion value associated with the focal length indicated by the focal length information in the real camera parameters.

[0097] According to this embodiment, it is possible to reproduce the same lens distortion, blur, and the like as with a real camera in a virtual viewpoint image.

[0098] [Other embodiments] The present disclosure can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions. The contents of the above-mentioned embodiments may be used in appropriate combination. [Explanation of symbols]

[0099] 700: Virtual camera control device 7011: Real camera parameter input section 7015: Camera path generation unit

Claims

1. an acquisition means for acquiring imaging information including information representing a position and an orientation of an imaging device, and motion information representing camera shake of the imaging device; a first generating means for generating viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint for generating the virtual viewpoint video, based on the imaging information and the motion information acquired by the acquiring means, when switching between an image captured by the imaging device and a virtual viewpoint video generated based on a plurality of images; a second generating means for generating the virtual viewpoint image that reproduces the camera shake of the imaging device based on the viewpoint information; 13. An information processing device comprising:

2. The imaging information further includes information indicating a zoom value.

2. The information processing apparatus according to claim 1,

3. the motion information includes at least one of information related to the rotation of the imaging device, information related to the movement of the imaging device, and information related to the zoom movement of the imaging device; 3. The information processing apparatus according to claim 2.

4. the motion information includes an amount of camera shake of the imaging device; the information regarding the rotation of the imaging device includes an amount of rotation of the imaging device; 4. The information processing apparatus according to claim 3.

5. The information regarding the movement of the imaging device includes a movement amount of the imaging device in each direction of a three-dimensional orthogonal coordinate system.

5. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

6. The information related to the zoom movement of the imaging device includes at least one of a zoom value, a movement direction, and a movement speed of the imaging device.

6. The information processing apparatus according to claim 3, wherein the information processing apparatus further comprises:

7. The camera shake of the imaging device includes a movement observed in a state where the position and the orientation of the imaging device are not controlled to be changed.

7. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises: a first input section;

8. The camera shake of the imaging device includes a movement in which the amount of change in the position and orientation of the imaging device is smaller than a threshold.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

9. the acquisition means acquires a plurality of images used to generate the virtual viewpoint video; Further, the image pickup apparatus includes an output unit for outputting either the captured image or the virtual viewpoint image.

9. The information processing apparatus according to claim 1, wherein the information processing apparatus further comprises:

10. The output means outputs the viewpoint information generated by the first generation means to the second generation means for each frame.

10. The information processing apparatus according to claim 9,

11. The viewpoint information transmitted for each frame is any one of first viewpoint information representing an operation by a first operator who operates the imaging device, second viewpoint information representing an operation by a second operator who operates the virtual viewpoint plus an operation according to the movement information, third viewpoint information representing an operation by the second operator, and fourth viewpoint information representing an operation by an autopilot mode that performs control so as to reduce a difference between a composition of the virtual viewpoint image and a composition of the captured image plus an operation according to the movement information.

11. The information processing apparatus according to claim 10,

12. When switching from the captured image to the virtual viewpoint image is executed, the output means outputs the virtual viewpoint image; In the output virtual viewpoint image, the camera shake of the imaging device before the switching is executed is reproduced.

12. The information processing device according to claim 9, wherein the information processing device is a device for processing information.

13. When an instruction to switch from the virtual viewpoint image to the captured image is received, the output means first outputs the virtual viewpoint image; In the output virtual viewpoint image, the camera shake of the imaging device after the switching is gradually reproduced.

13. The information processing device according to claim 9, wherein the information processing device is a device for processing information.

14. The acquisition means acquires virtual viewpoint operation information for a virtual viewpoint video generated based on a plurality of images, the viewpoint information is generated by adding a camera shake of the imaging device specified by the movement information to a position and orientation of a virtual viewpoint specified by the operation information.

14. The information processing apparatus according to claim 1, wherein the information processing apparatus comprises:

15. an acquisition step of acquiring imaging information including information representing a position and an orientation of an imaging device, and motion information representing camera shake of the imaging device; a first generation step of generating viewpoint information representing a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint for generating the virtual viewpoint video, based on the imaging information and the movement information acquired in the acquisition step, when switching between an image captured by the imaging device and a virtual viewpoint video generated based on a plurality of images; a second generation step of generating the virtual viewpoint image that reproduces the camera shake of the imaging device based on the viewpoint information; 13. An information processing method comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images