Information processing device, information processing method, and program

Abstract

It was difficult for one operator to control both the virtual and real cameras. [Solution] The display control device 130 includes setting means for setting the position and orientation of a virtual camera based on user operation, and output means for outputting information indicating the orientation of a second real camera, which is different from the first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the position and orientation of the virtual camera.

Description

Technical Field

[0001] The present invention relates to a method for controlling a virtual camera corresponding to a virtual viewpoint image and a real camera arranged in a real space.

Background Art

[0002] In recent years, a technique has been attracting attention in which a plurality of cameras are installed at different positions and synchronized shooting is performed from multiple viewpoints, and a virtual viewpoint image captured from an arbitrary viewpoint is generated using a plurality of captured images obtained by the shooting.

[0003] When distributing a virtual viewpoint image, there is a demand to switch and distribute the virtual viewpoint image and a captured image acquired from a camera installed in a real space (hereinafter referred to as a real camera). In performing such distribution, it is necessary to appropriately control a virtual camera corresponding to the virtual viewpoint image and the real camera.Means for Solving the Problems In Patent Document 1, a technique is disclosed in which when switching between a virtual viewpoint image and a captured image, the position and orientation of the virtual camera are controlled in accordance with the position and orientation of the real camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, in order to reduce the cost related to distribution, it has been required that an operator who operates a virtual camera also operates a real camera. However, it is not easy for one operator to operate both a virtual camera and a real camera.

[0006] The present disclosure aims to facilitate the control of both a virtual camera and a real camera by one operator.

Means for Solving the Problems

[0007] An information processing apparatus according to one aspect of the present disclosure has the following configuration: namely, setting means for setting the gaze point of a virtual camera based on user operation, and output means for outputting information indicating the orientation of a second real camera, which is different from a first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the gaze point of the virtual camera. [Effects of the Invention]

[0008] According to this disclosure, it becomes easier for one operator to control both virtual and real cameras. [Brief explanation of the drawing]

[0009] [Figure 1] This is a diagram showing an image processing system. [Figure 2] This figure shows an example of how to install an imaging device. [Figure 3] This diagram shows the hardware configuration of the display control device. [Figure 4] This figure shows the functional configuration of the display control device in the first embodiment. [Figure 5] This figure shows the operation flow of the display control device in the first embodiment. [Figure 6] This is a conceptual diagram of the first embodiment. [Figure 7] This figure shows the functional configuration of the display control device in the second embodiment. [Figure 8] This figure shows the operation flow of the display control device in the second embodiment. [Figure 9] This is a conceptual diagram of the second embodiment. [Figure 10] This figure shows the functional configuration of the display control device in the third embodiment. [Figure 11] This figure shows the operation flow of the display control device in the third embodiment. [Figure 12] This is a conceptual diagram of the third embodiment. [Modes for carrying out the invention]

[0010] <Embodiment> According to a preferred embodiment of the present disclosure, the information processing device has setting means for setting the position and orientation of a virtual camera based on user operation. The information processing device also has output means for outputting information indicating the orientation of a second real camera, which is different from the orientation of a first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the position and orientation of the virtual camera. The orientation of the second real camera may be set so that the imaging range of the second real camera overlaps with the imaging range specified from the position and orientation of the virtual camera. Alternatively, the orientation of the second real camera may be set so that the optical axis of the second real camera intersects with the optical axis of the second real camera, which is specified from the position and orientation of the virtual camera. Alternatively, the orientation of the second real camera may be set so that the optical axis of the second real camera passes through the point of fixation of the virtual camera. The point of fixation of the virtual camera is also called the gaze position of the virtual camera.

[0011] Furthermore, the position and orientation of the first physical camera do not change based on the gaze point of the virtual camera, while the orientation of the second physical camera changes based on the gaze point of the virtual camera, but its position does not change. Specifically, the first physical camera is a physical camera installed to generate a virtual viewpoint image, and its position and orientation do not change. The first physical camera may be one physical camera or multiple physical cameras. If the first physical camera is multiple physical cameras, these multiple physical cameras perform shooting in sync. The second physical camera is a camera whose pan, tilt, and zoom can be controlled remotely. In other words, the second physical camera's position is fixed and does not change, but its orientation can be changed remotely.

[0012] In this configuration, the user can change the orientation of the real camera by setting the position of the virtual camera's point of focus. In other words, one operator can easily control both the virtual camera and the real camera.

[0013] Further, the setting means sets the distance from the virtual camera to the fixation point of the virtual camera based on a user operation. For example, the user inputs the distance from the virtual camera to the fixation point. Further, the fixation point of the virtual camera is located on the optical axis of the virtual camera.

[0014] Further, the second real camera images an area different from the area where the 3D model of the subject is generated by the captured image of the first real camera.

[0015] In this aspect, the second real camera images an area different from the area where the virtual viewpoint image can be generated based on the 3D model of the subject, and it becomes possible to distribute videos of more various types of scenes. For example, the area where the 3D model of the subject can be generated is an area included in the imaging area of the first real camera. In other words, the 3D model of the subject cannot be generated for an area that cannot be imaged by the first real camera. By the second real camera imaging an area different from the area where the 3D model of the subject can be generated, that is, the area where the virtual viewpoint image can be generated, it becomes possible to distribute an image obtained by imaging a larger area. Therefore, by photographing and distributing sports or events in which competitions are held using a larger area, the viewing value can be improved.

[0016] Further, the virtual viewpoint image is generated based on the captured image of the first real camera and the virtual camera.

[0017] According to another preferred embodiment of the present embodiment, the information processing method includes a setting step of setting the fixation point of the virtual camera based on a user operation. Further, the information processing method includes an output step of outputting information indicating the posture of a second real camera different from the first real camera used to generate a virtual viewpoint image corresponding to the virtual camera based on the fixation point of the virtual camera.

[0018] According to another preferred embodiment of the present embodiment, the program causes a computer to function as the above-described information processing apparatus.

[0019] <Example> The embodiments of this disclosure will be described below with reference to the drawings. The components described in the following embodiments are merely examples of embodiments, and this disclosure is not limited to them.

[0020] <Example 1> Figure 1 shows an image processing system 100 according to this embodiment. The image processing system 100 includes a plurality of imaging devices 110, an image generation device 120, a display control device 130, a display 160, and an imaging device 150. The imaging devices 110, image generation device 120, display control device 130, and imaging device 150 are connected via communication cables such as LAN (Local Area Network) cables. In this embodiment, the communication cables are assumed to be LAN cables, but the communication cables are not limited to this embodiment. The image generation device 120 and the display 160 are connected via a video signal transmission cable.

[0021] The imaging devices 110 and 150 are, for example, digital cameras capable of capturing images (still images and videos). Figure 2 shows an example of the installation of the imaging devices 110 and 150.

[0022] The imaging device 110 is installed to surround a specific area in a stadium or similar location, and captures images (videos) of subjects within that area. The captured images are transmitted from the imaging device 110 to the image generation device 120. The imaging device 110's installation position, orientation, zoom, etc., are pre-set and are not changed when capturing subjects.

[0023] The imaging device 150 is installed around the imaging device 110 and positioned to capture images of the same subject as the imaging device 110. The installation position of the imaging device 150 is predetermined, and its posture, zoom, etc., can be changed when capturing a subject. The imaging device 150's posture, zoom, etc., can be controlled from the display control device 130 via a communication cable. The captured images are displayed on the display 160 via the communication cable. The imaging device 150 is a camera whose pan, tilt, and zoom can be controlled remotely, for example, a network camera. Since the imaging device 150 can change its posture and zoom during shooting, it can capture a different area than the imaging device 110. Because it can capture a different area than the imaging device 110 in this way, the area that can be captured by multiple imaging devices 110 and imaging device 150 is expanded, making it possible to display images showing sports or events that take place over a wide area.

[0024] The display 160 displays a virtual viewpoint image generated by the image generation device 120 or an image captured by the imaging device 150. The display 160 may also be a display device used by an operator to control a virtual camera, or a display device for viewers to view the virtual viewpoint image or the captured image. Furthermore, the display 160 may display both the virtual viewpoint image and the captured image.

[0025] The image generation device 120 stores the captured images obtained by the imaging device 110, and when virtual viewpoint information and playback time information are input by user operation of the display control device 130, it generates a virtual viewpoint image based on the captured image and the virtual viewpoint. Here, the virtual viewpoint information is information that indicates the three-dimensional position and angle of a virtual viewpoint (virtual viewpoint) in a virtual space constructed from the captured image, as well as the point of focus position that indicates which position is being photographed. The virtual viewpoint information includes the relative position to a predetermined origin position, such as the center of the stadium where the image was taken, that is, predetermined positions in front, behind, left, right, and up and down relative to the origin position. It also includes direction information in the direction from the predetermined position, that is, angle information with respect to the front, behind, left, right, and up and down axes. In other words, the direction from the predetermined position is information that indicates the current posture when a certain posture of the virtual camera is used as a reference. Furthermore, it shall include at least the front, behind, left, right, and up and down position information of the point of focus position, which is the position being photographed (focused on) by the virtual viewpoint, relative to the origin position.

[0026] The location information will be set to a predetermined origin point, such as the center of a stadium. The X-axis will be in the east-west direction with east being positive, the Y-axis in the north-south direction with north being positive, and the Z-axis in the up-down direction with up being positive. The unit will be meters (m), etc.

[0027] The playback time information consists of the time of capture of the image, including hours, minutes, seconds, and the number of frames per second. By specifying this playback time, the scene recorded at that time will be generated as a virtual viewpoint. For example, the number of frames per second may be set to 60.

[0028] The image generation device 120 is, for example, a server device equipped with database and image processing functions. The database stores images taken in advance of scenes where no subjects are present, such as before the start of subject photography in the stadium, as background images via the imaging device 110. In scenes where subjects are present, the foreground subject is separated as a specific object image through image processing. The specific object may be not only a person but also an object with a predetermined image pattern, such as a ball or other equipment.

[0029] The virtual viewpoint image corresponding to the virtual viewpoint information shall be generated from a background image and a specific object image managed in a database. For example, Model-Based Rendering (MBR) may be used as the virtual viewpoint image generation method. MBR is a method of generating a virtual viewpoint image using a three-dimensional shape generated based on multiple images taken of a subject from multiple directions. Specifically, it is a technique that uses the three-dimensional shape (model) of the target scene obtained by three-dimensional shape reconstruction methods such as the viewing volume cross-section method and Multi-View-Stereo (MVS) to generate an image of how the scene looks from a virtual viewpoint. Note that rendering methods other than MBR may also be used to generate the virtual viewpoint image. The generated virtual viewpoint image is transmitted to the display 160 via a video signal transmission cable.

[0030] The display control device 130 is, for example, a PC (Personal Computer) or a tablet. The viewpoint controller 131 is, for example, a mouse, keyboard, 6-axis controller, or touch panel, and the time controller 132 is, for example, an operating device equipped with a rotary dial. The user operates the viewpoint controller 131 and the time controller 132.

[0031] The display control device 130 also converts user operations from the connected viewpoint controller 131 and time controller 132 into virtual viewpoint movement instruction information (instructions regarding the amount and direction of movement) and playback time information, and transmits them to the image generation device 120.

[0032] Furthermore, the instructions for moving the virtual viewpoint and playback time are not limited to continuous movement; it is also possible to move to predetermined virtual viewpoints, such as the front position, back position, or top-down position of the subject in the virtual space. Additionally, by pre-setting the playback time, it is possible to instantly move to that time. The display control device 130 also displays objects in three-dimensional space based on user operations via an application displayed on the display 160 through the execution of a control program described later.

[0033] Furthermore, the display control device 130 acquires the virtual viewpoint image generated by the image generation device 120 and the captured image generated by the imaging device 150, selects the image to display on the display 160, and outputs the selected image. Alternatively, both images may be output by selecting both. Alternatively, this process may be omitted by the display control device 130, and a switcher (not shown) may be provided. This function allows for the distribution of the selected image.

[0034] Figure 3 shows the hardware configuration of the display control device 130.

[0035] The display control device 130 includes a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307. The CPU 301 reads control programs stored in the ROM 302 and executes various processes. The RAM 303 is used as the CPU 301's main memory, work area, and other temporary storage areas. The HDD 304 stores various data and programs. The display unit 305 displays various information. The input unit 306 can be connected to a keyboard, mouse, 6-axis controller, etc., and accepts various operations from the user. The communication unit 307 performs communication processing with external devices via a network. Ethernet (registered trademark) is one example of such a network. Alternatively, the communication unit 307 may communicate with external devices wirelessly.

[0036] The functions and processing of the display control device 130 are realized by the CPU 301 reading a program stored in the ROM 302 or HDD 304 and executing this program. The hardware configuration of the image generation device 120 is the same as that of the display control device 130.

[0037] Figure 4 shows the functional configuration of the display control device 130.

[0038] The viewpoint controller 131 can be, for example, a mouse, keyboard, 6-axis controller, or touch panel. For example, it periodically outputs information about the movement amount of the lever of a 6-axis controller such as a joystick to the controller operation acquisition unit 133.

[0039] The time controller 132 is, for example, an operating device equipped with a turntable. For example, it outputs information on the amount of movement of the turntable rotated by the user to the controller operation acquisition unit 133.

[0040] The controller operation acquisition unit 133 converts the movement amount information from the connected viewpoint controller 131 into virtual viewpoint movement amount information and outputs it to the virtual viewpoint position determination unit 134. It also converts the movement amount information from the connected time controller 132 into playback time movement amount information and outputs it to the time code determination unit 135.

[0041] The virtual viewpoint position determination unit 134 determines the current virtual viewpoint information based on the input virtual viewpoint movement amount information and outputs the virtual viewpoint information to the virtual viewpoint time information generation unit 136. The origin of the virtual space is set to the center of the shooting range surrounded by the imaging device 110, for example, the center of a stadium, and the three-dimensional movement position and direction from that center position are determined as the current virtual viewpoint information.

[0042] The timecode determination unit 135 determines the current playback time information based on the input playback time shift amount information and outputs the playback time information to the virtual viewpoint time information generation unit 136. The current playback time is defined as the start date and time when the imaging device 110 started shooting, and the current playback time is the time advanced from that start date and time.

[0043] The virtual viewpoint time information generation unit 136 generates virtual viewpoint time information indicating the virtual viewpoint position at the playback time by combining the virtual viewpoint information input from the virtual viewpoint position determination unit 134 and the playback time information input from the time code determination unit 135.

[0044] The virtual viewpoint information transmission unit 137 transmits the virtual viewpoint time information input from the virtual viewpoint time information generation unit 136 to the image generation device 120. The virtual viewpoint image generated by the image generation device 120 based on the virtual viewpoint time information is output to and displayed on the display 160.

[0045] The second real camera parameter management unit 138 pre-sets second real camera parameter information, including the second real camera position information of the location where the imaging device 150 is installed, and the controllable range of the imaging device 150, such as pan, tilt, and zoom. For the purposes of this explanation, the first real camera is referred to as the first real camera, and the imaging device 150 is referred to as the second real camera. The second real camera parameter information is also output to the second real camera control information generation unit 139. Furthermore, by aligning the positions of multiple first real cameras before shooting, a coordinate system in the virtual space is set that matches the coordinate system of the real space. This makes it possible to reproduce a real camera existing in the real space in the virtual space, and to simulate the position and orientation of the real camera in the virtual space. In this invention, the orientation of the imaging device 150 is calculated based on the virtual camera operated by the operator, and the imaging device 150 is controlled to achieve the calculated orientation.

[0046] The second actual camera position information is set to an initial position (X=0m, Y=50m, Z=10m) as a predetermined origin position, such as the center of the stadium.

[0047] Furthermore, the initial orientation is set to be, for example, horizontal to the Y-axis and facing the origin, with the left-right direction being positive pan (P) 0 degrees, the down-right direction being positive tilt (T) 0 degrees, and the zoom (Z) being 10 mm, i.e., (P=0, T=0, Z=10). Note that the scale of the second real camera position information and the position information in the virtual space are set to match.

[0048] The second real camera control information generation unit 139 generates second real camera control information from the input second real camera parameter information and virtual viewpoint time information. Specifically, it controls pan and tilt assuming that the virtual viewpoint point of focus position included in the input virtual viewpoint time information is captured from the second real camera position information included in the second real camera parameter information.

[0049] The generated second camera control information is also output to the second camera control information transmission unit 140.

[0050] The pan and tilt angles can be determined, for example, by using the following formula to calculate the inclination angle θ (unit: degrees "°") from the base a and height b of a right triangle. θ = tan⁻¹(b / a) For example, if the point of focus included in the virtual viewpoint time information is at (X=0, Y=0, Z=0) and the second actual camera position information is (X=0m, Y=50m, Z=10m), a right-angled triangle with a base of a=50m and a height of b=10m can be obtained from the difference in Y values. The above formula is used to calculate the inclination angle of that right-angled triangle, resulting in a calculation such as T=11.30°, which means that it is tilted 11.30° downwards relative to the initial direction which is horizontal to the Y axis and points towards the origin.

[0051] The second camera control information transmission unit 140 converts the input second camera control information into communication command information based on a communication protocol corresponding to the imaging device 150 and transmits it via the communication unit 307.

[0052] Next, the operation of the display control device 130 will be described.

[0053] Figure 5 is a flowchart showing the operation of the display control device 130 according to Embodiment 1. The following processes are performed when the CPU 301 reads and executes a program stored in the ROM 302 or HDD 304.

[0054] In step S501, the viewpoint controller 131 periodically outputs information about the movement amount of the lever of a 6-axis controller such as a joystick to the controller operation acquisition unit 133.

[0055] The time controller 132 outputs, for example, information on the amount of movement of a turntable rotated by the user to the controller operation acquisition unit 133. Next, the controller operation acquisition unit 133 outputs virtual viewpoint movement information to the virtual viewpoint position determination unit 134 based on the movement amount information from the connected viewpoint controller 131 and time controller 132 due to user operation. It also outputs playback time movement information to the time code determination unit 135.

[0056] In step S502, the virtual viewpoint position determination unit 134 determines the current virtual viewpoint information based on the input virtual viewpoint movement amount information and outputs the virtual viewpoint information to the virtual viewpoint time information generation unit 136.

[0057] In step S503, the timecode determination unit 135 determines the current playback time information based on the input playback time shift amount information and outputs the playback time information to the virtual viewpoint time information generation unit 136.

[0058] In step S504, the virtual viewpoint time information generation unit 136 generates virtual viewpoint time information indicating the virtual viewpoint position at the playback time by combining the virtual viewpoint information input from the virtual viewpoint position determination unit 134 and the playback time information input from the time code determination unit 135.

[0059] In step S505, the second real camera control information generation unit 139 sets the virtual viewpoint gaze point position included in the input virtual viewpoint time information as the second real camera gaze point position. The second real camera gaze point position is a position in the virtual space.

[0060] In step S506, the second real camera control information generation unit 139 generates second real camera control information from the input second real camera parameter information and the second real camera's gaze point position. Specifically, it sets the virtual space position corresponding to the second real camera's position in real space. If the coordinates in real space and virtual space are adjusted to be the same, the position in real space is used as the virtual space position. Next, the orientation of the second real camera is set so that the second real camera's gaze point position set in step S505 is located on the optical axis. Then, the set position and orientation of the second real camera are generated as real camera control information. If the coordinates in real space and virtual space are different, the real camera control information records the position converted to real space coordinates.

[0061] In step S507, the second actual camera control information transmission unit 140 converts the input second actual camera control information into communication command information based on a communication protocol corresponding to the imaging device 150 and transmits it via the communication unit 307.

[0062] Figure 6 is a conceptual diagram of the control of the imaging device 150 according to Embodiment 1. A virtual camera 601 in a virtual space is capturing a subject, and the virtual viewpoint point 602 is set directly below the subject.

[0063] Furthermore, by controlling the pan and tilt of the imaging device 150 so that it matches the second real camera's fixation point position to the virtual viewpoint fixation point 602, the scene captured by the virtual viewpoint image and the scene captured by the imaging device 150 will match.

[0064] As described above, in Example 1, it becomes possible to easily operate the second physical camera while manipulating the virtual viewpoint. Furthermore, it becomes possible to capture the same scene as the virtual viewpoint image.

[0065] <Example 2> Example 2 is an example of using a second real camera to capture images when there are limitations on the range of the second real camera's shooting capability.

[0066] Figure 7 shows the functional configuration of the display control device 700 according to Embodiment 2.

[0067] In addition to the configuration of the display control device 130 according to Embodiment 1, which was described with reference to Figure 4, the display control device 700 has the following functional components.

[0068] The second actual camera shooting range management unit 701 receives second actual camera parameter information from the second actual camera parameter management unit 138 and determines the second actual camera shooting range information, which is the shooting range, based on the camera controllable range such as pan-tilt-zoom of the imaging device 150.

[0069] Here, the second real camera shooting range information is defined as position information consisting of a three-dimensional space created when the space consisting of the shape of a viewing frustum, which is obtained when shooting in three-dimensional space from a position based on the second real camera position information, is continuously moved within a range where the pan and tilt can be controlled.

[0070] The second actual camera shooting range management unit 701 outputs second actual camera shooting range information to the second actual camera control information generation unit 702.

[0071] Furthermore, the second actual camera range information may be a three-dimensional space that excludes areas that are obstructed by buildings or objects installed within the stadium and cannot be photographed, or areas that will not be used as part of the image during shooting. It may also be a three-dimensional space that is excluded from the shooting range for reasons such as undesirable resolution.

[0072] The second real camera control information generation unit 702 generates second real camera control information from the input second real camera shooting range information, input second real camera parameter information, and virtual viewpoint time information.

[0073] Specifically, if the virtual viewpoint's gaze point position is not included within the three-dimensional space consisting of the second real camera's shooting range information, the second real camera's gaze point position is set to the position that is closest to the virtual viewpoint's gaze point position within the three-dimensional space consisting of the second real camera's shooting range information.

[0074] For example, the point where the line connecting the position of the second real camera in three-dimensional space and the virtual viewpoint point intersects with the boundary of the second real camera's shooting range is defined as the position of the second real camera's viewpoint point.

[0075] Furthermore, the system controls pan and tilt when capturing images at the second camera's point of focus, which is set based on the second camera's position information included in the second camera parameter information. The generated second camera control information is also output to the second camera control information transmission unit 140.

[0076] Figure 8 is a flowchart showing the operation of the display control device 700 according to Embodiment 2. Steps S501 to S504 and steps S505 to S507 are the same as those described in Figure 5, so their explanation is omitted.

[0077] In step S801, the second real camera control information generation unit 702 determines whether or not the virtual viewpoint gaze point is included in the three-dimensional space consisting of the second real camera shooting range information.

[0078] If the second real camera's targeting point is included within the three-dimensional space consisting of the second real camera's shooting range information (Yes in S801), in step S505, the second real camera's fixation point is set to the same position as the virtual viewpoint's fixation point.

[0079] If the data is not included in the three-dimensional space consisting of the second actual camera shooting range information (No. in S801), proceed to step S802.

[0080] In step 802, the second actual camera's gaze point position is set to the position that is closest to the virtual viewpoint gaze point position within the three-dimensional space consisting of the second actual camera's shooting range information.

[0081] Figure 9 is a conceptual diagram of the control of the imaging device 150 according to Embodiment 2. A virtual camera 601 in the virtual space is capturing the subject, and the virtual viewpoint point 602 is set directly below the subject.

[0082] Furthermore, if the virtual viewpoint fixation point 602 is not included in the second actual camera shooting range 901, the second actual camera fixation point 902 is set to the position closest to the virtual viewpoint fixation point 602. Based on the set second actual camera fixation point 902, the pan and tilt of the imaging device 150 are controlled.

[0083] As described above, according to Example 2, even if there is an area that the imaging device 150 cannot capture, it is possible to capture a scene as close as possible to the scene captured in the virtual viewpoint image. Furthermore, as soon as the subject enters the shooting range of the imaging device 150, the point of focus is immediately matched, and the same scene can be captured.

[0084] <Example 3> Example 3 is an example that addresses the time delay in the virtual space relative to the actual time. Here, the service provided by the virtual viewpoint image generation system is assumed to experience a delay time before it can be viewed by the service provider or user due to processing such as capture, transmission, 3D calculation processing, and image distribution. In other words, there is a predetermined delay between the actual time and the time in the virtual space, for example, a delay of 3 seconds.

[0085] Figure 10 shows the functional configuration of the display control device 1000 according to Embodiment 3.

[0086] In addition to the configuration of the display control device 130 according to Embodiment 1, which was described with reference to Figure 4, the display control device 1000 has the following functional components.

[0087] The estimated virtual viewpoint information generation unit 1001 generates estimated virtual viewpoint gaze point position information, which estimates the virtual viewpoint gaze point position after a predetermined time period, based on the virtual viewpoint gaze point position position information contained in the virtual viewpoint time information generation unit 136 input from the virtual viewpoint information transmission unit 137. For example, the virtual viewpoint gaze point position may be recorded for a predetermined period of time in the past, and the virtual viewpoint gaze point position after the predetermined time period may be estimated from the past virtual viewpoint gaze point positions. The estimation method may be, for example, linear interpolation or spline interpolation. The estimated virtual viewpoint gaze point position information is then output to the second real camera control information generation unit 139.

[0088] Figure 11 is a flowchart showing the operation of the display control device 1000 according to Embodiment 3. Steps S501 to S504 and steps S506 to S507 are the same as those described in Figure 5, so their explanation is omitted.

[0089] In step S1101, the estimated virtual viewpoint information generation unit 1001 generates estimated virtual viewpoint gaze position information, which estimates the virtual viewpoint gaze position after a predetermined time, based on the virtual viewpoint gaze position position information included in the input virtual viewpoint time information generation unit 136.

[0090] In step S1102, the second actual camera control information generation unit 139 sets the input estimated virtual viewpoint gaze point position as the second actual camera gaze point position.

[0091] Figure 12 is a conceptual diagram showing how to control the imaging device 150 according to Example 3.

[0092] Figure 12(a) is a conceptual diagram of a virtual viewpoint and estimated virtual viewpoint in a virtual space. A virtual camera 601 is capturing a subject, and the virtual viewpoint 602 is set directly below the subject. It also shows the virtual viewpoint 602 at the current time t seconds and the estimated virtual viewpoint 1201 at future time t+3 seconds, based on, for example, a past virtual viewpoint 603 at past time t-3 seconds.

[0093] For example, suppose that at past time t-3 seconds, the position of virtual viewpoint point 603 was Y=0m, and after moving along the Y-axis, the position of virtual viewpoint point 602 at present time t seconds is Y=10m. Using linear interpolation to estimate the position, the estimated position of virtual viewpoint point 1201 at future time t+3 seconds is estimated to be Y=20m.

[0094] Figure 12(b) is a conceptual diagram showing how to control the imaging device 150 based on the estimated virtual viewpoint gaze point 1201. The second actual camera gaze point position 1202 is set at a position based on the estimated virtual viewpoint gaze point 1201. The pan and tilt of the imaging device 150 are controlled relative to the second actual camera gaze point position 1202.

[0095] As described above, according to Example 3, it is possible to capture the same scene even when there is a delay in the virtual space relative to the real world.

[0096] In Examples 1 to 3, multiple second real cameras may be placed, and for example, all of the multiple placed second real cameras may be controlled.

[0097] Alternatively, the system may control only one of the multiple second-actual cameras that is closest to the virtual viewpoint's gaze point.

[0098] Although this disclosure has been described above based on several embodiments, this disclosure is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of this disclosure, and these modifications are not excluded from the scope of this disclosure.

[0099] Furthermore, in this embodiment, some or all of the control may be provided to an image processing system, etc., via a network or various storage media, by supplying a computer program that realizes the functions of the embodiment described above. The computer (or CPU, MPU, etc.) in the image processing system, etc., may then read and execute the program. In that case, the program and the storage medium storing the program constitute the present disclosure.

[0100] Furthermore, the disclosure of this embodiment includes the following configuration, method, and program.

[0101] (Composition 1) A setting means for setting the position and orientation of a virtual camera based on user operation, An output means that outputs information indicating the orientation of a second real camera, which is different from the orientation of a first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the position and orientation of the virtual camera. An information processing device characterized by having the following features.

[0102] (Configuration 2) The first physical camera's position and orientation remain unchanged based on the position and orientation of the virtual camera. The information processing apparatus according to Configuration 1, characterized in that the orientation of the second physical camera is changed without changing the position of the second physical camera, based on the position and orientation of the virtual camera.

[0103] (Composition 3) The information processing apparatus according to configuration 1, characterized in that the second physical camera is a network camera.

[0104] (Composition 4) The information processing device according to claim 1, characterized in that the orientation of the second real camera is determined based on the gaze point identified based on the position and orientation of the virtual camera.

[0105] (Composition 5) The setting means sets the distance from the virtual camera to the point of focus of the virtual camera based on user operation. The information processing device according to configuration 4, characterized in that the point of focus of the virtual camera is located on the optical axis of the virtual camera.

[0106] (Composition 6) The information processing apparatus according to configuration 1, characterized in that the second physical camera captures an area different from the area in which a 3D model of the subject is generated by the image captured by the first physical camera.

[0107] (Composition 7) The information processing apparatus according to configuration 1, characterized in that the virtual viewpoint image is generated based on the image captured by the first real camera and the virtual camera.

[0108] (method) A setup process to set the point of focus of the virtual camera based on user input, An output step that outputs information indicating the orientation of a second real camera, which is different from the first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the gaze point of the virtual camera; An information processing method characterized by having the following features.

[0109] (program) A program for causing a computer to function as an image processing device as described in any one of items 1 to 7 of the configuration. [Explanation of symbols]

[0110] 100 Image Processing Systems 110 Imaging device 150 Imaging device 120 Video Generation Device 130 Display control device

Claims

1. A setting means for setting the position and orientation of a virtual camera based on user operation, An output means that outputs information indicating the orientation of a second real camera, which is different from the orientation of a first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the position and orientation of the virtual camera. An information processing device characterized by having the following features.

2. The first physical camera's position and orientation remain unchanged based on the position and orientation of the virtual camera. The information processing apparatus according to claim 1, characterized in that the orientation of the second physical camera is changed without changing the position of the second physical camera, based on the position and orientation of the virtual camera.

3. The information processing apparatus according to claim 1, characterized in that the second physical camera is a network camera.

4. The information processing device according to claim 1, characterized in that the orientation of the second real camera is determined based on the gaze point identified based on the position and orientation of the virtual camera.

5. The setting means sets the distance from the virtual camera to the point of focus of the virtual camera based on user operation. The information processing apparatus according to claim 4, characterized in that the point of focus of the virtual camera is located on the optical axis of the virtual camera.

6. The information processing apparatus according to claim 1, characterized in that the second physical camera captures an area different from the area in which a 3D model of the subject is generated by the image captured by the first physical camera.

7. The information processing apparatus according to claim 1, characterized in that the virtual viewpoint image is generated based on the image captured by the first real camera and the virtual camera.

8. A setting process that sets the position and orientation of the virtual camera based on user input, An output step that outputs information indicating the orientation of a second real camera, which is different from the first real camera used to generate a virtual viewpoint image corresponding to the virtual camera, based on the position and orientation of the virtual camera; An information processing method characterized by having the following features.

9. A program for causing a computer to function as an information processing device according to any one of claims 1 to 7.

Images