Control device, control method, and program

Abstract

To provide a virtual viewpoint image in which a viewpoint of a person or the like is accurately reproduced.SOLUTION: An operation signal for specifying the position and direction in virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated on the basis of a plurality of captured images obtained by imaging by a plurality of imaging devices is acquired. A second virtual viewpoint facing a first virtual viewpoint specified by the acquired operation signal is decided. Virtual viewpoint information expressing the first virtual viewpoint or the second viewpoint is set in response to an instruction from an operator based on the virtual viewpoint image corresponding to the decided second virtual viewpoint.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a control technique for a virtual viewpoint used to generate a virtual viewpoint image. [Background technology]

[0002] There is a technology that uses multiple image capture devices installed in different positions to capture images synchronously and then uses the captured images to generate an image (virtual viewpoint image) that represents the view from a virtual image capture device (virtual viewpoint) that does not actually exist in the three-dimensional space of the image capture target. When generating a virtual viewpoint image, an operator typically refers to a generated virtual viewpoint image corresponding to the virtual viewpoint capturing the object (subject) on a UI screen, and sets the position and orientation of the new virtual viewpoint by operating a joystick or the like. The virtual viewpoint can be moved throughout the entire three-dimensional space of the image capture target, and the movement path of the virtual viewpoint, which is set continuously over time, is generally referred to as the "camera path." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-187797 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when generating a virtual viewpoint image of a sports game such as basketball, there is a demand for reproducing an image from a virtual viewpoint from a player's point of view. In this case, an operator operates a joystick or the like to specify the position and posture of the virtual viewpoint so that it matches the position and orientation of the target player's face on the UI screen. However, since the virtual viewpoint after such operation overlaps with the target player on the UI screen, the target player himself is naturally not displayed in the image corresponding to the virtual viewpoint. Therefore, an operator operating a joystick or the like while viewing an image corresponding to the specified virtual viewpoint cannot confirm whether the posture (line of sight) of the virtual viewpoint specified by the operator matches the target player's actual line of sight. In this regard, Patent Document 1 discloses a method for generating a free viewpoint image of a specific player as viewed from the ball by specifying, for example, the ball as a gaze point. With this method, it is sufficient for the player to actually look only at the gaze point specified by the operator (here, the ball). However, if the player actually looks at something other than the ball (such as another player or the goal), the player's line of sight cannot be reproduced in the strict sense.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to realize a virtual viewpoint image that accurately reproduces the viewpoint of a person or the like. [Means for solving the problem]

[0006] The virtual viewpoint control device of the present disclosure is characterized by having an acquisition means for acquiring an operation signal that specifies the position and orientation in virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices, a determination means for determining a second virtual viewpoint facing a first virtual viewpoint identified by the acquired operation signal, and a setting means for setting virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an operator's instruction based on the virtual viewpoint image corresponding to the determined second virtual viewpoint. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to realize a virtual viewpoint image that accurately reproduces the viewpoint of a person or the like. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image according to the first embodiment. [Figure 2] FIG. 1A is a diagram illustrating a UI screen used to specify the position and orientation of a virtual camera, and FIG. 1B is a diagram illustrating a UI screen used to specify an operation mode. [Figure 3] FIG. 1 is a block diagram showing a basic hardware configuration of an information processing apparatus. [Figure 4] 5 is a flowchart showing the flow of operations in the virtual viewpoint control device according to the first embodiment. [Figure 5] 10A and 10B are diagrams illustrating a specific example of converting virtual viewpoint information into face-to-face virtual viewpoint information. [Figure 6] FIG. 1A is a diagram illustrating a virtual viewpoint image corresponding to virtual viewpoint information, and FIG. 1B is a diagram illustrating a virtual viewpoint image corresponding to face-to-face virtual viewpoint information. [Figure 7] 10 is a flowchart showing details of processing by a camera path creation unit. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of an image processing system that generates a virtual viewpoint image according to a second embodiment. [Figure 9] 10 is a flowchart showing the flow of operations in the virtual viewpoint control device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0010] [Embodiment 1] Before describing each embodiment, a brief overview of a virtual viewpoint image will be provided. A virtual viewpoint image is an image that represents a view from a virtual viewpoint of an imaging device that is different from the viewpoint of an actual imaging device, and is also called a free viewpoint image. The virtual viewpoint is set by a method such as an operator directly specifying the virtual viewpoint by operating a controller or selecting from a plurality of preset virtual viewpoint candidates. The virtual viewpoint image includes both moving images and still images.

[0011] In the following embodiments, video is used as an example for explanation, but still images can also be used. Information indicating changes in the virtual viewpoint, necessary for video, is called a "camera path." The camera path defines the movement of a virtual imaging device (hereinafter referred to as a "virtual camera") in a video created by sequentially playing multiple virtual viewpoint images or CG images arranged in chronological order. The camera path is managed using frames and a timeline. Frames hold information necessary for generating each image that constitutes a video. Specifically, they hold information such as the scene time (time code) and the camera position and orientation. The scene time is expressed, for example, as a time code where the time when the game being filmed starts is 00:00:00:00 frame. The timeline shows the frame times on a single time axis. The number of frames included in the timeline is determined by the number of images played per second (frame rate). For example, if the frame rate is 60 frames per second, the timeline contains 60 frames per second. In particular, reference frames are called "key frames," and the times of the start and end key frames are shown on the timeline. Video content producers create a camera path by repeatedly determining the position and orientation of a virtual camera at a given time code and registering at least two key frames. When editing a camera path using key frames, frames are divided into two types: key frames and intermediate frames. For key frames, the user editing the camera path explicitly specifies information. On the other hand, for intermediate frames, which are frames between key frames, the system interpolates between the key frames to determine virtual viewpoint information.

[0012] <System configuration> 1 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint image according to this embodiment. The image processing system is composed of an imaging device group 100, a 3D model generation device 101, a virtual viewpoint image generation device 102, a display 103, a controller 104, and a virtual viewpoint control device 110. For convenience, a moving virtual viewpoint image will be referred to below as a "virtual viewpoint video."

[0013] The imaging device group 100 synchronizes the time of all devices and captures images from multiple directions to generate shape data (commonly referred to as a "3D model") representing the three-dimensional shape of an object (subject). Multiple pieces of video data obtained by this capture are sent to a 3D model generation device 101, where a 3D model is generated for each frame using a method such as volume intersection. Volume intersection is a method of extracting the silhouette of an object from each of multiple captured images, back-projecting each silhouette into the original three-dimensional space, and determining the intersection of the respective volumetric volumes to obtain the three-dimensional shape of the object. The generated 3D model is stored in a storage device (not shown) within the device and output in response to an acquisition request from a virtual viewpoint image generation device 102. The virtual viewpoint control device 110 receives a camera operation signal from the controller 104 and generates virtual viewpoint information indicating the position and orientation of the virtual camera. The generated virtual viewpoint information is then provided to the virtual viewpoint image generation device 102. The virtual viewpoint image generation device 102 performs rendering processing based on the 3D model of the object and the virtual viewpoint information to generate a virtual viewpoint image representing the view of the object from the virtual viewpoint indicated by the virtual viewpoint information. The generated virtual viewpoint image is displayed on the display 103.

[0014] The virtual viewpoint control device 110 is composed of a camera operation signal acquisition unit 111, a virtual viewpoint information setting unit 112, a virtual viewpoint information holding unit 113, a virtual viewpoint information conversion unit 114, and a virtual viewpoint information provision unit 115. Each unit of the virtual viewpoint control device 110 will be described below.

[0015] The camera operation signal acquisition unit 111 acquires from the controller 104 a camera operation signal generated in response to an operator operating the virtual camera using a joystick or the like. In this embodiment, the virtual viewpoint information includes parameters such as the position and attitude of the virtual camera, as well as zoom (focal length) and time. In this case, the position of the virtual camera is indicated by three-dimensional coordinate information according to a Cartesian coordinate system of three axes: X, Y, and Z. The origin in this case is an arbitrary position (e.g., the center of the court) in the three-dimensional space of the object to be imaged. The attitude of the virtual camera (the direction of the virtual viewpoint) is indicated by the angle formed with the three axes: pan, tilt, and roll. Pan is the angle of rotation parallel to the ground surface, tilt is the angle of rotation perpendicular to the ground surface, and roll is the angle of rotation around the optical axis of the virtual camera. The time is indicated by a time code. Zoom and time are each single-axis parameters. In other words, the virtual viewpoint at a given moment is specified by [X, Y, Z] (unit: m), [Pan, Tilt, Roll] (unit: degrees), [Zoom] (unit: mm), and a time code [HH, MM, SS, FF]. The camera operation signal input from the controller 104 represents the frame-to-frame changes in the position, orientation, and zoom of the virtual viewpoint [ΔX, ΔY, ΔZ, ΔPan, ΔTilt, ΔRoll, ΔZoom]. FIG. 2(a) shows an example of a user interface screen (UI screen) used by an operator to specify the position and orientation of the virtual camera. Currently, the UI screen 200 displays a virtual viewpoint image overlooking a basketball court, which is the imaging space. The operator registers key frames and creates a camera path while viewing this virtual viewpoint image corresponding to the current camera operation signal. The following description will be given using an example of creating a virtual viewpoint video from the perspective of player A. Note that parameters other than the above eight axes may be included, or all of the above eight axes may not be included.

[0016] The virtual viewpoint information setting unit 112 calculates virtual viewpoint information for the current frame based on the virtual viewpoint information stored in the virtual viewpoint information storage unit 113 and the camera operation signal acquired by the camera operation signal acquisition unit 111. The calculated virtual viewpoint information is sent to the virtual viewpoint information storage unit 113 and stored therein, and is used to calculate virtual viewpoint information for the next frame. The calculated virtual viewpoint information is also sent to the virtual viewpoint information conversion unit 114 depending on the operation mode, and converted into provisional virtual viewpoint information for confirming whether the line of sight direction of the virtual camera indicated by the virtual viewpoint information is appropriate. Here, there are two operation modes: a facing movement mode and a normal mode. The facing movement mode is an operation mode in which virtual viewpoint information generated based on a camera operation signal is converted into virtual viewpoint information (hereinafter referred to as "facing virtual viewpoint information") when the virtual camera represented by the virtual viewpoint information is moved to the facing position. The normal mode is a normal operation mode in which such conversion is not performed. FIG. 2(b) shows an example of a UI screen in which an operator selects one of the operation modes. The UI screen of FIG. 2(b) includes a pull-down menu 201 for switching the face-to-face movement mode between enabled (ON) and disabled (OFF), and an input field 202 for the operator to specify the distance D to the face-to-face position when the mode is enabled. The operator specifies the operation mode in advance via the UI screen as shown in FIG. 2(b). The operation mode may be configured to be switched, for example, by pressing a predetermined button on a controller that operates the virtual camera. The distance to the face-to-face position may also be configured to be changeable, for example, using a seek bar. The virtual viewpoint information setting unit 112 sets the calculated virtual viewpoint information for the current frame or the face-to-face virtual viewpoint information received from the virtual viewpoint information conversion unit 114 in accordance with the operation mode, and outputs the set information to the virtual viewpoint information providing unit 115.

[0017] The virtual viewpoint information holding unit 113 holds the virtual viewpoint information received from the virtual viewpoint information setting unit 112. When the virtual viewpoint information holding unit 113 receives new virtual viewpoint information from the virtual viewpoint information setting unit 112, it updates the virtual viewpoint information that it holds.

[0018] The virtual viewpoint information conversion unit 114 converts the virtual viewpoint information received from the virtual viewpoint information setting unit 112 into facing virtual viewpoint information that represents a virtual viewpoint moved to a facing position of the virtual viewpoint represented by the virtual viewpoint information. The conversion method will be described in detail later.

[0019] The virtual viewpoint information providing unit 115 provides the virtual viewpoint information and the face-to-face virtual viewpoint information set by the virtual viewpoint information setting unit 112 to the virtual viewpoint image generating device 102 via a network such as a LAN. In addition, the virtual viewpoint information providing unit 115 receives a clip playback instruction from an operator, acquires the corresponding camera path, and provides it to the virtual viewpoint image generating device 102 via a network such as a LAN.

[0020] The key frame registration unit 116 registers virtual viewpoint information at a specific time as a key frame based on a key frame registration instruction from an operator, and a camera path is created based on the key frame thus registered.

[0021] Based on an operator's instruction to create a camera path, the camera path creation unit 117 creates a camera path based on a plurality of key frames registered by the key frame registration unit 116. An ID is assigned to the created camera path, and the created camera path is stored in the auxiliary storage device 314, which will be described later.

[0022] <Hardware configuration> FIG. 3 is a block diagram showing a basic hardware configuration common to each device (the 3D model generation device 101, the virtual viewpoint image generation device 102, and the virtual viewpoint control device 110) that is an information processing device. Each device includes a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, a display unit 315, an operation unit 316, a communication I / F 317, and a bus 318. The CPU 311 controls the entire device using computer programs and data stored in the ROM 312 and the RAM 313, thereby realizing each function of each device. Note that one or more dedicated hardware components separate from the CPU 311 may be included, and at least a portion of the processing by the CPU 311 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage device 314 and data supplied from the outside via the communication I / F 317. The auxiliary storage device 314 is formed, for example, by a hard disk drive or the like, and stores various data such as image data and audio data. The display unit 315 is formed, for example, by an LCD display or LEDs, and displays a GUI (Graphical User Interface) for an operator to operate each device. The operation unit 316 is formed, for example, by a keyboard, mouse, touch panel, etc., and inputs various instructions to the CPU 311 in response to operations by the operator. The CPU 311 operates as a display control unit that controls the display unit 315 and an operation control unit that controls the operation unit 316. The communication I / F 317 is used for communication between each device and an external device. For example, when connected to an external device via a wired connection, a communication cable is connected to the communication I / F 317. Furthermore, when the communication I / F 317 has a function for wireless communication with an external device, it is equipped with an antenna. The bus 318 connects each unit within the device to transmit information. In this embodiment, the display unit 315 and the operation unit 316 are described as being present inside the device, but at least one of the display unit 315 and the operation unit 316 may be present as a separate device outside the device.

[0023] <Operation flow of virtual viewpoint control device> FIG. 4 is a flowchart showing the flow of operations in the virtual viewpoint control device 110 according to this embodiment. Below, the flow up to the creation of a camera pass including virtual viewpoint information from the player's point of view during a basketball game will be explained along with the flowchart in FIG. 4, with reference to specific examples as needed. The operation mode at the start of this flow is assumed to be normal mode, and it is executed when no clip is being played. Furthermore, this flow is executed on a frame-by-frame basis, but may also be executed at thinned frame intervals. In the following explanation, the symbol "S" means step.

[0024] In S401, the next process to be executed is determined based on whether or not a previous frame has already been processed. If there is a previous frame, S402 is executed; if there is no previous frame, S402 is skipped and S403 is executed next. For example, if processing has just started, this step is skipped because there is no previous frame.

[0025] In S402, the virtual viewpoint information setting unit 112 acquires the virtual viewpoint information of the previous frame from the virtual viewpoint information holding unit 113. [X prev ,Y prev ,Z prev ,Pan prev ,Tilt prev ,Roll prev ,Zoom prev ]=[-9.0, 2.0, 1.9, -95.0, 0.0, 0.0, 20.0] is acquired as the virtual viewpoint information of the previous frame.

[0026] In S403, the camera operation signal acquisition unit 111 acquires a camera operation signal corresponding to the current frame of interest as the processing target from the controller 104. Here, it is assumed that the amount of change from the previous frame [ΔX, ΔY, ΔZ, ΔPan, ΔTilt, ΔRoll, ΔZoom]=[0.0, 0.0, 0.0, -5.0, 0.0, 0.0, 0.0] has been acquired. The acquired camera operation signal is output to the virtual viewpoint information setting unit 112.

[0027] In S404, the virtual viewpoint information setting unit 112 calculates the virtual viewpoint information of the current frame based on the virtual viewpoint information of the previous frame acquired in S402 and the camera operation signal acquired in S403. The following formulas (1) to (7) are used to calculate the virtual viewpoint information. JPEG2025125421000002.jpg36126 In the above formulas (1) to (7), [X curr ,Y curr ,Z curr ,Pan curr ,Tilt curr ,Roll curr ,Zoom curr ] represents the virtual viewpoint information of the current frame. prev ,Y prev ,Z prev ,Pan prev ,Tilt prev ,Roll prev ,Zoom prev ] represents the virtual viewpoint information of the previous frame. Now, when the virtual viewpoint information of the previous frame acquired in S401 and the camera operation signal indicating the amount of change from all frames acquired in S402 are applied to the above equations (1) to (7), the result is shown in Table 1 below.

[0028] [Table 1] In the above example, only the pan angle of the virtual camera placed over the position of player A is adjusted. As a result, the virtual viewpoint information of the current frame, [X curr ,Y curr ,Z curr ,Pan curr ,Tilt curr ,Roll curr ,Zoom curr]=[-9.0, 2.0, 1.9, -100.0, 0.0, 0.0, 20.0] is calculated. The calculated virtual viewpoint information of the current frame is saved in the RAM 313. Then, it is held by the virtual viewpoint information holding unit 113 and is used as the virtual viewpoint information of the previous frame when calculating the virtual viewpoint information of the next current frame.

[0029] In S405, the next process to be executed is determined depending on the current operation mode. If the current operation mode is normal mode, S409 is executed next, and if it is face-to-face movement mode, S406 is executed next.

[0030] In S406, the virtual viewpoint information conversion unit 114 acquires the value of the distance D input in the input field 202 of the UI screen shown in Fig. 2(b) above. Now, it is assumed that the distance D = 5 m is acquired.

[0031] In S407, the virtual viewpoint information conversion unit 114 converts the virtual viewpoint information calculated in S404 into facing virtual viewpoint information. Here, the virtual camera at the facing position represented by the converted facing virtual viewpoint information is restricted to rotate around a vertical axis passing through the three-dimensional position of the original virtual camera represented by the virtual viewpoint information before conversion. Specifically, the facing virtual viewpoint information [X' curr ,Y' curr ,Z' curr ,Pan' curr ,Tilt' curr ,Roll' curr ,Zoom' curr ] is calculated. JPEG2025125421000004.jpg36124 Now, the virtual viewpoint information of the current frame [X curr ,Y curr ,Z curr ,Pan curr ,Tilt curr ,Roll curr ,Zoom curr]=[-9.0,2.0,1.9,-100.0,0.0,0.0,20.0] and distance D=5m are applied to the above equations (8) to (14), and the result is shown in Table 2 below.

[0032] [Table 2] As mentioned above, "π" in equation (11) is usually "180", but any value (for example, within the range of 160 to 200) that can realize a position that can be said to be roughly opposite each other may be used.

[0033] 5(a) and (b) illustrate the above specific example. In a virtual imaging space with the center of the basketball court as the origin, it can be seen that a virtual camera at the opposite position is specified at a distance D from the virtual camera specified to overlap with the position of player A. In this way, the opposite virtual viewpoint information, [X' curr ,Y' curr ,Z' curr ,Pan' curr ,Tilt' curr ,Roll' curr ,Zoom' curr ]=[-13.92,1.13,1.9,80.0,0.0,0.0,20.0] is calculated. Note that in the conversion using the above-mentioned formulas (8) to (14), the virtual camera at the facing position represented by the virtual viewpoint information after conversion becomes a virtual camera that is a mirror image of the virtual camera represented by the virtual viewpoint information before conversion. For example, the following formula (15) may be used instead of formula (13) so that the rotation parameters are not changed. JPEG2025125421000006.jpg6170 The face-to-face virtual viewpoint information thus obtained is stored in the RAM 313.

[0034] In S408, the virtual viewpoint information providing unit 115 transmits the face-to-face virtual viewpoint information of the current frame obtained in S407 or the virtual viewpoint information of the current frame calculated in S404 to the virtual viewpoint image generating device 102. The virtual viewpoint image generating device 115 generates a virtual viewpoint image corresponding to the received virtual viewpoint information / face-to-face virtual viewpoint information and returns the virtual viewpoint image to the virtual viewpoint control device 110. The virtual viewpoint control device 110 then displays the received virtual viewpoint image on the UI screen, and the operator checks the virtual viewpoint image displayed on the UI screen and, if OK, instructs the operator to register a key frame on the UI screen. Figure 6(a) is a diagram illustrating the virtual viewpoint image displayed when virtual viewpoint information is transmitted in the above specific example, and Figure 6(b) is a diagram illustrating the virtual viewpoint image displayed when face-to-face virtual viewpoint information is transmitted. Now, assume that player A is looking in the direction of player B in the real imaging space. When virtual viewpoint information for the current frame is transmitted in normal mode (S409), a virtual viewpoint image 610 representing the view from a virtual camera 601 (see the left diagram in FIG. 6(a)) facing player B and overlapping player A is displayed on the UI screen. On the other hand, when face-to-face virtual viewpoint information is transmitted in face-to-face movement mode (S408), a virtual viewpoint image 620 representing the view from a dashed virtual camera 602 (see the left diagram in FIG. 6(b)) facing player A and positioned 5 m away from player A is displayed on the UI screen. When the operator looks at the virtual viewpoint image 610 (see the right diagram in FIG. 6(a)), player A is not displayed therein, and therefore the operator cannot know whether the virtual viewpoint image 610 matches the scenery that player A actually sees. In contrast, in the case of the virtual viewpoint image 620 (see the right diagram in FIG. 6(b)), if player A is displayed in the virtual viewpoint image 620 from the so-called camera's line of sight, the operator can determine that the scenery matches the scenery that player A actually sees. In this way, the operator can actually check whether the direction of the specified virtual camera is aligned with the line of sight of the player or other person by looking at the virtual viewpoint image corresponding to the virtual camera in the opposite position, and then give instructions to register a key frame.6(a) and 6(b) are provided for reference purposes to visually indicate the position and orientation of the virtual camera / facing virtual camera corresponding to the virtual viewpoint image shown on the right, but these may be displayed together with the image on the right on the UI screen. That is, a CG image of the imaging space viewed from above may be displayed separately as a window for confirming the camera position, and an icon two-dimensionally representing the virtual camera to be operated may be superimposed on the CG image, allowing the operator to confirm the position and orientation of the virtual camera being operated.

[0035] In S410, the next process to be executed is determined depending on whether or not a key frame registration instruction has been input. If a key frame registration instruction has been input, S411 is executed next. On the other hand, if no instruction has been input, S411 is skipped and S412 is executed next. Note that the method by which the operator inputs the key frame registration instruction is not limited. For example, the operator may touch a predetermined button on the UI screen described above, or may press a predetermined key on the keyboard or a predetermined button on the controller 104.

[0036] In S411, the key frame registration unit 116 associates the virtual viewpoint information calculated in S404 with the time code of the current frame and registers it as a key frame. At this time, even in the face-to-face movement mode, the virtual viewpoint information calculated in S404 is registered as the key frame, rather than the face-to-face virtual viewpoint information obtained in S407. This eliminates the need for an operator who visually confirms the virtual viewpoint image from the face-to-face position to operate the virtual camera again and re-specify the desired virtual viewpoint, thereby enabling efficient key frame registration work. In this way, a camera path consisting of multiple key frames is obtained.

[0037] In S412, it is determined whether or not there is an instruction to switch the operation mode. If there is an instruction to switch, the operation mode is changed to the newly specified operation mode in S413. If there is no instruction to switch, S414 is executed next.

[0038] In S414, it is determined whether or not to end the registration of key frames. If an instruction to end the registration of key frames has been input, S415 is executed next. On the other hand, if an instruction to end the registration of key frames has not been input, the process returns to S402, and the current frame is set as the previous frame and the next frame is set as the current frame, and processing continues. Since the camera path creation process described below requires at least two key frames, a start frame and an end frame, if two or more key frames have not been registered, a warning may be displayed.

[0039] In S415, it is determined whether or not to start creating a camera path. If a command to create a camera path has been input, S416 is executed. On the other hand, if a command to create a camera path has not been input, the process waits for a certain period of time and then determines again whether or not an input has been made.

[0040] In S416, the camera path creation unit 117 creates a camera path using the key frames registered by the key frame registration unit 116. Fig. 7 is a flowchart showing details of the processing by the camera path creation unit 117. The following description will be given with reference to the flowchart shown in Fig. 7.

[0041] In S701, processing is assigned depending on whether multiple key frames necessary for creating a camera path have been registered. If two or more key frames have been registered, S702 is executed next. If not, the camera path cannot be created, so this flow ends.

[0042] In S702, interpolation is performed between adjacent key frames among the registered multiple key frames to generate virtual viewpoint information for intermediate frames and create a camera path. The interpolation method is not particularly limited, but may be, for example, linear interpolation, which connects the registered multiple key frames in order with straight lines, or nonlinear interpolation, which connects key frames with smooth curves. In this way, a camera path based on the registered key frames is obtained.

[0043] In S703, all registered key frames are cleared. This makes it possible to register new key frames and create the next camera path. Once the registered key frames have been cleared after creating the camera path, the process returns to the flow in FIG. 4, and the process ends. The above is the content of the operation flow of the virtual viewpoint control device 110 according to this embodiment.

[0044] <Modification> In the above-described embodiment, only the facing virtual viewpoint information is provided in the facing movement mode. However, the virtual viewpoint information on which the facing virtual viewpoint information is based may be provided as a set, and two types of virtual viewpoint images corresponding to the virtual viewpoint information and the facing virtual viewpoint information may be displayed on the UI screen. Two displays 103 may be used to display the virtual viewpoint image on each, or one display may display both virtual viewpoint images. In this case, the operator can also check the virtual viewpoint image that represents the view from the virtual camera that the operator has set.

[0045] Furthermore, when a key frame registration instruction is given, virtual viewpoint information is registered as a key frame regardless of the operation mode at that time, but it may be configured so that facing virtual viewpoint information is registered as a key frame in the facing movement mode.

[0046] As described above, according to this embodiment, the operator can sequentially check whether the virtual viewpoint he or she has specified is aligned with the person's line of sight by looking at images that represent the view from that facing position, and therefore can create a camera path for virtual viewpoint images that can accurately reproduce the person's viewpoint. [Embodiment 2] In the first embodiment, the operator confirms the position of an object of interest on a UI screen and then operates a joystick or the like to specify a virtual viewpoint, thereby calculating virtual viewpoint information representing the viewpoint of the object. Next, a second embodiment will be described in which tracking information of the object of interest is acquired to automatically calculate virtual viewpoint information representing the viewpoint of the object. Note that a description of the content common to the first embodiment will be omitted, and the following description will focus on the operational flow, which is the difference.

[0047] <System configuration> 8 is a diagram showing an example of the configuration of an image processing system for generating a virtual viewpoint image according to this embodiment. The basic configuration is the same as the image processing system according to the first embodiment, but the difference is that a tracking unit 801 is added to the virtual viewpoint control device 110.

[0048] The tracking unit 801 acquires a 3D model of each object from the 3D model generation device 101 and performs tracking processing for each object using, for example, a method using machine learning, which is a well-known technology. This tracking processing assigns a unique ID to the 3D model of each object, and the operator specifies an object of interest using the ID assigned to the 3D model. This acquires tracking information indicating the three-dimensional position [X, Y, Z] of the object of interest at each time during the imaging period.

[0049] <Operation flow of virtual viewpoint control device> FIG. 9 is a flowchart showing the flow of operations in the virtual viewpoint control device 110 according to this embodiment. This flow shows the flow of operations in the face-to-face position mode, which is a feature, and changes in operation mode are omitted. The following explanation will be made in accordance with the flowchart in FIG. 9. Note that this flow is executed on a frame-by-frame basis, but may also be executed at thinned frame intervals. In the following explanation, the symbol "S" means step.

[0050] In S901, the designation of an object of interest by the operator is accepted via the controller 104 or the like.

[0051] In S902, the virtual viewpoint information setting unit 112 acquires, from the tracking unit 801, tracking information of the current frame of interest as the processing target for the object of interest designated by the operator.

[0052] In S903, the virtual viewpoint information setting unit 112 generates virtual viewpoint information based on the information acquired in S902. At this time, the three-dimensional position [X, Y, Z] indicated by the tracking information is applied to the position of each parameter of the virtual viewpoint information. For parameters other than the position, such as the orientation [Pan, Tilt, Roll] and focal length [Zoom], preset default values ​​may be applied, or the operator may manually set these initial values.

[0053] In S904, similar to S406 in the flow of FIG. 4 in the first embodiment, the virtual viewpoint information conversion unit 114 acquires the value of the distance D input in the input field 202 of the UI screen shown in FIG. 2(b) described above.

[0054] In S905, similarly to S407 in the flow of FIG. 4 of the first embodiment, the virtual viewpoint information conversion unit 114 converts the virtual viewpoint information generated in S903 into facing virtual viewpoint information using the above-described formulas (8) to (15). The facing virtual viewpoint information of the current frame after conversion is saved in the RAM 313. The converted facing virtual viewpoint information is then saved in the virtual viewpoint information saving unit 113, and will be used as facing virtual viewpoint information of the previous frame in the calculation of facing virtual viewpoint information in S913, which will be described later.

[0055] In S906, the virtual viewpoint information providing unit 115 transmits the facing virtual viewpoint information of the current frame obtained in S905 to the virtual viewpoint image generating device 102. The virtual viewpoint image generating device 115 generates a virtual viewpoint image corresponding to the received facing virtual viewpoint information and returns the virtual viewpoint image to the virtual viewpoint control device 110. The virtual viewpoint control device 110 then displays the received virtual viewpoint image from the facing position on the UI screen, and the operator checks this and, if OK, gives an instruction to register a key frame on the UI screen.

[0056] In S907, the next process to be executed is determined depending on whether or not a key frame registration instruction has been input. If a key frame registration instruction has been input, S908 is executed next. On the other hand, if no instruction has been input, S908 is skipped and S909 is executed next.

[0057] In S908, the key frame registration unit 116 associates the virtual viewpoint information generated in S903 or the converted virtual viewpoint information obtained in S914 (described later) with the time code of the current frame and registers it as a key frame.

[0058] In S909, it is determined whether or not to end the key frame registration. If an instruction to end the key frame registration has been input, S916 is then executed. At this time, if two or more key frames have not been registered, a warning may be displayed. On the other hand, if an instruction to end the key frame registration has not been input, S910 is then executed.

[0059] In S910, the virtual viewpoint information setting unit 112 acquires the virtual viewpoint information of the previous frame from the virtual viewpoint information holding unit 113.

[0060] In S911, the virtual viewpoint information setting unit 112 acquires, from the tracking unit 801, tracking information of the target object in the current frame that is the processing target.

[0061] In S912, the camera operation signal acquisition unit 111 acquires a camera operation signal corresponding to the current frame from the controller 104. At this time, what the operator is operating is not a virtual camera corresponding to the viewpoint of the target object, but a virtual camera that has been moved to a facing position (hereinafter referred to as a "facing virtual camera").

[0062] In S913, the virtual viewpoint information setting unit 112 calculates the facing virtual viewpoint information of the current frame based on the facing virtual viewpoint information of the previous frame acquired in S910 and the camera operation signal acquired in S912. This calculation uses the following equations (16) to (22). JPEG2025125421000007.jpg36126 In the above formulas (16) to (22), [X' curr ,Y' curr ,Z' curr ,Pan' curr ,Tilt' curr,Roll' curr ,Zoom' curr ] represents the virtual viewpoint information of the current frame. prev ,Y' prev ,Z' prev ,Pan' prev ,Tilt' prev ,Roll' prev ,Zoom' prev ] represents the face-to-face virtual viewpoint information of the previous frame. The calculated face-to-face virtual viewpoint information is stored in the virtual viewpoint information storage unit 113 and is used as the face-to-face virtual viewpoint information of the previous frame when calculating face-to-face virtual viewpoint information for the next current frame.

[0063] In S914, the virtual viewpoint information conversion unit 114 converts the facing virtual viewpoint information generated in S913 into virtual viewpoint information. This conversion is in the opposite direction to the conversion in S905. In other words, the virtual camera represented by the virtual viewpoint information after conversion is restricted to rotate around a vertical axis passing through the three-dimensional position of the facing virtual camera represented by the facing virtual viewpoint information before conversion. The conversion in this step is carried out by, for example, modifying the above-mentioned formulas (8) to (14) to convert the known facing virtual viewpoint information [X' curr ,Y' curr ,Z' curr ,Pan' curr ,Tilt' curr ,Roll' curr ,Zoom' curr ], the unknown virtual viewpoint information [X curr ,Y curr ,Z curr ,Pan curr ,Tilt curr ,Roll curr ,Zoom curr The converted virtual viewpoint information of the current frame is stored in the RAM 313. Then, the process returns to S906, and the facing virtual viewpoint information calculated in S913 is transmitted to the virtual viewpoint image generation device 102.

[0064] In S915, it is determined whether or not to start creating a camera path. If a command to create a camera path has been input, S916 is executed. On the other hand, if a command to create a camera path has not been input, the process waits for a certain period of time and then determines again whether or not an input has been made.

[0065] In S916, the camera path creation unit 117 creates a camera path using the key frames registered by the key frame registration unit 116. The details of the camera path creation process are the same as those explained in the flow of Fig. 7 above, and there are no particular differences, so explanations will be omitted.

[0066] The above is the content of the operation flow of the virtual viewpoint control device 110 according to this embodiment. In the face-to-face movement mode of this embodiment, the operation content performed by the operator on the face-to-face virtual camera is reflected in the virtual viewpoint information.

[0067] As described above, according to this embodiment, virtual viewpoint information is calculated based on tracking information of the target object, and an image based on face-to-face virtual viewpoint information converted from the virtual viewpoint information is displayed on the UI screen. In this embodiment, the operator does not need to follow the target object moving freely around in three-dimensional space with his or her own eyes and operate a joystick or the like, but can register key frames by operating while viewing only an image capturing the target object from the front, making it easier to create camera passes from the player's perspective.

[0068] <Other embodiments> The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0069] The present disclosure also includes the following configurations and methods.

[0070] [Configuration 1] an acquisition means for acquiring an operation signal that specifies the position and orientation in a virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a determining means for determining a second virtual viewpoint facing the first virtual viewpoint specified by the acquired operation signal; a setting means for setting virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; A virtual viewpoint control device comprising:

[0071] [Configuration 2] 2. The virtual viewpoint control device according to configuration 1, wherein the determining means determines the position of the second virtual viewpoint to be a position obtained by rotating the first virtual viewpoint around a vertical axis passing through the position of the first virtual viewpoint.

[0072] [Configuration 3] 3. The virtual viewpoint control device according to configuration 2, wherein the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint is determined based on an instruction from the operator.

[0073] [Configuration 4] The determining means makes the determination using the following formula: JPEG2025125421000008.jpg36124 In the above formulas, D represents the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint, X' curr、 Y' curr、 Z' curr、 represents the position of the second virtual viewpoint in the three-dimensional coordinate system, Pan' curr、 Tilt' curr、 Roll' curr、 represents the orientation of the second virtual viewpoint, Zoom' curr represents the focal length of the second virtual viewpoint, Xcurr、 Y curr、 Z curr、 represents the position of the first virtual viewpoint in the three-dimensional coordinate system, Pan curr、 Tilt curr、 Roll curr、 represents the orientation of the first virtual viewpoint, Zoom curr represents the focal length of the first virtual viewpoint, 4. The virtual viewpoint control device according to configuration 3.

[0074] [Configuration 5] The determining means makes the determination using the following formula: JPEG2025125421000009.jpg36124 In the above formulas, D represents the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint, X' curr、 Y' curr、 Z' curr、 represents the position of the second virtual viewpoint in the three-dimensional coordinate system, Pan' curr、 Tilt' curr、 Roll' curr、 represents the orientation of the second virtual viewpoint, Zoom' curr represents the focal length of the second virtual viewpoint, X curr、 Y curr、 Z curr、 represents the position of the first virtual viewpoint in the three-dimensional coordinate system, Pan curr、 Tilt curr、 Roll curr、 represents the orientation of the first virtual viewpoint, Zoom curr represents the focal length of the first virtual viewpoint, 4. The virtual viewpoint control device according to configuration 3.

[0075] [Configuration 6] The operation mode includes a first mode and a second mode, In the first mode, The determining means makes the determination, the setting means sets virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; In the second mode, The determining means does not make the determination, the setting means sets virtual viewpoint information representing the first virtual viewpoint in response to an instruction from an operator. 6. The virtual viewpoint control device according to any one of configurations 1 to 5,

[0076] [Configuration 7] the virtual viewpoint image generated in accordance with the virtual viewpoint information set by the setting means is an image that reproduces a viewpoint of an object shown in the plurality of captured images, One of the first virtual viewpoint and the second virtual viewpoint is a virtual viewpoint corresponding to a viewpoint of the object. 7. The virtual viewpoint control device according to any one of configurations 1 to 6,

[0077] [Configuration 8] the acquiring means further acquires tracking information indicating a three-dimensional position of the object at each time during an imaging period of the plurality of captured images; the determining means determines the position of the first virtual viewpoint based on the acquired tracking information, and determines the orientation of the first virtual viewpoint based on the acquired operation signal. 8. The virtual viewpoint control device according to configuration 7.

[0078] [Configuration 9] the plurality of captured images and the virtual viewpoint image are moving images, the setting means sets the virtual viewpoint information corresponding to a specific frame in the plurality of captured images as a key frame constituting a camera path indicating a trajectory of the virtual viewpoint; 9. The virtual viewpoint control device according to any one of configurations 1 to 8,

[0079] [Configuration 10] 10. An image processing system including: the virtual viewpoint control device according to configuration 9; and a virtual viewpoint image generation device that performs rendering processing according to the camera path to generate the virtual viewpoint image.

[0080] [Method 1] an acquisition step of acquiring an operation signal that specifies the position and orientation in a virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices; a determining step of determining a second virtual viewpoint facing the first virtual viewpoint identified by the acquired operation signal; a setting step of setting virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; A virtual viewpoint control method comprising:

[0081] [Configuration 12] A program for causing a computer to function as the virtual viewpoint control device according to any one of configurations 1 to 9.

Claims

1. an acquisition means for acquiring an operation signal that specifies the position and orientation in a virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images by a plurality of imaging devices; a determining means for determining a second virtual viewpoint facing the first virtual viewpoint specified by the acquired operation signal; a setting means for setting virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; A virtual viewpoint control device comprising:

2. 2. The virtual viewpoint control device according to claim 1, wherein the determining means determines the position of the second virtual viewpoint to be a position obtained by rotating the first virtual viewpoint about a vertical axis passing through the position of the first virtual viewpoint.

3. 3. The virtual viewpoint control device according to claim 2, wherein the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint is determined based on an instruction from the operator.

4. The determining means makes the determination using the following formula: In the above formulas, D represents the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint, X' curr、 Y' curr、 Z' curr、 represents the position of the second virtual viewpoint in the three-dimensional coordinate system, Pan' curr、 Tilt' curr、 Roll' curr、 represents the orientation of the second virtual viewpoint, Zoom' curr represents the focal length of the second virtual viewpoint, X curr、 Y curr、 Z curr、 represents the position of the first virtual viewpoint in the three-dimensional coordinate system, Pan curr、 Tilt curr、 Roll curr、 represents the orientation of the first virtual viewpoint, Zoom curr represents the focal length of the first virtual viewpoint, 4. The virtual viewpoint control device according to claim 3.

5. The determining means makes the determination using the following formula: In the above formulas, D represents the distance between the position of the first virtual viewpoint and the position of the second virtual viewpoint, X' curr、 Y' curr、 Z' curr、 represents the position of the second virtual viewpoint in the three-dimensional coordinate system, Pan' curr、 Tilt' curr、 Roll' curr、 represents the orientation of the second virtual viewpoint, Zoom' curr represents the focal length of the second virtual viewpoint, X curr、 Y curr、 Z curr、 represents the position of the first virtual viewpoint in the three-dimensional coordinate system, Pan curr、 Tilt curr、 Roll curr、 represents the orientation of the first virtual viewpoint, Zoom curr represents the focal length of the first virtual viewpoint, 4. The virtual viewpoint control device according to claim 3.

6. The operation mode includes a first mode and a second mode, In the first mode, The determining means makes the determination, the setting means sets virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; In the second mode, The determining means does not make the determination, the setting means sets virtual viewpoint information representing the first virtual viewpoint in response to an instruction from an operator.

2. The virtual viewpoint control device according to claim 1.

7. the virtual viewpoint image generated in accordance with the virtual viewpoint information set by the setting means is an image that reproduces a viewpoint of an object shown in the plurality of captured images, one of the first virtual viewpoint and the second virtual viewpoint is a virtual viewpoint corresponding to a viewpoint of the object; 2. The virtual viewpoint control device according to claim 1.

8. the acquiring means further acquires tracking information indicating a three-dimensional position of the object at each time during an imaging period of the plurality of captured images; the determining means determines the position of the first virtual viewpoint based on the acquired tracking information, and determines the orientation of the first virtual viewpoint based on the acquired operation signal; 8. The virtual viewpoint control device according to claim 7, wherein:

9. the plurality of captured images and the virtual viewpoint image are moving images, the setting means sets the virtual viewpoint information corresponding to a specific frame in the plurality of captured images as a key frame constituting a camera path indicating a trajectory of the virtual viewpoint; 2. The virtual viewpoint control device according to claim 1.

10. 10. An image processing system comprising: the virtual viewpoint control device according to claim 9; and a virtual viewpoint image generation device that performs rendering processing according to the camera path to generate the virtual viewpoint image.

11. an acquisition step of acquiring an operation signal that specifies the position and orientation in a virtual space of a virtual viewpoint corresponding to a virtual viewpoint image generated based on a plurality of captured images obtained by capturing images using a plurality of imaging devices; a determining step of determining a second virtual viewpoint facing the first virtual viewpoint specified by the acquired operation signal; a setting step of setting virtual viewpoint information representing the first virtual viewpoint or the second virtual viewpoint in response to an instruction from an operator based on a virtual viewpoint image corresponding to the determined second virtual viewpoint; A virtual viewpoint control method comprising:

12. A program for causing a computer to execute the control method according to claim 11.

Images