Image processing apparatus, image processing method, and program

Abstract

To facilitate cooperation of virtual viewpoint images.SOLUTION: The image processing apparatus includes a first viewpoint determination unit configured to generate camera parameters of a first virtual camera based on an input value from a first input device, a reproduction position determination unit configured to determine a reproduction position of a virtual viewpoint image to be reproduced based on the input value from the first input device, and a first image generation unit configured to generate a first virtual viewpoint image based on the first virtual camera and the reproduction position. A second viewpoint determination unit configured to generate camera parameters of a second virtual camera based on an input value from the first input device and an input value from a second input device; and a second video image generation unit configured to generate a second virtual viewpoint video image based on the second virtual camera and the reproduction position.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method, and a program for processing a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology has emerged that uses multiple cameras installed in different locations to capture images from multiple viewpoints simultaneously, and then generates virtual viewpoint images from one or more arbitrary viewpoints, in addition to the images from the camera installation positions, using the multiple viewpoint images obtained through the capture. Services using virtual viewpoint images allow video producers to create content with powerful viewpoints from footage of, for example, a soccer or basketball game. Furthermore, users viewing the content can freely move their viewpoints while watching the game, allowing them to view virtual viewpoint images from any location.

[0003] Patent Document 1 discloses a method for controlling the position of a virtual camera to realize a composition desired by the operator of the virtual camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-109719 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 did not take into consideration the coordination of virtual viewpoint images among multiple people. For example, when a coach uses virtual viewpoint images generated from video footage of a basketball game to instruct a player, a use case is conceivable in which the coach and the player view the virtual viewpoint images on different devices. In this case, it was difficult for the coach to coordinate virtual viewpoint images, such as by having the player manipulate the virtual viewpoint image to some extent to understand the surrounding situation while reflecting the scenes the coach wants to show the player for instruction in the virtual viewpoint image on the player's side.

[0006] Therefore, an object of the present invention is to facilitate the linking of virtual viewpoint images. [Means for solving the problem]

[0007] The image processing device generates a virtual viewpoint image from a plurality of images captured by a plurality of imaging devices, and includes a first viewpoint determination means for generating camera parameters of a first virtual camera based on input values ​​from a first input device, a playback position determination means for determining a playback position of a virtual viewpoint image to be played back based on the input values ​​from the first input device, a first image generation means for generating a first virtual viewpoint image based on the first virtual camera and the playback position, a second viewpoint determination means for generating camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device, and a second image generation means for generating a second virtual viewpoint image based on the second virtual camera and the playback position. [Effects of the Invention]

[0008] The present invention makes it possible to easily link virtual viewpoint images. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration diagram of a virtual viewpoint image generation system according to a first embodiment. [Figure 2]FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing apparatus and an image processing apparatus according to the first embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing system according to a first embodiment. [Figure 4] FIG. 3 is a diagram illustrating an example of a display form of a virtual viewpoint image in the first embodiment. [Figure 5] 4 is a flowchart illustrating an example of processing performed by the image processing apparatus according to the first embodiment. [Figure 6] FIG. 10 is a diagram showing an example of a display form of a virtual viewpoint image in a modified example of the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of a functional configuration of an image processing system according to a second embodiment. [Figure 8] 10 is a flowchart illustrating an example of processing by an image processing apparatus according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations may be omitted.

[0011] First Embodiment In this embodiment, a system is described that generates a virtual viewpoint image representing a view from a specified virtual viewpoint based on a plurality of images captured by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user, and also includes, for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates. Furthermore, in this embodiment, the case where the virtual viewpoint is specified by a user operation is mainly described, but the virtual viewpoint may also be specified automatically based on the results of image analysis, etc.

[0012] In addition, in this embodiment, the description will be made using the term "virtual camera." The virtual camera is a virtual camera that is different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently describing a virtual viewpoint related to the generation of a virtual viewpoint image. In other words, the virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in this virtual imaging can be expressed as the position and orientation of the virtual camera. In other words, the virtual viewpoint image can be said to be an image that simulates an image captured by a camera when it is assumed that the camera exists at the position of the virtual viewpoint set in space.

[0013] [Configuration of the virtual viewpoint image generation system] First, the configuration of a virtual viewpoint image generation system according to this embodiment will be described with reference to FIGS.

[0014] The virtual viewpoint image generation system of this embodiment has n sensor systems, from sensor system 101a to sensor system 101n, and each sensor system has at least one camera, which is an image capturing device. Hereinafter, unless otherwise specified, the n sensor systems will not be distinguished from one another and will be referred to as multiple sensor systems 101.

[0015] FIG. 1(A) is a diagram showing an example of installation of multiple sensor systems 101. The multiple sensor systems 101 are installed to surround an area 120, which is an area to be photographed, and each sensor system photographs the area 120 from a different direction. In the example of this embodiment, the area to be photographed 120 is assumed to be the field of a stadium where a soccer match is played, and n (e.g., 100) sensor systems 101 are installed to surround the field. However, the number of sensor systems 101 to be installed is not limited, and the area to be photographed 120 is not limited to the stadium field. For example, the area 120 may include stadium seating, or the area 120 may be an indoor studio, stage, or the like.

[0016] Furthermore, the multiple sensor systems 101 do not have to be installed around the entire periphery of the area 120, and may be installed only in a part of the periphery of the area 120 depending on restrictions on installation location, etc. Furthermore, the multiple cameras of the multiple sensor system 101 may include imaging devices with different functions, such as a telephoto camera and a wide-angle camera.

[0017] The multiple cameras included in the multiple sensor systems 101 capture images synchronously. The multiple images obtained by capturing images from these cameras are called multi-view images. Note that each of the multi-view images in this embodiment may be a captured image, or may be an image obtained by performing image processing on the captured image, such as a process of extracting a predetermined area.

[0018] The multiple sensor systems 101 may have a microphone (not shown) in addition to a camera. The microphones of the multiple sensor systems 101 synchronously collect audio. Based on this collected audio, an acoustic signal can be generated that is played back together with the display of an image on the image display device 104. For the sake of simplicity, a description of audio will be omitted below, but it is assumed that images and audio are basically processed together.

[0019] The image processing system 1 is composed of an image processing device 20, a first information processing device 100, a second information processing device 200, etc., and acquires multi-viewpoint images from multiple sensor systems 101 and stores them in a database (not shown) together with the time code used for capturing the images. The time code is information for uniquely identifying the time when the image capturing device captured the image, and can be specified in a format such as "day:hour:minute:second.frame number." The image processing device 20 then generates a virtual viewpoint image from these stored multi-viewpoint images.

[0020] The first information processing device 100 displays the virtual viewpoint image received from the image processing device 20 on the first display device 30. The first information processing device 100 also receives a user's manual operation on the virtual camera via the first input device 10 and provides the image processing device 20. The second information processing device 200 displays the virtual viewpoint image received from the image processing device 20 on the second display device 50. The second information processing device 200 also receives a user's manual operation on the virtual camera via the second input device 40 and provides the image processing device 20. Here, the first information processing device 100 and the second information processing device 200 are, for example, portable terminals such as tablet terminals, smartphones, and head-mounted displays (HMDs).

[0021] The virtual camera 110 is set in a virtual space associated with the area 120, and can be set to a viewpoint different from that of any of the cameras of the multiple sensor systems 101. Here, different virtual cameras are set for the virtual viewpoint image provided to the first information processing device 100 and the virtual viewpoint image provided to the second information processing device 200. The virtual viewpoint image generated by the image processing device 20 is an image that represents the view from the virtual camera 110. Note that the virtual viewpoint image in the embodiment is also called a free viewpoint image.

[0022] In this embodiment, the virtual viewpoint image is mainly a moving image, but the virtual viewpoint image may be a still image.

[0023] [Configuration of image processing device and information processing device] Next, an example of the hardware configuration of the information processing device and image processing device according to this embodiment will be described with reference to FIGS. 2(A) and 2(B).

[0024] 2A shows an example of the hardware configuration of an information processing device. Here, the first information processing device 100 will be described, but the second information processing device 200 also has a similar configuration.

[0025] The CPU 101 uses the RAM 102 as a work memory, executes programs stored in the ROM 103 and / or the hard disk drive (HDD) 105, and controls each component described later via the system bus 112. This allows various processes described later to be executed.

[0026] The HDD interface (I / F) 104 is an interface such as Serial ATA (SATA) that connects the information processing device to a secondary storage device such as an HDD 105, an optical disk drive, an SSD, or a flash memory. In this embodiment, the HDD 105 is used as an example of a secondary storage device. The CPU 101 can read data from and write data to the HDD 105 via the HDD interface (I / F) 104. Furthermore, the CPU 101 loads data stored in the HDD 105 into the RAM 102. The CPU 101 can also save various data on the RAM 102 obtained by executing a program into the HDD 105.

[0027] The input interface (I / F) 106 connects the information processing device to an input device 107, such as a touch panel, keyboard, mouse, digital camera, or scanner, for inputting one or more coordinates. The input interface (I / F) 106 is, for example, a serial bus interface such as USB or IEEE1394. The CPU 101 can read data from the input device 107 via the input I / F 106. In this embodiment, the input interface 106 is connected to the first input device 10.

[0028] The output interface (I / F) 108 connects an output device 109 such as a display to the information processing device. The output interface (I / F) 108 is, for example, a video output interface such as DVI or HDMI (registered trademark). The CPU 101 can display the virtual viewpoint video by sending data related to the virtual viewpoint video to the output device 109 via the output I / F 108. The network interface (I / F) 110 is, for example, a network card such as a LAN card that connects the information processing device to an external server 111. The CPU 101 can read data from the external server 111 via the network I / F 110. In this embodiment, the output interface 108 is connected to the first display device 30.

[0029] 2B shows an example of the hardware configuration of the image processing device 20. A CPU 201 uses a RAM 202 as a work memory, executes a program stored in a ROM 203, and controls each component described below.

[0030] The communication unit 204 connects to an information processing device and performs data communication. The communication unit 204 performs communication in accordance with communication standards such as Ethernet or IEEE802.11 (so-called wireless LAN). The CPU 201 transmits and receives data to and from external devices such as information processing devices via the communication unit 204.

[0031] The input / output unit 205 inputs and outputs data via an input interface and an output interface, and is connected to devices such as a mouse, a keyboard, a display, and a digital camera.

[0032] The GPU 206 is a computing device or processor specialized for image processing. The GPU 206 performs image processing such as generating a virtual viewpoint image from images input from the multiple sensor systems 101.

[0033] [Explanation of the image processing system] Next, an example of the functional configuration of the image processing system 1 in this embodiment will be described with reference to FIGS. 3 and 4. In this embodiment, a use case will be described in which a user using the first display device 30 can also operate the virtual viewpoint image displayed on the second display device 50. This use case assumes, for example, that the first display device 30 is used by a sports coach, and the second display device 50 is used by an athlete receiving instruction from the coach. In such a case, it is expected that the coach will reflect the scene he wants to show the athlete in the athlete's virtual viewpoint image for instruction purposes, while allowing the athlete to operate the virtual viewpoint image to some extent to understand the surrounding situation. Therefore, in this embodiment, an image processing system will be described that is particularly suitable for such coaching.

[0034] The playback position determination unit 301 determines the playback position of the virtual viewpoint image in accordance with an instruction input from the first input device 10 of the first information processing device 100. The playback position is, for example, a time code.

[0035] The first viewpoint determination unit 302 determines the camera parameters of the virtual viewpoint camera according to instructions input from the first input device 10 of the first information processing device 100. The camera parameters of the virtual viewpoint camera are, for example, position and orientation. The position of the virtual camera is expressed, for example, by three-dimensional coordinates (x, y, z). The orientation of the virtual camera is expressed, for example, by three directions (pan, tilt, roll).

[0036] The first video generation unit 303 generates a virtual viewpoint image from the playback position determined by the playback position determination unit 301 and the position and orientation of the virtual camera determined by the first viewpoint determination unit 302. The virtual viewpoint image generated by the first video generation unit 303 is displayed on the first display device 30 of the first information processing device 100.

[0037] The second viewpoint determination unit 304 determines the position and orientation of the virtual viewpoint camera according to instructions input from the second input device 40 of the second information processing device 200. The position of the virtual camera is expressed, for example, by three-dimensional coordinates (x, y, z). The orientation of the virtual camera is expressed, for example, by three directions (pan, tilt, roll).

[0038] The second video generation unit 305 generates a virtual viewpoint image from the playback position determined by the playback position determination unit 301, the position and orientation of the virtual camera determined by the first viewpoint determination unit 302, and the position and orientation of the virtual camera determined by the second viewpoint determination unit 304. Unlike the first video generation unit 303, the second video generation unit 305 generates a virtual viewpoint image using information on the positions and orientations of the virtual cameras determined by the multiple viewpoint determination units. The virtual viewpoint image generated by the second video generation unit 305 is displayed on the first display device 30 of the first information processing device 100 and the second display device 50 of the second information processing device 200.

[0039] In this embodiment, the first video generation unit 303 and the second video generation unit 305 generate virtual viewpoint images using the playback positions determined by the playback position determination unit 301, but a playback position determination unit may be provided for each generation unit. That is, in the example of this embodiment, two playback position determination units may be provided, and the first video generation unit 303 and the second video generation unit 305 may each generate a virtual viewpoint image using the playback positions determined by the respective playback position determination units.

[0040] Next, the virtual viewpoint images and graphical user interfaces (GUIs) displayed on the first display device 30 and the second display device 50 will be described with reference to Figures 4(A) to 4(D). In this embodiment, a configuration will be described in which the first display device 30 is used by a coach and the second display device 50 is used by a player. Here, Figures 4(A) to 4(D) show game footage of a basketball game, with the rectangular parallelepiped objects being simplified figures that represent humans and the spheres being figures that represent basketballs.

[0041] Fig. 4(A) is an example of a GUI displayed on the first display device 30. Figs. 4(B) to 4(D) are each an example of a GUI displayed on the second display device 50. The GUI displayed on the second display device 50 can be switched depending on the device configuration or device type of the second information processing device 200, or on the mode selection from the second information processing device 200, such as window mode or full screen mode.

[0042] The virtual viewpoint image 401 is a virtual viewpoint image that is displayed on both the first display device 30 and the second display device 50. The virtual viewpoint image 402 is an image that is displayed only on the first display device 30. In other words, the virtual viewpoint image 401 is a virtual viewpoint image that is viewed by both the players and the coach, and the virtual viewpoint image 402 is a virtual viewpoint image that is viewed only by the coach.

[0043] When the coach is not performing any operation, the position of the virtual camera in the virtual viewpoint image 401 is fixed regardless of the player's operation. This is because the position specified by the coach is considered to be the viewpoint position that the coach wants to show to the player for coaching purposes. However, the orientation of the virtual camera in the virtual viewpoint image 401 can be changed by the player's operation. For example, if the second input device 40 is a touch panel, the player can change the orientation of the virtual camera in the virtual viewpoint image 401 by a swipe operation. Alternatively, if the second input device 40 is a mouse, the player can change the orientation of the virtual camera in the virtual viewpoint image 401 by a drag operation. This allows the coach to reflect the scene that the coach wants to show the player for coaching purposes in the player's virtual viewpoint image, while allowing the player to operate the virtual viewpoint image to some extent. Note that the position of the virtual camera does not have to be fixed, but may be changeable within a predetermined range centered on the virtual camera position specified by the coach.

[0044] The virtual viewpoint image 402 is a virtual viewpoint image used by the coach to prepare the next virtual viewpoint image that the coach wants to show to the players. The coach can change the position and attitude of the virtual camera of the virtual viewpoint image 402 using the first input device 10. For example, if the first input device 10 is a touch panel, the coach can change the attitude of the virtual camera of the virtual viewpoint image 402 by a swipe operation. Alternatively, if the first input device 10 is a mouse, the coach can change the attitude of the virtual camera of the virtual viewpoint image 402 by a drag operation.

[0045] Here, when the coach selects the viewpoint share button 408 using the first input device 10, the position and orientation of the virtual camera of the virtual viewpoint image 402 are applied to the position and orientation of the virtual viewpoint camera of the virtual viewpoint image 401. This allows the coach to switch the virtual viewpoint image to be shown to the players. Furthermore, when the coach selects the swap button 409 using the first input device 10, the position and orientation of the virtual camera of the virtual viewpoint image 402 and the position and orientation of the virtual viewpoint camera of the virtual viewpoint image 401 are swapped. This allows the coach to have the players view the virtual viewpoint images while switching between multiple virtual viewpoints, for example, when coaching while including the view of an opposing player.

[0046] It should be noted that when the viewpoint share button 408 is selected, the camera parameters of the virtual camera of the virtual viewpoint image 402 do not have to be immediately applied to the virtual camera of the virtual viewpoint image 401. For example, after the coach selects the viewpoint share button 408, a message informing the viewer that the viewpoint will be switched may be displayed for a few seconds in the virtual viewpoint image 401 before the camera parameters of the virtual camera of the virtual viewpoint image 402 are applied to the virtual camera of the virtual viewpoint image 401. This makes it possible to prevent viewers from becoming confused by sudden changes in the position and attitude of the virtual viewpoint of the virtual viewpoint image 401. It should be noted that this process can also be applied when the swap button 409 is selected.

[0047] When the viewpoint share button 408 is selected, a button for selecting whether or not to apply the camera parameters of the virtual camera of the virtual viewpoint image 402 may be superimposed on the virtual viewpoint image 401. When the button displayed on the virtual viewpoint image 401 is used to agree to apply the camera parameters of the virtual camera of the virtual viewpoint image 402 to the virtual viewpoint image 401, processing for viewpoint sharing is executed. This makes it possible to prevent the virtual viewpoint from being switched at a timing unintended by the player. This processing can also be applied when the swap button 409 is selected.

[0048] The first information processing device 100 may store the positions and orientations of multiple virtual cameras designated by the coach and determine the virtual viewpoint by selecting from the stored list. This allows the coach to change to a desired viewpoint without having to operate the virtual camera viewpoint again each time. For example, by storing the positions and orientations of virtual cameras in frequently-attracted locations, such as under the basket on a basketball court, the coach can switch to a virtual viewpoint image of a desired viewpoint simply by selecting from the list as needed. By providing such an operation method, the operation performed by the user can be simplified.

[0049] Furthermore, the image processing system 1 may save a series of virtual camera camera parameters and store them as a virtual camera path. However, the image processing system 1 is not limited to this method. Instead, the image processing system 1 may specify multiple camera parameters for basic frames and automatically interpolate the camera parameters between key frames to create and store a camera path. The image processing system 1 generates a virtual viewpoint image 401 that reproduces the path of the virtual camera based on the above-mentioned camera path. This allows the coach to set the viewpoint he or she wants the player to see across multiple frames.

[0050] The seek bar 403 is a GUI that indicates the playback position (time code) of the virtual viewpoint image 402. The coach can select any playback position by selecting a position on the seek bar 403. The reverse playback button 404 is a button for controlling reverse playback of the virtual viewpoint image 402. The pause / resume playback button 405 is a button for controlling pausing or resuming playback of the virtual viewpoint image 402. The forward playback button 406 is a button for forward playback of the virtual viewpoint image 402. The playback speed 407 is an item for changing the playback speed of the virtual viewpoint image 402. The coach can change the playback speed of the virtual viewpoint image 402 from options presented by direct input or a pull-down menu. For example, when 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played at normal speed. When a value less than 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played back slowly. When a value greater than 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played back at high speed.

[0051] 4(B) is an example of a GUI displayed on the second display device 50. This GUI is a GUI that is displayed by the second information processing device 200 on a PC, a tablet terminal, or the like, for example.

[0052] The seek bar 413 is a GUI that indicates the playback position (time code) of the virtual viewpoint image 401. A player can select any playback position by selecting a position on the seek bar 413. The reverse playback button 414 is a button for controlling reverse playback of the virtual viewpoint image 401. The pause / resume playback button 415 is a button for controlling pausing or resuming playback of the virtual viewpoint image 401. The forward playback button 416 is a button for forward playback of the virtual viewpoint image 401. The playback speed 417 is an item for changing the playback speed of the virtual viewpoint image 401. A player can change the playback speed of the virtual viewpoint image 401 by direct input or from options presented by a pull-down menu. For example, when 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played at normal speed. When a value less than 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played back slowly. When a value greater than 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played back quickly.

[0053] 4(B), the time range of the playback position that can be changed using seek bar 413 may be smaller than the time range of the playback position that can be changed using seek bar 403. This makes it possible for the coach to prevent the player from changing from the scene (playback position) that the coach wants to instruct the player to a different scene (playback position).

[0054] FIG. 4(C) is an example of a GUI displayed on the second display device 50. This GUI is, for example, a GUI displayed by the second information processing device 200 on a mobile device such as a smartphone or tablet terminal. Unlike the GUI shown in FIG. 4(B), the GUI shown in FIG. 4(C) does not display a GUI for controlling the virtual viewpoint image 401. In this case, the athlete can change the orientation of the virtual camera of the virtual viewpoint image 401 using the touch panel or acceleration sensor of the second information processing device 200, but cannot perform other operations. The coach can display a GUI on the second information processing device 200 according to the purpose of coaching by appropriately switching between the GUI shown in FIG. 4(B) and the GUI shown in FIG. 4(C) by issuing an instruction from the first information processing device 100.

[0055] Fig. 4(D) is an example of a GUI displayed on the second display device 50. This GUI is, for example, a GUI displayed by the second information processing device 200 on a wearable device such as an HMD. Alternatively, the GUI of Fig. 4(D) is displayed by transitioning from the GUI shown in Fig. 4(B) or Fig. 4(C) to a full-screen mode in the second information processing device 200. In Fig. 4(D), the virtual viewpoint image 401 is displayed on the full screen, and an operation GUI and the like are not displayed.

[0056] 4(C) and 4(D) is displayed on the second display device 50, the playback position of the virtual viewpoint image 401 is changed using a seek bar 403 displayed on the first display device 30. In this case, the playback state of the virtual viewpoint image 401 is controlled by a reverse playback button 404, a pause / resume playback button 405, a forward playback button 406, and a playback speed 407.

[0057] In this embodiment, it may be configured so that changing only the attitude of the virtual camera of the virtual viewpoint image 401 cannot be performed from the first input device 10. For example, when a player is viewing the virtual viewpoint image 401 using an HMD, if the coach changes only the attitude of the virtual camera, the player may lose track of where he or she is looking, which may result in a situation where the player is unable to concentrate on viewing the virtual viewpoint image.

[0058] The GUI (any of FIGS. 4(B) to (D)) displayed on the second information processing device may be selected from the first information processing device 100. This allows the coach to control the content displayed on the second information processing device 200 used by the athlete depending on the instruction he or she wishes to give.

[0059] In this embodiment, the virtual viewpoint image 401 and the virtual viewpoint image 402 are played back in a loop, which allows the coach or player to repeatedly view the same scene without any operation.

[0060] If the playback speed 407 is changed while the virtual viewpoint image 402 is paused, the virtual viewpoint image 402 may automatically resume playback. This saves the user the trouble of pressing the play button. Similarly, if the playback speed 417 is changed while the virtual viewpoint image 401 is paused, the virtual viewpoint image 401 may automatically resume playback.

[0061] [Processing of image processing device 20] An example of processing by the image processing device 20 will be described below using the flowchart shown in Fig. 5. The processing by the image processing device 20 is realized by loading software recorded in the ROM 203 into the RAM 102 and executing it by the CPU 201 or the GPU 206.

[0062] In S501, the CPU 201 determines the playback position of the first virtual viewpoint video based on a first input value input from the first information processing device 100. The first virtual viewpoint video is, for example, the virtual viewpoint image 402. The first input value that is the target of the processing in this step is, for example, a control value corresponding to an operation input to the seek bar 403 by the first input device 10. This control value is specifically a time code. Note that if the first input value is not an input value related to the playback position, the processing in this step is not executed, and the processing in the next step is executed.

[0063] In S502, the CPU 201 determines a first virtual viewpoint of the first virtual viewpoint video based on a first input value input from the first information processing device 100. The first virtual viewpoint is, for example, the position and orientation of the virtual camera of the virtual viewpoint image 402. The first input value that is the target of the processing in this step is, for example, a control value corresponding to an operation to change the position or orientation of the virtual camera of the virtual viewpoint image 402, input to the first information processing device 100 by the first input device 10. Note that when the swap button 409 is selected, parameters similar to those of the second virtual viewpoint are set as the first virtual viewpoint. Note that if the first input value is not an input value related to the first virtual viewpoint, the processing in this step is not executed.

[0064] In S503, the CPU 201 determines the playback position of the second virtual viewpoint video based on a second input value input from the second information processing device 200. The second virtual viewpoint video is, for example, the virtual viewpoint image 401. The second input value that is the target of the processing in this step is, for example, a control value corresponding to an operation input to the seek bar 413 by the second input device 40. Specifically, this control value is a time code. Note that, if the second information processing device 200 does not have a UI for controlling the playback position such as the seek bar 413, the CPU 201 determines the playback position of the second virtual viewpoint video to be the same as the playback position determined in S501. Note that, if the second input value is not an input value related to the playback position, the processing in this step is not executed, and the processing in the next step is executed.

[0065] In S504, the CPU 201 determines a second virtual viewpoint of the second virtual viewpoint video based on a first input value input from the first information processing device 100 and a second input value input from the second information processing device 200. The second virtual viewpoint is, for example, the position and orientation of the virtual camera of the virtual viewpoint image 401. The first input value to be processed in this step is, for example, a control value corresponding to an operation for changing the position of the virtual camera of the virtual viewpoint image 401, input to the first information processing device 100 by the first input device 10. The second input value to be processed in this step is, for example, a control value corresponding to an operation for changing the orientation of the virtual camera of the virtual viewpoint image 401, input to the second information processing device 200 by the second input device 40. Note that when the swap button 409 is selected, the same parameters as those of the first virtual viewpoint immediately before the processing of S502 is executed are set as the second virtual viewpoint. Note that, if the second input value is not an input value related to the second virtual viewpoint, the processing of this step is not executed. In this step, the position of the virtual camera is determined based on the input value of the first input device 10, but it is not limited to this and may be determined based on the input value of the second input device 40. In this case, the first input value is ignored, and the second virtual viewpoint of the second virtual viewpoint video is determined based only on the second input value.

[0066] In S505, the CPU 201 generates a first virtual viewpoint image using the GPU 206 based on the playback position determined in S501 and the first virtual viewpoint determined in S502.

[0067] In S506, the CPU 201 determines whether a trigger is detected from the first input value. This trigger is to reflect the virtual viewpoint of the first virtual viewpoint image in the second virtual viewpoint image. For example, the CPU 201 detects a trigger when the viewpoint share button 408 or the swap button 409 in FIG. 4A is selected. If a trigger is detected from the first input value, the process of S507 is executed. If a trigger is not detected from the first input value, the process of S508 is executed.

[0068] In S507, the CPU 201 generates a second virtual viewpoint image using the GPU 206 based on the playback position determined in S501 and the first virtual viewpoint determined in S502. In S508, the CPU 201 generates a second virtual viewpoint image using the GPU 206 based on the playback position determined in S503 and the second virtual viewpoint determined in S504.

[0069] In S509, the CPU 201 outputs the first virtual viewpoint image and the second virtual viewpoint image. For example, the CPU 201 outputs the first virtual viewpoint image to the first information processing device 100 and the second virtual viewpoint image to the second information processing device 200 via the input / output unit 205.

[0070] In S510, the CPU 201 determines whether or not to end the processing of this flowchart. For example, if an application for reproducing a virtual viewpoint image is ended, the CPU 201 determines to end the processing of this flowchart. If the processing of this flowchart is not to be ended, the processing of S511 is executed.

[0071] In S511, the CPU 201 changes the playback position. For example, in the case of forward playback, the CPU 201 advances the playback position by one frame. In the case of reverse playback, the CPU 201 moves the playback position back by one frame. After this process ends, the process of this flowchart is repeated from S501.

[0072] The processing of the image processing device 20 has been described above using Fig. 5. This processing makes it possible to easily link a plurality of virtual viewpoint images. For example, a coach can have the player manipulate the virtual viewpoint image to some extent to understand the surrounding situation, while reflecting the scene that the coach wants to show the player for instruction in the virtual viewpoint image on the player's side.

[0073] The CPU 201 executes playback processes such as playing back, pausing, and changing the playback speed of virtual viewpoint images as needed by interrupt processing during the processing of the above-described flowchart.

[0074] In this embodiment, the image processing device 20 generates both the first virtual viewpoint image (virtual viewpoint image 402) and the second virtual viewpoint image (virtual viewpoint image 401), but this is not limiting. For example, two image processing devices 20 may be provided, each generating a first virtual viewpoint image and a second virtual viewpoint image, respectively. In this case, the two image processing devices 20 communicate with each other, transmitting and receiving the camera parameters of the virtual cameras of the respective virtual viewpoint images and the playback positions of the virtual viewpoint images.

[0075] <Modification of the first embodiment> In this embodiment, the display and control of a virtual viewpoint image in the first information processing device 100 and the processing of the image processing device 20 will be described. The same reference numerals will be used for the configurations, processing, and functions as those in the first embodiment, including the virtual viewpoint image generation system in Fig. 1 and the image display device 104 in Fig. 2, and descriptions thereof will be omitted.

[0076] Fig. 6 shows an example of a GUI displayed on the first display device 30. A virtual viewpoint image 601 is newly displayed in addition to the GUI in Fig. 4(A).

[0077] The virtual viewpoint image 601 is a virtual viewpoint image in which a virtual viewpoint is set to overlook the stadium (basketball court). Here, the black rectangular parallelepipeds are shapes that resemble players playing. By displaying the virtual viewpoint image 601, the coach can easily grasp the relative positions of the players during the game. In this embodiment, the virtual viewpoint of the virtual viewpoint image 601 is fixed and does not change.

[0078] Furthermore, pictorial symbols and icons may be superimposed on the virtual viewpoint image 601. Icon 602 represents the position and posture of the virtual camera in the virtual viewpoint image 401. Icon 603 represents the position and posture of the virtual camera in the virtual viewpoint image 402. To distinguish between icon 602 and icon 603, different colors or patterns may be set. By superimposing icon 602 and icon 603 on the virtual viewpoint image 601, the coach and the players can each confirm the positions and postures of their virtual viewpoints.

[0079] Here, when changing the position of the virtual camera in the virtual viewpoint image 402, the coach can change it by selecting (e.g., left-clicking) a position on the virtual viewpoint image 601. This makes it easy to instantaneously move the virtual camera, for example, when moving the position of the virtual camera from one end of the stadium to the other. The coach may also set the gaze point 604 by selecting (e.g., right-clicking) a position on the virtual viewpoint image 601. If the coach sets the gaze point 604 and then sets a new virtual camera position using a selection method (e.g., left-clicking) different from the method used to set the gaze point 604, the virtual camera's line of sight will rotate horizontally left and right (pan direction) to face the gaze point 604. For example, if the gaze point 604 is set below the basket, setting the virtual camera position to any point on the sideline within the court will automatically change the line of sight of the virtual camera on the sideline to below the basket. Furthermore, when the position of the gaze point 604 is changed, the current virtual camera's line of sight may automatically rotate to face the gaze point 604. Although an example in which the attitude (line of sight) of the virtual camera rotates has been described here, the line of sight of the virtual camera may be set to face the gaze point 604 at the same time that the position of the virtual camera is determined.

[0080] This allows the user to set a point of interest in advance and instantly view the point of interest simply by setting the position of the virtual camera at any location. Note that if the first input device 10 is a touch panel, the coach may select a position by tapping and select a point of gaze by double tapping.

[0081] In this case, in the flowchart of FIG. 5, the CPU 201 performs the following process in step S502.

[0082] In S502, the CPU 201 acquires, as first input values, coordinates selected in the virtual viewpoint image 601. Here, the coordinates acquired by the CPU 201 are two-dimensional coordinates on a plane horizontal to the ground (xy plane), and coordinates in a direction perpendicular to the ground (z-axis direction) are not acquired. The CPU 201 then applies the acquired two-dimensional coordinates to the current coordinates of the first virtual viewpoint. That is, the first virtual viewpoint image is translated while the height and orientation of the viewpoint remain unchanged.

[0083] Also, in S502, the CPU 201 acquires, as first input values, the coordinates of the point of interest 604 selected in the virtual viewpoint image 601. Here, the coordinates acquired by the CPU 201 are two-dimensional coordinates on a plane parallel to the ground (xy plane), and coordinates in a direction perpendicular to the ground (z-axis direction) are not acquired. The CPU 201 then sets the orientation of the first virtual viewpoint so that it faces the acquired two-dimensional coordinates. That is, the first virtual viewpoint image rotates in the pan direction so that it faces the point of interest 604, while leaving the tilt and roll of the virtual viewpoint orientation unchanged. Note that the CPU 201 may rotate the orientation of the virtual viewpoint not only in the pan direction but also in the tilt direction so that the orientation of the first virtual viewpoint becomes parallel to the ground.

[0084] This makes it easier for coaches to give instructions to their players.

[0085] Second Embodiment In the first embodiment, a process was described in which the position and orientation of the virtual camera of the virtual viewpoint image 402 is reflected in the position and orientation of the virtual camera of the virtual viewpoint image 401 based on a predetermined trigger. Next, as a second embodiment, a mode in which some parameters of the virtual camera of the virtual viewpoint image 402 are reflected in the virtual viewpoint image 401 and some parameters of the virtual camera of the virtual viewpoint image 401 are reflected in the virtual camera of the virtual viewpoint image 402 based on a predetermined trigger will be described. Note that the same reference numerals are used for configurations, processes, functions, etc. that are the same as those in the first embodiment, and descriptions thereof will be omitted.

[0086] 7 is a diagram showing an example of the functional configuration of the image processing system 1 according to this embodiment. The image processing device 70 includes a playback position determination unit 301, a first viewpoint determination unit 702, a first video generation unit 303, a second viewpoint determination unit 704, and a second video generation unit 305.

[0087] The first viewpoint determination unit 702 determines the camera parameters of the virtual camera based on input values ​​from the first input device 10 of the first information processing device 100 and some parameters of the position and orientation of the virtual camera from the second viewpoint determination unit 704. For example, the first viewpoint determination unit 702 calculates the camera parameters based on the input values ​​from the input device 10, and then applies the height of the viewpoint of the virtual camera acquired from the second viewpoint determination unit 704 to the calculated camera parameters. This application process may be performed constantly, or may be performed only when a trigger, such as pressing the viewpoint share button 408, is detected from the input values ​​from the first input device 10. In this embodiment, this application process will be described later with reference to FIG. 8, taking the case where a trigger is detected from the input values ​​from the first input device 10.

[0088] The first viewpoint determination unit 702 outputs the generated camera parameters, excluding the parameter (viewpoint height) applied by the second viewpoint determination unit 704, to the second video image generation unit 305. The second video image generation unit 305 acquires camera parameters other than the camera parameters acquired from the first viewpoint determination unit 702 from the second viewpoint determination unit 704 and generates a virtual viewpoint image. This makes it possible to prevent the virtual viewpoint from being positioned at an unintended position, such as high up or underground, even when viewing VR using an HMD on the second display device 50 for viewing a life-size basketball court. Furthermore, this processing makes it possible to maintain the viewer's eye level. Note that the camera parameters acquired by the first viewpoint determination unit 702 from the second viewpoint determination unit 704 are not limited to viewpoint height, and may also include orientation information (tilt, etc.).

[0089] Furthermore, when a trigger is detected from the input values ​​from the first input device 10, the first viewpoint determination unit 702 may reset some parameters of the virtual camera to specific initial values. For example, the first viewpoint determination unit 702 resets the tilt and roll of the virtual camera to be parallel to the ground, and outputs the camera parameters of the virtual camera excluding the reset parameters to the second image generation unit 305. This prevents the tilt and roll of the virtual camera of the virtual viewpoint image displayed on the HMD as the second display device 50 from being affected by the posture of the coach's virtual viewpoint. This prevents a mismatch between the head posture of the VR viewer and the posture of the virtual camera, and reduces VR sickness and other symptoms of the viewer caused by a shift in their sense of balance.

[0090] Furthermore, the first viewpoint determination unit 702 may restrict changes to some camera parameters of the virtual camera. For example, the first viewpoint determination unit 702 may restrict the roll of the virtual viewpoint so that it remains parallel to the ground and cannot be changed. In this case, the first viewpoint determination unit outputs the virtual camera parameters excluding the restricted parameters to the second image generation unit 305. This makes it possible to reduce the discrepancy between the image of the VR viewer and the head posture. Note that the restricted parameters are not limited to roll, and tilt may also be included. Furthermore, a UI may be displayed on the first display device 30 so that the coach can select the parameters to be restricted.

[0091] The second viewpoint determination unit 704 has the same configuration as in the first embodiment. Furthermore, as described above, the second viewpoint determination unit 704 of this embodiment determines camera parameters of the virtual camera based on input values ​​from the second input device 40 of the second information processing device 200, and outputs the camera parameters to the first viewpoint determination unit 702.

[0092] Fig. 8 is a flowchart showing the flow of processing according to this embodiment in which the camera parameters of the virtual cameras of the virtual viewpoint image 401 and the virtual viewpoint image 402 are mutually reflected based on a predetermined trigger. Note that, in the flow of Fig. 8, the processing of S505 is moved to S802 and S801 is newly added compared to the flow of Fig. 5 of the first embodiment. The other processing is the same as in the first embodiment.

[0093] In S801, the CPU 201 changes the camera parameters of the virtual camera of the first virtual viewpoint image determined by the first viewpoint determination unit 702 based on some of the parameters of the position and orientation of the virtual camera determined by the second viewpoint determination unit 704.

[0094] In S802, the CPU 201 generates a first virtual viewpoint image based on the playback position determined in S501 and the first virtual viewpoint determined in S502 using the GPU 206. Note that the processing in this step is similar to the processing in S505 of FIG. 5 in the first embodiment.

[0095] The above has described the processing according to this embodiment for mutually reflecting the camera parameters of the virtual cameras of the virtual viewpoint image 401 and the virtual viewpoint image 402 based on a predetermined trigger. This processing prevents a viewpoint unintended by the coach from being reflected on the player, and when the player is viewing VR, it is possible to reduce VR sickness caused by a mismatch between the position and orientation of the head and the position and orientation of the virtual camera.

[0096] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

Claims

1. An image processing device that generates a virtual viewpoint video from a plurality of images captured by a plurality of imaging devices, a first viewpoint determination means for generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determination means for determining a playback position of a virtual viewpoint image to be played back based on an input value from the first input device; a first image generating means for generating a first virtual viewpoint image based on the first virtual camera and the playback position; a second viewpoint determination means for generating camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device; a second image generating means for generating a second virtual viewpoint image based on the second virtual camera and the playback position; 1. An image processing device comprising:

2. An image processing method for generating a virtual viewpoint video from a plurality of images captured by a plurality of imaging devices, comprising: a first viewpoint determination step of generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determination step of determining a playback position of a virtual viewpoint image to be played back based on an input value from the first input device; a first image generating step of generating a first virtual viewpoint image based on the first virtual camera and the playback position; a second viewpoint determination step of generating camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device; a second image generating step of generating a second virtual viewpoint image based on the second virtual camera and the playback position; An image processing method comprising:

3. a first viewpoint determination step of generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determination step of determining a playback position of a virtual viewpoint image to be played back based on an input value from the first input device; a first image generating step of generating a first virtual viewpoint image based on the first virtual camera and the playback position; a second viewpoint determination step of generating camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device; a second image generating step of generating a second virtual viewpoint image based on the second virtual camera and the playback position; A program for causing an image processing device to execute the above.

4. a first reflecting means for reflecting at least one of the camera parameters of the first virtual camera to the camera parameters of the second virtual camera; 2. The image processing apparatus according to claim 1, further comprising second reflecting means for reflecting at least one of the camera parameters of the second virtual camera other than those specified by said first reflecting means into the first camera parameters.

Images