Image processing apparatus, image processing method, system, computer program product, storage medium, and computer-implemented method

Abstract

The invention provides an image processing apparatus, an image processing method, a system, a computer program product, a storage medium, and a computer-implemented method. An image processing apparatus for generating a virtual viewpoint video includes: a first viewpoint determination unit arranged to generate a camera parameter of a first virtual camera based on an input value from a first input device; a playback position determination unit arranged to determine a playback position of the virtual viewpoint image based on an input value from the first input device; a first video generation unit arranged to generate a first virtual viewpoint video based on the first virtual camera and the playback position; a second viewpoint determination unit arranged to generate a camera parameter of a second virtual camera based on an input value from the first input device and an input value from the second input device; and a second video generation unit arranged to generate a second virtual viewpoint video based on the second virtual camera and the playback position.

Description

Technical Field

[0001] The present invention relates to an image processing device for virtual viewpoint images, a control method and a program thereof. Background Art

[0002] In recent years, technologies have emerged that generate virtual viewpoint images from any one or more viewpoints in addition to images at camera positions by using multiple viewpoint images captured by synchronized photography using multiple cameras installed at different locations. Services that use virtual viewpoint images enable video producers to create powerful viewpoint content from video recordings of, for example, football or basketball games. These services also enable users who are viewing content to view the game while freely moving their viewpoints, thereby enabling users to view virtual viewpoint images from any position.

[0003] Japanese Patent Application Laid-Open No. 2012-109719 discloses a method of controlling the position of a virtual camera so as to realize a composition desired by an operator of the virtual camera.

[0004] However, the technology disclosed in Japanese Patent Laid-Open No. 2012-109719 does not consider the coordination of virtual viewpoint images between multiple users. For example, in a usage scenario where a coach instructs a player using a virtual viewpoint image generated from a video recording of a basketball game, it is conceivable that the coach and the player are viewing the virtual viewpoint image on separate devices. In this case, it is difficult for the coach to coordinate the virtual viewpoint image so that the player can operate the virtual viewpoint image to a certain extent to grasp his surroundings, while also reflecting the desired shot for instructing the player on the virtual viewpoint image on the player side. Summary of the invention

[0005] According to one aspect of the present invention, an image processing device includes: an acquisition unit, which is arranged to obtain a plurality of images captured by a plurality of camera devices; a first viewpoint determination unit, which is arranged to generate camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determination unit, which is arranged to determine the playback position of a virtual viewpoint image to be played back based on input values ​​from the first input device; a first video generation unit, which is arranged to generate a first virtual viewpoint video based on the plurality of images, the camera parameters of the first virtual camera, and the playback position; a second viewpoint determination unit, which is arranged to generate camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device; and a second video generation unit, which is arranged to generate a second virtual viewpoint video based on the plurality of images, the camera parameters of the second virtual camera, and the playback position. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A and Figure 1B is a diagram showing an example of the configuration of a virtual viewpoint image generation system according to the first embodiment.

[0007] Figure 2A is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the first embodiment.

[0008] Figure 2B is a block diagram showing an example of the hardware configuration of the image processing apparatus according to the first embodiment.

[0009] Figure 3 is a diagram showing an example of a functional configuration of an image processing system according to the first embodiment.

[0010] 4A to 4D is a diagram showing an example of a display form of a virtual viewpoint image according to the first embodiment.

[0011] Figure 5 is a flowchart showing an example of a processing procedure of the image processing apparatus according to the first embodiment.

[0012] Figure 6 is a diagram showing an example of a display form of a virtual viewpoint image in a modification of the first embodiment.

[0013] Figure 7 is a diagram showing an example of a functional configuration of an image processing system according to the second embodiment.

[0014] Figure 8 is a flowchart showing an example of a processing procedure of the image processing apparatus according to the second embodiment. DETAILED DESCRIPTION

[0015] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the following embodiments do not limit the claimed invention. Although the embodiments describe multiple features, not all of the multiple features are required by the present invention, and any combination of multiple features is possible. In other words, the embodiments described below can be implemented separately, or can be implemented as a combination of multiple elements or features when necessary or when it is beneficial to combine elements or features in separate embodiments into a single embodiment. In the accompanying drawings, identical or similar parts are represented by identical reference numerals, and redundant descriptions can be omitted.

[0016] First embodiment

[0017] This embodiment shows a system for generating a virtual viewpoint image representing a view from a specified virtual viewpoint based on multiple images captured by multiple camera devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is not limited to an image corresponding to a viewpoint freely specified by a user, but also includes an image corresponding to a viewpoint selected by the user from multiple candidates (virtual viewpoint candidates). Although in this embodiment, the description mainly focuses on the case where the virtual viewpoint is specified by user operation, the virtual viewpoint can also be specified automatically based on the results of image analysis, etc.

[0018] In this embodiment, the term "virtual camera" is used for description. The virtual camera is different from the multiple camera devices actually installed around the target camera area, but is used for convenience to describe the concept of a virtual viewpoint used to generate a virtual viewpoint image. Specifically, the virtual viewpoint image can be regarded as an image taken from a virtual viewpoint set in a virtual space associated with the target camera area. The position and posture of the virtual viewpoint in the camera can be represented as the position and posture of the virtual camera. In other words, if there is a camera at the position of the virtual viewpoint set in the space, the virtual viewpoint image is a simulated image of the image taken by the camera.

[0019] Configuration of Virtual Viewpoint Image Generation System

[0020] First, refer to Figure 1A and Figure 1B The configuration of the virtual visual point image generation system according to the present embodiment is described.

[0021] The virtual viewpoint image generation system according to the present embodiment includes n sensor systems 101a to 101n each including at least one camera (imaging device). Unless otherwise specified, the n sensor systems are hereinafter referred to as a plurality of sensor systems 101 without distinguishing between the sensor systems.

[0022] Figure 1A 1 is a diagram showing an example of the installation of multiple sensor systems 101. Multiple sensor systems 101 are installed around an area 120 as a target camera area to capture images of the area 120 from different directions. In the example of the present embodiment, the target area 120 is a stadium field where a football game is played, and n (e.g., 100) sensor systems 101 are installed around the stadium field. However, the number of installed sensor systems 101 is not limited, and the target area 120 is not limited to the stadium field. For example, the target area 120 may include a stadium auditorium, or may be an indoor studio or a stage.

[0023] The plurality of sensor systems 101 need not necessarily be installed around the entire perimeter of the area 120 , but may be installed only around a portion of the perimeter of the area 120 due to location constraints or other limitations.

[0024] The multiple cameras of the multiple sensor system 101 may include imaging devices having different functions, such as a telephoto camera and a wide-angle camera.

[0025] Multiple cameras of the multiple sensor systems 101 synchronously capture images. The multiple images captured by these cameras are called multi-view images. The multi-view images in this embodiment may be captured images, or images obtained by performing image processing (such as processing to extract a predetermined area from a captured image).

[0026] In addition to the camera, the multiple sensor system 101 may also include a microphone (not shown). The microphones of the multiple sensor systems 101 collect sound synchronously. Based on the collected sound, the multiple sensor systems 101 may generate an acoustic signal to be played back together with the display of the image on the image display device. For ease of explanation, the description about the sound will be omitted. The image and the sound are basically processed together.

[0027] The image processing system 1 includes an image processing device 20, a first information processing device 100, and a second information processing device 200. The image processing system 1 acquires multi-viewpoint images from a plurality of sensor systems 101, and stores the multi-viewpoint images together with a time code used for photographing in a database (not shown). The time code is information for uniquely identifying the time when the camera captures the image. The time code can be specified in a format such as "day: hour: minute: second: frame number". The image processing device 20 generates a virtual viewpoint image based on the stored multi-viewpoint images.

[0028] The first information processing apparatus 100 displays the virtual viewpoint image received from the image processing apparatus 20 on the first display device 30. The first information processing apparatus 100 receives a manual operation of the virtual camera 210 by the user using the first input device 10, and provides the operation to the image processing apparatus 20. The second information processing apparatus 200 displays the virtual viewpoint image received from the image processing apparatus 20 on the second display device 50. The second information processing apparatus 200 receives a manual operation of the virtual camera by the user using the second input device 40, and provides the operation to the image processing apparatus 20. Examples of the first information processing apparatus 100 and the second information processing apparatus 200 include a tablet terminal, a smart phone, a head mounted display (HMD), and other portable terminals.

[0029] The virtual camera 210 may be set at a viewpoint different from the viewpoints of all cameras of the plurality of sensor systems 101 in the virtual space associated with the area 120. Here, different virtual cameras are set for the virtual viewpoint image to be provided to the first information processing device 100 and the virtual viewpoint image to be provided to the second information processing device 200. The virtual viewpoint image generated by the image processing device 20 is an image representing a view from the virtual camera 210. The virtual viewpoint image in this embodiment is also referred to as a free viewpoint video.

[0030] In the present embodiment, the description focuses on the case where the virtual viewpoint image is a moving image, but the virtual viewpoint image may be a still image.

[0031] Configuration of image processing device and information processing device

[0032] Next, we will refer to Figure 2A and Figure 2B An example of the hardware configuration of the information processing apparatus and the image processing apparatus according to the present embodiment is described.

[0033] Figure 2A An example of the hardware configuration of the information processing apparatus is shown. Although the configuration of the first information processing apparatus 100 is described, the second information processing apparatus 200 also has the same configuration.

[0034] A central processing unit (CPU) 101 controls the following configuration via a system bus 112 by executing a program stored in a read only memory (ROM) 103 and / or a hard disk drive (HDD) 105 using a random access memory (RAM) 102 as a working memory. Thus, the following various processes can be performed.

[0035] An example of HDD interface (I / F) 104 is Serial ATA (SATA), which connects the information processing device to HDD 105, an optical drive, a solid state drive (SSD), a flash memory or other secondary storage device. In the present embodiment, HDD 105 is used as an example of a secondary storage device. CPU 101 can read and write data with HDD 105 via HDD interface (I / F) 104. CPU 101 loads the data stored in HDD 105 into RAM 102. CPU 101 is also configured to store various data on RAM 102 obtained by executing a program in HDD 105.

[0036] An input interface (I / F) 106 connects the information processing apparatus to one or more input devices 107 for inputting coordinates, such as a touch panel, a keyboard, a mouse, a digital camera, and a scanner. Examples of the input interface (I / F) 106 include serial bus interfaces, such as a universal serial bus (USB) and IEEE1394. The CPU 101 is configured to read data from the input device 107 via the input I / F 106. In the present embodiment, the input interface 106 is connected to the first input device 10.

[0037] The output interface (I / F) 108 connects the information processing apparatus to an output device 109 (such as a display). Examples of the output interface (I / F) 108 include video output interfaces such as a digital visual interface (DVI) and a high-definition multimedia interface (HDMI). The CPU 101 sends data related to the virtual viewpoint video to the output device 109 via the output I / F 108 to display the virtual viewpoint video. The network interface (I / F) 110 is a network card, such as a local area network (LAN) card, that connects the information processing apparatus to an external server 111. The CPU 101 is configured to read data from the external server 111 via the network I / F 110. In the present embodiment, the output interface 108 is connected to the first display device 30.

[0038] Figure 2B 2 shows an example of the hardware configuration of the image processing apparatus 20. A CPU 201 controls the configuration described below by executing a program stored in a ROM 203 using a RAM 202 as a working memory.

[0039] The communication unit 204 is connected to the information processing device to perform data communication. The communication unit 204 performs communication according to a communication standard such as Ethernet or IEEE802.11 (generally referred to as wireless LAN). The CPU 201 transmits and receives data with the information processing device or other external devices via the communication unit 204.

[0040] The input / output section 205 inputs / outputs data via an input interface or an output interface. The input / output section 205 is connected to a mouse, a keyboard, a display, a digital camera, or other devices.

[0041] The graphics processing unit (GPU) 206 is a computing unit or a processor dedicated to image processing. The GPU 206 performs image processing such as generating a virtual viewpoint image from images input from the plurality of sensor systems 101 .

[0042] Image processing system

[0043] Next, we will refer to Figure 3 and 4A to 4DAn example of the functional configuration of the image processing system 1 according to the present embodiment is described. The present embodiment describes a use case in which a user using the first display device 30 can also operate a virtual viewpoint image displayed on the second display device 50. The use case assumes a case in which a coach of a sports competition uses the first display device 30, and a player instructed by the coach uses the second display device 50. In this case, the coach can enable the player to operate the virtual viewpoint image to a certain extent to grasp his surroundings, while also reflecting the desired lens for guiding the player to the virtual viewpoint image on the player side. The present embodiment describes an image processing system that is particularly suitable for such guidance. However, it will be understood that the present invention is not necessarily limited to guidance and sports applications. The terms "coach" or "player" used in the following embodiments are examples and can be regarded as an illustration of a situation in which the present invention may be useful. With respect to the coach, it can be regarded as a person who is responsible for showing or controlling the viewing of a specific virtual viewpoint image to one or more viewers, and the coach can be regarded as a first user. With respect to the player, it can be regarded as a person who receives a virtual viewpoint image from a person who is responsible for showing or controlling the viewing of a specific virtual viewpoint image, and the player can be regarded as a second user.

[0044] The playback position determination section 301 determines the playback position of the virtual viewpoint image according to an instruction input from the first input device 10 of the first information processing apparatus 100. An example of the playback position is a time code.

[0045] The first viewpoint determination unit 302 determines the parameters of the virtual camera according to the instruction input from the first input device 10 of the first information processing apparatus 100. Examples of the parameters of the virtual camera include position and posture. The position of the virtual camera is represented by three-dimensional coordinates (x, y, z), for example. The posture of the virtual camera is represented by three directions (pan, pitch, and roll), for example.

[0046] The first video generation unit 303 generates a virtual viewpoint image based on the playback position determined by the playback position determination unit 301 and the position and posture of the virtual camera determined by the first viewpoint determination unit 302. The virtual viewpoint image generated by the first video generation unit 303 is displayed on the first display device 30 of the first information processing apparatus 100.

[0047] The second viewpoint determination unit 304 determines the position and posture of the virtual camera according to the instruction input from the second input device 40 of the second information processing apparatus 200. The position of the virtual camera is represented by three-dimensional coordinates (x, y, z), for example. The posture of the virtual camera is represented by three directions (pan, pitch, and roll), for example.

[0048] The second video generation unit 305 generates a virtual viewpoint image based on the playback position determined by the playback position determination unit 301, the position and posture of the virtual camera determined by the first viewpoint determination unit 302, and the position and posture of the virtual camera determined by the second viewpoint determination unit 304. Unlike the first video generation unit 303, the second video generation unit 305 generates a virtual viewpoint image using information on the positions and postures of the virtual cameras determined by the plurality of viewpoint determination units 302 and 304. The virtual viewpoint image generated by the second video generation unit 305 is displayed on the first display device 30 of the first information processing apparatus 100 and the second display device 50 of the second information processing apparatus 200.

[0049] Although in the present embodiment, the first video generating section 303 and the second video generating section 305 generate virtual viewpoint images using the playback position determined by the playback position determining section 301, a playback position determining section may be prepared for each of the generating sections 303 and 305. In other words, in the example of the present embodiment, two playback position determining sections may be prepared, and the first video generating section 303 and the second video generating section 305 may generate virtual viewpoint images individually using the playback positions determined by the individual playback position determining sections.

[0050] Next, we will refer to 4A to 4D The description is made of a virtual viewpoint image and a graphical user interface (GUI) displayed on the first display device 30 and the second display device 50. In the present embodiment, the first display device 30 is used by a coach, and the second display device 50 is used by a player. 4A to 4D A video of a basketball game is shown, in which the rectangular parallelepiped object is a simplified human figure, and the sphere is a basketball figure.

[0051] Figure 4A An example of a GUI displayed on the first display device 30 is shown. FIG. 4B to FIG. 4D An example of a GUI displayed on the second display device 50 is shown. The GUI displayed on the second display device 50 may be switched according to the configuration and type of the second information processing apparatus 200, or a mode selected from the second information processing apparatus 200, such as a window mode or a full screen mode.

[0052] The virtual viewpoint image 401 is displayed on both the first display device 30 and the second display device 50. The virtual viewpoint image 402 (i.e., another virtual viewpoint image) is displayed only on the first display device 30. In other words, the virtual viewpoint image 401 is viewed by both the players and the coach, while the virtual viewpoint image 402 is viewed only by the coach.

[0053] If the coach does not perform any operation, the position of the virtual camera for the virtual viewpoint image 401 is fixed regardless of the operation of the team members. This is because the position specified by the coach is regarded as the position of the viewpoint that the coach wants to show to the team members for guidance. However, the posture of the virtual camera for the virtual viewpoint image 401 can be changed by the operation of the team members. For example, in the case where the second input device 40 is a touch panel, the team members can change the posture of the virtual camera of the virtual viewpoint image 401 by sliding operation. In another example, in the case where the second input device 40 is a mouse, the team members can change the posture of the virtual camera for the virtual viewpoint image 401 by dragging operation. Therefore, the coach can enable the team members to operate the virtual viewpoint image to a certain extent, while also reflecting the desired lens for guiding the team members to the virtual viewpoint image on the team member side. The position of the virtual camera is not necessarily fixed, and can be adjusted within a predetermined range centered on the virtual camera position specified by the coach.

[0054] The virtual viewpoint image 402 is a virtual viewpoint image for the coach to prepare the virtual viewpoint image that the coach wants the players to see next. The coach can change the position and posture of the virtual camera for the virtual viewpoint image 402 using the first input device 10. For example, in the case where the first input device 10 is a touch panel, the coach can change the posture of the virtual camera for the virtual viewpoint image 402 through a sliding operation. In another example, in the case where the first input device 10 is a mouse, the coach can change the posture of the virtual camera for the virtual viewpoint image 402 through a dragging operation.

[0055] When the coach selects the viewpoint sharing button 408 using the first input device 10, the position and posture of the virtual camera for the virtual viewpoint image 402 are applied to the position and posture of the virtual camera for the virtual viewpoint image 401. This enables the coach to switch virtual viewpoint images for the players. When the coach selects the swap button 409 using the first input device 10, the position and posture of the virtual camera for the virtual viewpoint image 402 are swapped with the position and posture of the virtual camera for the virtual viewpoint image 401. This configuration enables the coach to switch between multiple virtual viewpoints while having the players view the virtual viewpoint images, for example, to provide guidance while including the perspective of the opposing players.

[0056] In the case where the viewpoint sharing button 408 is selected, the parameters of the virtual camera for the virtual viewpoint image 402 are not necessarily immediately applied to the virtual camera for the virtual viewpoint image 401. For example, after the instructor selects the viewpoint sharing button 408 and before the parameters of the virtual camera for the virtual viewpoint image 402 are applied to the virtual camera for the virtual viewpoint image 401, a message notifying the switching of the viewpoint may be displayed for a few seconds on the virtual viewpoint image 401. This configuration prevents the viewer from being confused by the rapid change in the position and posture of the virtual viewpoint of the virtual viewpoint image 401. This process also applies to the case where the exchange button 409 is selected.

[0057] When the viewpoint sharing button 408 is selected, a button for selecting whether to apply the parameters of the virtual camera for the virtual viewpoint image 402 may be superimposed on the virtual viewpoint image 401. When the application of the parameters of the virtual camera for the virtual viewpoint image 402 to the virtual viewpoint image 401 is accepted using the button displayed on the virtual viewpoint image 401, a process for sharing the viewpoint is performed. This makes it possible to prevent the virtual viewpoint from being switched at a timing that is not expected by the players. This process also applies to the case where the exchange button 409 is selected.

[0058] The first information processing device 100 can store the positions and postures of multiple virtual cameras specified by the coach, and determine the virtual viewpoint from the stored list. This enables the coach to change the viewpoint to the desired viewpoint without having to perform the virtual camera viewpoint operation every time. For example, by storing the position and posture of the virtual camera at a striking position (such as under the basket of a basketball court), the coach can switch to the virtual viewpoint image of the desired viewpoint simply by selecting from the list as needed. Providing such an operation method simplifies the user's operation.

[0059] The image processing system 1 can store a series of virtual camera parameters as a path of a virtual camera (camera path). However, this is only exemplary. The image processing system 1 can specify multiple camera parameters in key frames, automatically interpolate the camera parameters between key frames to create a camera path, and store the camera path. The image processing system 1 reproduces the path of the virtual camera based on the camera path to generate a virtual viewpoint image 401. This enables the coach to set the viewpoint for the players across multiple frames.

[0060] The progress bar 403 is a GUI indicating the playback position (time code) of the virtual viewpoint image 402. The coach can select any playback position by selecting a position on the progress bar 403. The reverse playback button 404 is used to reversely playback the virtual viewpoint image 402. The pause / resume button 405 is used to pause or resume the playback of the virtual viewpoint image 402. The forward playback button 406 is used to forwardly playback the virtual viewpoint image 402. The playback speed 407 is an item for changing the playback speed of the virtual viewpoint image 402. The coach can change the playback speed of the virtual viewpoint image 402 by directly inputting or by selecting from the options presented in the drop-down menu. For example, if 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played back at normal speed. If a value less than 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played back slowly. If a value greater than 1 is selected for the playback speed 407, the virtual viewpoint image 402 is played back at high speed.

[0061] Figure 4B An example of a GUI displayed on the second display device 50 is shown. The GUI is displayed on the second information processing apparatus 200, which is, for example, a personal computer (PC) or a tablet terminal.

[0062] The progress bar 413 is a GUI indicating the playback position (time code) of the virtual viewpoint image 401. A player can select an arbitrary playback position by selecting a position on the progress bar 413. The reverse playback button 414 is used to reversely playback the virtual viewpoint image 401. The pause / resume button 415 is used to pause or resume the playback of the virtual viewpoint image 401. The forward playback button 416 is used to forwardly playback the virtual viewpoint image 401. The playback speed 417 is an item for changing the playback speed of the virtual viewpoint image 401. The player can change the playback speed of the virtual viewpoint image 401 by directly inputting or by selecting from the options presented in the drop-down menu. For example, if 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played back at normal speed. If a value less than 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played back slowly. If a value greater than 1 is selected for the playback speed 417, the virtual viewpoint image 401 is played back at high speed.

[0063] exist Figure 4B In the example, the time width of the playback position that can be changed using the progress bar 413 can be smaller than the time width of the playback position that can be changed using the progress bar 403. This enables the coach to prevent the shot (playback position) used to instruct the team members from being changed to a different shot (playback position) by the team members.

[0064] Figure 4C1 shows another example of a GUI displayed on the second display device 50. The GUI is displayed on the second information processing apparatus 200, which is a smartphone, a tablet terminal, or other portable device. Figure 4B The GUI shown is different. Figure 4C No GUI for controlling the virtual viewpoint image 401 is displayed. In this case, the player can change the posture of the virtual camera for the virtual viewpoint image 401 using the touch panel or acceleration sensor of the second information processing device 200, but cannot perform other operations. The coach can Figure 4B The GUI in Figure 4C An instruction to switch between GUIs in the information processing apparatus 200 is given to display a GUI suitable for the guidance purpose on the second information processing apparatus 200.

[0065] Figure 4D Another example of a GUI displayed on the second display device 50 is shown. The GUI is displayed on the second information processing apparatus 200, which is a wearable device such as an HMD. Figure 4B or Figure 4C The GUI shown in FIG. 1 is changed to a full screen mode and displayed on the second information processing device 200. Figure 4D In the GUI. Figure 4D In the example, the virtual viewpoint image 401 is displayed in full screen without displaying the operation GUI, etc.

[0066] Displayed on the second display device 50 Figure 4C or Figure 4D In the case of the GUI shown, the playback position of the virtual viewpoint image 401 is changed using the progress bar 403 displayed on the first display device 30. In this case, the playback state of the virtual viewpoint image 401 is controlled using the reverse playback button 404, the pause / resume button 405, the forward playback button 406, and the playback speed 407.

[0067] In the present embodiment, only the posture of the virtual camera for the virtual viewpoint image 401 may be changed without being operated from the first input device 10. This is because, for example, if a player views the virtual viewpoint image 401 through the HMD and the coach only changes the posture of the virtual camera, the player may not be able to track where he is looking, resulting in a situation where the player cannot concentrate on viewing the virtual viewpoint image.

[0068] Any GUI ( FIG. 4B to FIG. 4D ). This enables the coach to control the content displayed on the second information processing device 200 used by the players according to the details of the instruction.

[0069] In this embodiment, the virtual viewpoint image 401 and the virtual viewpoint image 402 are played back in a loop, so that the coach or the players can watch the same shot repeatedly.

[0070] When the playback speed 407 is changed while the virtual viewpoint image 402 is paused, the virtual viewpoint image 402 can automatically resume playback. This eliminates the need for the user to press a playback button. Similarly, when the playback speed 417 is changed while the virtual viewpoint image 401 is paused, the virtual viewpoint image 401 can automatically resume playback.

[0071] Processing by the image processing device 20

[0072] Will refer to Figure 5 The flowchart in 200 describes an example of processing performed by the image processing device 20. The processing of the image processing device 20 is realized by developing software recorded in the ROM 203 to the RAM 102 and causing the CPU 201 or the GPU 206 to execute the software.

[0073] In S501, the CPU 201 determines the playback position of the first virtual viewpoint video based on the first input value input from the first information processing apparatus 100. An example of the first virtual viewpoint video is the virtual viewpoint image 402. An example of the first input value to be processed in this step is a control value corresponding to an operation input to the progress bar 403 using the first input device 10. Specifically, the control value is a time code. If the first input value is not an input value related to the playback position, this step is not performed, and the next step is performed.

[0074] In S502, the CPU 201 determines the first virtual viewpoint of the first virtual viewpoint video based on the first input value input from the first information processing apparatus 100. The first virtual viewpoint is, for example, the position and posture of a virtual camera for the virtual viewpoint image 402. An example of the first input value to be processed in this step is a control value corresponding to an operation for changing the position or posture of the virtual camera for the virtual viewpoint image 402, which is input to the first information processing apparatus 100 using the first input device 10. If the exchange button 409 is selected, the same parameters as those of the second virtual viewpoint are set as the parameters of the first virtual viewpoint. If the first input value is not related to the first virtual viewpoint, the processing of this step is not performed.

[0075] In S503, the CPU 201 determines the playback position of the second virtual viewpoint video based on the second input value input from the second information processing apparatus 200. An example of the second virtual viewpoint video is the virtual viewpoint image 401. An example of the second input value to be processed in this step is a control value corresponding to an operation input to the progress bar 413 through the second input device 40. Specifically, the control value is a time code. If the second information processing apparatus 200 does not have a user interface (UI) for controlling the playback position as the progress bar 413, the CPU 201 determines the playback position of the second virtual viewpoint video as the same playback position as the playback position determined in S501. If the second input value is not an input value related to the playback position, the processing of this step is not performed, and the processing of the next step is performed.

[0076] In S504, the CPU 201 determines a second virtual viewpoint for the second virtual viewpoint video based on a first input value input from the first information processing apparatus 100 and a second input value input from the second information processing apparatus 200. An example of the second virtual viewpoint is the position and posture of a virtual camera for the virtual viewpoint image 401. An example of the first input value to be processed in this step is a control value corresponding to an operation for changing the position of the virtual camera for the virtual viewpoint image 401, which is input to the first information processing apparatus 100 by the first input device 10. An example of the second input value to be processed in this step is a control value corresponding to an operation for changing the posture of the virtual camera for the virtual viewpoint image 401, which is input to the second information processing apparatus 200 by the second input device 40.

[0077] If the exchange button 409 is selected, parameters similar to those of the first virtual viewpoint immediately before the processing of S502 are set as parameters of the second virtual viewpoint. If the second input value is not an input value related to the second virtual viewpoint, the processing of this step is not performed. Although in this step, the position of the virtual camera is determined based on the input value from the first input device 10, this is only exemplary. The position can be determined based on the input value on the second input device 40. In this case, the first input value is ignored, and the second virtual viewpoint for the second virtual viewpoint video is determined based only on the second input value.

[0078] In S505 , the CPU 201 generates a first virtual viewpoint image using the GPU 206 based on the playback position determined in S501 and the first virtual viewpoint determined in S502 .

[0079] In S506, the CPU 201 determines whether a trigger has been detected from the first input value. The trigger is used to reflect the virtual viewpoint for the first virtual viewpoint image to the second virtual viewpoint image. For example, if Figure 4A, the CPU 201 detects a trigger. If a trigger is detected from the first input value, the process of S507 is executed. If a trigger is not detected from the first input value, the process of S508 is executed.

[0080] In S507, the CPU 201 generates a second virtual viewpoint image based on the playback position determined in S501 and the first virtual viewpoint determined in S502 using the GPU 206. In S508, the CPU 201 generates a second virtual viewpoint image based on the playback position determined in S503 and the second virtual viewpoint determined in S504 using the GPU 206.

[0081] In S509 , the CPU 201 outputs the first virtual viewpoint image and the second virtual viewpoint image. For example, the CPU 201 outputs the first virtual viewpoint image to the first information processing apparatus 100 via the input / output section 205 , and outputs the second virtual viewpoint image to the second information processing apparatus 200 .

[0082] In S510, the CPU 201 determines whether to terminate the process of the flowchart. For example, when the application for playing back the virtual viewpoint image is terminated, the CPU 201 determines to terminate the process of the flowchart. If the process of the flowchart is not terminated, the process of S511 is executed.

[0083] In S511, CPU 201 changes the playback position. For example, for forward playback, CPU 201 moves the playback position forward by one frame. For reverse playback, CPU 201 returns the playback position by one frame. After completing this process, the process of this flowchart is repeated from S501.

[0084] Referenced Figure 5 The processing performed by the image processing device 20 is described. The processing facilitates coordination of multiple virtual viewpoint images. For example, a coach can enable a player to manipulate a virtual viewpoint image to a certain extent to grasp his surroundings, while also reflecting the desired lens for guiding the player to the virtual viewpoint image on the player side.

[0085] The CPU 201 performs playback processing such as playback, pause, and change of playback speed of virtual viewpoint images as necessary through interrupt processing during execution of the processing of the flowchart.

[0086] Although in the present embodiment, the image processing device 20 generates both the first virtual viewpoint image (virtual viewpoint image 402) and the second virtual viewpoint image (virtual viewpoint image 401), this is exemplary only. For example, two image processing devices 20 may be provided, which generate the first virtual viewpoint image or the second virtual viewpoint image separately. In this case, the two image processing devices 20 communicate with each other to send and receive parameters of the virtual camera for the corresponding virtual viewpoint image and the playback position of the corresponding virtual viewpoint image.

[0087] Modification of the first embodiment

[0088] This embodiment describes the display and control of the virtual viewpoint image in the first information processing device 100 and the processing of the image processing device 20. Figure 1A and Figure 1B Virtual viewpoint image generation system in Figure 2A and Figure 2B The same configurations, processes, and functions as those of the first information processing apparatus 100 and the image processing apparatus 20 of the first embodiment are given the same reference numerals, and redundant descriptions thereof will be omitted.

[0089] Figure 6 An example of a GUI displayed on the first display device 30 is shown. A virtual viewpoint image 601 is added to Figure 4A in the GUI.

[0090] The virtual viewpoint of the virtual viewpoint image 601 is set to look down at the sports field (basketball court). Here, the black rectangular block represents the players in action. Displaying the virtual viewpoint image 601 allows the coach to easily grasp the positional relationship between the players during the game. In this embodiment, the virtual viewpoint for the virtual viewpoint image 601 is fixed and does not change.

[0091] In addition, pictograms and icons may be superimposed on the virtual viewpoint image 601. Icon 602 indicates the position and posture of the virtual camera for the virtual viewpoint image 401. Icon 603 indicates the position and posture of the virtual camera for the virtual viewpoint image 402. Icon 602 and icon 603 may be set to different colors or patterns to distinguish each other. Superimposing icons 602 and 603 on the virtual viewpoint image 601 enables the coach to confirm the position and posture of the virtual viewpoints of the coach and the players.

[0092] When changing the position of the virtual camera for the virtual viewpoint image 402, the coach can change the position by selecting (e.g., left-clicking) a position on the virtual viewpoint image 601. This facilitates instant movement of the virtual camera, and is also convenient for, for example, moving the position of the virtual camera from one end of the sports field to the other end. The coach can also set the gaze point 604 by selecting (e.g., right-clicking) a position on the virtual viewpoint image 601. When the coach sets the gaze point 604 and sets another position of the virtual camera using a method different from the method for setting the gaze point 604 (left-clicking), the line of sight of the virtual camera rotates in the horizontal lateral direction (panning direction) to face the gaze point 604. For example, when the gaze point 604 is set below the basket and the position of the virtual camera is set at any position on the sideline in the field, the line of sight of the virtual camera on the sideline is automatically changed to below the basket. As the position of the gaze point 604 is changed, the current line of sight of the virtual camera can be automatically rotated to face the gaze point 604. Although in this example, the posture (line of sight) of the virtual camera is rotated, the line of sight of the virtual camera can be set to face the gaze point 604 while determining the position of the virtual camera.

[0093] Therefore, the target position is pre-set so that the user can immediately view the target position by only setting the position of the virtual camera to an arbitrary position. In the case where the first input device 10 is a touch panel, the coach can select the position by clicking and select the gaze point by double-clicking.

[0094] In this case, Figure 5 In the process of S502 of the flowchart in FIG. 1 , the CPU 201 performs the following process.

[0095] In S502, the CPU 201 obtains the coordinates selected in the virtual viewpoint image 601 as the first input value. The coordinates obtained by the CPU 201 are two-dimensional coordinates on a plane horizontal to the ground (xy plane), and the coordinates in a direction perpendicular to the ground (z-axis direction) are not obtained. The CPU 201 applies the obtained two-dimensional coordinates to the coordinates of the current first virtual viewpoint. In other words, the first virtual viewpoint image is translated, and the height and posture of the viewpoint remain unchanged.

[0096] In S502, the CPU 201 also obtains the coordinates of the gaze point 604 selected in the virtual viewpoint image 601 as the first input value. The coordinates obtained by the CPU 201 are two-dimensional coordinates on a plane (xy plane) horizontal to the ground, and the coordinates in the direction perpendicular to the ground (z-axis direction) are not obtained. The CPU 201 sets the first virtual viewpoint to face the obtained two-dimensional coordinates. In other words, in the first virtual viewpoint image, the first virtual viewpoint rotates in the pan direction so as to face the gaze point 604, and the pitch and roll of the posture remain unchanged. The CPU 201 can not only rotate the posture of the first virtual viewpoint in the pan direction, but also rotate the posture of the first virtual viewpoint in the pitch direction so that the posture of the first virtual viewpoint is parallel to the ground.

[0097] This makes it easier for the coach to provide instructions to the players.

[0098] Second embodiment

[0099] The first embodiment describes a process in which, based on a predetermined trigger, the position and posture of the virtual camera for the virtual viewpoint image 402 are reflected to the position and posture of the virtual camera for the virtual viewpoint image 401. In the second embodiment, based on a predetermined trigger, some parameters of the virtual camera for the virtual viewpoint image 402 are reflected to the virtual viewpoint image 401, and some parameters of the virtual camera for the virtual viewpoint image 401 are reflected to the virtual camera for the virtual viewpoint image 402.

[0100] The same reference numerals are used for the same configuration, processing, and function as those of the first embodiment, and redundant descriptions thereof will be omitted.

[0101] Figure 7 70 is a diagram showing an example of a functional configuration of the image processing system 1 according to the present embodiment. The image processing apparatus 70 includes a playback position determination section 301 , a first viewpoint determination section 702 , a first video generation section 303 , a second viewpoint determination section 704 , and a second video generation section 305 .

[0102] The first viewpoint determination unit 702 determines the parameters of the virtual camera based on the input value from the first input device 10 of the first information processing apparatus 100 and some parameters of the position and posture of the virtual camera from the second viewpoint determination unit 704. For example, the first viewpoint determination unit 702 calculates the camera parameters based on the input value from the first input device 10, and then applies the height of the viewpoint of the virtual camera obtained from the second viewpoint determination unit 704 to the calculated camera parameters. This application processing may be performed continuously, or may be performed only when a trigger (such as pressing the viewpoint sharing button 408) is detected from the input value from the first input device 10. For this application processing, this embodiment will be described later with reference to Figure 8A case where a trigger is detected from an input value from the first input device 10 is described.

[0103] The first viewpoint determination unit 702 outputs the camera parameters other than the parameters (the height of the viewpoint) applied from the second viewpoint determination unit 704 to the second video generation unit 305. The second video generation unit 305 obtains the camera parameters other than the camera parameters obtained from the first viewpoint determination unit 702 from the second viewpoint determination unit 704 to generate a virtual viewpoint image. In this way, even when the HMD is used as a second display device 50 for viewing life-size players in a basketball court for virtual reality (VR) viewing, the virtual viewpoint can be prevented from being located in an unexpected position such as a high place or the ground. This processing can also maintain the viewer's line of sight. The camera parameters obtained by the first viewpoint determination unit 702 from the second viewpoint determination unit 704 can be not only the viewpoint height, but also posture information (such as pitch).

[0104] When a trigger is detected from the input value from the first input device 10, the first viewpoint determination unit 702 may reset some parameters of the virtual camera to initial values. For example, the first viewpoint determination unit 702 resets the pitch and roll of the virtual camera to be parallel to the ground, and outputs the parameters of the virtual camera other than the reset parameters to the second video generation unit 305. This prevents the pitch and roll of the virtual camera for the virtual viewpoint image displayed on the HMD used as the second display device 50 from being affected by the posture of the virtual viewpoint of the instructor. This prevents the mismatch between the head posture of the VR viewer and the posture of the virtual camera, thereby reducing VR motion sickness caused by the mismatch of the viewer's sense of balance, etc.

[0105] In addition, the first viewpoint determination unit 702 can limit changes to some parameters of the virtual camera. For example, the first viewpoint determination unit 702 can prohibit changing the roll of the virtual viewpoint while keeping it parallel to the ground. In this case, the first viewpoint determination unit 702 outputs the parameters of the virtual camera other than the defined parameters to the second video generation unit 305. This reduces the misalignment between the image and the head posture of the VR viewer. The defined parameters can include not only roll but also pitch. A user interface (UI) can be displayed on the first display device 30 so that the coach can select the parameters to be limited.

[0106] The second viewpoint determination section 704 has a configuration similar to that of the first embodiment. In addition, as described above, the second viewpoint determination section 704 according to the present embodiment determines parameters of the virtual camera based on the input value from the second input device 40 of the second information processing apparatus 200 and outputs these parameters to the first viewpoint determination section 702.

[0107] Figure 8401 and 402 are flowcharts showing a process of reflecting parameters of virtual cameras for the virtual viewpoint image 401 and the virtual viewpoint image 402 to each other based on a predetermined trigger according to the present embodiment. Figure 8 The flowchart in FIG. 1 is the same as that according to the first embodiment. Figure 5 The flowchart in FIG. 8 is different in that the process in S505 is changed to S802, and the process in S801 is added. The other processes are the same as those of the first embodiment.

[0108] In S801 , the CPU 201 changes the parameters of the virtual camera for the first virtual viewpoint image determined by the first viewpoint determination section 702 based on some of the parameters of the position and posture of the virtual camera determined by the second viewpoint determination section 704 .

[0109] In S802, the CPU 201 generates a first virtual viewpoint image using the GPU 206 based on the playback position determined in S501 and the first virtual viewpoint determined in S502. The processing of this step is the same as that of the first embodiment. Figure 5 The processing of S505 in is the same.

[0110] Thus, the process of mutually reflecting the parameters of the virtual cameras for the virtual viewpoint image 401 and the virtual viewpoint image 402 based on a predetermined trigger according to the present embodiment has been described. This process prevents the coach's unintended viewpoint from being reflected to the players, and reduces VR motion sickness caused by the mismatch between the player's head position and posture and the position and posture of the virtual camera when the players watch VR content.

[0111] Other embodiments

[0112] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0113] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: an acquisition unit, arranged to obtain a plurality of images captured by a plurality of camera devices; a first viewpoint determination unit arranged to generate camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determination unit arranged to determine a playback position of a virtual viewpoint image to be played back based on an input value from the first input device; a first video generating unit arranged to generate a first virtual viewpoint video based on the plurality of images, camera parameters of the first virtual camera and the playback position; a second viewpoint determination unit arranged to generate camera parameters of a second virtual camera based on input values ​​from the first input device and input values ​​from a second input device; as well as A second video generating unit is arranged to generate a second virtual viewpoint video based on the plurality of images, camera parameters of the second virtual camera and the playback position.

2. The image processing device according to claim 1, in, The first viewpoint determination unit determines the position and posture of the first virtual camera based on an input value from the first input device, and The second viewpoint determination unit determines the position of the second virtual camera based on the input value from the first input device, and determines the posture of the second virtual camera based on the input value from the second input device.

3. The image processing device according to claim 1 or 2, further comprising: An output unit is arranged to output the first virtual viewpoint video and the second virtual viewpoint video to the first input device, and output the second virtual viewpoint video to the second input device.

4. The image processing device according to claim 3, wherein: The output unit changes the camera parameters of the second virtual camera to the camera parameters of the first virtual camera based on the instruction from the first input device.

5. The image processing device according to claim 3, wherein: The output unit exchanges the camera parameters of the first virtual camera and the camera parameters of the second virtual camera based on the instruction from the first input device.

6. The image processing device according to claim 3, wherein: The output unit changes the playback speed of the first virtual viewpoint video and the second virtual viewpoint video based on the instruction from the first input device.

7. The image processing apparatus according to claim 3, further comprising: a third video generating unit arranged to generate a third virtual viewpoint video according to the plurality of images and the playback position, the third virtual viewpoint video providing a bird's-eye view of a space captured according to the plurality of images, Wherein, the third virtual viewpoint video is output to the first input device.

8. The image processing device according to claim 7, wherein: The third video generation unit is arranged to generate the third virtual viewpoint video based on camera parameters of the first virtual camera and camera parameters of the second virtual camera to indicate a position of the first virtual camera and a position of the second virtual camera.

9. The image processing device according to claim 1 or 2, in, The first viewpoint determination unit determines the position and posture of the first virtual camera based on an input value from the first input device, and The second viewpoint determination unit determines the height of the posture and position of the second virtual camera based on the input value from the second input device, and determines other positions of the second virtual camera based on the input value from the first input device.

10. The image processing device according to claim 1 or 2, in, The first input device includes a tablet terminal, and Wherein, the second input device includes a head-mounted display.

11. An image processing method for generating a virtual viewpoint video based on a plurality of images captured by a plurality of camera devices, the image processing method comprising: An acquisition step for obtaining a plurality of images captured by the plurality of camera devices; A first viewpoint determination step for generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determining step, for determining a playback position of the virtual viewpoint image to be played back based on an input value from the first input device; A first video generating step, for generating a first virtual viewpoint video based on the plurality of images, the camera parameters of the first virtual camera and the playback position; A second viewpoint determination step for generating camera parameters of a second virtual camera based on an input value from the first input device and an input value from a second input device; as well as The second video generating step is used to generate a second virtual viewpoint video based on the multiple images, the camera parameters of the second virtual camera and the playback position.

12. A system comprising the image processing apparatus according to any one of claims 1 to 10, a first input device for receiving the first virtual viewpoint video, and a second input device for receiving the second virtual viewpoint video.

13. A computer program product comprising instructions for causing an image processing apparatus to perform the following steps: A first viewpoint determination step for generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determining step, for determining a playback position of the virtual viewpoint image to be played back based on an input value from the first input device; A first video generating step, for generating a first virtual viewpoint video based on a plurality of images, camera parameters of the first virtual camera and the playback position; A second viewpoint determination step for generating camera parameters of a second virtual camera based on an input value from the first input device and an input value from a second input device; as well as The second video generating step is used to generate a second virtual viewpoint video based on the multiple images, the camera parameters of the second virtual camera and the playback position.

14. A computer-readable storage medium having stored thereon instructions for causing an image processing apparatus to perform the following steps: A first viewpoint determination step for generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determining step, for determining a playback position of the virtual viewpoint image to be played back based on an input value from the first input device; A first video generating step, for generating a first virtual viewpoint video based on a plurality of images, camera parameters of the first virtual camera and the playback position; A second viewpoint determination step for generating camera parameters of a second virtual camera based on an input value from the first input device and an input value from a second input device; as well as The second video generating step is used to generate a second virtual viewpoint video based on the multiple images, the camera parameters of the second virtual camera and the playback position.

15. A computer-implemented method comprising: A first viewpoint determination step for generating camera parameters of a first virtual camera based on input values ​​from a first input device; a playback position determining step, for determining a playback position of the virtual viewpoint image to be played back based on an input value from the first input device; A first video generating step, for generating a first virtual viewpoint video based on a plurality of images, camera parameters of the first virtual camera and the playback position; A second viewpoint determination step for generating camera parameters of a second virtual camera based on an input value from the first input device and an input value from a second input device; as well as The second video generating step is used to generate a second virtual viewpoint video based on the multiple images, the camera parameters of the second virtual camera and the playback position.

Images