Processing system, its control method, and program

Abstract

To provide a technique for supporting an object (performer) when capturing images used for generating virtual viewpoint images.SOLUTION: A processing system for generating a virtual viewpoint image representing a view from a virtual viewpoint is provided, the system being configured to present an indication for determining the orientation of an object in a space including the object to be shot by multiple image capturing means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a processing system, a control method thereof, and a program.

Background Art

[0002] Recently, a technique of performing synchronized shooting with a plurality of cameras installed at different positions and generating virtual viewpoint content using a plurality of viewpoint images obtained by the shooting has attracted attention. According to such a technique, for example, highlight scenes of soccer or basketball can be viewed from various angles, so that a higher sense of presence can be given to the user as compared with ordinary images.

[0003] Patent Document 1 discloses a technique of arranging a plurality of cameras so as to surround a subject and generating an image of an arbitrary viewpoint using images of the subject taken by these plurality of cameras.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Conventionally, there is a so-called tally (tally lamp, tally light). A tally is a red lamp attached to a television camera or the like, which lights up when the device is in use and is usually interlocked with a switch or the like. Since the performer (subject) can know that he / she is being photographed at present, it can be used as a trigger for a line or an action.

[0006] However, in the system for generating the virtual viewpoint content described above, there is no actual camera (real camera) at the position of the virtual viewpoint. Therefore, for example, when shooting in a studio, the performer does not know where to direct their line of sight, resulting in limitations in terms of production.

[0007] The present disclosure has been made in view of the above problems, and aims to provide a technique for assisting a subject (performer) in shooting an image used for generating a virtual viewpoint image.

Means for Solving the Problem

[0008] To solve this problem, for example, the processing system of the present disclosure has the following configuration. That is, A processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, A plurality of imaging means for imaging a space including a subject, Specific means for specifying the position of the virtual viewpoint, Based on the position of the virtual viewpoint, Presentation means for performing a presentation for determining the orientation of the subject in the space.

Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a technique for assisting a subject (performer) in shooting an image used for generating a virtual viewpoint image.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

BEST MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] [First Embodiment] A system for installing a plurality of cameras in facilities such as stadiums, concert halls, and studios and performing shooting will be described with reference to FIG. 1.

[0013] FIG. 1 is an overall configuration diagram of an image processing system 100 to which a first embodiment is applied. This system 100 includes a photographing device 1, a real space information holding device 2, an image processing device 3, a user terminal 4, and a marker output device 5, which are assumed to be connected via a network. Note that the network may be wired or wireless.

[0014] In the above configuration, the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 can be realized by an information processing device typified by a personal computer (PC) and an application program executed on the information processing device. FIG. 9 is a hardware block configuration diagram of the information processing device 200. The information processing device 200 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display device 215, an operation unit 216, and a communication interface (I / F) 217, and these are structured to be connected to a system bus 218. When the power of this device is turned on, the CPU 211 executes the boot program in the ROM 212, loads the OS (operating system) from the auxiliary storage device 214 (for example, a hard disk) into the RAM 213, and executes the OS. As a result, the information processing device 200 can input various instructions from the user via the operation unit 26 and execute corresponding processing. Further, the CPU 211 can execute an application program pre-stored in the auxiliary storage device 214 under the control of the OS. For example, depending on the type of the application program, this information processing device 200 functions as the real space information holding device 2, the image processing device 3, the user terminal 4, or the marker output device 5. For example, when the information processing device 200 functions as the image processing device 3, the communication I / F 217 is communicably connected to a network (for communicating with the user terminal 4 and the marker output device 5), as well as the imaging device 1 and the real space information holding device 2. Also, when the information processing device 200 functions as the marker output device 5, the communication I / F 217 is connected to a network (for communicating with the user terminal 4 and the marker output device) and a projector. When the information processing device 200 functions as the real space information holding device 2, the auxiliary storage device 214 may be made to function as a file server after storing real space information (described in detail later).

[0015] The imaging device 1 is composed of a plurality of cameras installed so as to surround a competition field, a studio, etc., and they transmit the images obtained by synchronous imaging with each other to the image processing device 3.

[0016] The real space information holding device 2 holds information regarding the space within a predetermined range including the subject (performer). Specifically, it includes 3D model information of objects (background objects) that appear as the background in the virtual viewpoint image, such as the field and spectator seats in an arena, or the facilities in a studio, 3D space information indicating the range where the virtual viewpoint can be set, and the installation positions, shooting directions, focal lengths, etc. of each imaging device 1. Since the information held in this real space information holding device 2 is referred to when the image processing device 3 generates a virtual viewpoint image, it may be provided within the image processing device 3.

[0017] The image processing device 3 includes a virtual viewpoint image generation unit 301, a virtual camera path calculation unit 302, a virtual camera path information holding unit 303, a marker coordinate calculation unit 304, a display image generation unit 305, and a communication unit 306.

[0018] The virtual viewpoint image generation unit 301 generates a 3D model of the foreground object based on the images of a plurality of viewpoints acquired from the imaging device 1. Then, the virtual viewpoint image generation unit 301 maps the texture corresponding to the virtual viewpoint acquired from the virtual camera path calculation unit 302 to the generated foreground 3D model and the background 3D model obtained from the real space information holding device 2, and performs rendering to generate a virtual viewpoint image. In this generation process, the virtual viewpoint image generation unit 301 calculates the coordinates of the foreground object and the background object that will appear in the virtual viewpoint image to be generated, and executes texture mapping and rendering only for those coordinates. The virtual viewpoint image generation unit 301 passes these coordinate values to the virtual camera path information holding unit 303 described later as the subject foreground coordinates and the real space background coordinates.

[0019] The virtual camera path calculation unit 302 calculates temporally continuous virtual camera path parameters based on the user's instruction content for the virtual camera path instruction unit 403 of the user terminal 4. The virtual camera parameters are at least the position, orientation (gaze direction), and field of view angle (focal length) of the virtual camera, and are associated with the frame number or time code attached to the multi-viewpoint images so that the parameters at any moment in the shooting scene can be specified. When performing this calculation, the virtual camera path is set within the range where the virtual viewpoint can be set with reference to the real space information obtained from the real space information holding unit 2.

[0020] The virtual camera information holding unit 303 accumulates the subject foreground coordinates and real space background coordinates received from the virtual viewpoint image generation unit 301, and the virtual camera path parameters calculated by the virtual camera path calculation unit 302.

[0021] The marker coordinate calculation unit 304 generates information about the coordinate parameters of the marker information to be displayed in the real space based on the virtual camera parameters related to the virtual camera accumulated in the virtual camera information holding unit 303 and the three-dimensional space information (background information) held by the real space information holding unit 2.

[0022] The display image generation unit 305 generates a display image to be displayed on the image display unit 402 of the user terminal 4. The display image generated here is the virtual viewpoint image generated by the virtual viewpoint image generation unit 301. The virtual viewpoint image is an image representing the view from the virtual viewpoint.

[0023] The communication unit 306 communicates between the image processing apparatus 3, the user terminal 4, and the marker output apparatus 5 via a network (not shown). The communication unit 306 transmits and receives images, audio, text data, and instruction information such as virtual camera path instructions sent from the user side when generating the virtual viewpoint image between the image processing apparatus 3 and the user terminal 4. The communication unit 306 transmits and receives the coordinate information necessary for marker display, display parameters such as the shape, color, size, and display information of the marker between the image processing apparatus 3 and the marker output apparatus 5.

[0024] The user terminal 4 includes a communication unit 401, an image display unit 402, a virtual camera path instruction unit 403, a user information transmission unit 404, and a marker information transmission unit 405.

[0025] The communication unit 401 performs transmission and reception of various information with the device communication unit 306 of the aforementioned device as described above. The image display unit 402 receives and displays the image generated by the display image generation unit 305 described above. The virtual camera path instruction unit 403 receives an instruction from the user regarding the virtual camera path and passes it to the virtual camera path calculation unit 302 via the communication unit 401 and the communication unit 306. Here, the user does not necessarily need to strictly instruct all of the virtual camera parameters for the entire time they want to view. For example, it is also possible to input instructions based on various viewpoints such as wanting to view a virtual viewpoint image focused on a specific player or performer, always wanting to view a certain range around the ball, or wanting to view the location where a more notable event is occurring.

[0026] The user information transmission unit 404 attaches user information including the user ID to the information transmitted from the communication unit 401 towards the communication unit 306. The marker information transmission unit 405 attaches marker type parameter information such as the type of marker (including shape and color) to the information transmitted from the communication unit 401 towards the communication unit 306.

[0027] The marker output device 5 includes a communication unit 501, a control data generation unit 502, a marker control unit 503, and a display data output unit 504. The marker output device 5 displays (or presents) a marker that can be recognized by the person who is the subject. Note that the marker to be displayed is output at a position related to the virtual camera parameters. That is, in conjunction with the virtual camera parameters, the position of the marker is controlled by the marker output device 5.

[0028] As described above, the communication unit 501 communicates with the marker coordinate calculation unit 304 and the user terminal 4 via the device communication unit 306 of the image processing device 3 and a network or the like.

[0029] The control data generation unit 502 integrates the data output from the virtual viewpoint image generation unit 301 and the marker coordinate calculation unit 304 with the data output from the virtual camera path instruction unit 403 and the marker information transmission unit 405, generates control data such as the type of marker, display update time, and position coordinates, and outputs the control data to the marker control unit 503.

[0030] The marker control unit 503 converts the data into display data (screen coordinate system) based on display parameters such as the display position, size, shape, and display content of the marker, and outputs the display data to the display data output unit 504.

[0031] The display data output unit 504 is composed of a display device such as a projector, and outputs display data based on the output data of the marker control unit 503.

[0032] FIG. 2(a) is a schematic diagram of the real space surrounding the subject in the image processing system according to the present embodiment. The image processing system according to the present embodiment has a plurality of cameras (imaging devices 1) so as to surround a predetermined range of real space including the subject. Further, in order to display a marker recognizable by the subject (human) at a position based on the virtual camera path information, it has a projector (display data output unit 504).

[0033] The illustrated image processing system is an example when this configuration is implemented in a circular studio, and cameras 21a to 21f, projectors 22a to 22f, and screens 23a to 23f are arranged to surround the subject 24. The image processing device 3 generates a virtual viewpoint image from a plurality of viewpoint images acquired by the cameras 21a to 21f, and calculates the coordinates on the screen on the extension line connecting the subject 24 and the virtual viewpoint as marker coordinates. Then, the marker output device 5 receives the marker coordinates and displays a marker on the corresponding screen. Specifically, the display of the marker is projected onto the screen by the projectors 22a to 22f. The projector 22a projects onto the front screen area 23e. Similarly, the projector 22b projects onto the screen area 23d, the projector 22c projects onto the screen area 23f, and the projector 22d projects a marker onto the screen area 23b. Also, the projector 22e projects onto the screen area 23a, and the projector 22f projects a marker onto the screen area 23c. Note that the projectors 22a to 22f are connected by a network, and the display data output unit 504 is configured to transmit display data to the projector to be displayed. The screen is also configured on the floor surface and the ceiling surface. The projectors 22b and 22e project onto the floor screen 25. Also, the projectors 27a and 27b project onto the ceiling screen 26.

[0034] Figure 2(b) is a partial cross-sectional view of the circular studio of the present embodiment shown in Figure 2(a). As described above, the display of the marker is configured to be projected onto the screen by the projector. The projector 22 projects onto the screens 23 and 25, and the projector 27 projects the type of marker set by the user onto the screen 26.

[0035] Figure 3(a) is a diagram in which some components are extracted from the image processing system shown in Figure 2(a). The virtual viewpoint 31 indicates the virtual camera position when a certain virtual viewpoint image is generated. Also, the marker 32 is projected onto the screen area 23a by the projector 22e. Now, since the subject 24 cannot actually see the position of the virtual camera (virtual viewpoint 31), when trying to align the line of sight with the virtual camera, one will look at the projected marker 32. Note that the projection position of the marker 32 is controlled by the marker control unit 503 so as to be on the extension line connecting the face (eyes) of the subject 24 and the virtual viewpoint 31. Also, the distance from the face of the subject to the virtual viewpoint 31 is represented by the size of the circle of the marker 32b. In this way, it is possible to represent where the virtual viewpoint 31 is located between the marker projected on the screen and the face of the subject by the display form (here, the size) of the marker. For example, it can be realized by notifying the subject in advance that when the circle is enlarged, the virtual camera is at a position closer to the subject, and when the circle is reduced, the virtual camera is at a position closer to the marker.

[0036] Figures 3(b) to (e) are diagrams showing the positional relationship among the subject, the virtual camera, and the marker, and are views of Figure 3(a) seen from the side. In Figures 3(b) to (e), the white circle 33 indicates the virtual viewpoint, and the black circle 34 indicates the marker projected onto the screen. As shown in this figure, the face of the subject, the virtual viewpoint, and the marker are configured to be aligned on a straight line. Since the screen is also configured on the floor surface and the ceiling surface, as shown in Figures 3(b) to (e), even if the virtual camera is at an angle from above or below the subject, the subject 24 can align the line of sight with the virtual camera.

[0037] Figure 4 is a flowchart showing the processing of the control data generation unit 502 of the marker output device 5.

[0038] In S100, the control data generation unit 502 starts the generation process. In S101, based on the data output from the marker information transmission unit 405, the control data generation unit 502 performs the generation process of the marker display setting file. This setting file includes information defining the type of the marker and the display parameters (color, shape, etc.) of the marker.

[0039] In S102, the control data generation unit 502 counts the number of subjects from the 3D models of the subjects in the virtual viewpoint image output from the virtual viewpoint image generation unit 301 and assigns subject IDs.

[0040] In S103, the control data generation unit 502 calculates the coordinates of the face of the subject in the world coordinate system from the 3D model information of the subject. When there are multiple subjects, the same process is performed for the number of people. Also, the identification of the face from the 3D model of the subject can be performed using general image processing techniques based on the shape and feature points of the model.

[0041] In S104, the control data generation unit 502 calculates the marker coordinates to be displayed based on the setting file, the subject position (face coordinates), and the virtual camera coordinates. The calculation of the marker coordinates is executed in the world coordinate system, and the marker coordinates that intersect the screen on the extension line between the face coordinates and the virtual camera coordinates are calculated. As a result, the screen to be projected and the projector to be projected are determined.

[0042] In S105, the control data generation unit 502 generates display parameter information from the setting file and outputs it to the marker control unit 503 together with the marker coordinates.

[0043] As a result, the marker control unit 503 controls the display data output unit 504 to display the markers of the set type at the set position.

[0044] Note that since the marker to be displayed is a symbol indicating the direction in which the virtual viewpoint exists for the subject, it may be an icon representing the virtual viewpoint camera. In this case, the icon may be displayed in a size corresponding to the distance in the real space between the subject and the virtual viewpoint. As a result, for the subject, not only the direction in which the virtual viewpoint exists but also the distance to the virtual viewpoint can be grasped.

[0045] Also, when there are a plurality of subjects, marker coordinates are calculated for each subject and displayed with different settings (color and shape), so that the lines of sight can be aligned at the same position. This can be realized by notifying each subject of the marker settings in advance. Also, if a plurality of subjects always direct their lines of sight to the same position, there may be only one marker.

[0046] [First Modification Example of the First Embodiment] In the above embodiment, the image processing system has been described assuming that the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 are independent devices. However, when there is sufficient processing power, these may be realized by one information processing device and an application program. The configuration of the image processing system in this case is shown in FIG. 10. The image processing system includes an information processing device 300, a plurality of cameras 350a, 350b,... as imaging devices 1, and a plurality of projectors 360a, 360b,... as display data output units 504. The information processing device 300 includes a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a display device 305, an operation unit 306, a camera I / F 307, and a projector I / F. Here, the camera I / F 307 and the projector I / F 308 may be realized as one I / F. It is assumed that the auxiliary storage device 304 enables the OS and real space information. Also, the configuration related to imaging and projection surrounding the subject is the same as in FIG. 2(a).

[0047] When the power of this device is turned on, the CPU 301 loads the OS stored in the auxiliary storage device 304 into the RAM 303 according to the boot program stored in the ROM 302 and executes it. As a result, this device functions as a device that performs processing according to instructions from the user. Further, when the CPU 301 loads and executes the image processing program from the auxiliary storage device 304 into the RAM 303, this device functions as the virtual viewpoint image generation unit 301, virtual camera path calculation unit 302, virtual camera information holding unit 303, marker coordinate calculation unit 304, display image generation unit 305, image display unit 402, virtual camera path instruction unit 403, user information transmission unit 404, marker information transmission unit 405, control data generation unit 502, and marker control unit 503 shown in FIG. 1.

[0048] The processing procedure when the CPU 301 executes the image processing program will be described with reference to the flowchart of FIG. 11. For the sake of simplicity of explanation, it is assumed that the information regarding the marker has already been set.

[0049] In S401, the CPU 301 inputs information related to the virtual viewpoint via the operation unit 306. This information includes the coordinates of the virtual viewpoint in the real space, as well as the line-of-sight direction and field angle from the virtual viewpoint.

[0050] In S402, the CPU 301 receives the captured image data from each of the cameras 350a, 350b,... via the camera I / F 307. Then, in S403, the CPU 301 generates a virtual viewpoint image by referring to the real space information (especially information such as the installation position and direction of the cameras) stored in the auxiliary storage device 304. And in S404, the CPU 301 displays the generated virtual viewpoint image on the display device 305.

[0051] In S405, the CPU 301 calculates, as the marker display position, the intersection position between the screen surface and the line connecting the coordinates of the face of the subject and the coordinates of the virtual viewpoint in the real space. Then, in S406, the CPU 301 causes the projector, for which the calculated marker display position is the projection target, to display a marker at the marker position via the projector I / F 308.

[0052] After that, in S407, the CPU 301 determines whether there is an instruction to end the application by the user. If not, the process returns to S401 and the above process is repeated.

[0053] Note that in this modification example, it is assumed that the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 are integrated into the information processing device in FIG. 3, but some of them may be independent devices.

[0054] Also, for example, when generating a virtual viewpoint image of the movement of an actor in a movie or the like, since it is easier for the actor to act with a sense of presence, each projector may display not only the marker but also the surrounding scenery surrounding the actor.

[0055] Note that the presentation position of the marker may be determined regardless of the virtual viewpoint. In particular, the position of the marker may be a position that makes the subject look in a certain direction, and for example, it may be specified by the user. [Second Modification Example of the First Embodiment] In the above-described embodiment, the marker is displayed by projecting it from the projector onto the screen. However, the configuration of the display data output unit 504 is not limited to this. As shown in FIG. 12, a plurality of liquid crystal displays may be installed so as to be spread over the surfaces of the screens 23, 25, 26, and 28. In this case, in S104, the control data generation unit 502 determines the display 28a corresponding to the marker coordinates calculated in the world coordinate system, and calculates the display coordinates on the display screen based on the position and orientation of the display 28a that has been calibrated in advance. For the calibration of the display, the three-dimensional coordinates of the marker displayed on the display can be calculated by triangulation using the cameras 21a to 21f, and can be performed by associating them with the coordinates of the marker on the display. Also, in S105, the marker control unit 503 controls the display data output unit 504 to display the marker 32 of the type set at the display coordinates on the corresponding display 28a. On the screens other than the marker display coordinates on the display and on the screens of the other displays where the marker is not projected, the color of the screen may be displayed so as not to be conspicuous, or the surrounding scenery surrounding the performer may be displayed.

[0056] [Third Modification of the First Embodiment] In the first embodiment, the direction in which the virtual viewpoint exists is displayed using a dot marker. However, in order to further indicate the posture of the virtual camera, a three-dimensional model of the virtual camera may be projected onto the screen. At this time, the three-dimensional model of the virtual camera is stored in the auxiliary storage device 214 in advance and expanded in the RAM 213 when the application program is executed. The type of the marker becomes the three-dimensional model of the virtual camera. FIG. 13(a) shows a state in which the three-dimensional model of the virtual camera is projected. In S405, the CPU 301 virtually projects the three-dimensional model of the virtual camera from the coordinates 24f of the face of the subject in the real space toward the screen 23a, and calculates the three-dimensional coordinates of the image 32c. Then, by back-projecting the three-dimensional coordinates of the image 32c toward the projector 22d, an image of the marker to be displayed by the projector 22d is generated. Note that the three-dimensional model of the virtual camera may have an arbitrary shape, texture, and transmittance, and may also be a three-dimensional arrow model or the like instead of a camera.

[0057] As a result, in addition to the direction in which the virtual viewpoint exists, the subject can also recognize the posture of the virtual viewpoint.

[0058] [Fourth Modification of the First Embodiment] In the above embodiment, the direction and posture in which the virtual viewpoint exists are displayed using points or three-dimensional models. However, a virtual viewpoint image may be displayed to show the viewing angle of the virtual viewpoint. At this time, the display image generation unit 305 transmits the virtual viewpoint image generated immediately before to the projector or display. The type of the marker is the virtual viewpoint image. Fig. 13(b) shows how the virtual viewpoint image is projected. In S104, the control data generation unit 502 calculates the marker coordinates on the screen or display, and performs a projective transformation on the virtual viewpoint image so that the marker coordinates are at the center and the virtual viewpoint image can be seen facing the subject directly. The projective transformation uses an affine transformation or a homography transformation. In S105, the marker control unit 503 controls the display data output unit 504 to project the virtual viewpoint image 32d subjected to the projective transformation onto the corresponding display coordinates on the screen or display.

[0059] As a result, the subject can recognize the direction in which the virtual viewpoint exists, the viewing angle at the virtual viewpoint, and the appearance from the virtual viewpoint by looking at the virtual viewpoint image. Note that the user may change the display settings of the marker as needed to flip the virtual viewpoint image horizontally so that the virtual viewpoint image is displayed as if the subject itself is seen in a mirror image.

[0060] [Fifth Modification of the First Embodiment] When the number of virtual viewpoints is very large, such as 10 or more, and there are a plurality of virtual viewpoints moving at high speed, if the markers are displayed as they are, they may look cumbersome when viewed from the subject. In this modification, the virtual viewpoints that are close to each other are grouped and displayed, or the markers of the virtual viewpoints moving at high speed are displayed so as to be difficult to visually recognize.

[0061] The virtual camera information holding unit 303 transmits a plurality of virtual camera paths at the same time code to the marker coordinate calculation unit 304. The marker display change processing flow in the marker coordinate calculation unit 304 is shown in FIG. 14. At S201, a plurality of coordinates and speeds are acquired as virtual camera information. At S202, the distance threshold value (radius 2 m) and the quantity threshold value (= 10) described in the marker display setting are acquired, and it is determined whether there are 10 or more virtual cameras within 2 m of each other among the received plurality of virtual cameras. If No, the process proceeds to S204. If Yes, at S203, the center of gravity of the set of virtual cameras within 2 m of each other is determined as a representative point, and the marker is displayed only for the representative point. At this time, the color or marker type may be changed to indicate that it is a representative point, or the number of elements in the set may be displayed in text. At S204, based on the minimum speed threshold value (2 m / s), the maximum speed threshold value (10 m / s), and the maximum transmittance (90%) described in the marker display setting, the transmittance of the virtual camera with a speed of 2 to 10 m / s is linearly changed and determined. Note that the transmittance is 0% for 0 to 2 m / s and 90% for 10 m / s or more. The method for determining the transmittance is not limited to this. Also, not limited to transmittance, the marker display may be changed so that the faster the speed of the virtual camera, the more difficult it is to visually recognize it by changing the lightness, chroma, etc.

[0062] As a result, the subject can concentrate on the performance even when the number of virtual viewpoints is large.

[0063] Also, as an effect of this modification, when there is a delay from the shooting of the subject to the display of the marker, the position deviation due to the delay of the virtual camera that moves quickly is made less visible by the transmittance of the marker. And the virtual camera that does not move quickly has relatively little position deviation due to the delay and can be made easily visible without being transmitted.

[0064] [Second Embodiment] In the first embodiment and its modification described above, the position of the marker visually recognized by the subject is configured such that the line of sight (line of vision) matches the position of the virtual camera among the virtual camera parameters. The content expressed by the marker is not limited to what is described above.

[0065] In the second embodiment, although the marker to be displayed is related to the virtual camera parameters, it does not align the line of sight with the virtual camera, but indicates the position where the subject wants to align the line of sight (the position where the subject wants to direct the line of sight). By adopting such a configuration, it becomes possible to generate, as an image captured by the virtual camera, a scene of the line of sight directed by the subject towards the marker.

[0066] FIG. 5(a) is a diagram for explaining this embodiment. The marker 51 in the figure indicates the position where the subject wants to align the line of sight (the position where the subject wants to direct the line of sight) as described above. The type of marker in this embodiment is different from that in the first embodiment. Therefore, the switching of the marker type is performed based on the marker type parameter information transmitted by the marker information transmission unit 405 of the user terminal 4.

[0067] Reference numerals 52a and 52b in the figure are for the virtual camera. Although the subject 24 cannot actually be visually recognized, an image captured by the virtual camera of the line of sight directed at the marker 51 can be generated.

[0068] FIG. 5(b) is a diagram schematically showing the positional relationship between the virtual camera and the marker. The positional relationship between the virtual camera and the marker in this embodiment is defined in the marker information transmission unit 405 as a setting file in advance. In the setting file, for example, the marker 53 corresponding to the virtual camera path 55 is defined as an angle 57 formed on the extension line with the face of the subject. In this way, it becomes possible to create a free viewpoint image of the image of the subject looking at a certain target as images from various angles.

[0069] [Third Embodiment] In the first embodiment, its modification, and the second embodiment, the marker was configured to be visible to the subject by projecting it onto the screen with a projector. However, when generating free viewpoint video at a temporary studio or a filming location, it may not always be possible to configure a screen. Therefore, in the present embodiment, an indicator is provided on the mounting member equipped with the camera, and the position of the virtual camera is recognized by the subject based on the display manner of the indicator.

[0070] The third embodiment will be described with reference to FIG. 6. Cameras 61a to 61c are mounted on mounting members 62a to 62c arranged so as to surround the subject in the real space. These mounting members are provided with indicators, and the lighting position, blinking speed, display color, etc. in the height direction are controlled by the marker control unit 503. The reference numeral 64 in the figure indicates the position of the virtual camera, which cannot be directly seen by the subject. However, the subject can recognize the approximate position of the virtual camera based on the display of the indicator. For example, when indicators 63a and 63b are lit, it can be recognized that the position of the virtual camera is between the real cameras 61a and 61b and at the height of the approximate indicator. Furthermore, when indicator 63a blinks, it can be recognized that the position of the virtual camera is on the side closer to the real camera 61a.

[0071] By configuring the indicator on the mounting member of the real camera in this way, a configuration can be achieved in which the subject can recognize the approximate virtual camera position.

[0072] When adopting this third embodiment, the marker output device 5 in FIG. 1 will control the display of the indicator.

[0073] [Fourth Embodiment] In each of the above-described embodiments (including modified examples), the presentation of the marker was represented by a marker that two-dimensionally displays the position of the virtual camera in the real space. In the present embodiment, a configuration will be described in which the position of the virtual camera itself is presented three-dimensionally, that is, visually recognizable to the subject.

[0074] In the fourth embodiment, as the marker, for example, a self-flying device having a hovering function such as a drone is used. As the marker of the present embodiment, any marker may be used as long as it has power and can fly and has a configuration in which coordinate values can be controlled. That is, the marker of the present embodiment receives the virtual camera position coordinates output by the marker control unit 503 as the world coordinate values in the real space and flies to that position. Note that there may be a plurality of markers.

[0075] For example, as shown in FIG. 7, the marker has a configuration in which the flying object 71 has a mark (arrow display 72 and side marking 73 in the drawing) that allows the viewing direction of the virtual camera to be known.

[0076] With such a configuration, the subject can recognize the position of the virtual camera. When there are a plurality of markers and a free viewpoint video is generated, this marker may enter the viewing angle of the virtual camera. However, since the position of the virtual camera and the shape of the marker are known in advance, as a video, it is possible to generate a free viewpoint video by erasing the marker. At this time, the coordinates of the marker are projected onto each of the cameras 21, and the projected coordinates on the camera video are excluded from the model generation and rendering processes to prevent the image of the marker from affecting the model generation and rendering. In addition, the target area at that position may be erased by smoothing and compositing with the same texture as the surroundings, or may be erased by displaying the marker in the same color as the background in advance. [Modified Example of the Fourth Embodiment] In the above-described fourth embodiment, a drone is used. However, the position and orientation of the virtual camera may be displayed using other devices that can display three-dimensionally. For example, there are an aerial display that emits light from air by exciting plasma, an anaglyph, a 3D display of the polarized shutter method, and glasses-type or contact lens-type devices such as a holographic lens capable of AR display. In any case, the three-dimensional model of the marker is projected onto the coordinates of each eye of the subject on the left and right in the virtual space, and a virtual camera can be displayed three-dimensionally for the subject by generating an image for display on each device. Further, when there are a plurality of virtual cameras, the markers to be displayed for each device may be controlled to make it easier to recognize a specific virtual camera for each subject.

[0077] As a result, the subject can visually recognize the position and orientation of the virtual camera three-dimensionally while avoiding the risk of contacting the drone.

[0078] [Fifth Embodiment] In the above-described embodiments, the marker indicated information related to the line of sight of the subject. In the real space, when the subject performs, there may be cases where the line of sight does not necessarily match something. In that case, when the subject stands at a certain position in the real space and performs a scene, there may be a case where the subject wants to recognize the standing position. In the fifth embodiment, the display of the marker is projected onto the floor surface. The subject can recognize the standing position linked to the virtual camera based on the content displayed on the floor surface.

[0079] The configuration of this embodiment is shown in FIG. 8. The projector 83 is attached to the free pan-tilt head 84, and the projection position can be controlled by the pan angle, tilt angle, and zoom magnification. Further, as the display content 86, the standing position, the moving direction, etc. are displayed. Also, the display position and display content of the marker are controlled by the virtual camera position and the marker position according to the setting file defined by the marker information transmission unit 405. By configuring in this way, for example, it becomes possible to generate a free viewpoint video as if capturing the state of moving from one standing position to another with a virtual camera.

[0080] In the embodiment described so far, the subject visually recognizes the position of the marker when shooting, but the present invention is not necessarily limited to this configuration. For example, as described above, the marker position may be superimposed on the archive video as well, so that the viewer can also understand the relationship between the virtual camera and the line of sight, without being limited to a configuration in which the marker position is used only when shooting.

[0081] To achieve the above, multiple streams can be prepared for each virtual viewpoint. Furthermore, by preparing a video that overlooks the real space and configuring the marker position to be interactive, the user can switch the camera position and view the free viewpoint video.

[0082] In addition, images captured by multiple cameras may be stored as archived images, and when an operator generates a free viewpoint video at a later date, the face position (gaze) of the subject may be detected and a marker may be displayed at the position on the extension line in the virtual space. By configuring in this way, the operability of the operator is improved when determining the angle of view of the free viewpoint video.

[0083] In addition, the markers in the embodiments described so far will naturally appear in the images captured by each camera, and may be a hindrance in the generated free viewpoint video. However, since the display position, color, shape, etc. of the marker are known in the virtual space, it is possible to erase the marker by smoothing the area corresponding to the position with the same texture as the surroundings and performing a synthesis process. In addition, the above-mentioned problem can be easily solved by displaying the marker in the same color as the background in advance.

[0084] (Other Examples) The present disclosure can also be implemented by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be implemented by a circuit (for example, ASIC) that realizes one or more functions.

[0085] The invention is not limited to the above-described embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention.

Explanation of Signs

[0086] 1... Multi-viewpoint video holding unit, 2... Subject information holding unit, 3... Image processing apparatus, 301... Virtual viewpoint image generation unit, 302... Virtual camera path calculation unit, 303... Virtual camera information holding unit, 304... Marker coordinate calculation unit, 305... Display image generation unit, 306... Apparatus communication unit, 4... User terminal, 401... Terminal communication unit, 402... Image display unit, 403... Virtual camera path instruction unit, 404... User information transmission unit, 405... Marker information transmission unit, 5... Marker output apparatus, 501... Apparatus communication unit, 502... Control data generation unit, 503... Marker control unit, 504... Display data output unit

Claims

1. A processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, comprising: a plurality of photographing means for photographing a space including a subject; specifying means for specifying the position of the virtual viewpoint; presentation means for presenting, in the space, a presentation for determining the orientation of the subject based on the position of the virtual viewpoint.

2. The plurality of photographing means and a screen for projection are arranged so as to surround the subject, and the presentation means projects information for determining the orientation of the subject onto the screen. The processing system according to claim 1, wherein:

3. The processing system according to claim 2, wherein the presentation means projects information for determining the orientation of the subject at a position of an intersection of an extension line connecting the subject and the virtual viewpoint and the screen.

4. The processing system according to claim 2, wherein the presentation means projects information for determining the orientation of the subject at a position of an intersection of a line shifted in a predetermined direction with respect to an extension line connecting the subject and the virtual viewpoint and the screen.

5. The processing system according to any one of claims 1 to 4, wherein the presentation means presents information for determining the orientation of the subject in a size corresponding to the distance between the subject and the virtual viewpoint.

6. The plurality of photographing means are attached to a mounting member having an indicator that lights up a set position with respect to the position in the height direction, and are arranged so as to surround the subject, and the presentation means lights up a position for directing the line of sight of the subject. The processing system according to claim 1, wherein:

7. The processing system according to claim 1, wherein the presentation means is means for positioning a flying device at a position for directing the line of sight of the subject.

8. The processing system according to claim 7, wherein the flying device is provided with a mark representing the line-of-sight direction of the virtual viewpoint.

9. The presentation means is means for controlling the pan and tilt angles of a projector mounted on a mounting member capable of controlling the pan angle and tilt angle, and projects information for determining the orientation of the subject onto a floor corresponding to the position for directing the line of sight of the subject. The processing system according to claim 1, wherein:

10. The processing system according to any one of claims 1 to 9, wherein the presentation means presents information for determining the orientation of the subject at the position of the virtual viewpoint.

11. The processing system according to any one of claims 1 to 9, wherein the presentation means presents information for determining the orientation of the subject at a position different from the position of the virtual viewpoint and at a position in the direction in which the subject is facing.

12. The processing system according to any one of claims 1 to 11, wherein the orientation of the subject is the orientation of the face of the person who is the subject.

13. The processing system according to any one of claims 1 to 11, wherein the orientation of the subject is the orientation of the line of sight of the person who is the subject.

14. A user terminal having designation means for designating the virtual viewpoint and display means for displaying the virtual viewpoint image, An image processing apparatus for generating the virtual viewpoint image, A presentation apparatus having the presentation means The processing system according to any one of claims 1 to 13, characterized by comprising the above.

15. The processing system according to any one of claims 1 to 14, wherein the presentation means presents a virtual viewpoint image.

16. The processing system according to any one of claims 1 to 15, wherein the presentation means controls color or shape according to the number or speed of the virtual viewpoints.

17. A control method for a processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, comprising: A step of photographing a space including a subject with a plurality of photographing means; A specifying step of specifying the position of the virtual viewpoint; A presenting step of performing a presentation for determining the orientation of the subject in the space based on the position of the virtual viewpoint; A control method characterized by comprising the above.

18. A program for causing a computer to execute each step of the method according to claim 17.

Images