Information processing device, information processing method, and program

Abstract

To display a virtual viewpoint image suitable for an instrument for viewing.SOLUTION: An information processing apparatus acquires device information indicating the type of an instrument for viewing for a virtual viewpoint image, and performs control of causing the instrument for viewing to display a virtual viewpoint image that is created based on virtual viewpoint information determined based on the type of the instrument for viewing indicated by the acquired device information and a plurality of picked-up images obtained by picking up images of a predetermined area from a plurality of directions.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] A technology that uses multiple cameras installed at different positions to capture images synchronously and generate a virtual viewpoint image according to an arbitrarily specified virtual viewpoint using the multiple viewpoint images obtained by the capture is attracting attention. In a service that uses such a virtual viewpoint image, for example, a specific scene (such as a goal shot) in a soccer or basketball game can be viewed from various angles, providing a higher sense of realism to the user than with conventional captured images.

[0003] An image processing device such as a server that provides such a service collects images captured by multiple cameras of a specific position (area) in the space to be imaged, generates three-dimensional shape data, performs rendering, and transmits the generated virtual viewpoint image to a user terminal. This makes it possible to generate and view a virtual viewpoint image based on multiple viewpoint images.

[0004] Patent document 1 discloses a technology in which, upon receiving a user operation to change the virtual viewpoint position, a composite image of a vehicle viewed from the virtual viewpoint and a model diagram showing the virtual viewpoint position and its field of view range, which are displayed on a display device, are switched to the changed version. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2011-151446 A Summary of the Invention [Problem to be solved by the invention]

[0006] Various types of devices may be used when viewing a virtual viewpoint image. For example, devices such as a stationary display, a mobile terminal such as a smartphone or a tablet terminal, a head-mounted display (hereinafter also referred to as an HMD), a table-type display, etc. may be used to view a virtual viewpoint image. In this case where various devices may be used, depending on the terminal used, a virtual viewpoint image different from the virtual viewpoint image desired by the user may be displayed.

[0007] In view of the above-mentioned problems, the present disclosure provides a technique for displaying a virtual viewpoint image suitable for a viewing device. [Means for solving the problem]

[0008] An information processing device according to an embodiment of the present disclosure includes: The present invention has an acquisition means for acquiring device information indicating a type of viewing device for a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a specified area from a plurality of directions, and a determination means for determining a position and orientation of a virtual viewpoint corresponding to the virtual viewpoint image based on the type of viewing device indicated by the device information, wherein when the orientation of a display unit of the type of viewing device indicated by the device information acquired by the acquisition means is landscape orientation substantially parallel to a horizontal plane, the determination means determines the position and line of sight of the virtual viewpoint for observing a target object from the surroundings within the specified area. It is characterized by: Effect of the Invention

[0009] According to the present disclosure, it is possible to display a virtual viewpoint image suitable for a viewing device. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an outline of an image processing system. [Diagram 2] FIG. 1 illustrates an example of the configuration of an image processing system. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of each device. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an image generating apparatus. [Diagram 5] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Figure 6] FIG. 11 is a diagram illustrating an example of virtual viewpoint parameters. [Figure 7] FIG. 11 is a diagram illustrating an example of terminal information. [Figure 8] FIG. 13 is a diagram illustrating an example of metadata. [Figure 9] FIG. 13 is a diagram showing an example of object dimension information. [Figure 10]11A and 11B are diagrams illustrating examples of viewpoints determined by a virtual viewpoint determination unit. [Figure 11] 4 is a flowchart showing a flow of processing executed by the information processing device. [Figure 12] FIG. 13 is a diagram showing an example of thumbnail display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the following embodiment does not necessarily limit the present disclosure. In addition, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present disclosure.

[0012] [First embodiment] (System Configuration) An overview of an image processing system according to this embodiment will be described with reference to the drawings. FIG. 1 is a diagram showing a schematic example of an image processing system according to this embodiment. As shown in FIG. 1, an image processing system 100 has a plurality of cameras (imaging devices) 104 installed in a stadium 101 such as a soccer field. The stadium 101 includes a field 103 where a sporting event or the like is actually held, and spectator seats 102 surrounding the field. The multiple cameras 104 are arranged to surround the spectator seats 102 and the field 103.

[0013] Next, a configuration example of an image processing system will be described with reference to the drawings. Fig. 2 is a diagram showing a configuration example of an image processing system 100. As shown in Fig. 2, the image processing system 100 has a plurality of cameras 104 installed in a stadium 101, an image generating device 201, an information processing device 202, and a user terminal 203.

[0014] The multiple cameras 104 are arranged so that each captures at least a part or all of the range of the field 103, which is the imaging target area, and so that the angles of view of at least two cameras overlap, and are connected to each other, for example, via a transmission cable. The cameras 104 are also installed facing one or more real camera gaze points that are set in advance. In other words, each of the one or more real camera gaze points is captured from different directions by two or more cameras 104. The multiple cameras 104 are also connected to an image generating device 201, and transmit images acquired by capturing the field 103 to the image generating device 201.

[0015] The cameras 104 may be cameras that capture still images, cameras that capture moving images, or cameras that capture both still images and moving images. In this embodiment, unless otherwise specified, the term "image" includes the concepts of both still images and moving images.

[0016] The image generating device 201 is a device that generates a virtual viewpoint image. The virtual viewpoint image in this embodiment is also called a free viewpoint video, and is an image corresponding to a viewpoint freely (arbitrarily) specified by a user. However, the present invention is not limited to this, and for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates is also included in the virtual viewpoint image. In this embodiment, the virtual viewpoint image is mainly described as a moving image, but the virtual viewpoint image may be a still image. The virtual viewpoint may be specified by a user operation or automatically by a device. In this embodiment, an example in which an information processing device described later determines the virtual viewpoint will be described.

[0017] In addition, the method described in this embodiment will be a method of displaying a suitable virtual viewpoint image according to the terminal (device) used to view the virtual viewpoint image. For example, when a user uses an HMD (head mounted display), the user views a virtual viewpoint image displayed on a small display of the HMD with the small display placed in front of both eyes of the user. At this time, when a virtual viewpoint image corresponding to the viewpoint of a player playing soccer on the field or the viewpoint of a fictional character standing on the field is displayed on the small display of the HMD, the user wearing the HMD can experience virtual reality from the character's point of view. In addition, when a user uses a table-type display to display a virtual viewpoint image, it is considered that, for example, a virtual viewpoint image overlooking the entire field is often displayed. In this way, it is considered that the virtual viewpoint image to be displayed differs depending on the type of terminal.

[0018] Therefore, if the same virtual viewpoint image is always displayed regardless of the type of terminal, it may happen that the virtual viewpoint image desired by the user is not displayed. For example, when a virtual viewpoint image is displayed using an HMD, if an image that overlooks the entire field from above is displayed, the image may differ from the image desired by the user. Of course, after the virtual viewpoint image is displayed, the position of the virtual viewpoint and the line of sight direction may be changed by the user's operation, but this may increase the burden of the operation on the user. In this embodiment, a method for solving such a problem will be described.

[0019] The image generating device 201 stores a plurality of images captured by a plurality of cameras (hereinafter, the captured images may be referred to as "captured images") and generates a group of virtual viewpoint images using the stored plurality of images. The group of virtual viewpoint images is a collection of virtual viewpoint images corresponding to different virtual viewpoints. The image generating device 201 transmits the generated group of virtual viewpoint images and metadata to the information processing device 202. The metadata is position information indicating the positions of players playing on the field 103. The image generating device 201 is, for example, a server device, and has an image processing function for generating virtual viewpoint images, as well as a function as a database for storing a plurality of captured images and the generated group of virtual viewpoint images. In addition, the plurality of cameras 104 in the stadium 101 and the image generating device 201 are connected by a wired or wireless communication network line or a cable line such as SDI (Serial Digital Interface). The image generating device 201 receives the images captured by the plurality of cameras 104 through this line and stores them in a database.

[0020] The information processing device 202 selects a virtual viewpoint image to be provided to the user terminal 203 from the group of virtual viewpoint images generated by the image generating device 201. More specifically, the information processing device 202 determines the type of virtual viewpoint information used in the generation process of the virtual viewpoint image to be output to the user terminal 203 based on device information (terminal information) that identifies the user terminal 203. Then, the information processing device 202 selects a virtual viewpoint image according to the determined type of virtual viewpoint information as the virtual viewpoint image to be output to the user terminal 203. Examples of the user terminal 203 include viewing devices for virtual viewpoint images, such as stationary displays, mobile terminals such as smartphones and tablet terminals, head-mounted displays, table-type displays, and projectors. The user terminal 203 can be classified into three types, for example, as follows, according to the viewing form of the user. As the first type, a display device suitable for displaying an image corresponding to the viewpoint of a person in a predetermined area of ​​the field is classified, in which a small display that displays an image corresponding to each of both eyes of the user is arranged at a position relatively close to the user. For example, a head-mounted display is included. Therefore, such device information is stored in advance as data that is linked to information on a virtual viewpoint corresponding to the viewpoint position and line of sight of a specific player in a specific area of ​​the field. Alternatively, the device information may be linked to information on a virtual viewpoint corresponding to the viewpoint position and line of sight of a virtual person existing in the area of ​​the field. The viewpoint of the virtual person is set, for example, at the center of the field and at a height from the ground of the field that corresponds to the average height of a person. This allows the user to see an image as if he or she is standing on a field that he or she cannot actually enter.

[0021] The second type is a display device in which the display unit that displays the image is arranged horizontally, is arranged at a position relatively far from the user, and is suitable for displaying an image corresponding to the viewpoint position and line of sight direction for observing a target player from the surroundings within a specified area of ​​the field. For example, a stationary display and a projector are included. In addition, if detection information indicating that the display unit is horizontal is added to the device information, a mobile terminal such as a smartphone or a tablet terminal is also included. Therefore, such device information is stored in advance as data linked to information on a virtual viewpoint corresponding to the viewpoint position and line of sight direction for observing a target player from the surroundings within a specified area of ​​the field.

[0022] As the third type, display devices are classified as those in which the display unit that displays the image is arranged facing upward, in a position where the user can look down, and suitable for displaying an image corresponding to the viewpoint position and line of sight direction overlooking a part or all of a predetermined area of ​​the field. For example, a table-type display falls under this category. In addition, when detection information indicating that the display unit is facing upward is added to the device information, mobile terminals such as smartphones and tablet terminals also fall under this category. Therefore, such device information is stored in advance as data linked to information on a virtual viewpoint corresponding to the viewpoint position and line of sight direction overlooking a part or all of a predetermined area of ​​the field.

[0023] The virtual viewpoint information is information input from the user terminal 203, and is information including the virtual viewpoint position and line of sight direction (posture) used in the generation process of the virtual viewpoint image to be output to the user terminal 203.

[0024] When the information processing device 202 receives device information from the user terminal 203, the information processing device 202 determines the type of the user terminal 203 based on the device information, and determines virtual viewpoint information previously associated with the type of the user terminal 203 based on the determination result. Then, the information processing device 202 outputs a virtual viewpoint image corresponding to the determined virtual viewpoint information to the user terminal 203. That is, the information processing device 202 determines a type of virtual viewpoint information corresponding to a virtual viewpoint image considered to be an optimal image for the user when displayed on the user terminal. Then, the information processing device 202 selects a virtual viewpoint image according to the determined type of virtual viewpoint information from a group of virtual viewpoint images acquired from the image generating device 201. When the information processing device 202 receives an operation instruction to start playing back a virtual viewpoint image from the user terminal 203, the information processing device 202 transmits (outputs) to the user terminal 203 the virtual viewpoint image selected based on the determined type of virtual viewpoint information.

[0025] In addition, the information processing device 202 in this embodiment holds in advance a plurality of pieces of virtual viewpoint information corresponding to different types of devices, and determines appropriate virtual viewpoint information according to the type of the user terminal 203 determined based on the acquired device information. However, the present invention is not limited to this. For example, if it is determined in advance that the information processing device 202 will display a virtual viewpoint image on a specific type of device, there is no need to hold a plurality of pieces of virtual viewpoint information. In this case, the information processing device 202 checks whether the user terminal 203 that will display the virtual viewpoint image is a specific type of device based on the acquired device information, and if confirmed, displays the virtual viewpoint image corresponding to the virtual viewpoint information held in advance on the user terminal 203.

[0026] Furthermore, the information processing device 202 may display a virtual viewpoint image corresponding to a default virtual viewpoint when the user terminal 203 is not a type of device associated with the pre-stored virtual viewpoint information. As the default virtual viewpoint, for example, a viewpoint for viewing the field from the side may be set in advance. Furthermore, when the user terminal 203 is not a type of device associated with the pre-stored virtual viewpoint information, any virtual viewpoint information may be determined from the pre-stored virtual viewpoint information.

[0027] As a supplement, the information processing device 202 is, for example, a personal computer. Note that some or all of the functions of the information processing device 202 may be incorporated into the image generating device 201 or into the user terminal 203. Also, the reverse may be possible. That is, for example, some or all of the functions of the image generating device 201 may be incorporated into the information processing device 202. For example, the information processing device 202 may generate a virtual viewpoint image. In this way, the information processing device 202 and the image generating device 201 may be an integrated device, and the information processing device 202 and the user terminal 203 may be an integrated device. Also, any function among the functions of the image generating device 201, the information processing device 202, and the user terminal 203 in this embodiment may be executed by one device.

[0028] The user terminal 203 is an information processing device operated by a user who uses the image processing system 100. The user terminal 203 accepts operation instructions such as, for example, moving the virtual viewpoint position, changing the attitude of the virtual viewpoint (the line of sight direction of the virtual viewpoint), and switching the viewpoint based on the user's input. When the user terminal 203 accepts an operation instruction from the user, it generates a transmission signal related to information including a virtual viewpoint parameter indicating the content of the accepted operation instruction, and transmits the generated transmission signal to the information processing device 202. The virtual viewpoint parameter in this embodiment includes time information, virtual viewpoint position information, and attitude information. However, the information included in the virtual viewpoint parameter is not limited to this, and may further include, for example, angle of view information corresponding to the width of the field of view (angle of view) of the virtual viewpoint. In addition, the virtual viewpoint parameter may be configured to include any information among time information, position information, attitude information, and angle of view information.

[0029] The virtual viewpoint parameters in this embodiment will be described with reference to the drawings. FIG. 6 is a diagram showing an example of the configuration of the virtual viewpoint parameters. The time information is configured as HH (hours):MM (minutes):SS (seconds):FF (frames). The position information may be indicated by three-dimensional orthogonal coordinates of a coordinate system in which three coordinate axes (x-axis, y-axis, z-axis) intersect at the origin so as to be perpendicular to each other. The origin in this case may be, for example, the center of the center circle of the field 103, or any position in the imaging space. The attitude information may be indicated by angles relative to three axes, namely, pan (horizontal direction), tilt (vertical direction), and roll (direction in which the camera rotates). In addition, the user terminal 203 can receive a virtual viewpoint image from the information processing device 202 and display the received virtual viewpoint image on the display unit of a built-in (or, in some cases, external) display device.

[0030] The user terminal 203 is a terminal device, and is a viewing device for a virtual viewpoint image, such as a personal computer, a mobile terminal such as a smartphone or a tablet terminal, a head-mounted display, a table-type display, a stationary display, a projector, etc. The user terminal 203 has an interface for accepting a user operation such as at least one of a mouse, a keyboard, a six-axis controller, and a touch panel. The user terminal 203 also has terminal information (device information) for identifying the terminal device and a connected device connected to the terminal device. When the user terminal 203 is connected to the information processing device 202, it transmits its own terminal information to the information processing device 202. Here, the configuration of the terminal information will be described with reference to a diagram. FIG. 7 is a diagram showing an example of the configuration of the terminal information. The terminal information includes a terminal device name and a terminal device ID indicating an identification code corresponding to the terminal device name. The terminal device name is an arbitrary name. The terminal device ID is an identification code that uniquely identifies the type of viewing device, and is an identification symbol composed of alphabets and numbers, which is assigned to distinguish (identify) each device. In the example of FIG. 7, "HMD123456" is set as the terminal device ID for the terminal device name "head mounted display."

[0031] The image generating device 201, the information processing device 202, and the user terminal 203 are configured to be able to exchange information with each other via a network such as the Internet. Note that communication between the devices may be performed by either wireless communication or wired communication, or a combination of these.

[0032] (Device hardware configuration) Next, an example of the hardware configuration of each of the above-mentioned devices will be described with reference to the drawings. Fig. 3 is a diagram showing an example of the hardware configuration of the image generating device 201, the information processing device 202, and the user terminal 203. Each device has a common hardware configuration, and includes a controller unit 300, an operation unit 309, and a display device 310.

[0033] The controller unit 300 includes a CPU 301, a ROM 302, a RAM 303, a HDD 304, an operation unit I / F (interface) 305, a display unit I / F 306, and a communication I / F 307. These are also connected to each other via a system bus 308.

[0034] A central processing unit (CPU) 301 controls the operations of a ROM 302, a RAM 303, a HDD 304, an operation unit I / F 305, a display unit I / F 306, and a communication I / F 307 via a system bus 308. The CPU 301 starts an operating system (OS) by a boot program stored in a read only memory (ROM) 302. The CPU 301 executes an application program stored in a hard disk drive (HDD) 304, for example, on the started OS. The CPU 301 executes the application program to realize various processes of each device. A random access memory (RAM) 303 is used as a temporary storage area such as a main memory or a work area of ​​the CPU 301. The HDD 304 stores the application program and the like as described above. The CPU 301 may be configured with one processor or multiple processors.

[0035] The operation unit I / F 305 is an interface with the operation unit 309. The operation unit I / F 305 sends information input by a user in the operation unit 309 to the CPU 301. The operation unit 309 has devices capable of receiving user operations, such as a mouse, a keyboard, a touch panel, etc. The display unit I / F 306 is an interface with the display device 310. The display unit I / F 306 outputs, for example, image data to be displayed on the display device 310 to the display device 310. The display device 310 has a display such as a liquid crystal display.

[0036] The communication I / F 307 is an interface for performing communication such as Ethernet (registered trademark). The communication I / F 307 is connected to a transmission cable and includes a connector for receiving the transmission cable. The communication I / F 307 inputs and outputs information between the communication I / F 307 and an external device via the transmission cable. The communication I / F 307 may be, for example, a circuit for performing wireless communication such as a baseband circuit or an RF circuit, or an antenna. The controller unit 300 can also perform display control for displaying an image on an external display device 310 connected via a cable or a network. In this case, the controller unit 300 realizes display control by outputting display data to the external display device 310. The configuration of FIG. 3 is an example, and some of the configurations may be omitted or configurations not shown may be added, and the configurations shown in the figures may be combined. For example, the image generating device 201 may not have the display device 310.

[0037] Although the hardware configurations of the image generating device 201, the information processing device 202, and the user terminal 203 have been described above with reference to Fig. 3, none of the configurations shown in Fig. 3 are essential. In the above description, the controller unit 300 has been described as having the CPU 301, but this is not necessarily limited to this. For example, the controller unit 300 may have hardware such as an ASIC (Application Specific Integrated Circuit), a DSP, or an FPGA instead of or in addition to the CPU 301. Hardware such as an ASIC, a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array) may perform part or all of the processing performed by the CPU 301.

[0038] (Functional configuration of the image generating device) The functional configuration of the image generating device will be described with reference to the drawings. Fig. 4 is a block diagram showing an example of the functional configuration of the image generating device 201. Note that each function shown in Fig. 4 is realized, for example, by the CPU 301 of the image generating device 201 reading out various programs recorded in the ROM 302 and executing control of each unit. Also, a part or all of the configuration shown in Fig. 4 may be realized by dedicated hardware such as an ASIC or an FPGA.

[0039] 4, the image generating device 201 includes a control unit 401, an information storage unit 402, a captured image input unit 403, an image storage unit 404, a virtual viewpoint image generating unit 405, a metadata generating unit 406, and a data output unit 407. These are also interconnected by an internal bus 408, and can transmit and receive data to and from each other under the control of the control unit 401.

[0040] The control unit 401 controls the overall operation of the image generating device 201 in accordance with a computer program stored in the information storage unit 402. The information storage unit 402 is configured with a non-volatile memory, and stores information such as programs (e.g., computer programs that control the overall operation of the image generating device 201) and various data.

[0041] The captured image input unit 403 acquires images captured by the multiple cameras 104 installed in the stadium 101 at a predetermined frame rate, and outputs the images to the image storage unit 404. The captured image input unit 403 acquires captured images from the cameras 104 via a wired or wireless communication module, or an image transmission module such as SDI.

[0042] The image storage unit 404 is a large-capacity storage device such as a magnetic disk, an optical disk, or a semiconductor memory. The image storage unit 404 stores the captured images acquired by the captured image input unit 403, a group of virtual viewpoint images generated based on the captured images, and metadata indicating the position of an object. The image storage unit 404 may be provided physically outside the image generating device 201. The captured images stored in the image storage unit 404 and the group of virtual viewpoint images generated based on the captured images are stored in an image format such as MXF (Material eXchange Format). In addition, the captured images stored in the image storage unit 404 and the group of virtual viewpoint images generated based on the captured images are compressed in, for example, MPEG2 format. However, the data format is not necessarily limited to these.

[0043] The virtual viewpoint image generating unit 405 generates a group of virtual viewpoint images based on a plurality of captured images stored in the image storage unit 404. As a method for generating a group of virtual viewpoint images, for example, a method using image-based rendering can be mentioned. Image-based rendering is a rendering method for generating virtual viewpoint images based on captured images obtained by capturing images from a plurality of actual viewpoints, without performing a modeling process (i.e., a process for creating the shape of an object using geometric figures).

[0044] The method for generating the virtual viewpoint images is not limited to image-based rendering, and model-based rendering (MBR) may be used. Here, MBR is a rendering method for generating virtual viewpoint images using a three-dimensional model generated based on a plurality of captured images acquired by capturing an image of a subject from a plurality of directions. Specifically, MBR is a rendering method for generating an image of a target scene from a virtual viewpoint using a three-dimensional shape (model) of the target scene acquired by a three-dimensional shape reconstruction method. Examples of the three-dimensional shape reconstruction method include the shape-from-silhouette method and Multi-View-Stereo (MVS).

[0045] The generated virtual viewpoint image group includes virtual viewpoint images of various virtual viewpoints, viewing directions, and gaze points, and in this embodiment, such a virtual viewpoint image group is compressed and encoded in the spatial direction and the time direction as one image stream. However, in other embodiments, the virtual viewpoint image group may be composed of a plurality of independent images, rather than one image stream. Alternatively, the virtual viewpoint image group may not be compressed and encoded.

[0046] Furthermore, instead of the group of virtual viewpoint images, the image generating device 201 may generate information for generating virtual viewpoint images, such as information indicating a three-dimensional model, an image for mapping onto the three-dimensional model, etc. That is, instead of generating a rendered virtual viewpoint image, the virtual viewpoint image generating unit 405 may generate information necessary for rendering the virtual viewpoint image in the information processing device 202 or the user terminal 203.

[0047] The metadata generating unit 406 analyzes the captured images stored in the image storage unit 404 and acquires position information of a specific object. The acquisition of position information of a specific object is executed for all times from the start of capturing the captured images stored in the image storage unit 404 to the end of capturing the captured images. The specific object is a person such as a specific player or a referee, or a ball. The metadata generating unit 406 generates metadata including the acquired position information of the specific object, and outputs the generated metadata to the virtual viewpoint image generating unit 405.

[0048] Here, an example of the configuration of metadata will be described with reference to a diagram. FIG. 8 is a diagram showing an example of the configuration of metadata. The metadata includes, for example, time information 801 and position information 802. The time information 801 is configured as HH (hours):MM (minutes):SS (seconds).FF (frames). The position information 802 indicates the position of a specific object, and is expressed using three-dimensional orthogonal coordinates. Examples of the specific object include a ball, a player, and a referee. In the example of FIG. 8, the specific objects include a ball, a player 01, a player 02, and a player 03. The object name has an arbitrary name and an object ID indicating an identification code. The object ID is expressed by alphabets and numbers, and is an identification symbol assigned to each object so that each object can be identified. Note that the metadata is not limited to that acquired by the analysis of the captured image by the metadata generation unit 406, and may be that registered in advance in the image generation device 201 or the information processing device 202.

[0049] The data output unit 407 outputs the virtual viewpoint image group stored in the image storage unit 404 at a predetermined frame rate to the information processing device 202. In addition, the data output unit 407 outputs metadata stored in the image storage unit 404 to the information processing device 202 together with the virtual viewpoint image group.

[0050] (Functional configuration of information processing device) The functional configuration of the information processing device will be described with reference to the drawings. Fig. 5 is a block diagram showing an example of the functional configuration of the information processing device 202. Each function shown in Fig. 5 is realized, for example, by the CPU 301 of the information processing device 202 reading out various programs recorded in the ROM 302 and executing control of each unit. Also, a part or all of the configuration shown in Fig. 5 may be realized by dedicated hardware such as ASIC or FPGA.

[0051] 5, the information processing device 202 includes a control unit 501, an information storage unit 502, a data receiving unit 503, an image storage unit 504, an information acquisition unit 505, a virtual viewpoint determination unit 506, an image acquisition unit 507, a display control unit 508, and an input / output unit 509. These are also interconnected by an internal bus 510, and can transmit and receive data to and from each other under the control of the control unit 501.

[0052] The control unit 501 controls the overall operation of the information processing device 202 in accordance with a computer program stored in the information storage unit 502. The information storage unit 502 is configured with a non-volatile memory, and stores programs (e.g., computer programs that control the overall operation of the information processing device 202) and various data. The data receiving unit 503 receives a group of virtual viewpoint images and metadata from the image generating device 201, and outputs them to the image storage unit 504.

[0053] The image storage unit 504 is a large-capacity storage device such as a magnetic disk, an optical disk, or a semiconductor memory. The image storage unit 504 stores the virtual viewpoint image group and metadata acquired by the data receiving unit 503, the virtual viewpoint parameter group and terminal information acquired by the input / output unit 509, and the object dimension information acquired by the information acquisition unit 505. The virtual viewpoint parameter group is a collection of virtual viewpoint parameters. The image storage unit 504 may be provided physically outside the information processing device 202.

[0054] The information acquisition unit 505 acquires object dimension information from an external database (not shown) in which object dimension information obtained by measuring the dimensions (size) of a specific object is stored in advance. Here, the object dimension information will be described with reference to a diagram. FIG. 9 is a diagram showing an example of the configuration of the object dimension information. The object dimension information is object information indicating the size of an object present in a captured image, and includes an object name 901, an object ID 902, and a height (body height) 903. The object name 901 is an arbitrary name. The object ID 902 is an identification code represented by alphabets or numbers, and is indicated by an identification symbol assigned to each object so that each object can be identified. The height 903 is a numerical value indicating the length in the height direction (the size in the vertical direction) of the object. Here, the height is represented by a numerical value in centimeters (cm), but the height is not limited to being represented by a numerical value in this unit.

[0055] When the virtual viewpoint determination unit 506 receives an operation instruction to start displaying a virtual viewpoint image from the user terminal 203, it acquires terminal information from the image storage unit 504 and determines the type of virtual viewpoint information indicating at least the virtual viewpoint position and attitude based on the acquired terminal information. In the process by the virtual viewpoint determination unit 506, at least one type of virtual viewpoint information is determined.

[0056] Here, the relationship between the terminal device ID of the terminal information and the type of virtual viewpoint information to be determined will be described with reference to the drawings. FIG. 10 is a diagram for explaining the type of virtual viewpoint information determined by the virtual viewpoint determination unit. FIG. 10(a) shows an example of a virtual viewpoint image when a player viewpoint is determined as the virtual viewpoint indicated by a specific type of virtual viewpoint information. FIG. 10(b) shows an example of a virtual viewpoint image when a player observation viewpoint is determined as the virtual viewpoint indicated by a specific type of virtual viewpoint information. FIG. 10(c) shows an example of a virtual viewpoint image when a bird's-eye viewpoint is determined as the virtual viewpoint indicated by a specific type of virtual viewpoint information.

[0057] When the terminal device ID of the terminal information acquired from the image storage unit 504 is a head mounted display, the virtual viewpoint determination unit 506 determines a viewpoint from an arbitrary object (a specific player's viewpoint in the example of FIG. 10(a)) as a specific type of virtual viewpoint. The object at this time is, for example, a player, a referee, a ball, etc. Note that a configuration may be adopted in which the viewpoint of a virtual person not present on the field, different from the object, can be selected as the virtual viewpoint. The object may be selected via the user terminal 203, or a player favored by the user that has been registered in advance in the user terminal 203 may be automatically selected. The object may also be automatically selected by the virtual viewpoint determination unit 506. For example, a player holding the ball or a player near the ball may be selected as an automatically selected object by the virtual viewpoint determination unit 506 based on position information included in the metadata. The virtual viewpoint determination unit 506 identifies the position and moving direction of the selected object from the metadata, and determines the virtual viewpoint position and attitude (line of sight direction). The attitude is obtained from the direction in which the object moves within a certain time. Furthermore, the virtual viewpoint determination unit 506 specifies the height of the selected object from the object dimension information, and determines the virtual viewpoint position (three-dimensional coordinates) including the height. Note that it is desirable to set the height of the virtual viewpoint to a value slightly lower than the height of the object dimension information, so as to be closer to the viewpoint of the object. Furthermore, the virtual viewpoint determination unit 506 determines the height using the object dimension information, but this is not limiting, and the height may be a fixed height determined in advance.

[0058] When the terminal device ID of the terminal information acquired from the image storage unit 504 is a stationary display, the virtual viewpoint determination unit 506 determines the player imaging viewpoint (a viewpoint for observing an arbitrary object from behind in the example of FIG. 10(b)) as a specific type of virtual viewpoint. The selection of the object to be observed is the same as when the terminal device ID is a head-mounted display. The virtual viewpoint determination unit 506 specifies the position and moving direction of the selected object from the metadata. The moving direction is obtained from the direction in which the object moves within a certain time. The virtual viewpoint determination unit 506 determines a position that is a predetermined distance away from the reference in the opposite direction to the moving direction with the object position as the reference as the virtual viewpoint position. In addition, the virtual viewpoint determination unit 506 specifies the height of the selected object based on the object dimension information, and determines the virtual viewpoint position (three-dimensional coordinates) including the height. It is desirable to set the height of the virtual viewpoint to a value slightly higher than the height of the object dimension information. In addition, the virtual viewpoint determination unit 506 determines the attitude of the virtual viewpoint so that the line of sight of the virtual viewpoint is directed from the virtual viewpoint position to the object position. It should be noted that the virtual viewpoint determination unit 506 determines the height using object dimension information, but this is not limiting, and the height may be a constant height determined in advance.

[0059] When the terminal device ID of the terminal information acquired from the image storage unit 504 is a table-type display, the virtual viewpoint determination unit 506 determines a viewpoint overlooking the imaging range from directly above as a specific type of virtual viewpoint, as shown in Fig. 10(c). The virtual viewpoint determination unit 506 determines the orientation of the virtual viewpoint so that the virtual viewpoint faces directly below. In addition, the virtual viewpoint determination unit 506 raises the virtual viewpoint vertically upward from the center of the center circle of the soccer field, for example, and determines a position where the entire soccer field is within the imaging range as the virtual viewpoint position. Note that, although the bird's-eye view from directly above has been exemplified as the bird's-eye view, the present invention is not limited to this, and any viewpoint that overlooks the entire imaging range may be used.

[0060] The image acquisition unit 507 selects a virtual viewpoint image from the group of virtual viewpoint images acquired from the image storage unit 504, based on the virtual viewpoint position and orientation acquired from the virtual viewpoint determination unit 506. The image acquisition unit 507 may select a virtual viewpoint image from the group of virtual viewpoint images, based on virtual viewpoint parameters acquired via the input / output unit 509. The image acquisition unit 507 transmits the selected virtual viewpoint image to the display control unit 508.

[0061] When the display control unit 508 receives an operation instruction from the user terminal 203 to start displaying a virtual viewpoint image, it controls the user terminal 203 to display the virtual viewpoint image generated based on the virtual viewpoint position and posture determined by the virtual viewpoint determination unit 506.

[0062] The input / output unit 509 outputs the virtual viewpoint image acquired from the display control unit 508 to the user terminal 203. In addition, the input / output unit 509 receives input from the user terminal 203 regarding operation instructions, virtual viewpoint parameters, and terminal information.

[0063] Next, the flow of processing executed by the information processing device will be described with reference to the drawing. Fig. 11 is a flowchart showing the flow of processing executed by the information processing device 202. S represents a step.

[0064] In S1101, the information processing device 202 acquires various pieces of information.

[0065] In S1102, the information processing device 202 determines the type of virtual viewpoint information based on the device information acquired in S1101.

[0066] In S1103, the information processing device 202 selects a virtual viewpoint image based on the type of virtual viewpoint information determined in S1102. Specifically, the information processing device 202 selects a virtual viewpoint image according to the type of virtual viewpoint information determined from among a plurality of types of virtual viewpoint images.

[0067] In S1104, the information processing device 202 outputs the virtual viewpoint image generated in S1103 to the user terminal 203, and performs control to cause the user terminal 203 to display the virtual viewpoint image.

[0068] As described above, it is possible to determine the type of virtual viewpoint information including at least the virtual viewpoint position and attitude based on device information indicating the type of user terminal 203, generate a virtual viewpoint image according to the determined type of virtual viewpoint information, and perform control to display it on the user terminal 203. Therefore, a virtual viewpoint image suitable for the user terminal 203 can be automatically displayed simply by the user performing an operation to play the virtual viewpoint image on the user terminal 203. This eliminates the need to perform an operation to select the type of virtual viewpoint information, reducing the burden of user operations.

[0069] (Other embodiments) In the above embodiment, the virtual viewpoint determination unit 506 determines one type of virtual viewpoint information based on the received device information, but the present invention is not limited to this. For example, the virtual viewpoint determination unit 506 may determine a plurality of types of virtual viewpoint information as shown in Figs. 10(a) to 10(c), and then add a priority (priority order) to each virtual viewpoint based on the terminal device ID of the device information. The image acquisition unit 507 selects a virtual viewpoint image according to the virtual viewpoint information from the group of virtual viewpoint images acquired from the image storage unit 504 based on each type of virtual viewpoint information determined by the virtual viewpoint determination unit 506. When the display control unit 508 receives an operation instruction from the user terminal 203 to start displaying the virtual viewpoint image, the display control unit 508 controls to display the virtual viewpoint image according to the type of virtual viewpoint information with the highest priority. In addition, when a request to switch to another type of virtual viewpoint image (virtual viewpoint) is received from the user terminal 203, the display control unit 508 controls to display the virtual viewpoint image according to the type of virtual viewpoint information with the next highest priority. By controlling in this way, it becomes possible to display the virtual viewpoint images according to the virtual viewpoint information suitable for the viewing device in order of priority.

[0070] In addition, the virtual viewpoint determination unit 506 has been described as determining a plurality of types of virtual viewpoint information and then adding a priority to each piece of virtual viewpoint information based on the terminal device ID of the device information, but is not limited thereto. For example, the virtual viewpoint determination unit 506 may determine whether the type of virtual viewpoint information is appropriate for the viewing device based on the terminal device ID of the device information, and add the determination result to the virtual viewpoint information. The image acquisition unit 507 selects a virtual viewpoint image from the group of virtual viewpoint images acquired from the image storage unit 504 based on each piece of virtual viewpoint information acquired from the virtual viewpoint determination unit 506. The display control unit 508 controls the plurality of types of virtual viewpoint images acquired from the image acquisition unit 507 to be displayed as thumbnails on the display unit of the user terminal 203. Here, the display of thumbnails will be described with reference to figures. FIG. 12 is a diagram showing an example of thumbnail display. As shown in FIG. 12, thumbnail images 1201, 1202, and 1203 are displayed on the display unit of the user terminal 23. The thumbnail images 1201, 1202, and 1203 are reduced images of virtual viewpoint images generated according to the type of virtual viewpoint information determined by the virtual viewpoint determination unit 506. The selected image 1204 is a virtual viewpoint image of a thumbnail image selected by a user from among a plurality of thumbnail images. The display control unit 508 controls the selected image 1204 to display a warning based on the determination result attached to the virtual viewpoint image. If the determination result indicates that the virtual viewpoint information corresponds to a virtual viewpoint image that is not suitable for the terminal device, the display control unit 508 displays a warning 1205 including a warning message such as "Cannot be selected." By controlling in this manner, it is possible to prevent the user terminal 203 from reproducing (displaying) an inappropriate virtual viewpoint image.

[0071] In the above embodiment, the case of capturing an image of a soccer match is exemplified, but the image capturing subject is not necessarily limited to this. For example, the present embodiment can be applied to capturing images of other sports matches such as rugby, tennis, ice skating, and basketball, as well as live performances and concerts. In addition, in a live performance or concert, it is also possible to use a viewpoint for observing a specific artist as a virtual viewpoint, a viewpoint of a specific artist as a virtual viewpoint, or a viewpoint overlooking the stage where a performance is being performed as a virtual viewpoint.

[0072] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions. [Explanation of symbols]

[0073] 202 Information processing device 505 Information Acquisition Department 508 Display control unit

Claims

1. An acquisition means for acquiring device information indicating a type of viewing device for a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a predetermined area from a plurality of directions; a determination means for determining a position and an orientation of a virtual viewpoint corresponding to the virtual viewpoint image based on the type of the viewing device indicated by the device information; having When the orientation of a display unit of the viewing device of the type indicated by the device information acquired by the acquisition means is landscape orientation that is approximately parallel to a horizontal plane, the determination means determines the position and line of sight of the virtual viewpoint for observing a target object from the surroundings within the predetermined area.

23. An information processing apparatus comprising:

2. The acquisition means acquires detection information indicating an orientation of a display unit of the viewing device, When the orientation indicated by the detection information is a landscape orientation substantially parallel to a horizontal plane, the determining means determines the position and line of sight of the virtual viewpoint for observing a target object from the surroundings within the predetermined area.

2. The information processing apparatus according to claim 1,

3. The present invention further includes a control unit for controlling the display of a virtual viewpoint image on the viewing device, the virtual viewpoint image being generated based on virtual viewpoint information corresponding to a viewpoint position and a line of sight direction for observing a target object in the predetermined area from the surroundings.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. An acquisition means for acquiring device information indicating a type of viewing device for a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a predetermined area from a plurality of directions; a determination means for determining a position and an orientation of a virtual viewpoint corresponding to the virtual viewpoint image based on the type of the viewing device indicated by the device information; having When the orientation of a display unit of the audiovisual device of the type indicated by the device information acquired by the acquisition means is an upward orientation substantially perpendicular to a horizontal plane, the determination means determines the position and line of sight of the virtual viewpoint overlooking the predetermined area.

23. An information processing apparatus comprising:

5. The acquisition means acquires detection information indicating an orientation of a display unit of the viewing device, When the orientation indicated by the detection information is an upward orientation substantially perpendicular to a horizontal plane, the determining means determines the position and line of sight of the virtual viewpoint overlooking the predetermined area.

5. The information processing apparatus according to claim 4.

6. The present invention further includes a control unit for controlling the display of a virtual viewpoint image, which is generated based on virtual viewpoint information corresponding to a viewpoint position and a line of sight direction overlooking the predetermined area, on the viewing device.

6. The information processing apparatus according to claim 4, wherein:

7. The information processing apparatus according to claim 1 , wherein the device information includes an identification code indicating that the type of the audiovisual device is a tablet terminal or a smartphone.

8. further comprising an input means for receiving a user operation via the audiovisual device, When the control means receives the user's operation to reproduce the virtual viewpoint image through the input means, the control means performs control to automatically display the virtual viewpoint image on the viewing device.

7. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

9. The control means controls the viewing device to display a plurality of types of virtual viewpoint images, which are generated based on a plurality of types of virtual viewpoint information, as thumbnails.

7. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

10. The control means performs control to display, on the viewing device, a virtual viewpoint image generated based on virtual viewpoint information having a high priority indicating a priority order for display on the viewing device.

7. The information processing apparatus according to claim 3, wherein the information processing apparatus is a computer.

11. An acquisition step of acquiring device information indicating a type of device for viewing a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a predetermined area from a plurality of directions; determining a position and an orientation of a virtual viewpoint corresponding to the virtual viewpoint image based on the type of the viewing device indicated by the device information; Including, When the orientation of the display unit of the viewing device of the type indicated by the device information acquired in the acquisition step is landscape orientation that is approximately parallel to a horizontal plane, the determination step determines the position and line of sight of the virtual viewpoint for observing a target object from the surroundings within the predetermined area.

23. An information processing method comprising:

12. An acquisition step of acquiring device information indicating a type of device for viewing a virtual viewpoint image generated based on a plurality of captured images obtained by capturing an image of a predetermined area from a plurality of directions; determining a position and an orientation of a virtual viewpoint corresponding to the virtual viewpoint image based on the type of the viewing device indicated by the device information; Including, When the orientation of a display unit of the type of viewing device indicated by the device information acquired in the acquisition step is an upward orientation substantially perpendicular to a horizontal plane, the determination step determines the position and line of sight of the virtual viewpoint overlooking the predetermined area.

23. An information processing method comprising:

13. A program for causing a computer to function as the information processing device according to any one of claims 1 to 10.

Images