Information processing apparatus, information processing method, and program

Abstract

To generate a high-quality virtual viewpoint content.SOLUTION: An information processing apparatus determines an attention area for a range to be photographed by a plurality of photographing devices, and determines the roles of the plurality of photographing devices on the basis of the determined attention area. The information processing apparatus generates a three-dimensional model of a subject used for generation of a virtual viewpoint image, from images photographed by the plurality of photographing devices which are synchronized with each other, on the basis of the determined roles of the photographing devices.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to information processing techniques related to the generation of virtual viewpoint content. [Background technology]

[0002] Recently, a technology that generates virtual viewpoint content using a plurality of images obtained by synchronous shooting, by installing a plurality of shooting devices (hereinafter referred to as cameras) at different positions, has been attracting attention. In a system that generates virtual viewpoint content, a plurality of images synchronously shot by a plurality of cameras are collected in an image generating device such as a server, and the image generating device performs a three-dimensional model generation process, a rendering process, and the like using the plurality of images. For example, the image generating device generates a virtual viewpoint image seen from a specified viewpoint (hereinafter referred to as a virtual viewpoint) and transmits it to an end user terminal. This allows a user of the end user terminal to view, for example, highlight scenes of a rugby game from various directions, and to obtain a higher sense of realism than when viewing the images captured by the cameras as they are.

[0003] However, the quality of these virtual viewpoint images may vary depending on the arrangement of the multiple cameras. For example, the closer the position of the virtual viewpoint is to the arrangement of the actual cameras, the higher the quality of the virtual viewpoint image that can be generated, and conversely, the farther the position of the virtual viewpoint is from the arrangement of the actual cameras, the lower the quality of the virtual viewpoint image. In response to this, Patent Document 1 discloses a technology that makes it possible to suppress the deterioration of the quality of the virtual viewpoint image caused by the arrangement of the multiple cameras by moving the position of the viewpoint specified by the user to a position where a higher quality image can be generated. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-41594 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the case of the technology of Patent Document 1, although it is possible to suppress deterioration in the quality of the virtual viewpoint image, there is a possibility that a virtual viewpoint image from a viewpoint position different from the viewpoint position specified by the user is generated. In other words, it cannot be said that the virtual viewpoint content is necessarily a high-quality image seen from the viewpoint position desired by the user. On the other hand, when a virtual viewpoint image corresponding to the viewpoint position specified by the user is generated, as described above, the virtual viewpoint content may be low in quality depending on the arrangement of the multiple cameras, etc.

[0006] Therefore, an object of the present disclosure is to enable the generation of high-quality virtual viewpoint content. [Means for solving the problem]

[0007] The information processing device disclosed herein is characterized by having an area determination means for determining an area of ​​interest within a range to be photographed by a plurality of photographing devices, a role determination means for determining a role for each of the plurality of photographing devices based on the determined area of ​​interest, and a generation means for generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image from an image photographed synchronously by the plurality of photographing devices, based on the determined roles of the photographing devices. Effect of the Invention

[0008] According to the present disclosure, it becomes possible to generate high-quality virtual viewpoint content. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a content generation system according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a control station. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a control station according to the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a region of interest and a role of a camera in the first embodiment. [Diagram 5] FIG. 2 is a table showing the roles and attention areas of the cameras according to the first embodiment. [Figure 6] 4 is a flowchart showing a flow of information processing according to the first embodiment. [Figure 7] FIG. 11 is a diagram illustrating an example of a functional configuration of a control station according to a second embodiment. [Figure 8] FIG. 11 is an explanatory diagram of a region of interest and a role of a camera in the second embodiment. [Figure 9] FIG. 11 is a table showing the roles of the cameras in the second embodiment. [Figure 10] 10 is a flowchart showing a flow of information processing according to the second embodiment. [Figure 11] FIG. 13 is an explanatory diagram of the role of a camera in a region of interest in the third embodiment. [Figure 12] FIG. 13 is a table showing the roles of cameras in the third embodiment. [Figure 13] 13 is a flowchart of information processing according to the third embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of a functional configuration of a control station according to a fourth embodiment. [Figure 15] FIG. 13 is an explanatory diagram of time and the role of a camera in the fourth embodiment. [Figure 16] 13 is a flowchart showing a flow of information processing according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings. Each of the following embodiments does not limit the present disclosure, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present disclosure. The configuration of the embodiment may be appropriately modified or changed depending on the specifications of the device to which the present disclosure is applied and various conditions (conditions of use, environment of use, etc.). In addition, each of the embodiments described below can be appropriately combined. In each of the following embodiments, the same or similar configurations and processing steps are given the same reference symbols, and duplicated explanations are omitted.

[0011] <First embodiment> In this embodiment, a content generation system is exemplified in which a plurality of imaging devices and microphones are installed in a target facility such as a stadium (a stadium) or a concert hall, and virtual viewpoint content is generated based on image and audio data captured and collected by the imaging devices and microphones. In the following description, the imaging devices are referred to as cameras, microphones as microphones, image data as images, and audio data as audio. In the following description, a stadium where badminton or rugby matches are played is exemplified as a target facility where cameras and microphones are installed.

[0012] 1 is a diagram showing an example of the configuration of a content generation system 100 according to this embodiment. The content generation system 100 includes an image capture system 101, an image generation device 122, a controller 123, a switching hub 121, and a time server 127.

[0013] The photographing system 101 is composed of a set of a plurality of sensor systems (26 sets of sensor systems 110a-110z in the example of FIG. 1). As shown in FIG. 1, the sensor system 110a has a camera 112a, a microphone 111a, a camera platform 113a, an external sensor 114a, and a camera adaptor 120a. Similarly, the sensor systems 110b-110z each have one camera 112b-112z, one microphone 111b-111z, one camera platform 113b-113z, one external sensor 114b-114z, and one camera adaptor 120b-120z. The cameras 112a-112z, the camera platform 113a-113z, and the microphones 111a-111z are arranged at a plurality of positions so that the photographing target range corresponding to the entire surface of a court or field of a stadium or the like, which is a target facility, can be photographed and sound collected from various directions. That is, in the photographing system 101, the cameras 112a-112z and the pan heads 113a-113z photograph each subject from multiple directions within a photographing range such as a stadium, and the microphones 111a-111z collect sounds from multiple directions. The external sensors 114a-114z detect the temperature and vibration of the cameras in the sensor systems in which they are arranged. The camera adapters 120a-120z control the cameras 112a-112z, the pan heads 113a-113z, and the microphones 111a-111z in the sensor systems 110a-110z in which they are arranged, in response to commands sent from the controller 123. Each of the sensor systems 110a-110z is connected to a switching hub 121, and data is transmitted and received between the sensor systems 110 via the switching hub 121, forming a star-shaped network.

[0014] In the content generation system 100 of this embodiment, the sensor systems 110a-110z are connected to the image generation device 122 and the controller 123 via a switching hub 121. The switching hub 121 transmits 26 sets of images and sounds obtained by the sensor systems 110a-110z of the imaging system 101 to the image generation device 122.

[0015] In the following description, unless otherwise specified, the 26 sets of sensor systems 110a-110z will be referred to as sensor system 110 without distinction. Similarly, unless otherwise specified, the configuration within each sensor system 110 will be referred to as microphone 111, camera 112, pan head 113, external sensor 114, and camera adapter 120 without distinction. In FIG. 1, the number of sensor systems 110 is 26 sets, but this is only an example and is not limited to 26 sets. In this embodiment, the term "image" includes the concepts of video and still images unless otherwise specified. In other words, the content generation system 100 of this embodiment can process both still images and video. In this embodiment, the virtual viewpoint content generated by the content generation system 100 includes a virtual viewpoint image and a virtual viewpoint sound, but this is not limited to this, and for example, the virtual viewpoint content may not include sound. Furthermore, the sound included in the virtual viewpoint content may be, for example, sound collected by a microphone closest to the virtual viewpoint. Hereinafter, for the sake of brevity, descriptions relating to audio will be omitted where appropriate, but it is assumed that the content generation system 100 of this embodiment processes both images and audio.

[0016] Further, the sensor system 110 illustrated in FIG. 1 includes one microphone 111, one camera 112, one camera platform 113, one external sensor 114, and one camera adapter 120, but is not limited thereto. For example, the sensor system 110 may include at least one camera adapter 120, one camera 112, or one microphone 111. For example, the sensor system 110 may include one camera adapter 120 and multiple cameras 112, or may include one camera 112 and multiple camera adapters 120. That is, in the photographing system 101, the multiple cameras 112 and the multiple camera adapters 120 correspond to each other in an N to M ratio (N and M are both integers equal to or greater than 1). The sensor system 110 may include devices other than the microphone 111, the camera 112, the camera platform 113, and the camera adapter 120. The camera 112 and the camera adapter 120 may be configured as one unit. Further, each sensor system 110 is not limited to having the same configuration, and may have a different configuration. Although an example of a configuration in which the camera 112 and the camera adapter 120 are separate is shown in Fig. 1, they may be integrated in the same housing. Furthermore, in this case, the microphone 111 may be built in the integrated camera 112, or may be connected to the outside of the camera 112.

[0017] The time server 127 has a function of generating a time and a synchronization signal, and transmits the time and the synchronization signal to each sensor system 110a-110z via the switching hub 121. The camera adapters 120a-120z of the sensor systems 110a-110z that receive the time and the synchronization signal Genlock the cameras 112a-112z based on the time and the synchronization signal to synchronize the image frames. That is, the cameras 112a-112z of the sensor systems 110a-110z capture images at synchronized capture timings, thereby acquiring a plurality of synchronized captured images (i.e., multiple viewpoint images). The image generating device 122 described later generates a three-dimensional model based on the multiple viewpoint images captured at the synchronized timings, generates a virtual viewpoint image using the three-dimensional model, and so on, thereby suppressing deterioration in the quality of the virtual viewpoint image due to a difference in capture timing. In this embodiment, an example is given in which the time server 127 manages the time synchronization of each camera 112a-112z, but this is not limited thereto, and the processing for time synchronization may be performed independently by each camera 112a-112z or each camera adapter 120a-120z.

[0018] The controller 123 includes a control station 124 and a virtual camera operation UI 125 . The virtual camera operation UI 125 is a UI generating unit that generates a user interface for acquiring an instruction from a user. The virtual camera operation UI 125 acquires information on a virtual viewpoint from a user as user input information, and transmits it to an image generating device 122 described later. In this embodiment, the user is assumed to be an operator (system operator) who operates the content generation system 100 in an overall manner, or an operator (virtual viewpoint operator) who specifies a virtual viewpoint in the system. In this embodiment, the user who operates the virtual camera operation UI 125 is assumed to be an operator (virtual viewpoint operator) who specifies a virtual viewpoint. Note that, in this embodiment, an example is given in which a virtual viewpoint operator specifies a virtual viewpoint, but the specification of a virtual viewpoint may be performed by a user (end user) of an end user terminal 126 described later. When a user of the end user terminal 126 specifies a virtual viewpoint, a virtual camera operation UI is formed on the end user terminal 126.

[0019] The control station 124 controls the entire image capture system 101 connected via the switching hub 121. Although details will be described later, the control station 124 executes various processes such as determining an area of ​​interest, determining a role of the camera according to the area of ​​interest, and controlling the settings of the camera 112 and the camera platform 113 according to the role of the camera, as part of information processing related to the information processing device of this embodiment. The control station 124 also manages the operating state and controls parameter settings for each component of the content generation system 100 through networks 180a-180z and 190a-190c. The network may be GbE (Gigabit Ethernet) or 10GbE that conforms to the IEEE standard, which is Ethernet (registered trademark), or may be configured by combining interconnect Infiniband, industrial Ethernet, etc. The network is not limited to these examples and may be another type of network.

[0020] In addition, when the camera 112, the camera head 113, the microphone 111, etc. are installed in a target facility such as a stadium, the control station 124 also performs a calibration process for the camera 112, the camera head 113, the microphone 111, etc. When the camera 112 and the camera head 113 are calibrated, for example, markers are installed at multiple locations on the court or field, which is the shooting target range in the stadium, and each camera 112 shoots including the markers. Then, the control station 124 calculates the position, orientation, and focal length of each camera 112 in the world coordinates based on the image captured by each camera 112 including the markers, and performs a calibration process based on the information. Information on the position, orientation, and focal length of each camera calculated by the control station 124 is sent to the image generation device 122 described later and used when generating a three-dimensional model. In addition, the control station 124 transmits an image captured in advance by each camera 112 of the shooting target range such as the court or field of the stadium to the image generation device 122 as a background image.

[0021] The image generating device 122, as part of information processing related to the information processing device of this embodiment, generates a three-dimensional model based on the role of the camera determined by the control station 124, the captured image of each camera 112, and information on the position, orientation, and focal length of each camera 112. Furthermore, the image generating device 122 generates a virtual viewpoint image seen from a virtual viewpoint specified by a user, using the three-dimensional model and a background image prepared in advance. Then, the image generating device 122 transmits the generated virtual viewpoint image to the end user terminal 126. This allows the user of the end user terminal 126 to view (and listen to) the virtual viewpoint image seen from the virtual viewpoint.

[0022] In the present embodiment, an example is given in which the virtual viewpoint content includes sound data (audio data), but the virtual viewpoint content does not necessarily include sound data. The image generating device 122 may compress and encode the virtual viewpoint image using a standard technology such as H.264 or HEVC, and then transmit the image to the end user terminal 126 using the MPEG-DASH protocol. The image transmission protocol is not limited to MPEG-DASH, and may be, for example, HLS (HTTP Live Streaming) or other transmission methods. The virtual viewpoint image may be transmitted to the end user terminal 126 in an uncompressed state. In the case where the end user terminal 126 is compressed and encoded, a smartphone or a tablet terminal is assumed, and in the case where the end user terminal 126 is uncompressed, a display capable of displaying uncompressed images is assumed. That is, the content generating system 100 according to the present embodiment is capable of switching the image format according to the type of the end user terminal 126.

[0023] Fig. 2 is a diagram showing an example of a hardware configuration applicable to the control station 124 and the image generation device 122, which are components of the content generation system 100 shown in Fig. 1. Note that the hardware configuration of the control station 124 will be described as an example here. The control station 124 includes a central processing unit (CPU) 201, a random access memory (RAM) 202, a read only memory (ROM) 203, and a network interface 204. The CPU 201 executes various arithmetic processes and various processing programs using a control program stored in the ROM 203. The various processing programs include an information processing program according to this embodiment, and the CPU 201 executes the information processing program to realize each function of the control station 124 shown in FIG. 3, which will be described later. The control station 124 may have one or more dedicated hardware pieces different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0024] The RAM 202 is used as a main memory for the CPU 201 and as a temporary storage area such as a work area. The ROM 203 has a storage area for storing a boot program, a control program, an information processing program for implementing each functional unit of a control station 124 shown in FIG. 3 described later, parameters to be set for each functional unit of the control station 124, and the like. The network interface 204 is an interface unit for performing communication between the control station 124 and peripheral devices. The control station 124 transmits and receives various information via the network interface 204. Note that the network connected to the network interface 204 may be a wired network such as a LAN (Local Area Network) or a wireless network. Each component that constitutes the control station 124 is connected by an internal bus 205, and transmits and receives data via the internal bus 205.

[0025] FIG. 3 is a functional block diagram showing the functional configuration of the control station 124. The control unit 301 is a processing unit that controls the order of each information processing executed in the control station 124 . The area determination unit 302 performs a process of determining an attention area for a shooting target range, which is a range where shooting can be performed by the multiple cameras 112 of the shooting system 101. In the case of the first embodiment, the attention area is determined as an area for shooting an image to be used for generating a three-dimensional model. In the first embodiment, when a shooting area of ​​an image to be used for generating a three-dimensional model is specified by a user for a shooting target range such as a stadium, the area determination unit 302 determines the attention area based on the shooting area specified by the user. Note that the user in this case is assumed to be a system operator, but may be a virtual viewpoint operator. Details of the attention area determination process performed by the area determination unit 302 in the first embodiment will be described later.

[0026] The role determination unit 303 determines the role of each camera in the multiple sensor systems 110 according to the region of interest determined by the region determination unit 302. Determining the role of the camera means assigning the region to be photographed by each camera 112, including the setting of the photographing direction by the camera platform 113, and further assigning how the photographed image of each camera 112 is to be handled in the image generating device 122. In the case of the first embodiment, the camera role determination determines which camera 112 will photograph the region of interest, and further determines that the photographed image of each camera 112 will be used to generate a three-dimensional model in the image generating device 122. Details of the camera role determination process performed by the role determination unit 303 in the first embodiment will be described later.

[0027] The setting control unit 304 controls the settings of each camera 112 and the setting of the shooting direction by the camera platform 113 according to the role of the camera determined by the role determination unit 303. For example, the setting control unit 304 controls the camera settings such as the zoom ratio (shooting angle of view) of each camera 112, and the setting of the shooting direction of the camera 112 by the angle setting of the camera platform 113. The setting control unit 304 then transmits camera setting information, which includes information indicating the role of the camera in addition to information indicating the settings of the camera 112 and the settings of the camera platform 113, to the image generating device 122.

[0028] The operation input unit 305 performs a process of acquiring information (user input information) according to a user's operation. The user who operates the operation input unit 305 is assumed to be the above-mentioned system operator or virtual viewpoint operator. For example, when the operation input unit 305 acquires information designating a shooting area of ​​an image used for generating a three-dimensional model from the system operator as user input information, the operation input unit 305 sends the information designating the shooting area to the area determination unit 302. For example, when the operation input unit 305 acquires virtual viewpoint information indicating the position and direction of a virtual viewpoint from a virtual viewpoint operator through the virtual camera operation UI 125 as user input information, the operation input unit 305 sends the virtual viewpoint information to the image generation device 122 via the control unit 301.

[0029] As described above, the image generating device 122 generates a three-dimensional model based on the images captured by each camera 112 and the camera setting information including information indicating the role of the camera. The image generating device 122 also generates a virtual viewpoint image according to a virtual viewpoint specified by the user using the three-dimensional model and a background image prepared in advance. The image generating device 122 then transmits the generated virtual viewpoint image to the end user terminal 126.

[0030] 4(A) and 4(B) are diagrams used to explain the determination process of the region of interest performed by the region determination unit 302 of the first embodiment. In Fig. 4(A) and Fig. 4(B), an example is given in which there are two courts within the shooting range of a badminton match venue, and a badminton match is shot using these courts. In Fig. 4(A) and Fig. 4(B), cameras 401a-401n indicate some of the cameras 112 (14 in this example) including the pan head 113 described above, and the dotted lines extending from each camera 112 indicate the shooting direction. In addition, players 402-413 are players playing a badminton match, and are the main subjects of the cameras 401a-401n.

[0031] FIG. 4A shows an example in which the entire two courts set up in a badminton match venue, which is the range to be photographed, are specified by the user as the photographing range. As shown in FIG. 4A, when the entire match venue, which is the range to be photographed, is specified by the user as the photographing range of an image for generating a three-dimensional model, the region determination unit 302 determines the regions 420 and 430 corresponding to the two courts as the attention regions, respectively. That is, the region determination unit 302 determines the regions 420 and 430 obtained by dividing the photographing range in correspondence with the two courts as the attention regions, respectively. Note that, although two courts are given as an example in FIG. 4A, for example, when four courts are set up in the venue, the four regions corresponding to each of the courts are determined as the attention regions, respectively. Also, for example, when one field is the range to be photographed, such as a soccer or rugby match venue, and the photographing range is specified by the user as the photographing range, the region determination unit 302 determines the entire field as the attention region.

[0032] On the other hand, Fig. 4(B) shows an example in which only a part of the badminton match venue (for example, one court), which is the range to be photographed, is specified by the user as the photographing area. In the example of Fig. 4(B), the area determination unit 302 determines the area 440 corresponding to the court specified by the user as the photographing area as the attention area. Also, for example, in the case of a match being played on one field such as soccer or rugby, if the area in front of one of the goals installed in the field is specified as the photographing area, the area determination unit 302 determines the specified area in front of the goal as the attention area.

[0033] As described above, the region determination unit 302 determines the region of interest according to the shooting region designated by the user. Therefore, when the shooting region is changed from the entire shooting range in Fig. 4(A) to a part as in Fig. 4(B) or conversely, when the shooting region is changed from a part to the entire shooting range, the region determination unit 302 switches the region of interest according to the designated switching of the shooting region. Then, when the region determination unit 302 switches the region of interest, the region determination unit 302 determines the roles of each of the multiple cameras so as to correspond to the switched region of interest.

[0034] FIG. 5 is a diagram used to explain the camera role determination process performed by the role determination section 303 based on the attention area determined by the area determination section 302. In FIG. Table 501 shown in Fig. 5(A) shows the roles of the cameras 401a-401n that capture the regions 420 and 430 shown in Fig. 4(A) when the regions of interest determined by the region determination unit 302 are the regions 420 and 430 shown in Fig. 4(A). Table 502 shown in Fig. 5(B) shows the roles of the cameras 401a-401n that capture the region 440 shown in Fig. 4(B) when the region of interest determined by the region determination unit 302 is the region 440 shown in Fig. 4(B). In Tables 501 and 502, the camera number columns 401a-401n correspond to the cameras 401a-401n in Fig. 4(A) and 4(B), respectively. In addition, the capture target columns 420-440 correspond to the regions 420-440 in Fig. 4(A) and 4(B), respectively. Additionally, G1-G3 in the photography group column each represent a photography group, and indicate that a three-dimensional model will be generated using images captured by a camera belonging to that photography group.

[0035] For example, when the attention areas determined by the area determination unit 302 are the areas 420 and 430 in Fig. 4(A), the role determination unit 303 determines the role of each camera as shown in table 501 in Fig. 5(A). That is, the role determination unit 303 determines the role of cameras 401a and 401j-401n as the shooting group G1 to shoot the area 420, and determines the role of cameras 401b-401h as the shooting group G2 to shoot the area 430. As a result, in the image generation device 122, a three-dimensional model is generated based on the images of the area 420 captured by the cameras belonging to the shooting group G1, and a three-dimensional model is generated based on the images of the area 430 captured by the cameras belonging to the shooting group G2.

[0036] On the other hand, for example, when the attention area determined by the area determination unit 302 is the area 440 in Fig. 4(B), the role determination unit 303 determines the role of each camera as shown in table 502 in Fig. 5(B). That is, the role determination unit 303 determines the role of the cameras 401a-401n as the shooting group G3 to shoot the area 440. That is, in this case, the role determination unit 303 determines the role of all the cameras 401a-401n to be the role of concentrating on shooting one area 440. As a result, in the image generation device 122, a three-dimensional model is generated based on the images of the area 440 shot by each camera belonging to the shooting group G3.

[0037] As described above, the role determining section 303 can switch the role of each camera from table 501 shown in FIG. 5 to table 502, or vice versa, based on the region of interest determined by the region determining section 302.

[0038] Fig. 6 is a flowchart showing the flow of information processing executed by the control station 124 according to the first embodiment. The control station 124 of this embodiment performs information processing such as determining an attention area according to an imaging area designated by a user, determining the role of each camera 112 based on the attention area, and changing the settings of each camera 112 and controlling the camera platform 113. Note that the explanation of the flowchart in Fig. 6 will be given by taking as an example a case where the imaging area shown in Fig. 4(A) is designated in advance and then switched to the imaging area shown in Fig. 4(B) by a user instruction.

[0039] First, in step S601, the area determination unit 302 waits to receive a request to specify the shooting area of ​​an image to be used in generating a three-dimensional model. When a request to specify the shooting area is input from the user via the operation input unit 305, the processing proceeds to step S602.

[0040] When the process proceeds to step S602, the area determination unit 302 determines whether or not the shooting area needs to be changed in response to the user's request. Here, an example is given in which the shooting area shown in Fig. 4(A) has already been specified, so in step S602, the area determination unit 302 determines whether or not the shooting area needs to be changed (switched) from the shooting area shown in Fig. 4(A) to the shooting area shown in Fig. 4(B). If the area determination unit 302 determines that the shooting area does not need to be changed, the process returns to step S601, whereas if the area determination unit 302 determines that the shooting area needs to be changed, the process proceeds to step S603.

[0041] When the process proceeds to step S603, the region determination unit 302 determines the region of interest based on the information of the shooting region designated by the user in step S601. That is, in this example, the region determination unit 302 changes the region of interest from the region 420 and the region 430 shown in Fig. 4(A) to the region 440 shown in Fig. 4(B).

[0042] Next, in step S604, the role determination unit 303 determines the role of the camera based on the attention area determined by the area determination unit 302. That is, in this example, the role determination unit 303 changes (i.e. updates) the role of each camera from the role determined to capture the attention areas of areas 420 and 430 illustrated in Fig. 4(A) to the role of capturing the attention area of ​​area 440 illustrated in Fig. 4(B).

[0043] Next, in step S605, setting control unit 304 changes the settings of each camera and controls the head (i.e., controls the shooting direction of the camera) so that shooting is performed by each camera, based on the role information of the camera determined by role determination unit 303. That is, in this example, setting control unit 304 changes the camera setting values ​​including the zoom ratio of each camera and the setting values ​​such as the angle of the head from the setting values ​​for shooting areas 420 and 430 in Fig. 4(A) to the setting values ​​for shooting area 440 in Fig. 4(B).

[0044] Then, after the camera setting change and the pan head control are performed in step S605, in step S606, the control unit 301 transmits camera setting information including information indicating the role of the camera in addition to information indicating the settings of the camera and the angle of the pan head, to the image generating device 122. This allows the image generating device 122 to generate a three-dimensional model based on the camera setting information including the role information of the camera and the captured images of each camera 112.

[0045] As described above, in the content generation system 100 of the first embodiment, when a shooting area is specified by a user, a region of interest corresponding to the shooting area is determined, and further, a role of a camera suitable for shooting the region of interest is determined. In other words, in this embodiment, by appropriately changing the usage method of each camera of a group of multiple cameras according to the region of interest, the multiple cameras can be used efficiently, and a suitable three-dimensional model can be generated according to the user's specification. That is, according to this embodiment, a system is constructed that generates high-quality virtual viewpoint content adapted to the user's specification.

[0046] <Second embodiment> Next, as a second embodiment, an example will be described in which the situation of a subject within a shooting range is detected, an attention area according to the detected situation of the subject is determined, and the role of the camera is determined according to the attention area determined. In the following description, the situation of a subject within a shooting range is referred to as a "scene". The configuration of the content generation system 100 according to the second embodiment is the same as that in FIG. 1, and the hardware configuration of the control station 124 is the same as that in FIG. 2, so that illustration and description thereof will be omitted. Note that in the second embodiment, a case will be described in which the field where a rugby match is being played is taken as an example.

[0047] Fig. 7 is a functional block diagram showing each functional configuration in the control station 124 of the second embodiment. In the functional block diagram of Fig. 7, blocks that are roughly similar to the functional blocks shown in Fig. 3 are given the same reference numbers as in the example of Fig. 3, and their explanation will be omitted as appropriate. Below, functions and processes that are different from the example of the first embodiment will be explained.

[0048] The image receiving unit 701 receives an image from an external source. In this embodiment, the image received by the image receiving unit 701 is, for example, an overhead image acquired by an overhead camera (not shown) that captures an overhead image of the entire shooting range (the entire field in the case of a rugby match). Note that the overhead image of the entire shooting range is not limited to an image captured by an overhead camera, and may be, for example, a virtual viewpoint image generated based on a virtual viewpoint at a position that overlooks the entire shooting range.

[0049] The scene detection unit 702 detects a scene representing the situation of the subject from the overhead image received by the image receiving unit 701. In the case of a rugby match exemplified in this embodiment, the scene detection unit 702 detects from the overhead image a scene in which the players (rugby players) who are the subject are crowded together, and conversely, a scene in which the players are spread out and not crowded together. Then, the scene detection unit 702 sends information representing the detected scene (hereinafter referred to as scene information) to the region determination unit 302.

[0050] When the region determining unit 302 of the second embodiment receives scene information from the scene detecting unit 702, it determines a region of interest based on the scene information. When the region determination unit 302 of the second embodiment receives scene information indicating that players are crowded, it determines the region in the field of the shooting range where the players are crowded as a region of interest for directly using the image captured by the camera. Also, the region determination unit 302 determines the remaining region in the field excluding the region in which the players are crowded as a region of interest for capturing an image for generating a three-dimensional model. On the other hand, when the area determination unit 302 receives scene information indicating that the players are scattered across the entire field and not crowded together, it determines the entire field within the shooting range as the area of ​​interest for capturing images for generating a three-dimensional model. Then, the area determining section 302 sends information on the attention area determined according to the scene to the role determining section 303. The role determination unit 303 in the second embodiment determines the role of each camera 112 based on the information on the attention area determined according to the scene as described above.

[0051] 8(A) and 8(B) are diagrams used to explain the determination of the attention area and the role of the camera in the second embodiment. FIG. 8(A) is a diagram used to explain an example in which the entire field of the shooting range is determined as the attention area for taking an image for generating a three-dimensional model, and the role of the camera is determined based on the attention area. FIG. 8(B) is a diagram used to explain an example in which an area where subjects are densely packed is determined as the attention area for using the captured image as it is, and the other areas are determined as the attention area for taking an image for generating a three-dimensional model, and the role of the camera is determined according to the attention areas. Note that FIG. 8(A) and FIG. 8(B) show an example in which a rugby match is shot.

[0052] In Figures 8(A) and 8(B), cameras 801a-801r indicate some (18 in this example) of the cameras 112 including the above-mentioned pan head 113, and the dotted lines extending from each camera indicate the shooting direction. In Figures 8(A) and 8(B), 13 players 802 are players playing a rugby match, and are the main subjects of the cameras 801a-801r. The example in Figure 8(A) shows a scene in which the players 802 are scattered across the entire field of a rugby match, and the example in Figure 8(B) shows a scene in which the players 802 are concentrated in a certain area of ​​the field.

[0053] In the case of a scene in which the players 802 are scattered all over the field as in FIG. 8(A), the region determination unit 302 determines the entire field of the shooting target range as the attention region for capturing images for generating a three-dimensional model. Then, the role determination unit 303 determines all the cameras 801a-801r as cameras with the role of capturing images to be used for generating a three-dimensional model. In this way, in a scene in which the players 802 are scattered, the reason why all the cameras 801a-801r are determined as cameras with the role of capturing images to be used for generating a three-dimensional model is to generate a high-quality three-dimensional model. That is, in order to generate a high-quality three-dimensional model, it is necessary to capture each player 802 from multiple directions, and in the case of a scene in which the players 802 are scattered as in FIG. 8(A), it is highly likely that each player 802 can be captured from multiple directions. For this reason, the role determination unit 303 determines all the cameras 801a-801r as cameras with the role of capturing images for generating a three-dimensional model.

[0054] On the other hand, in the case of a scene in which the players 802 are crowded in a certain area as shown in FIG. 8B, the area determination unit 302 determines the crowded area as an attention area using the captured image as it is. As described above, in order to generate a high-quality three-dimensional model, it is necessary to capture each player 802 from multiple directions. However, when the players 802 are crowded, it is highly likely that images of the individual players 802 captured from multiple directions cannot be obtained. That is, in an area where the players are crowded, the players 802 on the inside are often hidden by the players 802 existing on the outside, and it is often impossible to capture the players 802 from multiple directions. If a three-dimensional model is generated based on an image of the players 802 in the crowded state, the quality of the model is highly likely to be low. For this reason, in the case of this embodiment, the area determination unit 302 determines the area where the players 802 are crowded as an attention area using the captured image as it is. Moreover, the area determination unit 302 determines the remaining area excluding the crowded area as an attention area for capturing an image for generating a three-dimensional model. In other words, since there is a high possibility that individual players 802 can be photographed from multiple directions in the remaining areas excluding the crowded areas, the area determination unit 302 determines these remaining areas as areas of interest for capturing images for generating a three-dimensional model.

[0055] 8(B), the area surrounded by the dotted circle 803 is a player-dense area, and the area determination unit 302 determines the player-dense area surrounded by the dotted circle 803 as a region of interest for using the captured image as is. The area determination unit 302 also determines the remaining area excluding the player-dense area surrounded by the dotted circle 803 as a region of interest for capturing an image for generating a three-dimensional model.

[0056] Then, the role determination unit 303 determines the role of the cameras 801a-801r based on the attention area for using the captured image as it is and the attention area for capturing an image to be used for generating a three-dimensional model. In the example of FIG. 8(B), the cameras 801a, 801e, 801i, and 801o are assumed to be cameras that have been determined to have the role of capturing images of the players 802 crowded together in the attention area of ​​the dotted circle 803, that is, the role of capturing images to be used as they are. In this way, by using the captured image of the area where the players 802 are crowded as it is, it is possible to suppress the overall deterioration of image quality. In addition, the role determination unit 303 sets each camera other than the cameras 801a, 801e, 801i, and 801o as a camera that has the role of capturing images for generating a three-dimensional model.

[0057] As described above, in the second embodiment, the attention area is determined based on the scene that represents the situation of the subject. Therefore, for example, when the scene changes from the scene shown in Fig. 8(A) to the scene shown in Fig. 8(B) or vice versa, the attention area is changed according to the scene change, and the camera settings are also changed.

[0058] FIG. 9 is a diagram used to explain the camera role determination process performed by the role determining unit 303 in the second embodiment. Table 901 shown in FIG. 9(A) shows the roles of each camera 801a-901r when the attention area is determined according to a scene in which there is no area crowded with players 802 as shown in FIG. 8(A). On the other hand, table 902 shown in FIG. 9(B) shows the roles of each camera 801a-901r when the attention area is determined according to a scene in which there is an area crowded with players as shown in FIG. 8(B). Note that 801a-801r in the camera number column in tables 901 and 902 correspond to each camera 801a-801r in FIG. 8(A) and FIG. 8(B), respectively. In addition, the "3D model generation" and "use of camera captured image" in the camera role column in tables 901 and 902 show whether the image captured by the camera is used to generate a 3D model or the captured image is used as it is.

[0059] That is, when there is no area crowded with players as in the example of Figure 8(A), the role determination unit 303 sets the cameras 801a-801r in Figure 8(A) to the role of capturing images for generating a three-dimensional model, as shown in table 901. 8(B), the role determination unit 303 determines the role of some of the cameras 801a, 801e, 801i, and 801o that capture the player-dense areas to be to use the captured images as they are, as shown in table 902. In addition, the role determination unit 303 determines the role of the remaining cameras, excluding the cameras 801a, 801e, 801i, and 801o that capture the player-dense areas, to be to capture images to be used in generating a three-dimensional model.

[0060] As described above, the role determination unit 303 in the second embodiment switches the role of the camera from table 901 illustrated in FIG. 9 to table 902, or vice versa, based on the area of ​​interest determined by the area determination unit 302.

[0061] Fig. 10 is a flowchart showing the flow of information processing executed in the control station 124 of the second embodiment shown in Fig. 7. Here, the flow of determining the area of ​​interest and the role of the camera will be described using as an example a case where the scene changes from the scene in Fig. 8(A) to the scene in Fig. 8(B) described above.

[0062] First, in step S1001, the image receiving unit 701 receives an overhead image captured by the overhead camera as described above. Next, in step S1002, the scene detection unit 702 performs scene detection as described above by analyzing the overhead image received by the image receiving unit 701. In this example, since the case where the scene changes from Fig. 8(A) to Fig. 8(B) is taken as an example, in step S1002 the scene detection unit 702 detects a change from the scene in Fig. 8(A) to the scene in Fig. 8(B).

[0063] Next, in step S1003, the control unit 301 judges whether or not it is necessary to change the role of the camera based on the information of the scene detected by the scene detection unit 702. That is, it judges whether or not the role of the camera determined at the present time is an appropriate role for the detected scene. Then, in step S1003, if the control unit 301 judges that it is not necessary to change the role of the camera, it returns the process to step S1001, whereas, if it judges that it is necessary to change the role of the camera, it advances the process to step S1004. Note that in this example, when the judgment process in step S1003 is performed, the role of each camera 801a-801r at the present time is to capture an image for generating a three-dimensional model. On the other hand, when the scene of FIG. 8(B) is detected in step S1002, the role of each camera 801a-801r at the present time is not an appropriate role for the detected scene. Therefore, in this example, the control unit 301 advances the process to step S1004.

[0064] When the process proceeds to step S1004, the area determination unit 302 determines an area of ​​interest based on the scene information received from the scene detection unit 702. In this example, the area determination unit 302 determines the area where the players 802 are densely located in Fig. 8(B) as an area of ​​interest for directly using the image captured by the camera, and determines the remaining areas as areas of interest for capturing images for generating a three-dimensional model.

[0065] Next, in step S1005, the role determination unit 303 determines (updates) the role of the camera based on the attention area determined by the area determination unit 302. In this example, the role determination unit 303 determines a camera whose role is to use the captured image of the area where the players 802 are crowded in FIG. 8(B) as it is, and a camera whose role is to use the captured image of the remaining area excluding the crowded area for generating a three-dimensional model. In this example, before the scene in FIG. 8(B) is detected, it is the scene in FIG. 8(A), and the entire field of the shooting target range has already been determined as the attention area for generating an image of a three-dimensional model. For this reason, in step S1005, the area determination unit 302 may determine the role of using the captured image as it is only for the camera that captures the attention area where the players 802 are crowded.

[0066] Next, in step S1006, setting control unit 304 changes the settings of each camera and controls the tripod head (controls the shooting direction of the camera) so that shooting is performed by each camera, based on the role information of the camera determined by role determination unit 303. In this example, setting control unit 304 changes the camera setting values ​​including the zoom ratio of each camera and the setting values ​​such as the angle of the tripod head from the setting values ​​corresponding to the scene in Fig. 8(A) to the setting values ​​for shooting according to the scene in Fig. 8(B).

[0067] Then, after the camera setting change and the pan head control are performed in step S1006, in step S1007, the control unit 301 transmits camera setting information including information indicating the role of the camera in addition to information indicating the settings of the camera and the pan head angle, etc., to the image generating device 122. This allows the image generating device 122 to generate a three-dimensional model based on the camera setting information including the role information of the camera and the images captured by each camera 112, and to generate a virtual viewpoint image using the three-dimensional model. However, in the case of the second embodiment, for the area where players 802 are densely packed shown in FIG. 8(B), the images captured by cameras 801a, 801e, 801i, and 801o in FIG. 8(B) are used as they are.

[0068] Furthermore, as shown in Fig. 8(B), it is desirable for the setting control unit 304 to set the shooting direction and zoom ratio of the camera that shoots a dense area of ​​players 802 where it is difficult to generate a high-quality three-dimensional model, so that the dense area fits as large as possible on the entire screen. Of course, the camera settings and the control of the camera head are not limited to this, and for example, a method of switching settings depending on the dense area or the number of cameras shooting the dense area may be used. Also, for example, the camera settings may be determined according to predetermined rules, or the settings may be dynamically changed based on the captured image.

[0069] As described above, in the content generation system 100 of the second embodiment, scene detection is performed, an attention area is determined according to the detected scene, and the role of the camera is set according to the attention area to perform shooting. In this way, according to the second embodiment, the usage method of the group of multiple cameras can be appropriately changed according to the scene. For example, in a scene where it is difficult to generate a high-quality three-dimensional model, such as a state where subjects are densely packed, appropriate shooting according to the situation and generation of a virtual viewpoint image can be realized by using the image of the dense area captured by some cameras as it is. That is, according to the second embodiment, appropriate shooting according to the situation of the subject and efficient use of multiple cameras are possible, and it is possible to build a system that generates high-quality virtual viewpoint content adapted to the situation of the subject. The information processing according to the first embodiment and the information processing according to the second embodiment may be combined.

[0070] <Third embodiment> Next, as a third embodiment, an example will be described in which, when a user specifies a subject of interest, the subject of interest is photographed by a specific camera and the photographed image is used as is to provide an image with a higher resolution. The configuration of the content generation system 100 of the third embodiment is the same as that of FIG. 1, the hardware configuration of the control station 124 is the same as that of FIG. 2, and the functional block configuration of the control station 124 is the same as that of FIG. 3, so that illustration and description thereof will be omitted. Note that, in the third embodiment, an example will be described in which a field where a rugby match is being played is photographed.

[0071] In the third embodiment, when user input information specifying a subject of interest is sent via operation input unit 305, region determination unit 302 determines the region of the specified subject of interest as a region of interest for directly using the captured image. Furthermore, region determination unit 302 determines the remaining region excluding the region of the subject of interest specified by the user as a region of interest for capturing an image for generating a three-dimensional model. Then, region determination unit 302 sends information on these regions of interest to role determination unit 303. The role determining unit 303 in the third embodiment determines the role of each camera 112 based on the information of the attention area determined by the area determining unit 302 .

[0072] 11(A) and 11(B) are diagrams used to explain the determination of the attention area and the role of the camera in the third embodiment. FIG. 11(A) is a diagram used to explain an example in which the entire object to be photographed is determined as the attention area for photographing an image for generating a three-dimensional model in the third embodiment, and the role of the camera is determined based on the attention area. On the other hand, FIG. 11(B) is a diagram used to explain an example in which the area of ​​the attention subject designated by the user is determined as the attention area for using the photographed image as it is, and the other areas are determined as the attention area for photographing an image for generating a three-dimensional model, and the role of the camera is determined according to the attention area. Note that FIGS. 11(A) and 11(B) show an example in which a rugby match is photographed.

[0073] 11(A) and 11(B), cameras 1101a-1101r indicate some (18 in this example) of cameras 112 including the above-mentioned pan head 113, and dotted lines extending from each camera indicate the shooting direction. Also, in Fig. 11(A) and 11(B), 13 players 1102 are players playing a rugby match, and are the main subjects of cameras 1101a-1101r. Also, in Fig. 11(B), player 1103 is a subject of interest designated by the user, and the area of ​​player 1103 (subject of interest) is the area of ​​interest in which the captured image is used as is.

[0074] 11(A), when none of the players 1102 are designated as the target subject, the area determining unit 302 determines the entire field of the shooting range as the target area for capturing images for generating a three-dimensional model. Then, the role determining unit 303 determines all of the cameras 1101a-1101r as cameras with the role of capturing images to be used in generating a three-dimensional model.

[0075] 11B, when a certain player 1103 is designated as a target subject, the area determination unit 302 determines the area of ​​the target subject as an area of ​​interest that uses the captured image as is. The area determination unit 302 also determines the remaining area excluding the target subject player 1103 as an area of ​​interest for capturing an image for generating a three-dimensional model.

[0076] In the example of FIG. 11B, the role determination unit 303 determines the roles of the cameras 1101a-1101r based on the attention area for using the captured image as it is and the attention area for capturing an image to be used for generating a three-dimensional model. In the example of FIG. 11B, it is assumed that the cameras 111a, 1101g, and 1101m are determined as cameras that use the captured image of the player 1103, who is the subject of interest, as it is. That is, for the player 1103 designated as the subject of interest, images of higher resolution can be provided by using the images captured by the cameras 1101a, 1101g, and 1101m as they are. In addition, the role determination unit 303 sets each of the cameras other than the cameras 1101a, 1101g, and 1101m as cameras that capture images for generating a three-dimensional model.

[0077] In the third embodiment, the area determination unit 302 tracks the area of ​​the player 1103 as the subject of interest even if the player 1103 moves, and continues to determine the area as the area of ​​interest in which the captured image is used as is. Similarly, the role determination unit 303 sets the role of the camera to track the player 1103 as the subject of interest. However, it is also assumed that the camera capable of capturing the player 1103 as the subject of interest moves may change. In this case, the role determination unit 303 appropriately selects a camera capable of tracking the player 1103 as the subject of interest from among the cameras 1101a-1101r, and determines the role of the selected camera to be the role of using the captured image as is.

[0078] FIG. 12 is a diagram used to explain the camera role determination process performed by the role determination unit 303 in the third embodiment. Table 1201 shown in FIG. 12(A) shows the roles of the cameras 1101a-1101r when no target subject is specified as shown in FIG. 11(A). On the other hand, table 1202 shown in FIG. 12(B) shows the roles of the cameras 1101a-1101r when any player is specified as a target subject as shown in FIG. 11(B). Note that 1101a-1101r in the camera number column in tables 1201 and 1202 correspond to the cameras 1101a-1101r in FIG. 11(A) and FIG. 11(B), respectively. In addition, the "3D model generation" and "use of camera captured image" in the camera role column in tables 1201 and 1202 indicate whether the captured image is used to generate a 3D model or is used as it is.

[0079] As in the example of Figure 11(A), when none of the players 1102 are designated as the subject of interest, the role determination unit 303 sets the cameras 1101a-1101r in Figure 11(A) to the role of capturing images for generating a three-dimensional model, as shown in table 1201. 11(B), when the player 1103 is designated as the subject of interest, the role determination unit 303 determines that some of the cameras 1101a, 1101g, and 1101m have a role in which the captured image of the player 1103 is used as is, as shown in table 1202. Moreover, the role determination unit 303 determines that the remaining cameras, excluding the cameras 1101a, 1101g, and 1101m that have a role in capturing images of the player 1103, have a role in which they capture images to be used in generating a three-dimensional model.

[0080] As described above, the role determination unit 303 of the third embodiment switches the role of the camera from table 1201 illustrated in FIG. 12 to table 1202, or vice versa, depending on whether or not a subject of interest has been specified by the user.

[0081] Fig. 13 is a flowchart showing the flow of information processing executed in the control station 124 of the third embodiment. Here, the flow of determining the attention area and the role of the camera will be described by taking as an example a case in which one of the players is designated as the attention subject as illustrated in Fig. 11(B) from the example of Fig. 11(A) described above.

[0082] First, in step S1301, the area determination unit 302 waits for a request to specify an object or the like from the user, and when the area determination unit 302 receives a specification request via the operation input unit 305, the process proceeds to step S1302. In step S1302, the area determination unit 302 determines whether the request from the user is a request to designate a subject of interest. In this example, in step S1302, the area determination unit 302 determines whether the request is to designate the player 1103 as a subject of interest, as illustrated in Fig. 11(B) above. If the area determination unit 302 determines that the request is not a request to designate a subject of interest, it returns the process to step S1301, whereas if it determines that the request is a request to designate a subject of interest, it proceeds to step S1303.

[0083] In step S1303, the region determining unit 302 determines the region of the subject of interest designated by the user in step S1301 as a region of interest for directly using the captured image, based on the information of the subject of interest. The region determining unit 302 also determines the remaining region other than the subject of interest as a region of interest for capturing an image for generating a three-dimensional model.

[0084] Next, in step S1304, the role determination unit 303 determines (updates) the role of the camera based on the attention area determined by the area determination unit 302. In this example, the role determination unit 303 determines a camera to be used as a camera that uses the captured image of the attention area of ​​the player 1103 in FIG. 11(B) as is, and a camera to be used as a camera that uses the captured image of the remaining attention areas excluding the player 1103 for generating a three-dimensional model. In this example, before the designation of the attention subject as shown in FIG. 11(B) is performed, the entire field of the shooting target range has already been determined as the attention area for generating an image of a three-dimensional model as shown in FIG. 11(A). Therefore, in step S1304, the area determination unit 302 may determine the role of using the captured image as is only for the camera that captures the area of ​​the player 1103, which is the attention subject.

[0085] Next, in step S1305, setting control unit 304 changes the settings of each camera and controls the tripod head (controls the shooting direction of the camera) so that shooting is performed by each camera based on the role information of the camera determined by role determination unit 303. In this example, setting control unit 304 changes the camera setting values ​​including the zoom rate of each camera and the setting values ​​such as the angle of the tripod head from the setting values ​​according to the example of Fig. 11(A) to the setting values ​​for shooting according to the example of Fig. 11(B). Note that setting control unit 304 sets the shooting direction and zoom rate of the camera shooting the target subject player 1103 so that player 1103 fits as large as possible on the entire screen, that is, it is desirable to use the camera for close-up shooting.

[0086] Furthermore, the setting control unit 304 of the third embodiment controls the tripod head of a camera that is to capture an image of the target subject, player 1103, in accordance with the movement of the target subject, thereby causing the capturing direction of the camera to follow the movement of the player 1103. Furthermore, for example, if the target subject, player 1103, moves and moves out of the range that the camera can capture, the role determination unit 303 sets another camera to the role of capturing an image of the target subject, player 1103, enabling the continuation of tracking.

[0087] Then, after the camera setting change and the pan head control are performed in step S1305, in step S1306, the control unit 301 transmits camera setting information including information indicating the role of the camera in addition to information indicating the settings of the camera and the angle of the pan head to the image generating device 122. This enables the image generating device 122 to generate a three-dimensional model based on the camera setting information including the role information of the camera and the captured images of each camera 112, and to generate a virtual viewpoint image using the three-dimensional model. However, in the case of the third embodiment, for the area of ​​the player 1103, which is the subject of interest in Fig. 11(B), the captured images of 1101a, 1101g, and 1101m in Fig. 11(B) are used as they are.

[0088] As described above, in the content generation system 100 of the third embodiment, the attention area is determined according to the designation of the target subject by the user, and the role of each camera is set according to the determined attention area and photographed. In the third embodiment, it is possible to obtain a high-definition photographed image of the target subject, thereby realizing appropriate photographing and generation of a three-dimensional model according to the designation of the target subject. That is, according to the third embodiment, appropriate photographing according to the designation of the target subject and efficient use of multiple cameras are possible, and it is possible to build a system that generates high-quality virtual viewpoint content adapted to the designation of the target subject.

[0089] In the third embodiment, an example is given in which it is assumed that the captured image of the target subject designated by the user is used as is, but the present invention is not limited to this. For example, the user may designate a specific location of interest instead of a target subject. In this case, the role of the camera may be set to determine a target area according to the specific location designated by the user and capture the target area in high definition. In addition, the designation of the target subject and the designation of the specific location as described above may be appropriately selected. In this case, the control station 124 determines whether the designation by the user is a designation of a target subject or a designation of a specific location, and determines the target area and the role of the camera according to the determination result. At least one of the information processing according to the first and second embodiments described above may be combined with the information processing according to the third embodiment.

[0090] <Fourth embodiment> When shooting a game such as badminton or rugby as described above, it is considered that the role of the camera is updated successively with the passage of time. Therefore, in the fourth embodiment, information indicating the role of the camera that changes with the passage of time is stored in a database together with time information, and the role information of the camera for each time is made available when generating a replay image later. According to the fourth embodiment, when replaying an image at a certain desired time, it is possible to refer to the role of the camera for each time, so that an effective and high-quality virtual viewpoint image can be generated. Note that the configuration of the content generation system 100 according to the fourth embodiment is the same as that of FIG. 1, and the hardware configuration of the control station 124 is the same as that of FIG. 2, so that illustration and description thereof are omitted.

[0091] Fig. 14 is a functional block diagram showing each functional configuration of a control station 124 according to the fourth embodiment. In the functional block diagram of Fig. 14, components having substantially the same functions as those of the functional blocks shown in Fig. 7 are given the same reference numerals as in Fig. 7, and their description will be omitted as appropriate. In this embodiment, an example is given in which a region of interest is determined based on a detected scene as in the second embodiment, and the role of a camera is determined, but the present embodiment is also applicable to an example in which a region of interest is determined based on user input information as in the first or third embodiment, and the role of a camera is determined. An information recording unit 1401 in FIG. 14 stores the camera role information similar to that described above together with time information managed by the time server 127 .

[0092] FIG. 15 is a diagram used to explain the camera role determination performed by the role determination unit 303 of the fourth embodiment, and the camera roles recorded in the information recording unit 1401 in association with time information. In FIG. 15, 801a-801r in the camera number column correspond to the cameras 801a-801r in FIG. 8, respectively. In addition, the three-dimensional model generation and the use of the camera captured image in the camera role column indicate whether the captured image is used to generate a three-dimensional model or is used as it is. In addition, in FIG. 15, the horizontal straight arrows represent the passage of time, the vertical straight line 1501 represents a certain shooting time T1, and similarly the vertical straight line 1502 represents a certain shooting time T2. That is, each of the cameras 801a-801r shown in the camera number column indicates that the role described in the camera role column was determined at the shooting time T1 represented by the straight line 1501 and the shooting time T2 represented by the straight line 1502. 15 shows that, for example, at the shooting time T1 indicated by the line 1501, all of the cameras 801a-801r have been determined to have the role of capturing images for generating a three-dimensional model. Furthermore, at the shooting time T2 indicated by the line 1502, some of the cameras (801a, 801e, 801l, and 801o) have been determined to have the role of capturing images to be used as is.

[0093] For example, when generating a replay image of a scene at shooting time T2, the image generating device 122 can determine that the cameras 801a, 801e, 801l, and 801o were the cameras that took the images to be used as they are, by referring to the information in the information recording unit 1401. Therefore, when generating a replay image of the scene at shooting time T2 using a three-dimensional model that has already been generated and stored, the image generating device 122 can use a high-quality image that uses the taken images as they are in areas where players are crowded together.

[0094] Fig. 16 is a flowchart showing the flow of information processing executed by the control station 124 according to the fourth embodiment shown in Fig. 14. Here, an example in which the attention area and the role of the camera are switched in response to detection of a scene in which players are crowded together will be described with reference to Fig. 8(A) and Fig. 8(B) used in the second embodiment described above.

[0095] First, in step S1601, the image receiving unit 701 receives an overhead image captured by an overhead camera. Next, in step S1602, the scene detection unit 702 performs scene detection by analyzing the received overhead image. In this example, since the scene changes from the scene in Fig. 8(A) to the scene in Fig. 8(B) as an example, in step S1602 the scene detection unit 702 detects that the scene has changed from the scene in Fig. 8(A) to the scene in Fig. 8(B).

[0096] Next, in step S1603, the control unit 301 determines whether or not it is necessary to change the role of the camera based on the information of the detected scene. If the control unit 301 determines in step S1603 that it is not necessary to change the role of the camera, the process returns to step S1601, whereas if it determines that it is necessary to change the role of the camera, the process proceeds to step S1604. In this example, the roles of the cameras 801a-801r when the scene of FIG. 8B was detected in step S1602 are not appropriate roles for the detected scene, so the control unit 301 proceeds to step S1604.

[0097] When the process proceeds to step S1604, the area determination unit 302 determines an area of ​​interest based on the scene information received from the scene detection unit 702. In this example, the area determination unit 302 determines the area where the players 802 are densely located in Fig. 8(B) as an area of ​​interest for directly using the image captured by the camera, and determines the remaining areas as areas of interest for capturing images for generating a three-dimensional model.

[0098] Next, in step S1605, the role determination unit 303 determines (updates) the role of the camera based on the attention area determined by the area determination unit 302. In this example as well, the role determination unit 303 determines a camera whose role is to use the captured image of the dense area of ​​players 802 in Fig. 8(B) as is, and a camera whose role is to use the captured image of the remaining area excluding the dense area for generating a three-dimensional model. In the next step S1606, the information recording unit 1401 records the information indicating the role of the camera in association with the time information.

[0099] Next, in step S1607, setting control unit 304 changes the settings of each camera, controls the tripod head (controls the shooting direction of the camera), etc. so that shooting is performed by each camera according to the role information of the camera determined by role determination unit 303. In this example, setting control unit 304 changes the camera setting values ​​including the zoom ratio of each camera and the setting values ​​such as the tripod head angle from the setting values ​​corresponding to the scene in Fig. 8(A) to the setting values ​​for shooting according to the scene in Fig. 8(B).

[0100] After that, in step S1608, the control unit 301 transmits camera setting information including information indicating the role of the camera in addition to information indicating the settings of the camera and the angle of the camera platform to the image generating device 122. This allows the image generating device 122 to generate a three-dimensional model based on the camera setting information including the role information of the camera and the images captured by each camera 112, and to generate a virtual viewpoint image using the three-dimensional model. However, in the fourth embodiment, as in the second embodiment, the images captured by the cameras 801a, 801e, 801i, and 801o in FIG. 8(B) are used as they are for the area crowded with players 802 shown in FIG. 8(B). In addition, the setting control unit 304 sets the shooting direction and zoom ratio for the camera capturing the area crowded with players 802, where it is difficult to generate a high-quality three-dimensional model, as shown in FIG. 8(B) so that the area is contained as large as possible in the entire screen.

[0101] In the content generation system 100 of the fourth embodiment, the camera role information is associated with time information and recorded in the information recording unit 1401, so that when generating a replay image, it is possible to refer to the camera role for each time. Therefore, according to the fourth embodiment, it is possible to generate higher quality virtual viewpoint content for a scene at a desired time. At least any one of the information processes according to the first to third embodiments described above may be combined with the information process according to the fourth embodiment.

[0102] The present disclosure can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions. The above-mentioned embodiments are merely examples of concrete implementations of the present disclosure, and the technical scope of the present disclosure should not be interpreted as being limited by them. In other words, the present disclosure can be implemented in various forms without departing from its technical concept or main features.

[0103] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) a region determining means for determining a region of interest within a range of an object to be photographed by a plurality of photographing devices; a role determination means for determining a role of each of the plurality of image capture devices based on the determined region of interest; a generating means for generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image from images captured by the plurality of image capturing devices in synchronization with each other, based on the determined roles of the image capturing devices; 13. An information processing device comprising: (Configuration 2) The area determination means determines the attention area according to a part of the photographing range based on a user's designation of the part of the photographing range, the role determination means determines a role of each of the plurality of image capture devices to be a role of capturing an image of the area of ​​interest corresponding to the partial area; 2. The information processing apparatus according to configuration 1, wherein the generating means generates the three-dimensional model from images captured by the plurality of image capturing devices determined to be responsible for capturing the region of interest. (Configuration 3) The region determining means determines two or more regions of interest for the subject range, the role determination means determines the roles of the plurality of image capture devices by dividing them into roles of capturing images of the two or more regions of interest, 2. The information processing apparatus according to configuration 1, wherein the generating means generates the three-dimensional model from the captured image for each of the regions of interest. (Configuration 4) The information processing device described in configuration 3, characterized in that the area determination means determines the two or more areas of interest for the shooting range based on an instruction from a user to specify the entire shooting range as an area for capturing images to be used in generating the three-dimensional model. (Configuration 5) The information processing device described in configuration 1, characterized in that the role determination means determines some of the multiple camera devices as camera devices whose role is to use the captured images as they are, based on the determined area of ​​interest, and determines the remaining camera devices excluding the some of the camera devices as camera devices whose role is to generate a three-dimensional model from the captured images. (Configuration 6) a detection means for detecting a state of a subject in the photographing range, 6. The information processing apparatus according to any one of configurations 1 to 5, wherein the area determining means determines the attention area based on a state of the subject detected by the detection means. (Configuration 7) The detection means detects a situation in which the subjects are crowded together within the photographing range, the region determining means determines, for the photographing range, a first attention region corresponding to a region where the objects are densely packed and a second attention region corresponding to a region where the objects are not densely packed; The information processing device according to configuration 6, wherein the role determination means determines some of the plurality of camera devices as camera devices having a role in which an image captured of the first area of ​​interest is used as is, and determines the remaining camera devices excluding the some of the camera devices as camera devices having a role in which the three-dimensional model is generated from an image captured of the second area of ​​interest. (Configuration 8) the area determining means determines, with respect to the photographing range, a first attention area corresponding to an area of ​​an attention subject designated by a user, and a second attention area corresponding to a remaining area excluding the area of ​​the attention subject; The information processing device described in any one of configurations 1 to 7, characterized in that the role determination means determines some of the multiple camera devices as camera devices with a role in which an image captured of the first area of ​​interest is used as is, and determines the remaining camera devices excluding the some of the camera devices as camera devices with a role in which the three-dimensional model is generated from an image captured of the second area of ​​interest. (Configuration 9) 9. The information processing device according to configuration 8, wherein the area determining means moves the first area of ​​interest in accordance with movement of a subject of interest. (Configuration 10) The information processing device described in configuration 9, characterized in that when the movement of the subject of interest causes the subject of interest to move out of the range in which the part of the camera devices can capture the subject of interest, the role determination means determines another camera device capable of capturing the subject of interest to be the camera device with the role of capturing the subject of interest. (Configuration 11) A setting control means for controlling the photographing direction and the angle of view of the photographing device is provided, The information processing device according to any one of configurations 7 to 10, characterized in that the setting control means controls the shooting direction and angle of view of the image capturing device determined to be responsible for capturing the first area of ​​interest so that the first area of ​​interest fits as large as possible within the entire screen. (Configuration 12) a recording means for recording the role of the camera device determined by the role determination means in association with time, The information processing device according to any one of configurations 1 to 11, wherein the generating means generates a virtual viewpoint image using the three-dimensional model based on the role of the imaging device recorded in the recording means corresponding to the time. (Method 1) A region determination step of determining a region of interest within a range of an object to be photographed by a plurality of photographing devices; a role determination step of determining a role of each of the plurality of image capture devices based on the determined region of interest; a generation step of generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image based on the determined roles of the image capturing devices from the images captured by the plurality of image capturing devices in synchronization; 13. An information processing method comprising: (Program 1) A program that causes a computer to function as the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0104] 100: content generation system, 101: photography system, 122: image generation device, 123: controller, 124: control station, 301: control unit, 302: area determination unit, 303: role determination unit, 304: setting control unit

Claims

1. a region determining means for determining a region of interest within a range of an object to be photographed by a plurality of photographing devices; a role determination means for determining a role of each of the plurality of image capture devices based on the determined region of interest; a generating means for generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image from images captured by the plurality of image capturing devices in synchronization with each other, based on the determined roles of the image capturing devices; 13. An information processing device comprising:

2. The area determination means determines the attention area according to a part of the photographing range based on a user's designation of the part of the photographing range, the role determination means determines a role of each of the plurality of image capture devices to be a role of capturing an image of the area of ​​interest corresponding to the partial area; 2 . The information processing apparatus according to claim 1 , wherein the generating means generates the three-dimensional model from images captured by the plurality of image capturing devices determined to be responsible for capturing images of the region of interest.

3. The region determining means determines two or more regions of interest for the subject range, the role determination means determines the roles of the plurality of image capture devices by dividing them into roles of capturing images of the two or more regions of interest, The information processing apparatus according to claim 1 , wherein the generating means generates the three-dimensional model from the captured image for each of the regions of interest.

4. The information processing device according to claim 3, characterized in that the area determination means determines the two or more areas of interest for the shooting range based on instructions from a user who specifies the entire shooting range as an area for capturing images to be used in generating the three-dimensional model.

5. The information processing device according to claim 1, characterized in that the role determination means determines some of the plurality of camera devices as camera devices whose role is to use the captured images as they are, based on the determined area of ​​interest, and determines the remaining camera devices excluding the some of the camera devices as camera devices whose role is to generate a three-dimensional model from the captured images.

6. a detection means for detecting a state of a subject in the photographing range, 2. The information processing apparatus according to claim 1, wherein the area determining means determines the attention area based on a state of the subject detected by the detecting means.

7. The detection means detects a situation in which the subjects are crowded together within the photographing range, the region determining means determines, for the photographing range, a first attention region corresponding to a region where the objects are densely packed and a second attention region corresponding to a region where the objects are not densely packed; The information processing device according to claim 6, characterized in that the role determination means determines some of the plurality of camera devices as camera devices with a role in which an image captured of the first area of ​​interest is used as is, and determines the remaining camera devices excluding the some of the camera devices as camera devices with a role in which the three-dimensional model is generated from an image captured of the second area of ​​interest.

8. the area determining means determines, with respect to the photographing range, a first attention area corresponding to an area of ​​a target object designated by a user, and a second attention area corresponding to a remaining area excluding the area of ​​the target object; The information processing device according to claim 1, characterized in that the role determination means determines some of the multiple camera devices as camera devices with a role in which an image captured of the first area of ​​interest is used as is, and determines the remaining camera devices excluding the some of the camera devices as camera devices with a role in which the three-dimensional model is generated from an image captured of the second area of ​​interest.

9. 9. The information processing apparatus according to claim 8, wherein the area determining means moves the first area of ​​interest in accordance with the movement of a subject of interest.

10. The information processing device according to claim 9, characterized in that, when the movement of the subject of interest causes the subject of interest to move out of the range in which the part of the photographing devices can photograph the subject of interest, the role determination means determines another photographing device capable of photographing the subject of interest to be the photographing device with the role of photographing the subject of interest.

11. A setting control means for controlling the photographing direction and the angle of view of the photographing device is provided, The information processing device according to any one of claims 7 to 10, characterized in that the setting control means controls the shooting direction and angle of view of an imaging device determined to be responsible for shooting the first area of ​​interest so that the first area of ​​interest fits as large as possible within the entire screen.

12. a recording means for recording the role of the camera device determined by the role determination means in association with time, The information processing device according to claim 1 , wherein the generating means generates a virtual viewpoint image using the three-dimensional model based on the role of the imaging device recorded in the recording means in correspondence with the time.

13. A region determination step of determining a region of interest within a range of an object to be photographed by a plurality of photographing devices; a role determination step of determining a role of each of the plurality of image capture devices based on the determined region of interest; a generation step of generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image based on the determined roles of the image capturing devices from the images captured by the plurality of image capturing devices in synchronization; 13. An information processing method comprising:

14. Computer, a region determining means for determining a region of interest within a range of an object to be photographed by a plurality of photographing devices; a role determination means for determining a role of each of the plurality of image capture devices based on the determined region of interest; a generating means for generating a three-dimensional model of a subject to be used for generating a virtual viewpoint image from images captured by the plurality of image capturing devices in synchronization with each other, based on the determined roles of the image capturing devices; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images