Control device, method and program

Abstract

To easily designate a virtual object appropriate for a current virtual-camera position.SOLUTION: A control device according to the present disclosure is to acquire a virtual-camera parameter including information of at least any of a position, direction and view angle of a virtual camera in a virtual space, generate a user interface for assisting a user in inputting to select one or more virtual object groups out of a plurality of virtual object groups to which one or more virtual objects belong to, based on the virtual-camera parameter, and furthermore determine the virtual object group based on the input to select the virtual object group from the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a user interface technology for generating a virtual viewpoint image. [Background technology]

[0002] There is a technology for generating a virtual viewpoint image capable of arbitrarily changing the viewpoint by generating three-dimensional shape data using data of a plurality of captured images obtained by synchronously capturing images by a plurality of imaging devices installed at different positions. This technology captures the generated three-dimensional shape data using a virtual camera (hereinafter referred to as a "virtual camera"), and generates an image of the three-dimensional shape data as viewed from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") as a virtual viewpoint image. It is possible to display virtual objects such as a virtual advertisement (hereinafter referred to as a "virtual advertisement") in such a virtual viewpoint image. Patent Document 1 discloses a technology for displaying a virtual advertisement in a virtual viewpoint image based on the designation of a user (hereinafter referred to as a "operator") who operates the display of a virtual advertisement in the virtual viewpoint image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-90786 Summary of the Invention [Problem to be solved by the invention]

[0004] Various modes are possible for the position, orientation, or size of a virtual object displayed in a virtual viewpoint image in a virtual space. Therefore, whether or not a virtual object such as a virtual advertisement is effectively displayed changes depending on at least one of the position (hereinafter referred to as "virtual camera position"), orientation, and direction of the virtual camera. Therefore, the operator is required to specify an appropriate virtual object according to the current virtual camera position, etc., and place it in the virtual space. However, it is difficult to instantly determine and specify an appropriate virtual object while operating the virtual camera, such as moving the virtual camera position. The present disclosure has been made in consideration of the above problem, and aims to provide a technology that enables the operator to easily specify an appropriate virtual object according to the current virtual camera position, etc. [Means for solving the problem]

[0005] The control device of the present disclosure includes a parameter acquisition means for acquiring virtual camera parameters including information on at least one of a position, orientation, and angle of view of a virtual camera in a virtual space, a UI generation means for generating a user interface based on the virtual camera parameters, the user interface assisting a user's input for selecting one or more virtual object groups from among a plurality of virtual object groups to which one or more virtual objects belong, and a determination means for determining the virtual object group based on an input from the user for selecting the virtual object group. Effect of the Invention

[0006] According to the present disclosure, the operator can easily specify an appropriate virtual object according to the current virtual camera position. [Brief description of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of a configuration of a control system according to a first embodiment. [Diagram 2]2 is a block diagram showing an example of a hardware configuration of each device included in the control system according to the first embodiment. FIG. [Diagram 3] 5 is a flowchart showing an example of a processing flow in the virtual camera control device according to the first embodiment. [Figure 4] 13 is a flowchart showing an example of a flow of a process of determining a virtual advertisement group ID in the virtual camera control device according to the first embodiment. [Diagram 5] 11 is a flowchart showing an example of the flow of a recommendation range determination process in the virtual camera control device according to the first embodiment. [Figure 6] 4 is a diagram showing an example of a UI for specifying a recommendation range, which is generated by a UI generating unit according to the first embodiment. FIG. [Figure 7] 13 is an example of a UI for specifying a virtual advertisement group, generated by a UI generation unit according to the first embodiment. [Figure 8] 5 is a flowchart showing an example of a processing flow in the image generating device according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing an example of the configuration of a control system according to a second embodiment. [Figure 10] 10 is a flowchart showing an example of a processing flow in a virtual camera control device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals.

[0009] [Embodiment 1] Fig. 1 is a block diagram showing an example of the configuration of a control system according to the first embodiment. Fig. 1 also shows the functional configuration of a virtual camera control device 120 and an image generating device 104. The imaging device group 101 includes a plurality of imaging devices, each of which is disposed at a different position from the others. Some of the imaging devices capture at least a part of the imaging target space, and perform imaging in synchronization with each other. Data of a plurality of captured images (hereinafter referred to as "multiple viewpoint image data") obtained by the synchronized imaging by each imaging device is transmitted to a shape data generating device 102.

[0010] The shape data generating device 102 receives the multi-view image data transmitted from the imaging device group 101, extracts an image area corresponding to an object for which 3D shape data is to be generated as a foreground from each captured image, and generates a silhouette image showing the silhouette of the foreground. The shape data generating device 102 also applies VisualHull to the generated silhouette image to generate 3D shape data showing the 3D shape of the foreground. Note that the method of extracting the foreground from the captured image and the method of generating 3D shape data using VisualHull are well known, so a description thereof will be omitted. The 3D shape data generated by the shape data generating device 102 is transmitted to the shape data storage device 103. The shape data storage device 103 has a large-capacity storage device such as an HDD (hard disk drive), receives the 3D shape data transmitted from the shape data generating device 102, and stores it in the storage device.

[0011] The virtual camera control device 120 is a device that allows an operator to operate a virtual camera for generating a virtual viewpoint image. The virtual camera control device 120 has, as functional components, a parameter calculation unit 121, a time code calculation unit 122, a group ID determination unit 123, a group ID holding unit 124, and a recommended range determination unit 125. In addition to the above-mentioned functional components, the virtual camera control device 120 also has a recommended range holding unit 126, a UI generation unit 127, and a virtual camera information transmission unit 128.

[0012] The parameter calculation unit 121 receives operation information output from the controller 141 by the operator operating the controller 141, and calculates virtual camera parameters including information on the position, orientation, and angle of view of the virtual camera based on the operation information. The virtual camera parameters calculated by the parameter calculation unit 121 are transmitted to the UI generation unit 127 and the virtual camera information transmission unit 128. As an example, when the virtual camera position is expressed by three-dimensional coordinates of [X, Y, Z] in the virtual space, the parameter calculation unit 121 calculates values ​​such as X=4.0, Y=9.0, and Z=1.5 as the virtual camera position. In this embodiment, the unit of each coordinate is meters [m], and the position of the origin is the center of the generation range of the three-dimensional shape data in the virtual space. In addition, when the X-axis is specified to be parallel to the ground surface, the Y-axis is specified to be parallel to the ground surface and perpendicular to the X-axis, and the Z-axis is specified to be perpendicular to the ground surface.

[0013] Furthermore, when the orientation of the virtual camera is expressed by three angles, [Pan, Tilt, Roll], the parameter calculation unit 121 calculates values ​​such as Pan=20.0, Tilt=10.0, and Roll=2.0 as the orientation of the virtual camera. In this embodiment, the unit of angle is degrees [°], and the range of each of Pan, Tilt, and Roll is from -180 to 180. Note that Pan is the angle of rotation in a direction parallel to the ground surface, Tilt is the angle of rotation in a direction perpendicular to the ground surface, and Roll is the angle of rotation around the optical axis of the virtual camera as the rotation axis. Furthermore, when the angle of view of the virtual camera is expressed by focal length [Zoom], the parameter calculation unit 121 calculates values ​​such as Zoom=6.0 as the angle of view of the virtual camera. In this embodiment, the unit of focal length is millimeters [mm]. The controller 141 includes a joystick or the like, and in this embodiment, the operator operates the joystick of the controller 141 to operate the virtual camera. For example, the virtual camera position changes depending on the amount and direction of tilt of the joystick.

[0014] The time code calculation unit 122 receives operation information output from the controller 141 as the operator operates the controller 141, and calculates a time code of the virtual viewpoint image based on the operation information. The calculated time code is transmitted to the virtual camera information transmission unit 128. The controller 141 includes a jog dial, and in this embodiment, as an example, the operator operates the jog dial of the controller 141 to input an instruction to play, stop, rewind, or fast forward the virtual viewpoint image, thereby determining the time code.

[0015] The group ID determination unit 123 determines a virtual ad group ID based on a designation from the operator. The virtual ad group ID (identifier) ​​is an identifier for uniquely identifying a virtual ad group that collectively handles one or more virtual advertisements. In this embodiment, the operator may designate a virtual ad group ID in the following manner, for example. For example, the operator designates a virtual ad group by pressing any one of buttons 708 to 713 using a mouse or the like in a UI (User Interface) 701 shown as an example in FIG. 7, which is displayed on the display 140. In the example of the UI 701 shown in FIG. 7, the operator presses a button 709 to designate a virtual ad group corresponding to the button 709, and the virtual ad group ID is determined to be 2 by the designation. The virtual ad group ID determined by the group ID determination unit 123 is transmitted to the group ID storage unit 124.

[0016] The group ID holding unit 124 receives the virtual advertisement group ID transmitted from the group ID determination unit 123, and stores and holds the virtual advertisement group ID in a storage device such as an HDD or a RAM (Random Access Memory). For example, when 2 is determined as the virtual advertisement group ID as in UI701 shown in Fig. 7, in the following description, it is assumed that 2 is held in the group ID holding unit 124.

[0017] The recommendation range determination unit 125 determines a recommendation range of the virtual ad group based on the operator's designation. The recommendation range indicates a range of a virtual camera position determined for each virtual ad group, and when the virtual camera position is within a certain recommendation range, a virtual ad group corresponding to the recommendation range is recommended to the operator. Information on the recommendation range determined by the recommendation range determination unit 125 is transmitted to the recommendation range holding unit 126. Note that, in this embodiment, the operator may designate the recommendation range in the following manner, for example. For example, the operator inputs a numerical value into any one of the text box groups 602 to 606 using a mouse, a keyboard, or the like in a UI 601 shown as an example in FIG. 6, which is displayed on the display 140. The recommendation range corresponding to the virtual ad group is designated by the input.

[0018] In the example of UI601 shown in FIG. 6, the text box group 602 shows a state where 1 is specified as the virtual ad group ID. Also, a state where 3.0 to 5.0 in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction are specified as the recommendation range of the virtual ad group ID with the virtual ad group ID being 1 is shown. In this case, a rectangular parallelepiped space having a width of 2.0 m in the X-axis direction, 15.0 m in the Y-axis direction, and 20.0 m in the Z-axis direction is set at the specified position as the recommendation range. Note that the method of specifying the recommendation range by the operator is not limited to the method of specifying by inputting numerical values ​​into the text boxes as in UI601 shown in FIG. 6. For example, the operator may specify the recommendation range graphically by transforming a solid object such as a rectangular parallelepiped in a three-dimensional space displayed as a UI using a mouse or the like. In this embodiment, the shape of the recommended range is described as a rectangular parallelepiped, but the shape of the recommended range is not limited to a rectangular parallelepiped, and may be any shape including any polyhedral shape, a sphere, etc. Also, the recommended range does not necessarily have to be specified by a closed space surrounded by a finite shape, and may be specified by a space with a part of the space being open.

[0019] The recommended range holding unit 126 receives information on the recommended range of each virtual advertisement group transmitted from the recommended range determination unit 125, and stores and holds this information in a storage device such as a HDD or a RAM.

[0020] The UI generation unit 127 generates a UI to be presented to the operator. The UI generated by the UI generation unit 127 is displayed on the display 140, and the operator can operate a mouse, a keyboard, or the like while checking the UI displayed on the display 140. Examples of the UI generated by the UI generation unit 127 include a UI 601 shown in FIG. 6 and a UI 701 shown in FIG. 7, which will be described later. The UI generation unit 127 acquires information on the recommended range of each virtual ad group held by the recommended range holding unit 126. In addition, the UI generation unit 127 receives virtual camera parameters transmitted from the parameter calculation unit 121. The UI generation unit 127 generates a UI that recommends virtual ad groups based on the recommended range of each virtual ad group and the virtual camera parameters.

[0021] Specifically, when the virtual camera position related to the generation of the virtual viewpoint image currently being processed is within the range of the recommendation range of a certain virtual ad group, the UI generation unit 127 generates a UI that recommends the virtual ad group. In the UI 701 shown as an example in Fig. 7, the recommendation state of the virtual ad group is expressed by whether or not the panels 714 to 719 are lit. For example, in the UI 701 shown in Fig. 7, when a certain panel among the panels 714 to 719 is lit, it is represented that the virtual ad group corresponding to the panel is recommended. Specifically, for example, the UI 701 shown in Fig. 7 represents a state in which the virtual ad groups with virtual ad group IDs of 1, 2, and 5 are recommended.

[0022] The virtual camera information transmission unit 128 acquires the virtual advertisement group ID held by the group ID holding unit 124. In addition, the virtual camera information transmission unit 128 receives the virtual camera parameters transmitted from the parameter calculation unit 121 and the time code transmitted from the time code calculation unit 122. Furthermore, the virtual camera information transmission unit 128 generates virtual camera information including the acquired virtual advertisement group ID and the received virtual camera parameters and time code, and transmits the generated virtual camera information to the image generation device 104.

[0023] The virtual advertisement generation device 106 generates data of a virtual advertisement created by a user as virtual object data using general-purpose application software capable of creating data of a virtual object, which is a three-dimensional object. Data of one or more virtual objects, which are virtual advertisements, generated by the virtual advertisement generation device 106 is transmitted to the image generation device 104.

[0024] The image generating device 104 generates a virtual viewpoint image, and outputs a signal indicating the generated virtual viewpoint image to the display 105 to display the virtual viewpoint image on the display 105. The image generating device 104 includes an image generating unit 110, a group registration unit 111, and a group holding unit 112.

[0025] The image generating unit 110 acquires three-dimensional shape data indicating the three-dimensional shapes of the foreground and background from the shape data storage device 103. The image generating unit 110 also receives virtual camera information transmitted from the virtual camera information transmitting unit 128. The image generating unit 110 generates a virtual viewpoint image based on the acquired three-dimensional shape data and the received virtual camera information, and outputs a signal indicating the generated virtual viewpoint image to the display 105. Note that the virtual camera information includes virtual camera parameters including information such as the position, orientation, and angle of view of the virtual camera, a time code, and a virtual advertisement group ID.

[0026] The process of generating a virtual viewpoint image includes the following processes. The first process is a process of acquiring, from the shape data storage device 103, three-dimensional shape data of the foreground and background corresponding to the time code included in the virtual camera information. The second process is a process of acquiring, from the group holding unit 112, data of a virtual ad group corresponding to a virtual ad group ID included in the virtual camera information. The third process is a process of generating a virtual viewpoint image based on the acquired three-dimensional shape data of the foreground and background, the data of the virtual ad group acquired from the group holding unit 112, and the virtual camera parameters included in the virtual camera information. Specifically, this process generates, as a virtual viewpoint image, an image of the three-dimensional shapes of the foreground and background indicated by the three-dimensional shape data and the virtual advertisement as a virtual object included in the virtual ad group, as viewed from a virtual viewpoint determined based on the virtual camera parameters. Note that the output destination of the virtual viewpoint image is not necessarily limited to the display 105, and for example, the data of the virtual viewpoint image may be output to an image distributor or a broadcasting station, etc.

[0027] The group registration unit 111 receives the data of the virtual advertisement as a virtual object transmitted from the virtual advertisement generation device 106, and registers the received data of the virtual advertisement in a virtual advertisement group. Specifically, the group registration unit 111 transmits the received data of the virtual advertisement to the group holding unit 112 in association with the virtual advertisement group ID. The group holding unit 112 receives the data of the virtual advertisement and the virtual advertisement group ID transmitted from the group registration unit 111. The group holding unit 112 stores and holds the received data of the virtual object in a storage device such as an HDD or a RAM in association with the received virtual advertisement group ID. Note that, in this embodiment, the data of the virtual advertisement as a virtual object includes information on the position of the virtual object in the virtual space, the shape of the virtual object, and the content indicated by the virtual object, but is not limited thereto. For example, the data of the virtual object may include only the information on the position and shape described above, and the information on the content described above may be configured to be separately specified. Note that, here, the information on the content is, for example, information on an image or the like indicating the content of the advertisement when the virtual object is a virtual advertisement.

[0028] Fig. 2 is a block diagram showing an example of a hardware configuration of each device included in the control system according to embodiment 1. The shape data generating device 102, the shape data storage device 103, the image generating device 104, the virtual advertisement generating device 106, and the virtual camera control device 120 can be realized by an information processing device 200 shown in Fig. 2. The information processing device 200 can include a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a system bus 208.

[0029] The CPU 201 realizes the functions of each device of the system shown in FIG. 1 by controlling the entire information processing device 200 using computer programs and data stored in the ROM 202 or the RAM 203. The information processing device 200 may have one or more dedicated hardware devices different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware devices. Examples of such dedicated hardware devices include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 202 stores programs and the like that do not require modification. The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, as well as data and the like supplied from the outside via the communication I / F 207. The auxiliary storage device 204 is composed of a hard disk drive or the like, and stores various data such as image data or audio data.

[0030] The display unit 205 is configured with a liquid crystal display or an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to give instructions to the information processing device 200. The operation unit 206 is configured with a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations from the user and inputs various instructions corresponding to the operations to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206.

[0031] The communication I / F 207 is used for communication with devices outside the information processing device 200, such as the imaging device group 101, the displays 105 and 140, the controller 141, and other information processing devices 200. When the information processing device 200 is connected to an external device by wire, a communication cable is connected to the communication unit 317. When the information processing device 200 has a function of wirelessly communicating with an external device, the communication I / F 207 includes an antenna. The system bus 208 connects each unit of the information processing device 200 to transmit information. In this embodiment, the display unit 205 and the operation unit 206 are assumed to exist inside the information processing device 200, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the information processing device 200.

[0032] The operation of the virtual camera control device 120 will be described with reference to Figs. 3 to 5. Fig. 3 is a flowchart showing an example of a processing flow of a process of recommending a virtual advertisement group to an operator in the virtual camera control device 120 according to the first embodiment. In the following description, the symbol "S" means a step. Each process shown in the flowchart is realized by developing a program code stored in the ROM 202 or the auxiliary storage device 204 or the like in the RAM 203 and executing the program by the CPU 201.

[0033] S301 and S311 represent repeated processing, and the virtual camera control device 120 repeatedly executes each process from S302 to S310 for each frame of the virtual viewpoint image generated by the image generation device 104. In this embodiment, the frame rate of the virtual viewpoint image is set to 59.94 fps (frames per second). That is, when the virtual viewpoint image is played back, the series of processes from S302 to S310 are processed every 1 / 59.94 seconds. The interval of the series of processes from S302 to S310 depends on the frame rate of the virtual viewpoint image.

[0034] First, in S302, the parameter calculation unit 121 calculates virtual camera parameters including information on the position, orientation, and angle of view of the virtual camera based on operation information from the controller 141 operated by the operator. For example, when the operator operates a controller having a joystick to operate the virtual camera, the parameter calculation unit 121 calculates the three-dimensional position of the virtual camera based on the amount and orientation of the joystick tilt. S303 and S306 represent repeated processing, and the virtual camera control device 120 repeatedly executes the processing of S304 and S305 for each virtual advertisement group. That is, after S302, the processing of S304 and S305 is repeatedly executed.

[0035] First, in S304, the UI generation unit 127 determines whether or not the virtual camera position determined based on the virtual camera parameters calculated in S302 is within the range of the recommendation range of the virtual ad group. Note that the recommendation range of each virtual ad group is determined in advance, for example, by a determination method described later with reference to Fig. 5, before the start of the processing of the flowchart shown in Fig. 3. Also, it is assumed that the UI generation unit 127 has previously acquired information on the recommendation range of each virtual ad group at any timing, such as before the start of the processing of the flowchart shown in Fig. 3.

[0036] As a specific example of the determination method, a case will be described in which the virtual camera position is expressed in three-dimensional coordinates of [X, Y, Z], and the values ​​are X=4.0, Y=9.0, and Z=1.5. In the case of UI 601 shown in FIG. 6 described later, in a text box group 602, 3.0 to 5.0 in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction are specified as the recommendation range of a virtual ad group whose virtual ad group ID is 1. The virtual camera position in the X-axis direction, 4.0, is included in the range X of 3.0 to 5.0 specified in the text box group 602. Similarly, the virtual camera position in the Y-axis direction, 9.0, is included in the range Y of 0.0 to 15.0, and the virtual camera position in the Z-axis direction, 1.5, is included in the range Z of 0.0 to 20.0.

[0037] Therefore, all virtual camera positions in the X-axis direction, the Y-axis direction, and the Z-axis direction are included within the range of the recommended range. Therefore, for a virtual ad group with a virtual ad group ID of 1, the UI generation unit 127 determines that the virtual camera is present within the range of the recommended range. In this way, when the X coordinate, the Y coordinate, and the Z coordinate of the virtual camera position are each within the range in all of the ranges X, Y, and Z, the UI generation unit 127 determines that the virtual camera position is within the range of the recommended range. On the other hand, when the virtual camera position is not included in one or more of the ranges X, Y, and Z, the UI generation unit 127 determines that the virtual camera position is not present within the range of the recommended range. The UI generation unit 127 performs the same determination for all virtual ad groups for which the recommended range is specified in S304 by repeating the above-mentioned process.

[0038] Note that, like the range X of the text box group 603, the upper limit of the range may not be specified. In this case, if the virtual camera position in the X-axis direction is -15.0 or more as indicated by the range X of the text box group 603, the UI generation unit 127 treats the virtual camera position in the X-axis direction as being within the range of the recommended range in the X-axis direction. The same applies to the range Y and the range Z. Similarly, the lower limit of the range may not be specified for at least one of the ranges X, Y, and Z. In this case, for an axis direction in which the lower limit of the range is not specified, if the virtual camera position in the axis direction is equal to or less than the upper limit of the range in the axis direction, the virtual camera position in the axis direction is treated as being within the range of the recommended range in the axis direction.

[0039] Also, like the range Z of the text box group 604, both the lower limit value and the upper limit value of the range may not be specified. For example, when both the upper limit value and the lower limit value of the range Z are not specified, the UI generation unit 127 judges whether or not the virtual camera position is within the range of the recommended range only in the X-axis direction and the Y-axis direction. Then, if the X coordinate of the virtual camera position is within the range X and the Y coordinate is within the range Y, the UI generation unit 127 judges that the virtual camera position is within the range of the recommended range. The same applies to the range X and the range Y when both the lower limit value and the upper limit value are not specified. Also, like the text box group 605, both the upper limit value and the lower limit value may not be specified for all of the range X, the range Y, and the range Z. In this case, the UI generation unit 127 judges that the virtual camera position is within the range of the recommended range regardless of the virtual camera position.

[0040] When it is determined in S304 that the virtual camera position is within the range of the recommendation range, in S305, the UI generation unit 127 sets the virtual ad group as a virtual ad group to be recommended. After S305, or when it is determined in S304 that the virtual camera position is not within the range of the recommendation range, the UI generation unit 127 determines whether or not the process of S304 has been executed for all virtual ad groups for which the recommendation range is specified. When it is determined that the process of S304 has not been executed for at least any virtual ad group, the UI generation unit 127 executes the process of S304 for virtual ad groups for which the process of S304 has not yet been executed, by the above-mentioned repeated process. When it is determined that the process of S304 has been executed for all virtual ad groups, the virtual camera control device 120 ends the repeated process, and the UI generation unit 127 executes the process of S307.

[0041] In S307, the UI generation unit 127 generates a UI that recommends all virtual ad groups set as recommendation targets in S305. For example, in a UI 701 shown in Fig. 7 described later, the operator can visually confirm the virtual ad groups that are the recommendation targets by turning on or off the panels 714 to 719. Specifically, for example, the example of the UI 701 shown in Fig. 7 shows a state in which virtual ad groups with virtual ad group IDs of 1, 2, and 5 are recommended.

[0042] Next, in S308, the time code calculation unit 122 calculates a time code of the virtual viewpoint image based on operation information from the controller 141 operated by the operator. For example, the operator operates the jog dial of the controller 141 to input an instruction to play, stop, rewind, or fast forward the virtual viewpoint image. The time code calculation unit 122 receives operation information corresponding to the input transmitted from the controller 141, and calculates a time code based on the operation information. Next, in S309, the virtual camera control device 120 performs a process of determining a virtual advertisement group ID based on a designation from the operator. Details of the determination process will be described later with reference to FIG. 4.

[0043] Next, in S310, the virtual camera information transmission unit 128 transmits the virtual camera information to the image generation device 104. Specifically, in the process of S310, the virtual camera information transmission unit 128 acquires the virtual advertisement group ID determined in S309. In addition, in the process of S310, the virtual camera information transmission unit 128 acquires the virtual camera parameters calculated by the parameter calculation unit 121 in S302 and the time code calculated by the time code calculation unit 122 in S308. Furthermore, in the process of S310, the virtual camera information transmission unit 128 generates virtual camera information including the acquired virtual advertisement group ID, virtual camera parameters, and time code, and transmits this to the image generation unit 110.

[0044] As described above, virtual camera control device 120 repeatedly executes the processes from S302 to S310 until, for example, the operator inputs an instruction to end the processes in virtual camera control device 120. In this case, when the operator inputs an instruction, virtual camera control device 120 ends the processes in the flowchart shown in FIG.

[0045] 4 is a flowchart showing an example of a flow of a process of determining a virtual advertisement group ID based on a designation from an operator in the virtual camera control device 120 according to embodiment 1. That is, the flowchart shows an example of a detailed flow of the process in S309, and the process of the flowchart is executed after S308. Note that each process shown in the flowchart is realized by loading a program code stored in the ROM 202 or the auxiliary storage device 204, etc., into the RAM 203, and executing the program by the CPU 201.

[0046] First, in S401, the group ID determination unit 123 determines a virtual advertisement group ID based on the operator's designation. For example, the operator designates a desired virtual advertisement group ID by using a mouse to select and press one of buttons 708 to 713 in a UI 701 shown as an example in Fig. 7 described later. Note that the example of the UI 701 shown in Fig. 7 shows a state in which the operator selects and presses button 709, and in this state, the group ID determination unit 123 determines the virtual advertisement group ID to be 2.

[0047] Next, in S402, the group ID holding unit 124 receives the virtual ad group ID determined in S401 and holds the virtual ad group ID. In the example of UI701 illustrated in Fig. 7, 2 is held as the virtual ad group ID by the group ID holding unit 124. Note that in the process of S310, when acquiring the virtual ad group ID determined in S309, the virtual camera information transmission unit 128 acquires the virtual ad group ID held in the group ID holding unit 124 in S402. After S402, the virtual camera control device 120 ends the process of the flowchart illustrated in Fig. 4, and then executes the process of S310.

[0048] Fig. 5 is a flowchart showing an example of a flow of a process of determining a recommendation range for each virtual advertisement group as specified by an operator in the virtual camera control device 120 according to embodiment 1. Note that the process of this flowchart is executed, for example, before the process of the flowchart shown in Fig. 3 is executed. Also, each process shown in the flowchart shown in Fig. 5 is realized by loading a program code stored in the ROM 202 or the auxiliary storage device 204, etc., into the RAM 203 and executing the program by the CPU 201.

[0049] First, in S501, the recommendation range determination unit 125 determines a recommendation range of a virtual ad group based on the operator's designation. For example, the operator designates a recommendation range of a desired virtual ad group by inputting a desired numerical value into any one of text box groups 602 to 606 using a mouse and a keyboard in a UI 601 shown as an example in FIG. 6 described later. The example of the text box group 602 of the UI 601 shown in FIG. 6 illustrates a state in which 1 is designated as a virtual ad group ID. In addition, the example illustrates a state in which 3.0 to 5.0 are designated in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction as recommendation ranges of a virtual ad group whose virtual ad group ID is 1. The recommendation range determination unit 125 determines a recommendation range of a virtual ad group whose virtual ad group ID is 1 based on this designation.

[0050] Next, in S502, the recommendation range holding unit 126 receives information on the recommendation range of each virtual advertisement group determined in S501 and holds this information. After S502, the virtual camera control device 120 ends the processing of the flowchart shown in Fig. 5. Note that the information on the recommendation range of each virtual advertisement group held in S502 is acquired by the UI generation unit 127 at any timing, such as before the start of the processing of the flowchart shown in Fig. 3.

[0051] 6 is a diagram showing an example of a UI601 generated by the UI generating unit 127 according to the first embodiment, which is an example of the UI601 for the operator to specify a recommendation range of a virtual ad group. Note that the display form of the UI601 is not limited in the present disclosure. For example, the UI601 may be displayed as one window, or may be displayed as a part of one UI including the UI601 and another UI related to the control of the virtual camera.

[0052] The text box groups 602 to 606 are areas where the operator specifies a recommendation range for each virtual ad group using an operation input device such as a mouse and a keyboard. In the UI 601, the text box group 602 specifies the recommendation range of a virtual ad group whose virtual ad group ID is 1. Similarly, the text box group 603 specifies the recommendation range of a virtual ad group whose virtual ad group ID is 2, the text box group 604 specifies the recommendation range of a virtual ad group whose virtual ad group ID is 3, and the text box group 605 specifies the recommendation range of a virtual ad group whose virtual ad group ID is 5. The text box group 606 is in a state where a virtual ad group ID has not been input and a recommendation range of a virtual ad group has not been specified.

[0053] For example, after inputting and specifying a virtual ad group ID and a recommendation range of a virtual ad group corresponding to a desired virtual ad group ID, the operator presses a decision button or the like not shown in Fig. 6. The recommendation range determination unit 125, triggered by the pressing of the decision button or the like, determines the recommendation range of each virtual ad group based on the information set in the text boxes 602 to 606 of the UI 601 shown in Fig. 6.

[0054] 7 is a diagram showing an example of a UI 701 generated by the UI generating unit 127 according to the first embodiment, which is used by the operator to specify a virtual ad group to be placed in a virtual space. Note that the display form of the UI 701 is not limited in the present disclosure. For example, the UI 701 may be displayed as one window, or may be displayed as a part of a UI including the UI 701 and another UI related to the control of the virtual camera.

[0055] The images 702 to 707 show the arrangement of each virtual object when one or more virtual objects belonging to a virtual ad group, that is, a virtual advertisement, are arranged in a virtual space. The operator selects a virtual ad group with reference to the images 702 to 707. In addition, the virtual ad group ID of the virtual ad group corresponding to each of the images 702 to 707 is displayed at the top of each of the images 702 to 707. Each of the images 702 to 707 may be automatically generated based on, for example, information on the position in the virtual space of one or more virtual advertisements (virtual objects) belonging to the virtual ad group. In addition, each of the images 702 to 707 may be manually created separately by a creator of the virtual advertisement. Note that the images 702 to 707 shown in the UI 701 of FIG. 7 show only the shape and arrangement of the virtual advertisement as simplified information, but the virtual advertisement in the images 702 to 707 may include the content of the virtual advertisement. In this case, the automatically generated images 702 to 707 are generated based on, for example, information on the position in the virtual space of each virtual advertisement belonging to each virtual advertisement group, and information on the content of each virtual advertisement.

[0056] The buttons 708 to 713 are for specifying a virtual ad group ID of a virtual ad group that the operator desires to display in the virtual viewpoint image. For example, the buttons 708 to 713 can be selected by a mouse click operation. The operator selects a button corresponding to a virtual ad group that the operator desires to display in the virtual viewpoint image from among the buttons 708 to 713 by this operation. The group ID determination unit 123 receives the designation by this operation and determines the virtual ad group ID corresponding to the selected button.

[0057] In the present disclosure, the method of designating the virtual ad group ID by the operator is not limited to this. For example, the operator may designate the virtual ad group ID by selecting any one of the images 702 to 707, or may designate the virtual ad group ID by directly inputting the value of the virtual ad group ID using a keyboard. In the example of UI701 shown in FIG. 7, the button 709 is selected, that is, the virtual ad group with the virtual ad group ID of 2 is selected. In this state, one or more virtual advertisements included in the virtual ad group with the virtual ad group ID of 2 may be displayed in the virtual viewpoint image.

[0058] Panels 714 to 719 indicate whether each virtual ad group is a target for recommendation. For example, among the multiple virtual ad groups corresponding to the panels 714 to 719, the virtual ad groups corresponding to the lit panels indicate that they are virtual ad groups that are targets for recommendation, i.e., recommended virtual ad groups. Also, the virtual ad groups corresponding to the unlit panels indicate that they are virtual ad groups that are not targets for recommendation, i.e., virtual ad groups that are not recommended. In the example of UI701 shown in FIG. 7, panels 714, 715, and 718 are lit, and panels 716, 717, and 719 are unlit. That is, in the example of UI701, it is indicated that the virtual ad groups corresponding to the virtual ad group IDs 1, 2, and 5 are recommended.

[0059] The operation of the image generating device 104 will be described with reference to Fig. 8. Fig. 8(a) is a flowchart showing an example of a flow of a virtual advertisement group registration process in the image generating device 104 according to the first embodiment. Fig. 8(b) is a flowchart showing an example of a flow of a virtual viewpoint image generation process in the image generating device 104 according to the first embodiment.

[0060] With reference to FIG. 8(a), a flow of a virtual advertisement group registration process in the image generating device 104 will be described. First, in S801, the group registration unit 111 receives data of virtual objects of one or more virtual advertisements transmitted from the virtual advertisement generating device 106, and registers the received data of one or more virtual objects as data of a virtual advertisement group. Specifically, the group registration unit 111 registers the data of the virtual objects of one or more virtual advertisements in association with a virtual advertisement group ID designated by a user of the virtual advertisement generating device 106. Next, in S802, the group holding unit 112 receives the data of the virtual objects associated with the virtual advertisement group ID registered in S801, and holds this as a virtual advertisement group. After S802, the image generating device 104 ends the process of the flowchart shown in FIG. 8(a). Note that the process of the flowchart is executed in advance at any timing before the start of the process of the flowchart shown in FIG. 8(b).

[0061] With reference to FIG. 8(b), the flow of the generation process of the virtual viewpoint image in the image generating device 104 will be described. The process of the flowchart shown in FIG. 8 is repeatedly executed for each frame of the virtual viewpoint image, for example. First, in S811, the image generating unit 110 receives the virtual camera control device 120, specifically, the virtual camera information transmitted from the virtual camera information transmitting unit 128. Next, in S812, the image generating unit 110 acquires three-dimensional shape data of the foreground and background corresponding to the time code included in the virtual camera information received in S811 by reading it from the shape data storage device 103. Next, in S813, the image generating unit 110 acquires data of the virtual advertisement group corresponding to the virtual advertisement group ID included in the virtual camera information received in S811 from the group holding unit 112.

[0062] Next, in S814, the image generating unit 110 generates a virtual viewpoint image. Specifically, the image generating unit 110 generates a virtual viewpoint image based on the virtual camera parameters included in the virtual camera information received in S811, the three-dimensional shape data of the foreground and background acquired in S812, and the data of the virtual advertisement group acquired in S813. The virtual viewpoint image generated in S814 is a frame of the virtual viewpoint image corresponding to the time code included in the virtual camera information received in S811. In addition, the virtual viewpoint image generated in S814 is displayed on the display 105. After S814, the image generating device 104 ends the processing of the flowchart shown in FIG. 8(b), and repeatedly executes the processing of the flowchart every time new virtual camera information is transmitted from the virtual camera control device 120.

[0063] As described above, the virtual camera control device 120 is configured to specify a recommendation range for each virtual ad group, and when the virtual camera position is within the range of the recommendation range of a certain virtual ad group, the virtual ad group is recommended to the operator. With such a virtual camera control device 120, the operator can easily or instantly determine the selection of the virtual ad group to be displayed on the virtual viewpoint image. As a result, the virtual camera control device 120 can improve the probability that an appropriate virtual advertisement corresponding to the virtual camera position is displayed on the virtual viewpoint image.

[0064] In the present embodiment, as an example, a form in which only one virtual ad group is selected has been described, but multiple virtual ad groups may be selected. In this case, for example, all of the virtual advertisements included in each of the multiple selected virtual ad groups are arranged in the virtual space.

[0065] In addition, in the present embodiment, as an example, the range of the virtual camera position is specified as the recommended range, but the specification of the recommended range is not limited to the virtual camera position. For example, the recommended range may be specified by the values ​​of Pan, Tilt, and Roll, which represent the orientation of the virtual camera, or Zoom, which represents the angle of view of the virtual camera, or by combining two or more of the position, orientation, and angle of view of the virtual camera. Even in such a case, the method of determining whether or not the recommended range is within the range, that is, whether or not to set as a recommended target, is the same as the method described in the present embodiment. For example, if the value of Pan is to be registered as the recommended range, a text box for specifying the range of Pan may be added to the UI601 shown in FIG. 6. Also, in the determination of S304 shown in FIG. 3, a determination of whether or not the value of Pan of the virtual camera is included in the range of Pan specified by the operator may be added. The same applies to Tilt, Roll, and Zoom.

[0066] In addition, in the present embodiment, as an example, a form in which a virtual advertisement is recommended based on a current virtual camera position has been described, but a configuration in which a virtual advertisement is recommended in association with a camera path created in advance may be used. Here, a camera path is a record of all changes over time in virtual camera parameters, and holds virtual camera parameters at all times during a certain period. For example, when the virtual camera position of a camera path at all times is within a range of a recommendation range of a certain virtual advertisement group at a rate equal to or greater than a predetermined threshold, the virtual advertisement group is configured to be the recommendation target for the camera path. Also, for example, a virtual advertisement group corresponding to a recommendation range in which the virtual camera position of a camera path at all times is present at the highest rate may be configured to be the recommendation target.

[0067] [Embodiment 2] In the first embodiment, a configuration for recommending a virtual ad group based on the position of a virtual camera and the like has been described. However, for example, when the position, orientation, or angle of view of the virtual camera changes drastically, such as when the virtual camera is operated to follow an object moving in a virtual space, the virtual ad group to be recommended may be frequently switched. In such a case, the panels 714 to 719 of the UI 701 shown in FIG. 7 may be switched on and off at short time intervals, which may make it difficult for the operator to visually recognize the virtual ad group to be recommended. Therefore, in the present embodiment, as an example, an example of a suitable method for recommending a virtual ad group when the moving speed of the virtual camera exceeds a predetermined threshold value will be described.

[0068] FIG. 9 is a block diagram showing an example of the configuration of a control system according to the second embodiment. In this embodiment, only the differences from FIG. 1 will be described. In FIG. 2, the configurations with the same reference numerals as those in FIG. 1 are the same as those in FIG. 1. The virtual camera control device 920 has a parameter calculation unit 921, a parameter holding unit 929, and a speed calculation unit 930 in addition to the functional configuration of the virtual camera control device 120 shown in FIG. 1. In addition, in the virtual camera control device 920, the UI generation unit 127 shown in FIG. 1 is changed to a UI generation unit 927. The processes of the parameter calculation unit 921, the parameter holding unit 929, the speed calculation unit 930, and the UI generation unit 927 are realized by the CPU 201 executing the programs expanded in the RAM 203, as in the first embodiment.

[0069] The parameter calculation unit 921 has the following functions in addition to the functions of the parameter calculation unit 121 according to the first embodiment. Specifically, the parameter calculation unit 921 transmits the calculated virtual camera parameters to the parameter storage unit 929. The parameter calculation unit 921 also transmits the calculated virtual camera parameters to the speed calculation unit 930, and receives the moving speed of the virtual camera as a return value. Furthermore, the parameter calculation unit 921 adds the received information on the moving speed of the virtual camera to the virtual camera parameters transmitted to the speed calculation unit 930, and transmits the virtual camera parameters after the addition to the UI generation unit 127.

[0070] The parameter storage unit 929 receives the virtual camera parameters transmitted from the parameter calculation unit 921, and stores the received virtual camera parameters. Specifically, the parameter storage unit 929 overwrites the virtual camera parameters previously received and stored with the newly received virtual camera parameters, and stores the latest virtual camera parameters.

[0071] The speed calculation unit 930 receives virtual camera parameters corresponding to the latest frame (hereinafter referred to as the "current frame") transmitted from the parameter calculation unit 921. The speed calculation unit 930 also acquires virtual camera parameters corresponding to the immediately previous frame (hereinafter referred to as the "previous frame") held by the parameter holding unit 929. Furthermore, the speed calculation unit 930 calculates the moving speed of the virtual camera based on the virtual camera parameters of the previous frame and the virtual camera parameters of the current frame. The speed calculation unit 930 transmits information on the calculated moving speed of the virtual camera to the parameter calculation unit 921 as a return value of the virtual camera parameters received from the parameter calculation unit 921.

[0072] The moving speed of the virtual camera can be calculated by dividing the distance between the virtual camera position in the previous frame and the virtual camera position in the current frame by the period between the previous frame and the current frame (hereinafter referred to as the "frame interval"). For example, when the frame rate is 59.94 fps, the frame interval is 1 / 59.94 seconds (sec). Let F be the frame rate, (X, Y, Z) be the three-dimensional coordinates of the virtual camera position in the current frame, and (X, Y, Z) be the three-dimensional coordinates of the virtual camera position in the previous frame. p ,Y p ,Z p ), the moving speed V of the virtual camera can be calculated by the following equation (1). V={(XX p ) 2 +(YY p ) 2 +(ZZ p ) 2} 1 / 2 / (1 / F)...Equation (1)

[0073] The UI generation unit 927 has the following functions in addition to the functions of the UI generation unit 127 according to the first embodiment. Specifically, when the moving speed of the virtual camera included in the received virtual camera parameters is equal to or greater than a predetermined threshold, the UI generation unit 927 does not recommend a virtual advertisement group. That is, the UI generated in the previous frame or earlier is maintained. Note that the threshold may be a value preset by the system, or may be a value appropriately set by the operator or other users.

[0074] 10 is a flowchart showing an example of a processing flow of a process of recommending a virtual advertisement group in the virtual camera control device 920 according to embodiment 2. Note that each process shown in the flowchart is realized by loading a program code stored in the ROM 202 or the auxiliary storage device 204, etc., into the RAM 203 and executing the program by the CPU 201.

[0075] In the flowchart shown in Fig. 10, steps S301 to S311 are similar to steps S301 to S311 in the flowchart shown in Fig. 3, and therefore description thereof will be omitted. In the repeated process from S302 to S306, if it is determined that the process of S304 has been executed for all virtual advertisement groups, the virtual camera control device 920 ends the repeated process, and the UI generation unit 927 executes the process of S1001.

[0076] In S1001, the speed calculation unit 930 calculates the moving speed of the virtual camera based on the virtual camera parameters corresponding to the previous frame and the virtual camera parameters corresponding to the current frame. Information on the calculated moving speed of the virtual camera is transmitted to the parameter calculation unit 121. In addition, the parameter calculation unit 121 transmits the virtual camera parameters including the information to the UI generation unit 927.

[0077] Next, in S1002, the UI generating unit 927 judges whether the moving speed of the virtual camera calculated in S1001 is equal to or greater than the threshold. If it is judged in S1001 that the moving speed is not equal to or greater than the threshold, that is, that the moving speed is less than the threshold, the UI generating unit 927 executes the process of S307. If it is judged in S1001 that the moving speed is equal to or greater than the threshold, the UI generating unit 927 executes the process of S303 to maintain the state of the recommendation in the previous frame. That is, the UI generated in the previous frame or earlier is maintained. In this embodiment, the method of maintaining the state of the recommendation is not important. For example, the UI generating unit 927 may generate a UI that maintains the on and off states in the previous frame for each of the panels 714 to 719 of the UI 701 shown as an example in FIG. 7. After S307 or S1003, the virtual camera control device 920 executes the process of S308.

[0078] As described above, the virtual camera control device 920 is configured not to change the state of the recommendation when the moving speed of the virtual camera is equal to or greater than the threshold. According to such a virtual camera control device 920, when the virtual camera position changes drastically, it is possible to suppress frequent switching of the recommendation state of the virtual ad group. As a result, according to the virtual camera control device 920, it is possible to suppress display of a UI that makes it difficult for the operator to visually recognize the virtual ad group to be recommended. Note that, in the present embodiment, as an example, a case where the moving speed of the virtual camera is equal to or greater than the threshold has been described, but it may be configured to maintain the state of the recommendation in the previous frame when the change speed of the orientation or angle of view of the virtual camera is equal to or greater than the threshold. In addition, it may be configured to determine whether or not to maintain the state of the recommendation in the previous frame by arbitrarily combining the moving speed of the virtual camera and the change speed of the orientation and angle of view.

[0079] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0080] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0081] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0082] <Configuration 1> a parameter acquisition means for acquiring virtual camera parameters including at least any one of information on a position, a direction, and an angle of view of a virtual camera in a virtual space; a UI generating means for generating a user interface for assisting a user's input for selecting one or more virtual object groups from among a plurality of virtual object groups to which one or more virtual objects belong, based on the virtual camera parameters; a determining means for determining the virtual object group based on an input from the user selecting the virtual object group; A control device comprising:

[0083] <Configuration 2> The virtual camera parameters include at least information on the position of the virtual camera; The control device according to configuration 1, characterized in that

[0084] <Configuration 3> the determining means determines identification information that uniquely identifies the virtual object group, thereby determining the virtual object group; 3. The control device according to configuration 1 or 2, characterized in that

[0085] <Configuration 4> the parameter acquisition means acquires the virtual camera parameters by calculating them; 4. The control device according to any one of configurations 1 to 3,

[0086] <Component 5> the virtual camera is a virtual viewpoint in the virtual space when a virtual viewpoint image is generated based on a plurality of image data obtained by imaging using a plurality of imaging devices; 5. The control device according to any one of configurations 1 to 4,

[0087] <Component 6> a range acquisition means for acquiring information on at least any range of a position, a direction, or an angle of view of the virtual camera included in the virtual camera parameters corresponding to each of the plurality of virtual object groups; Further comprising: the UI generating means generates the user interface based on the virtual camera parameters and the range information; 6. The control device according to any one of configurations 1 to 5,

[0088] <Component 7> The virtual camera parameters include at least information on the position of the virtual camera, the range acquisition means acquires, as the range information, at least information indicating a range of a position of the virtual camera; The control device according to configuration 6, characterized in that

[0089] <Component 8> the range information is based on an input from the user; 8. The control device according to configuration 6 or 7, characterized in that

[0090] <Component 9> The virtual camera parameters include at least information on the position of the virtual camera, a calculation means for calculating a moving speed of the virtual camera based on information on a position of the virtual camera included in the virtual camera parameters; Further comprising: the UI generating means does not change the user interface when the moving speed of the virtual camera is equal to or greater than a threshold; 9. The control device according to any one of configurations 1 to 8, characterized in that

[0091] <Component 10> The virtual objects belonging to each of the plurality of virtual object groups are virtual advertisement objects that can be placed in the virtual space; 10. The control device according to any one of configurations 1 to 9,

[0092] <Component 11> an information acquisition means for acquiring information on each of the plurality of virtual object groups, the information including information on the virtual objects belonging to each of the plurality of virtual object groups; Information on the plurality of virtual object groups acquired by the information acquisition means; and an image generating means for generating a virtual viewpoint image based on identification information capable of uniquely identifying the virtual object group determined by the determining means and information on the virtual camera parameters;

[0033] 11. The control device according to any one of configurations 1 to 10,

[0093] <Method> a parameter acquisition step of acquiring virtual camera parameters including at least any one of information on a position, an orientation, and an angle of view of a virtual camera in a virtual space; a generating step of generating, based on the virtual camera parameters, a user interface that assists a user in input for selecting one or more virtual object groups from among a plurality of virtual object groups to which the one or more virtual objects belong; determining the virtual object group based on an input from the user selecting the virtual object group; The method according to claim 1, further comprising:

[0094] <Program> A program for causing a computer to function as the control device according to any one of configurations 1 to 11. [Explanation of symbols]

[0095] 120 Virtual Camera Control Device 121 Parameter Calculation Unit 123 Group ID determination unit 127 UI generation part

Claims

1. a parameter acquisition means for acquiring virtual camera parameters including at least any one of information on a position, a direction, and an angle of view of a virtual camera in a virtual space; a UI generating means for generating a user interface that assists a user in an input for selecting one or more virtual object groups from a plurality of virtual object groups to which the one or more virtual objects belong, based on the virtual camera parameters; a determining means for determining the virtual object group based on an input from the user selecting the virtual object group; A control device comprising:

2. The virtual camera parameters include at least information on the position of the virtual camera; The control device according to claim 1 .

3. the determining means determines identification information that uniquely identifies the virtual object group, thereby determining the virtual object group; The control device according to claim 1 .

4. the parameter acquisition means acquires the virtual camera parameters by calculating them; The control device according to claim 1 .

5. the virtual camera is a virtual viewpoint in the virtual space when a virtual viewpoint image is generated based on a plurality of image data obtained by imaging using a plurality of imaging devices; The control device according to claim 1 .

6. a range acquisition means for acquiring information on at least any range of a position, a direction, or an angle of view of the virtual camera included in the virtual camera parameters corresponding to each of the plurality of virtual object groups; Further comprising: the UI generating means generates the user interface based on the virtual camera parameters and the range information; The control device according to claim 1 .

7. The virtual camera parameters include at least information on the position of the virtual camera, the range acquisition means acquires, as the range information, at least information indicating a range of a position of the virtual camera; The control device according to claim 6 .

8. the range information is based on an input from the user; The control device according to claim 6 .

9. The virtual camera parameters include at least information on the position of the virtual camera, a calculation means for calculating a moving speed of the virtual camera based on information on a position of the virtual camera included in the virtual camera parameters; Further comprising: the UI generating means does not change the user interface when the moving speed of the virtual camera is equal to or greater than a threshold; The control device according to claim 1 .

10. The virtual objects belonging to each of the plurality of virtual object groups are virtual advertisement objects that can be placed in the virtual space; The control device according to claim 1 .

11. an information acquisition means for acquiring information on each of the plurality of virtual object groups, the information including information on the virtual objects belonging to each of the plurality of virtual object groups; Information on the plurality of virtual object groups acquired by the information acquisition means; and an image generating means for generating a virtual viewpoint image based on identification information capable of uniquely identifying the virtual object group determined by the determining means and information on the virtual camera parameters; [0033] The control device according to claim 1 .

12. a parameter acquisition step of acquiring virtual camera parameters including at least any one of information on a position, an orientation, and an angle of view of a virtual camera in a virtual space; a generating step of generating, based on the virtual camera parameters, a user interface that assists a user in input for selecting one or more virtual object groups from among a plurality of virtual object groups to which the one or more virtual objects belong; determining the virtual object group based on an input from the user selecting the virtual object group; The method according to claim 1, further comprising:

13. A program for causing a computer to function as the control device according to any one of claims 1 to 11.

Images