Control apparatus, control method, storage medium, and computer program product

Abstract

The invention provides a control apparatus, a control method, a storage medium, and a computer program product. The purpose of the present disclosure is to make it easy to specify one or more virtual objects suitable for the current virtual camera position. A control device according to the present disclosure obtains virtual camera parameters including information of at least one of a position, an orientation, and a viewing angle of a virtual camera in a virtual space. Based on the virtual camera parameters, the control device generates a user interface that assists a user in making an input for selecting one or more than one virtual object group from a plurality of virtual object groups, where the one or more than one virtual object belongs to each of the plurality of virtual object groups. Further, the control device determines the virtual object group based on an input from a user for selecting the virtual object group.

Description

Technical Field

[0001] The present disclosure relates to a user interface (UI) technology for generating a virtual viewpoint image. Background Art

[0002] There is a technology for generating a virtual viewpoint image whose viewpoint can be freely changed by generating three-dimensional shape data using data of multiple captured images obtained by synchronously capturing images by multiple camera devices installed at different positions. The technology involves capturing an image of the generated three-dimensional shape data using a virtually created camera (hereinafter referred to as a "virtual camera") to generate an image of the three-dimensional shape data as seen from any virtually created viewpoint (hereinafter referred to as a "virtual viewpoint") as a virtual viewpoint image. In such a virtual viewpoint image, virtually created advertisements (hereinafter referred to as "virtual advertisements") as virtual objects can be displayed, etc. Japanese Patent Laid-Open No. 2022-090786 discloses a technology for displaying virtual advertisements in a virtual viewpoint image based on the designation of a user (hereinafter referred to as an "operator") who operates the display of virtual advertisements in a virtual viewpoint image.

[0003] The position, orientation, size, etc. of the virtual object to be displayed in the virtual viewpoint image in the virtual space may vary. For this reason, whether virtual objects such as virtual advertisements can be efficiently displayed varies depending on at least one of the position of the virtual camera (hereinafter referred to as the "virtual camera position"), the orientation of the virtual camera, and the direction of the virtual camera. Therefore, the operator is required to specify suitable virtual objects based on the current virtual camera position, etc. and place them in the virtual space. However, it is difficult to instantaneously determine and specify suitable virtual objects while operating the virtual camera (for example, moving the virtual camera position). The present disclosure is made in view of the above-mentioned problems, and an object of the present disclosure is to provide a technology that allows an operator to easily specify (one or more than one) virtual objects suitable for the current virtual camera position, etc. Summary of the invention

[0004] The control device according to the present disclosure includes: a parameter obtaining unit, which is used to obtain virtual camera parameters including information on at least one of the position, orientation and viewing angle of the virtual camera in the virtual space; a user interface generating unit, which is used to generate a user interface based on the virtual camera parameters, wherein the user interface assists a user in making inputs for selecting one or more virtual object groups from a plurality of virtual object groups, wherein one or more virtual objects belong to each of the plurality of virtual object groups; and a determining unit, which is used to determine the virtual object group based on inputs from the user for selecting the virtual object group.

[0005] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram showing an example of the configuration of a control system according to Embodiment 1;

[0007] Figure 2 is a block diagram showing an example of a hardware configuration of devices included in a control system according to Embodiment 1;

[0008] Figure 3 is a flowchart showing an example of the flow of processing of the virtual camera control device according to Embodiment 1;

[0009] Figure 4 is a flowchart showing an example of the flow of a process for determining a virtual advertisement group identifier (ID) by the virtual camera control device according to Embodiment 1;

[0010] Figure 5 is a flowchart showing an example of the flow of processing for determining a recommended range by the virtual camera control apparatus according to Embodiment 1;

[0011] Figure 6 is a diagram showing an example of a UI for designating a recommendation range generated by a UI generation unit according to Embodiment 1;

[0012] Figure 7 An example of a UI for specifying a virtual advertisement group generated by the UI generating unit according to the first embodiment is shown;

[0013] Fig. 8A and Figure 8B is a flowchart showing an example of the flow of processing of the image generating apparatus according to Embodiment 1;

[0014] Fig. 9 is a block diagram showing an example of a configuration of a control system according to Embodiment 2; and

[0015] Fig.10 : is a flowchart showing an example of the flow of processing of the virtual camera control device according to Embodiment 2. DETAILED DESCRIPTION

[0016] The present disclosure describes some typical embodiments in detail below with reference to the accompanying drawings. The configurations shown in the following embodiments are merely exemplary, and some embodiments of the present disclosure are not limited to the configurations shown schematically. Note that the same reference numerals will be used to describe the same components.

[0017] [Example 1]

[0018] Figure 1is a block diagram showing an example of the configuration of a control system according to Embodiment 1. Figure 1 Functional components of a virtual camera control device 120 and an image generation device 104 are also shown. The imaging device group 101 includes a plurality of imaging devices, and the respective imaging devices are placed at different positions from each other. The respective imaging devices are configured to capture images of at least a portion of an imaging target space, and perform imaging in synchronization with each other. Data of a plurality of captured images obtained by synchronous imaging by the imaging devices (hereinafter referred to as "multi-viewpoint image data") are sent to the shape data generation device 102.

[0019] The shape data generating device 102 receives the multi-viewpoint image data sent from the camera device group 101, extracts the image area covering the object to be generated with three-dimensional shape data as the foreground from each captured image, and generates a contour image indicating the contour of the foreground. In addition, the shape data generating device 102 generates three-dimensional shape data representing the three-dimensional shape of the foreground by using a visual hull (Visual Hull) based on the generated contour image. By the way, the method of extracting the foreground from each captured image and the method of generating three-dimensional shape data by Visual Hull are well known, and their description is omitted. The three-dimensional shape data generated by the shape data generating device 102 is sent to the shape data storage device 103. The shape data storage device 103 has a large-capacity storage device such as a hard disk drive (HDD), and receives the three-dimensional shape data sent from the shape data generating device 102 and stores it in the storage device.

[0020] The virtual camera control device 120 is a device used by an operator to operate a virtual camera for generating a virtual viewpoint image. The virtual camera control device 120 has a parameter calculation unit 121, a time code calculation unit 122, a group ID determination unit 123, a group ID holding unit 124, and a recommended range determination unit 125 as functional components. In addition to the above-mentioned functional components, the virtual camera control device 120 also has a recommended range holding unit 126, a UI generation unit 127, and a virtual camera information transmission unit 128.

[0021] The parameter calculation unit 121 receives operation information output from the controller 141 in response to the operator's operation of the controller 141, and calculates virtual camera parameters including information related to the position, orientation, and viewing angle of the virtual camera based on the operation information. The virtual camera parameters calculated by the parameter calculation unit 121 are sent to the UI generation unit 127 and the virtual camera information sending unit 128. In one example, in the case where the virtual camera position is represented by three-dimensional coordinates [X, Y, Z] in the virtual space, the parameter calculation unit 121 calculates values ​​such as X=4.0, Y=9.0, and Z=1.5 as the virtual camera position. It should be noted that in the present embodiment, the unit of each coordinate is meter [m], and the position of the origin is the center of the range in the virtual space where the three-dimensional shape data is to be generated. In addition, in the case where the X-axis is designated to be parallel to the ground, the Y-axis is designated to be parallel to the ground and perpendicular to the X-axis, and the Z-axis is designated to be perpendicular to the ground.

[0022] In addition, in the case where the orientation of the virtual camera is represented by three angles [Pan, Tilt, Roll], the parameter calculation unit 121 calculates values ​​such as Pan=20.0, Tilt=10.0, and Roll=2.0 as the orientation of the virtual camera. In the present embodiment, the unit of the angle is degree [°], and the value range of Pan, Tilt, and Roll is from -180 to 180. Note that Pan is an angle rotated in a direction parallel to the ground, Tilt is an angle rotated in a direction perpendicular to the ground, and Roll is an angle rotated around the optical axis of the virtual camera as the rotation axis. In addition, in the case where the angle of view of the virtual camera is represented by the focal length [Zoom], the parameter calculation unit 121 calculates a value such as Zoom=6.0 as the angle of view of the virtual camera. In the present embodiment, the unit of the focal length is millimeter [mm]. The controller 141 includes a joystick, etc. In the present embodiment, the operator operates the joystick of the controller 141 to operate the virtual camera. For example, the virtual camera position changes depending on the amount of joystick tilt and its direction.

[0023] The time code calculation unit 122 receives operation information output from the controller 141 in response to the operator's operation of the controller 141, and calculates the time code of the virtual viewpoint image based on the operation information. The calculated time code is sent to the virtual camera information sending unit 128. The controller 141 includes a shuttle key, etc. In the present embodiment, in one example, the operator operates the shuttle key of the controller 141 to input an instruction such as playing, pausing, rewinding or fast-forwarding the virtual viewpoint image and determine the time code.

[0024] The group ID determination unit 123 determines the virtual advertisement group ID based on the designation by the operator. The virtual advertisement group ID is an identifier for uniquely identifying a virtual advertisement group for collectively processing one or more virtual advertisements. Incidentally, in the present embodiment, examples of a method by which the operator designates the virtual advertisement group ID include the following methods. For example, using a mouse or the like, the operator presses a button displayed on the display 140 to be described later. Figure 7 One of buttons 708 to 713 in UI 701 is exemplarily shown to designate a virtual advertising group. Figure 7 The example of UI 701 shown represents a state in which the operator has pressed button 709, so that the virtual advertising group corresponding to button 709 has been specified, and as a result of the specification, 2 has been determined as the virtual advertising group ID. The virtual advertising group ID determined by group ID determination unit 123 is sent to group ID holding unit 124.

[0025] The group ID holding unit 124 receives the virtual advertisement group ID sent from the group ID determination unit 123, and stores and holds it in a storage device such as an HDD or a random access memory (RAM). Figure 7 In a case where 2 has been determined as the virtual advertisement group ID as in UI 701 shown, group ID holding unit 124 holds 2.

[0026] Recommended range determination unit 125 determines a recommended range for a virtual advertising group based on the operator's designation. The recommended range indicates a range of a virtual camera position determined individually for a virtual advertising group. In a case where the virtual camera position is within the range of the recommended range, a virtual advertising group corresponding to the recommended range is recommended to the operator. Information on the recommended range determined by recommended range determination unit 125 is sent to recommended range holding unit 126. Incidentally, in the present embodiment, examples of a method by which the operator designates a recommended range include the following methods. For example, using a mouse and a keyboard, the operator specifies a virtual advertising group displayed on display 140 to be described later. Figure 6 A numerical value is input in any of the text box groups 602 to 606 in the exemplarily shown UI 601. Based on the input, a recommendation range corresponding to the virtual advertisement group is specified.

[0027] Figure 6The example of UI 601 shown shows a state in which 1 is specified as the virtual ad group ID of text box group 602. The example also shows a state in which 3.0 to 5.0 in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction are specified as the recommended range of the virtual ad group having virtual ad group ID 1. In this case, a cubic space measuring 2.0 m in the X-axis direction, 15.0 m in the Y-axis direction, and 20.0 m in the Z-axis direction is set as the recommended range at the specified position. Note that the method used by the operator to specify the recommended range is not limited to the operator using Figure 6 The operator can specify the recommended range in a graphical manner by deforming a three-dimensional object such as a cube object in the three-dimensional space displayed as the UI 601 shown in the figure. Note that the present embodiment will be described on the assumption that the shape of the recommended range is a cube, but the shape of the recommended range is not limited to the cube shape, and may be any shape including any polyhedral shape and spherical shape, etc. In addition, each recommended range does not have to be specified as a closed space surrounded by a finite shape, and may be specified as a space that is partially open.

[0028] Recommendation range holding unit 126 receives the information of the recommendation range of each virtual advertisement group transmitted from recommendation range determination unit 125 , and stores and holds it in a storage device such as a HDD or a RAM.

[0029] The UI generation unit 127 generates a UI to be presented to the operator. The UI generated by the UI generation unit 127 is displayed on the display 140, and the operator can operate the UI displayed on the display 140 using a mouse and / or a keyboard, etc. Examples of the UI generated by the UI generation unit 127 include the following: Figure 6 The UI 601 shown and Figure 7 UI 701 shown, etc. UI generation unit 127 obtains information of the recommended range of each virtual advertisement group held by recommendation range holding unit 126. UI generation unit 127 also receives the virtual camera parameters sent from parameter calculation unit 121. UI generation unit 127 generates a UI for recommending virtual advertisement groups based on the recommended range of each virtual advertisement group and the virtual camera parameters.

[0030] Specifically, when the virtual camera position used to generate the virtual viewpoint image of the current processing target is within the recommendation range of the virtual advertising group, UI generation unit 127 generates a UI recommending the virtual advertising group. Figure 7In the exemplary UI 701, whether a virtual advertisement group is recommended is indicated based on whether each panel 714 to 719 is lit. Figure 7 When one of the panels 714 to 719 in the UI 701 is lit, it indicates that the virtual advertisement group corresponding to the panel is recommended. Figure 7 The illustrated UI 701 indicates a state in which virtual ad groups having virtual ad group IDs 1, 2, and 5 are recommended.

[0031] The virtual camera information sending unit 128 obtains the virtual advertisement group ID held by the group ID holding unit 124. Furthermore, the virtual camera information sending unit 128 receives the virtual camera parameters sent from the parameter calculation unit 121 and the time code sent from the time code calculation unit 122. Furthermore, the virtual camera information sending unit 128 generates virtual camera information including the obtained virtual advertisement group ID and the received virtual camera parameters and time code, and sends the generated virtual camera information to the image generating device 104.

[0032] The virtual advertisement generating device 106 generates data of (one or more) virtual advertisements created by the user as data of (one or more) virtual objects using multifunctional application software capable of creating data of virtual objects as three-dimensional objects. The data of at least one virtual object as a virtual advertisement generated by the virtual advertisement generating device 106 is sent to the image generating device 104.

[0033] The image generating device 104 generates a virtual viewpoint image and outputs a signal representing the generated virtual viewpoint image to the display 105 to display the virtual viewpoint image on the display 105. The image generating device 104 has an image generating unit 110, a group registering unit 111, and a group holding unit 112.

[0034] The image generation unit 110 obtains three-dimensional shape data representing the three-dimensional shapes of the foreground and the background from the shape data storage device 103. The image generation unit 110 also receives virtual camera information transmitted from the virtual camera information transmission unit 128. The image generation unit 110 generates a virtual viewpoint image based on the obtained three-dimensional shape data and the received virtual camera information, and outputs a signal representing the generated virtual viewpoint image to the display 105. Note that the virtual camera information includes virtual camera parameters including information such as the position, orientation, and viewing angle of the virtual camera, a time code, and a virtual advertising group ID.

[0035] The process of generating a virtual viewpoint image includes the following processes. The first is a process of obtaining the three-dimensional shape data of the foreground and background corresponding to the time code included in the virtual camera information from the shape data storage device 103. The second is a process of obtaining the data of the virtual advertising group corresponding to the virtual advertising group ID included in the virtual camera information from the group holding unit 112. The third is a process of generating a virtual viewpoint image based on the obtained three-dimensional shape data of the foreground and background, the data of the virtual advertising group obtained from the group holding unit 112, and the virtual camera parameters included in the virtual camera information. Specifically, the process generates an image of the three-dimensional shape of the foreground and background represented by the three-dimensional shape data and the virtual advertisement included in the virtual advertising group as a virtual object as seen from the virtual viewpoint determined based on the virtual camera parameters, as a virtual viewpoint image. By the way, the output destination of the virtual viewpoint image is not necessarily limited to the display 105, and the data of the virtual viewpoint image may be output to, for example, an image distributor or a broadcasting station, etc.

[0036] The group registration unit 111 receives the data of the virtual advertisement as the virtual object transmitted from the virtual advertisement generating device 106, and registers the received data of the virtual advertisement in the virtual advertisement group. Specifically, the group registration unit 111 associates the received data of the virtual advertisement with the virtual advertisement group ID, and transmits them to the group holding unit 112. The group holding unit 112 receives the data of the virtual advertisement and the virtual advertisement group ID from the group registration unit 111. The group holding unit 112 associates the received data of the virtual object with the received virtual advertisement group ID, and stores and holds them in a storage device such as an HDD or a RAM. Note that in the present embodiment, the data of the virtual advertisement as the virtual object includes the position of the virtual object in the virtual space, the shape of the virtual object, and the information of the content represented by the virtual object, but is not limited to these. For example, the configuration may be such that the data of the virtual object includes only the information of the above-mentioned position and shape, and the information of the above-mentioned content may be specified separately. Here, in the case where the virtual object is a virtual advertisement, the information of the content, for example, refers to information such as an image representing the content of the advertisement.

[0037] Figure 2 is a block diagram showing an example of the hardware configuration of devices included in the control system according to Embodiment 1. Figure 2 The information processing apparatus 200 shown implements the shape data generating apparatus 102, the shape data storing apparatus 103, the image generating apparatus 104, the virtual advertisement generating apparatus 106, and the virtual camera controlling apparatus 120. The information processing apparatus 200 may have a CPU 201, a read only memory (ROM) 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a system bus 208.

[0038] The CPU 201 controls the information processing device 200 as a whole by using computer programs and data stored in the ROM 202 or the RAM 203, etc., to realize Figure 1 201. The functions of each device included in the system shown. Note that the information processing device 200 may have one or more dedicated hardware in addition to the CPU 201, and at least a part of the processing of the CPU 201 may be performed by the dedicated hardware. Examples of such dedicated hardware include an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), and the like. The ROM 202 stores programs that do not need to be changed, etc. The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204 and data supplied from the outside via the communication I / F 207, etc. The auxiliary storage device 204 is a hard disk drive, etc., and stores various data such as image data or sound data.

[0039] The display unit 205 includes a liquid crystal display or a light emitting diode (LED) display, etc., and displays a graphical user interface (GUI) used by a user to give instructions to the information processing apparatus 200. The operation unit 206 includes a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 201 in response to receiving an operation from the user. The CPU 201 operates as a display control unit for controlling the display unit 205 and an operation control unit for controlling the operation unit 206.

[0040] The communication I / F 207 is used for communication between the information processing device 200 and external devices such as the camera device group 101, the displays 105 and 140, the controller 141, and other information processing devices 200. In the case where the information processing device 200 is connected to the external device by wire, a communication cable is connected to the communication I / F 207. In the case where the information processing device 200 has a function of wirelessly communicating with the external device, the communication I / F 207 includes an antenna. The system bus 208 connects the components of the information processing device 200 and transmits information to and from these components. By the way, in the present embodiment, the display unit 205 and the operation unit 206 exist in the information processing device 200, but at least one of the display unit 205 and the operation unit 206 may exist as a separate device outside the information processing device 200.

[0041] Now refer to Figures 3 to 5 The operation of the virtual camera control device 120 will be described. Figure 31 is a flowchart showing an example of the processing flow of the process of recommending a virtual advertising group to the operator by the virtual camera control apparatus 120 according to Embodiment 1. Note that each symbol "S" in the following description represents a step. In addition, the processing shown in the flowchart is realized by loading a program code stored in the ROM 202 or the auxiliary storage device 204 or the like into the RAM 203 and causing the CPU 201 to execute the program.

[0042] Note that S301 and S311 represent repeated processing. The virtual camera control device 120 repeatedly performs the processing of S302 to S310 for each frame in the virtual viewpoint image generated by the image generation device 104. In this embodiment, the frame rate of the virtual viewpoint image is set to 59.94fps (frames per second). That is, when the virtual viewpoint image is replayed, a series of processing from S302 to S310 is performed at intervals of 1 / 59.94 seconds. The interval of the series of processing from S302 to S310 depends on the frame rate of the virtual viewpoint image.

[0043] First, in S302, the parameter calculation unit 121 calculates virtual camera parameters including information on the position, orientation, and viewing angle of the virtual camera based on the operation information from the controller 141 operated by the operator. For example, in the case where the operator operates a controller including a joystick to operate the virtual camera, the parameter calculation unit 121 calculates the three-dimensional position of the virtual camera based on the tilt amount of the joystick and its orientation. Note that S303 and S306 represent repeated processing. The virtual camera control device 120 repeatedly performs the processing of S304 and S305 for each virtual advertising group. That is, the processing of S304 and S305 is repeatedly performed after S302.

[0044] First, in S304, the UI generation unit 127 determines whether the virtual camera position determined based on the virtual camera parameters calculated in S302 exists in the recommendation range of the virtual advertisement group. Figure 5 To illustrate the determination method, at the beginning Figure 3 The recommendation range of each virtual advertisement group is determined in advance before the processing of the flowchart shown in FIG. Figure 3 Information on the recommendation range of each virtual advertisement group is obtained in advance at any timing such as before the process of the flowchart shown is started.

[0045] A specific example of the determination method will be described based on a case where the virtual camera position is expressed by three-dimensional coordinates [X, Y, Z] and their values ​​are X = 4.0, Y = 9.0, and Z = 1.5. Figure 6In the case of UI 601 shown, 3.0 to 5.0 in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction are specified as recommended ranges for the virtual ad group having virtual ad group ID 1 in text box group 602. The virtual camera position in the X-axis direction, which is 4.0, falls within the range X of 3.0 to 5.0 specified in text box group 602. Similarly, the virtual camera position in the Y-axis direction, which is 9.0, falls within the range Y of 0.0 to 15.0, and the virtual camera position in the Z-axis direction, which is 1.5, falls within the range Z of 0.0 to 20.0.

[0046] This means that all virtual camera positions in the X-axis, Y-axis, and Z-axis directions fall within the recommended range. Thus, the UI generation unit 127 determines that the virtual camera exists within the recommended range of the virtual advertising group with the virtual advertising group ID 1. Therefore, in the case where the X-coordinate, Y-coordinate, and Z-coordinate of the virtual camera position fall within all of the ranges X, Y, and Z, respectively, the UI generation unit 127 determines that the virtual camera position exists within the recommended range. On the other hand, in the case where the virtual camera position does not fall within one or more of the ranges X, Y, and Z, the UI generation unit 127 determines that the virtual camera position does not exist within the recommended range. In S304, the UI generation unit 127 similarly makes a determination for each virtual advertising group for which a recommended range has been specified through the above-mentioned repeated processing.

[0047] Note that, like the range X in the text box group 603, the upper limit value of the range does not necessarily need to be specified. In this case, the UI generation unit 127 assumes that as long as the virtual camera position in the X-axis direction is -15.0 or more than -15.0 as indicated by the range X in the text box group 603, the virtual camera position in the X-axis direction exists in the recommended range in the X-axis direction. This also applies similarly to the range Y and the range Z. Likewise, the lower limit value of at least one of the range X, the range Y, and the range Z does not necessarily need to be specified. In this case, as long as the virtual camera position in the axis direction for which the lower limit value of the range is not specified is the upper limit value or less than the upper limit value, it is considered that the virtual camera position in the axis direction exists in the recommended range in the axis direction.

[0048] In addition, like the range Z in the text box group 604, it is not necessary to specify both the lower limit value and the upper limit value of the range. For example, in the case where both the upper limit value and the lower limit value of the range Z are not specified, the UI generation unit 127 only determines whether the virtual camera position in the X-axis direction and the Y-axis direction exists in the recommended range. Then, the UI generation unit 127 determines that the virtual camera position is within the recommended range when the X coordinate of the virtual camera position is within the range X and the Y coordinate of the virtual camera position is within the range Y. Similarly, for the range X and the range Y, the determination is made in the same way when both the lower limit value and the upper limit value are not specified. In addition, like the text box group 605, it is not necessary to specify the upper limit value and the lower limit value for any of the range X, the range Y, and the range Z. In this case, the UI generation unit 127 determines that the virtual camera position exists in the recommended range regardless of the virtual camera position.

[0049] If it is determined in S304 that the virtual camera position exists within the recommendation range, in S305, the UI generation unit 127 sets the virtual advertising group as the recommendation target. After S305 or in the case where it is determined in S304 that the virtual camera position does not exist within the recommendation range, the UI generation unit 127 determines whether the process of S304 has been performed on all virtual advertising groups for which the recommendation range has been specified. If it is determined that the process of S304 has not been performed on at least one of the virtual advertising groups, the UI generation unit 127 performs the process of S304 on the virtual advertising groups for which the process of S304 has not been performed through the above-mentioned repeated processing. If it is determined that the process of S304 has been performed on all virtual advertising groups, the virtual camera control device 120 terminates the repeated processing, and the UI generation unit 127 performs the process of S307.

[0050] In S307, the UI generation unit 127 generates a UI for recommending all virtual advertisement groups set as recommendation targets in S305. Figure 7 On the UI 701 shown, the operator can visually identify the recommended virtual advertising group based on whether the corresponding panels 714 to 719 are lit. Specifically, for example, Figure 7 The illustrated example of UI 701 indicates a state in which virtual ad groups having virtual ad group IDs 1 , 2 , and 5 are recommended.

[0051] Next, in S308, the time code calculation unit 122 calculates the time code of the virtual viewpoint image based on the operation information from the controller 141 operated by the operator. For example, the operator operates the shuttle key of the controller 141 to input an instruction to play, pause, rewind or fast-forward the virtual viewpoint image or other similar actions. The time code calculation unit 122 receives the operation information corresponding to the input sent from the controller 141, and calculates the time code based on the operation information. Next, in S309, the virtual camera control device 120 performs a process of determining the virtual advertisement group ID based on the operator's designation. This will be described later by using Figure 4 The details of this determination processing will be described below.

[0052] Next, in S310, the virtual camera information transmission unit 128 transmits the virtual camera information to the image generation device 104. Specifically, in the process of S310, the virtual camera information transmission unit 128 obtains the virtual advertising group ID determined in S309. In addition, in the process of S310, the virtual camera information transmission unit 128 obtains the virtual camera parameters calculated by the parameter calculation unit 121 in S302 and the time code calculated by the time code calculation unit 122 in S308. In addition, in the process of S310, the virtual camera information transmission unit 128 generates virtual camera information including the obtained virtual advertising group ID, virtual camera parameters, and time code, and transmits it to the image generation unit 110.

[0053] As described above, the virtual camera control apparatus 120 repeatedly executes the processing of S302 to S310, for example, until the operator inputs an instruction to terminate the processing of the virtual camera control apparatus 120. In this case, the virtual camera control apparatus 120 terminates the processing in response to the input from the operator. Figure 3 Processing of the flowchart shown.

[0054] Figure 4 1 is a flowchart showing an example of the flow of processing for determining a virtual advertising group ID based on the designation of the operator by the virtual camera control apparatus 120 according to Embodiment 1. Specifically, the flowchart shows an example of the flow of processing in S309, and the processing of the flowchart is performed after S308. Note that the processing shown in the flowchart is implemented by loading a program code stored in the ROM 202 or the auxiliary storage device 204 or the like into the RAM 203 and causing the CPU 201 to execute the program.

[0055] First, in S401, group ID determination unit 123 determines a virtual advertisement group ID based on a designation by an operator. For example, using a mouse, the operator selects and presses a button to be described later. Figure 7 One of buttons 708 to 713 in UI 701 is exemplarily shown to specify a desired virtual advertising group ID. Figure 7 The illustrated example of UI 701 indicates a state in which the operator has selected and pressed button 709. In this state, group ID determination unit 123 determines 2 as the virtual advertisement group ID.

[0056] Next, in S402, group ID holding unit 124 receives the virtual advertisement group ID determined in S401 and holds the virtual advertisement group ID. Figure 7 In the example of UI 701 shown, group ID holding unit 124 holds 2 as the virtual advertisement group ID. Note that in the process of S310 of obtaining the virtual advertisement group ID determined in S309, virtual camera information transmission unit 128 obtains the virtual advertisement group ID held by group ID holding unit 124 in S402. After S402, virtual camera control device 120 terminates Figure 4 The process of the flowchart shown in FIG. 1 is then performed, and the process of S310 is then performed.

[0057] Figure 5 1 is a flowchart showing an example of the flow of processing for determining the recommendation range of each virtual advertising group based on the operator's designation by the virtual camera control device 120 according to Embodiment 1. Note that the processing of this flowchart is performed, for example, Figure 3 In addition, the program code stored in the ROM 202 or the auxiliary storage device 204 is loaded into the RAM 203 and the CPU 201 executes the program. Figure 5 The process shown in the flowchart shown.

[0058] First, in S501, recommendation range determination unit 125 determines the recommendation range of the virtual advertisement group based on the operator's designation. For example, using a mouse and a keyboard, the operator specifies a virtual advertisement group to be described later. Figure 6 A desired numerical value is input in any of the text box groups 602 to 606 in the exemplarily illustrated UI 601 to specify a recommendation range of a desired virtual advertisement group. Figure 6 The example of text box group 602 in the illustrated UI 601 indicates a state in which 1 is designated as the virtual ad group ID. The example also indicates a state in which 3.0 to 5.0 in the X-axis direction, 0.0 to 15.0 in the Y-axis direction, and 0.0 to 20.0 in the Z-axis direction are designated as the recommendation range of the virtual ad group having the virtual ad group ID 1. Based on the designation, the recommendation range determination unit 125 determines the recommendation range of the virtual ad group having the virtual ad group ID 1.

[0059] Next, in S502, the recommended range holding unit 126 receives the information of the recommended range of each virtual advertisement group determined in S501 and holds the information. After S502, the virtual camera control device 120 terminates the operation. Figure 5 Note that in the process such as Figure 3 The information on the recommendation range of each virtual advertisement group held in S502 is obtained by UI generation unit 127 at any timing such as before the process of the illustrated flowchart is started.

[0060] Figure 6 6 is a diagram showing an example of a UI 601 for an operator to designate a recommendation range for each virtual advertisement group, generated by the UI generation unit 127 according to Embodiment 1. Note that in the present disclosure, the display form of the UI 601 is not limited. For example, the UI 601 may be displayed as a single window, or may be displayed as part of a single UI including the UI 601 and another UI for controlling a virtual camera.

[0061] Text box groups 602 to 606 are areas used by the operator to specify recommendation ranges of respective virtual advertising groups by using means for inputting operations such as a mouse and a keyboard, etc. On UI 601, a recommendation range of a virtual advertising group having a virtual advertising group ID 1 is specified in text box group 602. Likewise, recommendation ranges of virtual advertising groups having virtual advertising group IDs 2, 3, and 5 are specified in text box groups 603, 604, and 605, respectively. Text box group 606 is in a state where no virtual advertising group ID is input and a recommendation range is not specified for any virtual advertising group.

[0062] In one example, the operator inputs and specifies a virtual ad group ID and a recommended range of virtual ad groups corresponding to a desired virtual ad group ID, and then presses Figure 6 In response to pressing the confirmation button or the like as a trigger, the recommended range determination unit 125 determines the recommended range based on Figure 6 The information set in the text box group 602 to 606 in the illustrated UI 601 determines the recommendation scope of each virtual advertisement group.

[0063] Figure 7 701 is a diagram showing an example of a UI 701 for an operator to designate a virtual advertisement group to be placed in a virtual space, generated by the UI generation unit 127 according to Embodiment 1. Note that in the present disclosure, the display form of the UI 701 is not limited. For example, the UI 701 may be displayed as a single window, or as part of a UI including the UI 701 and another UI for controlling a virtual camera.

[0064] Images 702 to 707 each indicate the layout of one or more virtual advertisements as virtual objects belonging to a virtual advertising group when the virtual objects are placed in a virtual space. Referring to images 702 to 707, the operator selects a virtual advertising group. Virtual advertising group IDs of the virtual advertising groups corresponding to images 702 to 707 are displayed above images 702 to 707, respectively. Each of images 702 to 707 may be automatically generated, for example, based on information on the position of one or more virtual advertisements (virtual objects) belonging to the corresponding virtual advertising group in the virtual space. Alternatively, each of images 702 to 707 may be an image created individually and manually by a creator of (one or more) virtual advertisements, etc. Incidentally, in Figure 7 Only the shapes and layouts of virtual advertisements are shown as simplified information in images 702 to 707 shown in UI 701 of FIG. 1 , but the virtual advertisements in images 702 to 707 may include the contents of these virtual advertisements. In this case, the automatically generated images 702 to 707 are generated based on, for example, information on the positions of virtual advertisements belonging to the corresponding virtual advertisement group in the virtual space and information on the contents of the virtual advertisements.

[0065] Buttons 708 to 713 are intended for the operator to specify the virtual advertisement group ID of the virtual advertisement group that the operator wishes to display in the virtual viewpoint image. In one example, each of buttons 708 to 713 can be selected by a click operation with a mouse. With this operation, the operator selects a button corresponding to the virtual advertisement group that the operator wishes to display in the virtual viewpoint image from buttons 708 to 713. In response to the specification by the operation, group ID determination unit 123 determines the virtual advertisement group ID corresponding to the selected button.

[0066] Note that the method used by the operator to specify the virtual ad group ID in the present disclosure is not limited to this method. For example, the operator can specify the virtual ad group ID by selecting one of images 702 to 707, or by directly inputting the value of the virtual ad group ID using a keyboard. Figure 7 The illustrated example of UI 701 indicates a state in which button 709 is selected, that is, a state in which a virtual advertisement group having virtual advertisement group ID 2 is selected. In this state, one or more virtual advertisements included in the virtual advertisement group having virtual advertisement group ID 2 may be displayed in the virtual viewpoint image.

[0067] Each of panels 714 to 719 indicates whether the corresponding virtual advertising group is a recommendation target. For example, among the multiple virtual advertising groups corresponding to panels 714 to 719, the virtual advertising group corresponding to the lit panels indicates the virtual advertising group set as the recommendation target, that is, the recommended virtual advertising group. The virtual advertising group corresponding to the unlit panels indicates the virtual advertising group not set as the recommendation target, that is, the unrecommended virtual advertising group. Figure 7 In the example of UI 701 shown, panels 714, 715, and 718 are lit, and panels 716, 717, and 719 are not lit. That is, the example of UI 701 indicates a state in which virtual advertisement groups corresponding to virtual advertisement group IDs 1, 2, and 5 are recommended.

[0068] Will refer to Fig. 8A and Figure 8B The operation of the image generating device 104 will be described. Fig. 8A 1 is a flowchart showing an example of the flow of a process in which image generating device 104 according to Embodiment 1 registers a virtual advertisement group. Figure 8B 1 is a flowchart showing an example of the flow of a process in which the image generating device 104 of Embodiment 1 generates a virtual viewpoint image.

[0069] Will refer to Fig. 8A The following describes the process of image generation device 104 registering a virtual advertisement group. First, in S801, group registration unit 111 receives data of one or more virtual objects as virtual advertisements sent from virtual advertisement generation device 106, and registers the received data of one or more virtual objects as data of the virtual advertisement group. Specifically, group registration unit 111 registers the data of one or more virtual objects as virtual advertisements and a virtual advertisement group ID specified by a user of virtual advertisement generation device 106 in association with each other. Next, in S802, group holding unit 112 receives the data of the virtual objects associated with the virtual advertisement group ID registered in S801, and holds it as the virtual advertisement group. After S802, image generation device 104 terminates. Fig. 8A Note that the processing in the flowchart is Figure 8B Any timing before the processing of the illustrated flowchart is started is performed in advance.

[0070] Now refer to Figure 8B The flow of the process of generating a virtual viewpoint image by the image generating device 104 is described. For example, the process of repeatedly performing the process for each frame in the virtual viewpoint image is repeated. Figure 8B1. The process of the flowchart shown in FIG. 11 is performed. First, in S811, the image generation unit 110 receives the virtual camera information transmitted from the virtual camera control device 120 (specifically, from the virtual camera information transmission unit 128). Next, in S812, the image generation unit 110 obtains the three-dimensional shape data by reading out the three-dimensional shape data of the foreground and the background corresponding to the time code included in the virtual camera information received in S811 from the shape data storage device 103. Next, in S813, the image generation unit 110 obtains the data of the virtual advertising group corresponding to the virtual advertising group ID included in the virtual camera information received in S811 from the group holding unit 112.

[0071] Next, in S814, the image generation unit 110 generates a virtual viewpoint image. Specifically, the image generation unit 110 generates a virtual viewpoint image based on the virtual camera parameters included in the virtual camera information received in S811, the three-dimensional shape data of the foreground and background obtained in S812, and the data of the virtual advertising group obtained in S813. Note that the virtual viewpoint image generated in S814 is a frame in the virtual viewpoint image corresponding to the time code included in the virtual camera information received in S811. In addition, the virtual viewpoint image generated in S814 is displayed on the display 105. After S814, the image generation device 104 terminates Figure 8B The process of the flowchart shown is performed and is repeatedly executed each time new virtual camera information is transmitted from the virtual camera control apparatus 120.

[0072] As described above, a recommended range is specified for each virtual advertising group, and the virtual camera control device 120 is configured to recommend the virtual advertising group to the operator if the virtual camera position exists within the recommended range of the virtual advertising group. Such a virtual camera control device 120 enables the operator to easily and instantaneously determine which virtual advertising group is to be selected to be displayed in the virtual viewpoint image. Therefore, the virtual camera control device 120 increases the possibility of displaying (one or more than one) virtual advertisements suitable for the virtual camera position in the virtual viewpoint image.

[0073] Note that in this embodiment, the configuration of selecting only a single virtual advertisement group has been described as an example, but multiple virtual advertisement groups may be selected. In this case, for example, all virtual advertisements included in the selected multiple virtual advertisement groups are placed in the virtual space.

[0074] In addition, in the present embodiment, the configuration of designating the range of the virtual camera position as the recommended range has been described as an example, but the designation of the recommended range is not limited to the virtual camera position. For example, the values ​​of Pan, Tilt, and Roll indicating the orientation of the virtual camera or the value of Zoom indicating the angle of view of the virtual camera may be designated as the recommended range, or a combination of at least two of the position, orientation, and angle of view of the virtual camera may be designated as the recommended range. Although in this case, the method of determining whether any of the above parameters is within the recommended range (that is, whether to set the corresponding virtual advertising group as a recommendation target) is similar to the method described in the present embodiment. For example, Figure 6 The UI 601 shown adds a text box for specifying the range of Pan so that the Pan value can be registered as a recommended range. Figure 3 In the determination of S304 shown, it may be additionally determined whether the Pan value of the virtual camera is included in the Pan range specified by the operator. This also applies similarly to Tilt, Roll, and Zoom.

[0075] In addition, in the present embodiment, a configuration for recommending virtual advertisements based on the current virtual camera position has been described as an example. Alternatively, a configuration for recommending virtual advertisements in association with individual camera paths created in advance may be adopted. Here, a "camera path" is a record of all changes in virtual camera parameters over time, and the virtual camera parameters at all times in a given period are maintained. For example, the configuration is such that: in a case where the ratio of virtual camera positions throughout all times along the camera path as described above existing in the recommended range of the virtual advertisement group is a preset threshold or greater than a preset threshold, the virtual advertisement group is set as a recommendation target for the camera path. Alternatively, the configuration may be such that: for example, a virtual advertisement group corresponding to a recommendation range having the highest ratio of the presence of virtual camera positions throughout all times along the camera path may be set as a recommendation target.

[0076] [Example 2]

[0077] In Embodiment 1, a configuration has been described in which a virtual advertising group is recommended based on the position of a virtual camera, etc. Here, for example, in a case where the position, orientation, or viewing angle of a virtual camera changes drastically, such as in a case where a virtual camera is operated to follow an object moving inside a virtual space, etc., a virtual advertising group set as a recommendation target may be frequently switched. In this case, Figure 7Panels 714 to 719 in the illustrated UI 701 will become lit and unlit back and forth at short time intervals. This may present a UI that makes it difficult for the operator to visually recognize the virtual advertising group set as a recommendation target. In view of this, in Embodiment 2, a description will be given of an example of a preferred method of recommending a virtual advertising group, for example, in a case where the moving speed of the virtual camera exceeds a preset threshold.

[0078] Fig. 9 1 is a block diagram showing an example of the configuration of a control system according to Embodiment 2. Incidentally, in this embodiment, only the Figure 1 Different characteristics. Fig. 9 In, with Figure 1 The same reference numerals denote components similar to those in the Figure 1 The components shown are the same. The virtual camera control device 920 has the same Figure 1 In addition to the functional components included in the virtual camera control device 120 shown in FIG. 1 , a parameter calculation unit 921, a parameter holding unit 929, and a speed calculation unit 930 are also provided. In addition, in the virtual camera control device 920, Figure 1 The UI generating unit 127 shown is changed to a UI generating unit 927. As in Embodiment 1, the processing of the parameter calculating unit 921, the parameter holding unit 929, the speed calculating unit 930, and the UI generating unit 927 is realized by causing the CPU 201 to execute the program loaded to the RAM 203.

[0079] The parameter calculation unit 921 has the following functions in addition to the functions of the parameter calculation unit 121 according to Embodiment 1. Specifically, the parameter calculation unit 921 sends the calculated virtual camera parameters to the parameter holding unit 929. The parameter calculation unit 921 also sends the calculated virtual camera parameters to the speed calculation unit 930, and receives the moving speed of the virtual camera as a return value of the parameter. In addition, the parameter calculation unit 921 adds the information of the received moving speed of the virtual camera to the virtual camera parameters sent to the speed calculation unit 930, and sends the virtual camera parameters after the addition to the UI generation unit 927.

[0080] The parameter holding unit 929 receives the virtual camera parameters sent from the parameter calculation unit 921 and holds the received virtual camera parameters. Specifically, the parameter holding unit 929 overwrites the virtual camera parameters it previously received and held with the newly received virtual camera parameters and holds the latest virtual camera parameters.

[0081] The speed calculation unit 930 receives the virtual camera parameters corresponding to the latest frame (hereinafter referred to as "current frame") transmitted from the parameter calculation unit 921. In addition, the speed calculation unit 930 obtains the virtual camera parameters corresponding to the immediately previous frame (hereinafter referred to as "previous frame") held by the parameter holding unit 929. In addition, the speed calculation unit 930 calculates the moving speed of the virtual camera based on the virtual camera parameters in the previous frame and the virtual camera parameters in the current frame. The speed calculation unit 930 transmits information of the calculated moving speed of the virtual camera to the parameter calculation unit 921 as a return value of the virtual camera parameters received from the parameter calculation unit 921.

[0082] The moving speed of the virtual camera can be calculated by dividing the distance between the virtual camera position in the previous frame and the virtual camera position in the current frame by the time period between the previous frame and the current frame (hereinafter referred to as the "frame interval"). For example, in the case where the frame rate is 59.94fps, the frame interval is 1 / 59.94 seconds. With F as the frame rate, (X, Y, Z) as the three-dimensional coordinates of the virtual camera position in the current frame, and (X p , Y p , Z p ) as the three-dimensional coordinates of the virtual camera position in the previous frame, the moving speed of the virtual camera represented by V can be calculated by the following formula (1).

[0083] V={(XX p ) 2 +(YY p ) 2 +(ZZ p ) 2} 1 / 2 / (1 / F) Formula (1)

[0084] The UI generation unit 927 has the following functions in addition to the functions of the UI generation unit 127 according to Embodiment 1. Specifically, the UI generation unit 927 does not recommend a virtual advertising group when the moving speed of the virtual camera included in the received virtual camera parameters is a preset threshold or greater than a preset threshold. Specifically, the UI generation unit 927 maintains the UI generated for the previous frame or before the previous frame. Incidentally, the threshold value may be a value preset by the system, or a value appropriately set by an operator or another user.

[0085] Fig.101 is a flowchart showing an example of the processing flow of the process of recommending a virtual advertising group to the operator by the virtual camera control apparatus 920 according to Embodiment 2. Note that the process shown in the flowchart is implemented by loading a program code stored in the ROM 202 or the auxiliary storage device 204 or the like into the RAM 203 and causing the CPU 201 to execute the program.

[0086] exist Fig.10 In the flowchart shown, S301 to S311 are related to Figure 3 If it is determined that the process of S304 has been executed for all virtual advertising groups in the repeated process from S302 to S306, the virtual camera control device 920 terminates the repeated process, and the UI generation unit 927 executes the process of S1001.

[0087] In S1001, the speed calculation unit 930 calculates the moving speed of the virtual camera based on the virtual camera parameters in the previous frame and the virtual camera parameters in the current frame. The information of the calculated moving speed of the virtual camera is sent to the parameter calculation unit 921. In addition, the parameter calculation unit 921 sends the virtual camera parameters including the information to the UI generation unit 927.

[0088] Next, in S1002, the UI generation unit 927 determines whether the moving speed of the virtual camera calculated in S1001 is a threshold value or greater than the threshold value. If it is determined in S1002 that the moving speed is not above the threshold value, that is, it is less than the threshold value, the UI generation unit 927 performs the processing of S307. If it is determined in S1002 that the moving speed is the threshold value or greater than the threshold value, then in S1003, the UI generation unit 927 maintains the recommended state in the previous frame. Specifically, the UI generation unit 927 maintains the UI generated for the previous frame or before the previous frame. In the present embodiment, the method of maintaining the recommended state is not limited. For example, the UI generation unit 927 can generate the following UI, which is to maintain the lit or unlit state of each panel among panels 714 to 719 for the previous frame. Figure 7 The UI 701 is exemplarily shown. After S307 or S1003, the virtual camera control device 920 performs the process of S308.

[0089] As described above, the virtual camera control device 920 is configured not to change the recommended state when the moving speed of the virtual camera is a threshold value or greater than the threshold value. Such a virtual camera control device 920 prevents the recommended state of the virtual advertising group from being frequently switched when the position of the virtual camera changes drastically. Therefore, the virtual camera control device 920 prevents the display operator from visually recognizing the UI of the virtual advertising group set as the recommendation target. Note that in the present embodiment, the case based on whether the moving speed of the virtual camera is a threshold value or greater than the threshold value has been described as an example. Alternatively, the configuration may be such that: when the rate of change of the orientation or the angle of view of the virtual camera is a threshold value or greater than the threshold value, the recommended state in the previous frame is maintained. Also alternatively, the configuration may be such that: whether to maintain the recommended state in the previous frame is determined based on any combination of the moving speed of the virtual camera and the rate of change of the orientation and the rate of change of the angle of view.

[0090] [Other embodiments]

[0091] Some (one or more) embodiments of the present disclosure may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above (one or more) embodiments and / or including a system or device of one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above (one or more) embodiments and by the following method, wherein the computer of the system or device performs the above method by, for example, reading out and executing computer executable instructions from a storage medium to perform one or more functions in the above (one or more) embodiments and / or controlling one or more circuits to perform one or more functions in the above (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. For example, computer executable instructions may be provided to the computer from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage unit of a distributed computer system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)). TM etc.), one or more of a flash storage device and a memory card, etc.

[0092] According to the present disclosure, the operator can easily specify (one or more than one) virtual object suitable for the current virtual camera position.

[0093] Although the present disclosure has described exemplary embodiments, it is to be understood that some embodiments of the present disclosure are not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. A control device, comprising: a parameter obtaining unit for obtaining virtual camera parameters including information on at least one of a position, an orientation, and a viewing angle of the virtual camera in a virtual space; a user interface generating unit, configured to generate a user interface based on the virtual camera parameters, the user interface assisting a user in making an input for selecting one or more virtual object groups from a plurality of virtual object groups, wherein the one or more virtual objects belong to each virtual object group in the plurality of virtual object groups; as well as A determination unit is configured to determine the virtual object group based on an input from a user for selecting the virtual object group.

2. The control device according to claim 1, wherein: The virtual camera parameters include at least information about the position of the virtual camera.

3. The control device according to claim 1, wherein: The determination unit determines each virtual object group in the virtual object groups by determining identification information for uniquely identifying the virtual object group.

4. The control device according to claim 1, wherein: The parameter obtaining unit obtains the virtual camera parameters through calculation.

5. The control device according to claim 1, wherein: The virtual camera is a virtual viewpoint in the virtual space when a virtual viewpoint image is generated based on a plurality of image data obtained by imaging by a plurality of imaging devices.

6. The control device according to claim 1, further comprising a range obtaining unit, the range obtaining unit being configured to obtain, for each virtual object group in the plurality of virtual object groups, information on a range of at least one of a position, an orientation, and a viewing angle of the virtual camera included in the virtual camera parameters, in, The user interface generation unit generates the user interface based on the virtual camera parameters and the information of the range.

7. The control device according to claim 6, wherein: The virtual camera parameters include at least information about the position of the virtual camera, and The range obtaining unit obtains at least information indicating the range of the position of the virtual camera as the range information.

8. The control device according to claim 6, wherein: The range information is based on information input from the user.

9. The control device according to claim 1, wherein: The virtual camera parameters include at least information about the position of the virtual camera, and The control device further includes a calculation unit configured to calculate a moving speed of the virtual camera based on information on a position of the virtual camera included in the virtual camera parameters, and The user interface generation unit does not change the user interface if the moving speed of the virtual camera is a threshold value or more.

10. The control device according to claim 1, wherein: Each of the virtual objects belonging to the plurality of virtual object groups is an object in the form of a virtual advertisement that can be placed in the virtual space.

11. The control device according to claim 1, further comprising: an information obtaining unit, configured to obtain information related to each virtual object group in the plurality of virtual object groups, the information including information related to virtual objects belonging to each virtual object group in the plurality of virtual object groups; as well as An image generating unit is configured to generate a virtual viewpoint image based on the information related to the plurality of virtual object groups obtained by the information obtaining unit, identification information capable of uniquely identifying the virtual object group determined by the determining unit, and the information of the virtual camera parameter.

12. A control method comprising the following steps: obtaining virtual camera parameters including information of at least one of a position, an orientation, and a viewing angle of the virtual camera in a virtual space; generating a user interface based on the virtual camera parameters, the user interface assisting a user in making an input for selecting one or more virtual object groups from a plurality of virtual object groups, wherein the one or more virtual objects belong to each of the plurality of virtual object groups; and The group of virtual objects is determined based on input from a user for selecting the group of virtual objects.

13. A storage medium storing a program for causing a computer to execute the control method according to claim 12.

14. A computer program product comprising a program for causing a computer to execute the control method according to claim 12.

Images