Information processing device, control method for information processing device, and program

Abstract

To generate a virtual viewpoint image with desired information added at a desired position.SOLUTION: The information processing device disclosed herein generates parameters for generating a virtual viewpoint image with additional information displayed thereon. The information processing device sets an additional information display condition, identification information of the additional information, and virtual viewpoint information of the virtual viewpoint image to be generated as parameters and outputs the set parameters.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] Recently, a technology that generates a virtual viewpoint image (also called a free viewpoint video) that shows the view from any virtual viewpoint by using the multiple viewpoint images obtained by installing multiple cameras at different positions and capturing images synchronously from multiple viewpoints has been attracting attention. With such a technology, for example, it becomes possible to watch highlight scenes of a soccer or basketball game from various angles, and it becomes possible to give the user a high sense of realism compared to normal video content.

[0003] Adding various information to such virtual viewpoint images is being considered. Patent Document 1 discloses a technique for adding preset advertising information to a specific area of ​​a free viewpoint video when the viewpoint condition corresponding to the free viewpoint video is the same as a specific viewpoint condition that has been preset. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-41259 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, even if there is an area in the frame of a free viewpoint video where advertising information can be added, if the viewpoint condition corresponding to this free viewpoint video is different from a specific viewpoint condition that has been set in advance, it is not possible to add the advertising information.

[0006] An object of the present disclosure is to provide a technique for generating a virtual viewpoint image to which desired information is added at a desired position. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure is an information processing device that generates parameters for generating a virtual viewpoint image displaying additional information, and is characterized in having a setting means that sets display conditions for the additional information, identification information indicating the additional information, and virtual viewpoint information of the virtual viewpoint image to be generated as the parameters, and an output means that outputs the parameters set by the setting means. Effect of the Invention

[0008] According to the present disclosure, it is possible to generate a virtual viewpoint image to which desired information is added at a desired position. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram for explaining an image processing system. [Diagram 2] FIG. 1 shows an example of the hardware configuration of the device. [Diagram 3] A diagram explaining the generation of a virtual viewpoint image with a virtual advertisement [Figure 4] 1 is a flowchart showing a flow of processing executed by an image processing system. [Diagram 5] A sequence diagram showing a process flow between an information processing device and an image generating device. [Figure 6] FIG. 1 shows an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image. [Figure 7] A diagram showing an example of an obj file specified in the 3D model format of virtual advertising data [Figure 8] A diagram showing an example of a user interface [Figure 9] A diagram showing an example of a user operation screen. [Figure 10] A sequence diagram showing a process flow between an information processing device and an image generating device. [Figure 11] FIG. 13 is a diagram showing an example of a virtual advertisement image displayed on a virtual viewpoint image. [Figure 12] A diagram showing an example of a user interface [Figure 13] Diagram explaining the creation of virtual ad sets [Figure 14] FIG. 13 is a diagram showing an example of a virtual advertisement image displayed on a virtual viewpoint image. [Figure 15] A diagram showing an example of a user interface [Figure 16] FIG. 1 shows an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the following embodiment does not necessarily limit the present invention. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0011] [First embodiment] In this embodiment, an image processing system for generating a virtual viewpoint image displaying a virtual advertisement, which is a type of additional information that does not exist in the virtual viewpoint image to be added, will be described with reference to the drawings. Fig. 1 is a diagram for explaining the image processing system according to this embodiment, in which Fig. 1(a) shows an example of the configuration of the image processing system, and Fig. 1(b) shows a functional block of the image processing system.

[0012] The image processing system of this embodiment includes an information processing device 110, an image generating device 120 connected to the information processing device 110, and a user terminal 140 connected to the image generating device 120. The connections between the devices may be wired or wireless.

[0013] The information processing device 110 is a device that provides parameters for generating a virtual viewpoint image in which additional information is displayed. Specifically, the information processing device 110 is a device that issues an instruction to register virtual advertisement data, sets information on the display conditions of the set virtual advertisement data, a time code, and virtual viewpoint information to parameters, and transmits the information to the image generating device 120. The information processing device 110 has an operation receiving unit 111, a virtual advertisement setting unit 112, a registration instruction unit 113, a parameter setting unit 114, and a parameter transmission unit 118.

[0014] The operation reception unit 111 receives the input of a user operation (also called user input). Information generated according to the received input operation is sent to each functional unit according to the operation content. The virtual advertisement setting unit 112 sets virtual advertisement data to be synthesized with the virtual viewpoint video (added onto the virtual viewpoint image) according to the information sent from the operation reception unit 111. The virtual advertisement data is, for example, image data of an advertising signboard that does not exist in the virtual viewpoint image (basic 3D data) to be added. The virtual advertisement data to be set may be one or more, and may be one type or multiple types. Examples of the virtual advertisement data to be set include data of an image (see FIG. 3(b)) generated based on a 3D model of a virtual advertisement, which will be described later in detail. The virtual advertisement data set by the virtual advertisement setting unit 112 is sent to the registration instruction unit 113 and the display condition setting unit 115 of the parameter setting unit 114.

[0015] The registration instruction unit 113 instructs the image generating device 120 to register the virtual advertisement data set by the virtual advertisement setting unit 112. Specifically, the registration instruction unit 113 sends instruction information instructing the registration of the virtual advertisement data together with the set virtual advertisement data to the registration receiving unit 121 of the image generating device 120, and instructs the registration receiving unit 121 to register the sent virtual advertisement data in the image generating device 120.

[0016] The parameter setting unit 114 has a display condition setting unit 115, a time code setting unit 116, and a virtual viewpoint setting unit 117. The parameter setting unit 114 sets various information in virtual viewpoint image generation parameters (hereinafter also referred to as parameters) in units of frames of a virtual viewpoint image generated by the image generating device 120, in accordance with an input operation received by the operation receiving unit 111. That is, the parameters are information in units of frames of a virtual viewpoint image generated by the image generating device 120, and are information in which various information is set. The parameters in which various information is set are sent to a parameter sending unit 118.

[0017] The display condition setting unit 115 selects the virtual advertisement data to be set from the virtual advertisement data set by the virtual advertisement setting unit 112 according to the information received from the operation receiving unit 111. Then, the display condition setting unit 115 sets the display condition of the selected virtual advertisement data and identification information that uniquely indicates the selected virtual advertisement as parameters. The display condition includes the display start time and the display period of the selected virtual advertisement.

[0018] The time code setting unit 116 sets the time code for generating the virtual viewpoint image as a parameter in accordance with the information received from the operation receiving unit 111. The time code to be set corresponds to a time code linked to basic 3D data corresponding to the virtual viewpoint image, the details of which will be described later.

[0019] According to the information received from the operation receiving unit 111, the virtual viewpoint setting unit 117 sets virtual viewpoint information including the position and orientation of the virtual camera, which is the virtual viewpoint, and the angle of view corresponding to the virtual camera, as parameters.

[0020] The parameter transmission unit 118 outputs parameters in which information such as conditions has been set by the display condition setting unit 115, the time code setting unit 116, and the virtual viewpoint setting unit 117, to the outside of the information processing device 110. Specifically, the parameter transmission unit 118 transmits the parameters in which various information has been set to the parameter receiving unit 123 of the image generating device 120. Note that the parameter transmission unit 118 may transmit the parameters one by one or may transmit multiple parameters together. A set of multiple consecutive parameters from a certain time to another certain time is also called a virtual camera path because the trajectory of the virtual viewpoint forms a path.

[0021] The image generating device 120 is a device that generates a virtual viewpoint image in which a virtual advertisement is displayed frame by frame according to parameters received from the information processing device 110. In other words, the image generating device 120 can be said to be a device that generates a virtual viewpoint image in which a desired virtual advertisement is added at a desired position. The image generating device 120 has a registration receiving unit 121, a virtual advertisement storage unit 122, a parameter receiving unit 123, a parameter extracting unit 124, a virtual advertisement reading unit 128, a basic data storage unit 129, a basic data reading unit 130, a synthesis unit 131, and a rendering unit 132.

[0022] The registration reception unit 121 receives instruction information and virtual advertisement data instructed to be registered from the registration instruction unit 113 of the information processing device 110. The registration reception unit 121 sends the received virtual advertisement data to the virtual advertisement storage unit 122 to register the virtual advertisement data in accordance with the instruction information.

[0023] The virtual advertisement storage unit 122 stores the virtual advertisement data sent from the registration acceptance unit 121 .

[0024] The parameter receiving unit 123 acquires parameters in which a display condition of additional information to be displayed in a virtual viewpoint image, identification information uniquely indicating the additional information, virtual viewpoint information of a virtual viewpoint image to be generated, and a time code are set. Specifically, the parameter receiving unit 123 receives parameters transmitted from the parameter transmitting unit 118 of the information processing device 110.

[0025] The parameter extraction unit 124 has a display condition extraction unit 125, a time code extraction unit 126, and a virtual viewpoint extraction unit 127. The parameter extraction unit 124 extracts, from the parameters received by the parameter receiving unit 123, various pieces of information set in each functional unit.

[0026] The display condition extraction unit 125 extracts, from the parameters received by the parameter receiving unit 123, identification information of the virtual advertisement data set by the display condition setting unit 115 and information on the display conditions of the virtual advertisement data, which are included in the parameters. The identification information of the virtual advertisement data and the information on the display conditions of the virtual advertisement data are sent to the virtual advertisement reading unit 128.

[0027] The virtual advertisement reading unit 128 reads out the virtual advertisement data corresponding to the identification information extracted from the parameters by the display condition extraction unit 125 from the virtual advertisement storage unit 122. The read out virtual advertisement data is sent to the synthesis unit 131.

[0028] The basic data storage unit 129 stores 3D data of a basic model (hereinafter referred to as basic 3D data). The basic 3D data is data obtained by 3D modeling an object that actually exists. Note that, since a virtual viewpoint image is generated using the basic 3D data, the basic 3D data can also be said to be shape data indicating the shape of an object included in the generated virtual viewpoint image.

[0029] The time code extraction unit 126 extracts the time code contained in the parameter from the parameter received by the parameter receiving unit 123. The extracted time code is data set by the time code setting unit 116, and matches the time code of the basic 3D data. The extracted time code is sent to the basic data reading unit 130. Note that the parameters from which the time code is extracted are the same as the parameters from which the display condition extraction unit 125 extracts.

[0030] The basic data reading unit 130 reads out the basic 3D data that corresponds to (is related to) the time code extracted by the time code extraction unit 126 from the basic data storage unit 129. The read basic 3D data is sent to a synthesis unit 131.

[0031] The synthesis unit 131 synthesizes the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 to generate one piece of synthesized 3D data. The generated piece of synthesized 3D data is sent to the rendering unit 132.

[0032] The virtual viewpoint extraction unit 127 extracts virtual viewpoint information (virtual camera position information) indicating the position and orientation of the virtual camera, and the angle of view corresponding to the virtual camera, contained in the parameters received by the parameter receiving unit 123. The extracted virtual viewpoint information is sent to the rendering unit 132. Note that the parameters to be extracted for the virtual viewpoint information are the same as the parameters to be extracted by the display condition extraction unit 125.

[0033] The rendering unit 132 performs rendering processing on the composite 3D data generated by the composition unit 131 according to the virtual viewpoint information extracted by the virtual viewpoint extraction unit 127, and generates a virtual viewpoint image in which a virtual advertisement is displayed (virtual viewpoint image with virtual advertisement). The virtual viewpoint information includes the position and orientation of the virtual camera, and the angle of view corresponding to the virtual camera. The generated virtual viewpoint image with virtual advertisement is sent to a user terminal 140 or the like outside the image generation device 120.

[0034] The user terminal 140 is a device that displays the virtual viewpoint image with a virtual advertisement sent from the image generating device 120. The user terminal 140 may be an information processing device such as a personal computer including a liquid crystal display, a tablet terminal including a touch panel, or a smartphone.

[0035] Note that, although an image processing system having the information processing device 110, the image generating device 120, and the user terminal 140 has been described as an example, the present invention is not limited to this. The information processing device 110 may include the functions of the image generating device 120, the functions of the user terminal 140, or the functions of both the image generating device 120 and the user terminal 140. The image generating device 120 may include the functions of the user terminal 140.

[0036] <Hardware configuration of each device> The hardware configurations of the information processing device 110, the image generating device 120, and the user terminal 140 will be described with reference to the drawings. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device 110, the image generating device 120, and the user terminal 140. The information processing device 110, the image generating device 120, and the user terminal 140 have a common hardware configuration, and include a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication unit 217, and a bus 218.

[0037] The CPU 211 realizes the functions of the information processing device 110 and the image generating device 120 shown in FIG. 1(b) by controlling the entire image processing system using computer programs and data stored in the ROM 212 and the RAM 213. The information processing device 110 and the image generating device 120 may have one or more dedicated hardware different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication unit 217. The auxiliary storage device 214 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

[0038] The display unit 215 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to operate the information processing device 110. The operation unit 216 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user and inputs various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216.

[0039] The communication unit 217 is used for communication between the information processing device 110, the image generating device 120, and the user terminal 140 and devices external to the information processing device 110, the image generating device 120, and the user terminal 140. For example, when the information processing device 110, the image generating device 120, and the user terminal 140 are connected to the external devices by wire, a communication cable is connected to the communication unit 217. When the information processing device 110, the image generating device 120, and the user terminal 140 have a function of wirelessly communicating with the external devices, the communication unit 217 includes an antenna. The bus 218 connects each unit of the information processing device 110, the image generating device 120, and the user terminal 140 to transmit information.

[0040] In this embodiment, the display unit 215 and the operation unit 216 are assumed to exist inside the information processing device 110, but at least one of the display unit 215 and the operation unit 216 may exist as a separate device outside the information processing device 110. The display unit 215 and the operation unit 216 may be integrated.

[0041] <Generation of virtual viewpoint images with virtual advertisements> The generation of a virtual viewpoint image with a virtual advertisement according to a virtual viewpoint based on 3D data (three-dimensional data) will be explained using diagrams. FIG. 3 is a diagram explaining the generation of a virtual viewpoint image with a virtual advertisement according to a virtual viewpoint based on 3D data corresponding to an image (video) of a soccer game. FIG. 3(a) shows an example of a virtual viewpoint image generated according to a virtual viewpoint based on basic 3D data. FIG. 3(b) shows an example of an image of virtual advertisement data according to the virtual viewpoint of the virtual viewpoint image of FIG. 3(a). FIG. 3(c) shows an example of a virtual viewpoint image with a virtual advertisement obtained by combining the virtual viewpoint image of FIG. 3(a) and the image of the virtual advertisement data of FIG. 3(b).

[0042] 3(a) is an image of an actually existing object viewed from an arbitrary viewpoint (virtual viewpoint), and includes a field 311, players 312 on the field, and an advertising billboard 313 installed on the side of the field 311. The virtual viewpoint image 310 is an image generated according to a virtual viewpoint based on basic 3D data stored in advance in the basic data storage unit 129. The basic 3D data is, for example, shape data indicating the shape of an object generated by a known technology based on a plurality of captured images obtained by capturing an object area from different viewpoints using a plurality of imaging devices.

[0043] 3(b) is an image of a virtual advertisement that can be displayed in an area of ​​the virtual viewpoint image 310 that does not overlap with objects that are the focus of the viewer's attention, such as players 312 and advertising billboards 313 that are present in the virtual viewpoint image 310. That is, the image 320 of the virtual advertisement data includes two billboards 321 and 322 that are arranged side by side along the halfway line of the field 311 in the virtual viewpoint image 310. The billboard 321 is an image of a virtual advertisement with the texture "Abcde" written center-justified. The billboard 322 is an image of a virtual advertisement with the texture "ABCDE" written center-justified.

[0044] 3(c) is an image generated by the rendering unit 132 according to a virtual viewpoint based on composite 3D data obtained by combining basic 3D data and virtual advertisement data by the combining unit 131. Therefore, by preparing virtual advertisement data to be displayed in the virtual viewpoint image, it is possible to generate a virtual viewpoint image with a virtual advertisement in which an advertising billboard appears to exist near the halfway line of the field, a place that does not actually exist.

[0045] <Image processing system operation> Next, the flow of processing executed by the image processing system will be described with reference to the drawings. FIG. 4 is a flowchart showing the flow of processing executed by the image processing system. FIG. 4(a) shows the flow of processing executed by the image processing system, FIG. 4(b) shows a detailed flow of processing for setting parameters for generating a virtual viewpoint image, and FIG. 4(c) shows a detailed flow of processing for generating a virtual viewpoint image with a virtual advertisement. The CPU 211 reads out and executes a computer program stored in the ROM 212 or the auxiliary storage device 214, thereby performing the processing shown in FIG. 4. Hereinafter, in the description of FIG. 4, the processing steps will be simply denoted as S.

[0046] As shown in Fig. 4(a), in S401, the information processing device 110 of the image processing system accepts a user operation and sets various pieces of information generated according to the accepted operation content to parameters. A detailed flow of the setting process will be described later with reference to the drawings. The parameters in which various pieces of information have been set are transmitted to the image generating device 120 of the image processing system.

[0047] In S402, the image generating device 120 of the image processing system generates a virtual viewpoint image with a virtual advertisement according to various information set in the parameters. That is, the image generating device 120 generates a virtual viewpoint image with a virtual advertisement according to the virtual viewpoint information set in the parameters based on the composite 3D data obtained by combining the basic 3D data corresponding to the time code set in the parameters and the virtual advertisement data indicated by the identification information. A detailed flow of the generation process will be described later with reference to the drawings. The generated virtual viewpoint image with a virtual advertisement is sent to a display, which is the display unit 215 of the image generating device 120, a user terminal 140 connected to the image generating device 120, or the like.

[0048] Next, a detailed flow of the process of setting parameters for generating a virtual viewpoint image will be described with reference to the drawings. As shown in FIG. 4(b), in S411, the virtual advertisement setting unit 112 reads virtual advertisement data created in advance. The destination of the virtual advertisement data may be a virtual advertisement database (not shown) connected to the information processing device 110, or a USB memory connected to the information processing device 110. The read virtual advertisement data is sent to the registration instruction unit 113 and the display condition setting unit 115.

[0049] In S 412 , the registration instruction unit 113 transmits the target virtual advertisement data to the image generation device 120 together with instruction information for registering the virtual advertisement data read by the virtual advertisement setting unit 112 in the virtual advertisement storage unit 122 of the image generation device 120 .

[0050] In S413, the operation receiving unit 111 receives a user operation for inputting parameters for generating a virtual viewpoint image.

[0051] In S414, the operation reception unit 111 judges whether or not an instruction to display a virtual advertisement has been included in the information generated according to the input operation received by the operation reception unit 111 in S413. If the judgment result indicates that an instruction to display a virtual advertisement has been included in the information generated according to the input operation (YES in S414), the process proceeds to S415. On the other hand, if the judgment result indicates that an instruction to display a virtual advertisement has not been included in the information generated according to the input operation (NO in S414), S415 is skipped and the process proceeds to S416.

[0052] In S415, the display condition setting unit 115 sets the display conditions of the virtual advertisement data to be selected and the identification information of the virtual advertisement to be selected in the parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation received by the operation receiving unit 111 in S413.

[0053] In S416, the time code setting unit 116 sets a time code in the parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation received by the operation receiving unit 111 in S413.

[0054] In S417, the virtual viewpoint setting unit 117 sets virtual viewpoint information in the parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation accepted by the operation accepting unit 111 in S413. Note that the virtual viewpoint information includes the position and orientation of the virtual camera, and the angle of view corresponding to the virtual camera.

[0055] In S418, the parameter sending unit 118 sends to the parameter receiving unit 123 of the image generating device 120 the parameters in which the various information generated in the processes from S415 to S417 is set.

[0056] In S419, the information processing device 110 judges whether or not a new parameter has been input. If the judgment result indicates that a new parameter has been input (YES in S419), the process returns to S413, and a series of processes from S413 to S418 are executed for the input new parameter. On the other hand, if the judgment result indicates that a new parameter has not been input (NO in S419), the flow shown in FIG. 4(b) is terminated.

[0057] Next, a detailed flow of the process of generating a virtual viewpoint image with a virtual advertisement will be described with reference to the drawings. As shown in FIG. 4(c), in S421, the registration receiving unit 121 receives the target virtual advertisement data together with the instruction information transmitted from the registration instruction unit 113 of the information processing device 110.

[0058] In S422, the virtual advertisement storage unit 122 stores the virtual advertisement data accepted by the registration acceptance unit 121 in S421. As a result, the virtual advertisement data is registered in the image generation device 120.

[0059] In S423, the parameter receiving unit 123 receives the parameters transmitted by the parameter transmitting unit 118 in S418.

[0060] In S424, the basic data reading unit 130 reads out basic 3D data that corresponds to the time code extracted from the parameters by the time code extraction unit 126 of the parameter extraction unit 124, from the basic data storage unit 129. Note that the parameters from which the time code is extracted are the parameters received by the parameter receiving unit 123 in S423.

[0061] In S425, the display condition extraction unit 125 of the parameter extraction unit 124 judges whether or not the parameters received by the parameter receiving unit 123 in S423 include information on the display conditions of the virtual advertisement data. If the judgment result indicates that the parameters include information on the display conditions of the virtual advertisement data (YES in S425), the process proceeds to S426. On the other hand, if the judgment result indicates that the parameters do not include information on the display conditions of the virtual advertisement data (NO in S425), S426 and S427 are skipped and the process proceeds to S428.

[0062] In S426, the virtual advertisement reading unit 128 reads out virtual advertisement data corresponding to the identification information extracted from the parameters by the display condition extraction unit 125 from the virtual advertisement storage unit 122. Note that the parameters from which the identification information is extracted are the parameters received by the parameter receiving unit 123 in S423.

[0063] In S427, the synthesis unit 131 synthesizes the basic 3D data read by the basic data reading unit 130 in S424 and the virtual advertisement data read by the virtual advertisement reading unit 128 in S426 to generate synthesized 3D data.

[0064] In S428, the rendering unit 132 performs a rendering process on the composite 3D data generated in S427 or the basic 3D data read out in S424 according to the virtual viewpoint information extracted from the parameters by the virtual viewpoint extraction unit 127. This process generates a virtual viewpoint image with a virtual advertisement or a virtual viewpoint image. The generated virtual viewpoint image with a virtual advertisement or the virtual viewpoint image is sent as an image signal to the user terminal 140 or the like outside the image generating device 120. Note that the parameters from which the virtual viewpoint information is extracted are the parameters received by the parameter receiving unit 123 in S423.

[0065] In S429, the image generating device 120 judges whether or not new parameters have been received. If it is judged that new parameters have been received (YES in S429), the process returns to S423, and the series of processes from S423 to S428 are executed for the received new parameters. On the other hand, if it is judged that no new parameters have been received (NO in S429), the flow shown in FIG. 4(c) ends.

[0066] In the flow of Fig. 4(a) and Fig. 4(b), the parameter setting process including the reading and registration of the virtual advertisement data has been described, but the present invention is not limited to this. In other words, the virtual advertisement data may be read and registered in advance, and the parameter setting process including the setting of the display conditions of the virtual advertisement may be performed.

[0067] In addition, in the flow of Fig. 4(a) and Fig. 4(c), the process of generating a virtual viewpoint image including the reception and storage of an instruction to register virtual advertisement data has been described, but the present invention is not limited to this. That is, the reception and storage of an instruction to register virtual advertisement data may be performed in advance, and then the process of generating a virtual viewpoint image with a virtual advertisement may be performed.

[0068] <Processing flow between devices in the image processing system> The flow of processing between devices in an image processing system will be described with reference to the drawings. Fig. 5 is a sequence diagram showing the flow of processing between an information processing device 110 and an image generating device 120 in the image processing system. Hereinafter, in the description of Fig. 5 and Fig. 10 described later, the processing sequence will simply be represented as S.

[0069] In S501, the information processing device 110 (the registration instruction unit 113) instructs the image generating device 120 to register the virtual advertisement data "A" set by the virtual advertisement setting unit 112. S501 corresponds to the process of S412 in the flowchart of FIG. 4(b) and the process of S421 in the flowchart of FIG. 4(c). Here, an example of virtual advertisement data will be described with reference to the drawings. FIG. 6 is a diagram showing an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image, and FIG. 6(a) shows an example of a 3D model format of the virtual advertisement data transmitted at the time of registration. As shown in FIG. 6(a), in the 3D model format of the virtual advertisement data, the file names and file sizes of the obj file, the mtl file, and the png file are specified. The obj file stores information such as vertices and normals of the 3D model (three-dimensional model) of the virtual advertisement. The mtl (material) file stores color information and texture information of the 3D model of the virtual advertisement. The png file stores parameters of the 3D model of the virtual advertisement.

[0070] Here, the obj file will be described with reference to the drawings. FIG. 7 is a diagram showing an example of an obj file specified in the 3D model format of virtual advertisement data. In the obj file, the vertex coordinate value is specified by the keyword "v", the texture coordinate value is specified by the keyword "vt", and the normal vector is specified by the keyword "vn". In this embodiment, the obj file is used as an example of a file format that represents information related to the position, size, and orientation of the 3D model of the virtual advertisement data, but the file format is not limited to this. The file format may be any other format as long as it can represent the position, size, and orientation of the 3D model of the virtual advertisement data.

[0071] Returning to the description of FIG. 5, in S502, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 the parameters (without virtual advertisement) for generating a virtual viewpoint image without a display instruction for a virtual advertisement, which are set by the parameter setting unit 114. Here, an example of the parameter (without virtual advertisement) data for generating a virtual viewpoint image will be described with reference to the drawings. FIG. 6(b) is a diagram showing an example of the format of the parameter (without virtual advertisement) data for generating a virtual viewpoint image (hereinafter also referred to as parameter (without virtual advertisement) data). In the format of the parameter (without virtual advertisement) data, the position of the virtual camera is specified by "rotation quaternion" and the direction of the virtual camera is specified by "translation vector". In the format of the parameter (without virtual advertisement) data, further, information indicating the angle of view corresponding to the virtual camera is specified by "horizontal angle", and a time code is specified by "time code". That is, in the format of the parameter (without virtual advertisement) data shown in FIG. 6(b), virtual viewpoint information is specified, but information regarding the virtual advertisement is not specified.

[0072] In S503, similarly to S502, the information processing device 110 transmits to the image generating device 120 the parameters (without virtual advertisement) that are set by the parameter setting unit 114 and that do not include an instruction to display a virtual advertisement.

[0073] Here, an example of a virtual viewpoint image generated according to the extracted virtual viewpoint information by the image generating device 120 from the parameters (without virtual advertisement) in S502 and S503 will be described with reference to the drawings. When the rendering unit 132 performs rendering processing on the basic 3D data according to the virtual viewpoint information extracted from the parameters (without virtual advertisement), a virtual viewpoint image without an added virtual advertisement is generated as shown in FIG. 3(a). However, the time code extracted from the parameters (without virtual advertisement) received in S503 is different from the time code extracted from the parameters (without virtual advertisement) received in S502. That is, the parameters received in S503 and the parameters received in S502 target different frames.

[0074] In S504, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 the parameters for generating a virtual viewpoint image (virtual advertisement "A") accompanied by a display instruction of a virtual advertisement, which are set by the parameter setting unit 114. Here, an example of the parameter (virtual advertisement "A") data for generating a virtual viewpoint image will be described with reference to the drawings. FIG. 6(c) is a diagram showing an example of the format of the parameter (virtual advertisement "A") data for generating a virtual viewpoint image (hereinafter also referred to as parameter (virtual advertisement "A") data). In the format of the parameter (virtual advertisement "A") data, information indicating the position and direction of the virtual camera, the angle of view corresponding to the virtual camera, and a time code, which are also specified in the parameter (without virtual advertisement) shown in FIG. 6(b). In the format of the parameter (virtual advertisement "A") data, information regarding a display instruction of a virtual advertisement (virtual advertisement "A" in FIG. 6(c)) is further specified in "advertise display". This parameter (virtual advertisement "A") includes an instruction to display the virtual advertisement "A" ("advertisement ID":"A" in FIG. 6(a)) instructed to be registered in the processing of S501 in the virtual viewpoint image. Upon receiving such parameter (virtual advertisement "A") data, the image generating device 120 generates a virtual viewpoint image in which the virtual advertisement is displayed. However, the time code extracted from the parameters received in S504 differs from the time code extracted from the parameters received in S502 to S503. In other words, the parameters received in S502 to S504 each target a different frame.

[0075] In S505, similarly to S504, the information processing device 110 transmits to the image generating device 120 the parameters (virtual advertisement "A") set by the parameter setting unit 114 and accompanied by an instruction to display the virtual advertisement.

[0076] Here, an example of a virtual viewpoint image with a virtual advertisement, which the image generating device 120 extracts virtual viewpoint information from the parameters (virtual advertisement "A") in S504 and S505 and generates according to the extracted virtual viewpoint information, will be described with reference to the drawings. According to the virtual viewpoint information extracted from the parameters (virtual advertisement "A"), the rendering unit 132 performs rendering processing on the composite 3D data obtained by combining the basic 3D data and the virtual advertisement data. This processing generates a virtual viewpoint image with a virtual advertisement added, as shown in FIG. 3(c). However, the time code extracted from the parameters (virtual advertisement "A") received in S505 is different from the time code extracted from the parameters received in S502 to S504. That is, the parameters (virtual advertisement "A") received in S505 and the parameters received in S502 to S504 target different frames.

[0077] In S506, similarly to S502 and S503, the information processing device 110 again transmits to the image generating device 120 the parameters (without virtual advertisement) set by the parameter setting unit 114 and not accompanied by an instruction to display a virtual advertisement. In the image generating device 120, when the rendering unit 132 performs rendering processing on the basic 3D data according to the virtual viewpoint information extracted from the parameters (without virtual advertisement), a virtual viewpoint image without an added virtual advertisement is again generated. However, the time code extracted from the parameters (without virtual advertisement) received in S506 is different from the time code extracted from the parameters received in S502 to S505. That is, the parameters (without virtual advertisement) received in S506 and the parameters received in S502 to S505 are different in the target frames.

[0078] As described above using FIG. 5, it is possible to set a desired virtual advertisement to be added (displayed) at a desired position in a desired virtual viewpoint image, or to set a virtual advertisement not to be added (displayed), on a frame-by-frame basis.

[0079] <UI for setting virtual ad display conditions> An example of a user interface (UI) for setting conditions for displaying a virtual advertisement in a virtual viewpoint image will be described with reference to the drawings. FIG. 8 is a diagram showing an example of a UI for setting display conditions for a virtual advertisement, in which FIG. 8(a) shows a case where a virtual advertisement is set to be hidden, and FIG. 8(b) shows a case where a virtual advertisement is set to be displayed. When the display conditions for a virtual advertisement are set by a user operation in the UI for setting display conditions for a virtual advertisement, information regarding the display conditions for the virtual advertisement is set in parameters by the display condition setting unit 115.

[0080] The UI 801 for setting display conditions of a virtual advertisement includes two radio buttons 802 and 803. The radio button 802 is a button for setting a non-display state of the virtual advertisement, that is, a state in which the virtual advertisement is not displayed in the virtual viewpoint image. The radio button 803 is a button for additionally displaying the virtual advertisement with respect to the virtual viewpoint image, and for specifying the virtual advertisement to be displayed. The radio button 803 is arranged in a list box that shows a list of a plurality of registered virtual advertisements.

[0081] When a user does not want to display a virtual advertisement in the virtual viewpoint image, the user selects a radio button 802 corresponding to "None" for setting the display of the virtual advertisement to be hidden, as shown in Fig. 8(a). The selection of the radio button 802 in this manner corresponds to the processing sequence of S502, S503, and S506 in the sequence diagram of Fig. 5.

[0082] On the other hand, when a user wishes to display a virtual advertisement in the virtual viewpoint image, the user operates the virtual advertisement display at the desired timing, and selects a radio button 803 corresponding to the virtual advertisement to be displayed ("Virtual Advertisement A"), as shown in Fig. 8(b). The selection of the radio button 803 in this manner corresponds to the processing sequence of S504 and S505 in the sequence diagram of Fig. 5.

[0083] <User operation screen> Here, a user operation screen for setting a virtual viewpoint corresponding to a virtual viewpoint image and display conditions of a virtual advertisement will be described with reference to the drawings. FIG. 9 is a diagram showing an example of a user operation screen, in which FIG. 9(a) shows a case where a virtual advertisement is set to be hidden, and FIG. 9(b) shows a case where a virtual advertisement is set to be displayed. FIG. 9 shows a case where a virtual viewpoint image corresponding to the set display conditions is displayed on the left display, and a UI for setting display conditions of a virtual advertisement is displayed on the right display. Note that the display form of the virtual viewpoint image and the UI for setting display conditions of a virtual advertisement is not limited to this. The virtual viewpoint image and the UI for setting display conditions of a virtual advertisement may be displayed on one display.

[0084] As shown in Fig. 9(a), the left display 901 displays a virtual viewpoint image 911 generated according to the virtual viewpoint based on basic 3D data. The virtual viewpoint is initially set, and the initially set virtual viewpoint is used as the virtual viewpoint before a user operation related to the virtual viewpoint is accepted. The right display 902 displays a virtual advertisement display designation UI 912 in a state in which a radio button for setting the display of a virtual advertisement to be hidden for the image displayed on the left display 901 has been selected by a user operation. Since the display of a virtual advertisement has been selected, the left display displays a virtual viewpoint image without an added virtual advertisement.

[0085] As shown in Fig. 9(b), the left display 901 displays a virtual viewpoint image 921 generated according to a virtual viewpoint based on basic 3D data. The right display 902 displays a virtual advertisement display designation UI 922 in a state in which a radio button for setting the display of virtual advertisement A for the image displayed on the left display 901 has been selected by a user operation. Since the display of the virtual advertisement has been selected, the left display displays the virtual viewpoint image with the virtual advertisement added.

[0086] When a game is played over the entire field and players are present over the entire field, as in the virtual viewpoint image 911 displayed on the display 901, the radio button corresponding to "None" (no virtual advertisement) is selected in the UI 912. Therefore, no virtual advertisement is added to the virtual viewpoint image 911 displayed on the left display 901.

[0087] At the timing when players gather on one side of the field and there are no players on the other side, as in virtual viewpoint image 921 displayed on display 901, a radio button corresponding to virtual advertisement A is selected in UI 922. Therefore, a virtual advertisement is added to virtual viewpoint image 921 displayed on left display 901. By registering virtual advertisement data in advance and selecting a virtual advertisement at the desired timing on the UI displayed on right display 902 while viewing the virtual viewpoint image displayed on left display 901, the virtual advertisement can be instantly displayed in the virtual viewpoint image.

[0088] In this embodiment, when a radio button corresponding to virtual advertisement A is selected by a user operation, the parameter setting unit 114 sets a display instruction (display condition) for virtual advertisement A as a parameter, and the parameter transmission unit 118 transmits the parameter to the image generating device 120. However, the operation method is not limited to this, and a key assigned to a keyboard may be pressed, or an operation using a touch panel may be performed. In addition, the virtual viewpoint may be adjusted by a user operation using a virtual viewpoint image 911 displayed on the left display 901. When the virtual viewpoint is adjusted (set) by a user operation, the parameter setting unit 114 sets virtual viewpoint information as a parameter, and the parameter transmission unit 118 transmits the parameter to the image generating device 120.

[0089] This makes it possible to set, for example, in a penalty kick scene in a soccer game, to display virtual advertisements in areas of the field where no players are present, according to the camera angle, and generate a virtual viewpoint image with virtual advertisements according to this setting. In other words, it becomes possible to display virtual advertisements in areas of the field where no signboards can be placed, without overlapping with players present on the field.

[0090] As described above, according to this embodiment, the virtual advertisement data (3D model and still image texture) is registered in advance, and the identification information of the virtual advertisement data, the display conditions of the virtual advertisement, the virtual viewpoint information, and the time code are set as parameters, thereby achieving the following effects. A virtual viewpoint image with a virtual advertisement or a virtual viewpoint image without a virtual advertisement can be generated according to information extracted from the parameters. Therefore, it is possible to switch between displaying and hiding the virtual advertisement in real time on a frame-by-frame basis, and to display an effective virtual advertisement suited to the scene in the virtual viewpoint image. In other words, it is possible to generate a virtual viewpoint image in which a virtual advertisement is displayed at a desired position at a desired timing.

[0091] [Second embodiment] In this embodiment, a mode of switching and displaying a plurality of types of registered virtual advertisement data will be described with reference to the drawings. Fig. 10 is a sequence diagram showing the flow of processing between the information processing device 110 and the image generating device 120 when a plurality of types of virtual advertisement data are registered and display conditions for each of the registered plurality of types of virtual advertisement data are set. Note that in this embodiment, the processing of registering a virtual advertisement and setting display conditions for the registered virtual advertisement is the same as that in the image processing system of the above-mentioned first embodiment, and therefore a description thereof will be omitted.

[0092] In S1001, the information processing device 110 (registration instruction unit 113) instructs the image generating device 120 to register the virtual advertisement data “AA” set by the virtual advertisement setting unit 112. Note that S1001 corresponds to the process of S412 in the flowchart of FIG. 4(b) and the process of S421 in the flowchart of FIG. 4(c).

[0093] Similarly, in S1002, the information processing device 110 instructs the image generating device 120 to register the virtual advertisement data "BB" set by the virtual advertisement setting unit 112. In S1003, the information processing device 110 instructs the image generating device 120 to register the virtual advertisement data "CC" set by the virtual advertisement setting unit 112.

[0094] Here, examples of virtual advertisement data "AA", "BB", and "CC" will be described with reference to the figures. FIG. 11 shows example images of virtual advertisement data displayed in a virtual viewpoint image, with FIG. 11(a) showing an example image of virtual advertisement data "AA", FIG. 11(b) showing an example image of virtual advertisement data "BB", and FIG. 11(c) showing an example image of virtual advertisement data "CC". As shown in FIG. 11(a), FIG. 11(b) and FIG. 11(c), virtual advertisements 1101, 1102, and 1103 have the same texture but are displayed at different positions.

[0095] In the above, three types of data with different display positions of the virtual advertisement are registered, but the present invention is not limited to this. In other words, multiple types of virtual advertisement data corresponding to the desired display position and size and having the desired texture may be created in advance, and all of them may be registered in advance.

[0096] In S1004, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 the parameters (without virtual advertisement) set by the parameter setting unit 114 and not accompanied by an instruction to display a virtual advertisement.

[0097] In S1005, the information processing device 110 (parameter transmission unit 118) transmits parameters (virtual advertisement "AA") accompanied by a display instruction for the virtual advertisement "AA" set by the parameter setting unit 114 to the image generating device 120. Similarly, in S1006, the information processing device 110 transmits parameters (virtual advertisement "BB") accompanied by a display instruction for the virtual advertisement "BB" set by the parameter setting unit 114 to the image generating device 120. Also, in S1007, the information processing device 110 transmits parameters (virtual advertisement "CC") accompanied by a display instruction for the virtual advertisement "CC" set by the parameter setting unit 114 to the image generating device 120. However, the time codes extracted from the parameters received in S1004 to S1007 are different from the time codes extracted from the parameters received in the processing sequences other than the information processing device 110 itself. That is, the parameters received in S1004 to S1007 are targeted at different frames.

[0098] <Virtual ad display setting UI> An example of a user interface (UI) for setting display conditions of a virtual advertisement for a virtual viewpoint image will be described with reference to the drawings. FIG. 12 is a diagram showing an example of a UI for setting display conditions of a virtual advertisement. FIG. 12(a) shows an example of a UI for registering a virtual advertisement, FIG. 12(b) shows an example of a UI for creating a virtual advertisement set, and FIG. 12(c) shows an example of a UI for setting a virtual advertisement schedule. When a virtual advertisement is registered by a user operation in the UI for registering a virtual advertisement, the virtual advertisement setting unit 112 registers the virtual advertisement data in the image generating device 120. When a display condition of a virtual advertisement (virtual advertisement set) is created by a user operation in the UI for creating a virtual advertisement set, the display condition setting unit 115 sets information about the display condition of the virtual advertisement in a parameter. When a display condition of a virtual advertisement (virtual advertisement schedule) is set by a user operation in the UI for setting a virtual advertisement schedule, the display condition setting unit 115 sets information about the display condition of the virtual advertisement in a parameter.

[0099] As shown in FIG. 12(a), the virtual advertisement registration UI 1210 is a user interface for registering a virtual advertisement. The virtual advertisement registration UI 1210 has a list box 1211 that selectably displays virtual advertisements to be registered, and a "register" button 1212 for registering a selected virtual advertisement. The list box 1211 displays a set of images showing the virtual advertisement and its display position, the name of the virtual advertisement, and a radio button for selecting a virtual advertisement. In the example of FIG. 12(a), the list box 1211 displays virtual advertisements AA, BB, and CC.

[0100] Virtual advertisement AA is data indicating that a virtual advertisement with a center-aligned texture "Abcde" is displayed in a position in the center of the virtual viewpoint image in the vertical direction and toward the left of the center in the horizontal direction. Virtual advertisement BB is data indicating that a virtual advertisement with a center-aligned texture "Abcde" is displayed in a position in the center of the virtual viewpoint image in the vertical direction and toward the left of the center in the horizontal direction. Virtual advertisement CC is data indicating that a virtual advertisement with a center-aligned texture "Abcde" is displayed in a position in the center of the virtual viewpoint image in the vertical direction and toward the right of the center in the horizontal direction. By selecting the radio button corresponding to the virtual advertisement to be registered and pressing the "Register" button 1212, the selected virtual advertisement is registered in the image generating device 120.

[0101] As shown in FIG. 12(b), the virtual advertisement set creation UI 1220 has a list box 1221, a creation area 1222, and a confirmation area 1223. The list box 1221 displays a list of registered virtual advertisements. The creation area 1222 displays a list of virtual advertisements that can be created as a virtual advertisement set. The confirmation area 1223 displays a list of created virtual advertisement sets. In the example of FIG. 12(b), the confirmation area 1223 shows a state in which a virtual advertisement set 1237 that displays virtual advertisement AA, virtual advertisement BB, and virtual advertisement CC in that order is registered.

[0102] Here, a method for creating a virtual advertisement set will be described with reference to the drawings. Fig. 13 is a diagram for explaining a method for creating a virtual advertisement set. Fig. 13(a) shows a state in which registered virtual advertisements are displayed in a list, Fig. 13(b) shows a state in which a virtual advertisement to be the target of the virtual advertisement set is specified (identified), and Fig. 13(c) shows a state in which the virtual advertisement set is registered.

[0103] When a virtual advertisement is registered by a user operation, the virtual advertisement setting unit 112 displays a list of the virtual advertisements to be registered in a list box 1221 of a virtual advertisement set creation UI 1220 according to the operation contents of the user operation, as shown in Fig. 13(a) . In the example of Fig. 13(a), virtual advertisements AA1231, virtual advertisement BB1232, virtual advertisement CC1233, etc. are displayed in a list.

[0104] Then, by performing a drag-and-drop mouse operation on the virtual advertisement to be set as a virtual advertisement set by a user operation, the virtual advertisement to be set as a virtual advertisement set is moved from within the list box 1221 to the creation area 1222. In the example of Fig. 13(b), virtual advertisements AA1234, BB1235, and CC1236 are selected as the virtual advertisement set.

[0105] Then, when the user presses create button 1225, as shown in Fig. 13(c), the created virtual advertisement set 1237 is displayed in confirmation area 1223. In the example of Fig. 13(c), the virtual advertisement set 1237 is confirmed to display virtual advertisement AA 1234, virtual advertisement BB 1235, and virtual advertisement CC 1236 in this order.

[0106] Returning to the description of FIG. 12, as shown in FIG. 12(c), the virtual advertisement schedule setting UI 1240 is a UI for specifying a schedule for displaying virtual advertisements on a virtual viewpoint image. The virtual advertisement schedule setting UI 1240 has a virtual advertisement set designation area 1241 and a virtual advertisement schedule designation area 1242. In the virtual advertisement set designation area 1241, selectable virtual advertisement sets are displayed in a list. In the example of FIG. 12(c), the designation area 1241 shows a state in which one virtual advertisement set out of two or more types of virtual advertisement sets is selected. In the virtual schedule designation area 1242, a box for setting the order in which virtual advertisements included in the virtual advertisement set are displayed, an update unit in number of frames, and a pull-down menu for setting whether or not to perform loop playback are displayed. In the example of Fig. 12(c), in the designation area 1242, the top part of the left column shows a virtual advertisement set to be displayed in the order of virtual advertisement AA, virtual advertisement BB, and virtual advertisement CC, and the middle part of the left column shows a virtual advertisement set to be displayed in the order of virtual advertisement AA and virtual advertisement BB. Also, the right column shows that one frame is set as the update unit and loop playback is set as the loop playback. When the display conditions of the virtual advertisement (virtual advertisement set) are set by a user operation in the virtual advertisement schedule setting UI 1240, the display condition setting unit 115 sets the display conditions of the virtual advertisement as parameters.

[0107] In this way, by performing a user operation on the virtual advertisement schedule setting UI 1240, the display condition setting unit 115 can perform a process of setting a display schedule for a virtual advertisement. That is, the display condition setting unit 115 can select a virtual advertisement to be displayed in a virtual viewpoint image from among the virtual advertisement data registered in the virtual advertisement setting unit 112, and schedule the display time, display interval, display sequence, number of times of display, and the like of the selected virtual advertisement. The identification number of the virtual advertisement data set in the schedule by the display condition setting unit 115 is set in a parameter in which information regarding a display instruction for a virtual advertisement is set, together with virtual viewpoint information and a time code, according to this schedule. The virtual viewpoint information includes the position of the virtual camera (virtual viewpoint), the attitude (direction) of the virtual camera (virtual viewpoint), and the angle of view corresponding to the virtual camera. Then, the parameter transmission unit 118 transmits the parameters in which various information is set by the parameter setting unit 114 to the image generating device 120.

[0108] As described above, by registering multiple types of virtual advertisement data in advance and setting identification information for the multiple types of virtual advertisement data and information regarding the display conditions as parameters, a desired type of virtual advertisement can be displayed in the virtual viewpoint image at a desired timing.

[0109] In the virtual advertisement specification example shown in FIG. 12(b), it is possible to display a virtual advertisement such that the position of the advertising signboard moves from a left position to a right position in the image.

[0110] Moreover, by registering a plurality of types of virtual advertisement data in advance, it becomes possible to switch between and display a plurality of types of virtual advertisements.

[0111] Although the example of sequentially displaying a plurality of virtual advertisements having the same texture but different display positions on the virtual viewpoint image has been described, the present invention is not limited to this. For example, a plurality of virtual advertisements having a plurality of types of texture but the same display position, or a plurality of virtual advertisements having a plurality of types of texture but different display positions may be sequentially displayed on the virtual viewpoint image.

[0112] [Third embodiment] In this embodiment, another example of the image of the virtual advertisement will be described with reference to the drawings. FIG. 14 is a diagram showing an example of an image of virtual advertisement data displayed in a virtual viewpoint image. FIG. 14(a) shows a case where the texture of the virtual advertisement is right-justified, FIG. 14(b) shows a case where the texture of the virtual advertisement is center-justified, and FIG. 14(c) shows a case where the texture of the virtual advertisement is left-justified. A virtual advertisement 1401 shown in FIG. 14(a) is created as computer graphics from image data of an advertisement signboard that does not exist in the virtual viewpoint image (basic 3D data) to which the advertisement is to be added. In the virtual advertisement 1401, the texture "Abcde" is written (displayed) right-justified on the advertisement signboard. A virtual advertisement 1402 shown in FIG. 14(b) is created as computer graphics from image data of an advertisement signboard that does not exist in the virtual viewpoint image (basic 3D data) to which the advertisement is to be added. In the virtual advertisement 1402, the texture "Abcde" is written (displayed) center-justified on the advertisement signboard. The virtual advertisement 1403 shown in Fig. 14(c) is created as computer graphics from image data of an advertising billboard that does not exist in the virtual viewpoint image (basic 3D data) to which it is to be added. In the virtual advertisement 1403, the texture "Abcde" is written (displayed) left-justified on the advertising billboard. In other words, the above-mentioned virtual advertisements 1401, 1402, and 1403 show an example in which the 3D model showing the virtual advertising billboard is the same, but only the writing position (display position) of the texture is different.

[0113] In the case of this embodiment, the following effects can be obtained by using a sequence similar to that shown in Fig. 10. That is, by registering the data of the virtual advertisements 1401, 1402, and 1403 in the image generating device 120 and setting display conditions, including no virtual advertisement, for switching and displaying the virtual advertisement to be displayed as parameters, it is possible to switch between the virtual advertisements to be displayed. Also, in this embodiment, since the textures of the virtual advertisements 1401 to 1403 are present near the right side, the center, and the left side, it is possible to display the advertisements in a manner that moves by switching the display in the order of the virtual advertisements 1401, 1402, and 1403.

[0114] As described above, according to this embodiment, by setting display conditions of a virtual advertisement as parameters through user operation on the information processing device 110, it becomes possible to display or hide a virtual advertisement and to switch between different types of virtual advertisements.

[0115] [Fourth embodiment] In this embodiment, the time to display a virtual advertisement is set in advance as a schedule, and control is performed to switch the type of virtual advertisement to be displayed in the virtual viewpoint image according to the schedule without requiring immediate operation by the user. This embodiment will be described with reference to figures.

[0116] Regarding the information processing device possessed by the image processing system of this embodiment, functional units different from those of the first embodiment will be described with reference to FIG. 1(b). The display condition setting unit of this embodiment receives the virtual advertisement set by the virtual advertisement setting unit 112 and information generated according to the input operation received by the operation receiving unit, and also receives the time code set by the time code setting unit indicating the display time of the virtual advertisement. Next, the display condition setting unit of this embodiment links the parameters in which the display conditions of the predetermined virtual advertisement are set with the time code indicating the display time (display start time and display end time) of the predetermined virtual advertisement, and sends them to the parameter transmission unit of this embodiment. When the display start time of the predetermined virtual advertisement indicated by the time code is reached, the parameter transmission unit of this embodiment transmits the corresponding parameter (the predetermined virtual advertisement is present) to the parameter reception unit 123 of the image generating device 120. Furthermore, when the display end time of the predetermined virtual advertisement indicated by the time code is reached, the parameter transmission unit of this embodiment transmits the corresponding parameter (the predetermined virtual advertisement is not present) to the parameter reception unit 123 of the image generating device 120.

[0117] Moreover, with regard to the flow of processing executed by the information processing device when a schedule is set as a display condition for the above-mentioned virtual advertisement, processing that differs from that in the first embodiment will be described with reference to FIG. 4(b).

[0118] In S413, the operation receiving unit 111 receives a user operation for inputting parameters, i.e., a user operation for inputting (registering) a schedule in which the display start time and the display end time of the virtual advertisement to be displayed are set for the virtual advertisement to be displayed.

[0119] In S414, the operation reception unit 111 judges whether the current time is included in the range from the display start time to the display end time of the virtual advertisement set in the schedule accepted in S413. If the judgment result indicates that the current time is included in the display time range of the virtual advertisement set in the schedule (YES in S414), the process proceeds to S415. On the other hand, if the judgment result indicates that the current time is not included in the display time range of the virtual advertisement set in the schedule (NO in S414), S415 is skipped and the process proceeds to S416.

[0120] As described above, by setting the display time of the virtual advertisement in advance as a schedule, the information processing device 110 can transmit parameters (with virtual advertisement) or parameters (without virtual advertisement) set according to the schedule to the image generating device 120.

[0121] <Virtual ad display setting UI> An example of a user interface (UI) for setting the display conditions (display time) of a virtual advertisement for a virtual viewpoint image will be described with reference to a drawing. Fig. 15 is a diagram showing an example of a UI for setting the display conditions of a virtual advertisement.

[0122] The UI 1501 for setting display conditions of a virtual advertisement has a schedule setting field 1510 for setting the time to display the virtual advertisement. The schedule setting field 1510 has a start time setting field 1511, an end time setting field 1512, and a display target setting field 1513. The start time setting field 1511 is an area for setting the time to start displaying the target virtual advertisement by a user operation. The end time setting field 1512 is an area for setting the time to end displaying the target virtual advertisement by a user operation. The display target setting field 1513 is an area for setting the type of virtual advertisement to be displayed in the virtual viewpoint image by a user operation.

[0123] When a display schedule of the virtual advertisement is set by a user operation on the display condition setting UI, the display condition setting unit 115 sets information on the display conditions of the virtual advertisement in the parameters.

[0124] For example, a case will be described in which the broadcast time of a soccer match is from 6:00 p.m. to 8:00 p.m., and virtual advertisement AAA is displayed in the virtual viewpoint image from 6:00 p.m. to 7:00 p.m., and virtual advertisement BBB is displayed in the virtual viewpoint image from 7:00 p.m. to 8:00 p.m.

[0125] 15, by a user operation, "18:00" is set in the start time setting field 1511, "19:00" is set in the end time setting field 1512, and "virtual advertisement AAA" is set in the display target setting field 1513. In addition, "19:00" is set in the start time setting field 1511, "20:00" is set in the end time setting field 1512, and "virtual advertisement BBB" is set in the display target setting field 1513.

[0126] Information about the set display conditions of the virtual advertisement is set in parameters by the display condition setting section of the parameter setting unit when the designated time arrives, and the parameters are transmitted to the image generating device 120 by the parameter transmission section.

[0127] Therefore, by setting the display schedule for the virtual advertisements as described above, virtual advertisement AAA will be displayed in the virtual viewpoint image from 18:00 to 19:00, and virtual advertisement BBB will be displayed in the virtual viewpoint image from 19:00 to 20:00.

[0128] As described above, according to this embodiment, by setting the time for displaying a virtual advertisement in advance as a schedule, it is possible to control switching of the type of virtual advertisement to be displayed in the virtual viewpoint image according to the schedule without requiring immediate operation by the user.

[0129] In the above embodiment, a timer setting based on time is used as a display condition. However, it is also possible to set not only the time but also the elapsed time from when an instruction to display a virtual advertisement is given as a display condition.

[0130] Furthermore, it is also possible to use a display condition that combines a display instruction setting of a virtual advertisement by a timer and a display instruction setting of a virtual advertisement at any timing by the operator. For example, in a state where a schedule is set to display virtual advertisement AAA in the virtual viewpoint image from 18:00 to 19:00 and to display virtual advertisement BBB in the virtual viewpoint image from 19:00 to 20:00, it is also possible to combine the display instruction settings of the virtual advertisements shown below. In other words, it is also possible to switch the displayed virtual advertisement to virtual advertisement A by the operator's operation only during a penalty kick.

[0131] Furthermore, the present invention can be applied to a system in which a charge is billed to an advertiser of a virtual advertisement according to the display time of the virtual advertisement set by a timer.

[0132] [Fifth embodiment] In this embodiment, a 3D model and video data of a virtual advertisement are registered in advance, and the registered video data or a frame number of the video data is specified along with parameters to display a virtual advertisement with a moving texture in a virtual viewpoint image. This embodiment will be described with reference to drawings.

[0133] A virtual advertisement data registration example and a parameter example for generating a virtual viewpoint image will be described with reference to the drawings. FIG. 16 is a diagram showing a virtual advertisement data registration example and a parameter example for generating a virtual viewpoint image, in which FIG. 16(a) shows a 3D model format example of virtual advertisement data transmitted at the time of registration, and FIG. 16(b) shows a format example of parameter (with virtual advertisement) data. As shown in FIG. 16(a), in the 3D model format of virtual advertisement data, the file names and file sizes of the obj file, mtl file, and mp4 file are specified. The obj file and the mtl file are the same as the obj file and the mtl file described with reference to FIG. 6, and the description thereof will be omitted. The mp4 file stores video data (parameters) of the 3D model of the virtual advertisement. Here, a parameter (with virtual advertisement) data example will be described with reference to the drawings. In the format of the parameter (with virtual advertisement) data shown in FIG. 16(b), "AAAA:FWD" representing "advertisement ID: video playback method" is specified as a virtual advertisement display instruction. Here, FWD represents forward constant speed playback. In this way, by instructing the playback method of the video when setting various information for the parameters, it becomes possible to display a moving virtual advertisement.

[0134] As described above, according to this embodiment, by registering a video as virtual advertisement data in advance and setting identification information of the virtual advertisement data and information on the display conditions in the parameters, the following effects can be achieved. A virtual viewpoint image with a virtual advertisement (video data) or a virtual viewpoint image without a virtual advertisement can be generated according to the information set in the parameters. Therefore, it is possible to switch between displaying and hiding a virtual advertisement in real time on a frame-by-frame basis, and to display an effective virtual advertisement suited to the scene in the virtual viewpoint image. In other words, it is possible to generate a virtual viewpoint image in which a virtual advertisement (video data) is displayed in a desired position at a desired timing.

[0135] In the first to fifth embodiments, the advertising information (virtual advertisement) has been described as an example of the additional information added to the basic 3D data (moving image file), but the additional information is not limited to the advertising information. As another example of the additional information added to the basic 3D data, information on the player appearing in the virtual viewpoint image (hereinafter referred to as player information) may be added. When the player information is added to the moving image file, the virtual advertisement storage unit 122 stores information on the data type, the number of pieces of data, and the address of the player information database. The data type is information indicating that the additional information is information on the player. The number of pieces of data is the number of pieces of player information. The information on the address of the player information database is address information for connecting to the database in which the player information is accumulated. When such address information is not stored in the virtual advertisement storage unit 122, the virtual advertisement storage unit 122 stores information such as the player information ID, the player information model data of the player information, the player information material data, the pasting area of ​​the player information on the virtual viewpoint image, and the priority. As described above, this embodiment can be applied to things other than advertising information.

[0136] <Other embodiments> In the above, the information processing device 110 is described as setting the advertisement data and its display conditions, setting information related to the display conditions in parameters, and transmitting the data to the image generating device 120. However, the present invention is not limited to this. The information processing device may be configured to read out virtual advertisement data to be added to the virtual viewpoint image from the image generating device 120, set the display conditions thereof, set information related to the display conditions in parameters, and transmit the data to the image generating device 120.

[0137] Furthermore, the object of the present invention can also be achieved by supplying a storage medium on which a computer program code for implementing the above-mentioned functions is recorded to a system, and the system reads and executes the computer program code. In this case, the computer program code itself read from the storage medium implements the functions of the above-mentioned embodiment, and the storage medium on which the computer program code is stored constitutes the present disclosure. Also included is a case in which an operating system (OS) running on a computer performs part or all of the actual processing based on the instructions of the program code, and the above-mentioned functions are implemented by that processing.

[0138] Furthermore, the image processing system of this embodiment may be realized in the following form. That is, the computer program code read from the storage medium is written to a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the computer program code, a CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing to realize the above-mentioned functions.

[0139] When the present disclosure is applied to the above-mentioned storage medium, the storage medium stores computer program code corresponding to the flowchart described above.

[0140] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions. [Explanation of symbols]

[0141] 111 Operation reception section 114 Parameter setting section 118 Parameter transmission unit

Claims

1. An information processing device that generates parameters for generating a virtual viewpoint image on which additional information is displayed, a setting means for setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output means for outputting the parameters set by the setting means; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein said setting means sets said parameters for each frame in which an image generating device generates said virtual viewpoint image.

3. The virtual viewpoint information includes a position of a virtual viewpoint, a direction of the virtual viewpoint, and a field of view corresponding to the virtual viewpoint.

3. The information processing device according to claim 1, wherein the information processing device is a computer.

4. The setting means further sets a time code of the virtual viewpoint image generated by the image generating device as the parameter. The information processing device according to claim 1 ,

5. The information processing device according to claim 1 , wherein the additional information includes information related to an advertisement that is shown in a 3D model format or texture.

6. The additional information includes information about objects included in the virtual viewpoint image, the objects being represented in a 3D model format or texture. The information processing device according to claim 1 .

7. The information represented by the texture includes video data.

7. The information processing device according to claim 5 or 6.

8. The display conditions of the additional information include at least one of a display time of the additional information, a display interval of the additional information, a display order of the additional information, and a display count of the additional information. The information processing device according to claim 1 .

9. a user interface for accepting user input; The setting means sets the parameters in accordance with a user input received via the user interface. The information processing device according to claim 1 .

10. an acquisition means for acquiring parameters in which a display condition of additional information to be displayed in a virtual viewpoint image, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated are set; an extraction means for extracting a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information from the parameters; a first reading means for reading additional information indicated by the identification information extracted by the extraction means from among the pre-stored additional information based on a display condition of the additional information extracted by the extraction means; a generating means for generating a virtual viewpoint image in which the additional information is displayed, in accordance with the additional information read by the first reading means, prestored shape data indicating a shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted by the extracting means; An image generating device comprising:

11. 11. The image generating apparatus according to claim 10, wherein the acquisition means acquires the parameters set for each frame for which the generation means generates the virtual viewpoint image.

12. The virtual viewpoint information includes a position of a virtual viewpoint, a direction of the virtual viewpoint, and a field of view corresponding to the virtual viewpoint.

12. An image generating device according to claim 10 or 11.

13. The extraction means extracts the time code from the parameter acquired by the acquisition means, the parameter being set with a time code of the virtual viewpoint image to be generated by the generation means; The generating means generates a virtual viewpoint image in which the additional information is displayed in accordance with shape data of the object corresponding to the time code extracted by the extracting means, among shape data of the object stored in advance.

13. An image generating device according to any one of claims 10 to 12, characterized in that

14. 14. An image generating device according to claim 10, characterized in that the additional information includes information about an advertisement shown in a 3D model format or texture.

15. The additional information includes information about objects included in the virtual viewpoint image, the objects being represented in a 3D model format or texture.

15. An image generating device according to any one of claims 10 to 14, characterized in that

16. The information represented by the texture includes video data.

16. An image generating device according to claim 14 or 15.

17. The display conditions of the additional information include at least one of a display time of the additional information, a display interval of the additional information, a display order of the additional information, and a display count of the additional information.

17. An image generating device according to any one of claims 10 to 16, characterized in that

18. An information processing method for generating parameters for generating a virtual viewpoint image on which additional information is displayed, comprising the steps of: a setting step of setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output step of outputting the parameters set in the setting step; An information processing method comprising:

19. an acquisition step of acquiring parameters in which display conditions of additional information to be displayed in a virtual viewpoint image, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated are set; an extraction step of extracting a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information from the parameters; a reading step of reading, from among the pre-stored additional information, additional information indicated by the identification information extracted in the extraction step, based on a display condition of the additional information extracted in the extraction step; a generating step of generating a virtual viewpoint image in which the additional information is displayed, according to the additional information read in the reading step, pre-stored shape data indicating a shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted in the extracting step; 1. An image generating method comprising:

20. A program causing a computer to function as the information processing device according to any one of claims 1 to 9.

21. A program causing a computer to function as the image generating device according to any one of claims 10 to 17.

22. An image processing system including: an information processing device that generates parameters for generating a virtual viewpoint image in which additional information is displayed; and an image generating device that generates the virtual viewpoint image in accordance with the parameters generated by the information processing device, The information processing device includes: a setting means for setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output means for outputting the parameters set by the setting means; Have The image generating device comprises: an acquisition means for acquiring the parameters output by the output means; an extraction means for extracting a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information from the parameters acquired by the acquisition means; a readout means for reading out additional information indicated by the identification information extracted by the extracting means from among the pre-stored additional information based on a display condition of the additional information extracted by the extracting means; a generating means for generating a virtual viewpoint image in which the additional information is displayed, in accordance with the additional information read by the reading means, prestored shape data indicating a shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted by the extracting means; An image processing system comprising:

Images