Information processing apparatus, generation apparatus, information processing method, generation method, and program

Abstract

To generate a virtual viewpoint image in which desired information is added to a desired position.SOLUTION: The information processing apparatus generates a parameter for generating the virtual viewpoint image on which the additional information is displayed. The information processing apparatus sets a display condition of additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated as parameters, and outputs the set parameters.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image. [Background technology]

[0002] Recently, a technology that generates a virtual viewpoint image (also called a free viewpoint video) that shows the view from any virtual viewpoint using the multiple viewpoint images obtained by installing multiple cameras at different positions and capturing images synchronously has been attracting attention. With this technology, for example, it becomes possible to watch highlight scenes from a soccer or basketball game from various angles, and it is possible to give the user a more realistic feeling than with ordinary video content.

[0003] Adding various information to such virtual viewpoint images has been studied. Patent Document 1 discloses a technique for adding preset advertising information to a specific area of ​​a free viewpoint video when the viewpoint condition corresponding to the free viewpoint video is the same as a specific viewpoint condition that has been preset. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-41259 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the technology of Patent Document 1, even if there is an area in the frame of the free viewpoint video where advertising information can be added, if the viewpoint conditions corresponding to this free viewpoint video are different from the specific viewpoint conditions that have been set in advance, the advertising information cannot be added.

[0006] An object of the present disclosure is to provide a technology for generating a virtual viewpoint image to which desired information is added at a desired position. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present disclosure is an information processing device that generates parameters for generating a virtual viewpoint image in which additional information is displayed, and is characterized by having a setting means that sets display conditions for the additional information, identification information indicating the additional information, and virtual viewpoint information of the virtual viewpoint image to be generated as the parameters, and an output means that outputs the parameters set by the setting means. [Effects of the Invention]

[0008] According to the present disclosure, it is possible to generate a virtual viewpoint image to which desired information is added at a desired position. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an image processing system. [Figure 2] A diagram showing an example of the hardware configuration of the device. [Figure 3] A diagram explaining the generation of a virtual viewpoint image with a virtual advertisement [Figure 4] 1 is a flowchart showing the flow of processing executed by an image processing system; [Figure 5] A sequence diagram showing the flow of processing between an information processing device and an image generating device. [Figure 6] FIG. 1 shows an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image. [Figure 7] A diagram showing an example of an obj file specified in the 3D model format of virtual advertising data [Figure 8] A diagram showing an example of a user interface [Figure 9] A diagram showing an example of a user operation screen [Figure 10] A sequence diagram showing the flow of processing between an information processing device and an image generating device. [Figure 11] A diagram showing an example of a virtual advertisement image displayed on a virtual viewpoint image. [Figure 12] A diagram showing an example of a user interface [Figure 13] Diagram illustrating the creation of a virtual ad set [Figure 14] A diagram showing an example of a virtual advertisement image displayed on a virtual viewpoint image. [Figure 15] A diagram showing an example of a user interface [Figure 16] FIG. 1 shows an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present embodiment will be described below with reference to the drawings. Note that the following embodiment does not necessarily limit the present invention. Furthermore, not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention.

[0011] [First embodiment] In this embodiment, an image processing system for generating a virtual viewpoint image displaying a virtual advertisement, which is a type of additional information that does not exist in the virtual viewpoint image to be added, will be described with reference to the drawings. Fig. 1 is a diagram for explaining the image processing system according to this embodiment, in which Fig. 1(a) shows an example of the configuration of the image processing system, and Fig. 1(b) shows functional blocks of the image processing system.

[0012] The image processing system of this embodiment includes an information processing device 110, an image generating device 120 connected to the information processing device 110, and a user terminal 140 connected to the image generating device 120. The connections between the devices may be wired or wireless.

[0013] The information processing device 110 is a device that provides parameters for generating a virtual viewpoint image in which additional information is displayed. Specifically, the information processing device 110 is a device that issues an instruction to register virtual advertisement data, sets information on the display conditions of the set virtual advertisement data, a time code, and virtual viewpoint information in parameters, and transmits the information to the image generating device 120. The information processing device 110 has an operation receiving unit 111, a virtual advertisement setting unit 112, a registration instruction unit 113, a parameter setting unit 114, and a parameter transmission unit 118.

[0014] The operation receiving unit 111 receives user operation input (also referred to as user input). Information generated according to the received input operation is sent to each functional unit according to the operation content. The virtual advertisement setting unit 112 sets virtual advertisement data to be synthesized with the virtual viewpoint video (added onto the virtual viewpoint image) according to the information sent from the operation receiving unit 111. The virtual advertisement data is, for example, image data of an advertising billboard that does not exist in the virtual viewpoint image (basic 3D data) to be added. The virtual advertisement data to be set may be one or more, and may be one type or multiple types. Examples of the virtual advertisement data to be set include data of an image (see FIG. 3(b)) generated based on a 3D model of a virtual advertisement, which will be described in detail later. The virtual advertisement data set by the virtual advertisement setting unit 112 is sent to the registration instruction unit 113 and the display condition setting unit 115 of the parameter setting unit 114.

[0015] The registration instruction unit 113 instructs the image generation device 120 to register the virtual advertisement data set by the virtual advertisement setting unit 112. Specifically, the registration instruction unit 113 sends instruction information instructing the registration of this virtual advertisement data to the registration acceptance unit 121 of the image generation device 120 together with the set virtual advertisement data, and instructs the registration acceptance unit 121 to register the sent virtual advertisement data in the image generation device 120.

[0016] The parameter setting unit 114 has a display condition setting unit 115, a time code setting unit 116, and a virtual viewpoint setting unit 117. The parameter setting unit 114 sets various types of information in virtual viewpoint image generation parameters (hereinafter also referred to as parameters) for each frame of the virtual viewpoint image generated by the image generation device 120, in accordance with an input operation received by the operation receiving unit 111. That is, the parameters are information for each frame of the virtual viewpoint image generated by the image generation device 120, and are information in which various types of information are set. The parameters in which various types of information are set are sent to a parameter sending unit 118.

[0017] The display condition setting unit 115 selects virtual advertisement data to be set from the virtual advertisement data set by the virtual advertisement setting unit 112 in accordance with the information received from the operation receiving unit 111. Then, the display condition setting unit 115 sets the display conditions of the selected virtual advertisement data and identification information that uniquely identifies the selected virtual advertisement as parameters. The display conditions include the display start time and display period of the selected virtual advertisement, etc.

[0018] The time code setting unit 116 sets a time code for generating a virtual viewpoint image as a parameter in accordance with information received from the operation receiving unit 111. The time code to be set corresponds to a time code linked to basic 3D data corresponding to the virtual viewpoint image, the details of which will be described later.

[0019] According to the information received from the operation receiving unit 111, the virtual viewpoint setting unit 117 sets virtual viewpoint information including the position and orientation of the virtual camera, which is the virtual viewpoint, and the angle of view corresponding to the virtual camera, as parameters.

[0020] The parameter transmission unit 118 outputs parameters for which information such as conditions has been set by the display condition setting unit 115, the time code setting unit 116, and the virtual viewpoint setting unit 117 to the outside of the information processing device 110. Specifically, the parameter transmission unit 118 transmits the parameters for which various information has been set to the parameter reception unit 123 of the image generation device 120. Note that the parameter transmission unit 118 may transmit the parameters one by one or multiple parameters together. A set of multiple consecutive parameters from one time to another time is also called a virtual camera path, as the trajectory of the virtual viewpoint forms a path.

[0021] The image generating device 120 is a device that generates a virtual viewpoint image in which a virtual advertisement is displayed frame by frame in accordance with parameters received from the information processing device 110. In other words, the image generating device 120 can also be said to be a device that generates a virtual viewpoint image in which a desired virtual advertisement is added at a desired position. The image generating device 120 has a registration accepting unit 121, a virtual advertisement storage unit 122, a parameter receiving unit 123, a parameter extracting unit 124, a virtual advertisement reading unit 128, a basic data storage unit 129, a basic data reading unit 130, a synthesizing unit 131, and a rendering unit 132.

[0022] The registration receiving unit 121 receives instruction information and virtual advertisement data instructed to be registered from the registration instruction unit 113 of the information processing device 110. The registration receiving unit 121 sends the received virtual advertisement data to the virtual advertisement storage unit 122 to register the virtual advertisement data in accordance with the instruction information.

[0023] The virtual advertisement storage unit 122 stores the virtual advertisement data sent from the registration acceptance unit 121 .

[0024] The parameter receiving unit 123 acquires parameters in which display conditions for additional information to be displayed in the virtual viewpoint image, identification information that uniquely identifies the additional information, virtual viewpoint information of the virtual viewpoint image to be generated, and a time code are set. Specifically, the parameter receiving unit 123 receives parameters transmitted from the parameter transmitting unit 118 of the information processing device 110.

[0025] The parameter extraction unit 124 has a display condition extraction unit 125, a time code extraction unit 126, and a virtual viewpoint extraction unit 127. The parameter extraction unit 124 extracts various pieces of information set in each functional unit from the parameters received by the parameter reception unit 123.

[0026] The display condition extraction unit 125 extracts, from the parameters received by the parameter receiving unit 123, identification information of the virtual advertisement data set by the display condition setting unit 115 and information on the display conditions of the virtual advertisement data, which are included in the parameters. The identification information of the virtual advertisement data and the information on the display conditions of the virtual advertisement data are sent to the virtual advertisement reading unit 128.

[0027] The virtual advertisement reading unit 128 reads out the virtual advertisement data corresponding to the identification information extracted from the parameters by the display condition extraction unit 125 from the virtual advertisement storage unit 122. The read out virtual advertisement data is sent to the synthesis unit 131.

[0028] The basic data storage unit 129 stores 3D data of basic models (hereinafter referred to as basic 3D data). The basic 3D data is data obtained by 3D modeling an object that actually exists. Note that, since a virtual viewpoint image is generated using the basic 3D data, the basic 3D data can also be said to be shape data that indicates the shape of an object included in the generated virtual viewpoint image.

[0029] The time code extraction unit 126 extracts the time code included in the parameter from the parameter received by the parameter receiving unit 123. The extracted time code is data set by the time code setting unit 116 and matches the time code of the basic 3D data. The extracted time code is sent to the basic data reading unit 130. Note that the parameters from which the time code is extracted are the same as the parameters from which the display condition extraction unit 125 extracts.

[0030] The basic data reading unit 130 reads out the basic 3D data corresponding to the time code extracted by the time code extraction unit 126 from the basic data storage unit 129. The read basic 3D data is sent to a synthesis unit 131.

[0031] The synthesis unit 131 synthesizes the basic 3D data read by the basic data reading unit 130 and the virtual advertisement data read by the virtual advertisement reading unit 128 to generate one piece of synthesized 3D data. The generated piece of synthesized 3D data is sent to the rendering unit 132.

[0032] The virtual viewpoint extraction unit 127 extracts virtual viewpoint information (virtual camera position information) indicating the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera, which are included in the parameters received by the parameter receiving unit 123. The extracted virtual viewpoint information is sent to the rendering unit 132. Note that the parameters to be extracted as the virtual viewpoint information are the same as the parameters to be extracted by the display condition extraction unit 125.

[0033] The rendering unit 132 performs rendering processing on the composite 3D data generated by the composition unit 131 in accordance with the virtual viewpoint information extracted by the virtual viewpoint extraction unit 127, and generates a virtual viewpoint image in which a virtual advertisement is displayed (a virtual viewpoint image with a virtual advertisement). The virtual viewpoint information includes the position and orientation of the virtual camera and the angle of view corresponding to the virtual camera. The generated virtual viewpoint image with a virtual advertisement is sent to a user terminal 140 or the like external to the image generation device 120.

[0034] The user terminal 140 is a device that displays the virtual viewpoint image with the virtual advertisement sent from the image generating device 120. The user terminal 140 may be any information processing device such as a personal computer including a liquid crystal display, a tablet terminal including a touch panel, or a smartphone.

[0035] Although the image processing system having the information processing device 110, the image generating device 120, and the user terminal 140 has been described as an example, the present invention is not limited to this. The information processing device 110 may include the functions of the image generating device 120, the functions of the user terminal 140, or the functions of both the image generating device 120 and the user terminal 140. The image generating device 120 may also include the functions of the user terminal 140.

[0036] <Hardware configuration of each device> The hardware configurations of the information processing device 110, the image generating device 120, and the user terminal 140 will be described using the drawings. Fig. 2 is a diagram showing an example of the hardware configuration of the information processing device 110, the image generating device 120, and the user terminal 140. The information processing device 110, the image generating device 120, and the user terminal 140 have a common hardware configuration, including a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication unit 217, and a bus 218.

[0037] The CPU 211 controls the entire image processing system using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing the functions of the information processing device 110 and the image generating device 120 shown in FIG. 1(b). The information processing device 110 and the image generating device 120 may each have one or more dedicated hardware components different from the CPU 211, and at least a portion of the processing performed by the CPU 211 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a digital signal processor (DSP). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, as well as data supplied from the outside via the communication unit 217. The auxiliary storage device 214 is configured, for example, by a hard disk drive or the like, and stores various data such as image data and audio data.

[0038] The display unit 215 is configured with, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to operate the information processing device 110. The operation unit 216 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user to input various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216.

[0039] The communication unit 217 is used for communication between the information processing device 110, the image generation device 120, and the user terminal 140 and external devices. For example, if the information processing device 110, the image generation device 120, and the user terminal 140 are connected to external devices via a wired connection, a communication cable is connected to the communication unit 217. If the information processing device 110, the image generation device 120, and the user terminal 140 have the function of wirelessly communicating with external devices, the communication unit 217 is equipped with an antenna. The bus 218 connects the various components of the information processing device 110, the image generation device 120, and the user terminal 140 to transmit information.

[0040] In this embodiment, the display unit 215 and the operation unit 216 are assumed to exist inside the information processing device 110, but at least one of the display unit 215 and the operation unit 216 may exist as a separate device outside the information processing device 110. The display unit 215 and the operation unit 216 may also be integrated.

[0041] <Generation of virtual viewpoint images with virtual advertisements> The generation of a virtual viewpoint image with a virtual advertisement according to a virtual viewpoint based on 3D data (three-dimensional data) will be explained using diagrams. FIG. 3 is a diagram explaining the generation of a virtual viewpoint image with a virtual advertisement according to a virtual viewpoint based on 3D data corresponding to an image (video) of a soccer game. FIG. 3(a) shows an example of a virtual viewpoint image generated according to a virtual viewpoint based on basic 3D data. FIG. 3(b) shows an example of an image of virtual advertisement data according to the virtual viewpoint of the virtual viewpoint image of FIG. 3(a). FIG. 3(c) shows an example of a virtual viewpoint image with a virtual advertisement obtained by combining the virtual viewpoint image of FIG. 3(a) and the image of the virtual advertisement data of FIG. 3(b).

[0042] 3(a) is an image of an actually existing object viewed from an arbitrary viewpoint (virtual viewpoint), and includes a field 311, players 312 on the field, and advertising billboards 313 installed on the side of the field 311. The virtual viewpoint image 310 is an image generated in accordance with a virtual viewpoint based on basic 3D data pre-stored in the basic data storage unit 129. The basic 3D data is, for example, shape data indicating the shape of an object generated by known technology based on multiple captured images obtained by capturing an object area from different viewpoints using multiple imaging devices.

[0043] Image 320 of the virtual advertisement data shown in FIG. 3(b) is an image of a virtual advertisement that can be displayed in an area of ​​virtual viewpoint image 310 that does not overlap with objects that are the focus of the viewer's attention, such as players 312 and advertising billboards 313 that are present in virtual viewpoint image 310. That is, image 320 of the virtual advertisement data includes two billboards 321 and 322 that are arranged side by side along the halfway line of field 311 in virtual viewpoint image 310. Billboard 321 is an image of a virtual advertisement with the texture "Abcde" written in the center. Billboard 322 is an image of a virtual advertisement with the texture "ABCDE" written in the center.

[0044] 3(c) is an image generated by the rendering unit 132 according to the virtual viewpoint based on the synthesized 3D data obtained by synthesizing the basic 3D data and the virtual advertisement data by the synthesizing unit 131. Therefore, by preparing the virtual advertisement data to be displayed in the virtual viewpoint image, it is possible to generate a virtual viewpoint image with a virtual advertisement in which an advertising billboard appears to exist near the halfway line of the field, a place that does not actually exist.

[0045] <Image processing system operation> Next, the flow of processing executed by the image processing system will be explained using the drawings. FIG. 4 is a flowchart showing the flow of processing executed by the image processing system. FIG. 4(a) shows the flow of processing executed by the image processing system, FIG. 4(b) shows a detailed flow of processing for setting parameters for generating a virtual viewpoint image, and FIG. 4(c) shows a detailed flow of processing for generating a virtual viewpoint image with a virtual advertisement. The processing shown in FIG. 4 is performed by the CPU 211 reading and executing a computer program stored in the ROM 212 or the auxiliary storage device 214. Hereinafter, in the explanation of FIG. 4, processing steps will be simply denoted as S.

[0046] 4(a), in S401, the information processing device 110 of the image processing system accepts a user operation and sets various pieces of information generated in accordance with the accepted operation content as parameters. The detailed flow of the setting process will be described later with reference to the drawings. The parameters in which the various pieces of information have been set are transmitted to the image generating device 120 of the image processing system.

[0047] In S402, the image generating device 120 of the image processing system generates a virtual viewpoint image with a virtual advertisement according to various information set in the parameters. That is, the image generating device 120 generates a virtual viewpoint image with a virtual advertisement according to the virtual viewpoint information set in the parameters based on synthesized 3D data obtained by synthesizing basic 3D data corresponding to the time code set in the parameters with the virtual advertisement data indicated by the identification information. A detailed flow of the generation process will be described later using figures. The generated virtual viewpoint image with a virtual advertisement is sent to a display, which is the display unit 215 of the image generating device 120, a user terminal 140 connected to the image generating device 120, or the like.

[0048] Next, a detailed flow of the process of setting parameters for generating a virtual viewpoint image will be described with reference to the drawings. As shown in FIG. 4(b), in S411, the virtual advertisement setting unit 112 reads virtual advertisement data created in advance. The virtual advertisement data may be read from a virtual advertisement database (not shown) connected to the information processing device 110, or from a USB memory connected to the information processing device 110. The read virtual advertisement data is sent to the registration instruction unit 113 and the display condition setting unit 115.

[0049] In S412 , the registration instruction unit 113 transmits the target virtual advertisement data to the image generation device 120 together with instruction information for registering the virtual advertisement data read by the virtual advertisement setting unit 112 in the virtual advertisement storage unit 122 of the image generation device 120 .

[0050] In S413, the operation receiving unit 111 receives a user operation for inputting parameters for generating a virtual viewpoint image.

[0051] In S414, the operation receiving unit 111 determines whether or not an instruction to display a virtual advertisement has been included in the information generated in accordance with the input operation received by the operation receiving unit 111 in S413. If the determination result indicates that an instruction to display a virtual advertisement has been included in the information generated in accordance with the input operation (YES in S414), the process proceeds to S415. On the other hand, if the determination result indicates that an instruction to display a virtual advertisement has not been included in the information generated in accordance with the input operation (NO in S414), S415 is skipped and the process proceeds to S416.

[0052] In S415, the display condition setting unit 115 sets the display conditions of the virtual advertisement data to be selected and the identification information of the virtual advertisement to be selected as parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation received by the operation receiving unit 111 in S413.

[0053] In S416, the time code setting unit 116 sets a time code in the parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation received by the operation receiving unit 111 in S413.

[0054] In S417, the virtual viewpoint setting unit 117 sets virtual viewpoint information in the parameters to be transmitted to the image generating device 120 based on the information generated in accordance with the input operation received by the operation receiving unit 111 in S413. The virtual viewpoint information includes the position and orientation of the virtual camera, and the angle of view corresponding to the virtual camera.

[0055] In S418, the parameter transmission unit 118 transmits to the parameter reception unit 123 of the image generation device 120 the parameters in which the various information generated in the processes from S415 to S417 is set.

[0056] In S419, the information processing device 110 determines whether or not a new parameter has been input. If the determination result indicates that a new parameter has been input (YES in S419), the process returns to S413, and a series of processes from S413 to S418 are executed for the input new parameter. On the other hand, if the determination result indicates that a new parameter has not been input (NO in S419), the flow shown in FIG. 4(b) ends.

[0057] Next, a detailed flow of the process of generating a virtual viewpoint image with a virtual advertisement will be described with reference to the drawings. As shown in Fig. 4(c), in S421, the registration receiving unit 121 receives the target virtual advertisement data together with the instruction information transmitted from the registration instruction unit 113 of the information processing device 110.

[0058] In S422, the virtual advertisement storage unit 122 stores the virtual advertisement data accepted by the registration acceptance unit 121 in S421. As a result, the virtual advertisement data is registered in the image generation device 120.

[0059] In S423, the parameter receiving unit 123 receives the parameters transmitted by the parameter transmitting unit 118 in S418.

[0060] In S424, the basic data reading unit 130 reads, from the basic data storage unit 129, basic 3D data that corresponds to the time code extracted from the parameters by the time code extraction unit 126 of the parameter extraction unit 124. Note that the parameters from which the time codes are extracted are the parameters received by the parameter receiving unit 123 in S423.

[0061] In S425, the display condition extraction unit 125 of the parameter extraction unit 124 determines whether or not the parameters received by the parameter receiving unit 123 in S423 include information related to the display conditions of the virtual advertisement data. If the determination result indicates that the parameters include information related to the display conditions of the virtual advertisement data (YES in S425), the process proceeds to S426. On the other hand, if the determination result indicates that the parameters do not include information related to the display conditions of the virtual advertisement data (NO in S425), the process skips S426 and S427 and proceeds to S428.

[0062] In S426, the virtual advertisement reading unit 128 reads out the virtual advertisement data corresponding to the identification information extracted from the parameters by the display condition extraction unit 125 from the virtual advertisement storage unit 122. Note that the parameters from which the identification information is extracted are the parameters received by the parameter receiving unit 123 in S423.

[0063] In S427, the synthesis unit 131 synthesizes the basic 3D data read by the basic data reading unit 130 in S424 and the virtual advertisement data read by the virtual advertisement reading unit 128 in S426 to generate synthesized 3D data.

[0064] In S428, the rendering unit 132 performs rendering processing on the composite 3D data generated in S427 or the basic 3D data read out in S424 in accordance with the virtual viewpoint information extracted from the parameters by the virtual viewpoint extraction unit 127. This processing generates a virtual viewpoint image with a virtual advertisement or a virtual viewpoint image. The generated virtual viewpoint image with a virtual advertisement or a virtual viewpoint image is sent as an image signal to the user terminal 140 or the like external to the image generating device 120. Note that the parameters from which the virtual viewpoint information is extracted are the parameters received by the parameter receiving unit 123 in S423.

[0065] In S429, the image generating device 120 determines whether or not new parameters have been received. If the determination result indicates that new parameters have been received (YES in S429), the process returns to S423, and the series of processes from S423 to S428 are executed for the new parameters that have been received. On the other hand, if the determination result indicates that new parameters have not been received (NO in S429), the flow shown in FIG. 4(c) ends.

[0066] 4(a) and 4(b), the parameter setting process including the reading and registration of virtual advertisement data has been described, but the present invention is not limited to this. That is, the virtual advertisement data may be read and registered in advance, and then the parameter setting process including the setting of the display conditions of the virtual advertisement may be performed.

[0067] 4(a) and 4(c) have described the process of generating a virtual viewpoint image including the reception and storage of an instruction to register virtual advertisement data, but the present invention is not limited to this. That is, the reception and storage of an instruction to register virtual advertisement data may be performed in advance, and then the process of generating a virtual viewpoint image with a virtual advertisement may be performed.

[0068] <Processing flow between devices in the image processing system> The flow of processing between devices in an image processing system will be described using a diagram. Fig. 5 is a sequence diagram showing the flow of processing between an information processing device 110 and an image generating device 120 in the image processing system. Hereinafter, in the description of Fig. 5 and Fig. 10 described later, the processing sequence will simply be represented as S.

[0069] In S501, the information processing device 110 (registration instruction unit 113) instructs the image generation device 120 to register the virtual advertisement data "A" set by the virtual advertisement setting unit 112. S501 corresponds to the processing of S412 in the flowchart of FIG. 4(b) and the processing of S421 in the flowchart of FIG. 4(c). Here, an example of virtual advertisement data will be described using figures. FIG. 6 is a diagram showing an example of virtual advertisement data registration and examples of parameters for generating a virtual viewpoint image, and FIG. 6(a) shows an example of a 3D model format of the virtual advertisement data transmitted at the time of registration. As shown in FIG. 6(a), the 3D model format of the virtual advertisement data specifies the file names and file sizes of the obj file, mtl file, and png file. The obj file stores information such as vertices and normals of the 3D model (three-dimensional model) of the virtual advertisement. The mtl (material) file stores color information and texture information of the 3D model of the virtual advertisement. The png file stores parameters of the 3D model of the virtual advertisement.

[0070] Here, the obj file will be described with reference to the drawings. FIG. 7 is a diagram showing an example of an obj file specified in the 3D model format of virtual advertisement data. In the obj file, vertex coordinate values ​​are specified by the keyword "v", texture coordinate values ​​are specified by the keyword "vt", and normal vectors are specified by the keyword "vn". In addition, in this embodiment, the obj file is used as an example of a file format representing information related to the position, size, and orientation of the 3D model of virtual advertisement data, but this is not limiting. The file format may be any other format as long as it can represent the position, size, and orientation of the 3D model of virtual advertisement data.

[0071] Returning to the explanation of FIG. 5, in S502, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 parameters for generating a virtual viewpoint image (without virtual advertisement) that are set by the parameter setting unit 114 and do not accompany an instruction to display a virtual advertisement. An example of parameter (without virtual advertisement) data for generating a virtual viewpoint image will now be described with reference to a diagram. FIG. 6(b) is a diagram showing an example format of parameter (without virtual advertisement) data for generating a virtual viewpoint image (hereinafter also referred to as parameter (without virtual advertisement) data). In the format of the parameter (without virtual advertisement) data, the position of the virtual camera is specified by "rotation quaternion" and the orientation of the virtual camera is specified by "translation vector". In the format of the parameter (without virtual advertisement) data, furthermore, information indicating the angle of view corresponding to the virtual camera is specified by "horizontal angle", and a time code is specified by "time code". That is, in the format of the parameter (without virtual advertisement) data shown in FIG. 6(b), virtual viewpoint information is specified, but information related to the virtual advertisement is not specified.

[0072] In S503, similarly to S502, the information processing device 110 transmits to the image generating device 120 the parameters (without virtual advertisement) set by the parameter setting unit 114 and not accompanied by an instruction to display the virtual advertisement.

[0073] Here, an example of a virtual viewpoint image generated according to the extracted virtual viewpoint information by the image generating device 120 in S502 and S503, extracted from the parameters (without virtual advertisement), will be described with reference to the drawings. When the rendering unit 132 performs rendering processing on the basic 3D data according to the virtual viewpoint information extracted from the parameters (without virtual advertisement), a virtual viewpoint image without an added virtual advertisement, as shown in FIG. 3(a), is generated. However, the time code extracted from the parameters (without virtual advertisement) received in S503 is different from the time code extracted from the parameters (without virtual advertisement) received in S502. In other words, the parameters received in S503 and the parameters received in S502 target different frames.

[0074] In S504, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 the parameters for generating a virtual viewpoint image (virtual advertisement "A") accompanied by a display instruction for a virtual advertisement, which have been set by the parameter setting unit 114. An example of the parameter (virtual advertisement "A") data for generating a virtual viewpoint image will now be described with reference to the drawings. FIG. 6(c) is a diagram showing an example format of the parameter (virtual advertisement "A") data for generating a virtual viewpoint image (hereinafter also referred to as parameter (virtual advertisement "A") data). The format of the parameter (virtual advertisement "A") data specifies the position and orientation of the virtual camera, the angle of view corresponding to the virtual camera, and a time code, which are also specified in the parameters (without virtual advertisement) shown in FIG. 6(b). The format of the parameter (virtual advertisement "A") data further specifies information regarding a display instruction for the virtual advertisement (virtual advertisement "A" in FIG. 6(c)) in "advertise display". This parameter (virtual advertisement "A") includes an instruction to display the virtual advertisement "A" ("advertise ID": "A" in FIG. 6(a)) instructed to be registered in the processing of S501 in the virtual viewpoint image. Upon receiving such parameter (virtual advertisement "A") data, the image generating device 120 generates a virtual viewpoint image in which the virtual advertisement is displayed. However, the time code extracted from the parameters received in S504 is different from the time code extracted from the parameters received in S502 to S503. In other words, the parameters received in S502 to S504 each target different frames.

[0075] In S505, similarly to S504, the information processing device 110 transmits to the image generating device 120 the parameters (virtual advertisement "A") set by the parameter setting unit 114 and accompanied by an instruction to display the virtual advertisement.

[0076] Here, an example of a virtual viewpoint image with a virtual advertisement that the image generating device 120 extracts virtual viewpoint information from the parameters (virtual advertisement "A") in S504 and S505 and generates according to the extracted virtual viewpoint information will be described with reference to the drawings. According to the virtual viewpoint information extracted from the parameters (virtual advertisement "A"), the rendering unit 132 performs a rendering process on the composite 3D data that combines the basic 3D data and the virtual advertisement data. This process generates a virtual viewpoint image with a virtual advertisement added, as shown in FIG. 3(c). However, the time code extracted from the parameters (virtual advertisement "A") received in S505 is different from the time code extracted from the parameters received in S502 to S504. In other words, the parameters (virtual advertisement "A") received in S505 and the parameters received in S502 to S504 target different frames.

[0077] In S506, similarly to S502 and S503, the information processing device 110 again transmits to the image generating device 120 the parameters (without virtual advertisement) set by the parameter setting unit 114 that do not include an instruction to display a virtual advertisement. In the image generating device 120, when the rendering unit 132 performs rendering processing on the basic 3D data in accordance with the virtual viewpoint information extracted from the parameters (without virtual advertisement), a virtual viewpoint image without an added virtual advertisement is again generated. However, the time code extracted from the parameters (without virtual advertisement) received in S506 is different from the time code extracted from the parameters received in S502 to S505. In other words, the parameters (without virtual advertisement) received in S506 and the parameters received in S502 to S505 target different frames.

[0078] As explained above using FIG. 5, it is possible to set a desired virtual advertisement to be added (displayed) at a desired position in a desired virtual viewpoint image, or to set a virtual advertisement not to be added (displayed), on a frame-by-frame basis.

[0079] <UI for setting virtual ad display conditions> An example of a user interface (UI) for setting conditions for displaying a virtual advertisement in a virtual viewpoint image will be described with reference to the drawings. Fig. 8 is a diagram showing an example of a UI for setting display conditions for a virtual advertisement, where Fig. 8(a) shows a case where the virtual advertisement is set to be hidden, and Fig. 8(b) shows a case where the virtual advertisement is set to be displayed. When the display conditions for a virtual advertisement are set by a user operation on the UI for setting display conditions for a virtual advertisement, information regarding the display conditions for the virtual advertisement is set in parameters by the display condition setting unit 115.

[0080] The UI 801 for setting display conditions for a virtual advertisement includes two radio buttons 802 and 803. The radio button 802 is a button for setting a virtual advertisement not to be displayed in the virtual viewpoint image. The radio button 803 is a button for additionally displaying a virtual advertisement in the virtual viewpoint image and for specifying the virtual advertisement to be displayed. The radio button 803 is arranged in a list box that lists a plurality of registered virtual advertisements.

[0081] If the user does not want to display a virtual advertisement in the virtual viewpoint image, the user selects a radio button 802 corresponding to "None" to hide the virtual advertisement, as shown in Fig. 8(a). The selection of the radio button 802 in this manner corresponds to the processing sequence of S502, S503, and S506 in the sequence diagram of Fig. 5.

[0082] On the other hand, when a user wants to display a virtual advertisement in the virtual viewpoint image, the user selects a radio button 803 corresponding to the virtual advertisement to be displayed ("Virtual Advertisement A") at the desired timing, as shown in Fig. 8(b). The selection of the radio button 803 in this manner corresponds to the processing sequence of S504 and S505 in the sequence diagram of Fig. 5.

[0083] <User operation screen> Here, a user operation screen for setting a virtual viewpoint corresponding to a virtual viewpoint image and display conditions of a virtual advertisement will be described with reference to the drawings. FIG. 9 is a diagram showing an example of a user operation screen, where FIG. 9(a) shows a case where the non-display of a virtual advertisement is set, and FIG. 9(b) shows a case where the display of a virtual advertisement is set. FIG. 9 shows a case where a virtual viewpoint image corresponding to the set display conditions is displayed on the left display, and a UI for setting display conditions of a virtual advertisement is displayed on the right display. Note that the display form of the virtual viewpoint image and the UI for setting display conditions of a virtual advertisement is not limited to this. The virtual viewpoint image and the UI for setting display conditions of a virtual advertisement may be displayed on a single display.

[0084] As shown in Fig. 9(a), the left display 901 displays a virtual viewpoint image 911 generated in accordance with the virtual viewpoint based on basic 3D data. The virtual viewpoint is initially set, and before a user operation related to the virtual viewpoint is accepted, the initially set virtual viewpoint is used as the virtual viewpoint. The right display 902 displays a virtual advertisement display designation UI 912 in a state in which a radio button for setting whether to hide a virtual advertisement for the image displayed on the left display 901 has been selected by a user operation. Since whether to hide a virtual advertisement has been selected, the left display displays a virtual viewpoint image without any virtual advertisement added.

[0085] As shown in Fig. 9(b), the left display 901 displays a virtual viewpoint image 921 generated in accordance with a virtual viewpoint based on basic 3D data. The right display 902 displays a virtual advertisement display designation UI 922 in a state in which a radio button for setting the display of virtual advertisement A for the image displayed on the left display 901 has been selected by a user operation. Note that since the display of the virtual advertisement has been selected, the left display displays the virtual viewpoint image with the virtual advertisement added.

[0086] When a game is played over the entire field and players are present across the entire field, as in the virtual viewpoint image 911 displayed on the display 901, the radio button corresponding to "None" (no virtual advertisement) is selected in the UI 912. Therefore, no virtual advertisement is added to the virtual viewpoint image 911 displayed on the left display 901.

[0087] As shown in virtual viewpoint image 921 displayed on display 901, when players gather on one side of the field and there are no players on the other side, the radio button corresponding to virtual advertisement A is selected in UI 922. Therefore, a virtual advertisement is added to virtual viewpoint image 921 displayed on left display 901. By registering virtual advertisement data in advance and selecting a virtual advertisement at the desired timing on the UI displayed on right display 902 while viewing the virtual viewpoint image displayed on left display 901, the virtual advertisement can be instantly displayed on the virtual viewpoint image.

[0088] In this embodiment, when a radio button corresponding to virtual advertisement A is selected by a user operation, the parameter setting unit 114 sets a display instruction (display condition) for virtual advertisement A as a parameter, and the parameter transmission unit 118 transmits the parameter to the image generation device 120. However, the operation method is not limited to this, and a key assigned to a keyboard may be pressed, or an operation using a touch panel may be used. Furthermore, the virtual viewpoint may be adjusted by a user operation using a virtual viewpoint image 911 displayed on the left display 901. When the virtual viewpoint is adjusted (set) by a user operation, the parameter setting unit 114 sets virtual viewpoint information as a parameter, and the parameter transmission unit 118 transmits the parameter to the image generation device 120.

[0089] This makes it possible to set a virtual advertisement to be displayed in an area where no players are present on the field, depending on the camera angle, in a scene where a penalty kick is being taken in a soccer match, for example, and to generate a virtual viewpoint image with a virtual advertisement according to this setting. In other words, it becomes possible to display a virtual advertisement in an area of ​​the field where no signboard can be placed, without overlapping with the players present on the field.

[0090] As described above, according to this embodiment, virtual advertisement data (3D models and still image textures) are registered in advance, and identification information for the virtual advertisement data, display conditions for the virtual advertisement, virtual viewpoint information, and a time code are set as parameters, thereby achieving the following effects: A virtual viewpoint image with a virtual advertisement or a virtual viewpoint image without a virtual advertisement can be generated according to information extracted from the parameters. Therefore, it is possible to switch between displaying and hiding a virtual advertisement in real time on a frame-by-frame basis, and display an effective virtual advertisement suited to the scene in the virtual viewpoint image. In other words, it is possible to generate a virtual viewpoint image in which a virtual advertisement is displayed at a desired position at a desired timing.

[0091] [Second embodiment] In this embodiment, a mode of switching and displaying multiple types of registered virtual advertisement data will be described using the drawings. Fig. 10 is a sequence diagram showing the flow of processing between the information processing device 110 and the image generating device 120 when multiple types of virtual advertisement data are registered and display conditions for each of the multiple types of registered virtual advertisement data are set. In this embodiment, the processing of registering a virtual advertisement and setting display conditions for the registered virtual advertisement is the same as that in the image processing system of the first embodiment described above, and therefore a description thereof will be omitted.

[0092] In S1001, the information processing device 110 (registration instruction unit 113) instructs the image generating device 120 to register the virtual advertisement data "AA" set by the virtual advertisement setting unit 112. Note that S1001 corresponds to the processing of S412 in the flowchart of FIG. 4(b) and the processing of S421 in the flowchart of FIG. 4(c).

[0093] Similarly, in S1002, the information processing device 110 instructs the image generating device 120 to register the virtual advertisement data "BB" set by the virtual advertisement setting unit 112. In S1003, the information processing device 110 instructs the image generating device 120 to register the virtual advertisement data "CC" set by the virtual advertisement setting unit 112.

[0094] Here, examples of virtual advertisement data "AA", "BB", and "CC" will be explained using figures. FIG. 11 shows example images of virtual advertisement data displayed in a virtual viewpoint image, where FIG. 11(a) shows an example image of virtual advertisement data "AA", FIG. 11(b) shows an example image of virtual advertisement data "BB", and FIG. 11(c) shows an example image of virtual advertisement data "CC". As shown in FIGS. 11(a), 11(b), and 11(c), virtual advertisements 1101, 1102, and 1103 have the same texture but are displayed at different positions.

[0095] Although the above description has been given of a case where three types of data with different display positions of the virtual advertisement are registered, the present invention is not limited to this. In other words, multiple types of virtual advertisement data corresponding to the desired display position and size and having the desired texture may be created in advance, and all of them may be registered in advance.

[0096] In S1004, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 the parameters (without virtual advertisement) set by the parameter setting unit 114 and not accompanied by an instruction to display a virtual advertisement.

[0097] In S1005, the information processing device 110 (parameter transmission unit 118) transmits to the image generating device 120 parameters (virtual advertisement "AA") set by the parameter setting unit 114 and accompanied by a display instruction for the virtual advertisement "AA". Similarly, in S1006, the information processing device 110 transmits to the image generating device 120 parameters (virtual advertisement "BB") set by the parameter setting unit 114 and accompanied by a display instruction for the virtual advertisement "BB". Furthermore, in S1007, the information processing device 110 transmits to the image generating device 120 parameters (virtual advertisement "CC") set by the parameter setting unit 114 and accompanied by a display instruction for the virtual advertisement "CC". However, the time codes extracted from the parameters received in S1004 to S1007 are different from the time codes extracted from the parameters received in the other processing sequences other than the information processing device itself. That is, the parameters received in S1004 to S1007 each target different frames.

[0098] <Virtual ad display setting UI> An example of a user interface (UI) for setting display conditions of a virtual advertisement for a virtual viewpoint image will be described with reference to the drawings. FIG. 12 shows an example of a UI for setting display conditions of a virtual advertisement. FIG. 12(a) shows an example of a UI for registering a virtual advertisement, FIG. 12(b) shows an example of a UI for creating a virtual advertisement set, and FIG. 12(c) shows an example of a UI for setting a virtual advertisement schedule. When a virtual advertisement is registered by a user operation in the UI for registering a virtual advertisement, the virtual advertisement setting unit 112 registers the virtual advertisement data in the image generating device 120. When display conditions of a virtual advertisement (virtual advertisement set) are created by a user operation in the UI for creating a virtual advertisement set, the display condition setting unit 115 sets information about the display conditions of the virtual advertisement in parameters. When display conditions of a virtual advertisement (virtual advertisement schedule) are set by a user operation in the UI for setting a virtual advertisement schedule, the display condition setting unit 115 sets information about the display conditions of the virtual advertisement in parameters.

[0099] As shown in FIG. 12(a), the virtual advertisement registration UI 1210 is a user interface for registering a virtual advertisement. The virtual advertisement registration UI 1210 has a list box 1211 that selectably displays virtual advertisements to be registered, and a "Register" button 1212 for registering a selected virtual advertisement. The list box 1211 displays a plurality of sets of images showing the virtual advertisements and their display positions, the names of the virtual advertisements, and radio buttons for selecting the virtual advertisements. In the example of FIG. 12(a), the list box 1211 displays virtual advertisements AA, BB, and CC.

[0100] Virtual advertisement AA is data indicating that a virtual advertisement with a centered texture "Abcde" is to be displayed in the vertical center and to the left of the horizontal center in the virtual viewpoint image. Virtual advertisement BB is data indicating that a virtual advertisement with a centered texture "Abcde" is to be displayed in the vertical center and to the left of the horizontal center in the virtual viewpoint image. Virtual advertisement CC is data indicating that a virtual advertisement with a centered texture "Abcde" is to be displayed in the vertical center and to the right of the horizontal center in the virtual viewpoint image. By selecting the radio button corresponding to the virtual advertisement you want to register and pressing the "Register" button 1212, the selected virtual advertisement is registered in the image generating device 120.

[0101] As shown in FIG. 12(b), virtual advertisement set creation UI 1220 has a list box 1221, a creation area 1222, and a confirmation area 1223. List box 1221 displays a list of registered virtual advertisements. Creation area 1222 displays a list of virtual advertisements that can be created as virtual advertisement sets. Confirmation area 1223 displays a list of created virtual advertisement sets. In the example of FIG. 12(b), confirmation area 1223 shows a state in which virtual advertisement set 1237 that displays virtual advertisement AA, virtual advertisement BB, and virtual advertisement CC in that order has been registered.

[0102] Here, a method for creating a virtual advertisement set will be described using diagrams. Fig. 13 is a diagram for explaining a method for creating a virtual advertisement set. Fig. 13(a) shows a state in which registered virtual advertisements are displayed as a list, Fig. 13(b) shows a state in which virtual advertisements that are targets of the virtual advertisement set have been specified (identified), and Fig. 13(c) shows a state in which the virtual advertisement set has been registered.

[0103] When a virtual advertisement is registered by a user operation, the virtual advertisement setting unit 112 displays a list of the virtual advertisements to be registered in a list box 1221 of a virtual advertisement set creation UI 1220, in accordance with the operation content of the user operation, as shown in Fig. 13(a). In the example of Fig. 13(a), virtual advertisements AA1231, virtual advertisement BB1232, virtual advertisement CC1233, etc. are displayed in a list.

[0104] Then, by a user operation, a drag-and-drop mouse operation is performed on the virtual advertisement to be used as a virtual advertisement set, and the virtual advertisement to be used as a virtual advertisement set is moved from within list box 1221 to creation area 1222. In the example of Fig. 13(b), virtual advertisement AA1234, virtual advertisement BB1235, and virtual advertisement CC1236 are selected as the virtual advertisement set.

[0105] Then, when the user presses create button 1225, the created virtual advertisement set 1237 is displayed in confirmation area 1223, as shown in Fig. 13(c) . In the example of Fig. 13(c) , virtual advertisement set 1237 is confirmed to display virtual advertisement AA 1234, virtual advertisement BB 1235, and virtual advertisement CC 1236 in this order.

[0106] Returning to the explanation of FIG. 12 . As shown in FIG. 12( c), UI 1240 for setting a virtual advertisement schedule is a UI for specifying a schedule for displaying virtual advertisements on a virtual viewpoint image. UI 1240 for setting a virtual advertisement schedule has a virtual advertisement set specification area 1241 and a virtual advertisement schedule specification area 1242. In virtual advertisement set specification area 1241, a list of selectable virtual advertisement sets is displayed. In the example of FIG. 12( c), specification area 1241 shows a state in which one virtual advertisement set out of two or more types of virtual advertisement sets has been selected. In virtual schedule specification area 1242, boxes for setting the order in which the virtual advertisements included in the virtual advertisement set are to be displayed and the update unit in terms of the number of frames, and a pull-down menu for setting whether or not to perform loop playback are displayed are displayed. 12(c), in the designation area 1242, the top row of the left column shows a virtual advertisement set to be displayed in the order of virtual advertisement AA, virtual advertisement BB, and virtual advertisement CC, and the middle row of the left column shows a virtual advertisement set to be displayed in the order of virtual advertisement AA and virtual advertisement BB. The right column also shows that one frame is set as the update unit and loop playback is set as the loop playback. When the display conditions of the virtual advertisement (virtual advertisement set) are set by a user operation on the virtual advertisement schedule setting UI 1240, the display condition setting unit 115 sets the display conditions of the virtual advertisement as parameters.

[0107] In this way, by performing a user operation on the virtual advertisement schedule setting UI 1240, the display condition setting unit 115 can perform processing to set a display schedule for a virtual advertisement. That is, the display condition setting unit 115 can select a virtual advertisement to be displayed in a virtual viewpoint image from the virtual advertisement data registered by the virtual advertisement setting unit 112, and schedule the display time, display interval, display order, number of times of display, etc. of the selected virtual advertisement. The identification number of the virtual advertisement data set in the schedule by the display condition setting unit 115 is set in parameters in which information regarding a display instruction for a virtual advertisement is set, along with virtual viewpoint information and a time code, in accordance with this schedule. The virtual viewpoint information includes the position of the virtual camera (virtual viewpoint), the attitude (direction) of the virtual camera (virtual viewpoint), and the angle of view corresponding to the virtual camera. Then, the parameter transmission unit 118 transmits the parameters in which various information has been set by the parameter setting unit 114 to the image generation device 120.

[0108] As described above, by registering multiple types of virtual advertisement data in advance and setting the identification information and information regarding the display conditions of the multiple types of virtual advertisement data as parameters, it is possible to display the desired type of virtual advertisement in the virtual viewpoint image at the desired timing.

[0109] In the example of virtual advertisement designation shown in FIG. 12(b), it is possible to display a virtual advertisement such that the position of the advertising signboard moves from the left position to the right position in the image.

[0110] Furthermore, by registering a plurality of types of virtual advertisement data in advance, it becomes possible to switch between and display a plurality of types of virtual advertisements.

[0111] Although the example in which a plurality of virtual advertisements with the same texture but different display positions are sequentially displayed on the virtual viewpoint image has been described, the present invention is not limited to this. For example, a plurality of virtual advertisements with different textures but the same display position, or a plurality of virtual advertisements with different textures but different display positions may be sequentially displayed on the virtual viewpoint image.

[0112] [Third embodiment] In this embodiment, other examples of virtual advertisement images will be described using drawings. FIG. 14 is a diagram showing an example of an image of virtual advertisement data displayed in a virtual viewpoint image. FIG. 14(a) shows a case where the texture of the virtual advertisement is right-justified, FIG. 14(b) shows a case where the texture of the virtual advertisement is center-justified, and FIG. 14(c) shows a case where the texture of the virtual advertisement is left-justified. A virtual advertisement 1401 shown in FIG. 14(a) is created as computer graphics from image data of an advertising billboard that does not exist in the virtual viewpoint image (basic 3D data) to which the advertisement is to be added. In the virtual advertisement 1401, the texture "Abcde" is written (displayed) right-justified on the advertising billboard. A virtual advertisement 1402 shown in FIG. 14(b) is created as computer graphics from image data of an advertising billboard that does not exist in the virtual viewpoint image (basic 3D data) to which the advertisement is to be added. In the virtual advertisement 1402, the texture "Abcde" is written (displayed) center-justified on the advertising billboard. The virtual advertisement 1403 shown in Figure 14(c) is created as computer graphics from image data of an advertising billboard that does not exist in the virtual viewpoint image (basic 3D data) to which it is to be added. In the virtual advertisement 1403, the texture "Abcde" is written (displayed) left-justified on the advertising billboard. In other words, the above-mentioned virtual advertisements 1401, 1402, and 1403 show an example in which the 3D model representing the virtual advertising billboard is the same, but only the writing position (display position) of the texture is different.

[0113] In the case of this embodiment, too, the following effects can be obtained by using a sequence similar to that shown in Fig. 10. That is, by registering the data of virtual advertisements 1401, 1402, and 1403 in the image generating device 120 and setting display conditions, including no virtual advertisement, for switching and displaying the desired virtual advertisement as a parameter, it is possible to switch between the virtual advertisements to be displayed. Furthermore, in this embodiment, since the textures of virtual advertisements 1401 to 1403 are present near the right side, center, and left side, by switching the display in the order of virtual advertisement 1401, virtual advertisement 1402, and virtual advertisement 1403, it is possible to display the advertisement content in a manner that moves.

[0114] As described above, according to this embodiment, by setting the display conditions of the virtual advertisement as parameters through user operation on the information processing device 110, it becomes possible to display or hide the virtual advertisement and to switch between different types of virtual advertisements.

[0115] [Fourth embodiment] In this embodiment, the time to display a virtual advertisement is set in advance as a schedule, and control is performed to switch the type of virtual advertisement displayed in the virtual viewpoint image according to the schedule without requiring immediate operation by the user. This embodiment will be explained using figures.

[0116] Regarding the information processing device included in the image processing system of this embodiment, functional units different from those of the first embodiment will be described with reference to FIG. 1(b). The display condition setting unit of this embodiment receives the virtual advertisement set by the virtual advertisement setting unit 112 and information generated in accordance with an input operation received by the operation receiving unit, as well as a time code set by the time code setting unit indicating the display time of the virtual advertisement. Next, the display condition setting unit of this embodiment associates parameters for which the display conditions of a specific virtual advertisement are set with time codes indicating the display times (display start time and display end time) of the specific virtual advertisement, and sends them to the parameter sending unit of this embodiment. When the display start time of the specific virtual advertisement indicated by the time code arrives, the parameter sending unit of this embodiment sends the corresponding parameters (presence of the specific virtual advertisement) to the parameter receiving unit 123 of the image generating device 120. Furthermore, when the display end time of the specific virtual advertisement indicated by the time code arrives, the parameter sending unit of this embodiment sends the corresponding parameters (absence of the specific virtual advertisement) to the parameter receiving unit 123 of the image generating device 120.

[0117] Furthermore, with regard to the flow of processing executed by the information processing device when a schedule is set as a display condition for the virtual advertisement described above, processing that differs from that in the first embodiment will be described with reference to FIG. 4(b).

[0118] In S413, the operation receiving unit 111 receives a user operation for inputting parameters, i.e., a user operation for inputting (registering) a schedule for the virtual advertisement to be displayed, in which the display start time and the display end time of the virtual advertisement are set.

[0119] In S414, the operation receiving unit 111 determines whether the current time is within the range from the display start time to the display end time of the virtual advertisement set in the schedule received in S413. If the determination result indicates that the current time is within the display time range of the virtual advertisement set in the schedule (YES in S414), the process proceeds to S415. On the other hand, if the determination result indicates that the current time is not within the display time range of the virtual advertisement set in the schedule (NO in S414), S415 is skipped and the process proceeds to S416.

[0120] As described above, by setting the display time of the virtual advertisement as a schedule in advance, the information processing device 110 can transmit parameters (with virtual advertisement) or parameters (without virtual advertisement) set according to the schedule to the image generating device 120.

[0121] <Virtual ad display setting UI> An example of a user interface (UI) for setting the display conditions (display time) of a virtual advertisement for a virtual viewpoint image will be described with reference to the drawings. Fig. 15 is a diagram showing an example of a UI for setting the display conditions of a virtual advertisement.

[0122] The UI 1501 for setting display conditions for a virtual advertisement has a schedule setting field 1510 for setting the time at which the virtual advertisement is to be displayed. The schedule setting field 1510 has a start time setting field 1511, an end time setting field 1512, and a display target setting field 1513. The start time setting field 1511 is an area where the time at which the display of the target virtual advertisement is to start is set by a user operation. The end time setting field 1512 is an area where the time at which the display of the target virtual advertisement is to end is set by a user operation. The display target setting field 1513 is an area where the type of virtual advertisement to be displayed in the virtual viewpoint image is set by a user operation.

[0123] When a display schedule for the virtual advertisement is set by a user operation on the display condition setting UI, the display condition setting unit 115 sets information about the display conditions of the virtual advertisement in the parameters.

[0124] For example, we will explain the case where the broadcast time of a soccer match is from 6:00 p.m. to 8:00 p.m., and the setting is made to display virtual advertisement AAA in the virtual viewpoint image from 6:00 p.m. to 7:00 p.m., and virtual advertisement BBB in the virtual viewpoint image from 7:00 p.m. to 8:00 p.m.

[0125] 15, by user operation, "18:00" is set in the start time setting field 1511, "19:00" is set in the end time setting field 1512, and "virtual advertisement AAA" is set in the display target setting field 1513. Furthermore, "19:00" is set in the start time setting field 1511, "20:00" is set in the end time setting field 1512, and "virtual advertisement BBB" is set in the display target setting field 1513.

[0126] Information about the set display conditions of the virtual advertisement is set in parameters by the display condition setting section of the parameter setting unit when the designated time arrives, and the parameters are transmitted to the image generating device 120 by the parameter transmission section.

[0127] Therefore, by setting the display schedule for virtual advertisements as described above, control is performed so that virtual advertisement AAA is displayed in the virtual viewpoint image from 6:00 p.m. to 7:00 p.m., and virtual advertisement BBB is displayed in the virtual viewpoint image from 7:00 p.m. to 8:00 p.m.

[0128] As described above, according to this embodiment, by setting the time for displaying a virtual advertisement in advance as a schedule, it becomes possible to control switching of the type of virtual advertisement to be displayed in the virtual viewpoint image according to the schedule without requiring immediate operation by the user.

[0129] In the above embodiment, a timer setting based on time is used as a display condition. However, it is also possible to set the elapsed time since the instruction to display the virtual advertisement as a display condition, in addition to the time.

[0130] Furthermore, it is also possible to use a display condition that combines a virtual advertisement display instruction setting by a timer with a virtual advertisement display instruction setting at any timing by the operator. For example, with a schedule set to display virtual advertisement AAA in the virtual viewpoint image from 6:00 PM to 7:00 PM and virtual advertisement BBB in the virtual viewpoint image from 7:00 PM to 8:00 PM, it is also possible to combine the following virtual advertisement display instruction settings. In other words, it is also possible to switch the displayed virtual advertisement to virtual advertisement A by the operator's operation only during penalty kicks.

[0131] Furthermore, the present invention can be applied to a system in which a charge is billed to the advertiser of a virtual advertisement according to the display time of the virtual advertisement set by a timer.

[0132] [Fifth embodiment] In this embodiment, a 3D model and video data of a virtual advertisement are registered in advance, and the registered video data or frame number of the video data is specified along with parameters to display a virtual advertisement with a moving texture in a virtual viewpoint image. This embodiment will be described with reference to the accompanying drawings.

[0133] An example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image will be described with reference to the drawings. FIG. 16 shows an example of virtual advertisement data registration and an example of parameters for generating a virtual viewpoint image. FIG. 16(a) shows an example of a 3D model format of virtual advertisement data transmitted at the time of registration, and FIG. 16(b) shows an example of a format of parameter (with virtual advertisement) data. As shown in FIG. 16(a), the 3D model format of virtual advertisement data specifies the file names and file sizes of the obj file, mtl file, and mp4 file. The obj file and mtl file are the same as the obj file and mtl file described with reference to FIG. 6, and therefore their description will be omitted. The mp4 file stores video data (parameters) of the 3D model of the virtual advertisement. An example of parameter (with virtual advertisement) data will now be described with reference to the drawings. In the format of parameter (with virtual advertisement) data shown in FIG. 16(b), "AAAA:FWD" representing "advertisement ID: video playback method" is specified as the virtual advertisement display instruction. Here, FWD indicates constant speed playback in the forward direction. In this way, by specifying the video playback method when setting various information for the parameters, it becomes possible to display a moving virtual advertisement.

[0134] As described above, according to this embodiment, by registering a video as virtual advertisement data in advance and setting identification information of the virtual advertisement data and information related to display conditions in parameters, the following effects can be achieved. A virtual viewpoint image with a virtual advertisement (video data) or a virtual viewpoint image without a virtual advertisement can be generated according to the information set in the parameters. Therefore, it is possible to switch between displaying and hiding a virtual advertisement in real time on a frame-by-frame basis, and to display an effective virtual advertisement suited to the scene in the virtual viewpoint image. In other words, it is possible to generate a virtual viewpoint image in which a virtual advertisement (video data) is displayed at a desired position at a desired timing.

[0135] Furthermore, in the first to fifth embodiments, advertising information (virtual advertisements) has been described as an example of additional information added to basic 3D data (video file), but the additional information is not limited to advertising information. Another example of additional information added to basic 3D data is information about players appearing in virtual viewpoint images (hereinafter referred to as player information). When player information is added to a video file, the virtual advertisement storage unit 122 stores information about the data type, the number of pieces of data, and the address of the player information database. The data type is information indicating that the additional information is information about players. The number of pieces of data is the number of pieces of player information. The information about the address of the player information database is address information for connecting to the database in which the player information is accumulated. When such address information is not stored in the virtual advertisement storage unit 122, the virtual advertisement storage unit 122 stores information such as a player information ID, player information model data of the player information, player information material data, an area where the player information is pasted on the virtual viewpoint image, and priority information. As described above, this embodiment can be applied to things other than advertising information.

[0136] <Other embodiments> Note that, in the above description, the information processing device 110 is described as setting the advertisement data and its display conditions, setting information related to the display conditions in parameters, and transmitting the data to the image generating device 120. However, the present invention is not limited to this. The information processing device may be configured to read out virtual advertisement data to be added to the virtual viewpoint image from the image generating device 120, set the display conditions thereof, set information related to the display conditions in parameters, and transmit the data to the image generating device 120.

[0137] Furthermore, the object of the present invention can be achieved by providing a system with a storage medium containing computer program code for implementing the above-described functions, and having the system read and execute the computer program code. In this case, the computer program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium containing the computer program code constitutes the present disclosure. This also includes cases where an operating system (OS) running on a computer performs some or all of the actual processing based on the instructions of the program code, thereby implementing the above-described functions.

[0138] Furthermore, the image processing system of this embodiment may be realized in the following manner. That is, computer program code read from a storage medium is written to memory provided on a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on instructions in the computer program code, a CPU or the like provided on the function expansion card or function expansion unit performs some or all of the actual processing to realize the above-mentioned functions.

[0139] When the present disclosure is applied to the storage medium, the storage medium stores computer program code corresponding to the flowcharts described above.

[0140] The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0141] 111 Operation reception section 114 Parameter setting section 118 Parameter transmission unit

Claims

1. An information processing device that generates parameters for generating a virtual viewpoint image on which additional information is displayed, a setting means for setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output means for outputting the parameters set by the setting means; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein said setting means sets said parameters for each frame in which an image generating device generates said virtual viewpoint image.

3. The virtual viewpoint information is information including a position of a virtual viewpoint, a direction of the virtual viewpoint, and a field angle corresponding to the virtual viewpoint.

3. The information processing device according to claim 1, wherein the information processing device is a computer.

4. The setting means further sets a time code of the virtual viewpoint image generated by the image generating device as the parameter. The information processing device according to claim 1 , wherein the information processing device is a computer.

5. The information processing device according to claim 1 , wherein the additional information includes information about an advertisement that is displayed in a 3D model format or texture.

6. The additional information includes information about objects included in the virtual viewpoint image, which are represented in a 3D model format or texture. The information processing device according to claim 1 , wherein the information processing device is a computer.

7. The information represented by the texture includes video data.

7. The information processing device according to claim 5, wherein:

8. The display conditions of the additional information include at least one of a display time of the additional information, a display interval of the additional information, a display order of the additional information, and a display count of the additional information. The information processing device according to claim 1 , wherein the information processing device is a computer.

9. a user interface for accepting user input; The setting means sets the parameters in accordance with a user input received via the user interface. The information processing device according to claim 1 , wherein the information processing device is a computer.

10. an acquisition means for acquiring parameters in which display conditions for additional information to be displayed in a virtual viewpoint image, identification information indicating the additional information, and virtual viewpoint information of the virtual viewpoint image to be generated are set; an extraction means for extracting, from the parameters, a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information; a first reading means for reading, from among the pre-stored additional information, additional information indicated by the identification information extracted by the extraction means, based on a display condition for the additional information extracted by the extraction means; a generating means for generating a virtual viewpoint image in which the additional information is displayed, in accordance with the additional information read by the first reading means, pre-stored shape data indicating the shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted by the extracting means; An image generating device comprising:

11. 11. The image generating device according to claim 10, wherein the acquisition means acquires the parameters set for each frame for which the generation means generates the virtual viewpoint image.

12. The virtual viewpoint information is information including a position of a virtual viewpoint, a direction of the virtual viewpoint, and a field angle corresponding to the virtual viewpoint.

12. An image generating device according to claim 10 or 11.

13. the extraction means extracts the time code from the parameter acquired by the acquisition means, the parameter being a time code of the virtual viewpoint image generated by the generation means; The generating means generates a virtual viewpoint image on which the additional information is displayed in accordance with shape data of the object corresponding to the time code extracted by the extracting means, from among shape data of the object stored in advance.

13. An image generating device according to any one of claims 10 to 12.

14. 14. The image generating device according to claim 10, wherein the additional information includes information about an advertisement shown in a 3D model format or texture.

15. The additional information includes information about objects included in the virtual viewpoint image, which are represented in a 3D model format or texture.

15. An image generating device according to any one of claims 10 to 14.

16. The information represented by the texture includes video data.

16. An image generating device according to claim 14 or 15.

17. The display conditions of the additional information include at least one of a display time of the additional information, a display interval of the additional information, a display order of the additional information, and a display count of the additional information.

17. An image generating device according to any one of claims 10 to 16.

18. An information processing method for generating parameters for generating a virtual viewpoint image on which additional information is displayed, comprising: a setting step of setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output step of outputting the parameters set in the setting step; An information processing method comprising:

19. an acquisition step of acquiring parameters in which display conditions for additional information to be displayed in a virtual viewpoint image, identification information indicating the additional information, and virtual viewpoint information of the virtual viewpoint image to be generated are set; an extraction step of extracting, from the parameters, a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information; a reading step of reading, from the pre-stored additional information, additional information indicated by the identification information extracted in the extracting step, based on a display condition of the additional information extracted in the extracting step; a generating step of generating a virtual viewpoint image in which the additional information is displayed, in accordance with the additional information read in the reading step, pre-stored shape data indicating the shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted in the extracting step; An image generating method comprising:

20. A program that causes a computer to function as the information processing device according to any one of claims 1 to 9.

21. A program that causes a computer to function as the image generating device according to any one of claims 10 to 17.

22. An image processing system including: an information processing device that generates parameters for generating a virtual viewpoint image on which additional information is displayed; and an image generating device that generates the virtual viewpoint image in accordance with the parameters generated by the information processing device, The information processing device includes: a setting means for setting, as the parameters, a display condition of the additional information, identification information indicating the additional information, and virtual viewpoint information of a virtual viewpoint image to be generated; an output means for outputting the parameters set by the setting means; have The image generating device an acquisition means for acquiring the parameters output by the output means; an extracting means for extracting a display condition of the additional information, identification information of the additional information, and the virtual viewpoint information from the parameters acquired by the acquiring means; a reading means for reading out the additional information indicated by the identification information extracted by the extracting means from the pre-stored additional information based on the display conditions of the additional information extracted by the extracting means; a generating means for generating a virtual viewpoint image in which the additional information is displayed, in accordance with the additional information read by the reading means, pre-stored shape data indicating the shape of an object included in the virtual viewpoint image, and the virtual viewpoint information extracted by the extracting means; An image processing system comprising:

Images